WatchDirectory home page
WatchDirectory Startseite (Deutsche Version)
Site WatchDirectory (Français)
  Welcome, Guest. Please Login or Register
YaBB - Yet another Bulletin Board
   
  HomeHelpSearchLoginRegister  
 
Page Index Toggle Pages: 1
Wait for files is not.... waiting for files (Read 8079 times)
andrewb
YaBB Newbies
*
Offline



Posts: 9
Wait for files is not.... waiting for files
Feb 25th, 2008 at 9:10pm
 
Hello,

New professional version license holder here.  Love the program as it allows us to automate some mundane tasks.  I've noticed a couple issues that I'll post here separately.

I have a task set up to run a program on new files when they appear in a network directory.  I have the task set to wait for files (both methods) as the file size method didn't seem to do anything.  I also put in a 30 second delay to try to make sure that the files were done writing before the program acts on it.  Problem is that the program is still telling me that the file is empty when it's being asked to act on it.  If I try the exact same command line that WD is trying on the file a couple minutes later it always works fine.  It would appear that WD is jumping the gun and that the 30 second wait is only occasionally enough time for the file to actually be complete and readable.

Any thoughts?
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Wait for files is not.... waiting for files
Reply #1 - Feb 26th, 2008 at 7:13am
 
Hello Andrew,

It seems you have set it up correctly (wait-for-files combined with a long enough delay period). The way it works is that WD will check the file-size (and/or lock-mode) of the file every "delay period" seconds. As soon as WD determines the size doesn't change anymore, it will start your task for the "new file event".
Now, for this to work properly, there are several factors that come into it....

For example, some program that create a file will pre-allocate all disk-storage before writing to the file (the file is immediatly 500MB upon creation, but no data is actually written yet). In this case the "file size" option will not really help.

Another problem is that some programs (mostly FTP servers) create files like this:

create file
write a few bytes
close file
... few milliseconds or seconds delay ...
open file
write a few more bytes
close file
... few milliseconds or seconds delay ...
open file
write a few more bytes
close file
... few milliseconds or seconds delay ...

and so on... In this case, the lock-mode option can start your task to early - when WD checks if it can open the file, it could be checking during the delay where the file is closed.

If you have such a problematic setup, maybe you can use the "File Age Monitoring Method" instead, and let it trigger when the file is 1 minute old (based on Create and Last Write date/time), I think that should work.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
andrewb
YaBB Newbies
*
Offline



Posts: 9
Re: Wait for files is not.... waiting for files
Reply #2 - Feb 26th, 2008 at 5:56pm
 
Thanks for the details.  Again, what you said makes perfect sense.

I looked into what's happening with the file writing and discovered that it's quite a bit more complicated than I thought.  Maybe you have an idea of how to deal with this:

I have a program processing a video file from one directory and putting the result file into a directory that WD is watching.  What happens is that as soon as the processing on the input file begins a file named input.ext and input.exttemp are written into the watch directory.  input.ext is created with 0 bytes and input.exttemp is written into as the processing is done.  When the processing is done input.exttemp is then copied into input.ext.  When the file is done copying the input.exttemp file is deleted and THAT'S when processing should begin on input.ext

Any ideas?
Back to top
 
 
IP Logged
 
eric
YaBB Newbies
*
Offline



Posts: 3
Re: Wait for files is not.... waiting for files
Reply #3 - Feb 26th, 2008 at 10:19pm
 
Did you try the Events, FILEDEL trigger? 

Seems like that would work if you trigger on the .exttemp file - but then you have to change the filename from .exttemp to .ext some how to do your operation on the .ext file.

just from a fellow user here...
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Wait for files is not.... waiting for files
Reply #4 - Feb 27th, 2008 at 5:48am
 
Ah, that explains it.
The FILEDEL option is a good idea as well.

Let WD monitor for deleted files with a exttemp extension. Now configure the "run any program" plugin as the attached picture.
I have highlighted the most important part, the parameter to pass to the program. It uses the following variables:

%WD_FILE_D% - the directory where the deleted file is found
%WD_FILE_B% - the file title (this will be FileName for the file FileName.exttemp)
ext - a hardcoded file extension

So, when the file C:\Monitor\Hello.exttemp is deleted, the program will run for C:\Monitor\Hello.ext

Gert
Back to top
 

sap.png (7 KB | )
sap.png

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
andrewb
YaBB Newbies
*
Offline



Posts: 9
Re: Wait for files is not.... waiting for files
Reply #5 - Feb 27th, 2008 at 4:30pm
 
Excellent work.  I set it up as you suggested and it's working well.  Thanks very much for your help.
Back to top
 
 
IP Logged
 
andrewb
YaBB Newbies
*
Offline



Posts: 9
Re: Wait for files is not.... waiting for files
Reply #6 - Feb 29th, 2008 at 6:07pm
 
I thought it was working well but for some reason it's missing files now and again.  As it's critical that it doesn't I need to make it more robust.  I tried another method of recognizing the files by having it trigger on FILECHANGE only - not FILENEW - and wait until the file size has stopped changing and/or it's available to read.  This seems to work as well as the method you explained with FILEDEL but both methods will sometimes not trigger and I don't know why.  I was using the POLL monitoring method - the directory being monitored is on a network - and have tried the default method as well with the same results.  I'm sure that there's a combination of settings that will make this work every time... I just don't know what they are...

Andrew
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Wait for files is not.... waiting for files
Reply #7 - Mar 1st, 2008 at 7:37am
 
Quote:
...for some reason it's missing files now and again...

There are 2 possibilities here:
1. WD does not detect the file
2. WD does detect the file and calls your program, but the program doesn't properly process the file

Lets first determine if WD detected the file.
Right-click your task and select "View History"
  this opens the history viewer on the page with messages for this task
double click any of the first messages to open the next window with all detected files

Do you see the file there?
Note that you can press CTRL+F to open the "Find" dialog where you can enter part of the filename.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
andrewb
YaBB Newbies
*
Offline



Posts: 9
Re: Wait for files is not.... waiting for files
Reply #8 - Mar 2nd, 2008 at 4:03am
 
I checked the history and the issue was that it wasn't seeing the files.  I changed the monitor method back to polling every 5 seconds and it seems to be okay now.

Two new things though:  With small files they are written so quickly that polling every 5 seconds isn't enough to catch the change.  I'd have to write some sort of catch-all to watch the same directory and if the files are there and readable after 15 minutes or something then they're done being processed.  Is there any problem having two (or more) watches watch the same directory as long as they don't act on the same events?

Secondly, is there any way (this may be a feature request) to filter files by file size?  Another option for achieving what I'm trying to is to have new files with the wanted extension ignored until they are bigger than 0 bytes.

Thanks.
Andrew
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Wait for files is not.... waiting for files
Reply #9 - Mar 4th, 2008 at 8:47am
 
What you can do instead is use the File Age monitoring method on the ext file (not the exttemp file).
Use the option to fire for files older than 5 minutes (latest of creation/modified), and  don't select the option to "remember triggered files". This will fire for the file every time the directory is polled. The trick will be to use another task:
Automatically Start a BAT file - http://www.watchdirectory.net/wdhelp/plugins/wdopAutoRunBat.html

Let the bat file first check if there is a corresponding exttemp file, in that case it should stop (your conversion process is still running).
The bat file would look like this:

Code:
rem -- check if there is an exttemp file present. if it is - quit the script
SET CHK=%WD_FILE_D%%WD_FILE_B%.exttemp
IF EXIST "%CHK%" GOTO :EOF
rem -- exttemp not present, continue
"C:\Program Files\Processor\ConvertIt.exe" "%WD_FILE%"
rem delete (or move) the original file
DEL "%WD_FILE%"
 



You obviously need to change "C:\Program Files\Processor\ConvertIt.exe" to the real program you use. The trick here is that the script will also remove the original file
to prevent WD to call the script again. You can also move the file to another directory, instead of deleting it, for example:
MOVE "%WD_FILE%" "D:\Backups"

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Wait for files is not.... waiting for files
Reply #10 - Mar 4th, 2008 at 1:09pm
 
one thing I just thought of....
If the converter you call is a "real" windows program (not a console program), you should call it like this:

Code:
start "converter" /min /wait "C:\Program Files\Processor\ConvertIt.exe" "%WD_FILE%"
 



because otherwise control is immediately returned to your batchscript, causing it to delete the file before your converter is done with it.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
Page Index Toggle Pages: 1