WatchDirectory home page
WatchDirectory Startseite (Deutsche Version)
Site WatchDirectory (Français)
  Welcome, Guest. Please Login or Register
YaBB - Yet another Bulletin Board
   
  HomeHelpSearchLoginRegister  
 
Page Index Toggle Pages: 1
Could I be Overloading it? (Read 6238 times)
ian
YaBB Newbies
*
Offline



Posts: 9
Could I be Overloading it?
Feb 3rd, 2006 at 6:57pm
 
Hi there, I need some assistance figuring out a strange problem with a watchdirectory task that i have set up.  I am monitoring a folder on a local disk, asking WD to make a copy of any files that are new or change to a network drive.  

The strange thing is that it does this for 90% of files, but there are some files or folders are missing from the network drive.

It is not a permissions issue, all files are always written with EVERYONE readwrite.  

The folder i am monitoring is about 60 gig made up of tens of thousands of files and hundreds of directories.   is there a limit?

Thanks
Ian
Back to top
 
 
IP Logged
 
ian
YaBB Newbies
*
Offline



Posts: 9
Re: Could I be Overloading it?
Reply #1 - Feb 3rd, 2006 at 7:35pm
 
Ok, after some more research, i may be onto something

If i stop the job, create some new files, change some files then restart the task, Sometimes the files copy and sometimes they dont maybe that is where the discrepancy between the two folders is coming in.

This is very confusing to troubleshoot. 

I tried turning on the "when starting up check for missed events" and i can see it in the debug log going through all the directories, but it never does any copies.  

If i turn on "filenew" it gets nasty because then it leaves all the renamed backup copies of files all over the place which is undesirable, but it does get these new files copied over.

This tells me it is not a problem with trigger masks (set to *)..... ????

If the task is running and i add/change a file, it is copied immeditately.

Any suggestions?
Thanks
Ian
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Could I be Overloading it?
Reply #2 - Feb 4th, 2006 at 8:16am
 
When you use the fast monitoring method there is no practical limit for the directory size. "Fast" doesn't scan the directory tree, but receives notifications from Windows for all changes.

The other 2 methods (polling and default) work by scanning the directory tree and comparing the results to the previous scan. These 2 methods can have trouble with large directories (they need a bit of memory for each file/directory present).

The check for missed events and "fire filenew at startup" options always scan the directory tree even if you use the "Fast" monitoring method. Note that the "missed events" option will fire events for those files with a creation/modification date later than the date/time the previous watchDirectory run stopped. This date/time is stored inside
C:\Documents and Settings\All Users\Application Data\watchDirectory\TASKNAME\synchpoint.txt

If you enable the "write debug messages" option, you should be able to search the task history database (use Ctrl+F to find the file you are interested in). When "debug" is enabled, the "missed events" option will write messages like:
ThreadSyncPoint found C:\Temp\TheFile.txt
The above message doesn't say it will fire an event for this file, just that it is "investigating" this file.
If this file (or directory) is added to the queue of files to fire an event for (start the auto copy task), it will be followed by a message like:
added as filenew: C:\Temp\TheFile.txt

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
ian
YaBB Newbies
*
Offline



Posts: 9
Re: Could I be Overloading it?
Reply #3 - Feb 6th, 2006 at 4:08pm
 
Hi there, i am using the FAST monitoring method to monitor a local drive and send the files to a network drive.

I have enabled debug again, and when i start the task it runs the "check for missed events". 

Nither of the messages displayed above appear.
When i look at the log, and search for one of the folders that has not been copied, i see the following in the MESSAGE TEXT column: c:\directory\directory\directory\file.txt 1/20/2006 10:00:00 AM

The date shown is previous to the last run of the job .  I get no mention of "ThreadSyncPoint found " or "added as filenew".

The files do not copy.

Please assist.  Thank you
Back to top
 
 
IP Logged
 
ian
YaBB Newbies
*
Offline



Posts: 9
Re: Could I be Overloading it?
Reply #4 - Feb 6th, 2006 at 4:09pm
 
interestingly enough, as i spent 10 minutes typing that last email, the job has stopped.
The final writes to the log file is

"SyncPoint ThreadID: 0xD4C -- Stopping"
"TSP: ending"



Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Could I be Overloading it?
Reply #5 - Feb 6th, 2006 at 4:19pm
 
ian wrote on Feb 6th, 2006 at 4:08pm:
I have enabled debug again, and when i start the task it runs the "check for missed events". 

Nither of the messages displayed above appear.

I forgot to mention, you should make sure that the "D" button is pressed - see picture
Back to top
 

diwe.jpg (3 KB | )
diwe.jpg

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
ian
YaBB Newbies
*
Offline



Posts: 9
Re: Could I be Overloading it?
Reply #6 - Feb 6th, 2006 at 4:22pm
 
Thanks for your response, the D button was pressed all along.
Thanks
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Could I be Overloading it?
Reply #7 - Feb 6th, 2006 at 4:23pm
 
ian wrote on Feb 6th, 2006 at 4:09pm:
interestingly enough, as i spent 10 minutes typing that last email, the job has stopped.
The final writes to the log file is

"SyncPoint ThreadID: 0xD4C -- Stopping"
"TSP: ending"


The synchpoint thread is just one of many threads in the program. TSP and SyncPoint both refer to the code that runs at startup to "check for missed events", and "fire filenew/dirnew at startup".

If you can not figure it out, please send me your taskfile and history database (ZIPPED!):
taskfile: <task directory>\TASKNAME.config
database: <task directory>\TASKNAME\history.db
replace TASKNAME by the name you gave the task.
you can explore the <task directory> by pressing Ctrl+9 inside the Control Center.

Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Could I be Overloading it?
Reply #8 - Feb 6th, 2006 at 4:25pm
 
ian wrote on Feb 6th, 2006 at 4:22pm:
Thanks for your response, the D button was pressed all along.
Thanks

If it is pressed, you should see messages for each file it is checking.
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
ian
YaBB Newbies
*
Offline



Posts: 9
Re: Could I be Overloading it?
Reply #9 - Feb 6th, 2006 at 4:26pm
 
i do see each message as it is checking each file, but it is not preceeded by anything other than the directory name.

The files only exist on the source, not the destination, and are never copied.
Back to top
 
 
IP Logged
 
ian
YaBB Newbies
*
Offline



Posts: 9
Re: Could I be Overloading it?
Reply #10 - Feb 6th, 2006 at 4:29pm
 
Files have been sent.
Thanks
Back to top
 
 
IP Logged
 
Page Index Toggle Pages: 1