WatchDirectory home page
WatchDirectory Startseite (Deutsche Version)
Site WatchDirectory (Français)
  Welcome, Guest. Please Login or Register
YaBB - Yet another Bulletin Board
   
  HomeHelpSearchLoginRegister  
 
Page Index Toggle Pages: 1
Missed events (Read 5852 times)
MMaynard
YaBB Newbies
*
Offline



Posts: 5
Missed events
Jun 7th, 2011 at 7:41pm
 
We are currently on V4.6.2 using WD to do a lot of automated tasks. We do not run WD as a service. We have been using WD for about 3 years... maybe longer.

We have about 50 WD tasks on a virtual Win2008 64-bit server. Most of the directories that are being monitored are on another server accessed using UNC through DFS. All network folders are provided by an OpenFiler box.

There are other things running on this server; scheduled tasks doing things like FTP pulls and initiating CMD files. CPU utilization on this server is relatively low most of the time. WD runs 24x7 on this server.

The problem is that some of the WD tasks will terminate for no reason (icon turns red). Sometimes the tasks will stay green but will stop working. Most of these tasks are configured for the 'FAST' monitoring method but we have tried the various polling methods & intervals with no improvement.  The failures are about 1 or 2 tasks each week and there is no pattern to the tasks that fail.

About 35 of these tasks are similar and could be combined into 1 CMD file that monitors the primary subdirectory and initiates a specific CMD file based on which subdirectory received a new file. This will take some work but could be done if the number of WD tasks is the problem.  

Do you have any suggestions?
-Mike-
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Missed events
Reply #1 - Jun 8th, 2011 at 11:31am
 
Hello Mike,

50 tasks is a lot so if you can combine a few of them that will help.

Do you use the Upload Task (FTP)? The current beta (will be released today or tomorrow) has a few fixes where a task could hang (and stays green) if it failed to connect to the FTP server.

In some (network) cases the "Default" and "Fast" monitoring methods can "wait forever" without detecting anything. This can happens if the remote computer closes the handle to the directory without telling WD. The Polling and File Age methods should work fine in those cases.
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
MMaynard
YaBB Newbies
*
Offline



Posts: 5
Re: Missed events
Reply #2 - Jun 8th, 2011 at 9:31pm
 
I will change the tasks to the "polling" method for new file monitoring.  We have about 3 of the tasks that use WD to do FTP. The others initiate a CMD file and the FTP is part of those scripts.  One of the FTP tasks failed earlier this week (turned red though...)

I was hoping you would say WD could easily handle 100+ concurrent tasks. Combining them is certainly possible but a good bit of work and testing since most of these tasks have been in production for a long time.

-Mike-
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Missed events
Reply #3 - Jun 9th, 2011 at 10:23am
 
MMaynard wrote on Jun 8th, 2011 at 9:31pm:
I was hoping you would say WD could easily handle 100+ concurrent tasks.


WD can handle 6 billion tasks. It is the operating system/network that can cause issues.

Make sure to look at
http://www.watchdirectory.net/wdhelp/plugins/wdopMultiPlugin.html

This will let you combine tasks so they all use the same (only 1) "monitoring engine".
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
MMaynard
YaBB Newbies
*
Offline



Posts: 5
Re: Missed events
Reply #4 - Jun 9th, 2011 at 7:49pm
 
I will try setting these as multiple tasks. Thanks for the suggestion.
-Mike-
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Missed events
Reply #5 - Jun 9th, 2011 at 7:56pm
 
A few more tips to reduce the number of tasks are here:
http://blog-en.gdpsoftware.com/2009/05/how-many-tasks-does-watchdirectory.html

If you let a new WD task monitor a "higher" (parent) directory, your new script may want to make use of the %WD_RELDIR% variable ( http://www.watchdirectory.net/wdhelp/plugins/wdopAutoRunBatEnv.html ) to determine which subdir the script is called for.


Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
MMaynard
YaBB Newbies
*
Offline



Posts: 5
Re: Missed events
Reply #6 - Jul 8th, 2011 at 9:58pm
 
Gert,
Would it be possible to add functionality to WatchDirectory to enable *both* the FAST and the POLLING methods of detection?  We continue to struggle with WD tasks missing important files and having to manually watch the directories that WD is supposed to be watching. We moved to the polling method and things are slightly improved but not perfect yet.  There are anomalies with polling that are creating issues.

We also suspect a memory leak with WD. The VM that runs WD has grown to over 8GB and my network guy says this is highly unusual and he thinks it is WD.
-Mike-
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Missed events
Reply #7 - Jul 9th, 2011 at 8:59am
 
Quote:
The VM that runs WD has grown to over 8GB

Memory use? I agree that is highly unusual. WD is a 32bit program and can at most address 2gb memory (per task).

What you can do is look at Windows Task Manager and note the number of handles and memory of the wdrun.exe processes (each is one task). A day later check them again.

Quote:
We continue to struggle with WD tasks missing important files

Have you looked at the Task History? See http://blog-en.gdpsoftware.com/2008/05/how-task-history-works.html
In almost all cases people say WD didn't detect a file, the file is detected but the task could not "operate" on it (copy, zip, email - whatever the task is supposed to do with the file).

Can you email me a backup of a "problem task" (see http://watchdirftp.blogspot.com/2010/06/send-task-backup-to-support.html ), make sure to mention a filename WD didn't detect. Or even better: enable "debug" for the task (as described in the blogpost) and send the backup after it failed to detect the file.
Do not post the backup on this forum, email it to me and mention this forum post.
Gert
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
MMaynard
YaBB Newbies
*
Offline



Posts: 5
Re: Missed events
Reply #8 - Jul 12th, 2011 at 10:20pm
 
Gert wrote on Jul 9th, 2011 at 8:59am:
What you can do is look at Windows Task Manager and note the number of handles and memory of the wdrun.exe processes (each is one task). A day later check them again.


Quote:
Or even better: enable "debug" for the task (as described in the blogpost) and send the backup after it failed to detect the file. Do not post the backup on this forum, email it to me and mention this forum post.


I will try both of these the next time I am back working on this issue. It might be a few days.

-Mike-
Back to top
 
 
IP Logged
 
Page Index Toggle Pages: 1