WatchDirectory home page
WatchDirectory Startseite (Deutsche Version)
Site WatchDirectory (Franais)
  Welcome, Guest. Please Login or Register
YaBB - Yet another Bulletin Board
   
  HomeHelpSearchLoginRegister  
 
Page Index Toggle Pages: 1
WD jobs occasionally stop running (Read 11971 times)
coop
YaBB Newbies
*
Offline



Posts: 13
WD jobs occasionally stop running
Aug 21st, 2012 at 4:54pm
 
Not sure if this is a bug as such, but we have WD running on a server which monitors various folders (all local to the server but entered as UNC paths) performing a variety of tasks - moving files, zipping, renaming, FTPing and so on.

We've been using WD for around 15 months and up until a few months ago never had any issues, but recently we are getting occasions where certain jobs just stop working - the control centre says they are still running but they stop performing the expected actions.

Each week we will have these odd jobs failing to run but it won't necessarily be the same ones on the same day, and while some stop working others will carry on running fine! We have raised this with our network support people and they say there are no server errors that may explain this.

We move a lot of files every day (circa 6000) and I am wondering if these problems may be due to the history log being too large? Is there anything we can do to clear it down, or is there some way to run diagnostics on the program to see what is happening when the jobs stop working?

Any thoughts would be appreciated as currently we are having to check everything daily and restart the jobs regularly to ensure they carry on working.

Cheers

Mat
Back to top
 
 
IP Logged
 
Dirk
YaBB Administrator
*****
Offline



Posts: 658
South Germany
Re: WD jobs occasionally stop running
Reply #1 - Aug 21st, 2012 at 5:00pm
 
Which OS are you using and are those jobs running as windows service? If so, under which user account?
Back to top
 

Viele Gre / Best regards
Dirk - GdP Software

dirk [at] gdpsoftware [dot] com
http://blog-de.gdpsoftware.com
Webseite: http://www.gdpsoftware.com
WWW WWW  
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2225
The Netherlands
Re: WD jobs occasionally stop running
Reply #2 - Aug 21st, 2012 at 5:02pm
 
Hello Mat,

That sounds strange, especially because you say those are local drives. Any reason why you use UNC names (not that it matters for local drives)?

To keep the history db small (should not matter if you have enough disk space), open the menu
View -> Options
and select the tab History.

Set it to (for example) 2 days. Leave "how much" set to Max.
When tasks are restarted they will cleanup.

Is there a task that has this problem often?
Please follow this blog for that task (enable debug and run the task): http://blog-en.gdpsoftware.com/2010/06/send-task-backup-to-support.html

When the problem happens again, email me the backup.
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #3 - Aug 23rd, 2012 at 10:23am
 
Dirk wrote on Aug 21st, 2012 at 5:00pm:
Which OS are you using and are those jobs running as windows service? If so, under which user account?


The OS is Windows Server 2007 SP1. The jobs are running as a Windows Service under a user called 'Network Service'.

I'm not sure why the jobs were set up to use the UNC paths when the directories are all on the server that the application is installed on - I think this is something we could change if it is identified as a cause of the problems.

I'll follow the blog instructions for a couple of the jobs and provide an update next time they fail to run.

Thanks for the feedback so far.
Back to top
 
 
IP Logged
 
Dirk
YaBB Administrator
*****
Offline



Posts: 658
South Germany
Re: WD jobs occasionally stop running
Reply #4 - Aug 23rd, 2012 at 10:27am
 
Just to clarify:
there is no Windows Server 2007 - I assume you mean Windows 2008 Server.

As default all windows services are running under "Local System". Have you changed this manually?
Back to top
 

Viele Gre / Best regards
Dirk - GdP Software

dirk [at] gdpsoftware [dot] com
http://blog-de.gdpsoftware.com
Webseite: http://www.gdpsoftware.com
WWW WWW  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #5 - Aug 23rd, 2012 at 10:32am
 
Sorry, yes I meant 2008 - typo!

I didn't set the jobs up initially so I'm not sure whether this change was an intentional one. What impact might it have to run these under the Netwrok Service account rather than the Local System?
Back to top
 
 
IP Logged
 
Dirk
YaBB Administrator
*****
Offline



Posts: 658
South Germany
Re: WD jobs occasionally stop running
Reply #6 - Aug 23rd, 2012 at 10:40am
 
It should run fine if the user "network service" have sufficient rights to all folder that are involved it the task (watched folder, target folder, ..).
How many tasks have you running as windows service under this account? I asked because there is an undocumented windows limitation how many services can run under the same user account - which can end up in the problems you saw.
Back to top
 

Viele Gre / Best regards
Dirk - GdP Software

dirk [at] gdpsoftware [dot] com
http://blog-de.gdpsoftware.com
Webseite: http://www.gdpsoftware.com
WWW WWW  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #7 - Aug 23rd, 2012 at 10:46am
 
There are about 35 services running under that one user, of which around 20 are WatchDirectory jobs.
Back to top
 
 
IP Logged
 
Dirk
YaBB Administrator
*****
Offline



Posts: 658
South Germany
Re: WD jobs occasionally stop running
Reply #8 - Aug 23rd, 2012 at 10:53am
 
That is a lot.
Even if Windows 2008 handles this problem much more better than W2003 did, you can read here about the issue: http://blogs.msdn.com/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx

I cannot say that this *is* the problem you have, but it could. So if it happens again, please create a new domain user with sufficient rights and let some of the WatchDirectory services (~10) run under this new account and check if the problem went away.
Back to top
 

Viele Gre / Best regards
Dirk - GdP Software

dirk [at] gdpsoftware [dot] com
http://blog-de.gdpsoftware.com
Webseite: http://www.gdpsoftware.com
WWW WWW  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #9 - Aug 23rd, 2012 at 10:58am
 
Unfortunately I don't have sufficient access to create new users, but if I reverted all the WD jobs to Local System and changed the UNC paths to local paths, do you think this would help?
Back to top
 
 
IP Logged
 
Dirk
YaBB Administrator
*****
Offline



Posts: 658
South Germany
Re: WD jobs occasionally stop running
Reply #10 - Aug 23rd, 2012 at 11:06am
 
You can *only* watch local folders when you use "Local System" user for the service accounts (and also only copy to local folders). The User "Local System" has no rights on remote systems. This could be a cause why someone has changed it.
At all it will not help - also "Local System" can run into this problem. The solution is to split the services into different accouts. Before you do so - just look at a few days if it happens again.
Back to top
 

Viele Gre / Best regards
Dirk - GdP Software

dirk [at] gdpsoftware [dot] com
http://blog-de.gdpsoftware.com
Webseite: http://www.gdpsoftware.com
WWW WWW  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #11 - Aug 23rd, 2012 at 3:19pm
 
OK, what i think we'll do is wait for it to fail again and send the task backup to Gert for review, and if he can see that there are no other issues then we'll get some additional domain accounts set up and share the WD jobs between them - I'd rather not do this until we're sure there are no other explanations, even though I agree that seems the most likely reason.

Thanks!
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2225
The Netherlands
Re: WD jobs occasionally stop running
Reply #12 - Aug 23rd, 2012 at 3:26pm
 
Yes, that's a good idea.

By the way. If it is the number of tasks set to run as a Service you would be seeing errors like
StartService fails: Error Code:1053
Description: The service did not respond to the start or control request in a timely fashion.

Anyway what is the cause, it is always a good idea to reduce the number of tasks to lighten the burden on the computer. See this post for some ideas:
http://blog-en.gdpsoftware.com/2009/05/how-many-tasks-does-watchdirectory.html
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #13 - Aug 24th, 2012 at 10:55am
 
Some of the jobs failed to run this morning, so I have emailed you the backup of one of the two that I put the extra debug info setting on.

The other one (an FTP job that sends over 2000 files a day) created a 72Mb zip file which I cannot email. Inside this file was a history.db that was over 730Mb!

I hope this helps and I look forward to seeing what you can find.
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2225
The Netherlands
Re: WD jobs occasionally stop running
Reply #14 - Aug 24th, 2012 at 11:13am
 
Not much to see in history (no errors or warnings).

The task monitors \\larnaca\Sunlight_File_Share\live\outbound and moves/renames them to \\larnaca\Sunlight_File_share\live\outbound\ftp\

You mentioned earlier those directories are actually local directories (on C: or such).

Can you please make the following changes to this task:
- use the actual local directory names (C:\.....)
- change the monitoring method to "Fast"

About the ftp job - debug mode writes a lot of data and especially "write FTP conversation to history". Lets first concentrate on this rename task.
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #15 - Aug 24th, 2012 at 4:26pm
 
OK, I've modified the job to use local paths and Fast monitoring as advised.

I've left the additional debugging option unchecked - should I switch it on?

The job should run over the weekend so we should have some feedback on Monday...
Back to top
 
 
IP Logged
 
Dirk
YaBB Administrator
*****
Offline



Posts: 658
South Germany
Re: WD jobs occasionally stop running
Reply #16 - Aug 24th, 2012 at 4:30pm
 
yes, switch it on, otherwise we won't see something.
Back to top
 

Viele Gre / Best regards
Dirk - GdP Software

dirk [at] gdpsoftware [dot] com
http://blog-de.gdpsoftware.com
Webseite: http://www.gdpsoftware.com
WWW WWW  
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #17 - Aug 24th, 2012 at 4:32pm
 
OK, I've done that. Will update again on Monday...
Back to top
 
 
IP Logged
 
coop
YaBB Newbies
*
Offline



Posts: 13
Re: WD jobs occasionally stop running
Reply #18 - Aug 28th, 2012 at 11:08am
 
The changes to the job seem to ahve fixed the issues we were experiencing. We do however want to monitor this for a few more days to be sure it isn't a coincidence.

I will update again early next week or sooner if there are any issues, but thanks for now.
Back to top
 
 
IP Logged
 
Page Index Toggle Pages: 1