WatchDirectory Forum
http://www.watchdirectory.net/cgi-bin/yabb25/YaBB.pl
watchDirectory forums >> Bug Reports >> WD jobs occasionally stop running
http://www.watchdirectory.net/cgi-bin/yabb25/YaBB.pl?num=1345560896

Message started by coop on Aug 21st, 2012 at 4:54pm

Title: WD jobs occasionally stop running
Post by coop on Aug 21st, 2012 at 4:54pm
Not sure if this is a bug as such, but we have WD running on a server which monitors various folders (all local to the server but entered as UNC paths) performing a variety of tasks - moving files, zipping, renaming, FTPing and so on.

We've been using WD for around 15 months and up until a few months ago never had any issues, but recently we are getting occasions where certain jobs just stop working - the control centre says they are still running but they stop performing the expected actions.

Each week we will have these odd jobs failing to run but it won't necessarily be the same ones on the same day, and while some stop working others will carry on running fine! We have raised this with our network support people and they say there are no server errors that may explain this.

We move a lot of files every day (circa 6000) and I am wondering if these problems may be due to the history log being too large? Is there anything we can do to clear it down, or is there some way to run diagnostics on the program to see what is happening when the jobs stop working?

Any thoughts would be appreciated as currently we are having to check everything daily and restart the jobs regularly to ensure they carry on working.

Cheers

Mat

Title: Re: WD jobs occasionally stop running
Post by Dirk on Aug 21st, 2012 at 5:00pm
Which OS are you using and are those jobs running as windows service? If so, under which user account?

Title: Re: WD jobs occasionally stop running
Post by Gert on Aug 21st, 2012 at 5:02pm
Hello Mat,

That sounds strange, especially because you say those are local drives. Any reason why you use UNC names (not that it matters for local drives)?

To keep the history db small (should not matter if you have enough disk space), open the menu
View -> Options
and select the tab History.

Set it to (for example) 2 days. Leave "how much" set to Max.
When tasks are restarted they will cleanup.

Is there a task that has this problem often?
Please follow this blog for that task (enable debug and run the task): http://blog-en.gdpsoftware.com/2010/06/send-task-backup-to-support.html

When the problem happens again, email me the backup.

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 23rd, 2012 at 10:23am

Dirk wrote on Aug 21st, 2012 at 5:00pm:
Which OS are you using and are those jobs running as windows service? If so, under which user account?


The OS is Windows Server 2007 SP1. The jobs are running as a Windows Service under a user called 'Network Service'.

I'm not sure why the jobs were set up to use the UNC paths when the directories are all on the server that the application is installed on - I think this is something we could change if it is identified as a cause of the problems.

I'll follow the blog instructions for a couple of the jobs and provide an update next time they fail to run.

Thanks for the feedback so far.

Title: Re: WD jobs occasionally stop running
Post by Dirk on Aug 23rd, 2012 at 10:27am
Just to clarify:
there is no Windows Server 2007 - I assume you mean Windows 2008 Server.

As default all windows services are running under "Local System". Have you changed this manually?

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 23rd, 2012 at 10:32am
Sorry, yes I meant 2008 - typo!

I didn't set the jobs up initially so I'm not sure whether this change was an intentional one. What impact might it have to run these under the Netwrok Service account rather than the Local System?

Title: Re: WD jobs occasionally stop running
Post by Dirk on Aug 23rd, 2012 at 10:40am
It should run fine if the user "network service" have sufficient rights to all folder that are involved it the task (watched folder, target folder, ..).
How many tasks have you running as windows service under this account? I asked because there is an undocumented windows limitation how many services can run under the same user account - which can end up in the problems you saw.

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 23rd, 2012 at 10:46am
There are about 35 services running under that one user, of which around 20 are WatchDirectory jobs.

Title: Re: WD jobs occasionally stop running
Post by Dirk on Aug 23rd, 2012 at 10:53am
That is a lot.
Even if Windows 2008 handles this problem much more better than W2003 did, you can read here about the issue: http://blogs.msdn.com/ntdebugging/archive/2007/01/04/desktop-heap-overview.aspx

I cannot say that this *is* the problem you have, but it could. So if it happens again, please create a new domain user with sufficient rights and let some of the WatchDirectory services (~10) run under this new account and check if the problem went away.

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 23rd, 2012 at 10:58am
Unfortunately I don't have sufficient access to create new users, but if I reverted all the WD jobs to Local System and changed the UNC paths to local paths, do you think this would help?

Title: Re: WD jobs occasionally stop running
Post by Dirk on Aug 23rd, 2012 at 11:06am
You can *only* watch local folders when you use "Local System" user for the service accounts (and also only copy to local folders). The User "Local System" has no rights on remote systems. This could be a cause why someone has changed it.
At all it will not help - also "Local System" can run into this problem. The solution is to split the services into different accouts. Before you do so - just look at a few days if it happens again.

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 23rd, 2012 at 3:19pm
OK, what i think we'll do is wait for it to fail again and send the task backup to Gert for review, and if he can see that there are no other issues then we'll get some additional domain accounts set up and share the WD jobs between them - I'd rather not do this until we're sure there are no other explanations, even though I agree that seems the most likely reason.

Thanks!

Title: Re: WD jobs occasionally stop running
Post by Gert on Aug 23rd, 2012 at 3:26pm
Yes, that's a good idea.

By the way. If it is the number of tasks set to run as a Service you would be seeing errors like
StartService fails: Error Code:1053
Description: The service did not respond to the start or control request in a timely fashion.

Anyway what is the cause, it is always a good idea to reduce the number of tasks to lighten the burden on the computer. See this post for some ideas:
http://blog-en.gdpsoftware.com/2009/05/how-many-tasks-does-watchdirectory.html

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 24th, 2012 at 10:55am
Some of the jobs failed to run this morning, so I have emailed you the backup of one of the two that I put the extra debug info setting on.

The other one (an FTP job that sends over 2000 files a day) created a 72Mb zip file which I cannot email. Inside this file was a history.db that was over 730Mb!

I hope this helps and I look forward to seeing what you can find.

Title: Re: WD jobs occasionally stop running
Post by Gert on Aug 24th, 2012 at 11:13am
Not much to see in history (no errors or warnings).

The task monitors \\larnaca\Sunlight_File_Share\live\outbound and moves/renames them to \\larnaca\Sunlight_File_share\live\outbound\ftp\

You mentioned earlier those directories are actually local directories (on C: or such).

Can you please make the following changes to this task:
- use the actual local directory names (C:\.....)
- change the monitoring method to "Fast"

About the ftp job - debug mode writes a lot of data and especially "write FTP conversation to history". Lets first concentrate on this rename task.

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 24th, 2012 at 4:26pm
OK, I've modified the job to use local paths and Fast monitoring as advised.

I've left the additional debugging option unchecked - should I switch it on?

The job should run over the weekend so we should have some feedback on Monday...

Title: Re: WD jobs occasionally stop running
Post by Dirk on Aug 24th, 2012 at 4:30pm
yes, switch it on, otherwise we won't see something.

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 24th, 2012 at 4:32pm
OK, I've done that. Will update again on Monday...

Title: Re: WD jobs occasionally stop running
Post by coop on Aug 28th, 2012 at 11:08am
The changes to the job seem to ahve fixed the issues we were experiencing. We do however want to monitor this for a few more days to be sure it isn't a coincidence.

I will update again early next week or sooner if there are any issues, but thanks for now.

WatchDirectory Forum » Powered by YaBB 2.5.2!
YaBB Forum Software © 2000-2017. All Rights Reserved.