WatchDirectory Forum
http://www.watchdirectory.net/cgi-bin/yabb25/YaBB.pl
watchDirectory forums >> Bug Reports >> watchDirectory hangs website and network access
http://www.watchdirectory.net/cgi-bin/yabb25/YaBB.pl?num=1283440781

Message started by mmarx82 on Sep 2nd, 2010 at 5:19pm

Title: watchDirectory hangs website and network access
Post by mmarx82 on Sep 2nd, 2010 at 5:19pm
I'm testing watchDirectory 4.6.2 on a Windows 2003 standard edition server, The server is running 2 Intel Xeon E5335 Processors with 4 GB of Ram.
Additionally the server runs IIS, FTP, WSUS 3.2 on it. I'm the only user with direct network access to it.

I've setup 1 task to run as a service, I had it doing something else but to eliminate the possiblity of the first subtask being the culprit I removed it only to find I still have the issue. So now all it's doing is what you see in the attached image.

The problem is when watchDirectory does detect the new files in the monitored Directory and then it starts to run the configured tasks we cannot access our hosted website or get access to the network drives on this server. The funny part is network and cpu utilization are less than 5-10% during this time so you would think there shouldn't be an issue but there is. After the tasks end everything is fine again. Any suggestions or need any more info?


Capture.JPG (50 KB | )

Title: Re: watchDirectory hangs website and network acces
Post by Dirk on Sep 2nd, 2010 at 8:32pm
Hi mmarx82,

can you please tell us if you see the same behavior when the task is *not* running as a service (started "normal, F5")?

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 8:42pm
Ok, that was better, I still noticed a network access hickup though it was much better (98% better).
The website though was still affected, causing navigation on the site to hang 30-60 seconds.

Title: Re: watchDirectory hangs website and network acces
Post by Dirk on Sep 2nd, 2010 at 8:53pm
is it also a terminal server? Under which user account was the task running and how much more non- WatchDirectory services are running under the same account?

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 8:54pm
No other WatchDirectory Services, logged in locally as domain admin.

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 9:02pm
I should also note that this server was completely reloaded about 2 weeks ago and is fully updated.

Title: Re: watchDirectory hangs website and network acces
Post by Dirk on Sep 2nd, 2010 at 9:06pm
I am asking under which account your service was running, not which user was logged it at server console. Please create a new domain user in AD, switch the task to run as service but do not start the task. Open you service configuration menu ( services.msc ) and search your task, its named "watchdirectory:[name of your task]". Double click it, change to the second tab and change the user account under which the task will be running from "local system" into your newly created domain user. Make sure that this user has enough rights for the things you want to do.
Start the task ( or start the service) and let me know what the performance is doing now.

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 9:16pm
The service was running under the system account, it is now running under a domain admin account doing the steps you described.

The performance was the same as the first time and I even got "not responding" on the watchDirectory Control Center app this time.

Title: Re: watchDirectory hangs website and network acces
Post by Dirk on Sep 2nd, 2010 at 9:28pm
please do not use admin account, use a dedicated new account. I cannot promise you that this will solve your problem, we need to experiment a little bit.
Is the server running as terminal server too? (i am not talking about about admin rdp access)
Note that we have hundert of installations on W2k3 servers (including my own) without a problem. We need to find out what is different at your server. Can you tell me a little bit about your hardware? Type of server, type of hhd's, controller, raid?

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 9:50pm
ok, RDP and Remote Assistance is turned off.
The server is a Gateway E-9520T
LSI MegaRAID 8308ELP RAID controller (SCSI)
Running Raid 5
Has 2 partitions (Drives C, D)
Drive C - 50 GB (39.1 GB free)
Drive D - 1.31 TB (1.07 TB free)
Has Intel Pro/1000 NIC - Driver Date: 3/26/2010 - Version: 9.13.41.0
There is a NIC driver feature that affects our 2008 servers Network Access that has to be disabled (Receive-Side Scaling), went ahead already and did that on this server even though it's not a 2008 server. Doesn't make a difference though on or off with this issue.

Using Sophos for AV, program and system firewall is off right now.

I installed watchDirectory to Drive C, but the configurations are saved to Drive D. (For backup purposes, We backup D not C)

The wmv file I'm testing with right now is 780MB.

Just tried a dedicated new account, still receiving the same issue.

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 9:54pm
No not running as terminal server either.

Title: Re: watchDirectory hangs website and network acces
Post by Dirk on Sep 2nd, 2010 at 10:10pm
thanks for your info. Actually I am out of ideas. Are you copying your wmv locally or to a network destination? If network, please try copy locally just for testing.
Please also stop virus scanner for wmv files just to see if something changes.
I am writing from Germany, it is late night here, we will continue next day.

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 2nd, 2010 at 10:21pm
the copy is a local copy

Folder Structure is

Videos
-Upload
-Videos

Anything dropped in the Upload folder will be copied/renamed (appending the date to the filename) and moved to the other videos folder. When this gets to working properly I'm going to add the convert video subtask back in there so the non-wmv files will be converted to mp4 with h264 encoding. Which btw, is a feature request of mine (converting video). I'm currently using HandbrakeCLI to do that with your "Start A Program" pluggin.

My workday is about to end anyways, then I have a 4 day weekend so back to testing on Tuesday. Thanks for trying.

Title: Re: watchDirectory hangs website and network acces
Post by Gert on Sep 3rd, 2010 at 8:12am
Hi,

I strongly suspect this is caused by your virus scanner. Most scanners allow you to exclude complete directories from scanning, so that may help.

I see you use "Copy" as the first subtask and after that "Delete".
A "Copy" will normally copy bit-by-bit to its destination (a process that may take a lot of time), however, if you select the option to "[X] Delete after copy" and "[X] Implement as move" the copy process does not copy the bits of the file, but only changes the directory entry of the file (only when moving on the same disk). If it can do a Move instead of Copy, it should take less than a millisecond.

Hope this helps a bit.

Title: Re: watchDirectory hangs website and network acces
Post by Dirk on Sep 3rd, 2010 at 8:39am
I agree with Gert about virus scanner. Please stop it for a test. Note that the scanner will maybe scan twice: on reading and on writing the file. Please also try to copy this large file manually (without Watchdirectory) and with activated scanner. You should see a similar slow performance.

You are talking about MegaRAID 8308ELP: SAS or SATA? Does your controller have "write cache" enabled?

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 7th, 2010 at 4:02pm
I can copy the file manually and the problem not occur, disabling AV didn't help. It's SAS and the write cache was disabled so I enabled it to no avail.

What has seemed to help is I unistalled watchDirectory and re-installed it, this time using the default configuration location. The issues are still there but much more tolerable. It's more of a hickup now than a hang.

I also noticed for the website that the html pages don't experience the problem it's just the asp and aspx pages. Our site is mostly asp which is why I didn't notice that before.

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 7th, 2010 at 4:13pm
Is it possible to limit the number of CPU's or cores watchDirectory runs on? I would like to try lowering it by 1.

Title: Re: watchDirectory hangs website and network acces
Post by Gert on Sep 7th, 2010 at 6:00pm
You can use the windows task manager to set the "Processor Affinity".

Start the task and right-click wdrun.exe in task manager.

Title: Re: watchDirectory hangs website and network acces
Post by mmarx82 on Sep 7th, 2010 at 6:23pm
Duh, I can't say for sure but I think that's what the doctor ordered. Right now it's fine.

Title: Re: watchDirectory hangs website and network acces
Post by Gert on Sep 7th, 2010 at 6:28pm
mmm, very strange. It sounds like your disk (raid) doesn't appreciate multiple accesses by different processes. You should run into the same issue if you run other software that does heavy disk access I guess.

If I am correct, the processor affinity set on Task Manager is NOT saved by Windows. This thread has a few solutions (I didn't try them): http://www.tomshardware.co.uk/forum/179198-10-processor-affinity-automatically

WatchDirectory Forum » Powered by YaBB 2.5.2!
YaBB Forum Software © 2000-2017. All Rights Reserved.