WatchDirectory Forum
watchDirectory forums >> How to... >> File/Folder copy leaves empty folders behind.

Message started by rsistla on Mar 23rd, 2012 at 7:02pm

Title: File/Folder copy leaves empty folders behind.
Post by rsistla on Mar 23rd, 2012 at 7:02pm
We are evaluating the product and very pleased with it, just does exactly what we wanted. However before ordering licenses i would like to get a bug resolved.

We are planning to move selected files and folders from a file share on one file server (SOURCESRV) in DOMAIN-A to a Target Server (DESTSRV) in DOMAIN-B

The Directory structure on source (\\SOURCESRV\SHARE) is provided in the screenshot, what we want to achieve is this.

The last level of the directory structure has 3 folders
1. Untested
2. QAed
3. Certified

As per the process
1. New requests for changes come in the form of folders with files and documents required for implementation, Users in the domain populate the "UnTested" folder with such change requests.
2. Developers promote (move these folders) these requests to "QA'ed" folder when they are being tested before implementation
3. Further the changes that are fully tested are moved into "Certified" folder (ready for production implementation)

Now the requirement.

When any folders (files are contained within) are moved from "QAed" folder into "Certified" folder they should be moved to a new server for diskspace management purpose.

The problem !!
We have set the monitoring at source (\\SOURCESRV\SHARE) level and monitoring all subdirectories for changes, when new files/folders are added to any of the "Certified" folders they need to be moved (copied and deleted from source) to target (\\DESTSRV\DESTSHARE).  We are seeing the following.

1. All files copied to "Certified" folder are properly getting moved maintaining the folder structure.
2. But all folders on the source are left without getting removed (empty folder structure is left on Source)

This is causing some issues, can we have a resolution for this in terms of settings required for the task???
Quick help is appreciated

Thanks and sorry for the lengthy detail.

FolderStructure.JPG (27 KB | )

Title: Re: File/Folder copy leaves empty folders behind.
Post by rsistla on Mar 23rd, 2012 at 7:04pm
Also the Remove empty directories setting is checked in the Task configuration

Title: Re: File/Folder copy leaves empty folders behind.
Post by Gert on Mar 23rd, 2012 at 10:14pm

Actually this may be hard to accomplish in one task (I assume you use the "copy files" task). Your setting to remove the directory if empty is only "tried once", each time a file is moved (or copied/deleted). If there is a nested folder structure below "certified", I think you will see only the deepest folders get removed, depending on the order the files are moved.

The real problem is caused by the fact that "copy files" is created just for that: to copy files. The "remove empty directory" option was added as an afterthought (feature request) and did not have those nested subfolders in mind.

I think what you need is a 2nd task:
* Delete File task (do not set its only option to "remove empty dirs" - same issue as above).
* monitor the same directory
* on "Events" set it to only trigger "DIRNEW" (only directories)
* monitoring method: File Age, make sure NOT to select the option "remember triggered files", let it trigger (for example) for files/dirs older than 30 minutes
* Filter events - see picture below
"when filename equals certified do NOT process the file/dir"
"when directory contains "\certified" DO process the file/dir"
make sure to set it to "all other files/dirs are ignored"

I think that should work.
evts.png (17 KB | )

Title: Re: File/Folder copy leaves empty folders behind.
Post by ggbrown on Mar 26th, 2012 at 2:32pm
this appears to have worked with some additional tweaking around trigger times,  folder ages and directory polling.
We are running this migration realtime, ie the move of source from qa'd to certified is expected to result in immediate migration of this source to our off server storage,  so our source folder age is going to be very 'Young'.)

The subsequent task for cleardown of the tree structure now effectively happens in chunks..
ie A test was run against a tree of 93 folders containing 148MB data.

The intitial copy task ran successfully but left us with some 50 empty folders in our source directory tree, the subsequent cleanup task then ran a number of times resulting in a gradually staged cleardown of these empty directories, but does now result in complete clearance of the transient source folders.

Happy customer..thanks!

Title: Re: File/Folder copy leaves empty folders behind.
Post by rsistla on Mar 27th, 2012 at 11:46am
Thanks Gert, Appreciate your swift response.

Gerry and I are in the same team.  Your solution really settles the empty folders on source issue. So far our tests have been in the same domain which works perfectly, but here is the actual solution we wanted.

However we have a major issue using the tool which is working across domains.

Source server : SOURCESRV in DOMAIN-A
Target server : DESTSRV in DOMAIN-B
Source Share : \\SOURCESRV\SHARE

We would like to install the WatchDirectory tool on Target server(DESTSRV) and configure to run it as a service which has access to Source share on the Source server.  

I can currently logon using the admin account on DESTSRV and access the source share on SOURCESRV (RW allowed) however Watch Directory does not allow the source folder to be configured for monitoring using the the "Copy files" task.

On closer inspection, V4.2.5 shows error "folder not reacheable" while configuring the task itself, while V4.6.6 lets to configure the source share but when the task runs it shows the error in the Task logs that source share is unavailable.

Is there a requirement for the file share access both ways (from SOURCESRV -> DESTSRV and DESTSRV -> SOURCESRV), currently we dont have ACLs enabled for SOURCESRV -> DESTSRV.  We need more details to make the tool work in our environment.


Title: Re: File/Folder copy leaves empty folders behind.
Post by Dirk on Mar 27th, 2012 at 12:11pm
Hi Ramesh,

do you have a transitive trust between Domain A and Domain B? Can the user under which the service is started at target server access the share on source server?

Title: Re: File/Folder copy leaves empty folders behind.
Post by rsistla on Mar 27th, 2012 at 12:36pm
Hi Dirk,

Yes the account in question can log on to the DESTSRV (Domain B) as administrator and has the access to the Domain A source share (\\SOURCESRV\SHARE).  


Title: Re: File/Folder copy leaves empty folders behind.
Post by Dirk on Mar 27th, 2012 at 1:30pm
Hi Ramesh,

just to make it clear:
WatchDirectory is installed on a server in domain B. You run the task as a windows service. You want to copy files from a server in domain A to a folder in domain B, so your watched folder is in domain A, your target folder is in domain B.

When you connect with explorer from the server in domain B to the server in domain A, are you promted for credentials? Windows is maybe caching those credentials, so you were maybe asked for the credential only the first time you connected. Please let me know if the domains have a transitive (both ways) trust relationship. Please also look at windows service panel (services.msc) and let me know under which account the service was configured to run. Is it "local system" or "\\domainA\administrator" or "\\domainB\administartor" ? You can also try to run this task "normal (F5)" (not as windows service); are you able to connect with that?

You mentioned above that all was running well when you tested it in a single domain (same domain). So it is a credential issue.
You said: "currently we dont have ACLs enabled for SOURCESRV", so I assume you have set it to "everyone -> full access" in NTFS rights settings and the same for share rights settings, correct?

Title: Re: File/Folder copy leaves empty folders behind.
Post by ggbrown on Mar 27th, 2012 at 2:05pm
dirk, just a a possible alternative approach, i read somwhere that WD can have sub tasking setup.

if we reverted to running WD on the source server and detected a suitable trigger event, is it possible to have a parent task that launches a simple .bat/vbs/exe that opens a temporary unc share access (declaring the correct \\domain\credentials) and then run the copy task as a sub task ..effectively running it in a pre-permissioned shell pointing to the declared unc target path.  Would this get us around the lack of ACl issue.?

For info, The source should be locked down so ntfs "everyone" would be readonly...thus preventing any deletion post copy so not an option.

Service will run under a \\domain\serviceaccount  There are currently no ACLS open between domains.. in Ramesh's tests he is using domainb\useraccount but with pre-cached unc access under \\domaina\useraccount

Title: Re: File/Folder copy leaves empty folders behind.
Post by Dirk on Mar 27th, 2012 at 2:30pm
Hi Gerry,

with the "run any .bat file" plugin you are free to write any script to make your login and mapping. It can also contains a 'net use' command to map a share from a different domain (note that your password is stored in clear text in this script!).

You said: 'There are currently no ACLS open between domains'. The task tries to build a connection to the remote domain by using the credentials of the service or the user who starts the Control Center and task manually. So if your service is running unter \\domainB\serviceaccount and you have *no* trust between the domains, this (service)user will not be able to connect to any share in domain A. But this you have allready seen ;-)

We have maybe a solution for that problem in our actual beta:

Just install "on top" of your current install.

stop the task (lets say it is called HELLO)
select the menu "tools ->  explore task directory" (windows explorer opens)
open the settings file named HELLO.confg with notepad
scroll to the bottom and add a section (sections) like this:


If you add multiple sections for multiple shares, increase the number ([netuse_1] ... [netuse_7]), there can be little "gaps" in this numbering. If you omit one of the entries (unc, usr, pwd) the entire section is ignored.

The above should work the same as if the following command is issued

NET USE \\delllap\share passwd /user:username /persistent:NO

No drive-letter is mapped, you need to use the UNC name in the rest of your task.

Currently the WatchDirectory Control Center has no window to maintain those settings and the password must be plain text. If we get this working, we will add a window for it and make sure the passwords are stored encrypted.

Please let us know if that helps.

Title: Re: File/Folder copy leaves empty folders behind.
Post by Kerry on May 4th, 2012 at 5:20pm
I've used WatchDir for a few years and recently had this problem of accessing a remote machine on a different domain. After reading this post, I downloaded your beta, made the [netuse_0] additions to the .config file, and it works perfectly! Thank you.


Title: Re: File/Folder copy leaves empty folders behind.
Post by Dirk on May 4th, 2012 at 7:29pm
Hi Kerry,

great to hear, thanks for the feedback!

WatchDirectory Forum » Powered by YaBB 2.5.2!
YaBB Forum Software © 2000-2017. All Rights Reserved.