WatchDirectory home page
WatchDirectory Startseite (Deutsche Version)
Site WatchDirectory (Français)
  Welcome, Guest. Please Login or Register
YaBB - Yet another Bulletin Board
   
  HomeHelpSearchLoginRegister  
 
Page Index Toggle Pages: 1
Synchronised FTP uploads (Read 6203 times)
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Synchronised FTP uploads
Oct 27th, 2010 at 12:47pm
 
Hello,
We are using WatchDirectory to upload files (video file and metadata XML file) through FTP to publish video on a website.

Our own software exports the metadata XML and the video file to 2 separate watch-directories, which are monitored by 2 separate Watchdirectory tasks for FTP upload to 2 separate destinations.
We upload the video file to a hosting party and the corresponding metadata XML file is uploaded to another hosting party.

I want to upload the metadata XML after uploading the video file has finished. Is it possible to somehow synchronise this?
How?
Can I monitor the video watch-directory for finished FTP upload and after that, copy the corresponding XML from a directory to the other watch-directory?
How should I configure that?

Thanks for your help!
Renzo.
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Synchronised FTP uploads
Reply #1 - Oct 27th, 2010 at 1:14pm
 
Hello Renzo,

If your software can create those *.xml and (I guess) *.avi files in the same directory (or a sub-directories of the same parent directory), you can use the Subtask task ( http://www.watchdirectory.net/wdhelp/plugins/wdopMultiPlugin.html ).

Use your original FTP task for avi files as the first subtask, followed by your xml upload task as the second upload Subtask.

The first subtask (avi files) has *.AVI as "only run this subtask for files matching....", the second subtask has *.xml. Note these match-patterns match against the full path of detected files, so you can also use something like *\movies\* and *\xml\* for those patterns.
...
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Re: Synchronised FTP uploads
Reply #2 - Oct 29th, 2010 at 11:48am
 
Hello Gert,
Thanks for your help, this almost does the job.

One additional question:
It is possible that multiple exports, one export contains 1 MP4 video file and 1 XML metadata file, are placed into the watchdirectory simultaneously. After an MP4 file is uploaded Watchdirectory will upload the XML file, but if there are multiple XML's it will upload all of them?
Both files are named the same, except for the extension.

If above is the case, I have to make sure the correct XML file is uploaded after the MP4 is uploaded. can I pass the complete filename from the first task to the second?

Thanks again!
Renzo.
Back to top
 
 
IP Logged
 
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Re: Synchronised FTP uploads
Reply #3 - Oct 29th, 2010 at 3:19pm
 
Hello Gert (again),
See attached my settings for the main task, for both sub tasks and for the subtask configuration.
I thought that subtasks are executed sequentially because they are subtasks but I see that it does not wait for the mp4 to be uploaded before the xml is uploaded.
Do I need to do more to have behavior?
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Synchronised FTP uploads
Reply #4 - Oct 30th, 2010 at 8:55am
 
Hi Renzo,

Zie ik daar 2ekamergemist en debatgemist?

Anyway, I think I made a thinking error with my first reply, you are correct it won't work as I described it.
If I understand you correctly, those files are created in pairs like
123.mp4 123.xml
124.mp4 124.xml
125.mp4 125.xml
but they (the 6 files above) can be created in any order. The mp4's can be uploaded immediately, as they arrive. However, the corresponding xml may only be uploaded after its mp4 is completely uploaded. Correct?

I think I know how to implement this, but it requires a little batch scripting (I will do that for you).

Can you confirm the above description is correct?
Am I right those "pairs of files" are in the same directory (c:\whatever\123.mp4 and c:\whatever\123.xml) ?

Feel free to email me in Dutch, if you want.
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Re: Synchronised FTP uploads
Reply #5 - Oct 30th, 2010 at 9:58am
 
Hallo Gert,
Dan in het Nederlands  Smiley.

Je veronderstelling klopt, het zijn pairs van mp4 en xml en ze worden inmiddels in dezelfde directory geplaatst. Het gaat om de Tweede Kamer inderdaad, waar we debatten opnemen en publiceren op het internet.
Je kunt je voorstellen dat wanneer een lang debat (3 uur of zo) wordt gevolgd door een korte dat terwijl het lange debat nog wordt geFTPed er al weer een nieuw debat klaar staat (mp4 met XML). Wanneer de video dan is geFTPed dan moet alleen het bijbehorende xml bestand worden geupload.
Daarna het volgende pair, enz.

"The mp4's can be uploaded immediately, as they arrive. However, the corresponding xml may only be uploaded after its mp4 is completely uploaded. Correct?"
Dit klopt dus inderdaad.

Misschien wel leuk, debatgemist.tweedekamer.nl zal binnenkort voor het publiek toegankelijk worden. WatchDirectory Professional wordt dan dus gebruikt voor uploads naar de hosting partijen voor video enerzijds en metadata anderzijds.

Bedankt maar weer!
Groeten Renzo.
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Synchronised FTP uploads
Reply #6 - Oct 30th, 2010 at 10:24am
 
I actually meant you could email me in Dutch, I would like to keep the forum English.
Renzo's answer above basically said my assumptions are correct.

The way I think this can be implemented:

Task 1, using the Subtask plugin, monitoring C:\Video for MP4 files (NOT xml files, but they are still created there).
SubTask 1.1 - Your current subtask that uploads the MP4
SubTask 1.2 - The "Run Any Program" subtask ( http://www.watchdirectory.net/wdhelp/plugins/wdopAutoLaunch.html ), configured to run CMD.EXE which starts a bat file

Subtask 1.2 runs a bat file and this bat file has all file information for the detected MP4 file (not the xml file). The bat file just waits until the corresponding XML is found (I guess it will be present already in 99.99% of cases) and when found it copies (or moves) this XML to a totally different directory (C:\XML).
Let "Run any Program" start "C:\Windows\System32\cmd.exe"
and set parameters to
/C "C:\Bin\CopyXML.bat"

(this bat file is shown further down below)

Task 2 - your current FTP task to upload XML. Let it monitor C:\XML instead (where Subtask 1.2 copies the xml files of completed MP4's)


The bat file for Subtask 1.2 is simple enough:
Code:
SET TARGET=C:\XML
SET XML=%WD_FILE_D%%WD_FILE_B%.xml
:TryAgain
IF EXIST "%XML%" GOTO :CopyXml
rem sleep a bit
"%WD_INSTDIR%\GdPUtil.exe" -sleep 3
GOTO :TryAgain

:CopyXml
COPY "%XML%" "%TARGET%"
EXIT
 




I have not tested the above but I think it should work.
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Re: Synchronised FTP uploads
Reply #7 - Nov 1st, 2010 at 8:42pm
 
Hi Gert,
Sorry for the post in Dutch, I misread that you meant email.

Still, the solution you provided does not work.
Are you sure the environment variables %WD_FILE_D%%WD_FILE_B% exist? If I, during FTP upload of the MP4 file, execute the command 'env' I do never find these environment variables.
Am I missing something?
My task/subtask configuration is correct up until the copying of the XML, uploading the MP4 does work.

And, where can I get GpdUtil.exe?

Hope you can help me (again Smiley).

Greetz Renzo.
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Synchronised FTP uploads
Reply #8 - Nov 2nd, 2010 at 6:45am
 
Yes, those WD_ variables all exist (see http://www.watchdirectory.net/wdhelp/plugins/wdopAutoRunBatEnv.html ).
GdPUtil is installed with WatchDirectory so "%WD_INSTDIR%\GdPutil.exe" should be fine (WD_INSTDIR is WD's install directory).

I just did a small test with this bat file:
[code]SET WD_ > "C:\Temp\wdvars.txt"
[/code]
and let the Subtask plugin run the "start any program" plugin. When I dropped a file into the monitored directory, I got this in C:\Temp\wdvars.txt:
[code]WD_AGECREATED_DAYS=0
WD_AGECREATED_HOURS=0
WD_AGEMODIFIED_DAYS=206
WD_AGEMODIFIED_HOURS=4962
WD_CONFIG=startpe
WD_CUREVT=1
WD_CURRUN=1
WD_DATE=20101102
WD_DIR=f:\Testing\Source
WD_DTTM=20101102 053417
WD_FILE=f:\Testing\Source\12 - kopie (3).pdf
WD_FILE_B=12 - kopie (3)
WD_FILE_D=f:\Testing\Source\
WD_FILE_DATE_CR=20101102
WD_FILE_DATE_LM=20100409
WD_FILE_E=PDF
WD_FILE_N=12 - kopie (3).pdf
WD_FILE_OWNER=Gert
WD_FILE_SHORTNAME=12-KOP~4.PDF
WD_FILE_SHORTPATH=f:\Testing\Source\12-KOP~4.PDF
WD_FILE_SIZE=63932
WD_FILE_SIZE_GB=0
WD_FILE_SIZE_KB=62
WD_FILE_SIZE_MB=0
WD_FILE_TIME_CR=053414
WD_FILE_TIME_LM=113124
WD_FIRSTDIR=Source
WD_INSTDIR=C:\Program Files (x86)\watchDirectory
WD_OFILE=
WD_OFILE_B=
WD_OFILE_D=
WD_OFILE_E=
WD_OFILE_N=
WD_ORELDIR=
WD_PARENTDIR=Source
WD_REASON=FILENEW
WD_RELDIR=
WD_TASKDIR=C:\ProgramData\watchDirectory\startpe
WD_TIME=053417
[/code]

I wonder if you have setup the subtask mask correctly for the 2nd subtask (*.mp4).
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Re: Synchronised FTP uploads
Reply #9 - Nov 3rd, 2010 at 9:29am
 
Hello Gert,
Thanks again for your help, now I've got it working, the subtask 1.2 mask was indeed incorrect  Embarrassed.
Thanks!

One question, is it possible that Watchdirectory keeps the XML file, after upload, open so that other applications cannot use it? I did not find a setting related to this behavior.

At the upload site, we have our own software monitoring the directory to where the XML's are uploaded to import them into a database. Before, when files were placed in that directory, they were automatically picked up but not anymore.
Any idea?
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Synchronised FTP uploads
Reply #10 - Nov 3rd, 2010 at 9:36am
 
No, WD doesn't keep those files open at the FTP server-side.

However, some settings you may want to experiment with, perhaps they influence how the FTP Server software works:

* Enable or Disable the option "keep connection open as long as possible"
* Enable or Disable the option "upload files with a .partial extension"

Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
renzobekken
YaBB Newbies
*
Offline



Posts: 7
Re: Synchronised FTP uploads
Reply #11 - Nov 3rd, 2010 at 10:28am
 
Hello Gert,

* Enable or Disable the option "upload files with a .partial extension"

This setting was the problem, I changed that setting yesterday.
Now everything works fine, thanks for your help!

Greetz Renzo.
Back to top
 
 
IP Logged
 
Gert
YaBB Administrator
*****
Offline



Posts: 2239
The Netherlands
Re: Synchronised FTP uploads
Reply #12 - Nov 3rd, 2010 at 12:16pm
 
A little protest from this side...

> * Enable or Disable the option "upload files with a .partial extension" 
> This setting was the problem, I changed that setting yesterday.

YOUR application running on the FTP server is the problem. Depending on the ".partial" setting, it means
* Your application doesn't handle locked/in-use files correctly
or
* Your application doesn't handle renamed files correctly
Back to top
 

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&...
WWW WWW GdPSoftware  
IP Logged
 
Page Index Toggle Pages: 1