WatchDirectory Forum - Losing my mind with an sftp transfer

Oct 12^th, 2024 at 11:21am

Choose Language:

WatchDirectory home page WatchDirectory Startseite (Deutsche Version) Site WatchDirectory (Français)
		Welcome, Guest. Please Login or Register

Home

Help

WatchDirectory Forum › watchDirectory forums › Bug Reports › Losing my mind with an sftp transfer

(Moderators: Gert, Forum Admin)

‹ Previous Topic | Next Topic ›

Pages: 1

Losing my mind with an sftp transfer (Read 569 times)

dracosalpha

YaBB Newbies

Offline

Posts: 7
Montreal, Canada

Losing my mind with an sftp transfer
Jul 21^st, 2023 at 10:08pm

Holy crap I have been at this for hours!

I'm posting this here because I assume it's some kind of bug. Feel free to point and laugh if this is not the case. I'm a little punchy now.

I need to monitor a folder for the presence of a file and transfer it via SFTP using SSH keys, NOT a user name and password. This is a client requirement.

My understanding is that I can't do that using the "Upload Files" task type, because it can only work with usernames and password, even in SFTP.

So I thought I could use a batch file using OpenSSH SFTP.
I tested the process with a batch file from a cmd command window, and it works correctly.
But when I run the batch file from WatchDirectory, the file doesn't transfer. I can't figure out why. Everything else works perfectly, except the file doesn't go.

The batch file creates a script file, then invokes SFTP from OpenSSH to send the file.

The batch file is basically:

<commands to concatenate the control file>

C:\Windows\System32\OpenSSH\sftp.exe -b c:\FTP_ROOT\clientname\prod\export\control\sftpcontrol.scr user@suppliersite.com

<commands to copy the transferred file to archives>

The control file contains three commands:

cd inbound
put c:/FTP_ROOT/clientname/prod/client/export/filename.CSV
quit

This works great if I run the batch file from the command line.

If I run it through WatchDirectory, the command gives me an error.

Here is the batch file capture log:
C:\Windows\system32>GOTO :StartOfBat
C:\Windows\system32>"C:\Program Files (x86)\watchDirectory\wdAnnotate.exe" info "Starting C:\php\MEDUpload.bat for event"

C:\Windows\system32>Call "C:\php\MEDUpload.bat" FILENEW
The system cannot find the path specified.
1 file(s) copied.

I can't figure out why this command works fine as a batch file, but not as a batch file run through WatchDirectory.
What is it about the WatchDirectory command execution environment that is different?

Any help would be appreciated.
Also, if this is not a bug, and I should post this elsewhere, please let me know.

IP Logged

Gert

YaBB Administrator

Offline

Posts: 2362
The Netherlands

Re: Losing my mind with an sftp transfer
Reply #1 - Jul 22^nd, 2023 at 5:59am

Hi,

I bet you are running the task as a Windows Service....

Make sure your scripts and command lines use full path names and if those paths have spaces in their name, "properly quote" them.

So do not use
Code:

hello inputfile.txt

but use this instead
Code:

"c:\Program Files (x86)\Hello Inc\hello.exe" "c:\temp\inputfile.txt"

Make sure to read http://blog-en.gdpsoftware.com/2010/04/watchdirectory-tasks-as-windows-service.h...

(not laughing here, you are not the first one to make this mistake)

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&

WWW

IP Logged

dracosalpha

YaBB Newbies

Offline

Posts: 7
Montreal, Canada

Re: Losing my mind with an sftp transfer
Reply #2 - Jul 22^nd, 2023 at 12:44pm

Hi Gert,

Thanks a lot for your answer.
Yes, I was trying to run it as a service, but since it didn't work, I also tried other modes, results are the same.
I'm from the very old school of 8.3, and don't usually have spaces in anything.
Here is the whole batch file:

----
@echo off
TITLE MEDSFTPUpload
cd \FTP_ROOT\clientname\prod\client\export\control

SET SOURCEDIR=C:\FTP_ROOT\clientname\prod\client\export\
SET TARGETDIR=C:\FTP_ROOT\clientname\prod\client\export\archives\

set SCRIPT=C:\FTP_ROOT\clientname\prod\client\export\control\sftpcontrol.scr

(
echo cd inbound
echo put c:/FTP_ROOT/clientname/prod/client/export/%WD_FILE_N%
echo quit
) > %SCRIPT%

cd \FTP_ROOT\clientname\prod\client\export
REM timeout /t 10 /nobreak
C:\Windows\System32\OpenSSH\sftp.exe -b c:\FTP_ROOT\clientname\prod\client\export\control\sftpcontrol.scr varinc@cssh.automation.website.ca

copy "%SOURCEDIR%%WD_FILE_N%" "%TARGETDIR%%WD_FILE_B%%WD_DTTM%.%WD_FILE_E%"
erase "%SOURCEDIR%%WD_FILE_N%"

----

Files generated by the process don't have spaces either, as you can see in the latest script file I tested: sftpcontrol.scr

----

cd inbound
put c:/FTP_ROOT/clientname/prod/client/export/MED-DM-WF-SUIVI-ACHAT-20230719-Q0-ex.C
SV
quit

----

Obviously I can't execute this batch file manually, since the runtime variables aren't available if not initiated by WatchDirectory, so I run this edited version instead to test the command. In it I REM out the lines that have runtime variables, and the script picks up the control file from the previous execution.

----
@echo off
TITLE MEDSFTPUpload
cd \FTP_ROOT\clientname\prod\client\export\control

SET SOURCEDIR=C:\FTP_ROOT\clientname\prod\client\export\
SET TARGETDIR=C:\FTP_ROOT\clientname\prod\client\export\archives\

set SCRIPT=C:\FTP_ROOT\clientname\prod\client\export\control\sftpcontrol.scr

REM (
REM echo cd inbound
REM echo put c:/FTP_ROOT/clientname/prod/client/export/%WD_FILE_N%
REM echo quit
REM ) > %SCRIPT%

cd \FTP_ROOT\clientname\prod\client\export
REM timeout /t 10 /nobreak
C:\Windows\System32\OpenSSH\sftp.exe -b c:\FTP_ROOT\clientname\prod\client\export\control\sftpcontrol.scr varinc@cssh.automation.website.ca

REM copy "%SOURCEDIR%%WD_FILE_N%" "%TARGETDIR%%WD_FILE_B%%WD_DTTM%.%WD_FILE_E%"
REM erase "%SOURCEDIR%%WD_FILE_N%"

----

This batch file successfully transfers the file. I tried waiting 10, 20, and 30 seconds between the creation of the control file and the initiation of the command, just in case the file system doesn't realize the file is there yet one batch file line later. I've seen something like that happen before.
That wasn't it though.

Unless you can see something, my fallback will probably be to generate the script file only, and to try running a batch file through the Windows Task Scheduler every 10 minutes or so, but that leaves me open to missing files, if multiple files are dropped in the same 10 minutes. Shouldn't happen but, again, I've had surprises before.

Let me know if you see anything.
And, as always, I really appreciate your excellent knowledge of your product.

Alex.

IP Logged

dracosalpha

YaBB Newbies

Offline

Posts: 7
Montreal, Canada

Re: Losing my mind with an sftp transfer
Reply #3 - Jul 22^nd, 2023 at 1:01pm

UPDATE

After reading the information in the link you sent me, I tried changing the user to the administrator instead of the local user.
This time I got a different error message in the log.

----

C:\Windows\system32>GOTO :StartOfBat
C:\Windows\system32>"C:\Program Files (x86)\watchDirectory\wdAnnotate.exe" info "Starting C:\php\MEDUpload.bat for event"

C:\Windows\system32>Call "C:\php\MEDUpload.bat" FILENEW
ERROR: Input redirection is not supported, exiting the process immediately.
The system cannot find the path specified.
1 file(s) copied.

----

It's not from the timeout, because I have it REMmed out.

IP Logged

Gert

YaBB Administrator

Offline

Posts: 2362
The Netherlands

Re: Losing my mind with an sftp transfer
Reply #4 - Jul 22^nd, 2023 at 2:13pm

Alex,

I obviously can not see what is happening. Just a quick look at what you wrote, I see

cd \nvnnkknv

are you sure you are on the DISK you think you are?
Maybe do it like

C:
cd \whatever

A debug tactic I use with scripts called by WD is the "debugger.bat" script below.
Code:

ECHO ON
SET DEBUGOUTPUT=C:\LOGS\%WD_CUREVT%.txt

rem the batch file to call (bat or cmd) - this is your original script
SET SCRIPTTOCALL=C:\Scripts\MyScript.bat

rem output all WD variables to DEBUGOUTPUT
ECHO ----- start environment listing ----- >> "%DEBUGOUTPUT%"
SET WD_ >> "%DEBUGOUTPUT%"
ECHO ----- end environment listing ----- >> "%DEBUGOUTPUT%"

ECHO ----- start output of script %SCRIPTTOCALL% ----- >> "%DEBUGOUTPUT%"

rem call the batch script, redirecting all output to DEBUGOUTPUT
Call "%SCRIPTTOCALL%" >> "%DEBUGOUTPUT%" 2>&1

ECHO ----- end output of script %SCRIPTTOCALL% ----- >> "%DEBUGOUTPUT%"

and let WD call this script instead. You should change it so debugger.bat will call your script. It will first dump all environment variables whose name start with WD_ to a log. Then call your script, capturing all output (including error output, that is what the 2>&1 does) into the same log. Each event (detected file) will create a new log file with names like 123.txt, 124.txt, 125.txt (incrementing number based on the event number).

Make sure you remove any "echo off" from your own script. If you want to log something from your own script, just do
ECHO just before upload
to it - without redirecting to DEBUGOUTPUT!!!

Hope this debug method gives you a bit more insight in what is happening.

Gert Rijs - gert (at) gdpsoftware (dot) com
Blog: http://blog-en.gdpsoftware.com/
End Alzheimer's: http://www.alz.org&&

WWW

IP Logged

dracosalpha

YaBB Newbies

Offline

Posts: 7
Montreal, Canada

Re: Losing my mind with an sftp transfer
Reply #5 - Jul 25^th, 2023 at 1:02pm

Thanks Gert, for the detailed help to debug my script.

I did have a feeling that the control file that is created at execution time may not yet be available to the SFTP command...
I create C:\FTP_ROOT\medicus\prod\client\export\control\sftpcontrol.scr and then invoke C:\Windows\System32\OpenSSH\sftp with that control file. (highlighted below)
There's a chance the file is not yet "available" in that same second.

When I run that same command a moment later manually, the file transfers with no problems.
Is there a way to introduce a timeout using WatchDirectory? It's not letting me use timeout inside the batch file. If I do, I get this:
C:\Windows\system32>Call "C:\php\MEDKopelUpload.bat" FILENEW
ERROR: Input redirection is not supported, exiting the process immediately.

Another possibility may be that the control file needs to be addressed by an 8.3 filename. I'll try that too.

Let me know if you see anything else.

Also, can I shoot you some thank you cash for your help so far?

----- start environment listing -----
WD_CONFIG=MEDKopelSFTPUpload
WD_CUREVT=2
WD_CURRUN=8
WD_DATE=20230725
WD_DIR=C:\FTP_ROOT\medicus\prod\client\export
WD_DTTM=20230725 074621
WD_EVENTNO=2
WD_FILE=C:\FTP_ROOT\medicus\prod\client\export\MED-DM-WF-SUIVI-ACHAT-20230725-Q4
73-ex.csv
WD_FILE_B=MED-DM-WF-SUIVI-ACHAT-20230725-Q473-ex
WD_FILE_D=C:\FTP_ROOT\medicus\prod\client\export\
WD_FILE_DATE_CR=20230725
WD_FILE_DATE_LM=20230725
WD_FILE_E=CSV
WD_FILE_N=MED-DM-WF-SUIVI-ACHAT-20230725-Q473-ex.csv
WD_FILE_OWNER=admin
WD_FILE_SHORTNAME=MED-DM~1.CSV
WD_FILE_SHORTPATH=C:\FTP_ROOT\medicus\prod\client\export\MED-DM~1.CSV
WD_FILE_SIZE=89933
WD_FILE_SIZE_GB=0
WD_FILE_SIZE_KB=87
WD_FILE_SIZE_MB=0
WD_FILE_TIME_CR=073002
WD_FILE_TIME_LM=073003
WD_INSTDIR=C:\Program Files (x86)\watchDirectory
WD_REASON=FILENEW
WD_RELDIR=
WD_TASKDIR=c:\programdata\watchdirectory\medkopelsftpupload
WD_TIME=074621
----- end environment listing -----
----- start output of script C:\php\MEDKopelUpload.bat -----

C:\Windows\system32>REM ECHO OFF

C:\Windows\system32>TITLE MEDKopelUpload

C:\Windows\system32>cd \FTP_ROOT\medicus\prod\client\export\control

C:\FTP_ROOT\medicus\prod\client\export\control>SET SOURCEDIR=C:\FTP_ROOT\medicus\prod\client\export\

C:\FTP_ROOT\medicus\prod\client\export\control>SET TARGETDIR=C:\FTP_ROOT\medicus\prod\client\export\archives\

C:\FTP_ROOT\medicus\prod\client\export\control>set SCRIPT=C:\FTP_ROOT\medicus\prod\client\export\control\sftpcontrol.scr

C:\FTP_ROOT\medicus\prod\client\export\control>(
echo cd inbound
echo put c:/FTP_ROOT/medicus/prod/client/export/MED-DM-WF-SUIVI-ACHAT-20230725-Q473-ex.cs
v
echo quit
) 1>C:\FTP_ROOT\medicus\prod\client\export\control\sftpcontrol.scr

C:\FTP_ROOT\medicus\prod\client\export\control>C:

C:\FTP_ROOT\medicus\prod\client\export\control>cd \FTP_ROOT\medicus\prod\client\export

C:\FTP_ROOT\medicus\prod\client\export>C:\Windows\System32\OpenSSH\sftp.exe -b c:\FTP_ROOT\medicus\prod\client\export\control\sftpcontrol.scr varinc@cssh.automation.kopel.ca
The system cannot find the path specified.

C:\FTP_ROOT\medicus\prod\client\export>copy "C:\FTP_ROOT\medicus\prod\client\export\MED-DM-WF-SUIVI-ACHAT-20230725-Q473-ex.c
sv" "C:\FTP_ROOT\medicus\prod\client\export\archives\MED-DM-WF-SUIVI-ACHAT-20230725-
Q473-ex20230725 074621.CSV"
1 file(s) copied.

C:\FTP_ROOT\medicus\prod\client\export>erase "C:\FTP_ROOT\medicus\prod\client\export\MED-DM-WF-SUIVI-ACHAT-20230725-Q473-ex.c
sv"
----- end output of script C:\php\MEDKopelUpload.bat -----

« Last Edit: Jul 25^th, 2023 at 3:21pm by dracosalpha »

IP Logged

dracosalpha YaBB Newbies Offline Posts: 7 Montreal, Canada	Re: Losing my mind with an sftp transfer Reply #6 - Jul 25^th, 2023 at 3:44pm Ok, I've pretty much given up. The commands work when run by the event manager, so I'm going to just run it every few minutes to transfer this file when it arrives. Thanks nevertheless for your help.
Back to top	IP Logged

Pages: 1

‹ Previous Topic | Next Topic ›

« Home

‹ Board

Top of this page