UDF to support SFTP protocol using PSFTP

[Lupo73]

Note that I forgot to write an answer, but the last "red corrections" are implemented in the code.

[iCode]

i like it when it all works the first time

very nice work Adventurer -- you made it really easy to use psftp

[GreenCan]

Hi Lupo73,

I have been doing some tests with your SFTPEx UDF and I find it very useful. I appreciate that you are the first one to come up with a UDF covering SFTP, and you've got my   for this.

I found an inconsistency in _SFTP_FileMove which I tried to correct, see code hereunder.

The first issue occurred while trying to rename a file to an already existing one, then the function did not return an error, but the file was not renamed.

The second issue occurred while moving (multiple) files to a directory using wildcards, and one of the files already existed in the destination directory.

I reshuffled the If/Else loop so that it tackles all error situations and I believe it captures all errors now.

I added extra error return values, I am not sure if Error 5 is still needed but I left it there, it doesn't harm anyway.

Correct Version (added dummy stdoutread at the end so that other functions are not affected by the remaining data in the buffer,  sorry again)

Update 12/12/2013

New issue found when trying to move multiple files with wildcards.

In some cases, for example when using wildcards, file processing time increases, StdoutRead could still be empty, so i added a loop waiting fro the 'psftp>' prompt before going forward with error checking.

One possible issue: if the prompt never comes, the function will ends into an indefinite loop.  I have no solution for this but adding a time-out as a parameter would be a partial answer.  Question is then 'which time out should be used?'

You will also notice that I empty the console buffer before and at the end of the function. I noticed that  previous failed function, like an error on creating a directory will abort the process with an error but do not clear the console buffer.  I would recommend to always clear the console on exit of a function.

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _SFTP_FileMove
; Description ...: Move/Rename a file on a SFTP server.
; Syntax.........: _SFTP_FileMove ( $hConnection, $sSourceFile, $sDestinationFile )
; Parameters ....: $hConnection - as returned by _SFTP_Connect().
;                  $sSourceFile - The source file to move/rename. Wildcards are allowed if move to a directory
;                  $sDestinationFile - The destination file or directory to move/rename.
; Return values .: Success - 1
;                  Failure - 0, sets @error
;                  |1 - The connection is closed
;                  |2 - File not found
;                  |3 - Failure to rename or move a file. In case of wildcards, failure to move at least one file.
;                  |4 - no file name matched the source file name(s)
;                  |5 - Other error
; Author ........: Lupo73, GreenCan
; Modified.......: Corrected issue while trying to rename a file to a new file that already exists
;                   Issue with StdoutRead being empty still, causing malfunction of _SFTP_FileMove, added wait loop.
; Remarks .......:
; Related .......: _SFTP_Connect
; Link ..........:
; Example .......: No
; ===============================================================================================================================
Func _SFTP_FileMove($hConnection, $sSourceFile, $sDestinationFile)
    If ProcessExists($hConnection) = 0 Then
        Return SetError(1, 0, 0)
    EndIf

    Local $sLine
    $sLine = StdoutRead($hConnection) ; dummy StdoutRead to empty the buffer from previously failed commands

    StdinWrite($hConnection, 'mv "' & $sSourceFile & '" "' & $sDestinationFile & '"' & @CRLF)
    ;loop required to wait for console response
    While 1
        sleep(100)
        $sLine = StdoutRead($hConnection, True) ; keep the stream buffer intact
;~         If $sLine <> "" Then ExitLoop
        If StringInStr($sLine, "psftp>") Then ExitLoop
    WEnd
    While 1
        $sLine = StdoutRead($hConnection)
        If ProcessExists($hConnection) = 0 Then
            Return SetError(1, 0, 0)
        ElseIf StringInStr($sLine, $sSourceFile & " -> ") Then
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            ContinueLoop
        ElseIf StringInStr($sLine, "no such file or directory") Then
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            Return SetError(2, 0, 0)
        ElseIf StringInStr($sLine, "failure") Then
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            Return SetError(3, 0, 0)
        ElseIf StringInStr($sLine, "nothing matched") Then
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            Return SetError(4, 0, 0)
        ElseIf StringInStr($sLine, "psftp>") Then
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            ExitLoop
        ElseIf $sLine <> "" Then ; any other failure
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            Return SetError(5, 0, 0)
        ElseIf $sLine == "" Then
            ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " " & $sLine & @CR)
            ExitLoop
        EndIf
        Sleep(10)
        ;If Not @Compiled Then ConsoleWrite( @ScriptLineNumber & " >" & $sLine & "<" & @CR)
    WEnd
    $sLine = StdoutRead($hConnection) ; dummy StdoutRead to empty the buffer
    Return 1
EndFunc   ; ==>_SFTP_FileMove

I hope this helps.

GreenCan

Edited December 12, 2013 by GreenCan

[GreenCan]

An example file is always better...  here is one

remember to fill in these lines:

Global $sServerName = "server name"                 ; ==> fill in
Global $sUsername = "user name"                        ; ==> fill in
Global $sPassword = "user password"                  ; ==> fill in
Global $iServerPort = 22                                       ; ==> fill in optionally, default should be 22, not 0 as far as i know
Global $cRemoteDir = "/some remote paths"        ; ==> fill in

and create a 'test.txt' file in your script folder.

Update 12/12/2013

cleanup up example

SFTPEx example - Generic.au3

Edited December 12, 2013 by GreenCan

[Lupo73]

Sorry for the late and thanks for these improvements!

A question: why it is needed a second loop before the main one? the check of the "psftp>" string is already included in the main loop.

If you would like to help me to improve and complete this UDF, I think it could be useful to several users. 

Edited January 5, 2014 by Lupo73

[GreenCan]

It's my turn to apologize for replying late to your query...

A question: why it is needed a second loop before the main one? the check of the "psftp>" string is already included in the main loop.

I added the loop because I noticed that for connections using slow sftp hosts and especially when using wildcards or large files, I got unresolved and replicated errors.  The simplest explanation is that the psftp> prompt might not yet be in the buffer while the function performs the error checking.

I don't remember exactly what kind of error I got, but the file move failed without returning any of the 5 error codes.

Ideally you want to exit here

        ElseIf StringInStr($sLine, "psftp>") Then
or here;
        ElseIf $sLine == "" Then
but either you receive the prompt or you don't get it and your function fails in any or the other case.

If you really need assurance, I can try to replicate the issue but  I don't know when I will have time to check this.

GreenCan

 

[Lupo73]

Don't worry.. but obviously feel free to propose (or directly do) other improvements and fixes to the code.

[Neutro]

Thank you very much Lupo73, this is just what I needed and it's working flawlessly

[MuffettsMan]

THANK YOU Green Can lol I'm a hack and this in your face example gave me enough examples to get me started using the udf!!!

 

An example file is always better...  here is one

remember to fill in these lines:

Global $sServerName = "server name"                 ; ==> fill in
Global $sUsername = "user name"                        ; ==> fill in
Global $sPassword = "user password"                  ; ==> fill in
Global $iServerPort = 22                                       ; ==> fill in optionally, default should be 22, not 0 as far as i know
Global $cRemoteDir = "/some remote paths"        ; ==> fill in

and create a 'test.txt' file in your script folder.

Update 12/12/2013

cleanup up example

SFTPEx example - Generic.au3

[GreenCan]

My pleasure

But the credits go to Lupo73 as he came with the SFTPEx udf, which by the way, I use every day in my company...

GreenCan

[Bitnugger]

I have discovered a truly insidious error in:

_SFTP_ListToArrayEx ()

Multiple spaces in file and directory names are filtered out by the splitting of the string.

$aSubStringSplit = _StringExplode(StringStripWS($aStringSplit[$A], 7), " ", 8)

But this should not happen, because the names must remain unchanged in all cases.

As a workaround, I have therefore the above line changed to:

$sSplit = $aStringSplit[$A]
$aSubStringSplit = _StringExplode(StringStripWS(StringLeft($sSplit, StringInStr($sSplit, ":") + 3), 7), " ", 7)
_ArrayAdd($aSubStringSplit, StringMid($sSplit, StringInStr($sSplit, ":") + 4))

A very interesting project that will hopefully continue to be cultivated.
 

Edited March 4, 2015 by Bitnugger

[Gauss]

I have started using this UDF SFTPEx.au3 to download a bunch of files from a remote server to local. 

First, Great job with the UDF and this really works! awesome work Lupo73...

Second, there is no timestamps on the listed files when I use the function _SFTP_ListToArrayEx. Screenshots attached

Third, Is the _SFTP_ProgressDownload function still not working?

I am using the version 1.0 beta 9

      using _ArrayDisplay function

    using Filezilla

Edited December 8, 2015 by Gauss
snapshot labels updated

[Gauss]

No replies yet on my question. Anybody please help.

[argumentum]

No replies yet on my question. Anybody please help.

As a workaround, I have therefore the above line changed to:

$sSplit = $aStringSplit[$A]
$aSubStringSplit = _StringExplode(StringStripWS(StringLeft($sSplit, StringInStr($sSplit, ":") + 3), 7), " ", 7)
_ArrayAdd($aSubStringSplit, StringMid($sSplit, StringInStr($sSplit, ":") + 4))

A very interesting project that will hopefully continue to be cultivated.

did you try that ? 

[nill]

Please help

I want upload folder with all content from local comp to server by sftp

use _SFTP_DirPutContents but on server i dont see new folder

_SFTP_DirPutContents dont return any errors

$hSession = _SFTP_Open()
$hSession = _SFTP_Connect($hSession, $sServerName, $sUsername, $sPassword, $iServerPort)

$server="/home/admin/web/local/public_html/"
$local="C:\test\"
$err =_SFTP_DirPutContents($hSession, $local, $server, 1)

When use _SFTP_FilePut all work fine

[t0nZ]

Hi Lupo73 and thank  you for your work, this UDF saves my life.

This said, I found the function _SFTP_DirSetCurrent() "works" but doesn't return *exactly*  the new DIR, because I think the output inside the func is not correctly trimmed.

So when I try to set "/20170517073216" as current DIR the Func returns something like "/20170517073216/
psftp> " (with all the spaces) so to test if  the FUNC has worked well I must do something like 

If StringInStr(_SFTP_DirSetCurrent($Conn, $ftpfolder),$ftpfolder) Then...
 ;ok 
else
 ;not OK
endif

Cheers.

Edited June 19, 2017 by t0nZ

[MimiOne]

Hello Lupo73, thank you very much for this UDF
The _SFTP_DirDelete function loops to infinity trying to erase "." and ".."
I added a test:

If ($ aFileList [$ A] [0] <> ".") And ($ aFileList [$ A] [0] <> "..") Then ...

 do not know not so "." and ".." are sytematically present at the top of the list on all the servers

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _SFTP_DirDelete
; Description ...: Deletes a Directory on a SFTP server.
; Syntax.........: _SFTP_DirDelete ( $hConnection, $sRemoteDir )
; Parameters ....: $hConnection - as returned by _SFTP_Connect().
;                  $sRemoteDir - The remote Directory to be deleted.
; Return values .: Success - 1
;                  Failure - 0, sets @error
;                  |1 - The connection is closed
;                  |2 - Directory not found
;                  |3 - Directory probably contains not removable files
;                  |4 - Failed listing Directory
;                  |5 - Other error
; Author ........: Lupo73
; Modified.......:
; Remarks .......:
; Related .......: _SFTP_Connect, _SFTP_FileDelete, _SFTP_ListToArrayEx
; Link ..........:
; Example .......: No
; ===============================================================================================================================
Func _SFTP_DirDelete($hConnection, $sRemoteDir)
    If ProcessExists($hConnection) = 0 Then
        Return SetError(1, 0, 0)
    EndIf

    Local $aFileList = _SFTP_ListToArrayEx($hConnection, $sRemoteDir)
    If @error Then
        Return SetError(4, 0, 0)
    EndIf

    If $aFileList[0][0] > 0 Then
        For $A = 1 To $aFileList[0][0]
            If StringLeft($aFileList[$A][2], 1) <> "d" Then
                _SFTP_FileDelete($hConnection, $sRemoteDir & "/" & $aFileList[$A][0])
            Else
                ;-- Do not try to delete "." and ".." ---------------------------
                If ($aFileList[$A][0] <> ".") And ($aFileList[$A][0] <> "..") Then _SFTP_DirDelete($hConnection, $sRemoteDir & "/" & $aFileList[$A][0])
            EndIf
            If @error Then
                Return SetError(5, 0, 0)
            EndIf
        Next
    EndIf

;-- Etc ....   --------------

EndFunc

 

Edited February 20, 2018 by MimiOne

[MimiOne]

A small step for "_SFTP_ProgressDownload" (which works properly for files).
I do not understand lines 42 / 70 (in Spoiler)
cordially

Spoiler

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _SFTP_ProgressDownload
; Description ...: Downloads a file and shows a Progress window or by Calling a User defined Function.
; Syntax.........: _SFTP_ProgressDownload ( $hConnection, $sRemoteFile [, $sLocalFile = "" [, $fFailIfExists = False [, $fRecursiveGet = 1 [, $FunctionToCall = ""]]]] )
; Parameters ....: $hConnection - as returned by _SFTP_Connect().
;                  $sRemoteFile - The remote file.
;                  $sLocalFile - Optional, The local file (or use the source name if not defined).
;                  $fFailIfExists - Optional, True: do not overwrite existing (default = False)
;                  $fRecursiveGet - Optional, Recurse through sub-dirs: 0 = Non recursive, 1 = Recursive (default)
;                  $FunctionToCall - Optional, A function which can update a ProgressBar and react on UserInput like Click on Abort or Close App.
; Return values .: Success - 1
;                  Failure - 0, sets @error
;                  |1 - The connection is closed
;                  |2 - Local folder exists and $fFailIfExists = True
;                  |3 - Remote folder not found
;                  |4 - Download aborted by PercentageFunc and Return of Called Function
;                  |5 - Other error
; Author ........: Lupo73
; Modified.......:
; Remarks .......:
; Related .......: _SFTP_Connect, _SFTP_ProgressUpload, _SFTP_FileGetSize
; Link ..........:
; Example .......: No
; ===============================================================================================================================
Func _SFTP_ProgressDownload($hConnection, $sRemoteFile, $sLocalFile = "", $fFailIfExists = False, $fRecursiveGet = 1, $FunctionToCall = "")
    If ProcessExists($hConnection) = 0 Then
        Return SetError(1, 0, 0)
    EndIf

    If $sLocalFile <> "" Then
        $sLocalFile = __WinAPI_GetFullPathName($sLocalFile)
        If FileExists($sLocalFile) Then
            If $fFailIfExists Then
                Return SetError(2, 0, 0)
            EndIf
        EndIf
        $sLocalFile = ' "' & $sLocalFile & '"'
    EndIf
    If $FunctionToCall = "" Then
        ProgressOn("SFTP Download", "Downloading " & $sRemoteFile)
    EndIf
    Local $iSize = _SFTP_FileGetSize($hConnection, $sRemoteFile) ; <<<<<<<<<<<<<<<<<<<<<< 0 for folders
    Local $sLine, $iInitialBytes, $iReadBytes, $iError, $iPercent
    If $fRecursiveGet Then
        $sLine = '-r '
    EndIf

    StdinWrite($hConnection, 'get ' & $sLine & '-- "' & $sRemoteFile & '"' & $sLocalFile & @CRLF)
    $iReadBytes = ProcessGetStats($hConnection, 1) ; <<<<<<<<<<<<<<<<<<<<<<<< it may needs to load from another stat
;~     $iInitialBytes = $iReadBytes[3]

    While 1
        $sLine = StdoutRead($hConnection)
        If ProcessExists($hConnection) = 0 Then
            $iError = 1
            ExitLoop
        ElseIf StringInStr($sLine, "psftp>") Then
            ExitLoop
        ElseIf StringInStr($sLine, "=> local:") Then
            ContinueLoop
        ElseIf StringInStr($sLine, "no such file or directory") Then
            $iError = 3
            ExitLoop
        ElseIf $sLine <> "" Then
            $iError = 5
            ExitLoop
        EndIf

        $iReadBytes = ProcessGetStats($hConnection, 1)
        $iPercent = int(($iReadBytes[4] / $iSize) * 100) ;<<< !!!! $iSize = 0 for folders !!!! >>>>

        If $FunctionToCall = "" Then
            ProgressSet($iPercent) ;(($iReadBytes[3] - $iInitialBytes) / $iSize * 100)
        Else
;~          If Call($FunctionToCall, ($iReadBytes[3] - $iInitialBytes) / $iSize * 100) <= 0 Then
            If Call($FunctionToCall, $iPercent) <= 0 Then
                ProcessClose($hConnection)
                $iError = 4
                ExitLoop
            EndIf
        EndIf
        Sleep(10)
    WEnd

    If $FunctionToCall = "" Then
        ProgressOff()
    EndIf

    If $iError Then
        Return SetError($iError, 0, 0)
    EndIf

    Return 1
EndFunc   ;==>_SFTP_ProgressDownload

 

 

Edited February 21, 2018 by MimiOne

[Slash12]

I needed a quick way to publish files to SFTP and this saved the day. Thank you Lupo73!

I wanted to keep the file date as the original but that does not happen, all date-time values changes to upload date and time. I guess there is a limitation in psftp functions. using a normal sftp client (like FlashFXP) preserves the original file date and times. If there is a way to come around this I'm interested. 

 

[Slash12]

This UDF is working fine but I eventually will need to have the file creation date and time preserved during transfer. Is there anyone that can add this to the UDF? Or have a tip on other ways (using autoit) to transfer files to sftp remote host and get the file creation date and time preserved. I'm not that keen on adding winscp or any other client to accomplish this but if there is no way around it I may have to. 

Thanks.

Edited September 29, 2018 by Slash12