Getting list of files from an FTP site

[jussie]

Hi all,

I have been given a task to connect to an FTP site and download all the contents on a daily basis. I would be using the InetGet command for the connection and download

InetGet("ftp://USERNAME:PASSWORD@WEBSITE", "SAVED FILE", 1)

How can I get a list of the files on the remote site so I can set them into an array and download them one by one?

I am using v3.2.1.11 (beta)

Thanks in advance

Justin

[jussie]

I think I might have found my answer at http://www.autoitscript.com/forum/index.php?showtopic=34470

[jussie]

Actually it didn't work.

Can someone help me write a script to connect to an FTP site (with credentials) and download all the files from the root directory.

I would assume listing the files to an array or something like that. I can't get my head around some of the commands and the paramaters that go with them.

I was hoping to use InetGet command but you need to know what the file names would be to download.

Thanks in advance

Justin

[flyingboz]

Jussie:

I would seriously consider writing or using a wrapper for an ftp client, or using an ftp client with scripting capabilities.

While not free, Kermit-95 has an exceptional scripting engine and is largely compatible with cKermit, which runs on just about every platform known to man.

ncftp (www.ncftp.com) has an excellent freeware command line ftp engine, as well as specialized get / put command line tools.

SmartFTP, FileZilla, and many other windows ftp clients exist, with varying levels of scripting support and licensing.

the m$ ftp client on later o/s has a /s switch, which allows you to script a set series of commands.

Basic handling of sucking down files from a web / ftp site could also easily be handled by curl or wget in a batch file.

To specifically answer your question as to how to get the list of files on an ftp site, one sends the ftp server a LIST command and parses the results.

[ChrisL]

Actually it didn't work.

Can someone help me write a script to connect to an FTP site (with credentials) and download all the files from the root directory.

I would assume listing the files to an array or something like that. I can't get my head around some of the commands and the paramaters that go with them.

I was hoping to use InetGet command but you need to know what the file names would be to download.

Thanks in advance

Justin

Did you download the FTP.au3 file attached at the bottom of the post you linked to?

[jussie]

Yes I did download the ftp.au3 file and saw all the new functions. I don't want passwords to be visible at all as it is for a government department and they would reject if if they were visible as in being parsed in like the ftp -s command.

Thanks anyway.

[sykes]

Check these links out:

http://www.15seconds.com/issue/981203.htm

http://15seconds.com/files/981203.zip

http://support.genotec.ch/downloads/AspFTP2_Doc.htm

Heres a sample of code that downloads all files in the root directory to the directory the script resides in:

*Note - This code requires aspftp.dll to be installed ... the download link is above.*

$oMyError = ObjEvent("AutoIt.Error","MyErrFunc")

$oFtp = ObjCreate("NIBLACK.ASPFTP")
$oFtp.sServerName = "<ip address here>"
$oFtp.sUserID = "<username here>"
$oFtp.sPassword = "<password here>"
$oFtp.lTransferType = 2; sets transfer type to binary
$oFtp.bConnect

$oDirList = $oFtp.sListDir("/", 128)
    $x = StringSplit($oDirList, ";")
        For $i = 1 to $x[0]
            $oFtp.bGetFile($x[$i], @ScriptDir & "\" & $x[$i])
        Next
$oFtp.bDisconnect

Func MyErrFunc()
  $HexNumber=hex($oMyError.number,8)
    $errlog = FileOpen(@ScriptDir & "\error.log", 1)
    FileWriteLine($errlog, "Error captured - " & @MON & "/" & @MDAY & "/" & @YEAR)
    FileWriteLine($errlog, @TAB & "Error Description: " & $oMyError.description & @CRLF & _
                           @TAB & "Error Number: " & $HexNumber           & @CRLF & _
                           @TAB & "ScriptLine: " & $oMyError.scriptline  & @CRLF & _
                           @TAB & "Source: " & $oMyError.source)
  SetError(1)
Endfunc

Edited December 7, 2006 by sykes

[jussie]

Hi,

I have been trying to get some different FTP scripts to be able to connect to outside FTP sites. I have been able to get to one site by using FtpSetProxy followed by InetGet to get to a file.

I was hoping to be able to use http://www.autoitscript.com/forum/index.php?showtopic=34470 to show me an array of filenames and then download each file but it fails at _FTPConnect. I have tried it with FtpSetProxy and HttpSetProxy separately and it makes no difference. I assume the SetProxy functions only work for InetGet.

I think I am having all these troubles because my work internet has proxy authentication but it is not linked to AD so everytime I open IE or Firefox I have to put in my username and password. Maybe it is to help stop possible virus outbreaks or something.

I could test it at home without authentication but I do enough work and don't want to take it home with me.

I could use an FTP client with a scheduler but it would have to be security checked also and besides I would like to be able to make the program do what I want it to do.

I will try the code you provided sykes but I think I will still have a problem with proxy authentication again but I will see how I go.

Cheers

Justin

[Richard Robertson]

They have you log in to the proxy so they can monitor your activity. They tried it at my school but gave up because they did it inefficiently.

This may be a bad suggestion, but you could try proxy to another computer through a random port (58473) and proxy some FTP. It shouldn't be hard to implement, as long as other ports aren't blocked.

Edited December 8, 2006 by Mr Icekirby

[jussie]

Thanks mate but according to my work and the security limitations I have that would be classed as a huge security issue and would probably be marched out immediately. I guess I'll just have to do it at home.

I should have mentioned that I work for an outsourced IT company that supports company X. My company has proxy authentication which is why I wanted to test it with proxy authentication to make sure it works at all but Company X has it tied to AD and is setup via access accounts (forms and signatures required).

I have been using this as a tester (at work with proxy problems). I am not #including the functions I am just adding them to the script

#include<array.au3>

;===============================================================================

;

; Function Name: _FtpFilesListToArray()

; Description: Get Filenames, Directorys, or Both of a Directory.

; Parameter(s): $l_FTPSession - Long From _FileConnect

; $Return_type - 0 = Both Files and Directorys, 1 = Directorys, 2 = Files

; Requirement(s): DllCall, wininet.dll

; Return Value(s): On Success - 1

; On Failure - 0

; Author(s): Me

;

;===============================================================================

Func _FTPFilesListToArray($l_FTPSession, $Return_Type = 0, $l_Flags = 0, $l_Context = 0)

Dim $array, $array2d

$array = _ArrayCreate($array)

$array2d = _ArrayCreate($array2d)

$str = "dword;int64;int64;int64;dword;dword;dword;dword;char[256];char[14]"

$WIN32_FIND_DATA = DllStructCreate($str)

Local $callFindFirst = DllCall('wininet.dll', 'int', 'FtpFindFirstFile', 'long', $l_FTPSession, 'str', "", 'ptr', DllStructGetPtr($WIN32_FIND_DATA), 'long', $l_Flags, 'long', $l_Context)

If Not $callFindFirst[0] Then

MsgBox(0, "Folder Empty", "No Files Found ")

SetError(-1)

Return 0

EndIf

$ret = ""

While 1

Select

Case $Return_Type = 0 ; Folders and files

If DllStructGetData($WIN32_FIND_DATA, 1) = 16 Then

_ArrayInsert($array, 1, DllStructGetData($WIN32_FIND_DATA, 9)) ; Add Folder to top of array

Else

_ArrayAdd($array, DllStructGetData($WIN32_FIND_DATA, 9)) ; Add folder to array

EndIf

Case $Return_Type = 1 ; Folders only

If DllStructGetData($WIN32_FIND_DATA, 1) = 16 Then _ArrayAdd($array, DllStructGetData($WIN32_FIND_DATA, 9))

Case $Return_Type = 2 ; Files only

If DllStructGetData($WIN32_FIND_DATA, 1) <> 16 Then _ArrayAdd($array, DllStructGetData($WIN32_FIND_DATA, 9))

EndSelect

Local $callFindNext = DllCall('wininet.dll', 'int', 'InternetFindNextFile', 'long', $callFindFirst[0], 'ptr', DllStructGetPtr($WIN32_FIND_DATA))

If Not $callFindNext[0] Then

ExitLoop

EndIf

WEnd

$WIN32_FIND_DATA = 0

$array[0] = UBound($array) - 1

Return $array

EndFunc ;==>_FTPFilesListToArray

;===============================================================================

;

; Function Name: _FTPOpen()

; Description: Opens an FTP session.

; Parameter(s): $s_Agent - Random name. ( like "myftp" )

; $l_AccessType - I dont got a clue what this does.

; $s_ProxyName - ProxyName.

; $s_ProxyBypass - ProxyByPasses's.

; $l_Flags - Special flags.

; Requirement(s): DllCall, wininet.dll

; Return Value(s): On Success - Returns an indentifier.

; On Failure - 0 and sets @ERROR

; Author(s): Wouter van Kesteren.

;

;===============================================================================

Func _FTPOpen($s_Agent, $l_AccessType = 1, $s_ProxyName = '', $s_ProxyBypass = '', $l_Flags = 0)

Local $ai_InternetOpen = DllCall('wininet.dll', 'long', 'InternetOpen', 'str', $s_Agent, 'long', $l_AccessType, 'str', $s_ProxyName, 'str', $s_ProxyBypass, 'long', $l_Flags)

If @error OR $ai_InternetOpen[0] = 0 Then

SetError(-1)

Return 0

EndIf

Return $ai_InternetOpen[0]

EndFunc ;==> _FTPOpen()

;===============================================================================

;

; Function Name: _FTPConnect()

; Description: Connects to an FTP server.

; Parameter(s): $l_InternetSession - The Long from _FTPOpen()

; $s_ServerName - Server name/ip.

; $s_Username - Username.

; $s_Password - Password.

; $i_ServerPort - Server port ( 0 is default (21) )

; $l_Service - I dont got a clue what this does.

; $l_Flags - Special flags.

; $l_Context - I dont got a clue what this does.

; Requirement(s): DllCall, wininet.dll

; Return Value(s): On Success - Returns an indentifier.

; On Failure - 0 and sets @ERROR

; Author(s): Wouter van Kesteren

;

;===============================================================================

Func _FTPConnect($l_InternetSession, $s_ServerName, $s_Username, $s_Password, $i_Passive, $i_ServerPort = 0, $l_Service = 1, $l_Flags = 0, $l_Context = 0)

If $i_Passive == 1 Then $l_Flags = 0x08000000

Local $ai_InternetConnect = DllCall('wininet.dll', 'long', 'InternetConnect', 'long', $l_InternetSession, 'str', $s_ServerName, 'int', $i_ServerPort, 'str', $s_Username, 'str', $s_Password, 'long', $l_Service, 'long', $l_Flags, 'long', $l_Context)

If @error Or $ai_InternetConnect[0] = 0 Then

SetError(-1)

Return 0

EndIf

Return $ai_InternetConnect[0]

EndFunc ;==>_FTPConnect

;===============================================================================

;

; Function Name: _FTPGetFile()

; Description: Get file from a FTP server.

; Parameter(s): $l_FTPSession - The Long from _FTPConnect()

; $s_RemoteFile - The remote Location for the file.

; $s_LocalFile - The local file.

; $l_Flags - Special flags.

; $l_Context - I dont got a clue what this does.

; Requirement(s): DllCall, wininet.dll

; Return Value(s): On Success - 1

; On Failure - 0

; Author(s): Wouter van Kesteren

;

;===============================================================================

Func _FTPGetFile($l_FTPSession, $s_RemoteFile, $s_LocalFile, $l_Flags = 0, $l_Context = 0)

Local $ai_FTPGetFile = DllCall('wininet.dll', 'int', 'FtpGetFile', 'long', $l_FTPSession, 'str', $s_RemoteFile, 'str', $s_LocalFile, 'long', $l_Flags, 'long', $l_Context)

If @error Or $ai_FTPGetFile[0] = 0 Then

SetError(-1)

Return 0

EndIf

Return $ai_FTPGetFile[0]

EndFunc ;==>_FTPGetFile

;===============================================================================

;

; Function Name: _FTPClose()

; Description: Closes the _FTPOpen session.

; Parameter(s): $l_InternetSession - The Long from _FTPOpen()

; Requirement(s): DllCall, wininet.dll

; Return Value(s): On Success - 1

; On Failure - 0

; Author(s): Wouter van Kesteren

;

;===============================================================================

Func _FTPClose($l_InternetSession)

Local $ai_InternetCloseHandle = DllCall('wininet.dll', 'int', 'InternetCloseHandle', 'long', $l_InternetSession)

If @error OR $ai_InternetCloseHandle[0] = 0 Then

SetError(-1)

Return 0

EndIf

Return $ai_InternetCloseHandle[0]

EndFunc ;==> _FTPClose()

$server = 'xxx.xxx.xxx.xxx'

$username = 'username'

$pass = 'password'

$dllhandle = DllOpen('wininet.dll'); I have seen that this is required with BETA Versions

$Open = _FTPOpen('MyFTP Control')

If $Open = 0 Then

MsgBox(0,"FTP_Open Error",@error)

Else

MsgBox(0,"FTP Open completed","OK")

EndIf

$Conn = _FTPConnect($Open, $server, $username, $pass, 1)

If $Conn = 0 Then

MsgBox(0,"FTP_Connect Error",@error)

Else

MsgBox(0,"FTP Connect completed","OK")

EndIf

$Fileslist = _FTPFilesListToArray($Conn, 2)

_ArrayDisplay($Fileslist, 'Files List')

;after this then I would use _FTPGetFile() with the array list to download each file.

$Ftpc = _FTPClose($Open)

DllClose($dllhandle)

_FtpOpen returns OK

_FTPConnect returns -1

I am using XPSP2 and AutoIt v3.2.1.13 (beta)

Should this script work when there is no proxy AUTH??

Can someone check it for me please. I would realy need this to work as I have been working on this for about 5 days and it needs to be done by Christmas (someone before me had this project for 6 months and passed it on to me).

Thanks in advance

Justin

Edited December 8, 2006 by jussie