_FTP_... functions use codepage (zOS) (solved)

[Xenobiologist]

Hi folks,

I'm using the udf #include <FTPEx.au3> to "download" some files from our zOS (MVS) system via autoit FTP with _FTP_FileGet.

The FileGet works, but the codepage is wrong. Special German chars like öäüß are not transfered correctly.

Any idea how to set the codepage? Swichting between binary or ascii doesn't solve the problem.

Using the ftp command in a cmd.exe window it works like this : quote site sbd=(IBM-273,iso8859-1)

Thanks

Mega

Edited August 11, 2016 by Xenobiologist

[mLipok]

Please make any testing file (the best will be simple txt file) zip !!! them and post this ZIP archive here..
 

[Xenobiologist]

I don't know how you can test this. This is a FTP download from a zOS system.  Use a file which is in codepage EBCDIC with some öäüß and try to download it via FTPGet.

I'm using somethig like this:

#include <Array.au3>
#include <FTPEx.au3>
#include <MsgBoxConstants.au3>
#include <Process.au3>

Local $sServer = 'etet' ; zoS MVS System (IBM)
Local $sUsername = @UserName
Local $sPass = 'xxx'

Local $hOpen = _FTP_Open('MyFTP Control')
Local $hConn = _FTP_Connect($hOpen, $sServer, $sUsername, $sPass, 0, 21, $INTERNET_SERVICE_FTP, $INTERNET_FLAG_TRANSFER_ASCII)
;~ Local $hConn = _FTP_Connect($hOpen, $sServer, $sUsername, $sPass, 0, 21, $INTERNET_SERVICE_FTP, $INTERNET_FLAG_TRANSFER_BINARY)
If @error Then
    MsgBox($MB_SYSTEMMODAL, '_FTP_Connect', 'ERROR=' & @error)
Else
;~  ConsoleWrite(_FTP_Command($hConn, 'quote site sbd=(IBM-273,iso8859-1)') & @CRLF)
    ConsoleWrite(_FTP_FileGet($hConn, "'SRV.DWS.TEMP.BNKINFOD'", @ScriptDir & '\Daten1.txt', False, 0, $FTP_TRANSFER_TYPE_ASCII) & @CRLF)
EndIf
Local $iFtpc = _FTP_Close($hConn)
Local $iFtpo = _FTP_Close($hOpen)

in the file "downloaded" all looks fine, but the special chars do not: ü is } ß is ~ and so on.

 

 

Edited August 10, 2016 by Xenobiologist

[mLipok]

Did you try on another FTP Server to confirm that this is not related to "zoS MVS System (IBM)"
You can just use CesarFTP or FileZillaServer instaled on your own PC.

btw.
Please make some TXT test file zip them and post it here.

 

[Xenobiologist]

Here is a file.

I do not need to test it with another server. It should work with the one I test on.

Daten1.txt

[mLipok]

7 minutes ago, Xenobiologist said:

It should work with the one I test on.

I understand but, this is normal testing procedure. There is a need to eliminate all possible issues.

So you are talking that this: Daten1.ext was also tested by you and the problem occurs, when you get them via _FTP_FileGet(), I will do some test.

 

Edited August 10, 2016 by mLipok

[Xenobiologist]

Yes exactly, I got the file via _FTP_FileGet from the mainframe. On my windows PC the special chars are not displayed corretly in notepad++ or ultraedit or whatever.

I thought, that I have to set the codepage before using the FileGet like I did with the ftp command in a dosprompt.

Maybe I'm wrong and there is no other chance then using a StringReplace for every special char afterwards.

Edited August 10, 2016 by Xenobiologist

[BrewManNH]

In the $iFlags try setting it to $FTP_TRANSFER_TYPE_BINARY instead of the default.

[Xenobiologist]

11 minutes ago, BrewManNH said:

In the $iFlags try setting it to $FTP_TRANSFER_TYPE_BINARY instead of the default.

I did. That doesn't help. The file on the mainfraime is a normal text file. Transfering it with binary_mode creates a file like this:

 

ÇÁÄÕÙ^ÒäÙéÕÁÔÅ^ÂÁÕÒÕÁÔÅ^×Óé^ÖÙã^ãÅÉÓÖÙã^âãÙÁââÅ^ãÅÓÅÆÖÕ^ãÅÓÅÆÁç^ÒÖÕãÁÒã^ÂÕÒÂÅÙ^ÂÕÒÂÅÙäÉ^ÂÓé^ÒÙÅÉâ^ãÖäÙÉÔâ^Èãã×^ÕÙÂâÖÙÄé^ãÄÂâ^Áâðôð^Áâðóð^Áâðòð^Áâðñð^×éÂÉÓñð^×éÂÉÓñö^×éÇäåðñ^×éÂÉÓòó^ãÄÁÕÇð÷^ÁéÁÕÇðö^ÁéÁÕÇñò^ÁéÁÕÇñô^×éÇäåðó^×éÇäåðø^×éÇäåñð^×éÇäåñõ^×éÇäåñö^×éÇäåñ÷^ÕÙåÅÙÂ^æÇéÕÙ^åÖÙâãñ^åÖÙâãò^åÖÙâãó^åÖÙâãô^åÖÙâãõ^åÖÙâãö^åÖÙâã÷^åÖÙâãø^åÖÙâãù^×Óé×Æ^×ÖâãÆÁÃÈ^ÇÅÕÖÉÄ^éÅÕãÙÁÓÂÁÕÒ^åÅÙÂÁÕÄ^ÇæÔÓÉé^âÓÓÉé^ÁÙåÖÙâ^ÂÂó^@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Edited August 10, 2016 by Xenobiologist

[BrewManNH]

I just noticed you're using the ASCII flag in the FTP_Open as well, should probably change it there as well
.

[jchd]

The solution is to get the text as binary and use Windows primitives to translate between EBCDIC (which one BTW, there are so many of versions!) and native AutoIt (Unicode).

Func _CodepageToStr($sCP, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & BinaryLen($sCP) & "]")
    DllStructSetData($tText, 1, $sCP)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", BinaryLen($sCP), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", BinaryLen($sCP), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_CodepageToStr

It's now up to you to select the correct EBCDIC codepage identifier from here.

Edited August 10, 2016 by jchd

[Xenobiologist]

I tried all four combinations.

_FTP_Connect with $INTERNET_FLAG_TRANSFER_ASCII, $INTERNET_FLAG_TRANSFER_BINARY

_FTP_FileGet with $FTP_TRANSFER_TYPE_ASCII, $FTP_TRANSFER_TYPE_BINARY

Edited August 10, 2016 by Xenobiologist

[jchd]

Use Binary mode instead.

[jchd]

Please note that the example file Daten1.txt is not EBCDIC at all, as a quick look on it's hex content immediately shows.

Nonetheless you'll have to determine which 8-bit charset the zOS machine uses for this file and pass the corresponding identifier to the function posted.

[Xenobiologist]

Thanks, I tried this:

 

#include <Array.au3>
#include <FTPEx.au3>
#include <MsgBoxConstants.au3>

Local $sServer = 'etet' ; zoS MVS System (IBM)
Local $sUsername = @UserName
Local $sPass = 'xxx'

Local $hOpen = _FTP_Open('MyFTP Control')
;~ Local $hConn = _FTP_Connect($hOpen, $sServer, $sUsername, $sPass, 0, 21, $INTERNET_SERVICE_FTP, $INTERNET_FLAG_TRANSFER_ASCII)
Local $hConn = _FTP_Connect($hOpen, $sServer, $sUsername, $sPass, 0, 21, $INTERNET_SERVICE_FTP, $INTERNET_FLAG_TRANSFER_BINARY)
If @error Then
    MsgBox($MB_SYSTEMMODAL, '_FTP_Connect', 'ERROR=' & @error)
Else
;~  ConsoleWrite(_FTP_Command($hConn, 'quote site sbd=(IBM-273,iso8859-1)') & @CRLF)
;~  ConsoleWrite(_FTP_FileGet($hConn, "'SRV.DWS.TEMP.BNKINFOD'", @ScriptDir & '\Daten1.txt', False, 0, $FTP_TRANSFER_TYPE_ASCII) & @CRLF)
    ConsoleWrite(_FTP_FileGet($hConn, "'SRV.DWS.TEMP.BNKINFOD'", @ScriptDir & '\Daten1.txt', False, 0, $FTP_TRANSFER_TYPE_BINARY) & @CRLF)
EndIf
Local $iFtpc = _FTP_Close($hConn)
Local $iFtpo = _FTP_Close($hOpen)

$c = FileRead(@ScriptDir & '\Daten1.txt')

FileWrite(@ScriptDir & '\Daten2.txt', _CodepageToStr($c, 1141) & @CRLF)

Func _CodepageToStr($sCP, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & BinaryLen($sCP) & "]")
    DllStructSetData($tText, 1, $sCP)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", BinaryLen($sCP), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", BinaryLen($sCP), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_CodepageToStr

This way I get a second file Daten2.txt which now shows öäüß corretly. The only problem I've to figure out is how to insert a CRLF back into the file. The file contains currently only one long line.

[jchd]

There is already a Windows-style line termination at the end of the Daten1.txt file (0x0D 0x0A).

Can you post the binary data as received (written as binary file)?

Edited August 10, 2016 by jchd

[Xenobiologist]

Good morning,

#include <Array.au3>
#include <FTPEx.au3>
#include <MsgBoxConstants.au3>

Local $sServer = 'etet' ; zoS MVS System (IBM)
Local $sUsername = @UserName
Local $sPass = 'xxx'

Local $hOpen = _FTP_Open('MyFTP Control')
;~ Local $hConn = _FTP_Connect($hOpen, $sServer, $sUsername, $sPass, 0, 21, $INTERNET_SERVICE_FTP, $INTERNET_FLAG_TRANSFER_ASCII)
Local $hConn = _FTP_Connect($hOpen, $sServer, $sUsername, $sPass, 0, 21, $INTERNET_SERVICE_FTP, $INTERNET_FLAG_TRANSFER_BINARY)
If @error Then
    MsgBox($MB_SYSTEMMODAL, '_FTP_Connect', 'ERROR=' & @error)
Else
;~  ConsoleWrite(_FTP_Command($hConn, 'quote site sbd=(IBM-273,iso8859-1)') & @CRLF)
;~  ConsoleWrite(_FTP_FileGet($hConn, "'SRV.DWS.TEMP.BNKINFOD'", @ScriptDir & '\Daten1.txt', False, 0, $FTP_TRANSFER_TYPE_ASCII) & @CRLF)
    ConsoleWrite(_FTP_FileGet($hConn, "'SRV.DWS.TEMP.BNKINFOD'", @ScriptDir & '\Daten1.txt', False, 0, $FTP_TRANSFER_TYPE_BINARY) & @CRLF)
EndIf
Local $iFtpc = _FTP_Close($hConn)
Local $iFtpo = _FTP_Close($hOpen)

$c = FileRead(@ScriptDir & '\Daten1.txt')
FileDelete(@ScriptDir & '\Daten3.txt')
FileWrite(@ScriptDir & '\Daten3.txt' , StringStripWS(StringReplace(_CodepageToStr($c, 1141), ';      ', ';' & @CRLF), 7) & @CRLF)

Func _CodepageToStr($sCP, $iCodepage = Default)
    If $iCodepage = Default Then $iCodepage = Int(RegRead("HKLM\SYSTEM\CurrentControlSet\Control\Nls\Codepage", "OEMCP"))
    Local $tText = DllStructCreate("byte[" & BinaryLen($sCP) & "]")
    DllStructSetData($tText, 1, $sCP)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", BinaryLen($sCP), _
            "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", BinaryLen($sCP), _
            "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc   ;==>_CodepageToStr

This way the file looks the way it should. Of course it is not the best way to add the CR but hey, it works :-)

Thanks to all!

Mega

[Xenobiologist]

Still interested in how to set a codepage before using FileGet or FilePut.

Without setting it I always need to manipulate the files before transfering them.

Edited August 11, 2016 by Xenobiologist

[jchd]

AFAIK FTP protocol has only two modes: text or nibary (Oops, binary!). I.e. it has no knowledge or say about the content of what's being got/put. Such "decoration" is higher in the ISO model, so belongs to code processing things before put or after get.

Edited August 11, 2016 by jchd

[Xenobiologist]

Then what is this?

When I use the ftp.exe from C:\Windows\System32 in a DosPrompt.

ftp etet

Type in user and password

Then it looks like this

ftp>

Now I can type in this: ftp> quote site sbd=(IBM-273,iso8859-1)
and get this --> 200 SITE command was accepted

After that the transfer is managed correctly. The umlauts etc. are shown correctly.

It seems the command tells the server to act different.