$FO_BINARY question about encoding

[mLipok]

While reading FileOpen() documentation I notice this following remark:

Quote

A file can be read as binary (byte) data by using FileOpen() with the binary flag.

My questions:

Q1:
Does the data read in this way have any encoding?

Q2:
Does it make sense to use this type of combination:

Local $hFileOpen = FileOpen($sFilePath, $FO_BINARY + $FO_UTF8)
Local $hFileOpen = FileOpen($sFilePath, $FO_BINARY + $FO_ANSI)

 

Edited November 11, 2023 by mLipok

[Andreik]

A1: Reading as binary without other flags I suppose AutoIt will try internally to detect the encoding.

A2: Yes, it does.

[mLipok]

So if I have stored data in binary blob in databases when I get the BinaryData I start to asking:
How to determine encoding to save data to file or even show data content ?

Edited November 11, 2023 by mLipok

[Andreik]

Jon wrote some time ago about this and I think it's how AutoIt detects encoding.

Take a look here: https://github.com/AutoItConsulting/text-encoding-detect

[jchd]

All this doesn't seem to make sense (for me at least).

If you specify reading as binary, all you get is a verbatim (binary) stream of bytes just like you could see using an hex file editor.

Mixing file encoding parameters doesn't make sense. I suspect that the core function tries hard to be clever and does overwrite binary flag when ANSI or UTFx are also there, but this is nonsensical. At a restaurant, ask your meat both raw, blue, rare, medium and well done to watch the eyes of the embarrassed waiter.

2 hours ago, Andreik said:

A1: Reading as binary without other flags I suppose AutoIt will try internally to detect the encoding.

No, it reads the file as binary, verbatim from bytes from disc. A binary file has no encoding, zero, none, nada, ketchi. For instance a jpeg image uses a file format, but no encoding.

2 hours ago, mLipok said:

So if I have stored data in binary blob in databases when I get the BinaryData I start to asking:
How to determine encoding to save data to file or even show data content ?

That question remains yours! You are supposed to know your data and act consistantly: if you read in a .gif, correctly store it as a blob, are you really going to save back the bytes converted to UTF8?

File encoding is a metadata and you'd have to handle that extra information by yourself to avoid breaking things down.

Suppose I give you a text file without telling you which encoding it was used to create it (e.g. codepage 20905, turkish IBM EBCDIC), then you're unable to process it correctly or, rather, have very little chance of doing so!

https://learn.microsoft.com/en-us/windows/win32/intl/code-page-identifiers

[Andreik]

1 hour ago, jchd said:

No, it reads the file as binary, verbatim from bytes from disc. A binary file has no encoding, zero, none, nada, ketchi. For instance a jpeg image uses a file format, but no encoding.

It's true that binary data by itself doesn't have an encoding but AutoIt seems to make an attempt to guess the encoding when you read any kind of data from file. At least this is stated in help file:

Quote

When reading without an explicit unicode mode flag, the content of the file is examined and a guess is made whether the file is UTF8, UTF16 or ANSI.

Even if it doesn't attempt to guess the encoding it makes sense to use for FileOpen() combinations like FO_BINARY + FO_UNICODE. If you don't open the file for binary read the result it's not binary data at all even if you specify some other flags regarding the encoding. So if you want binary data as result FO_BINARY is mandatory.

Edit: jchd seems to be right and AutoIt would make an attempt to guess the encoding when you read from file NOT in binary mode.

Edited November 11, 2023 by Andreik

[mLipok]

1 hour ago, jchd said:

if you read in a .gif, correctly store it as a blob, are you really going to save back the bytes converted to UTF8?

You wana say to me if I know that this is GIF, JPG, PDF then I know that it should be processed only as a BINARY which is set of bit and bytes.
In this case I can use them to save them to file (just us binary), or pass the binary to QuickPDF or any other component or API which requires BINARY not string.
Thus there is no need to convert BINARY to string and nothing more, as BINARY are set of BYTES / BIT.

If I have saved TEXT, RTF, XML or JSON file as a BINARY in databases then I still need to know what kind of data I have (and it is know as in DB i have meta data about file extension).

There meay be some untypical cases.
For example if I have XML file saved as BINARY (BLOB in DB) then I can use BinaryToString with predefined ANSI decoding.
Then why I get XML as TEXTUAL data I can notice that some tags letting me know about encoding for example:

Quote

<?xml version="1.0" encoding="UTF-8"?>

Thus I understand that I must use BinaryToString again but this time with UTF8 decoding.

 

Edited November 11, 2023 by mLipok

[mLipok]

55 minutes ago, Andreik said:

it makes sense to use for FileOpen() combinations like FO_BINARY + FO_UNICODE.

After reading @jchd answer I think now that $FO_BINARY should be used together only with:

Quote

$FO_READ (0) = Read mode (default)     $FO_APPEND (1) = Write mode (append to end of file)     $FO_OVERWRITE (2) = Write mode (erase previous contents)     $FO_CREATEPATH (8) = Create directory structure if it doesn't exist (See Remarks).

and not with:

Quote

$FO_UNICODE or $FO_UTF16_LE (32) = Use Unicode UTF16 Little Endian reading and writing mode.     $FO_UTF16_BE (64) = Use Unicode UTF16 Big Endian reading and writing mode.     $FO_UTF8 (128) = Use Unicode UTF8 (with BOM) reading and writing mode.     $FO_UTF8_NOBOM (256) = Use Unicode UTF8 (without BOM) reading and writing mode.     $FO_ANSI (512) = Use ANSI reading and writing mode.     $FO_UTF16_LE_NOBOM (1024) = Use Unicode UTF16 Little Endian (without BOM) reading and writing mode.     $FO_UTF16_BE_NOBOM (2048) = Use Unicode UTF16 Big Endian (without BOM) reading and writing mode.     $FO_FULLFILE_DETECT (16384) = When opening for reading and no BOM is present, use the entire file to determine if it is UTF8 or UTF16. If this is not

because the other one are fo the cases when you wana get textual data and not binary data.
I even suppose that when you use $FO_BINARY flag, than all higher flag are omminted by FileOpen() function

Hope @jpm will be able to look here and confirm my guessess.
 

[mLipok]

@jchd please take a look:

expandcollapse popup

#Region ; *** Dynamically added Include files ***
#include <FileConstants.au3>                                 ; added:11/11/23 23:33:38
#EndRegion ; *** Dynamically added Include files ***
#include <MsgBoxConstants.au3>
#include <StringConstants.au3>

Example()

Func Example()
    Local Const $sString = "grzegrzółka" ; $sString is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    FileDelete('Testing.txt')
    FileWrite('Testing.txt', $sString) ; here I do not care about encoding

    ; after a time I need to read file as BINARY
    Local $hFile = FileOpen('Testing.txt', $FO_READ + $FO_BINARY)
    Local $dBinary = FileRead($hFile)
    FileClose($hFile)

    Local $sConverted

    ; guessing $dBinary is ANSI trying to convert $dBinary to $sConverted using BinaryToString
    $sConverted = BinaryToString($dBinary, $SB_ANSI) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    DisplayResults($sString, $dBinary, $sConverted, "$SB_ANSI")

    ; guessing $dBinary is UTF8 trying to convert $dBinary to $sConverted using BinaryToString
    $sConverted = BinaryToString($dBinary, $SB_UTF8) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    DisplayResults($sString, $dBinary, $sConverted, "$SB_UTF8")
EndFunc   ;==>Example

; Helper function which formats the message for display.  It takes the following parameters:
; $sOriginal - The original string before conversions.
; $dBinary - The original string after it has been converted to binary.
; $sConverted- The string after it has been converted to binary and then back to a string.
; $sConversionType - A human friendly name for the encoding type used for the conversion.
Func DisplayResults($sOriginal, $dBinary, $sConverted, $sConversionType)
    MsgBox($MB_SYSTEMMODAL, "", "Original:" & @CRLF & $sOriginal & @CRLF & @CRLF & "Binary:" & @CRLF & $dBinary & @CRLF & @CRLF & $sConversionType & ":" & @CRLF & $sConverted)
EndFunc   ;==>DisplayResults

and please especially focus on my guesses about UCS2 in the comments:

Local Const $sString = "grzegrzółka" ; $sString is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
$sConverted = BinaryToString($dBinary, $SB_ANSI) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
$sConverted = BinaryToString($dBinary, $SB_UTF8) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.

Can you verify that I'm right or wrong here ?

Edited November 11, 2023 by mLipok
example modified, few times....

[Andreik]

18 minutes ago, mLipok said:

I even suppose that when you use $FO_BINARY flag, than all higher flag are omminted by FileOpen() function

This might be true and it makes more sense now what @jchd said above.

[mLipok]

sorry for the last example (modified few times)
hope finally the current version is finall.

[mLipok]

After reading:
https://stackoverflow.com/a/76229071/5314940

I saw few discussion about IsTextUnicode:
 

Spoiler

 

 

 

 

even mentioned here:

https://github.com/AutoItConsulting/text-encoding-detect/blob/master/README.md

 

 

And nothing about: IMultiLanguage2::DetectCodepageInIStream method

https://learn.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/aa740985(v=vs.85)

 

Any body know if IMultiLanguage2::DetectCodepageInIStream will be usefull to work with BinaryData ?

[mLipok]

Here's another additional puzzle:

expandcollapse popup

#include <FileConstants.au3>
#include <MsgBoxConstants.au3>
#include <StringConstants.au3>

Example()

Func Example()
    Local Const $sString = "grzegrzółka" ; $sString is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    FileDelete('Testing.txt')
    ; here I do not care about encoding
    FileWrite('Testing.txt', $sString)
    ; from HelpFile: When writing text AutoIt will write using UTF8 (without BOM) by default.

    ; after a time I need to read file as BINARY
    Local $hFile = FileOpen('Testing.txt', $FO_READ + $FO_BINARY)
    Local $dBinary = FileRead($hFile)
    FileClose($hFile)

    Local $sConverted

    ; guessing $dBinary is ANSI trying to convert $dBinary to $sConverted using BinaryToString
    $sConverted = BinaryToString($dBinary, $SB_ANSI) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    DisplayResults($sString, $dBinary, $sConverted, "$SB_ANSI - got error as Binary was UTF8 (without BOM)")

    ; guessing $dBinary is UTF8 trying to convert $dBinary to $sConverted using BinaryToString
    $sConverted = BinaryToString($dBinary, $SB_UTF8) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    DisplayResults($sString, $dBinary, $sConverted, "$SB_UTF8")

    ; save $dBinary to $sStream_Textual as ANSI even though we know it is UTF8
    Local $sStream_Textual = BinaryToString($dBinary, $SB_ANSI)
    ; here some processing on $sStream_Textual
    $sStream_Textual = StringReverse($sStream_Textual, $STR_UCS2)
    $sStream_Textual = StringReverse($sStream_Textual, $STR_UCS2)
    ; restore $sStream_Textual to binary
    $dBinary = StringToBinary($sStream_Textual, $SB_ANSI)
    ; restore $sStream_Textual to binary

    $sConverted = BinaryToString($dBinary, $SB_UTF8) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    DisplayResults($sString, $dBinary, $sConverted, "$SB_UTF8 - why this still works good even after $sStream_Textual processing as ANSI")

EndFunc   ;==>Example

; Helper function which formats the message for display.  It takes the following parameters:
; $sOriginal - The original string before conversions.
; $dBinary - The original string after it has been converted to binary.
; $sConverted- The string after it has been converted to binary and then back to a string.
; $sConversionType - A human friendly name for the encoding type used for the conversion.
Func DisplayResults($sOriginal, $dBinary, $sConverted, $sConversionType)
    MsgBox($MB_SYSTEMMODAL, "", "Original:" & @CRLF & $sOriginal & @CRLF & @CRLF & "Binary:" & @CRLF & $dBinary & @CRLF & @CRLF & $sConversionType & ":" & @CRLF & $sConverted)
EndFunc   ;==>DisplayResults

Please focus on the question in the code:
why this still works good even after $sStream_Textual processing as ANSI

I mean why using:

Local $sStream_Textual = BinaryToString($dBinary, $SB_ANSI)
    ; here some processing on $sStream_Textual
    $sStream_Textual = StringReverse($sStream_Textual, $STR_UCS2)
    $sStream_Textual = StringReverse($sStream_Textual, $STR_UCS2)
    ; restore $sStream_Textual to binary
    $dBinary = StringToBinary($sStream_Textual, $SB_ANSI)

did not broke UTF8 encoding ?
Why did I expect something to break?

Because this first conversion shows that something was broken:

$sConverted = BinaryToString($dBinary, $SB_ANSI) ; $sConverted is UCS-2 as: Native AutoIt strings use the UCS-2 character set and encoding.
    DisplayResults($sString, $dBinary, $sConverted, "$SB_ANSI - got error as Binary was UTF8 (without BOM)")

btw.
I'm sorry for my stupid questions, but I just don't understand it.

Edited November 12, 2023 by mLipok