Jump to content

BinaryToString for unsupported language. [Solved]

carl1905
 Share

Recommended Posts

jchd Universalist

jchd

Posted
Posted

These are ANSI codepages which will be supported iff your system is setup for such support. Which binary encoding is used in your input? UTF8, UTF16-LE, UTF16-BE?

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites
carl1905 Seeker

carl1905

Posted
  • Author
Posted
39 minutes ago, jchd said:

These are ANSI codepages which will be supported iff your system is setup for such support. Which binary encoding is used in your input? UTF8, UTF16-LE, UTF16-BE?

Hi, jchd.

The one that I want to do it is, extract text from *.dat files. These are script files which is written by Shift-JIS (Japanese).

Image

I made a tool to extract the text. However, BinarytoString doesn't support change binary to Shift-JIS. 

$Str = BinaryToString("0x" & StringRegExpReplace($Str,"0x",""),5)

There are no Global Const for Shift-JIS in StringConstants.au3. There are only ANSI, UTF16LE/BE, and UTF-8. 

Global Const $SB_ANSI = 1
Global Const $SB_UTF16LE = 2
Global Const $SB_UTF16BE = 3
Global Const $SB_UTF8 = 4

However, I'm look forward to add it if it is possible. 

Global Const $SB_Shift_JIS = 5

 

Link to comment
Share on other sites
jchd Universalist

jchd

Posted
Posted

You can't do it that way. Rather try this: 

#include <FileConstants.au3>

Global Const $CP_SHIFT_JIS = 932

; get your JIS input from full binary file
Local $hInp = FileOpen("Shift-JIS.txt", $FO_BINARY)
Local $Inp = FileRead($hInp)
FileClose($hInp)

; or fake it as in your example
$Inp = Binary("0x8366815B835E82CC835F83458393838D815B836882C98EB8947382B582DC82B582BD42")
Local $tInp = DllStructCreate("byte[" & BinaryLen($Inp) & "]")
DllStructSetData($tInp, 1, $Inp)

Local $sJIS = _CodepageStructToString($tInp, $CP_SHIFT_JIS)
MsgBox(0, "JIS text", $sJIS)

Func _CodepageStructToString($tText, $iCodepage)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", DllStructGetSize($tText), _
                                "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("Kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", DllStructGetSize($tText), _
                        "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc

 

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites
carl1905 Seeker

carl1905

Posted
  • Author
Posted (edited)
On 2016. 4. 20. at 10:09 AM, jchd said:

You can't do it that way. Rather try this: 

#include <FileConstants.au3>

Global Const $CP_SHIFT_JIS = 932

; get your JIS input from full binary file
Local $hInp = FileOpen("Shift-JIS.txt", $FO_BINARY)
Local $Inp = FileRead($hInp)
FileClose($hInp)

; or fake it as in your example
$Inp = Binary("0x8366815B835E82CC835F83458393838D815B836882C98EB8947382B582DC82B582BD42")
Local $tInp = DllStructCreate("byte[" & BinaryLen($Inp) & "]")
DllStructSetData($tInp, 1, $Inp)

Local $sJIS = _CodepageStructToString($tInp, $CP_SHIFT_JIS)
MsgBox(0, "JIS text", $sJIS)

Func _CodepageStructToString($tText, $iCodepage)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", DllStructGetSize($tText), _
                                "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("Kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", DllStructGetSize($tText), _
                        "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc

 

Hmm.. I'm not sure how to adjust your code for my code.

Sample file

Here is my au3 code for extract text from *.dat file. It works well for English.dat. But it doesn't work for Japanese.dat file. 

#include <FileConstants.au3>

Global Const $CP_SHIFT_JIS = 932

$Path = FileOpenDialog("Select the dat file", @ScriptDir, "dat files (*.dat)",1)
If @error = 1 Then Exit
$File = fileopen($Path,16)
$Name = CompGetFileName($Path)
$Id = FileRead($File,2)
If $Id <> "XL" Then
    MsgBox(0,"Error!",'Wrong id: "'&BinaryToString($Id)&'"'&@CR&'Expected: "XL"')
    Exit
 EndIf
 FileSetPos($File,8,0)
 $Files = FileRead($File,2)
 FileSetPos($File,12,0)
 $Base = _BinaryToInt32(FileRead($File,4))
FileSetPos($File,$Base,0)

Dim $Text
For $i = 1 to $Files
    $Offset = _BinaryToInt32(FileRead($File,4)) + $Base
    $pos = FileGetPos($File)
    FileSetPos($File,$Offset,0)
    $S = ""
    $Str = ""
    Do
        $Str &= $S
        $S = FileRead ($File, 1)
     Until $S = 0
       $Str = BinaryToString("0x" & StringRegExpReplace($Str,"0x",""),4)
       $Str = StringRegExpReplace($Str,@CRLF,"<cf>")
       $Str = StringRegExpReplace($Str,@LF,"<lf>")
       $Str = StringRegExpReplace($Str,@CR,"<cr>")
       $Text &= $Str & @CRLF
    FileSetPos($File,$pos,0)
Next
$hAllTextFil = FileOpen ($Name&".txt", 2+32)
FileWrite ($hAllTextFil, $Text)
FileClose ($hAllTextFil)
TrayTip ("Exporter", "Finish!", 3)
sleep (3000)
Func CompGetFileName($Path)
If StringLen($Path) < 4 Then Return -1
$ret = StringSplit($Path,"\",2)
If IsArray($ret) Then
Return $ret[UBound($ret)-1]
EndIf
If @error Then Return -1
EndFunc
Func _BinaryToInt32($Binary)
    Return BitAND(Int(Binary($Binary)), 0xFFFFFFFF)
EndFunc

 

Edited by carl1905
Link to comment
Share on other sites
jchd Universalist

jchd

Posted
Posted

Oops, sorry I missed the notification for your previous post. Yet you're lucky: I never answer question related to games (see forum rules why), but I considered your script isn't actually "interacting with one".

Glad to see you finally got it working. It's now up to you to decide which codepage constant has to be used with each file.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites
carl1905 Seeker

carl1905

Posted
  • Author
Posted (edited)
3 hours ago, jchd said:

Oops, sorry I missed the notification for your previous post. Yet you're lucky: I never answer question related to games (see forum rules why), but I considered your script isn't actually "interacting with one".

Glad to see you finally got it working. It's now up to you to decide which codepage constant has to be used with each file.

It's fine. Thank you.

 

Edited by carl1905
Link to comment
Share on other sites
Trong Universalist

Trong

Posted
Posted (edited)

Your  Sample file  not is Binary file, You can read file as text and show it.
 

Please create new topic!

29 minutes ago, carl1905 said:

It's fine. Thank you. However, can I ask one more question? For example, uhm.. there is a sample.txt file.

What I want to convert

And I want to change the text to another form using some kind of table.txt How can I transform it?

 

Edited by VIP

Regards,
 

Link to comment
Share on other sites
jchd Universalist

jchd

Posted
Posted

@VIP,

Why do you believe the sample isn't binary? It is. Period.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites
jchd Universalist

jchd

Posted
Posted

That's utterly unreliable! Look at the file with a hex editor and you see why.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites
Trong Universalist

Trong

Posted
Posted

Good Lucky: 

#EndRegion Author C490C3A06F2056C4836E2054726F6E67
; Export Language PS XL

Local $sFilePath = FileOpenDialog("Select the dat file", @ScriptDir, "dat files (*.dat)", 1)
If @error = 1 Then Exit

XLangPSExport($sFilePath)
If Not @error Then
    TrayTip("Exporter", "Finish!", 3)
Else
    Exit MsgBox(0, "Error!", 'Wrong Lang ID: Only for XL')
EndIf

Func XLangPSExport($sFilePath, $sOutFile = Default)
    Local $hFile = FileOpen($sFilePath, 16)
    If FileRead($hFile, 2) <> "XL" Then Return SetError(1, 0, 0)
    FileSetPos($hFile, 8, 0)
    Local $sLineContent = FileRead($hFile, 2)
    FileSetPos($hFile, 12, 0)
    Local $sOffset = BitAND(Int(Binary(FileRead($hFile, 4))), 0xFFFFFFFF)
    FileSetPos($hFile, $sOffset, 0)
    If $sOutFile = Default Then $sOutFile = $sFilePath & ".lang"
    Local $sNewText, $sPos, $tmpHex, $Offset, $tmpStr
    For $i = 1 To $sLineContent
        $Offset = BitAND(Int(Binary(FileRead($hFile, 4))), 0xFFFFFFFF) + $sOffset
        $sPos = FileGetPos($hFile)
        FileSetPos($hFile, $Offset, 0)
        While 1
            $tmpHex = FileRead($hFile, 1)
            If @error Or $tmpHex = 0 Then ExitLoop
            $tmpStr &= $tmpHex
        WEnd
        FileSetPos($hFile, $sPos, 0)
        $tmpStr = StringReplace($tmpStr, StringToBinary(@CRLF), StringToBinary("<CF>"));~ 0x0D0A
        $tmpStr = StringReplace($tmpStr, StringToBinary(@LF), StringToBinary("<LF>"));~ 0x0A
        $tmpStr = StringReplace($tmpStr, StringToBinary(@CR), StringToBinary("<CR>"));~ 0x0D
        $sNewText &= $tmpStr & "0D0A";@CRLF
        $tmpStr = ""
    Next
    While StringInStr($sNewText, "0D0A0D0A")
        $sNewText = StringReplace($sNewText, "0D0A0D0A", "0D0A")
    WEnd
    $sNewText = "0x" & StringReplace($sNewText, "0x", "")
    FileClose($hFile)
    $hFile = FileOpen($sOutFile, 2 + 8 + 16)
    FileWrite($hFile, Binary($sNewText))
    FileClose($hFile)
EndFunc   ;==>XLangPSExport

 

If your problem has been resolved, please modify Topic Title add  [Solved]
Click LIKE THIS to thanks!

Regards,
 

Link to comment
Share on other sites
carl1905 Seeker

carl1905

Posted
  • Author
Posted
56 minutes ago, VIP said:

Good Lucky: 

#EndRegion Author C490C3A06F2056C4836E2054726F6E67
; Export Language PS XL

Local $sFilePath = FileOpenDialog("Select the dat file", @ScriptDir, "dat files (*.dat)", 1)
If @error = 1 Then Exit

XLangPSExport($sFilePath)
If Not @error Then
    TrayTip("Exporter", "Finish!", 3)
Else
    Exit MsgBox(0, "Error!", 'Wrong Lang ID: Only for XL')
EndIf

Func XLangPSExport($sFilePath, $sOutFile = Default)
    Local $hFile = FileOpen($sFilePath, 16)
    If FileRead($hFile, 2) <> "XL" Then Return SetError(1, 0, 0)
    FileSetPos($hFile, 8, 0)
    Local $sLineContent = FileRead($hFile, 2)
    FileSetPos($hFile, 12, 0)
    Local $sOffset = BitAND(Int(Binary(FileRead($hFile, 4))), 0xFFFFFFFF)
    FileSetPos($hFile, $sOffset, 0)
    If $sOutFile = Default Then $sOutFile = $sFilePath & ".lang"
    Local $sNewText, $sPos, $tmpHex, $Offset, $tmpStr
    For $i = 1 To $sLineContent
        $Offset = BitAND(Int(Binary(FileRead($hFile, 4))), 0xFFFFFFFF) + $sOffset
        $sPos = FileGetPos($hFile)
        FileSetPos($hFile, $Offset, 0)
        While 1
            $tmpHex = FileRead($hFile, 1)
            If @error Or $tmpHex = 0 Then ExitLoop
            $tmpStr &= $tmpHex
        WEnd
        FileSetPos($hFile, $sPos, 0)
        $tmpStr = StringReplace($tmpStr, StringToBinary(@CRLF), StringToBinary("<CF>"));~ 0x0D0A
        $tmpStr = StringReplace($tmpStr, StringToBinary(@LF), StringToBinary("<LF>"));~ 0x0A
        $tmpStr = StringReplace($tmpStr, StringToBinary(@CR), StringToBinary("<CR>"));~ 0x0D
        $sNewText &= $tmpStr & "0D0A";@CRLF
        $tmpStr = ""
    Next
    While StringInStr($sNewText, "0D0A0D0A")
        $sNewText = StringReplace($sNewText, "0D0A0D0A", "0D0A")
    WEnd
    $sNewText = "0x" & StringReplace($sNewText, "0x", "")
    FileClose($hFile)
    $hFile = FileOpen($sOutFile, 2 + 8 + 16)
    FileWrite($hFile, Binary($sNewText))
    FileClose($hFile)
EndFunc   ;==>XLangPSExport

 

If your problem has been resolved, please modify Topic Title add  [Solved]
Click LIKE THIS to thanks!

Thanks you. Your coed can extract text and save it as Shift-JIS from binary file.

My codes(add function from jchd) will extract Japanese text and save it as UTF-16LE. 

#include <FileConstants.au3>
Global Const $CP_SHIFT_JIS = 932

$Path = FileOpenDialog("Select the dat file", @ScriptDir, "dat files (*.dat)",1)
If @error = 1 Then Exit
$File = fileopen($Path,16)
$Name = CompGetFileName($Path)
$Id = FileRead($File,2)
If $Id <> "XL" Then
    MsgBox(0,"Error!",'Wrong id: "'&BinaryToString($Id)&'"'&@CR&'Expected: "XL"')
    Exit
 EndIf
 FileSetPos($File,8,0)
 $Files = FileRead($File,2)
 FileSetPos($File,12,0)
 $Base = _BinaryToInt32(FileRead($File,4))
FileSetPos($File,$Base,0)

Dim $Text
For $i = 1 to $Files
    $Offset = _BinaryToInt32(FileRead($File,4)) + $Base
    $pos = FileGetPos($File)
    FileSetPos($File,$Offset,0)
    $S = ""
    $Str = ""
    Do
        $Str &= $S
        $S = FileRead ($File, 1)
     Until $S = 0
       $Inp = Binary("0x" & StringRegExpReplace($Str,"0x",""))

       Local $tInp = DllStructCreate("byte[" & BinaryLen($Inp) & "]")
       DllStructSetData($tInp, 1, $Inp)

       Local $Str = _CodepageStructToString($tInp, $CP_SHIFT_JIS)
       $Str = StringRegExpReplace($Str,@CRLF,"<cf>")
       $Str = StringRegExpReplace($Str,@LF,"<lf>")
       $Str = StringRegExpReplace($Str,@CR,"<cr>")
       $Text &= $Str & @CRLF
    FileSetPos($File,$pos,0)
Next
$hAllTextFil = FileOpen ($Name&".txt", 2+32)
FileWrite ($hAllTextFil, $Text)
FileClose ($hAllTextFil)
TrayTip ("Exporter", "Finish!", 3)
sleep (3000)
Func CompGetFileName($Path)
If StringLen($Path) < 4 Then Return -1
$ret = StringSplit($Path,"\",2)
If IsArray($ret) Then
Return $ret[UBound($ret)-1]
EndIf
If @error Then Return -1
EndFunc
Func _BinaryToInt32($Binary)
    Return BitAND(Int(Binary($Binary)), 0xFFFFFFFF)
 EndFunc
 Func _CodepageStructToString($tText, $iCodepage)
    Local $aResult = DllCall("kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", DllStructGetSize($tText), _
                                "ptr", 0, "int", 0)
    Local $tWstr = DllStructCreate("wchar[" & $aResult[0] & "]")
    $aResult = DllCall("Kernel32.dll", "int", "MultiByteToWideChar", "uint", $iCodepage, "dword", 0, "struct*", $tText, "int", DllStructGetSize($tText), _
                        "struct*", $tWstr, "int", $aResult[0])
    Return DllStructGetData($tWstr, 1)
EndFunc

 

Link to comment
Share on other sites
Trong Universalist

Trong

Posted
Posted (edited)

You can use it to convert:

Function: 

#include <WinAPI.au3>

;~ _ConvertEncodingFile(@DesktopDir & "\sample.txt", @DesktopDir & "\NEW_sample.txt", 932) ;iCodepage: SHIFT_JIS = 932

Func _ConvertEncodingFile($sFilePathIN, $sFilePathOUT = Default, $iCodePage = 65001) ;iCodepage: UTF-8=65001
    ; iCodepage: https://msdn.microsoft.com/en-us/library/windows/desktop/dd317756(v=vs.85).aspx
    If Not FileExists($sFilePathIN) Then Return SetError(1, 0, 0)
    Local $hOpen = FileOpen($sFilePathIN, 16384)
    If $sFilePathOUT = Default Then $sFilePathOUT = $sFilePathIN
    Local $sContent = FileRead($hOpen)
    If @error Or StringStripWS($sContent, 8) = "" Then Return SetError(2, 0, 0)
    FileClose($hOpen)
    $sContent = _WinAPI_WideCharToMultiByte($sContent, $iCodePage, 1)
    If @error Then Return SetError(3, @error, 0)
    $hOpen = FileOpen($sFilePathOUT, 2 + 8 + 16)
    FileWrite($hOpen, $sContent)
    If @error Then Return SetError(4, @error, 0)
    FileClose($hOpen)
    Return 1
EndFunc   ;==>_ConvertEncodingFile

 

Edited by Trong
Function Convert

Regards,
 

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...