Issue with accented characters (CLI → clibpoard)

[littlebigman]

Hello,

I use the following script to 1) run a command-line application (ie. "DOS" app), 2) copy its output to the clipboard, and 3) paste it to Notepad as a simple way to search the data.

I notice that accented chars are wrong, eg. "é" becomes "é"

Is it because of the DOS to Windows character pages?

[…]
;run CLI app
Local $iPID = Run($WholeLine, $szDir, @SW_HIDE,$STDOUT_CHILD)
If @error Then Exit MsgBox($MB_ICONERROR, "Error", "Running failed.")
Local $sOutput = StdoutRead($iPID)
If @error Then Exit ConsoleWrite("Error returned by StdOutRead! @error=" & @error & ", @extended=" & @extended & @CRLF)
While True
    $sOutput &= StdoutRead($iPID)
    If @error Then ExitLoop
WEnd

;Poor man's search: copy STDOUT into clipboard, launch Notepad, paste
ClipPut($sOutput)
$iPID = Run("notepad.exe")
Opt("WinTitleMatchMode", 2) ;1=start, 2=subStr, 3=exact, 4=advanced, -1 to -4=Nocase
Local $hHdl = WinWait("[TITLE:Untitled - Notepad2; CLASS:Notepad2]","",10)
if  not $hHdl then Exit MsgBox($MB_ICONERROR,"Error","Windows not found")
Sleep(500)
Send("^v") ;paste
Sleep(1000)
Send("^{HOME}") ;jump to first line in notepad

Thank you.

[Nine]

Take a look at  _WinAPI_OemToChar()

[littlebigman]

Thanks. Looks like I'm using it wrong: This time, "é" turns into "+¬".

;run CLI app
Local $iPID = Run($WholeLine, $szDir, @SW_HIDE,$STDOUT_CHILD)
If @error Then Exit MsgBox($MB_ICONERROR, "Error", "Running failed.")
Local $sOutput = StdoutRead($iPID)
If @error Then Exit ConsoleWrite("Error returned by StdOutRead! @error=" & @error & ", @extended=" & @extended & @CRLF)
While True
    $sOutput &= StdoutRead($iPID)
    If @error Then ExitLoop
    ;ConsoleWrite($sOutput & @CRLF)
WEnd

;convert from DOS to ANSI
$sOutput = _WinAPI_OemToChar ($sOutput)

;Poor man's search: copy STDOUT into clipboard, launch Notepad, paste
ClipPut($sOutput)

[water]

I use the following function (taken from the forum):

; #INTERNAL_USE_ONLY# ===========================================================================================================
; Name ..........: _Ansi2Oem
; Description ...: Translates a string from ANSI to OEM.
; Syntax ........: _Ansi2Oem($sText)
; Parameters ....: $sText - Text to translate
; Return values .: Success - Returns the translated text
;                  Failure - None
; Author ........: Zedna
; Modified ......:
; Remarks .......:
; Related .......: Oem2Ansi (see link below)
; Link ..........: https://www.autoitscript.com/forum/topic/201398-german-umlauts-and-console-programs/?do=findComment&comment=1445253
; Example .......:
; ===============================================================================================================================
Func _Ansi2Oem($sText)
    Local $aText
    $aText = DllCall('user32.dll', 'Int', 'CharToOem', 'str', $sText, 'str', '')
    Return ConsoleWrite($aText[2])
EndFunc   ;==>_Ansi2Oem

 

[Nine]

Lets try something simple that you and I can actually run :

#include <WinAPIConv.au3>
#include <Constants.au3>

Local $iPID = Run(@ComSpec & " /c dir c:\", "", @SW_SHOW, $STDERR_MERGED)
ProcessWaitClose($iPID)
Local $sStream = StdoutRead($iPID)
MsgBox($MB_OK, "", _WinAPI_OemToChar($sStream))

Does that work ?

[ioa747]

Out of curiosity, have you checked this? or unchecked?

Settings > Time & Language >  Language >  Administrative language settings >  Change system locale

[littlebigman]

2 hours ago, water said:

I use the following function (taken from the forum):

; #INTERNAL_USE_ONLY# ===========================================================================================================
; Name ..........: _Ansi2Oem
; Description ...: Translates a string from ANSI to OEM.
; Syntax ........: _Ansi2Oem($sText)
; Parameters ....: $sText - Text to translate
; Return values .: Success - Returns the translated text
;                  Failure - None
; Author ........: Zedna
; Modified ......:
; Remarks .......:
; Related .......: Oem2Ansi (see link below)
; Link ..........: https://www.autoitscript.com/forum/topic/201398-german-umlauts-and-console-programs/?do=findComment&comment=1445253
; Example .......:
; ===============================================================================================================================
Func _Ansi2Oem($sText)
    Local $aText
    $aText = DllCall('user32.dll', 'Int', 'CharToOem', 'str', $sText, 'str', '')
    Return ConsoleWrite($aText[2])
EndFunc   ;==>_Ansi2Oem

 

The input file is UTF8, so that didn't work.

2 hours ago, Nine said:

Does that work ?

Nope, still wrong characters.

;Local $iPID = Run(@ComSpec & " /c dir c:\", "", @SW_SHOW, $STDERR_MERGED)
Local $iPID = Run(@ComSpec & " /c type test.utf8.txt", "", @SW_SHOW, $STDERR_MERGED)

 

1 hour ago, ioa747 said:

Out of curiosity, have you checked this? or unchecked?

Settings > Time & Language >  Language >  Administrative language settings >  Change system locale

That solved it (reboot required.)

Thanks all.

[jchd]

FYI, on my system (Win10 x64 20h2) checking this setting has side effects: when I run a very simple piece of code from Scite, random strings are displayed in Scite console.

[ioa747]

14 minutes ago, littlebigman said:

That solved it (reboot required.)

checked ? or unchecked?

[littlebigman]

1 minute ago, ioa747 said:

checked ? or unchecked?

From unchecked to checked.

7 minutes ago, jchd said:

FYI, on my system (Win10 x64 20h2) checking this setting has side effects: when I run a very simple piece of code from Scite, random strings are displayed in Scite console.

Thanks for the tip. If it occurs on mine, I'll revert back, and use an alternative — the application I need to use also as the option of saving the output to a file instead.

Checking the option in Windows added a second keyboard layout while I only had one before. No biggie, though.

[Nine]

As @jchd, checking this box does not work for me either.  I am unable to write french characters correctly in Scite console.  Did not test it extensively as I have no benefit of using it, or I may say, I have no issue without it...

[jchd]

@Nine, I fixed that by 1) setting this In SciTeUserProperties:

code.page=65001
output.code.page=65001

and using a good Unicode font:

font.monospace=font:DejaVu Sans Mono,size:11

Then I use CW() instead of ConsoleWrite():

Func CW($s = "")
    If @Compiled Then
        _CUI_ConsoleWrite($s)
    Else
        _ConsoleWrite($s)
    EndIf
EndFunc   ;==>CW


Func _CUI_ConsoleWrite(ByRef $s)
    Local Static $hDll = DllOpen("kernel32.dll")
    Local Static $hCon = __CUI_ConsoleInit($hDll)
    DllCall($hDll, "bool", "WriteConsoleW", "handle", $hCon, "wstr", $s & @LF, "dword", StringLen($s) + 1, "dword*", 0, "ptr", 0)
    Return
EndFunc   ;==>_CUI_ConsoleWrite


; internal use only
Func __CUI_ConsoleInit(ByRef $hDll)
    DllCall($hDll, "bool", "AllocConsole")
;~  The following 2 lines don't work for compiled scripts due to a MS bug
;~      see last post in thread https://developercommunity.visualstudio.com/t/setconsoleoutputcpcp-utf8-results-in-code-page-850/413190
;~      where MS support acknowledges it's a (still unfixed) bug in Windows itself
;~      So you can only display current locale codepage using current console font in a console when your script is compiled
;~      When running your script from SciTE, the UTF8 setting of SciTE overrides this bug and Unicode can be displayed
;~  DllCall("Kernel32.dll", "bool", "SetConsoleCP", "uint", 65001)
;~  DllCall("Kernel32.dll", "bool", "SetConsoleOutputCP", "uint", 65001)
    Return DllCall($hDll, "handle", "GetStdHandle", "int", -11)[0]
EndFunc   ;==>__CUI_ConsoleInit


; Unicode-aware ConsoleWrite
Func _ConsoleWrite(ByRef $s)
    ConsoleWrite(BinaryToString(StringToBinary($s & @LF, 4), 1))
EndFunc   ;==>_ConsoleWrite

 

[jchd]

Using the above changes, run this:

$s = "Большая проблема  大问题  बड़ी समस्या  مشكلة كبيرة Test Title 😭" & @LF
$s &= "Our familly " & ChrW(0xD83D) & ChrW(0xDC68) & ChrW(0xD83C) & ChrW(0xDFFB) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC69) & ChrW(0xD83C) & ChrW(0xDFFF) & ChrW(0x200D) & ChrW(0xD83D) & ChrW(0xDC66) & ChrW(0xD83C) & ChrW(0xDFFD & @LF)

CW($s)

Result:

Большая проблема  大问题  बड़ी समस्या  مشكلة كبيرة Test Title 😭
Our familly 👨🏻‍👩🏿‍👦🏽

[ioa747]

more...

https://www.softwarert.com/change-language-windows-command-prompt/

[littlebigman]

On 4/21/2024 at 6:12 PM, littlebigman said:

Checking the option in Windows added a second keyboard layout while I only had one before. No biggie, though.

This seems to work to remove that second keyboard and go back to normal. I had a bad feeling about making changes to the Region section…and wasn't disappointed.