Encoding problem - detecting/converting

[mLipok]

I have such a snipet:

_Example()
Func _Example()
    Local $sStringToCheck1 = 'Jeżeli'
    Local $sStringToCheck2 = 'Jeżeli'
    MsgBox(0, '', StringToBinary($sStringToCheck1) & @CRLF & StringToBinary($sStringToCheck2))
EndFunc

This should demonstrate that first and second string are not the same (I hope so).
They have diferent encoding.

The problem is that I have single text with mixed strings.

The questions are how to:

1. detect if file contain first encoding (which is bad in my case)
2. how to convert bad (first) encoding to proper (second)

Remarks:
Remember that single file contains both strings.

 

[PACaleala]

@OP You will see the difference between strings using an editor with 'composite fonts'.

The first string contains a combining dot above(Ux0307).

The second string has a latin small letter z with dot above(Ux017C).

If you go for a solution dealing with a file have a look here: 

Have fun !

[argumentum]

On 11/13/2017 at 9:01 AM, mLipok said:

This should demonstrate that first and second string are not the same (I hope so).

_Example()
Func _Example()
    Local $sStringToCheck1 = 'Jeżeli'
    Local $sStringToCheck2 = 'Jeżeli'
    MsgBox(0, '', BinaryToString(StringToBinary($sStringToCheck1,1),4) & @CRLF & BinaryToString(StringToBinary($sStringToCheck2,1),4))
EndFunc

Hope it helps

2nd try:

_Example()
Func _Example()
    Local $sStringToCheck1 = 'Jeżeli'
    Local $sStringToCheck2 = 'Jeżeli'

    MsgBox(262144, '', BinaryToString(StringToBinary($sStringToCheck1,1),4) & " = " & isItOk($sStringToCheck1) & " = " & $sStringToCheck1 & @CR & _
        BinaryToString(StringToBinary($sStringToCheck2,1),4)  & " = " & isItOk($sStringToCheck2) & " = " & $sStringToCheck2 )
EndFunc

Func isItOk($s) ; maybe ?
    If StringToBinary(BinaryToString(StringToBinary($s,1),3)) = "" Then Return 0
    Return 1
EndFunc

 

Edited November 20, 2017 by argumentum
thinking

[jchd]

Both string use the same encoding but they differ in Unicode normalization form.

Unicode Normalization Form: see https://r12a.github.io/scripts/tutorial/part3 and search for "Precomposed vs. decomposed" to see how this works in Latin scripts. Read about Windows support: https://msdn.microsoft.com/en-us/library/windows/desktop/dd374126(v=vs.85).aspx

Change normalization form of a string: https://msdn.microsoft.com/en-us/library/windows/desktop/dd319093(v=vs.85).aspx

[argumentum]

8 minutes ago, jchd said:

Change normalization form of a string

If any of you get to write the DllCall() for the  Normaliz.dll please post the code  

[jchd]

Do you really need that?

_Example()
Func _Example()
    Local $sStringToCheck1 = 'Jeżeli'
    Local $sStringToCheck2 = 'Jeżeli'
    MsgBox(0, 'Binary', StringToBinary($sStringToCheck1) & @CRLF & StringToBinary($sStringToCheck2))
    MsgBox(0, 'Using ==', $sStringToCheck1 == $sStringToCheck2)
    MsgBox(0, 'Using =', $sStringToCheck1 = $sStringToCheck2)
    MsgBox(0, 'Using StringCompare', StringCompare($sStringToCheck1, $sStringToCheck2) ? 'Distinct' : 'Same')
EndFunc

 

[jchd]

For who wants it:

; Unicode Normalization Forms
Global Enum $UNF_NormC = 1, $UNF_NormD, $UNF_NormKC = 5, $UNF_NormKD

Func _UNF_Change($sIn, $iForm)
    If $iForm = $UNF_NormC Or $iForm = $UNF_NormD Or $iForm = $UNF_NormKC Or $iForm = $UNF_NormKD Then
        Local $aRet = DllCall("Normaliz.dll", "int", "NormalizeString", "int", $iForm, "wstr", $sIn, "int", -1, "ptr", 0, "int", 0)
        Local $tOut = DllStructCreate("wchar[" & $aRet[0] & "]")
        $aRet = DllCall("Normaliz.dll", "int", "NormalizeString", "int", $iForm, "wstr", $sIn, "int", -1, "ptr", DllStructGetPtr($tOut, 1), "int", $aRet[0])
        Return DllStructGetData($tOut, 1)
    Else
        SetError(1, 0, $sIn)
    EndIf
EndFunc


_Example()
Func _Example()
    Local $sStringToCheck1 = 'Jeżeli'
    Local $sStringToCheck2 = 'Jeżeli'
    MsgBox(0, 'Binary', StringToBinary($sStringToCheck1) & @CRLF & StringToBinary($sStringToCheck2))
    MsgBox(0, 'Using ==', $sStringToCheck1 == $sStringToCheck2)
    MsgBox(0, 'Using =', $sStringToCheck1 = $sStringToCheck2)
    MsgBox(0, 'Using StringCompare', StringCompare($sStringToCheck1, $sStringToCheck2) ? 'Distinct' : 'Same')
    MsgBox(0, '== after normalization', _UNF_Change($sStringToCheck1, $UNF_NormC) == _UNF_Change($sStringToCheck2, $UNF_NormC))
EndFunc

 

Edited May 3, 2019 by jchd