Unicode and StringRegExpReplace issues

[lgwapnitsky]

No problem but I'm beginning to think you might want to do another replacement after the loop and that is to delete invalid characters. That way you probably wouldn't need to validate. I can whip up that SRER quickly since I already had it earlier but didn't think to save it.

That's what this is for:

$Testarray = StringRegExp($sStr, "([[:alnum:]\-\s\x27\x9e\x9f\xc9-\xff]+)", 3)
MsgBox(0, "Result2", _ArrayToString($Testarray, ""))

Actually, this code just concatenates out the bad characters. My SRER is working fine right now...for the characters I need.

Edited May 19, 2010 by lgwapnitsky

[GEOSoft]

Instead of botherng to create the array with the RegEx you might want to think about just deleting enything except the allowable characters.

$sUserName = StringRegExpReplace($sUserName, "[^[:alnum:]\-\s\x27\x9e\x9f\xc9-\xff]", "")

In this one, anything inside the first "[" and the last "]" is excluded from deletion.

[lgwapnitsky]

Instead of botherng to create the array with the RegEx you might want to think about just deleting enything except the allowable characters.

$sUserName = StringRegExpReplace($sUserName, "[^[:alnum:]\-\s\x27\x9e\x9f\xc9-\xff]", "")

In this one, anything inside the first "[" and the last "]" is excluded from deletion.

Very clean...I like it. Thanks.

[lgwapnitsky]

Very clean...I like it. Thanks.

btw - shouldn't it be \xc0 instead of \xc9? That's where A with grave is...

[GEOSoft]

btw - shouldn't it be \xc0 instead of \xc9? That's where A with grave is...

Should it? I just copied the characters you were using, however \xc0 is correct. Easily tested if you follow that link I gave you.

[Ascend4nt]

Though I don't want to threadjack this erm thread, I'm curious.. if we want to check for specific unicode characters, we can do it as long as we know the range of hex codes, like for Cyrillic, the range is something like 0x400-0x4FFF and some in the 0x5xxx range.

I just did a test on Cyrillic characters, mixed with Japanese and english, and the following PCRE picked out the correct Cyrillic characters, but I'm kinda lost because I can't get hex codes for all character sets.

Grab (a subset of) Cyrillic Unicode characters: "[\x{0400}-\x{04FF}]"

GEOSoft, do you know of any good sites which list the hex-code ranges for different languages? I just sorta stumbled on what I did, and it's really a general overview. And I can't get my brain around this 'U+' notation. Plus now I'm reading that 16-bits isn't going to capture the entire Unicode character set (there's 32-bit versions though..)

[GEOSoft]

Though I don't want to threadjack this erm thread, I'm curious.. if we want to check for specific unicode characters, we can do it as long as we know the range of hex codes, like for Cyrillic, the range is something like 0x400-0x4FFF and some in the 0x5xxx range.

I just did a test on Cyrillic characters, mixed with Japanese and english, and the following PCRE picked out the correct Cyrillic characters, but I'm kinda lost because I can't get hex codes for all character sets.

Grab (a subset of) Cyrillic Unicode characters: "[\x{0400}-\x{04FF}]"

GEOSoft, do you know of any good sites which list the hex-code ranges for different languages? I just sorta stumbled on what I did, and it's really a general overview. And I can't get my brain around this 'U+' notation. Plus now I'm reading that 16-bits isn't going to capture the entire Unicode character set (there's 32-bit versions though..)

I don't have a link to tha page I found when I first went looking for this information. I searched the forums earlier to find the script I did for getting the Unicode characters (Full as I recall) using a RegEx. I still haven't found it and I think that may be becuase I either didn't have the correct serch term or the limit on the number of search returns cut off the list.

[GEOSoft]

Wait a second here. I do remember that I found the correct codes for the sets some place on http://unicode.org/

[jchd]

Ascend4nt,

The right spot for much Unicode stuff is there. Be sure to visit UniView, Converters and other shortcuts Ishida offers.

You can also benefit looking there to discover in detail which script maps where and in which Unicode plane.

Finally, a Unicode primer, not very tech but correct. While you are there, take a few hours to read the 1075 articles Joel posted on his blog. Most are worth it.

[Ascend4nt]

Awesome, thanks for the links guys!

[GEOSoft]

Awesome, thanks for the links guys!

In case you are interested I Pmed you another link that you might be intereset in keeping track of.Also there was a minor update to the toolkit today, same link.

[lgwapnitsky]

Should it? I just copied the characters you were using, however \xc0 is correct. Easily tested if you follow that link I gave you.

Strange- I thought I posted my final testing code yesterday, but I don't see it now:

#include <String.au3>

Global $UnicodeReplacements[19][2] = [["\xc0-\xc5","A"],["\xe0-\xe5","a"], ["\xc6","AE"], ["\xc7","C"],["\xe7","c"], _
                                        ["\xc8-\xcb","E"], ["\xe8-\xeb","e"], ["\xcc-\xcf","I"], ["\xec-\xef","i"], _
                                        ["\xd1","N"],["\xf1","n"], ["\xd2-\xd6\xd8","O"], ["\xf2-\xf6\xf8","o"], _
                                        ["\xd9-\xdc","U"], ["\xf9-\xfc","u"], ["\xdd","Y"], ["\xfd","y"]]


            
$ib = InputBox("Test","enter test string")

$ib = StringRegExpReplace($ib, "[^[:alnum:]\-\s\x27\x9e\x9f\xc9-\xff]", "")

for $ur = 0 to UBound($UnicodeReplacements) -1
    $ib = StringRegExpReplace($ib, "[" & $UnicodeReplacements[$ur][0] & "]", $UnicodeReplacements[$ur][1])
next

ConsoleWrite(_StringProper($ib) & @CRLF)

Also, I'm curious about the toolkit link you PM'ed to Ascend4nt...

Edited May 20, 2010 by lgwapnitsky

[GEOSoft]

I think you did and if I'm correct it's way back on page 1 of this thread. Your array is different and I'm guessing that you already allowed for the changes.

ConsoleWrite(), contrary to popular belief is not a great method of testing. Use MsgBoxes instead. For me it doesn't matter becuase it's only a single click to convert them anyway and in your case there is only one to be done. This portion of the code seems to be fine when I test it but I'm curious to know how it is with th rest of your script.

It was just a link to the revisions for the latest version of PCRE which AutoIt isn't using yet.

Edited May 20, 2010 by GEOSoft

[lgwapnitsky]

I think you did and if I'm correct it's way back on page 1 of this thread. Your array is different and I'm guessing that you already allowed for the changes.

ConsoleWrite(), contrary to popular belief is not a great method of testing. Use MsgBoxes instead. For me it doesn't matter becuase it's only a single click to convert them anyway and in your case there is only one to be done. This portion of the code seems to be fine when I test it but I'm curious to know how it is with th rest of your script.

It was just a link to the revisions for the latest version of PCRE which AutoIt isn't using yet.

Ahhh...i'm not awake yet.

One more question...for now...

\x2122 doesn't seem to be matching ™ (which I'm also still trying to figure out why I can't type it on my keyboard). This is for a different portion of the utility.

Thanks

[GEOSoft]

Ahhh...i'm not awake yet.

One more question...for now...

\x2122 doesn't seem to be matching ™ (which I'm also still trying to figure out why I can't type it on my keyboard). This is for a different portion of the utility.

Thanks

Give me a while to get some caffeine into me and I'll look at that.

[jchd]

You can type it with your keyboard using the compose sequence Alt0153 and it indeed is \x2122 or 8482 decimal.

[lgwapnitsky]

You can type it with your keyboard using the compose sequence Alt0153 and it indeed is \x2122 or 8482 decimal.

That works...some help this is: http://www.fileformat.info/info/unicode/char/2122/index.htm

[GEOSoft]

Actually that should be \x99 which also doesn't work. I suspect it's a bug in the PCRE engine and not in AutoIt.

[jchd]

Hi George,

Beware that \x99 is the Latin1 ANSI assignment for ™ (the Unicode code point \U2122).

Charmap.exe shows it nicely, once you check the "advanced view" checkbox (I don't know how it gets called in non frenchy Windows). You can switch the displayed charset between Unicode, Ansi (select your codepage), OEM, ...

[lgwapnitsky]

Actually that should be \x99 which also doesn't work. I suspect it's a bug in the PCRE engine and not in AutoIt.

I also tried the literal (\xe2\x84\xa2) which did not work either.