UTF-8 encoding ? !! \u2019 \u2013 \u2018

[Alexxander]

Hello

 

i have lots of text like this

 

With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n

i have no idea what does this \u stuff means, after googling it i found that it is some sort of utf encoding

 

how could i write this text to a file without loosing any characters and getting rid of \u  ?

 

 

 

EDIT: after more digging i found that i need "Converting Unicode Entities to Unicode Text" any ideas hoe to do that with autoit ?

Edited June 4, 2015 by Alexxander

[abberration]

I am not sure that you can convert this text because it is already formatted as ansi. You could use StringRegExpReplace to replace the unwanted characters. I made a simple one that gets rid of most of the annoying ones, not just the \u. The $str is the string that you posted.

$str = "With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n"

$str = StringRegExpReplace($str, "(\\n\\u2022|\\u2013|\\u2019|\\u2018|\\n|\\)", "")

MsgBox(0, "", $str)

 

[jchd]

$string = "With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n"
$string = StringReplace($string, '\n', @CRLF)
$string = StringReplace($string, '\/', '/')
$string = Execute('"' & StringRegExpReplace($string, '\\u(\d\d\d\d)', '" & Chr($1) & "') & '"')

 

[Alexxander]

$string = "With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n"
$string = StringReplace($string, '\n', @CRLF)
$string = StringReplace($string, '\/', '/')
$string = Execute('"' & StringRegExpReplace($string, '\\u(\d\d\d\d)', '" & Chr($1) & "') & '"')

 

​thank you for your simple solution ...

but this way i loosed the symbols 

ex: 

\u2022

 

this is a bullet symbol "•"

 

any other ideas ?

[jchd]

Oh sorry, it was a typo. Chr($1) make that ChrW(0x$1)

Edited June 5, 2015 by jchd

[Alexxander]

Thanks you very much, you are awesome !

[Alexxander]

Bro i'am facing a problem  with this text using your edited code

 

The incredible new Dungeon Crawler taking the world by storm! Over 10 million downloads worldwide, with all new exciting content now available! Grow your character and restore the cosmic balance in an exciting free-to-play online adventure! AppleTell \u2013 named \u201cBest iOS Game of E3 2014\u201d\n\nTouchArcade \u2013 \u201cTaichi Panda has all the necessary components that make up a great hack 'n slash action game.\u201d\n\n148Apps \u2013 \u201cThe game looks like it has some pretty nice graphics and fast play style.\u201d\n\n==Features==\n\nA Stunning Fantasy Adventure\n\nEnter a world of goblins, bandits, and pandas as you adventure through graphically stunning maps in full 3D environments. Bash your way to the center of the mystery with a fast and fluid animation style built on full-body motion capture.\n\nHack \u2018n Slash Action\n\nFight through hordes of enemies with a fast-paced combat system. Use a mixture of speed, strength and skill, combining aerial attacks and combo moves to devastate the enemy. Explore twisting labyrinths, avoid precarious traps, and defeat powerful bosses!\n\nUpgrade Everything\n\nLevel up your character to unlock powerful new abilities. Upgrade and refine your gear to bolster your power, and unleash elemental magic through ancient runes. Develop your Pets, special battle companions with their own unique progression and skills.\n\nPlay in Real-time with Friends\n\nPlay in real-time multiplayer with friends and strangers. Become the ultimate Taichi champion with PvP arenas and battlefields. Team up with fellow adventurers in group dungeons to face off against formidable Bosses. Form Guilds with friends and compatriots, and expand your territory in Guild vs Guild war mode!\n\n==Connect==\n\nVisit our official site to learn more about Taichi Panda: http:\/\/panda.snail.com\/en \nJoin the Taichi Panda community and learn more on Facebook and our official Forum \nFacebook Page: https:\/\/www.facebook.com\/TaichiPandaEN \nCommunity Forum: http:\/\/pandabbs.snail.com\/\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Any ideas ? 

[jchd]

Jeez, I apologize for not drinking a second coffee for the second eye!

$string = "The incredible new Dungeon Crawler taking the world by storm! Over 10 million downloads worldwide, with all new exciting content now available! Grow your character and restore the cosmic balance in an exciting free-to-play online adventure! AppleTell \u2013 named \u201cBest iOS Game of E3 2014\u201d\n\nTouchArcade \u2013 \u201cTaichi Panda has all the necessary components that make up a great hack 'n slash action game.\u201d\n\n148Apps \u2013 \u201cThe game looks like it has some pretty nice graphics and fast play style.\u201d\n\n==Features==\n\nA Stunning Fantasy Adventure\n\nEnter a world of goblins, bandits, and pandas as you adventure through graphically stunning maps in full 3D environments. Bash your way to the center of the mystery with a fast and fluid animation style built on full-body motion capture.\n\nHack \u2018n Slash Action\n\nFight through hordes of enemies with a fast-paced combat system. Use a mixture of speed, strength and skill, combining aerial attacks and combo moves to devastate the enemy. Explore twisting labyrinths, avoid precarious traps, and defeat powerful bosses!\n\nUpgrade Everything\n\nLevel up your character to unlock powerful new abilities. Upgrade and refine your gear to bolster your power, and unleash elemental magic through ancient runes. Develop your Pets, special battle companions with their own unique progression and skills.\n\nPlay in Real-time with Friends\n\nPlay in real-time multiplayer with friends and strangers. Become the ultimate Taichi champion with PvP arenas and battlefields. Team up with fellow adventurers in group dungeons to face off against formidable Bosses. Form Guilds with friends and compatriots, and expand your territory in Guild vs Guild war mode!\n\n==Connect==\n\nVisit our official site to learn more about Taichi Panda: http:\/\/panda.snail.com\/en \nJoin the Taichi Panda community and learn more on Facebook and our official Forum \nFacebook Page: https:\/\/www.facebook.com\/TaichiPandaEN \nCommunity Forum: http:\/\/pandabbs.snail.com\/\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
$string = StringReplace($string, '\n', @CRLF)
$string = StringReplace($string, '\/', '/')
$string = Execute('"' & StringRegExpReplace($string, '\\u([[:xdigit:]]{4})', '" & ChrW(0x$1) & "') & '"')

 

[Alexxander]

@jchd

thanks bro, appreciate your help...

PS: Do you think we could face problems like this in the future with other texts ? (i'am not nagging), i only want to know if i should use the latest code for all my text or make more tests

[jchd]

All you may now encounter are different escapes, like \r \t or similar. Now the \uxxxx are correctly changed into the corresponding Unicode character. Once again sorry for not giving my previous answers enough attention.

[Jury]

Of course just replacing the \u with &#x and ending the code point with ; will also result in the correct char being displaied in the html - try:

<html>
<head>
<title></title>
</head>
<body>
&#x2022;<br>
&#x201d;
</body>
</html>

 

Edited June 9, 2015 by Jury