[resolved] Charset problem with POST method

[phamhoangthi]

Hello, everybody, I want to use Post method, something like Curl in Php, to post the data to the server and then, receive the respond to do something.
But the respond is not in Charset utf-8, so I can't read it.
Here is my code (after seeing some example sample codes):

func POST($URL, $DATA)

        Local $oHTTP = ObjCreate("WinHttp.WinHttpRequest.5.1")
        $oHTTP.Open("POST", $URL, False)
        If (@error) Then Return SetError(1, 0, 0)
        $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
        $oHTTP.SetRequestHeader("Accept-Charset", "UTF-8,*")
        $oHTTP.Send($Data)
        If (@error) Then Return SetError(2, 0, 0)
        If ($oHTTP.Status <> 200) Then Return SetError(3, 0, 0)
        Return SetError(0, 0, $oHTTP.ResponseText) //Resolved: Change this to ResponseBody and everything's ok  
EndFunc


Global $TEXT = POST("http://mylink.com/ba.php","var1=13567567290")
  FileWriteLine("result.html",$TEXT)

and the result file, I can't read anything.

How can I get passed it?
Please help me.

Edited October 5, 2015 by phamhoangthi

[jchd]

Since you asked for UTF8, you got UTF8, which is exactly what you display. Convert $TEXT as received from UTF8 to native AutoIt string (UTF16-LE) and you'll be fine.

$TEXT = BinaryToString(StringToBinary($TEXT, <correct parameter #1>), <correct parameter #2>)

Your mission -if you accept it- consists in reading the help to find the missing bits.

[phamhoangthi]

Since you asked for UTF8, you got UTF8, which is exactly what you display. Convert $TEXT as received from UTF8 to native AutoIt string (UTF16-LE) and you'll be fine.

$TEXT = BinaryToString(StringToBinary($TEXT, <correct parameter #1>), <correct parameter #2>)

Your mission -if you accept it- consists in reading the help to find the missing bits.

Thanks for helping.

I tried to convert my text to ANSI, and then, UTF8,
But "số điện thoại" became "S��? �?i��?n thoại" after converting

[jchd]

Do you display the string with ConsoleWrite or MsgBox? Conversion to ANSI turns non-ANSI characters into question marks; don't do that.

The former displays strings converted to Windows charset (so-called ANSI) by default while the second displays UTF16 Unicode strings directly.

To make SciTE display UTF8 you have to put the following in User Properties file (see Options):

code.page=65001
output.code.page=65001

Then you can use ConsoleWrite directly to display UTF8 data. To display native AutoIt strings (UTF16) you can use this:

Local $str = "số điện thoại"

Func _StrToUtf8($s)
    Return BinaryToString(StringToBinary($s, 4), 1)
EndFunc

ConsoleWrite(_StrToUtf8($str) & @LF)

NOTE: I justy checked and it seems that the latest release of SciTE doesn't display UTF8 correctly regardless of code.page setting. So use MsgBox with plain UTF16 strings (AutoIt native) for now until this is fixed.

[phamhoangthi]

Do you display the string with ConsoleWrite or MsgBox? Conversion to ANSI turns non-ANSI characters into question marks; don't do that.

The former displays strings converted to Windows charset (so-called ANSI) by default while the second displays UTF16 Unicode strings directly.

To make SciTE display UTF8 you have to put the following in User Properties file (see Options):

code.page=65001
output.code.page=65001

Then you can use ConsoleWrite directly to display UTF8 data. To display native AutoIt strings (UTF16) you can use this:

Local $str = "số điện thoại"

Func _StrToUtf8($s)
    Return BinaryToString(StringToBinary($s, 4), 1)
EndFunc

ConsoleWrite(_StrToUtf8($str) & @LF)

NOTE: I justy checked and it seems that the latest release of SciTE doesn't display UTF8 correctly regardless of code.page setting. So use MsgBox with plain UTF16 strings (AutoIt native) for now until this is fixed.

Sorry, My SciTE editor can't display unicode anymore after adding code.page=65001, it shows: "s? di?n tho?i" instead of "số điện thoại" and I don't know how to use UTF16 string.
I am originally a web coding and I am still new at this...
In html document, I just need to put a <meta charset='utf8' to solve my problem

Edited October 3, 2015 by phamhoangthi

[jchd]

That was the point of my bottom note in bold. This is being investigated and will be fixed shortly.

BUT, if it displays what you show, that means that you don't feed UTF8 but a conversion to ANSI. Show a short sample of failing code.

[phamhoangthi]

That was the point of my bottom note in bold. This is being investigated and will be fixed shortly.

BUT, if it displays what you show, that means that you don't feed UTF8 but a conversion to ANSI. Show a short sample of failing code.

Yes, I did see your notice and I just didn't understand why.
This is my simple example, these written both in a file and in the console

 

$TEXT = _StrToUtf8("Ä�ã Ä�ến thÆ° viá»�n") ;This is result of POST method from a webpage. This originally is: "đã đến thư viện" that I can read in my language.

Func _StrToUtf8($s)
    Return BinaryToString(StringToBinary($s, 1), 4)
EndFunc
ConsoleWrite($TEXT)
FileDelete("Google.html")
  FileWriteLine("Google.html",$TEXT)

 

 

Console:

--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
??ã ???n thu vi??n+>17:31:57 AutoIt3.exe ended.rc:0
+>17:31:57 AutoIt3Wrapper Finished.

Output file:

�?ã �?ến thư vi�?n

But I need

đã đến thư viện

 

Edited October 3, 2015 by phamhoangthi

[jchd]

Yes, I did see your notice and I just didn't understand why.

Most likely a bug in the SciTE settings.

Your script is mixing things up. The original string "đã đến thư viện" has the following UTF8 representation (in hex):

0xC491C3A320C491E1BABF6E207468C6B0207669E1BB876E

The string in the firrst line of your example is already UTF8 but gets loaded as a series of individual UTF16 characters in AutoIt. So you shouldn't convert that into UTF8, that won't work.

; say $UTF8 is the result you get from the server
Local $UTF8 = Binary("0xC491C3A320C491E1BABF6E207468C6B0207669E1BB876E")
MsgBox(0, "", BinaryToString($UTF8, 4))

 

[phamhoangthi]

My first line, "�ã �ến thư vi�n" is not only like this in SciTE editor consolewrite but also in output.html file.
I'm sorry that nothing is new, With 

BinaryToString($UTF8, 4)

in html output file, I see this:

"Cảm ơn bạn �?ã �?ến thư vi�?n, hẹn gặp bạn lần sau!"
meaning:
"Thank you for coming to library, see next time!"

"Cảm ơn bạn and hẹn gặp bạn lần sau! " is great, but not "�?ã �?ến thư vi�?n,"
 I think  those codes can't convert these characters: "Đ, ệ etc..."

 

 

 

Edited October 3, 2015 by phamhoangthi

[jchd]

What do you use to "see" that? UTF8 text has to be displayed accordingly to this encoding. When you "see" "Cảm ơn bạn �?ã �?ến thư vi�?n, hẹn gặp bạn lần sau!" that means that the text is somewhere converted to ANSI, where non-ANSI characters show up as �

 I think  those codes can't convert these characters: "Đ, ệ etc..."

Of course yes! Correct display only need to handle conversions correctly. The code in the first post results in an english page for me (coded in UTF8).

[phamhoangthi]

What do you use to "see" that? UTF8 text has to be displayed accordingly to this encoding. When you "see" "Cảm ơn bạn �?ã �?ến thư vi�?n, hẹn gặp bạn lần sau!" that means that the text is somewhere converted to ANSI, where non-ANSI characters show up as �

Of course yes! Correct display only need to handle conversions correctly. The code in the first post results in an english page for me (coded in UTF8).

 

Yeah! I view the output by using Notepad++, Chrome browser, same result.
And now I have to use Stringreplace or regurlar expression or something like that to fix some error characters? 
I don't think this is a good idea, why I can't receive the respond from a server as Utf8?
Why in Php, The result always looks good?
I am still new in this and I have many questions to answer.

[jchd]

Can you post a working example of a website which returns a vietnamese UTF8 answer (I wild guess it's what you target)?

There's no need to StringReplace or regexp anything.

[phamhoangthi]

yeah, I use a Php class to do that, (curl)
Of course this is only an example that I won't  stop at method process()  but anyway this method will return the source of the page after sending POST request. Example: The search result.

class test{
                private $ch; 
                private $page; 
                private $name; 
                private $domain;
                private $parse;
                
            function __construct($var, $studentid){  //declare the curl
                $parse= parse_url($var); //parse_url
                $param = array(
                'var1'=> '123456',  
                'var2' => '654321');                
                $this->domain = $parse['host'];
                $this->ch = curl_init($var);
                curl_setopt($this->ch, CURLOPT_RETURNTRANSFER,true);                
                curl_setopt($this->ch, CURLOPT_POST, count($param));  
                curl_setopt($this->ch,CURLOPT_POSTFIELDS, $param);
    
            }
            
            function process(){ //exec the curl
                
                return $this->page = curl_exec($this->ch); //This variable is result from POST METHOD, and Ofcourse I just need to set a Meta charset html tag to display to browser. No need converting.        
                curl_close($this->ch); //close curl
            }
        
        }
        
        
            $test = new test('http://link.com',$_REQUEST['input']); //Start 'curling'
            $test->process();

 

There's no need to StringReplace or regexp anything.
so how to fix the problem?

 

 

Edited October 3, 2015 by phamhoangthi

[jchd]

Still nothing AutoIt reproducible here.

BTW, you can download the new beta AutoIt3Wrapper just produced for fixing this issue with displaying UTF8.

[phamhoangthi]

Still nothing AutoIt reproducible here.

BTW, you can download the new beta AutoIt3Wrapper just produced for fixing this issue with displaying UTF8.

I can't find new version while the latest version is still in 20-9-2015 
Do you have any solutions for my problem about receiving respond from POST request?

Edited October 3, 2015 by phamhoangthi

[Danyfirex]

does your browser show the correct output? I'm sure is server side issue.

Also you can use utf8_encode/utf8_decode in server side.

Regards

[phamhoangthi]

does your browser show the correct output? I'm sure is server side issue.

Also you can use utf8_encode/utf8_decode in server side.

Regards

I am sorry that I can't touch to  the server side.
There is no way else?
Update: thank you for your answer, 
it's probably the cause of the problem.
If I post directly to an online website, the result, as far as jchd said, is already in Utf8, no need any converting funtion, it shows well.
But If I download the page (saving from browser) and put to localhost, the problem starts happening.

Edited October 3, 2015 by phamhoangthi

[jchd]

Can you post the output verbatim, or a reproducer script? There may be a simple way to restore the content.

[mLipok]

I can't find new version while the latest version is still in 20-9-2015 Do you have any solutions for my problem about receiving respond from POST request?

http://www.autoitscript.com/autoit3/scite/download/beta_SciTE4AutoIt3/

 

[phamhoangthi]

Yes, Here I have 2 files, source and result,
I send POST request to source file (on my localhost) and receive respond as result file.
There is only one paragraph in each file.

result.html

source.html

Edited October 3, 2015 by phamhoangthi