File handling functions not recognising files with non ASCII/ANSI characters in filename

[Greengenie]

I've just upgraded to Windows 11 and many apps save files with pictograms and other non ASCII/ANSI characters (even periods seem to be encoded with '46' in there somewhere, possibly %46). If I create an array of file paths (_FileListToArrayRec()) the paths are all in the array, but the file handling functions (e.g. FileExists()) do not recognize the file paths that contain special characters.

How can we reference these file paths in standard AutoIt3 functions?

My reason for doing this is to simply FileMove() to eliminate the special characters so my custom function looks at each character in the $sFileName and either transforms top-bit-set characters to a similar character in the range ASCII 32 to 126 or eliminates troublesome characters like ":". This results in transforming "6.2.25" to "64624625", which looks like it the period is encoded using the ASCII value. In a URL it would be %46. On my old Windows 7 machine the apps generally save files with names only using ASCII 32 to 255 so I never had a problem.

[argumentum]

make a zip file with these filenames ( just 10 or so files, and should be zero size/empty ) otherwise, I personally have nothing to troubleshoot.
Post that file somewhere and I'll figure something out.

Look into _WinAPI_MultiByteToWideChar() ?

Edited February 9 by argumentum

[Jos]

There normally is no need to mess around with their names. 

Post some script have the issue you mention and we can have a look.

[Greengenie]

Thanks for the reply. I tried zipping a couple of files but Windows standard compression refuses to zip them ... because the filenames contain special characters!

[Greengenie]

The script simply uses _FileListToArrayRec() then for each path in the array I do the following (script simplified to reduce size of this post)

$allowable = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 !'()-[]_`"

$new="" 

_PathSplit($f[$i],$sDrive,$sDir,$sFileName,$sExtension) ; $f is the array of path names
    For $j=1 To StringLen($sFileName)
        $c=StringMid($sFileName, $j, 1)
        If StringInStr($allowable, $c)=0 Then
          $a=Asc($c)
          If $a=160 Then $a = " "  ; hard space to normal space

          many other similar expressions in ElseIfs to change top bit set chars to ASCII 32 to 126 equivalent

        Else

           $new &= $c  ; leave character as it was

        EndIf

        $new &= $a

  Next

  $new is then expanded to full path by concatenating $sDrive & $sDir & $sFileName & $sExtension

  FileMove($f[$i], $new)

I tried pasting the file names into SciTE but it just shows the special characters as "?".

Sorry, I'm not familiar with the formatting of these posts to show nice colored code snippets.

My main reason for doing this is that many files get save to a NAS which is accessed by a number of computers running various operating systems that may not be able to handle the special characters (e.g. Linux, RISCOS, Windows7).

The attached screen shot shows what the filenames look like in Windows 11 File Explorer.  Thanks for your interest in helping.

Edited February 9 by Jos
added codebox

[Jos]

This is not a complete runnable script as it is missing the input array. Please post something that replicates your problem even when you manually fill the array and describe what is going wrong.  

Use <> to post code . 

[Greengenie]

Here is a test script. I've simplified the character code changing If statement. The path needs to be entered without quote marks. Try a file with a name including periods. Should work. Then try a file with special characters (probably won't work). Note that even if the FileMove() is successful @error seems to be 0, which in the AutoIt3 documentation says "Failure: 0 if source cannot be moved or if dest already exists and flag=0". Strange because flag=1 to enable overwrite. Anyway, that's another issue.

expandcollapse popup

#include <File.au3>
#include <Array.au3>

Global $sDrive=""
Global $sDir=""
Global $sFileName=""
Global $sExtension=""
Global $allowable = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 !'()-[]_`"

$file = InputBox("Test", "Enter file path")
rename($file)
Exit

Func rename($f)
  _PathSplit($f,$sDrive,$sDir,$sFileName,$sExtension)
  Local $new = "",$c,$a,$j
  For $j=1 To StringLen($sFileName)
    $c=StringMid($sFileName,$j,1)
    If StringInStr($allowable,$c)=0 Then
      $a=Asc($c)
      If $a=160 Then
        $c = " "
      ElseIf $a>=192 And $a<=198 Then
        $c="A"
      ; other similar ElseIf lines
      Else
        $c=""
      EndIf
    EndIf
    $new = $new & $c
  Next

  If $new <> $sFileName Then
    $new = $sDrive & $sDir & $new & $sExtension
    FileMove($f, $new, 1)
    $err = @error
    MsgBox(0,"Debug","err=" & $err)
  EndIf
EndFunc

 

[Jos]

I have tested this script, and it is working fine for me. I've tested with a filename containing a Special character ä and got renamed correctly.
Furthermore, I don't see anything in your testscript with FileExists() usage and not working as you claimed in your initial post?

It is possible that there is a rights issue or the target file is already opened by another program, like Explore, as that can cause the file being used when "looking" at its directory.

Edited February 9 by Jos

[Greengenie]

OK, thanks for looking at this. It turns out there are several issues that somewhat masked what was happening...

First, the original script exhibiting the issue is quite a lot larger than the test script I posted, and it obtains the path (either a directory or a single file) from the clipboard. Next, it turns out that the path names with the special characters exhibit different character codes when the path goes through the clipboard from when they are obtained using _FileListToArrayRec(). This was confusing.

In order to simplify the script for posting here, I used InputBox but then I was having to use the "Copy as path" and paste it into the InputBox, but that was encapsulating the path in quote marks which were being passed into the rename() function and then FileExists() said it didn't exist.

Finally, as I mentioned earlier, I thought that FileMove() was returning 0 (failure) even when the renaming worked. It was my error, I was looking at @error instead of the return value. So I while I was monitoring @error it appeared to be failing but actually it was doing the rename. I was checking file attributes and permissions, all which turned out to be irrelevant, but while doing that I found that FileSetAttribute() was failing, but it wasn't the issue anyway.

So, the edited original script is now working (now that I understand what is going on). And AutoIt3 is handling the path names correctly. I just wish people wouldn't use all the fancy characters and encodings!

Thanks again for your help. You know what its like when stuff that worked on an old machine fails on a new machine and you think its a real problem, but when you start discussing it with others, you end up fixing your own problem.

[Greengenie]

Following up on this issue, in case anyone finds it useful:

If a file path name contains special characters and the path is put onto the clipboard, FileExists($path) returns False. However, if the path is loaded into an array with $a=_FileListToArrayRec($dir...), then file handling functions can use parameters like $a[$n] and they work.

So, on Windows 7, I am able to copy a file to the clipboard, extract FileNameW (the full path name) from it and pass that to my script, where FileExists($path) returns True and other functions can also handle $path. All is fine. 

If I do the exact same thing on Windows 11, the FileNameW text does not exist in the clipboard and it is necessary to get the path using

$clipCF_Text = _ClipBoard_GetDataEx($CF_TEXT)
 $data = DllStructCreate("char Text[8192]", $clipCF_TEXT)
 $path = DllStructGetData($data, "Text")

but in this case FileExists($path) returns False. There is a difference between "Copy as path" and "Copy", where the former delimits the path with double quotes, but even after stripping them the path cannot be found using the file handling functions.

What I've had to do, is load an array with all the path names in the containing directory using _FileListToArrayRec($dir...), then search the array for the file I want to process, and use the found element in file handling functions. This works, but its a lot of additional programming.

I expect this issue has something to do with character encoding but I've no idea how path names in clipboard data should be handled.

[jchd]

Grab the content of the clipboard with ClipGet() or use _ClipBoard_GetData($CF_UNICODETEXT).

Also if you insist on using _ClipBoard_GetDataEx, then also use $CF_UNICODETEXT and $data = DllStructCreate("wchar Text[8192]", $clipCF_TEXT)

Else you're  requesting Windows to convert the (native) Unicode text in the clipboard to the default 8-bit codepage, making "special" characters (as you name them) either emasculated or lost in translation.