This is a weird one

[TheSaint]

Not sure where to put this one, so I guess here is as good a place as any. I am not sure if it is even strictly an AutoIt question, but it only happens with my AutoIt created programs.

I use INI files a lot, have done so for many years. They are quick and simple and ideal most of the time. When necessary I use XML or SQL.

Every now and then, not often, I have what appears to be some kind of corruption issue. Over the years it has happened to me at least half a dozen times, perhaps more and perhaps more have gone unnoticed. This is with various programs I have coded.

What happens, is that the first entry in an INI file is no longer being read by INI functions.

I don't know the cause, but I have found four ways to fix the issue.

1. Put a blank line at the start of the file. NOTE - If I remove that line, it goes back to not working.

2. Copy & paste into another file, and replace the original with that.

3. Use FileOpen, FileRead, FileWrite to create a new file to replace the original with. Some of my programs now have a '/fix' parameter on the command-line, to do this at need, via a program shortcut.

4. Do the same as '3' above but wipe original file and write back to that.

I am guessing it is some unprintable character or a header issue maybe. But nothing is obvious or seen when opening in an editor like SciTE.

You can delete the first few entries of the file, and the new first entry becomes unreadable via INI functions.

Can anyone here definitively say what is happening?
If I can prevent this issue, it would be much very appreciated, and you will gain my heartfelt gratitude.

The source for some of the INI file entries comes from a calibre XML type 'metadata.opf' file, which I read line by line, rather than bother with XML functions. I only extract from about 6 lines, and unnecessary content at start and end of each line is stripped.

Without further ado, here is my latest program, as a script and executable, which I am still developing. Plus the INI file in question - Ebooks.ini. I've also included the 'metadata.opf' file for that first (non-read entry) ebook.

Booklist To HTML.zip

Just select the first Author List entry shown - A. D. Davies.
In the combo control for Titles, you will only see one ebook entry.
At the start of the INI file, there are two ebooks listed for that author, and only the second one is the one listed in program.
If you create and rename the 'Test.ini' file to 'Ebooks.ini', and use that, you will see that both are now listed.

NOTE - You can create a shortcut to the script or EXE file, with the parameter '/fix' and run the program with that to get a fixed file called 'Test.ini'.

WARNING - Script is written using AutoIt v3.3.0.0 ... an oldy but a goody. I have tested it with almost the latest version of AutoIt (v3.3.14.2) as well.

EDIT
Per request of TheDcoder who is on Linux right now. Test.ini

Edited July 31, 2018 by TheSaint

[careca]

Never happened to me. And i converted your ebooks ini to utf-8 instead of what it was which is utf-8-bom, and it works, showing 2 titles.

[TheSaint]

17 minutes ago, careca said:

Never happened to me. And i converted your ebooks ini to utf-8 instead of what it was which is utf-8-bom, and it works, showing 2 titles.

Mmm that's interesting and possibly a clue. Perhaps IniWrite is responsible when the file is first created ... or a later update?

Ahhhh a light bulb just went off. I have wiped the file using the _FileCreate function, I think. That said, I have also used that in the 4th fix option.

I just use the AutoIt INI functions much the same as I have always used them, and created thousands of files, and only a few give this rare issue.

Thanks for checking and the feedback.

Edited July 31, 2018 by TheSaint

[TheDcoder]

Looks like @careca has beat me to it, I examined both the good and bad ini files in a hex editor to look at the raw bytes inside the file. Surprise surprise, the bad file has a byte ordering mark (BOM) at the start of the file (EF BB BF)... this is missing in the good file. It is a common practice to use the BOM to assist programs in identifying the encoding correctly, but unfortunately it has side-effects if the file isn't identified correctly.

I reckon it is a classic case of an encoding mis-interpertation with the Ini functions in AutoIt. it is probably best to use FileOpen with the $FO_UTF8_NOBOM flag to create a file without the (probably) offending BOM character.

Also I would like to mention again that I have not confirmed anything but the glaring difference both the files have. I am running Linux at the moment and cannot test anything.

Edited July 31, 2018 by TheDcoder

[TheSaint]

20 minutes ago, TheDcoder said:

Looks like @careca has beat me to it, I examined both the good and bad ini files in a hex editor to look at the raw bytes inside the file. Surprise surprise, the bad file has a byte ordering mark at the start of the file (EF BB BF)...

I reckon it is a classic case of an encoding mis-interpertation with the Ini functions in AutoIt. it is probably best to use FileOpen with the $FO_UTF8_NOBOM flag to create a file without the (probably) offending BOM character.

Thanks bud.

I'm not sure i understand what all that means, but I am guessing one of the AutoIt functions caused the issue - IniWrite or _FileCreate or both together.

It is certainly not intuitive to use FileOpen when using INI functions ... it is usually an either or ... and using the right tool for the job.

So really, this kind of puts a spanner in the works and a trap for the unwary. The INI file seemed to be working fine to me, until I noticed the missing entry when testing my HTML creating code.

Perhaps wiping the contents of the INI file using _FileCreate, put in in a state that IniWrite later made a mistake with, when I started to populate the INI file again? I say that, because the INI file was working fine when I first populated it ... first few entries anyway.

Perhaps as you suggested on IRC, I should do something like use the $FC_UTF8 flag with _FileCreate, when I use it.

Edited July 31, 2018 by TheSaint

[TheDcoder]

5 minutes ago, TheSaint said:

I'm not sure i understand what all that means, but I am guessing one of the AutoIt functions caused the issue - IniWrite or _FileCreate or both together.

_FileCreate might be the culprit, IniWrite if it is written correctly (and I think it is) then it should be automatically be able to handle new file create with the appropriate encoding. I'd recommend going the FileOpen route to create a new file... or even better just use IniWrite while being file agnostic in all situations, it should handle file creation.

---

8 minutes ago, TheSaint said:

It is certainly not intuitive to use FileOpen when using INI functions ... it is usually an either or ... and using the right tool for the job.

That is correct, like I have said above, use IniWrite for best results. But FileOpen is also a better alternative when used with appropriate flags:

22 minutes ago, TheDcoder said:

it is probably best to use FileOpen with the $FO_UTF8_NOBOM flag to create a file without the (probably) offending BOM character.

---

9 minutes ago, TheSaint said:

Perhaps as you suggested on IRC, I should do something like use the $FC_UTF8 flag with _FileCreate, when I use it.

Sorry, I mis-remembered the function and flag names in IRC, please refer to my post for the correct names

[TheSaint]

It does make you ponder, how many INI files exist out there, where the first entry is not being read.

It is perhaps not something you would immediately notice, especially with lots of entries, and whatever usage was involved.

[Earthshine]

this is why we regression test with standard baseline data... lol and we try to automate such testing if possible.

[TheSaint]

4 minutes ago, TheDcoder said:

Sorry, I mis-remembered the function and flag names in IRC, please refer to my post for the correct names

Mmmm there are no flags for _FileCreate.

FileOpen is not really appropriate for INI stuff, and certainly no replacement for _FileCreate.

I am kind of only left with IniDelete, which is painful for a large INI file.

[TheSaint]

9 minutes ago, Earthshine said:

this is why we regression test with standard baseline data... lol and we try to automate such testing if possible.

I have no clue what you mean by that. It's probably above my paygrade.

I failed my Master's Degree in Nerdspeak.

But thanks ...... I think.

Edited July 31, 2018 by TheSaint

[Earthshine]

LOL. When we make changes to our source code and regenerated, or build the binaries, we need to test it to make sure we didn't break anything. For you, reading a standard ini file and processing it correctly (known input and known output vs actual output) would be that particular test to see if you are missing anything, or if it added anything you don't want or misbehaves in general.

I try to automate using AutoIt wherever I can to test our stuff so our other teams can test all the new functionality.

Edited July 31, 2018 by Earthshine

[TheSaint]

Mmm, that sounds wise, but I am not sure I can test something that is not gonna tell me there is an error.

It has to know that first INI entry is there but unreadable.

And even if i could devise a test, am I really gonna have that run every time I use an INI function?
Just not intuitive or feasible.

Surely this is some kind of bug that needs fixing?
And rather have it fixed where it should be, than me apply the bandaid approach.

P.S. At this point, I could just elect for the simple approach, and make sure all my INI files have a blank line at the start. I do that once, and then only again, if _FileCreate is used. I say that method, because at this point, I don't know for sure what creates the issue, and it could potentially happen any time. Sure, I could go to the extra trouble of making sure, every INI file is created first with FileOpen, but what if that isn't enough?

Of course, I could just FileDelete the file instead of using _FileCreate, if that is indeed the culprit.

Edited July 31, 2018 by TheSaint

[Earthshine]

no, you only test when you are building new versions of your tools/apps. you most certainly could code up a test with a known ini, you would know what to expect on the output, no?

software dev/build/test is my life so sorry if I get ahead. my dream is to FULLY automate all testing of our software products. it's happening, gradually

 

so, you make changes to your Teracopy Timer app, you should: regression test it fully to make sure you have not broken anything that used to work, then test all your new functionality.

 

that may be too much but that is the idea. cheers.

Edited July 31, 2018 by Earthshine

[TheSaint]

2 minutes ago, Earthshine said:

no, you only test when you are building new versions of your tools/apps. you most certainly could code up a test with a known ini, you would know what to expect on the output, no?

That doesn't seem applicable to me. What am I testing for and what with? This is a unique INI that is always changing. How does it know what the output should be?

[TheSaint]

31 minutes ago, Earthshine said:

so, you make changes to your Teracopy Timer app, you should: regression test it fully to make sure you have not broken anything that used to work, then test all your new functionality.

Test it with what?

I always manually test my programs after updating, time permitting of course, which effects how fully that is. The bigger and more complex my programs become the more that needs testing ... or because they are free, I leave that up to time and my usual usage ... or other users. I certainly don't retest every single aspect of what usually develops into a complex beast, and I bet no-one else does either ... only so many hours in a day etc.

Many of my programs have features I never or rarely use. I share because i like to, but essentially my programs are made for me ... and others are free to accept as is or adapt/improve.

I'm not sure you understand the issue here though, as it is certainly not about updating or even that program you mentioned, which is not related at all to this issue.
Neither is it a fault of my coding. If anything, it is an AutoIt bug or a Windows one. That said, I am self-taught and there are many gaps in my knowledge, regarding concepts. Still, I am using the AutoIt functions as exampled by the Help file, and have been doing so for years, without much issue. You rarely see me post a question here, so i must be doing something right. I have created hundreds of programs, many of which i use each and every day.

Edited July 31, 2018 by TheSaint

[Earthshine]

testing reveals bugs in not only your code but other peoples code, such as this case. in many cases we had to keep old versions of server software because when other teams changed their stuff and ours broke we would know what broke it by testing against the old and new stuff.

good luck with the bug, I hope they fix it for you.

Edited July 31, 2018 by Earthshine

[BrewManNH]

The INI functions come from a time before Windows was Unicode aware, INI files could only be written in straight ANSI by design. The BOM is throwing the function off because the file is  being read as ANSI text.

You can make INI files work with Unicode by using FileOpen with the $FO_UNICODE parameter set. I guess it just won't work if you use the UTF-8 w/BOM option though.

[careca]

Everytime i used ini's the only thing i do is set the path and file to a variable, and then iniwrite.

Never had an issue like you mention, the applications i use ini with have preferences, so if i had an issue i would notice right away.

What im trying to say is maybe ditch all the other functions and stick to iniwrite, let it create the file.

[TheSaint]

3 hours ago, careca said:

Everytime i used ini's the only thing i do is set the path and file to a variable, and then iniwrite.

Never had an issue like you mention, the applications i use ini with have preferences, so if i had an issue i would notice right away.

What im trying to say is maybe ditch all the other functions and stick to iniwrite, let it create the file.

Sounds like you use INI files more as settings files, while I often use them for a lot lot more, for many years now, and this issue would seem a pretty rare event.

Anyway there is merit in what you suggest, and I think I will now ditch _FileCreate for FileDelete, as I suspect it is the culprit. Just for the record, IniWrite always created the INI file, I just used _FileCreate to wipe or zero byte its content ... as a quick Remove All option.

It has been suggested by BM that the antiquity of INI files is also at play, though I'm not entirely sure about that aspect, as the INI standard has been updated at least once, perhaps more than that, and while I can only speak for Win 7 right now, I would not be surprised if even Windows 10 and certainly Linux, still use INI files ... certainly many programs still do. There was a time when INI files had a maximum file size limit of around 64 kb's. They appear to be pretty limitless in size now, though I know AutoIt still has some limits with them.

Thanks to all who replied.

Edited August 1, 2018 by TheSaint

[TheDcoder]

3 hours ago, TheSaint said:

Anyway there is merit in what you suggest, and I think I will now ditch _FileCreate for FileDelete, as I suspect it is the culprit. Just for the record, IniWrite always created the INI file, I just used _FileCreate to wipe or zero byte its content ... as a quick Remove All option.

Wow... an assumption, you never mentioned that you were using _FileCreate to zero it's contents

9 hours ago, TheSaint said:

FileOpen is not really appropriate for INI stuff, and certainly no replacement for _FileCreate.

I beg to differ, I only suggested FileOpen to create or empty the file, you need not to use it for Ini functions. I went the extra mile to show you how _FileCreate could be replaced in such a way that it never produces a BOM:

; Drop in replacement for _FileCreate
Func _FileCreateNoBOM($sFilePath)
    Local $hFile = FileOpen($sFilePath, $FO_OVERWRITE + $FO_UTF8_NOBOM)
    FileClose($hFile)
EndFunc

Another surprise: The _FileCreate function also uses FileOpen, have a look at it yourself in File.au3 include