Jump to content

string search takes too loong (bible search)=SOLVED


dirty
 Share

Recommended Posts

i have a file with 36105 lines to search for a string.

enough said.

it takes too long to find a string given, so

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#Include <File.au3>
#Include <GuiListView.au3>
FileInstall ("Bibles\Russian.txt","Russian.txt",1)
$MainGUI = GUICreate ("Bible Search",400,435)
GUICtrlCreateGroup ("Search ?",5,5,390,45)
$Input = GUICtrlCreateInput ("",10,20,300,20)
$SearchButton = GUICtrlCreateButton ("Go !",315,20,70,20)
GUICtrlCreateGroup ("Results.",5,50,390,325)
$ListView = GUICtrlCreateListView ("Results of your search displayed here.|",10,65,380,300)
_GUICtrlListView_SetColumnWidth ($ListView,0,375)
$ShowButton = GUICtrlCreateButton ("Show Selected.",5,380,395,30)
$Status = GUICtrlCreateLabel ("Whats going on ?",5,420,390,20)
GUISetState (@SW_SHOW,$MainGUI)
While 1
$msg = GUIGetMsg()
If $msg = $GUI_EVENT_CLOSE Then
  FileDelete ("Russian.txt")
  Exit
EndIf
If $msg = $SearchButton Then
  $SearchFor = GUICtrlRead ($Input)
  _GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($ListView))
  For $i = 1 to 36105
   GUICtrlSetData ($Status,"Whats going on ?         Working")
   $SearchIn = FileReadLine ("Russian.txt",$i)
   If StringInStr ($SearchIn,$SearchFor) <> @error Then
    GUICtrlCreateListViewItem ($SearchIn,$ListView)
   EndIf
   $Loopmsg = GUIGetMsg()
   If $Loopmsg = $SearchButton Then
    MsgBox(0,'','search stoped')
    ExitLoop
   EndIf
  Next
  GUICtrlSetData ($Status,"Whats going on ?      Nothing")
EndIf
If $msg = $ShowButton Then
  MsgBox(0,'',GUICtrlRead (GUICtrlRead ($ListView)))
EndIf
WEnd

as simple as i could yet its too long.

Is there a way to search faster ?

I noticed searching for a whole phrase takes just as long as it takes to find a word in this phrase

Edited by dirty
Link to comment
Share on other sites

Hello dirty,

I'm not sure if there is a limitation of how many lines can be read into an array with this function, however I know this function is faster for reading lines from a text file

_FileReadToArray()

this will read your text file into a 1 dimensional array with each element containing a line from your text file...then loop as you did in your example, only this time checking on your array. Largest file I read into an array was 18k lines, and looping for a check too all took only a few seconds. Depending on the severity of your check this could change the timing, however I still say that this way is faster than reading each line from the file individually

My Contributions: Unix Timestamp: Calculate Unix time, or seconds since Epoch, accounting for your local timezone and daylight savings time. RegEdit Jumper: A Small & Simple interface based on Yashied's Reg Jumper Function, for searching Hives in your registry. 

Link to comment
Share on other sites

Why not use _FileReadToArray, because currently you're opening the file ... reading the line ... closing the file (repeat 36105 times) also the Array returned has the number of lines in the index 0 item, so you don't have to hard code the number of lines.

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Link to comment
Share on other sites

There is an example in your help file.

#include <file.au3>
Dim $aRecords
If Not _FileReadToArray("error.log",$aRecords) Then
   MsgBox(4096,"Error", " Error reading log to Array    error:" & @error)
   Exit
EndIf
For $x = 1 to $aRecords[0]
    Msgbox(0,'Record:' & $x, $aRecords[$x])
Next

My Contributions: Unix Timestamp: Calculate Unix time, or seconds since Epoch, accounting for your local timezone and daylight savings time. RegEdit Jumper: A Small & Simple interface based on Yashied's Reg Jumper Function, for searching Hives in your registry. 

Link to comment
Share on other sites

Look at the Example in the Help file for _FileReadToArray.

UDF List:

 
_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_ArrayFilter/_ArrayReduce_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 22/04/2018

Link to comment
Share on other sites

Try this faster version. You can certainly go without arrays.

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <File.au3>
#include <GuiListView.au3>
FileInstall("BiblesRussian.txt", "Russian.txt", 1)
$MainGUI = GUICreate("Bible Search", 400, 435)
GUICtrlCreateGroup("Search ?", 5, 5, 390, 45)
$Input = GUICtrlCreateInput("", 10, 20, 300, 20)
$SearchButton = GUICtrlCreateButton("Go !", 315, 20, 70, 20)
GUICtrlCreateGroup("Results.", 5, 50, 390, 325)
$ListView = GUICtrlCreateListView("Results of your search displayed here.|", 10, 65, 380, 300)
_GUICtrlListView_SetColumnWidth($ListView, 0, 375)
$ShowButton = GUICtrlCreateButton("Show Selected.", 5, 380, 395, 30)
$Status = GUICtrlCreateLabel("Whats going on ?", 5, 420, 390, 20)
Local $source = FileRead("Russian.txt")  ; read the whole text at once
Local $res
GUISetState(@SW_SHOW, $MainGUI)
While 1
$msg = GUIGetMsg()
If $msg = $GUI_EVENT_CLOSE Then
  FileDelete("Russian.txt")
  Exit
EndIf
If $msg = $SearchButton Then
  $SearchFor = GUICtrlRead($Input)
  _GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($ListView))
  GUICtrlSetData($Status, "Whats going on ?      Working")
  $SearchFor = StringRegExpReplace($SearchFor, "([.*+()?^$|{}[]])", "$1")
  $res = StringRegExp($source, "(?im)^.*" & $SearchFor & ".*$", 3)
  If Not @error Then
   For $line In $res
    GUICtrlCreateListViewItem($line, $ListView)
   Next
  EndIf
  GUICtrlSetData($Status, "Whats going on ?   Nothing")
EndIf
If $msg = $ShowButton Then
  MsgBox(0, '', GUICtrlRead(GUICtrlRead($ListView)))
EndIf
WEnd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

I imagine even just keeping the file open until the search is complete would speed it up.

This part.

_GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($ListView))
  For $i = 1 to 36105
   GUICtrlSetData ($Status,"Whats going on ?         Working")
   $SearchIn = FileReadLine ("Russian.txt",$i)
   If StringInStr ($SearchIn,$SearchFor) <> @error Then
    GUICtrlCreateListViewItem ($SearchIn,$ListView)
   EndIf
   $Loopmsg = GUIGetMsg()
   If $Loopmsg = $SearchButton Then
    MsgBox(0,'','search stoped')
    ExitLoop
   EndIf
  Next

To

_GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($ListView))
  $hfile = FileOpen("Russian.txt")
  For $i = 1 to 36105
   GUICtrlSetData ($Status,"Whats going on ?         Working")
   $SearchIn = FileReadLine ($hfile,$i)
   If StringInStr ($SearchIn,$SearchFor) <> @error Then
    GUICtrlCreateListViewItem ($SearchIn,$ListView)
   EndIf
   $Loopmsg = GUIGetMsg()
   If $Loopmsg = $SearchButton Then
    MsgBox(0,'','search stoped')
    ExitLoop
   EndIf
  Next
FileClose("Russian.txt")
Edited by JohnOne

AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Link to comment
Share on other sites

Try this faster version. You can certainly go without arrays.

Man this was instant result.

But these two lines

$SearchFor = [url="../../../autoit3/docs/functions/StringRegExpReplace.htm"]StringRegExpReplace[/url]($SearchFor, "([.*+()?^$|{}[]])", "$1")
  $res = [url="../../../autoit3/docs/functions/StringRegExp.htm"]StringRegExp[/url]($source, "(?im)^.*" & $SearchFor & ".*$", 3)

driving me nuts !.

They are as complex as your avatar.

Speaking of me being stupid, how come there is no smiley face for being stupid ? It would be very appropriate in my case.

Edited by dirty
Link to comment
Share on other sites

They are not as complex as they seem.

The speed is gained by loading the whole file in memory once for all. Then using a Regular Expression (RE) to extract all lines which contain what the user looks for.

You were talking about 36K lines, so with 100 characters/line it's just a 3.6Mb block. I tested it with 50Mb files on a slow laptop and that still gave decent results.

The first line you quote is needed due to the fact that the search string may contain special characters which the RE engine would normally interpret as "commands". This is achived by "escaping" them all (prefixing them with a backslash).

The second statement performs the actual extraction of every line which contain, from the start of line followed by the search string followed by followed by an end of line or end of text.

BTW if you better wish to ignore punctuation in the search string, it's possible to change that very easily. Just say so.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Using your example:

How do i step back each time a * is found to previously defined chapter=#?

For example my lisview has 4 columns.

$SearchFor = StringRegExpReplace($SearchFor, "([.*+()?^$|{}[]])", "$1")
$res = StringRegExp($source, "(?im)^.*" & $SearchFor & ".*$", 3)
        If Not @error Then
            For $line In $res
                $NewLine = StringTrimLeft ($line,StringInStr($line,"*")) ;becuase each line starts with Versus=1* or Versus=2* etc so i remove that to display in first column.
                GUICtrlCreateListViewItem($NewLine & "|Chapter|Book|Testament|", $ListView)
            Next
        EndIf

Here with line generated by (your doing) i can trim it by finding special character i placed in each line *

Because this is a bible, lines look like this

Versus=30* а всем зверям земным, и всем птицам небесным, и всякому [гаду,] пресмыкающемуся по земле, в котором душа живая, дал Я всю зелень травную в пищу. И стало так.
Versus=31* И увидел Бог все, что Он создал, и вот, хорошо весьма. И был вечер, и было утро: день шестой.

@Chapter=2`

Versus=1* Так совершены небо и земля и все воинство их.
Versus=2* И совершил Бог к седьмому дню дела Свои, которые

I somehow need to assign Chapter # into the second column of listview

Note that each @Chapter=# and has a special character at the end ` and @ at the beginning to help me to find it, but with your code i cant really figure out how to go back to this symbol and read whats behind it (kinda go backwards i guess)

I also dont understand why did u use $SearchFor twice ? Maybe thats how it works, i dont know

Edited by dirty
Link to comment
Share on other sites

Here's a version showing found text, verse, chapter, book & testament.

As explained in my previous post, I use StringRegExpReplace to replace several special characters in what the user searches for to keep them from being intrepreted wrongly by the actual search. This replacement is very fast as it only deals with the text entered by the user, not on the whole Bible text.

I need to make you aware of something that may be important to you: the search expression is performing caseless (that's the i in (?ims) options). BUT the search engine AutoIt currently uses only cares of lower ASCII (the 127 first characters in the ANSI table). There is a possibility to search for Unicode text while taking care of non-english letters (over the most used part of Unicode) but this compile option is not currently usable in AutoIt.

Thus I'm unsure how caseless search will work for you. It will depend of your file being ANSI or UTF-8 (with BOM), which codepage you use if using ANSI. If that causes havoc, say so and I'll do something to keep it efficient and simple, but your users will have to use a russian codepage.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Search works fine both for english and russian.

by looking at

$SearchFor = StringRegExpReplace($SearchFor, "([.*+()?^$|{}[]])", "$1")

$res = StringRegExp($source, "(?im)^.*" & $SearchFor & ".*$", 3)

i cant figure out how to make it find "@" that represents the beginning of the line with Chapter#`

$line returns matching line of text u searched for and $res returns nothing at all.

So i was thinking about doing it this way.

$LastFoundChapter = "" ;declare previously so it gets changed when found
$SearchInput = GUICtrlRead($Input)
$SearchFor = StringRegExpReplace($SearchInput, "([.*+()?^$|{}[]])", "$1")
$res = StringRegExp($source, "(?im)^.*" & $SearchFor & ".*$", 3)
If Not @error Then
For $line In $res
If $line = "@" Then Assign ("LastFoundChapter",$line) ;when chapter line is found, assign it to the variable
$NewLine = StringTrimLeft ($line,StringInStr($line,"*")) ;remove Versus=#* from the line to make it clean
GUICtrlCreateListViewItem($NewLine & "|" & $LastFoundChapter & "|Book|Testament|", $ListView) ;write to listview

but i cant figure out how to find the @ while looking for what is in the $SearchInput

Edited by dirty
Link to comment
Share on other sites

Ooops, I seem to have left the example code away from the post... Apologies for that.

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <File.au3>
#include <GuiListView.au3>
FileInstall("BiblesRussian.txt", "Russian.txt", 1)
$MainGUI = GUICreate("Bible Search", 400, 435)
GUICtrlCreateGroup("Search ?", 5, 5, 390, 45)
$Input = GUICtrlCreateInput("", 10, 20, 300, 20)
$SearchButton = GUICtrlCreateButton("Go !", 315, 20, 70, 20)
GUICtrlCreateGroup("Results.", 5, 50, 390, 325)
$ListView = GUICtrlCreateListView("Results of your search displayed here.|Verse|Chapter|Book|Testament", 10, 65, 380, 300)
_GUICtrlListView_SetColumnWidth($ListView, 0, 375)
$ShowButton = GUICtrlCreateButton("Show Selected.", 5, 380, 395, 30)
$Status = GUICtrlCreateLabel("Whats going on ?", 5, 420, 390, 20)
Local $source = FileRead("Russian.txt")  ; read the whole text at once
Local $res
Local $book = "Paul", $testament = "New" ; I don't know where you get this from
GUISetState(@SW_SHOW, $MainGUI)
While 1
$msg = GUIGetMsg()
If $msg = $GUI_EVENT_CLOSE Then
;~   FileDelete("Russian.txt")
  Exit
EndIf
If $msg = $SearchButton Then
  $SearchFor = GUICtrlRead($Input)
  _GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($ListView))
  GUICtrlSetData($Status, "Whats going on ?      Working")
  $SearchFor = StringRegExpReplace($SearchFor, "([.*+()?^$|{}[]])", "$1")
  $res = StringRegExp($source, "(?ims)@chapter=(d+)`.*?(?-s)^versus=(d+)*s(.*?" & $SearchFor & ".*?)$", 3)
  If Not @error Then
   For $i = 0 To UBound($res) - 1 Step 3
    GUICtrlCreateListViewItem($res[$i + 2] & '|' & $res[$i + 1] & '|' & $res[$i] & '|' & $book & '|' & $testament, $ListView)
   Next
  EndIf
  GUICtrlSetData($Status, "Whats going on ?   Nothing")
EndIf
If $msg = $ShowButton Then
  MsgBox(0, '', GUICtrlRead(GUICtrlRead($ListView)))
EndIf
WEnd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Man i really appreciate you helping me out :)

I attached English version if the file for you to look at. Maybe this will help you to help me how to get this right.

English.rar

Right now i have no clue why you predefine local $book, $testament = "New" and how would they work.

Those are being located in the process of a search, kinda like searching for 4 things at same time.

For example:

1st you search for what ever text you need. Once found, look back and see what chapter it belongs to, then what book and then the testament.

I guess i could change each line in the text file from this:

Versus=1* In the beginning God created the heaven and the earth.

into:

Book=Genesis Chapter=1 Versus=1* In the beginning God created the heaven and the earth.

but it will take me forever. With help of autoit i can probably make that a matter of a day of coding and few hours of text processing (with my level of experience working with strings)

This is how i replaced

1 In the beginning God created the heaven and the earth.

with

Versus=1* In the beginning God created the heaven and the earth.

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <File.au3>
#Include <GuiStatusBar.au3>
$MainGUI = GUICreate("Bible Search", 400, 435)
$SearchButton = GUICtrlCreateButton("Go !", 315, 20, 70, 20)
GUICtrlCreateGroup("Results.", 5, 50, 390, 325)
$ListView = GUICtrlCreateEdit("", 10, 65, 380, 300)
GUISetState(@SW_SHOW, $MainGUI)
While 1
$msg = GUIGetMsg()
If $msg = $GUI_EVENT_CLOSE Then
  Exit
EndIf
If $msg = $SearchButton Then
  For $i = 1 to 150 ;150 because psalms have 150 versus
   $source = FileRead("English.txt")
   $Replaced = StringReplace ($source, @CRLF & "CHAPTER " & $i & @CRLF,@CRLF & "@Chapter=" & $i & "`" & @CRLF)
   GUICtrlSetData($ListView,$Replaced)
   FileClose ($source)
   _GUICtrlStatusBar_SetText ($StatusBar,"Going "& $i)
   $OpenToWrite = FileOpen ("English.txt",2+128)
   FileWrite ($OpenToWrite,$Replaced)
   FileClose ($OpenToWrite)
  Next
  FileClose ("English.txt")
EndIf
WEnd

This loops 150 times just to change 1 to Versus=1*

so imagine how long its gonna take me to change it to:

Book=Genesis Chapter=1 Versus=1* In the beginning God created the heaven and the earth. :D

The good thing is, by the time its all over i will probably play guitar like a pro :oops: Unless you have a better plan ?

Because i still have no clue (sorry for being dumb) how your script works and u are my only hope :rip:

Edited by dirty
Link to comment
Share on other sites

Your biggest problem is in the file format you use.

You definitely need a better one, as you correctly identified.

First, split things in two: Bible and NT to ease processing.

Then we'll index each versus with a short complete reference as you devised, but there is no need to be unduly verbose here.

Imagine you abbreviate things this way:

BI = Bible

NT = New Testament

GEN = Genesis

...

MAT = Mathiew

PAU = Paul

...

Affect every book a unique three-letter acronym (preferably using uppercase english letters). This will be used for any translation.

Then we'll write a one-time utility (once for every language file) to produce two files having the required format, something like:

BI,GEN,1,1§In the beginning God created the heaven and the earth.

NT,MAT,3,14§But John forbad him, saying, I have need to be baptized of thee, and comest thou to me?

Once done (it'll be very fast) we merge both files into one.

Then I'll change the search script to find text and fill in references in a much more simple and efficient way.

The idea is that extracting matching complete lines with this reference convention is easy, but "going backward" (there is no going backward with REs) to find chapter and book and testament is utterly slow, up to being worst than every other approach when searching very common substrings like 'he', due to RE backtraking intensely.

Does that sound reasonable to you?

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

(there is no going backward with REs)

No wonder reverse engineering is a science :D

I will start formatting text files, but wouldnt it be easier to have whole name of a testament and book instead of abbreviated one ?

This way we wont have to rename NT,MAT,3,14§ into NEW TESTAMENT,MATHEW,3,14§ or how were you gonna go about setting whole name into listview column i dont know.

Let me know before i start

Link to comment
Share on other sites

Ok i got it

Here is how i did it so i could understand.

$SearchFor = StringRegExpReplace($SearchInput, "([.*+()?^$|{}[]])", "$1")
  $res = StringRegExp($source, "(?im)^.*" & $SearchFor & ".*$", 3)
  If Not @error Then
   For $line In $res
    $Loopmsg = GUIGetMsg()
    If $Loopmsg = $SearchButton Then
     GUICtrlSetData ($SearchButton,"Go !")
     ExitLoop
    EndIf
    $NewLine = StringTrimLeft ($line,StringInStr($line,"*"))
    $Chapter = StringLeft ($line,StringInStr($line,",","",3))
    $TrimmedChapter = StringTrimLeft ($Chapter,StringInStr ($Chapter,"=",'',3))
    $FixedChapter = StringTrimRight ($TrimmedChapter,1)
    $Book = StringLeft ($line,StringInStr($line,",","",2))
    $TrimmedBook = StringTrimLeft ($Book,StringInStr ($Book,"=",'',2))
    $FixedBook = StringTrimRight ($TrimmedBook,1)
    $Testament = StringLeft ($line,StringInStr($line,",","",1))
    $TrimmedTestament = StringTrimLeft ($Testament,StringInStr ($Testament,"="))
    $FixedTestament = StringTrimRight ($TrimmedTestament,1)
    GUICtrlCreateListViewItem($NewLine & "|" & $FixedChapter & "|" & $FixedBook & "|" & $FixedTestament & "|", $ListView)
    Assign ("Found",$Found+1)
   Next
  EndIf

This is how each line of text looks like

T=ВЕТХИЙ ЗАВЕТ,B=БЫТИЕ,C=1,V=1* В начале сотворил Бог небо и землю.

So from here i simply count the instance of an = and trim it twise.

I know trimming twice could be done once somehow, but hey it works like a champ ! all thanks to you

Link to comment
Share on other sites

One perhaps last question:

How would you make the advanced search to work with your example ? I mean for instance searching for:

"hell" will also return hellicopter as well as hellicoptering and hellicapture and last but not least helliscloriousis.

Is it possible to change your part of the code to search differently if * is provided ?

For example *hell returns anystringbeforhell or hell* returns hellanystringafterhell.

Right now it works like both, *hell* and returns anystringbeforhellanystringafterhell.

So i was thinking to narrow the search result down by exact match like google does, for example

"hell" will return only those lines that will have the word hell in them and ignore those with hellium and hellerious :oops: by wrapping a word into quotes.

I had that feature in my other project managment app and it works like this:

$GetNameFromGUI = GUICtrlRead($SearchByName) ;Name. Can be "Name"
$Part1 = StringTrimLeft($GetNameFromGUI, 1) ;removes 1 quote and returns Name"
$Part2 = StringTrimRight($Part1, 1) ;removes another quote from Name" and returns Name
If $GetNameFromGUI = ('"' & $Part2 & '"') Then ;If $GetNameFromGUI had quotes surrounding it then
;whatever code to display it on gui or elsewhere using $GetNameFromResult
Else
;whatever code to display it on gui or elsewhere using $GetNameFromResult
EndIf

Its funny how now that i am looking at it, i dont understand it :D but am sure you get the idea.

In your case i dont know how to have that thing work the same way.

Edited by dirty
Link to comment
Share on other sites

Try this improved version which uses any * character in the searched string just like a classical DOS/Windows joker.

Whith it, you can search *e*h*ee* and have all occurences of Bethleem appear enclosed in double quotes (and possibly other word matches).

#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <File.au3>
#include <GuiListView.au3>
FileInstall("BiblesRussian.txt", "Russian.txt", 1)
$MainGUI = GUICreate("Bible Search", 400, 435)
GUICtrlCreateGroup("Search ?", 5, 5, 390, 45)
$Input = GUICtrlCreateInput("", 10, 20, 300, 20)
$SearchButton = GUICtrlCreateButton("Go !", 315, 20, 70, 20)
GUICtrlCreateGroup("Results.", 5, 50, 390, 325)
$ListView = GUICtrlCreateListView("Results of your search displayed here.|Versus|Chapter|Book|Testament", 10, 65, 380, 300)
_GUICtrlListView_SetColumnWidth($ListView, 0, 375)
$ShowButton = GUICtrlCreateButton("Show Selected.", 5, 380, 395, 30)
$Status = GUICtrlCreateLabel("Whats going on ?", 5, 420, 390, 20)
Local $source = FileRead("Russian.txt") ; read the whole text at once
Local $res, $refs, $match, $Found, $Versustext, $Versus, $Chapter, $book, $testament
GUISetState(@SW_SHOW, $MainGUI)
While 1
$msg = GUIGetMsg()
If $msg = $GUI_EVENT_CLOSE Then
;~   FileDelete("Russian.txt")
  Exit
EndIf
If $msg = $SearchButton Then
  $SearchFor = GUICtrlRead($Input)
  _GUICtrlListView_DeleteAllItems(GUICtrlGetHandle($ListView))
  GUICtrlSetData($Status, "Whats going on ?      Working")
  $SearchFor = StringReplace($SearchFor, "*", "§")
  $SearchFor = StringRegExpReplace($SearchFor, "([.*+()?^$|{}[]])", "$1")
  $SearchFor = "b" & StringReplace($SearchFor, "§", ".*?") & "b"
  $res = StringRegExp($source, "(?im)^.*" & $SearchFor & ".*$", 3)
  If Not @error Then
   $Found = UBound($res)
   For $line In $res
    $Loopmsg = GUIGetMsg()                 ;
    If $Loopmsg = $SearchButton Then                 ;
     GUICtrlSetData($SearchButton, "Go !")         ; what purpose does that serve ?
     ExitLoop                             ;
    EndIf                           ;
    $refs = StringRegExp($line, "(?i)^s*ts*=s*([^,]*)s*,s*bs*=s*([^,]*)s*,s*cs*=s*(d*)s*,s*vs*=s*(d*)*s*(.*)", 1)
    $testament = $refs[0]
    $book = $refs[1]
    $Chapter = $refs[2]
    $Versus = $refs[3]
    $Versustext = StringRegExpReplace($refs[4], "(?i)(" & $SearchFor & ")", '"$1"')
    GUICtrlCreateListViewItem($Versustext & "|" & $Versus & "|" & $Chapter & "|" & $book & "|" & $testament & "|", $ListView)
   Next
  EndIf
  GUICtrlSetData($Status, "Whats going on ?   Nothing")
EndIf
If $msg = $ShowButton Then
  MsgBox(0, '', GUICtrlRead(GUICtrlRead($ListView)))
EndIf
WEnd
Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...