Jump to content

Recommended Posts

Posted (edited)

Hello,

 

I am currently writing a program that parses a massive table from a website, and need a way to add a progress bar while parsing.

I am currently using the function _IETableWriteToArray($oObj, True) to parse the array. I need the progress bar to update as the table is parsed, not just at the end of the parsing.

Any help at all would be very much appreciated!

 

*EDIT --> The array I am left with after parsing is $array[0-50000][16]

Edited by rm4453
[SOLVED]
Posted (edited)

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

#include <Array.au3>
#include <IE.au3>

Local $oIE = _IECreate("https://en.wikipedia.org/wiki/List_of_North_American_settlements_by_year_of_foundation")
Local $oTable = _IETableGetCollection($oIE, 1)
Local $aTableData = _IETableWriteToArray_mod($oTable, True)
_ArrayDisplay($aTableData)

_IEQuit($oIE)

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_mod(ByRef $oObject, $bTranspose = False)
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", "processing rows ", 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_mod

 

Edited by Chimp

 

image.jpeg.9f1a974c98e9f77d824b358729b089b0.jpeg Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Posted
3 minutes ago, Chimp said:

Hi @rm4453

a bad idea might be to change the IE library.... ( I suggest not to do it :thumbsdown:) , a better idea is to only clone the _IETableWriteToArray function and modify it by adding the progressbar and then use the modified function function instead of the original one (in the modified function I marked with "; <-------- mod #" the lines I added):

 

Hello, @Chimp

 

I added your modified function to the IE library, and renamed it to _IETableWriteToArray_ProgressBar. I am testing it right now to see how well it works out, and appreciate all the help so far!

Posted (edited)

@Chimp

I have added one more modification, and it is working perfectly thank you!

*The modification allows you to append a message to be on the progress bar at the end of the function inputs.*

; #FUNCTION# ====================================================================================================================
; Author ........: Dale Hohm
; modified ......: Chimp
; ===============================================================================================================================
Func _IETableWriteToArray_ProgressBar(ByRef $oObject, $bTranspose = False, $message = "Processing Data!") ; <------- If copied into IE.au3 can call normally with _ProgressBar appended to have progress bar pop up, you can also include a message as the last arg if wanted!
    If Not IsObj($oObject) Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidDataType")
        Return SetError($_IESTATUS_InvalidDataType, 1, 0)
    EndIf
    ;
    If Not __IEIsObjType($oObject, "table") Then
        __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_InvalidObjectType")
        Return SetError($_IESTATUS_InvalidObjectType, 1, 0)
    EndIf
    ;
    Local $iCols = 0, $oTds, $iCol
    Local $oTrs = $oObject.rows
    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $iCol = $iCol + $oTd.colSpan
        Next
        If $iCol > $iCols Then $iCols = $iCol
    Next
    Local $iRows = $oTrs.length
    Local $aTableCells[$iCols][$iRows]
    Local $iRow = 0
    Local $iPercent ; <-------- mod 1

    ProgressOn("Progress Meter", $message, 0 & "/" & $iRows, -1, -1, 16) ; <-------- mod 2

    For $oTr In $oTrs
        $oTds = $oTr.cells
        $iCol = 0
        For $oTd In $oTds
            $aTableCells[$iCol][$iRow] = String($oTd.innerText)
            If @error Then ; Trap COM error, report and return
                __IEConsoleWriteError("Error", "_IETableWriteToArray", "$_IESTATUS_COMError", @error)
                Return SetError($_IESTATUS_ComError, @error, 0)
            EndIf
            $iCol = $iCol + $oTd.colSpan
        Next
        $iRow = $iRow + 1
        $iPercent = Int($iRow / $iRows * 100) ; <-------- mod 3
        ProgressSet($iPercent, $iRow & "/" & $iRows & @TAB & "(" & $iPercent & "%)") ; <-------- mod 4
    Next
    If $bTranspose Then
        Local $iD1 = UBound($aTableCells, $UBOUND_ROWS), $iD2 = UBound($aTableCells, $UBOUND_COLUMNS), $aTmp[$iD2][$iD1]
        For $i = 0 To $iD2 - 1
            For $j = 0 To $iD1 - 1
                $aTmp[$i][$j] = $aTableCells[$j][$i]
            Next
        Next
        $aTableCells = $aTmp
    EndIf
    ProgressOff() ; <-------- mod 5
    Return SetError($_IESTATUS_Success, 0, $aTableCells)
EndFunc   ;==>_IETableWriteToArray_ProgressBar

 

Edited by rm4453
Fixed Spelling Mishap
Posted

You are welcome :)
p.s. better to also add an ProgressOff() statement right after the line "If @error Then ; Trap COM error, report and return" so that the progress bar will be closed also in case of errors.

 

image.jpeg.9f1a974c98e9f77d824b358729b089b0.jpeg Chimp

small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...