Code to extract plain text from a PDF file

[mLipok]

I find an example in C

http://www.codeproject.com/Articles/7056/Code-to-extract-plain-text-from-a-PDF-file

As I'm not a C programmer, so I wanted to ask somebody about transferring this functionality to the AutoIt code.

I'll be grateful.

Best regards
mLipok

 

edit:

some more info (code inf VB):

http://www.codeproject.com/Articles/31627/Decode-FlateDecode-PDF-Stream-To-Plain-Text-using

Edited April 16, 2014 by mLipok

[Exit]

I use this when I need the text of a PDF.

;~ PDF2Text

$PDF = @DesktopDir & "\Intenso 2TB Betriebsanleitung.pdf"

ShellExecute($PDF)
$hPDF = WinWaitActive("[class:AcrobatSDIWindow]")
ControlSend($hPDF, "", "", "{ALT}{down 6}{right}{enter}")
Sleep(2000)
ControlSend($hPDF, "", "", "{enter}")
Sleep(20000) ; allow Textconversion to finish
WinClose($hPDF)
ShellExecute(StringTrimRight($PDF, 3) & "txt")

[mLipok]

So the method is not practical when you have to extract (as soon as possible) text of 200 pdf files.

[jguinch]

I have already made something like this...

Download the XPDF tools (pdfinfo and pdftotext.) and try my script :

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _XFDF_Info
; Description....: Retrives informations from a PDF file
; Syntax.........: _XFDF_Info ( "File" [, "Info"] )
; Parameters.....: File    - PDF File.
;                  Info    - The information to retrieve
; Return values..: Success - If the Info parameter is not empty, returns the desired information for the specified Info parameter
;                          - If the Info parameter is empty, returns an array with all available informations
;                  Failure - 0, and sets @error to :
;                   1 - PDF File not found
;                   2 - Unable to find the external programm
; Remarks........: The array returned is two-dimensional and is made up as follows:
;                   $array[1][0] = Label of the first information (title, author, pages...)
;                   $array[1][1] = value of the first information
;                   ...
; ===============================================================================================================================
Func _XFDF_Info($sPDFFile, $sInfo = "")
    Local $sXPDFInfo = @ScriptDir & "\pdfinfo.exe"

    If NOT FileExists($sPDFFile) Then Return SetError(1, 0, 0)
    If NOT FileExists($sXPDFInfo) Then Return SetError(2, 0, 0)
    $sXPDFInfo = FileGetShortName($sXPDFInfo)

    Local $iPid = Run(@ComSpec & ' /c ' &  $sXPDFInfo & ' "' & $sPDFFile & '"', @ScriptDir, @SW_HIDE, 2)

    Local $sResult
    While 1
        $sResult &= StdoutRead($iPid)
        If @error Then ExitLoop
    WEnd

    Local $aInfos = StringRegExp($sResult, "(?m)^(.+?): +(.*)$", 3)
    If @error Or Mod( UBound($aInfos, 1), 2) = 1 Then Return SetError(3, 0, 0)

    Local $aResult [ UBound($aInfos, 1) / 2][2]

    For $i = 0 To UBound($aInfos) - 1 Step 2
        If $sInfo <> "" AND $aInfos[$i] = $sInfo Then Return $aInfos[$i + 1]
        $aResult[$i / 2][0] = $aInfos[$i]
        $aResult[$i / 2][1] = $aInfos[$i + 1]
    Next

    If $sInfo <> "" Then Return ""

    Return $aResult
EndFunc ; ---> _XFDF_Info


; #FUNCTION# ====================================================================================================================
; Name...........: _XPDF_Search
; Description....: Retrives informations from a PDF file
; Syntax.........: _XFDF_Info ( "File" [, "String" [, Case = 0 [, Flag = 0 [, FirstPage = 1 [, LastPage = 0]]]]] )
; Parameters.....: File    - PDF File.
;                  String    - String to search for
;                  Case      - If set to 1, search is case sensitive (default is 0)
;                  Flag      - A number to indicate how the function behaves. See below for details. The default is 0.
;                  FirstPage  - First page to convert (default is 1)
;                  LastPage   - Last page to convert (default is 0 = last page of the document)
; Return values..: Success -
;                   Flag = 0 - Returns 1 if the search string was found, or 0 if not
;                   Flag = 1 - Returns the number of occcurrences found in the whole PDF File
;                   Flag = 2 - Returns an array containing the number of occurrences found for each page
;                              (only pages containing the search string are returned)
;                              $array[0][0] - Number of matching pages
;                              $array[0][1] - Number of occcurrences found in the whole PDF File
;                              $array[n][0] - Page number
;                              $array[n][1] - Number of occcurrences found for the page
;                  Failure - 0, and sets @error to :
;                   1 - PDF File not found
;                   2 - Unable to find the external programm
; ===============================================================================================================================
Func _XPDF_Search($sPDFFile, $sSearch, $iCase = 0, $iFlag = 0, $iStart = 1, $iEnd = 0)
    Local $sXPDFToText = @ScriptDir & "\pdftotext.exe"
    Local $sOptions = " -layout -f " & $iStart
    Local $iCount = 0, $aResult[1][2] = [[0, 0]], $aSearch, $sContent, $iPageOccCount
   
    If NOT FileExists($sPDFFile) Then Return SetError(1, 0, 0)
    If NOT FileExists($sXPDFToText) Then Return SetError(2, 0, 0)
   
    If $iEnd > 0 Then $sOptions &= " -l " & $iEnd
   
    Local $iPid = Run($sXPDFToText & $sOptions & ' "' & $sPDFFile & '" -', @ScriptDir, @SW_HIDE, 2)
    While 1
        $sContent &= StdoutRead($iPid)
        If @error Then ExitLoop
    WEnd
   
   
    Local $aPages = StringSplit($sContent, chr(12) )
   
    For $i = 1 To $aPages[0]
        $iPageOccCount = 0
        While StringInStr($aPages[$i], $sSearch, $iCase, $iPageOccCount + 1)
            If $iFlag <> 1 AND $iFlag <> 2 Then
                $aResult[0][1] = 1
                ExitLoop
            EndIf
            $iPageOccCount += 1
        WEnd

        If $iPageOccCount Then
            Redim $aResult[ UBound($aResult, 1) + 1][2]
            $aResult[0][1] += $iPageOccCount
            $aResult[0][0] = UBound($aResult) - 1
            $aResult[ UBound($aResult, 1) - 1 ][0] = $i + $iStart - 1
            $aResult[ UBound($aResult, 1) - 1 ][1] = $iPageOccCount
        EndIf
    Next
   
    If $iFlag = 2 Then Return $aResult
    Return $aResult[0][1]
   
EndFunc ; ---> _XPDF_Search



; #FUNCTION# ====================================================================================================================
; Name...........: _XPDF_ToText
; Description....: Converts a PDF file to plain  text.
; Syntax.........: _XPDF_ToText ( "PDFFile" , "TxtFile" [ , FirstPage [, LastPage [, Layout ]]] )
; Parameters.....: PDFFile    - PDF Input File.
;                  TxtFile    - Plain text file to convert to
;                  FirstPage  - First page to convert (default is 1)
;                  LastPage   - Last page to convert (default is last page of the document)
;                  Layout     - If true, maintains (as  best as possible) the original physical layout of the text
;                               If false, the behavior is to 'undo'  physical  layout  (columns, hyphenation, etc.)
;                                 and output the text in reading order.
;                               Default is True
; Return values..: Success - 1
;                  Failure - 0, and sets @error to :
;                   1 - PDF File not found
;                   2 - Unable to find the external program
; ===============================================================================================================================
Func _XPDF_ToText($sPDFFile, $sTXTFile, $iFirstPage = 1, $iLastPage = 0, $bLayout = True)
    Local $sXPDFToText = @ScriptDir & "\pdftotext.exe"
    Local $sOptions
   
    If NOT FileExists($sPDFFile) Then Return SetError(1, 0, 0)
    If NOT FileExists($sXPDFToText) Then Return SetError(2, 0, 0)
   
    If $iFirstPage <> 1 Then $sOptions &= " -f " & $iFirstPage
    If $iLastPage <> 0 Then $sOptions &= " -l " & $iLastPage
    If $bLayout = True Then $sOptions &= " -layout"
   
    Local $iReturn = ShellExecuteWait ( $sXPDFToText , $sOptions & ' "' & $sPDFFile & '" "' & $sTXTFile & '"', @ScriptDir, "", @SW_HIDE)
    If $iReturn = 0 Then Return 1
   
    Return 0
   
EndFunc ; ---> _XPDF_ToText

Original post on the french forum

Edited May 27, 2015 by jguinch

[mLipok]

but your solution is not really free.

  Quote

 

If you're interested in commercial licensing, please see the Glyph & Cog web site.

 

And I can not integrate them into a script.

In addition, the action you propose creates a TXT file, and I would like to avoid that.
I would like to retrieve the text directly from PDF to a variable in the script.

[Danp2]

Have you looked at this?

[jguinch]

Not really free, yes..

But it's possible to read the stdout from pdftotext.exe (there is a command line parameter for), not necessary convert to a text file.

My _XPDF_Search function does not use a text file...

[mLipok]

  On 4/16/2014 at 5:54 PM, Danp2 said:

Have you looked at this?

 

Thanks.

I just analyzing this.

Its look like DeltaRocked wants to do exactly the same thing.

Edited April 16, 2014 by mLipok

[mLipok]

for now I have this:

expandcollapse popup

#include<string.au3>
#include <array.au3>
#include "zlib_udf.au3"

_Example()
Exit

Func _Example()
    _Zlib_Startup("zlib1.dll")
;~ Normal PDFs
    Local $file = FileOpenDialog('Select PDF file', @ScriptDir, 'Searchable PDF File (*.pdf)', 0, 'test.pdf')
    Local $var = _GetTxtFromPDF($file)
    MsgBox(1, 'test', $var)
EndFunc   ;==>_Example

Func _GetTxtFromPDF($fullfilename)
    Local $start_obj = '(?i)\d* \d* obj'
    Local $start_pt = '(?i) obj'
    Local $end_pt = '(?i)endobj'
    Local $strpos = 0, $length = 10, $count_loop, $ex_data, $Decompressed, $header, $binlen, $aDecomperessed, $iRegExp_Error
    Local $start_ex_pt = '(?i)>>\s*stream' ; & '\r\n' ;at the end of the string. this will include @CR @LF in the search.
    Local $end_ex_pt = '(?i)endstream' ;'\r\n' & ; at the start of the string
    Local Enum $CRYPT_STRING_BASE64X509CRLHEADER = 9, $CRYPT_STRING_HEXADDR, $CRYPT_STRING_HEXASCIIADDR, $CRYPT_STRING_HEXRAW

    If Not FileExists($fullfilename) Then Return SetError(1, 0, 0)
    $sData = FileRead($fullfilename)
;~  MsgBox(1, '$sData', $sData)
    $start_array = StringRegExp($sData, $start_pt, 3)
    $start_obj_array = StringRegExp($sData, $start_obj, 3)
    $end_array = StringRegExp($sData, $end_pt, 3)
    FileDelete(@ScriptDir & '\test.log')
    FileWrite(@ScriptDir & '\test.log', 'Analyzing ' & $fullfilename & @CRLF)

    If IsArray($start_array) And IsArray($end_array) Then
        If UBound($start_array) == UBound($end_array) Then
            $count_loop = 1
            While $count_loop <= UBound($start_array)
                $start_pos = StringInStr($sData, $start_array[$count_loop - 1], 2, $count_loop) + StringLen($start_array[$count_loop - 1])
                $end_pos = StringInStr($sData, $end_array[$count_loop - 1], 2, $count_loop)
                $ex_data = StringMid($sData, $start_pos, $end_pos - $start_pos)
                If StringInStr($ex_data, 'stream', 2) And StringInStr($ex_data, '/flatedecode', 2) And StringInStr($ex_data, '/predictor', 2) == 0 And StringInStr($ex_data, '/BBox', 2) == 0 And StringInStr($ex_data, '/ASCIIHexDecode', 2) == 0 Then
;~              If StringInStr($ex_data, 'stream', 2) Then
                    $start_extract_array = StringRegExp($ex_data, $start_ex_pt, 3)
                    $end_extract_array = StringRegExp($ex_data, $end_ex_pt, 3)
                    If IsArray($start_extract_array) And IsArray($end_extract_array) Then
                        $start_ex_pos = StringInStr($ex_data, $start_extract_array[0], 2, 1) + StringLen($start_extract_array[0])
                        $end_ex_pos = StringInStr($ex_data, $end_extract_array[0], 2, 1)
                        $ex_ex_data = StringStripWS(StringMid($ex_data, $start_ex_pos, $end_ex_pos - $start_ex_pos), 3)
;~                      MsgBox(1, '$ex_ex_data', $ex_ex_data)
                        $binlen = BinaryLen($ex_ex_data)
                        $header = StringStripWS(StringLeft($ex_data, $start_ex_pos), 7) ; used for writing logs and has got nothing to do with the exracted stream
;~                      $Decompressed = zlib($ex_ex_data, $binlen)
                        $Decompressed = _Zlib_UncompressBinary($ex_ex_data, $binlen)
;~                      MsgBox(1, 'Info1', VarGetType($Decompressed))
;~                      MsgBox(1, 'Info2', $Decompressed)
;~                      ClipPut($Decompressed)

                        $Decompressed = BinaryToString($Decompressed)
;~                      Local $stest = @error
;~                      MsgBox(1, 'BinaryToString Error', $stest)
;~                      MsgBox(1, '$Decompressed', $Decompressed)
;~                      ClipPut($Decompressed)
;~                      MsgBox(1, '$Decompressed', $Decompressed)

;~                      $aDecomperessed = StringRegExp($Decompressed, '(?is)Td\R\[\((.*?)\)\] TJ', 3)
                        $aDecomperessed = StringRegExp($Decompressed, '(?s)(?(?=Td\R\[\()Td\R\[\((.*?)\] Tj|Tc\R\((.*?)\) Tj)', 3)
                        $iRegExp_Error = @error
;~                      If StringInStr($Decompressed, '/javascript', 2) <> 0 Or StringInStr($Decompressed, 'else if', 2) <> 0 Then
;~                          MsgBox(0,'Info','JS Decrypted')
;~                      FileWrite(@ScriptDir & '\test.log', $start_obj_array[$count_loop - 1] & @CRLF & $header & @CRLF & _
;~                              'BinaryLen of the extracted compressed stream = ' & $binlen & @CRLF & '-------------------------' & _
;~                              @CRLF & StringReplace(BinaryToString($Decompressed), '>><<', '>>' & @CRLF & '<<') & @CRLF & '-------------------------' & @CRLF)
;~                      EndIf
                        ConsoleWrite('! TESTING' & @CRLF)
                        ConsoleWrite($start_obj_array[$count_loop - 1] & @CRLF & $header & @CRLF & _
                                'BinaryLen of the extracted compressed stream = ' & $binlen & @CRLF & '-------------------------' & _
                                @CRLF & _
                                StringReplace($Decompressed, '>><<', '>>' & @CRLF & '<<') & _
                                @CRLF & '-------------------------' & @CRLF)
                        If $iRegExp_Error Then
;~                          MsgBox(1,'test1','')
                            Return SetError(3, $iRegExp_Error, '')
                        Else
;~                          MsgBox(1,'test2','')
                            Return SetError(0, 0, _ArrayToString($aDecomperessed, @CRLF))
                        EndIf
                    EndIf
                EndIf
                $count_loop += 1
            WEnd
;~             Return UBound($start_array)
            Return $Decompressed
        Else
            Return SetError(1, 0, 0)
        EndIf
    Else
        Return SetError(2, 0, 0)
    EndIf
EndFunc   ;==>_GetTxtFromPDF

Remarks:

use the modifications done by ProgAndy

I still have some issue

ie.: 

'?do=embed' frameborder='0' data-embedContent>>

[dexto]

So, I'm looking into this and got it to decompress text in PDFs:

expandcollapse popup

; YOU JUST NEED zlib1.dll
__Zlib_Startup()



$t = GetPdfStreamContent("MY.PDF")

ConsoleWrite($t & @CRLF)



Func GetPdfStreamContent($fullfilename) ;Finds streams within a PDF. Returns an Array with the streams starting at $strm_Array[1]
    Local $regex = '(?s)<<[^<]*?FlateDecode[^<]+?Length \d+[^<]*?>>[\r\n]*stream[\r\n]*(.*?)[\r\n]*endstream' ;this is regex, (\d+) will return length, @extended the position of the stream
    Local $sData = FileRead($fullfilename)
    Local $arr = StringRegExp($sData, $regex, 3)
    Local $out = ''
    For $i = 0 To UBound($arr) - 1
        $out &= BinaryToString(__ZLib_UncompressBinary($arr[$i]))
    Next
    Return $out
EndFunc   ;==>GetPdfStreamContent


Func __Zlib_Startup()
    Local $dllPath = @TempDir & '\' & @ScriptName & 'zlib1.dll'
    FileInstall('zlib1.dll', $dllPath, 1)
    Global $Zlib_Dll = DllOpen($dllPath)
EndFunc   ;==>__Zlib_Startup


; Decompresses data, you need to know how large the decompressed data will be.
Func __Zlib_Uncompress($CompressedPtr, ByRef $CompressedSize, $UncompressedPtr, $UncompressedSize)
    ; modified by ProgAndy
    $call = DllCall($Zlib_Dll, "int:cdecl", "uncompress", "ptr", $UncompressedPtr, "long*", $UncompressedSize, "ptr", $CompressedPtr, "long", $CompressedSize)
    If @error Then Return SetError(1, 0, -7)
    $CompressedSize = $call[2]
    Return $call[0]
EndFunc   ;==>__Zlib_Uncompress

Func __ZLib_UncompressBinary($bBinary, $iLength = 0)
    ; ProgAndy
    Local $i = 1, $tBuf, $iSize, $iRes
    Local $tBin = DllStructCreate("byte[" & BinaryLen($bBinary) & "]")
    DllStructSetData($tBin, 1, $bBinary)
    If $iLength < 1 Then $iLength = DllStructGetSize($tBin) * 2
    $bBinary = 0
    Do
        $tBuf = DllStructCreate("byte[" & $iLength * $i & "]")
        $iSize = DllStructGetSize($tBin)
        $iRes = __Zlib_Uncompress(DllStructGetPtr($tBin), $iSize, DllStructGetPtr($tBuf), DllStructGetSize($tBuf))
        $i += 1
    Until $iRes <> -5
    If $iRes <> 0 Then Return SetError($iRes, 0, "")
    $tBin = 0
    Return DllStructGetData(DllStructCreate("byte[" & $iSize & "]", DllStructGetPtr($tBuf)), 1)
EndFunc   ;==>__ZLib_UncompressBinary

Expecting text in PDF: "AU3"

Instead getting scary data structure:

/CIDInit /ProcSet findresource begin
12 dict begin
begincmap
/CMapType 2 def
/CMapName/R19 def
1 begincodespacerange
<00><ff>
endcodespacerange
3 beginbfrange
<01><01><0041>
<02><02><0055>
<03><03><0033>
endbfrange
endcmap
CMapName currentdict /CMap defineresource pop
end end

I can see ASCII in it spelling 41 55 33 or AU3 but CMap is more complicated than that. Any idea if there some something to decode CMap?

Edited May 20, 2015 by dexto
formating