Search the Community

Hi, I thought I would never post a C/WinAPI related question in this forum ever, but here we are after a few years and me having learnt enough of C to write a basic console program My issue is that I am trying to read my child process's stdout output but ReadFile never returns if the child exits or if it is killed... very strange , I have been trying to work my way around this. The options I can think of are: Create a new thread and check for existance of the process constantly while reading Somehow make the pipe asynchronous (overlapped) so that I can read it in a non-blocking manner Fix ReadFile to return when the process ends Obviously I would prefer No. 3, I just want to make my program work. Here is my code if you guys want to take a look: // No text highlighting for C/C++ but we have it for C#? Blasphemy! bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) { char *cmd; // Figure out the command string for execution if (config) { char *parameters = " -f -"; cmd = malloc(strlen(instance->tor_path) + strlen(parameters) + 1); if (!cmd) return false; strcpy(cmd, instance->tor_path); strcat(cmd, parameters); } else cmd = instance->tor_path; // Prepare startup info with appropriate information SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info); instance->startup_info.dwFlags = STARTF_USESTDHANDLES; SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true}; HANDLE pipes[2]; if (output_pipes == NULL) { CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0); output_pipes = pipes; } instance->startup_info.hStdOutput = output_pipes[1]; instance->startup_info.hStdError = output_pipes[1]; instance->stdout_pipe = output_pipes[0]; // Stored for internal reference if (config) { // Reuse the pipes array to store standard input pipes CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0); instance->startup_info.hStdInput = pipes[0]; } // Create the process bool success = CreateProcessA( NULL, cmd, NULL, NULL, config ? true : false, 0, NULL, NULL, &instance->startup_info, SecureZeroMemory(&instance->process, sizeof instance->process) ); // Free command string if needed if (config) free(cmd); // Write config to Tor's standard input unsigned long bytes_written; if (success) { WriteFile(pipes[1], config, strlen(config), &bytes_written, NULL); // Work around for simulating Ctrl + Z which sends the substitution character (ASCII 26), // this is needed in order for Tor to detect EOT/EOF while reading the config WriteFile(pipes[1], &(char){26}, 1, &bytes_written, NULL); } CloseHandle(pipes[1]); // Return on failure if (!success) return false; } char *allium_read_stdout_line(struct TorInstance *instance) { char *buffer = instance->buffer.data; // Check for valid buffer and allocate if needed if (instance->buffer.size == 0 || !buffer) { buffer = instance->buffer.data = malloc(instance->buffer.size = 80 + 1); if (!buffer) return NULL; } // Process the input unsigned int read_len = 0; while (true) { // Read data unsigned long bytes_read; if (ReadFile(instance->stdout_pipe, buffer, 1, &bytes_read, NULL) == false || bytes_read == 0) return NULL; // Check if we have reached end of line if (buffer[0] == '\n') break; // Proceed to the next character ++buffer; ++read_len; // Resize buffer if it is full if (read_len == instance->buffer.size) { char *new_buffer = malloc(instance->buffer.size += 50); if (new_buffer) memcpy(new_buffer, instance->buffer.data, read_len); free(instance->buffer.data); if (!new_buffer) return NULL; instance->buffer.data = new_buffer; buffer = instance->buffer.data + read_len; } } // Terminate the new line with null character and return // Special handling for Windows, terminate at CR if present buffer[read_len >= 2 && buffer[-1] == '\r' ? -1 : 0] = '\0'; } The allium_start function creates the redirection pipes and the child process, the other allium_read_stdout_line function reads from the stdout pipe created by the first function, ReadFile in this function does not return when the child ends or gets killed. I appriciate the help of the WinAPI gurus here, thanks in advance!

Hi guys, i am having an issue reading a notepadd++ file... #include <FileConstants.au3> #include <MsgBoxConstants.au3> #include <WinAPIFiles.au3> Example() Func Example() ; Open the file for reading and store the handle to a variable. Local $hFileOpen = FileOpen("test123.html", $FO_READ) If $hFileOpen = -1 Then MsgBox($MB_SYSTEMMODAL, "", "An error occurred when reading the file.") Return False EndIf ; Read the contents of the file using the handle returned by FileOpen. Local $sFileRead = FileRead($hFileOpen) ; Close the handle returned by FileOpen. FileClose($hFileOpen) ; Display the contents of the file. MsgBox($MB_SYSTEMMODAL, "", "Contents of the file:" & @CRLF & $sFileRead) EndFunc ;==>Example It does seem that i get an error but why?

HiHo Forum, in my program SMF - Search my Files I use a special method to calculate fake (and fast) md5 hashes to identify duplicate files. With the standard settings SMF reads the first 8KB, 8KB from the middle and the last 8KB from a file and calculates a md5 hash on the data. Now I've noticed that the larger the files, the slower the calculation (makes sense ). I assume that's because also the FileSetPointer() operation takes it's time for large files. Calculation drops from somewhere about 300 files / sec to 30 files / sec (of course depending on the filesize and also your machine's power). Now I thought about a way to further improve search & hashing speed and stumbled over the possibility for an "Overlapped" (that's Microsoft's term for asynchronous) file access. You request several portions of a file and wait for the results to show up, the ReadFile() function itself returns instantaneously and does not wait for the return buffer to be filled (opposed to the standard behavior where the function only returns when the operation has been finished). Currently in SMF I'm requesting the 3 blocks of data and after they've been read I calculate the hash. What I will try out and test is to request all 3 blocks in overlapped mode and start calculating the hash for each block independently (maybe even in a different thread as pointed out by Ward in ) as soon as they pop up. My assumption is that this might improve the performance for large files (> 10MB? I just have to give it a shoot), for smaller files I think the standard operation should be superior. As I've not found any example for asynchronous file access on the forum I thought I just post the WIP code for those interested to take a look and help me improve it ... It's just a crude and raw example, but at least it works (on my machines anyhow). Yashied's most excellent is required for the example to work. #region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_UseUpx=n #AutoIt3Wrapper_UseX64=n #AutoIt3Wrapper_Res_requestedExecutionLevel=asInvoker #endregion ;**** Directives created by AutoIt3Wrapper_GUI **** ; [url="http://msdn.microsoft.com/en-us/library/windows/desktop/aa365467(v=vs.85).aspx"]http://msdn.microsoft.com/en-us/library/windows/desktop/aa365467(v=vs.85).aspx[/url] ; [url="http://msdn.microsoft.com/en-us/library/windows/desktop/aa365683(v=vs.85).aspx"]http://msdn.microsoft.com/en-us/library/windows/desktop/aa365683(v=vs.85).aspx[/url] ; [url="http://support.microsoft.com/kb/156932"]http://support.microsoft.com/kb/156932[/url] #include <StructureConstants.au3> #include <WinAPIEx.au3> #include <Memory.au3> #include <array.au3> Global $nBytes, $hFile Global Const $ERROR_IO_INCOMPLETE = 996 ; Overlapped I/O event is not in a signaled state Global Const $ERROR_IO_PENDING = 997 ; Overlapped I/O operation is in progress $sFile = FileOpenDialog("Select a large file to open for overlapped (asynchronous) reading...", StringLeft(@WindowsDir, 3), "All (*.*)", 3) If @error Then Exit $sFile = StringReplace($sFile, "|", @CRLF) If FileGetSize($sFile) < 1024 * 100 Then MsgBox(0, "", "Larger than 100kb would make sense...") Exit EndIf $aDrive = _WinAPI_GetDriveNumber(StringLeft($sFile, 2)) $aData = _WinAPI_GetDriveGeometryEx_RO($aDrive[1]) ConsoleWrite(_WinAPI_GetLastErrorMessage() & @TAB & @error & @CRLF) ; 'Bytes per Sector: ' & $aData[4] ; FILE_FLAG_NO_BUFFERING ; File access sizes, including the optional file offset in the OVERLAPPED structure, if specified, ; must be for a number of bytes that is an integer multiple of the volume sector size $iBytesToRead = 16 * $aData[4]; ~ 8.192 bytes with 512 bytes per sector ; Because buffer addresses for read and write operations must be sector-aligned, the application must have direct control of how these buffers are allocated. ; One way to sector-align buffers is to use the VirtualAlloc function to allocate the buffers $pBuffer_Mem = _MemVirtualAlloc(0, $iBytesToRead, $MEM_COMMIT, $PAGE_READWRITE) ; $tBuffer = DllStructCreate("byte[" & $iBytesToRead & "];byte[" & $iBytesToRead & "];byte[" & $iBytesToRead & "]") ; $GENERIC_READ = 0x80000000 ; $FILE_ATTRIBUTE_NORMAL = 0x00000080 ; $FILE_FLAG_OVERLAPPED = 0x40000000 ; $FILE_FLAG_NO_BUFFERING = 0x20000000 Global Const $FILE_FLAG_OVERLAPPED = 0x40000000 Global Const $FILE_FLAG_NO_BUFFERING = 0x20000000 $iTimer = TimerInit() $hFile = _WinAPI_CreateFileEx($sFile, 3, $GENERIC_READ, 7, BitOR($FILE_ATTRIBUTE_NORMAL, $FILE_FLAG_OVERLAPPED, $FILE_FLAG_NO_BUFFERING)) $iFileGetSize = _WinAPI_GetFileSizeEx($hFile) ;ConsoleWrite(@CRLF & "+ Filesize " & $iFileGetSize & @CRLF & @CRLF) $iTimer_0 = TimerDiff($iTimer) ; Global Const $tagOVERLAPPED = "int Internal;int InternalHigh;int Offset;int OffsetHigh;int hEvent" $tOverlapped1 = DllStructCreate($tagOVERLAPPED) $pOverlapped1 = DllStructGetPtr($tOverlapped1) _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer, 1), $iBytesToRead, $nBytes, $pOverlapped1) $iTimer_1 = TimerDiff($iTimer) $sTimer_1 = _WinAPI_GetLastError() $tOverlapped2 = DllStructCreate($tagOVERLAPPED) $pOverlapped2 = DllStructGetPtr($tOverlapped2) $iOffset = Int(($iFileGetSize / 2) - ($iBytesToRead / 2 + 1)) $iOffset = (Floor($iOffset / $aData[4])) * $aData[4] DllStructSetData($tOverlapped2, "Offset", _WinAPI_LoDWord($iOffset)) ; Setting "Filepointer" to the middle of the file (SetFilePointer is not valid for overlapped operations) DllStructSetData($tOverlapped2, "OffsetHigh", _WinAPI_HiDWord($iOffset)) _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer, 2), $iBytesToRead, $nBytes, $pOverlapped2) $iTimer_2 = TimerDiff($iTimer) $sTimer_2 = _WinAPI_GetLastError() $tOverlapped3 = DllStructCreate($tagOVERLAPPED) $pOverlapped3 = DllStructGetPtr($tOverlapped3) $iOffset = $iFileGetSize - $iBytesToRead $iOffset = (Floor($iOffset / $aData[4])) * $aData[4] DllStructSetData($tOverlapped3, "Offset", _WinAPI_LoDWord($iOffset)) ; Setting "Filepointer" to the end of the file (SetFilePointer is not valid for overlapped operations) DllStructSetData($tOverlapped3, "OffsetHigh", _WinAPI_HiDWord($iOffset)) _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer, 3), $iBytesToRead, $nBytes, $pOverlapped3) $iTimer_3 = TimerDiff($iTimer) $sTimer_3 = _WinAPI_GetLastError() $sRes1 = _WinAPI_GetOverlappedResult($hFile, $pOverlapped1, $nBytes) $sRes1 = $sRes1 & @TAB & $nBytes $iTimer_4 = TimerDiff($iTimer) $sTimer_4 = _WinAPI_GetLastError() $sRes2 = _WinAPI_GetOverlappedResult($hFile, $pOverlapped2, $nBytes) $sRes2 = $sRes2 & @TAB & $nBytes $iTimer_5 = TimerDiff($iTimer) $sTimer_5 = _WinAPI_GetLastError() $sRes3 = _WinAPI_GetOverlappedResult($hFile, $pOverlapped3, $nBytes) $sRes3 = $sRes3 & @TAB & $nBytes $iTimer_6 = TimerDiff($iTimer) $sTimer_6 = _WinAPI_GetLastError() _WinAPI_CloseHandle($hFile) $iTimer_7 = TimerDiff($iTimer) $sTimer_7 = _WinAPI_GetLastError() $sRes4 = _WinAPI_GetOverlappedResult($hFile, $pOverlapped1, $nBytes) $sRes4 = $sRes4 & @TAB & $nBytes $sRes5 = _WinAPI_GetOverlappedResult($hFile, $pOverlapped2, $nBytes) $sRes5 = $sRes5 & @TAB & $nBytes $sRes6 = _WinAPI_GetOverlappedResult($hFile, $pOverlapped3, $nBytes) $sRes6 = $sRes6 & @TAB & $nBytes _MemVirtualFree($pBuffer_Mem, $iBytesToRead, $MEM_DECOMMIT) $iTimer = TimerInit() $t_ReadFile_Standard = _ReadFile_Standard($sFile, $iFileGetSize) ConsoleWrite("_WinAPI_CloseHandle - Before " & $sRes1 & @TAB & $sRes2 & @TAB & $sRes3 & @CRLF) ConsoleWrite("_WinAPI_CloseHandle - After " & $sRes4 & @TAB & $sRes5 & @TAB & $sRes6 & @CRLF & @CRLF) ConsoleWrite($iTimer_0 & @CRLF & $iTimer_1 & @CRLF & $iTimer_2 & @CRLF & $iTimer_3 & @CRLF & $iTimer_4 & @CRLF & $iTimer_5 & @CRLF & $iTimer_6 & @CRLF & $iTimer_7 & @CRLF & @CRLF & TimerDiff($iTimer) & @CRLF & @CRLF) ConsoleWrite($sTimer_1 & @CRLF & $sTimer_2 & @CRLF & $sTimer_3 & @CRLF & $sTimer_4 & @CRLF & $sTimer_5 & @CRLF & $sTimer_6 & @CRLF & $sTimer_7 & @CRLF) MsgBox(0, "", StringLeft(DllStructGetData($tBuffer, 1), 5) & StringRight(DllStructGetData($tBuffer, 1), 5) & @CRLF _ & StringLeft(DllStructGetData($tBuffer, 2), 5) & StringRight(DllStructGetData($tBuffer, 2), 5) & @CRLF _ & StringLeft(DllStructGetData($tBuffer, 3), 5) & StringRight(DllStructGetData($tBuffer, 3), 5) & @CRLF _ & @CRLF & @CRLF _ & StringLeft(DllStructGetData($t_ReadFile_Standard, 1), 5) & StringRight(DllStructGetData($t_ReadFile_Standard, 1), 5) & @CRLF _ & StringLeft(DllStructGetData($t_ReadFile_Standard, 2), 5) & StringRight(DllStructGetData($t_ReadFile_Standard, 2), 5) & @CRLF _ & StringLeft(DllStructGetData($t_ReadFile_Standard, 3), 5) & StringRight(DllStructGetData($t_ReadFile_Standard, 3), 5)) Func _ReadFile_Standard($sFile, $iFileGetSize, $iFlag = 0) ; Local $hFile = _WinAPI_CreateFile($Checksum_Filename, 2, 2, 7), $Checksum_Result, $nBytes ; FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000 Local $hFile = _WinAPI_CreateFileEx($sFile, 3, $GENERIC_READ, 7, $iFlag), $nBytes If $hFile = 0 Then Return SetError(4) ;"File was locked and could not be analyzed..." Local $tBuffer = DllStructCreate("byte[" & $iBytesToRead & "];byte[" & $iBytesToRead & "];byte[" & $iBytesToRead & "]") _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer, 1), $iBytesToRead, $nBytes) $iOffset = Int(($iFileGetSize / 2) - ($iBytesToRead / 2 + 1)) $iOffset = (Floor($iOffset / $aData[4])) * $aData[4] _WinAPI_SetFilePointerEx($hFile, $iOffset) _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer, 2), $iBytesToRead, $nBytes) $iOffset = $iFileGetSize - $iBytesToRead $iOffset = (Floor($iOffset / $aData[4])) * $aData[4] _WinAPI_SetFilePointerEx($hFile, $iOffset) ; $iBytesToRead/2 +1 _WinAPI_ReadFile($hFile, DllStructGetPtr($tBuffer, 3), $iBytesToRead, $nBytes) _WinAPI_CloseHandle($hFile) Return $tBuffer EndFunc ;==>_ReadFile_Standard Func _WinAPI_GetDriveGeometryEx_RO($iDrive) Local $hFile = _WinAPI_CreateFileEx('.PhysicalDrive' & $iDrive, 3, 0, 0x01) If Not $hFile Then Return SetError(1, 0, 0) EndIf Local $tDGEX = DllStructCreate('int64;dword;dword;dword;dword;int64') Local $Ret = DllCall('kernel32.dll', 'int', 'DeviceIoControl', 'ptr', $hFile, 'dword', 0x000700A0, 'ptr', 0, 'dword', 0, 'ptr', DllStructGetPtr($tDGEX), 'dword', DllStructGetSize($tDGEX), 'dword*', 0, 'ptr', 0) If (@error) Or (Not $Ret[0]) Then $Ret = 0 EndIf _WinAPI_CloseHandle($hFile) If Not IsArray($Ret) Then Return SetError(2, 0, 0) EndIf Local $Result[6] For $i = 0 To 5 $Result[$i] = DllStructGetData($tDGEX, $i + 1) Next Return $Result EndFunc ;==>_WinAPI_GetDriveGeometryEx_RO Edit: Just wanted to point out that this technique only makes sense in very special cases. The default synchronous FileRead utilizes the internal cache manager and for sure is much faster (and easier) to handle than this asynchronous FileRead method and should be the choice for 99.9% of your needs. For some further info take a look at these pages: MSDN - ReadFile function MSDN - Synchronous and Asynchronous I/O MSKB - Asynchronous Disk I/O Appears as Synchronous One observation I've made is that it seems like the _WinAPI_CloseHandle($hFile) call is blocking until the operation has been finished. Before that, the progress of the single requests can be monitored using the _WinAPI_GetOverlappedResult() function. Best Regards

i've been playing with this for awhile and have finally gotten where i can read the file i want, but i still can't WRITE the characters i need to write. i can open file "Contoso.vuw" and process to the end of file, displaying both the character position number and associated character, IF i open the file as read only. If i open the file as FileWrite("contoso.vuw", 1) the file runs to the end, but displays nothing. If i open as FileWrite("contoso.vuw", 16) i get a binary representation of all characters. i'm stuck. i need to replace characters 17, 18 and 19 with whatever the user types in (i can do that part), a 3-character userid. If i can get these three positions done i can do the other positions further down in the file (characters 2247, 2248, and 2249, all represented as "abc"). Can someone point me in the right direction please. Thanks, Dave BinaryWrite.zip