Reading a child process's output is impossible without blocking and Windows won't return even after the child ends

[TheDcoder]

Hi, I thought I would never post a C/WinAPI related question in this forum ever, but here we are after a few years and me having learnt enough of C to write a basic console program

My issue is that I am trying to read my child process's stdout output but ReadFile never returns if the child exits or if it is killed... very strange , I have been trying to work my way around this. The options I can think of are:

1. Create a new thread and check for existance of the process constantly while reading
2. Somehow make the pipe asynchronous (overlapped) so that I can read it in a non-blocking manner
3. Fix ReadFile to return when the process ends

Obviously I would prefer No. 3, I just want to make my program work. Here is my code if you guys want to take a look:

expandcollapse popup

// No text highlighting for C/C++ but we have it for C#? Blasphemy!

bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) {
    char *cmd;
    
    // Figure out the command string for execution
    if (config) {
        char *parameters = " -f -";
        cmd = malloc(strlen(instance->tor_path) + strlen(parameters) + 1);
        if (!cmd) return false;
        strcpy(cmd, instance->tor_path);
        strcat(cmd, parameters);
    } else cmd = instance->tor_path;
    
    // Prepare startup info with appropriate information
    SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info);
    instance->startup_info.dwFlags = STARTF_USESTDHANDLES;
    
    SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true};
    
    HANDLE pipes[2];
    if (output_pipes == NULL) {
        CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
        output_pipes = pipes;
    }
    instance->startup_info.hStdOutput = output_pipes[1];
    instance->startup_info.hStdError = output_pipes[1];
    instance->stdout_pipe = output_pipes[0]; // Stored for internal reference
    
    if (config) {
        // Reuse the pipes array to store standard input pipes
        CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
        instance->startup_info.hStdInput = pipes[0];
    }
    
    // Create the process
    bool success = CreateProcessA(
        NULL,
        cmd,
        NULL,
        NULL,
        config ? true : false,
        0,
        NULL,
        NULL,
        &instance->startup_info,
        SecureZeroMemory(&instance->process, sizeof instance->process)
    );
    
    // Free command string if needed
    if (config) free(cmd);
    
    // Write config to Tor's standard input
    unsigned long bytes_written;
    if (success) {
        WriteFile(pipes[1], config, strlen(config), &bytes_written, NULL);
        // Work around for simulating Ctrl + Z which sends the substitution character (ASCII 26),
        // this is needed in order for Tor to detect EOT/EOF while reading the config
        WriteFile(pipes[1], &(char){26}, 1, &bytes_written, NULL);
    }
    CloseHandle(pipes[1]);
    
    // Return on failure
    if (!success) return false;
}

char *allium_read_stdout_line(struct TorInstance *instance) {
    char *buffer = instance->buffer.data;
    
    // Check for valid buffer and allocate if needed
    if (instance->buffer.size == 0 || !buffer) {
        buffer = instance->buffer.data = malloc(instance->buffer.size = 80 + 1);
        if (!buffer) return NULL;
    }
    
    // Process the input
    unsigned int read_len = 0;
    while (true) {
        // Read data
        unsigned long bytes_read;
        if (ReadFile(instance->stdout_pipe, buffer, 1, &bytes_read, NULL) == false || bytes_read == 0) return NULL;
        
        // Check if we have reached end of line
        if (buffer[0] == '\n') break;
        
        // Proceed to the next character
        ++buffer; ++read_len;
        
        // Resize buffer if it is full
        if (read_len == instance->buffer.size) {
            char *new_buffer = malloc(instance->buffer.size += 50);
            if (new_buffer) memcpy(new_buffer, instance->buffer.data, read_len);
            free(instance->buffer.data);
            if (!new_buffer) return NULL;
            instance->buffer.data = new_buffer;
            buffer = instance->buffer.data + read_len;
        }
    }
    
    // Terminate the new line with null character and return
    // Special handling for Windows, terminate at CR if present
    buffer[read_len >= 2 && buffer[-1] == '\r' ? -1 : 0] = '\0';
}

The allium_start function creates the redirection pipes and the child process, the other allium_read_stdout_line function reads from the stdout pipe created by the first function, ReadFile in this function does not return when the child ends or gets killed.

I appriciate the help of the WinAPI gurus here, thanks in advance!

[Earthshine]

hey, have you read this yet?

http://www.microhowto.info/howto/capture_the_output_of_a_child_process_in_c.html

[TheDcoder]

@Earthshine Yup, was one of the first results that I got a few weeks back when investigating how output redirection works in POSIX.

Sadly it only talks about the POSIX functions to do this, Windows doesn't provide this functionality as far as I am aware. My code for POSIX works like a charm, the issue is only on Windows .

The ReadFile documentation mentions:

Quote

When a synchronous read operation reaches the end of a file, ReadFile returns TRUE and sets *lpNumberOfBytesRead to zero.

This does not happen when the process ends... it just never returns!

I wonder how the internal AutoIt functions like Run and StdoutRead work... I am especially keen on how StdoutRead can be non-blocking, does AutoIt use multi-threading to maintain to retrieve the output later? I might use this method, but I would like to avoid it if possible.

[TheDcoder]

Just posted this question over at StackOverflow in hopes of getting answers: https://stackoverflow.com/questions/54416116/readfile-does-not-return-while-reading-stdout-from-a-child-process-after-it-ends

[TheDcoder]

It turns out that I lacked basic understanding of how handles work... and that they are duplicated when passed through CreateProcess, so you need to close your own copies of the handles in order for the pipe to properly close when the program exits. I also discovered that inheritence is a bit more complicated, as both handles (each for an end of the pipe) are marked as inheritable by the CreatePipe function during creation... this causes problems yet again by passively duplicating the handles which were not even passed to CreateProcess, you can prevent this by using the SetHandleInformation function. Here is the working code:

expandcollapse popup

bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) {
    // Prepare startup info with appropriate information
    SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info);
    instance->startup_info.dwFlags = STARTF_USESTDHANDLES;

    SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true};

    HANDLE pipes[2];
    if (output_pipes == NULL) {
        CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
        output_pipes = pipes;
    }
    SetHandleInformation(output_pipes[0], HANDLE_FLAG_INHERIT, 0);
    instance->startup_info.hStdOutput = output_pipes[1];
    instance->startup_info.hStdError = output_pipes[1];
    instance->stdout_pipe = output_pipes[0]; // Stored for internal reference

    // Create the process
    bool success = CreateProcessA(
        NULL,
        cmd,
        NULL,
        NULL,
        config ? true : false,
        0,
        NULL,
        NULL,
        &instance->startup_info,
        SecureZeroMemory(&instance->process, sizeof instance->process)
    );

    // Close the write end of our stdout handle
    CloseHandle(output_pipes[1]);

    // Return on failure
    if (!success) return false;
}

You can refer to the accepted answer in my StackOverflow question for more details, many thanks to Remy Lebeau and RbMm for answering my questions there.

[arnoldfields]

On 1/29/2019 at 8:50 AM, TheDcoder said:

Just posted this question over at StackOverflow in hopes of getting answers:  -snip- Hope this may help.

Thanks for sharing this link. I have found a couple of answers that are useful and could be implemented in work.

Edited May 4, 2019 by Jos
spam removed but we still expect that answer whenever you return

[JLogan3o13]

@arnoldfields care to explain why you specifically changed TheDcoder's link in his quote? 

[SenaAka]

Thank you!

[TheDcoder]

@SenaAka Glad to see that I have helped... and that you created an account to say thank you