Jump to content

Reading a child process's output is impossible without blocking and Windows won't return even after the child ends


TheDcoder
 Share

Recommended Posts

Hi, I thought I would never post a C/WinAPI related question in this forum ever, but here we are after a few years and me having learnt enough of C to write a basic console program :)

My issue is that I am trying to read my child process's stdout output but ReadFile never returns if the child exits or if it is killed... very strange :unsure:, I have been trying to work my way around this. The options I can think of are:

  1. Create a new thread and check for existance of the process constantly while reading
  2. Somehow make the pipe asynchronous (overlapped) so that I can read it in a non-blocking manner
  3. Fix ReadFile to return when the process ends

Obviously I would prefer No. 3, I just want to make my program work. Here is my code if you guys want to take a look:

// No text highlighting for C/C++ but we have it for C#? Blasphemy!

bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) {
    char *cmd;
    
    // Figure out the command string for execution
    if (config) {
        char *parameters = " -f -";
        cmd = malloc(strlen(instance->tor_path) + strlen(parameters) + 1);
        if (!cmd) return false;
        strcpy(cmd, instance->tor_path);
        strcat(cmd, parameters);
    } else cmd = instance->tor_path;
    
    // Prepare startup info with appropriate information
    SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info);
    instance->startup_info.dwFlags = STARTF_USESTDHANDLES;
    
    SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true};
    
    HANDLE pipes[2];
    if (output_pipes == NULL) {
        CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
        output_pipes = pipes;
    }
    instance->startup_info.hStdOutput = output_pipes[1];
    instance->startup_info.hStdError = output_pipes[1];
    instance->stdout_pipe = output_pipes[0]; // Stored for internal reference
    
    if (config) {
        // Reuse the pipes array to store standard input pipes
        CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
        instance->startup_info.hStdInput = pipes[0];
    }
    
    // Create the process
    bool success = CreateProcessA(
        NULL,
        cmd,
        NULL,
        NULL,
        config ? true : false,
        0,
        NULL,
        NULL,
        &instance->startup_info,
        SecureZeroMemory(&instance->process, sizeof instance->process)
    );
    
    // Free command string if needed
    if (config) free(cmd);
    
    // Write config to Tor's standard input
    unsigned long bytes_written;
    if (success) {
        WriteFile(pipes[1], config, strlen(config), &bytes_written, NULL);
        // Work around for simulating Ctrl + Z which sends the substitution character (ASCII 26),
        // this is needed in order for Tor to detect EOT/EOF while reading the config
        WriteFile(pipes[1], &(char){26}, 1, &bytes_written, NULL);
    }
    CloseHandle(pipes[1]);
    
    // Return on failure
    if (!success) return false;
}

char *allium_read_stdout_line(struct TorInstance *instance) {
    char *buffer = instance->buffer.data;
    
    // Check for valid buffer and allocate if needed
    if (instance->buffer.size == 0 || !buffer) {
        buffer = instance->buffer.data = malloc(instance->buffer.size = 80 + 1);
        if (!buffer) return NULL;
    }
    
    // Process the input
    unsigned int read_len = 0;
    while (true) {
        // Read data
        unsigned long bytes_read;
        if (ReadFile(instance->stdout_pipe, buffer, 1, &bytes_read, NULL) == false || bytes_read == 0) return NULL;
        
        // Check if we have reached end of line
        if (buffer[0] == '\n') break;
        
        // Proceed to the next character
        ++buffer; ++read_len;
        
        // Resize buffer if it is full
        if (read_len == instance->buffer.size) {
            char *new_buffer = malloc(instance->buffer.size += 50);
            if (new_buffer) memcpy(new_buffer, instance->buffer.data, read_len);
            free(instance->buffer.data);
            if (!new_buffer) return NULL;
            instance->buffer.data = new_buffer;
            buffer = instance->buffer.data + read_len;
        }
    }
    
    // Terminate the new line with null character and return
    // Special handling for Windows, terminate at CR if present
    buffer[read_len >= 2 && buffer[-1] == '\r' ? -1 : 0] = '\0';
}

The allium_start function creates the redirection pipes and the child process, the other allium_read_stdout_line function reads from the stdout pipe created by the first function, ReadFile in this function does not return when the child ends or gets killed.

I appriciate the help of the WinAPI gurus here, thanks in advance! :D

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

@Earthshine Yup, was one of the first results that I got a few weeks back when investigating how output redirection works in POSIX.

Sadly it only talks about the POSIX functions to do this, Windows doesn't provide this functionality as far as I am aware. My code for POSIX works like a charm, the issue is only on Windows :(.

The ReadFile documentation mentions:

Quote

When a synchronous read operation reaches the end of a file, ReadFile returns TRUE and sets *lpNumberOfBytesRead to zero.

This does not happen when the process ends... it just never returns!

I wonder how the internal AutoIt functions like Run and StdoutRead work... I am especially keen on how StdoutRead can be non-blocking, does AutoIt use multi-threading to maintain to retrieve the output later? I might use this method, but I would like to avoid it if possible.

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

It turns out that I lacked basic understanding of how handles work... and that they are duplicated when passed through CreateProcess, so you need to close your own copies of the handles in order for the pipe to properly close when the program exits. I also discovered that inheritence is a bit more complicated, as both handles (each for an end of the pipe) are marked as inheritable by the CreatePipe function during creation... this causes problems yet again by passively duplicating the handles which were not even passed to CreateProcess, you can prevent this by using the SetHandleInformation function. Here is the working code:

bool allium_start(struct TorInstance *instance, char *config, allium_pipe *output_pipes) {
    // Prepare startup info with appropriate information
    SecureZeroMemory(&instance->startup_info, sizeof instance->startup_info);
    instance->startup_info.dwFlags = STARTF_USESTDHANDLES;

    SECURITY_ATTRIBUTES pipe_secu_attribs = {sizeof(SECURITY_ATTRIBUTES), NULL, true};

    HANDLE pipes[2];
    if (output_pipes == NULL) {
        CreatePipe(&pipes[0], &pipes[1], &pipe_secu_attribs, 0);
        output_pipes = pipes;
    }
    SetHandleInformation(output_pipes[0], HANDLE_FLAG_INHERIT, 0);
    instance->startup_info.hStdOutput = output_pipes[1];
    instance->startup_info.hStdError = output_pipes[1];
    instance->stdout_pipe = output_pipes[0]; // Stored for internal reference

    // Create the process
    bool success = CreateProcessA(
        NULL,
        cmd,
        NULL,
        NULL,
        config ? true : false,
        0,
        NULL,
        NULL,
        &instance->startup_info,
        SecureZeroMemory(&instance->process, sizeof instance->process)
    );

    // Close the write end of our stdout handle
    CloseHandle(output_pipes[1]);

    // Return on failure
    if (!success) return false;
}

You can refer to the accepted answer in my StackOverflow question for more details, many thanks to Remy Lebeau and RbMm for answering my questions there.

EasyCodeIt - A cross-platform AutoIt implementation - Fund the development! (GitHub will double your donations for a limited time)

DcodingTheWeb Forum - Follow for updates and Join for discussion

Link to comment
Share on other sites

  • 2 weeks later...
On 1/29/2019 at 8:50 AM, TheDcoder said:

Just posted this question over at StackOverflow in hopes of getting answers:  -snip- Hope this may help.

Thanks for sharing this link. I have found a couple of answers that are useful and could be implemented in work.

Edited by Jos
spam removed but we still expect that answer whenever you return
Link to comment
Share on other sites

  • Moderators

@arnoldfields care to explain why you specifically changed TheDcoder's link in his quote? 

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to comment
Share on other sites

  • 2 months later...

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...