Implementing an equivalent of DllCall() in C++

[twitchyliquid64]

Hi all,

One of my projects requires the use of dynamically calling a Dll where the name, or parameters of the Dll are not known at compile time. In other words, I'm trying to implement the dynamic functionality of DllCall().

This is a challenge I'm not quite sure how to approach.

Loading up the the Dll and getting the address of the function is trivial, it's just pushing the values to the stack, and getting the return values that would be the problem.

I have a feeling I will need assembly.

Does anyone know how I can do this? I simply have no idea where to start. If anyone has any example code on this it would be most appreciated

Thank you very much in advance,

Hyperzap

[Mat]

What calling convention is being used?

[twitchyliquid64]

Both cdecl and stdcall. I need support for both.

[Mat]

You shouldn't need assembly, provided you are using a language where you can call function pointers. It may take a bit of work to write something the complier allows though. Simple summary of function pointers in C:

http://publications.gbdirect.co.uk/c_book/chapter5/function_pointers.html

For a quick guide to stack and major calling conventions I'd go here:

http://www.unixwiz.net/techtips/win32-callconv-asm.html

Or use AutoIt.

Edit: For using calling conventions with function pointers, there is no standard way to do it, but see here: http://www.newty.de/fpt/fpt.html

Edited March 28, 2012 by Mat

[twitchyliquid64]

You shouldn't need assembly, provided you are using a language where you can call function pointers. It may take a bit of work to write something the complier allows though. Simple summary of function pointers in C:

http://publications.gbdirect.co.uk/c_book/chapter5/function_pointers.html

For a quick guide to stack and major calling conventions I'd go here:

http://www.unixwiz.net/techtips/win32-callconv-asm.html

Or use AutoIt.

Edit: For using calling conventions with function pointers, there is no standard way to do it, but see here: http://www.newty.de/fpt/fpt.html

The problem is not calling the function. The problem is calling the function dynamically, and passing the arguments at runtime. The arguments are not known at compile time, so I cannot simply create a function pointer with the relevant arguments.

[Shaggi]

I'm working on something you maybe could use. Sec.

[twitchyliquid64]

I'm working on something you maybe could use. Sec.

Thank you

Edited March 28, 2012 by twitchyliquid64

[Shaggi]

Okay, here goes. First off, this is nontested, and just so you can get an idea of how to implement it. I think it works though lol. There are some comments around, but basically, you need to assemble asm in your function, you need to know convention, return type, etc. etc. It's not that hard to do with standard types, but floats and doubles use other registers and stacks, so you might wanna read up on that. functions, that returns structs, is usually very implementation defined and often optimized away so you probably wont have luck with that.

All in all, if you are not 100% about what you do, this is not the road you wanna go down if you need a stable system. Nevertheless, this is quite fun

expandcollapse popup

//by shaggi: quick dirty nontested example of dllcall
//x86 ofc
//only works with very primitive types and stdcall atm
//should be used like:
#include <windows.h>
#include <iostream>
/*
    int args [] = {1, 2};
    int result;
    result_t res = DllCall<rsint>(ret_t::rint, conv_t::cstdcall, (void*)0xDEADBEEF, ARRAYSIZE(args), args, & result);
*/

enum conv_t
{
    ccdecl,
    cstdcall,
    cfastcall,
    cthiscall
};

enum ret_t
{
    rint,
    rsint,
    rfloat,
    rdouble,
    rptr
};

enum result_t
{
    success,
    bad_address,
    bad_arguments,
    bad_return,
    nothing_happened
};
#define OPCODE_PUSH_SIZE 0x1
#define OPCODE_CALL_SIZE 0x1
#define CALL_ASM_SIZE 0x5
#define MOV_PTR_EAX_SIZE 0x5
#define RETN_SIZE 0x1
#define OPCODE_RETN 0xC3
#define POS_BUF 0x10
#define OPCODE_PUSH 0x68
#define OPCODE_CALL 0xE8
#define OPCODE_MOV_DWORD_PTR_FROM_EAX 0xA3
#define OPCODE_MOV_DWORD_PTR_FROM_EAX_SIZE 1
#define OPCODE_MOV_EAX 0xB8
#define MOV_EAX_SIZE 0x1
#define OPCODE_CALL_EAX 0xD0FF
#define CALL_EAX_SIZE 0x2
#define MOV_EAX_TOTAL_SIZE 0x5
//this might be wrong can't remember lol.
#define addressOf(abs, eip) 
    ((abs - eip) - CALL_ASM_SIZE)

typedef void (_stdcall * empty_func)();

template<typename T> //retain SOME type security :p

result_t DllCall(ret_t return_type, conv_t convention, void * address, int argc, int args [], T * _return)
{
    /* some checks */
    if(!address)
        return bad_address;
    if(argc && !args)
        return bad_arguments;
    if(!_return)
        return bad_return;
    /* now we need to compile our own assembly, and floating point types differs a lot from standard dwords
        so we split here */


    switch (return_type)
    {
        case ret_t::rint:
        case ret_t::rsint:
        case ret_t::rptr:
            /* these look quite similar for a start */
        {
            /* next up, conventions differ. */
            switch(convention)
            {
                case conv_t::cstdcall:
                /* in stdcall, callee cleans up so we dont have to worry about that.
                    arguments pushed right to left (?) */
                {
                    /* so now we need a buffer. it needs to be:
                        (sizeof(native_int) + push_size) * argc +
                        mov eax, call_address (5)
call eax(2)
                        mov [void_buf], eax; // should be 5, cant remember
                        return (always just ret, we define the asm as taking no arguments and stdcall
                    */

volatile DWORD buf = 0; // <- this will hold the return type
volatile DWORD * void_buf = &buf;

                    int iSize = (sizeof(int) + OPCODE_PUSH_SIZE) * argc + MOV_EAX_TOTAL_SIZE + CALL_EAX_SIZE + MOV_PTR_EAX_SIZE + RETN_SIZE + POS_BUF;

                    BYTE * asm_buf = (BYTE*)VirtualAlloc(NULL,iSize,MEM_RESERVE | MEM_COMMIT, PAGE_EXECUTE_READWRITE);
                    //now, asm_buf points to out little function
int temp = 0;
                    for(int i = argc - 1; i >= 0; i--) {
                        //Here we fill in push, and the param for each argc
                        *(BYTE*) & asm_buf[i * 5] = OPCODE_PUSH;
                        *(DWORD*) & asm_buf[i * 5 + OPCODE_PUSH_SIZE] = args[temp++];
                    }
                    int args_offset = argc ? argc * 5 : 0 ;
                    // assemble mov eax, address
                    *(BYTE*) & asm_buf[args_offset] = OPCODE_MOV_EAX;
                    *(DWORD*) & asm_buf[args_offset + MOV_EAX_SIZE] = (DWORD)address;

//assemble call eax
*(WORD*) & asm_buf[args_offset + MOV_EAX_TOTAL_SIZE] = (WORD)OPCODE_CALL_EAX;

int current_offset = args_offset + MOV_EAX_TOTAL_SIZE + CALL_EAX_SIZE;
                    // assemble mov [void_buf], eax
                    *(BYTE*) & asm_buf[current_offset] = OPCODE_MOV_DWORD_PTR_FROM_EAX;

                    *(DWORD*) & asm_buf[current_offset + OPCODE_MOV_DWORD_PTR_FROM_EAX_SIZE] = reinterpret_cast<DWORD>(void_buf);

                    *(BYTE*) & asm_buf[current_offset + MOV_PTR_EAX_SIZE] = OPCODE_RETN;
                    /* now we have hopefully assembled a small function that looks like this:
                        for each arg in args
                            push arg
mov eax, address
                        call eax
                        mov [void_buf], eax
                        retn 0

                        lets call it! lol
                    */

                    empty_func  asm_func = (empty_func)asm_buf;

                    (*asm_func)(); //omg

                    // now buf should hold the return value...

                    *_return = (T) buf;

                    //all done! now clean up
VirtualFree(asm_buf,iSize,MEM_DECOMMIT);

                    //and return
                    return result_t::success;
                }




            }


            break;
        }

        case ret_t::rfloat:
        {

            break;
        }
        case ret_t::rdouble:
        {


            break;
        }

    }



    return result_t::nothing_happened;
}

int __stdcall func_one(int a, int b, int c) {
    std::cout << "a: " << a  << ", b: " << b << ", c: " << c << std::endl;
    return a + b + c;
}

int main() {
    int args[] = {1,2,3};
    int result;
    result_t res = DllCall<signed int>(ret_t::rsint, conv_t::cstdcall, reinterpret_cast<void*>(func_one), ARRAYSIZE(args), args, & result);

    std::cout << "result of DllCall: " << res << ", returned " << (int)result << std::endl;
    system("pause");

return 0;
}

E: if you're having a hard time reading this, i suggest you look at my processcall udf, which does the same thing, only in autoit

E2: small error calculating the relative address, fixed

E3: It actually works now

Edited March 28, 2012 by Shaggi

[twitchyliquid64]

Okay, here goes. First off, this is nontested, and just so you can get an idea of how to implement it. I think it works though lol. There are some comments around, but basically, you need to assemble asm in your function, you need to know convention, return type, etc. etc. It's not that hard to do with standard types, but floats and doubles use other registers and stacks, so you might wanna read up on that. functions, that returns structs, is usually very implementation defined and often optimized away so you probably wont have luck with that.

All in all, if you are not 100% about what you do, this is not the road you wanna go down if you need a stable system. Nevertheless, this is quite fun

expandcollapse popup

//by shaggi: quick dirty nontested example of dllcall

//x86 ofc
//only works with very primitive types and stdcall atm
//should be used like:

/*
    int args [] = {1, 2};
    int result;
    result_t res = DllCall<rsint>(ret_t::rint, conv_t::cstdcall, (void*)0xDEADBEEF, ARRAYSIZE(args), args, &amp; result);
*/

enum conv_t
{
    ccdecl,
    cstdcall,
    cfastcall,
    cthiscall
};

enum ret_t
{
    rint,
    rsint
    rfloat,
    rdouble,
    rptr
};

enum result_t
{
    success,
    bad_address,
    bad_arguments,
    bad_return,
    nothing_happened
};
#define OPCODE_PUSH_SIZE 0x1
#define OPCODE_CALL_SIZE 0x1
#define CALL_ASM_SIZE 0x5
#define MOV_PTR_EAX_SIZE 0x5
#define RETN_SIZE 0x1
#define OPCODE_RETN 0xC3
#define POS_BUF 0x10
#define OPCODE_PUSH 0x68
#define OPCODE_CALL 0xE8
#define OPCODE_MOV_DWORD_PTR_FROM_EAX 0xA3
#define OPCODE_MOV_DWORD_PTR_FROM_EAX_SIZE 0xA3

//this might be wrong can't remember lol.
#define addressOf(abs, eip) 
    ((abs - eip) - CALL_ASM_SIZE)

typedef void (_stdcall * empty_func)();

template<typename T> //retain SOME type security :p

result_t DllCall(ret_t return_type, conv_t convention, void * address, int argc, int args [], T * _return)
{
    /* some checks */
    if(!address)
        return bad_address;
    if(argc &amp;&amp; !args)
        return bad_arguments;
    if(!_return)
        return bad_return;
    /* now we need to compile our own assembly, and floating point types differs a lot from standard dwords
        so we split here */

    DWORD buf; // <- this will hold the return type
    DWORD * void_buf = &amp; buf;
    switch (return_type)
    {
        case ret_t::rint:
        case ret_t::sint:
        case ret_t::rptr:
            /* these look quite similar for a start */
        {
            /* next up, conventions differ. */
            switch(convention)
            {
                case conv_t::cstdcall:
                /* in stdcall, callee cleans up so we dont have to worry about that.
                    arguments pushed right to left (?) */
                {
                    /* so now we need a buffer. it needs to be:
                        (sizeof(native_int) + push_size) * argc +
                        call address (constant, 5 bytes) +
                        mov [void_buf], eax; // should be 5, cant remember
                        return (always just ret, we define the asm as taking no arguments and stdcall
                    */
                    int iSize = (sizeof(int) + OPCODE_PUSH_SIZE) * argc + CALL_ASM_SIZE + MOV_PTR_EAX_SIZE + RETN_SIZE + POS_BUF;

                    BYTE * asm_buf = new BYTE[iSize];

                    //now, asm_buf points to out little function

                    for(int i = argc; i ; i--) {
                        //Here we fill in push, and the param for each argc
                        *(BYTE*) &amp; asm_buf[i * 5] = OPCODE_PUSH;
                        *(DWORD*) &amp; asm_buf[i * 5 + OPCODE_PUSH_SIZE] = args[i];
                    }
                    int args_offset = argc ? 0 : argc * 5 + 1;
                    // assemble CALL address
                    *(BYTE*) &amp; asm_buf[args_offset] = OPCODE_CALL;
                    *(DWORD*) &amp; asm_buf[args_offset + OPCODE_CALL_SIZE] = (DWORD*)address;

                    // assemble mov [void_buf], eax
                    *(BYTE*) &amp; asm_buf[args_offset + CALL_ASM_SIZE] = OPCODE_MOV_DWORD_PTR_FROM_EAX;

                    *(DWORD*) &amp; asm_buf[args_offset
                                        + CALL_ASM_SIZE
                    + OPCODE_MOV_DWORD_PTR_FROM_EAX_SIZE] = void_buf;

                    *(BYTE*) &amp; asm_buf[args_offset + CALL_ASM_SIZE + MOV_PTR_EAX_SIZE] = OPCODE_RETN;
                    /* now we have hopefully assembled a small function that looks like this:
                        for each arg in args
                            push arg
                        call address
                        mov [void_buf], eax
                        retn 0

                        lets call it! lol
                    */

                    empty_func * asm_func = (empty_func*)asm_buf;

                    asm_func(); //omg

                    // now buf should hold the return value...

                    *_return = (T) buf;

                    //all done! now clean up

                    delete[] asm_func;

                    //and return
                    return result_t::success;
                }




            }


            break;
        }

        case ret_t::rfloat:
        {

            break;
        }
        case ret_t::fdouble:
        {


            break;
        }

    }



    return result_t::nothing_happened;
}

E: if you're having a hard time reading this, i suggest you look at my processcall udf, which does the same thing, only in autoit

Hang on a sec ... Your generating an asm stub ... ON THE FLY to deal with dynamic calls. THAT'S FUCKING AWESOME! thanks, i hope autoit does it in just as cool a way

[Shaggi]

Hang on a sec ... Your generating an asm stub ... ON THE FLY to deal with dynamic calls. THAT'S FUCKING AWESOME! thanks, i hope autoit does it in just as cool a way

I think they did at some point, but hey... who knows ^^ maybe devs could give some insight/help?

btw beware, it seems quation of the code returns invalid code...

// assemble mov [void_buf], eax                  *(BYTE*) & asm_buf[args_offset + CALL_ASM_SIZE] = OPCODE_MOV_DWORD_PTR_FROM_EAX;

compared to:

// assemble mov [void_buf], eax                  *(BYTE*) & asm_buf[args_offset + CALL_ASM_SIZE] = OPCODE_MOV_DWORD_PTR_FROM_EAX;

Edited March 28, 2012 by Shaggi

[ProgAndy]

By the way, C/invoke is a quite complete implementation for runtime binding: https://bitbucket.org/bogen/cinvoke/ It even has a mechanism similar to DLLStructCreate.

Edited March 28, 2012 by ProgAndy

[Richard Robertson]

I'm curious as to how you could possibly deal with arbitrary function prototypes and still know what to pass it and what it gives back.

[ProgAndy]

I'm curious as to how you could possibly deal with arbitrary function prototypes and still know what to pass it and what it gives back.

You have to know the calling convention, the size and the type of the parameters. Then you use asm to push the data in the correct order on the stack and fill the necessary registers. Now call the function pointer and finally do the cleanup required by the calling convention.

Edit: libffi is another alternative to C/invoke: http://sourceware.org/libffi/

Edited March 28, 2012 by ProgAndy

[Richard Robertson]

You have to know the calling convention, the size and the type of the parameters. Then you use asm to push the data in the correct order on the stack and fill the necessary registers. Now call the function pointer and finally do the cleanup required by the calling convention.

Edit: libffi is another alternative to C/invoke: http://sourceware.org/libffi/

I understand how to call arbitrary things. My question is how is his compiled program going to know what to execute in the future? That's my question.

[Shaggi]

I understand how to call arbitrary things. My question is how is his compiled program going to know what to execute in the future? That's my question.

External input?

[Richard Robertson]

External input?

Such as what? I'm not trying to say that your efforts are pointless, just trying to understand. Is it like a scripting language? I can't think of any other situation where arbitrary calls are necessary.

[Shaggi]

Such as what? I'm not trying to say that your efforts are pointless, just trying to understand. Is it like a scripting language? I can't think of any other situation where arbitrary calls are necessary.

You can ask the OP

[Richard Robertson]

The OP is the one I intended to answer the question.

[twitchyliquid64]

The OP is the one I intended to answer the question.

Its pretty much for a scripting language interpeter. Thats the easiest way of thinking of it.

[Richard Robertson]

That makes sense then. Other than dynamic assembly or runtime compiler calling, there is no way to do it. Do you have experience in either?