This post will demonstrate how to intercept unmanaged calls in the executing process. There are many reasons for intercepting unmanaged calls, among them monitoring, debugging and some other hacks.
This post will demonstrate how to intercept calls to CreateFile from Kernel32 library. The CreateFile method is called for opening an existing file and creating new one. For example, calls to File.OpenText will initiate a call to CreateFile.

Hooking CreateFile in unmanaged code

We’ll define a function pointer type for CreateFile:

typedef HANDLE (WINAPI *FileCreateFunction)(
    LPCWSTR lpFileName,
    DWORD dwDesiredAccess,
    DWORD dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD dwCreationDisposition,
    DWORD dwFlagsAndAttributes,
    HANDLE hTemplateFile);

Afterwards, we’ll store the original CreateFile method and create a hook which will redirect calls:

FileCreateFunction OriginalCreateFile = (FileCreateFunction)GetProcAddress(GetModuleHandle(L"kernel32"), "CreateFileW");

HANDLE WINAPI CreateFileHook(
    LPCWSTR lpFileName,
    DWORD dwDesiredAccess,
    DWORD dwShareMode,
    LPSECURITY_ATTRIBUTES lpSecurityAttributes,
    DWORD dwCreationDisposition,
    DWORD dwFlagsAndAttributes,
    HANDLE hTemplateFile)
{
    bool hasListener = createFileCallback != NULL;
    if(hasListener)
    {
        createFileCallback(lpFileName);
    }

    return OriginalCreateFile(
        lpFileName,
        dwDesiredAccess,
        dwShareMode,
        lpSecurityAttributes,
        dwCreationDisposition,
        dwFlagsAndAttributes,
        hTemplateFile);
}

For now, ignore the code about the callback; we’ll use this code later for notifying the managed when a file is created.

As we can see, the hook function has the same signature as the original one. This is obvious since the function caller will not be changed, just the target method. The hook observes the function arguments and forwards the call to the original call.

Now, we’ll forward the calls from CreateFile to the new hook using mhook library:

BOOL APIENTRY DllMain( HMODULE hModule,
    DWORD  ul_reason_for_call,
    LPVOID lpReserved)
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
        Mhook_SetHook((PVOID*)&OriginalCreateFile, CreateFileHook);
        break;
    case DLL_PROCESS_DETACH:
        createFileCallback = NULL;
        Mhook_Unhook((PVOID*)&OriginalCreateFile);
        break;
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
        break;    
    }

    return TRUE;
}

This code will hook the CreateFile method (OriginalCreateFile) to our new defined hook when the library is loaded into the process.

Using the library in managed code

In order to load the unmanaged library we’ll use P/invoke calls:

[DllImport("kernel32", SetLastError = true)]
static extern IntPtr LoadLibrary(string lpFileName);

[DllImport("kernel32.dll", SetLastError = true)]
static extern bool FreeLibrary(IntPtr hModule);

Loading the library:

static void Main(string[] args)
{
    IntPtr library = LoadLibrary(@"..\..\..\Debug\InterceptionLibrary.dll");
    FreeLibrary(library);
}

Right now, all the calls will behave the same, but all will be routed through our new hook (a user of the software will experience no difference).

Preparing a callback in unmanaged code

We would like to know what file is being created, so we’ll define a callback function pointer that matches it:

typedef void (*NotifyCallbackFunction)(const TCHAR* fileName);

The managed code will register a callback using the method:

extern "C" __declspec(dllexport) void RegisterFileCreateListener(
    NotifyCallbackFunction callback )
{
    createFileCallback = callback;
}

This method is exposed so it can be used by the managed code using:

extern "C" __declspec(dllexport)

Registering the callback through managed code

First, in order to call the register method, we’ll need to define a delegate into the register method will be loaded (ignore the callback delegate for now):

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
private delegate void RegisterFileListenerDel(OnFileCreatedDel callback);

In order to load the register method into the managed assembly, we’ll need to use:

[DllImport("kernel32", SetLastError = true)]
static extern IntPtr GetProcAddress(IntPtr hModule, string procName);

Loading the method is done by:

IntPtr pAddressOfFunctionToCall = GetProcAddress(library, "RegisterFileCreateListener");
var registerListener = (RegisterFileListenerDel)
                       Marshal.GetDelegateForFunctionPointer(
                           pAddressOfFunctionToCall,
                           typeof (RegisterFileListenerDel));

Now, we’ll have to define a callback delegate and method:

[UnmanagedFunctionPointer(CallingConvention.Cdecl, CharSet = CharSet.Unicode)]
private delegate void OnFileCreatedDel(string fileName);

private static void OnFileCreated(string fileName)
{
    Console.WriteLine("File opened: {0}", fileName);
}

Now, all that’s needed is to register the callback using the method loaded in the previous step:

registerListener(OnFileCreated);

That’s all, as long as the library is loaded our callback we’ll be notified on every CreateFile call.

Example

This is our final version of the main method in the managed code:

static void Main(string[] args)
{
    IntPtr library = LoadLibrary(@"..\..\..\Debug\InterceptionLibrary.dll");
    
    IntPtr pAddressOfFunctionToCall = GetProcAddress(library, "RegisterFileCreateListener");
    var registerListener = (RegisterFileListenerDel)
                           Marshal.GetDelegateForFunctionPointer(
                               pAddressOfFunctionToCall,
                               typeof (RegisterFileListenerDel));
    registerListener(OnFileCreated);

    File.OpenText(@"C:\Development\wow.txt").Dispose();
    File.CreateText(@"C:\Development\wow2.txt").Dispose();

    FreeLibrary(library);
}

Running this code will notify these files created:

Summary

In order to be notified about a native call in managed code:

1. Create an unmanaged library
2. Hook the requested method using mhook
3. Create and expose a callback registration method
4. In managed code load the library
5. Register a callback

You can download the source code example here.

Alternatives

There several alternatives. For example, another possible way to intercept calls in unmanaged calls is Detours, a library developed by Microsoft. Another possible solution is EasyHook, a library that allows intercepting unmanaged calls directly from managed code.