Saturday, June 16, 2012

C# calling a C++ DLL, things NOT to forget

Whatever you do, no matter how rusty you are at doing this, remember to set the calling convention on your DllImport statements.  I spent two hours last night trying to track down a bizarre AMD ATI driver bug in a 3D rendering dll I was writing - it only occured when the platform was x86 and the requested feature level was DirectX 11, all other feature levels worked fine and everything worked fine on x64 from the get go.)

I went to bed frustrated (and tired - it was 1:45AM and I wasn't planning on that last night - lol) and woke up annoyed, and after finding very little mention of the problem on the inter-tubes figured I was doing something stupid (since it is unlikely I'm the one discovering a 'new' problem with an ATI driver and Direct3D.)  As usual, assuming I screwed up has paid off in spades.

It turns out that I was suffering some sort of stack corruption that was screwing up the creating of devices and swap chains in DirectX 11 and the root cause was my ASSumption about the default calling convention of a DllImport.  I figured it was cdecl, and my C++ dll is sprinkled with __cdecl export definitions; unfortunately, (and here's where I wasn't thinking very well) the default convention is OF COURSE WinAPI.

Once I specified the calling convention my mysterious and bizarro AMD driver issues suddenly vanished, and I have now moved on to more mysterious and problematic bugs.

No comments:

Post a Comment