The .dll should be the correct one. .lib files are not used like that (On Linux you would use .so rather than .lib too).
However in Unity C# you do not seem to need to pass in the .dll suffix (may be optional). Perhaps try:
Have a look here: http://docs.unity3d.com/Manual/Plugins.html
Btw, this is a Unity Pro only feature so that could be your issue.
Perhaps also make sure you are not trying to compile for the Web Player. That has extra restrictions preventing native code for security sandboxing reasons.
Can you make sure your .dll does not require any other dlls? Usually if they require more .dlls then it could fail to load. There are tools on Windows that should help you find out (perhaps try depends (http://www.dependencywalker.com)).
Perhaps also make sure that you have built a native .dll. It is quite likely that you already know this but if you generated a .NET dll using Microsoft C++/clr:safe or pure, it will not work when loaded in like this.
Finally, I once noticed that when I made a Unity plugin with wxWidgets, it actually had to have the wxWidgets dll in the same directory as Unity.exe. Perhaps try this if none of the other suggestions work.