I've been thinking in similar lines. ?@ would be a nice way telling the compiler only to accept handles. Another option that I've thought about would be to have some callback so the application can decide at compile time whether the passed in type should be allowed or not. The latter would require more work by the application developer, but would provide exact control.
The problem with the constructor sounds like a bug in the library. I'll investigate it.