Was recently getting this as well. It came down to having a single resource not getting cleaned up correctly (a Constant Shader buffer).
The key thing on this, as stated before, is going through and ensuring that any item with an ExtRef > 0 should be investigated. One thing that you can do to help is add a proper name to the resource, like so:
ZeroMemory(&constDesc, sizeof(constDesc) );
constDesc.BindFlags = D3D11_BIND_CONSTANT_BUFFER;
constDesc.ByteWidth = sizeof( XMMATRIX );
constDesc.Usage = D3D11_USAGE_DEFAULT;
result = mD3DD3Device->CreateBuffer( &constDesc, 0, &mWVPConstBuf );
if ( FAILED(result) )
mWVPConstBuf->SetPrivateData( WKPDID_D3DDebugObjectName, sizeof("mWVPConstBuf")-1, "mWVPConstBuf" );
Calling SetPrivateData on the D3D resource allows you to get a proper name in the debug spew. Saved me a great deal of time in trying to figure out what's what.