Are you somehow CPU bound? If so, adding Frustum Culling to the work the CPU has to do has just made the problem worse.

Do you know where your current bottleneck lies?

Are you running in Debug or with the Debug layer turned on by any chance?

You do alot of math, I think the MatrixDecompose isn't the best way to go about this

EDIT: You'll probably be better off doing frustum culling outside of your Map(), I think it's good practice to keep Map() as fast as possible :) As in: setup a list from frustum checks before doing Map(), and simply copy that list into the resource..

Also: You get the Determinant in your first line, then get it again from the XMMatrixInverse() so the first line is redundant..

How I setup a frustum from a view projection matrix:

VOID Frustum::SetFromViewProjMatrix(DirectX::XMMATRIX vp)
{
	XMFLOAT4X4 mvp;
	XMVECTOR planes[6];
	XMStoreFloat4x4(&mvp, vp);
	planes[0] = XMVectorSet(mvp._14 + mvp._11, mvp._24 + mvp._21, mvp._34 + mvp._31, mvp._44 + mvp._41);
	planes[1] = XMVectorSet(mvp._14 - mvp._11, mvp._24 - mvp._21, mvp._34 - mvp._31, mvp._44 - mvp._41);
	planes[2] = XMVectorSet(mvp._14 - mvp._12, mvp._24 - mvp._22, mvp._34 - mvp._32, mvp._44 - mvp._42);
	planes[3] = XMVectorSet(mvp._14 + mvp._12, mvp._24 + mvp._22, mvp._34 + mvp._32, mvp._44 + mvp._42);
	planes[4] = XMVectorSet(mvp._13, mvp._23, mvp._33, mvp._43);
	planes[5] = XMVectorSet(mvp._14 - mvp._13, mvp._24 - mvp._23, mvp._34 - mvp._33, mvp._44 - mvp._43);
	for (INT i = 0; i < 6; i++)
		XMStoreFloat4(&Planes[i], XMPlaneNormalize(planes[i]));
    XMVECTOR corner[8];
    corner[0] = XMVectorSet(-1.0f, -1.0f, 0.0f, 1.0f);
    corner[1] = XMVectorSet(1.0f, -1.0f, 0.0f, 1.0f);
    corner[2] = XMVectorSet(-1.0f, 1.0f, 0.0f, 1.0f);
    corner[3] = XMVectorSet(1.0f, 1.0f, 0.0f, 1.0f);
    corner[4] = XMVectorSet(-1.0f, -1.0f, 1.0f, 1.0f);
    corner[5] = XMVectorSet(1.0f, -1.0f, 1.0f, 1.0f);
    corner[6] = XMVectorSet(-1.0f, 1.0f, 1.0f, 1.0f);
    corner[7] = XMVectorSet(1.0f, 1.0f, 1.0f, 1.0f);
    XMMATRIX vpinv = XMMatrixInverse(NULL, vp);
    // The 8 corner points of the frustum
    for (INT i = 0; i < 8; i++)
        XMStoreFloat3(&Points[i], XMVector3TransformCoord(corner[i], vpinv));
}

Here's how I test a box against the frustum:

BOOL PointInsideBox(DirectX::XMVECTOR p, DirectX::XMVECTOR box, DirectX::XMVECTOR boxsize)
{
	XMVECTOR box2 = XMVectorAdd(box, boxsize);
	if (XMVector3GreaterOrEqual(p, box) && XMVector3LessOrEqual(p, box2))
		return TRUE;
	return FALSE;
}

BOOL Frustum::BoxInsideFrustum(DirectX::XMVECTOR p, FLOAT sizex, FLOAT sizey, FLOAT sizez)
{
        // Test wether any box corners are inside all frustum planes or not
	BOOL test = TRUE;
	for (INT i = 0; i < 6; i++)
	{
		XMVECTOR plane = XMLoadFloat4(&Planes[i]);
		XMVECTOR axisvert = XMVectorSet((Planes[i].x < 0.0f) ? XMVectorGetX(p) : XMVectorGetX(p) + sizex,
										(Planes[i].y < 0.0f) ? XMVectorGetY(p) : XMVectorGetY(p) + sizey,
										(Planes[i].z < 0.0f) ? XMVectorGetZ(p) : XMVectorGetZ(p) + sizez, 0.0f);
		if ((XMVectorGetX(XMPlaneDot(plane, axisvert)) + Planes[i].w) < 0.0f)
		{
			test = FALSE;
			break;
		}
	}
	if (test)
		return TRUE;
        // Check frustum corner points against the box (for cases where the box is larger than the frustum)
	XMVECTOR vsize = XMVectorSet(sizex, sizey, sizez, 0.0f);
	for (INT i = 0; i < 8; i++)
	{
		if (PointInsideBox(XMLoadFloat3(&Points[i]), p, vsize))
			return TRUE;
	}
	return FALSE;
}

Oh wow, that's some increase

It mattered only because I couldn't come up with any other reasonable explanation for drawing 100 objects per frame causing you to be CPU bound and running in Debug might have been one reason. But since it's fixed we needn't delve into it any further!