Methods described by HappyCoder and David Eberly as well as the method I described in my previous post (slower), worked and gave identical results thanks
@Postie: no I couldn't get that to work, I used barycentric coordinates to calculate circum circle of three points, and chose different set of points, but none of them gave minimum sphere of the frustum
@Mussi: I solved them using Maple, you should give it a try, nice app
@Owl: I have tried checking miniball algorithm out, Dave Eberly also has an implementation in his website, which is very complicated and as long as frustum has 8 verts and is symmetric I don't think I need to implement that.
@Dave, @HappyCoder thanks guys, as long as I got HappyCoder's method more clearly, I'll try to implement that first and check the results
another method I'm currently using is that, I use barycentric minimum bounding spheres method (described here) for two tetrahedrons inside of the frustum. if indices 0~3 is vertices of the frustum's near plane, and 4~7 for far plane, I calculate one sphere for 0-2-5-7, and one for 1-3-4-6 (opposite tetrahedrons) , and merge two spheres. I think get acceptable results from this, but it's still pretty big especially for frustums with high aspect ratio (fov is high and (far-near) is small), although I don't think I can get small bounding spheres for those, because of the nature of the shape.