Hello Helixirr, I am also doing a game all by myself. My advice is: screw the technical part.
What do we need to make our games? It is imperative you start from a design and a clear target. The only features we need to support are those required to make the game work. On top of that, we want to support all those features which can be implemented at low cost (to avoid adding them in a rush when we need them) and those we need to avoid go crazy.
When I stopped thinking about my design, I figured out I really required very little. Here are some lessons learned anyway.
File formats.
There are two variations. First is the aforementioned FBX. I've never used it myself but everyone keeps telling me I should. FBX has its own loading SDK and it appears to be very functional, especially when it comes to animation. There have been quite some threads about FBX in the forum and I'd say it's a very solid option.
What I'm currently using is Collada, accompanied by a JSON file. I use AssImp to load it up but the AssImp developers are focused on the graphics options only. Therefore, I had to find a way to encode my engine-specific data with the asset. There's a major issue with Collada (or perhaps with AssImp, I'm not really sure). Due to coordinate system transforms, you cannot really be sure about how to transform your vertices. The only robust solution I've found so far is the presence of special annotations.
OBJ is very old and relatively basic most of the time. You can get quite some mileage out of it but animations are a no-go.
Regardless of what you import from, runtime must use a condensed engine-specific representation.
Physics integration.
Every rigid mesh is to be enclosed in a bounding volume. In practice, hulls are sufficient in most cases. I suggest against using your physics API blob directly. Using an intermediate representation isn't very costly in terms of effort and buys some independence in my opinion. On top of everything, there's no need to wander through the API code (supposing you can for first) to figure out what's in the blob.
Polygon count and textures.
Supporting "nearly arbitrary" textures should take very little compared to supporting a fixed number. Such as 4. My models currently don't support textures at all, they rather pull up a shader for each draw-call and the shader itself causes the textures to load. Ideally each shader supports an unlimited amount of textures but of course this is a different thing in practice.
I've set the limit to 64*1024-4 vertices per-drawcall, I've set an hard limit of 1024 drawcalls per-model.
Skeletal animation.
Believe it or not, it wasn't necessary for my game so I left it out!
Tools, vertex painting.
Leave those to the DCC tool of choice. Your artist will be at home, will have all the tools he needs, access to all the documentation he needs and likely be happier. Mesh editing is hard. Leave it to the pros.
edit: minor layout