Just to kind of echo Promit, that is how I separate the two.
A render lib can essentially be as simple as this:
function render(material, geometry)
//rendering done here
The your game/app is just a system where by you organise a series of calls the the renderer.render function by whichever means suits your purpose.
The renderer knows nothing of how you structure your game/app, it just sits there waiting for things to draw.
What you draw, the order you draw it, where you draw it... all this information is constructed outside of the renderer.
(obviously this is just high level speak and things can get a little more complex but in general you can make things work like this and it is very flexible)
For instance, a wrote a quick space shooter game using a bitmap blitting renderer and once it was done I was able to swap out the renderer like for like for a GPU based one. Took next to no time at all because there was so very few connections between the two systems.
Many opt for an entity component system as it is flexible, you could take a look at unity as an example of one. Each game object has a number of components such as meshes, transforms, renderers etc...
It might give you some ideas anyway.
But start small! Don't try and solve every problem at once or you will struggle!
The idea of an engine is generally to run though a set of objects created by the user and update them based on some rules (such as physics) then draw them to screen.
Get that working first, then flesh it out as you go along exposing more control to the user as you go. Let them tweak positions, velocities, material textures... simple things like that to start... get that working and you will have a nice starting point!
I use the 2nd suggested method with a slight twist, I have different vertices (duplicates) used for the back so they can have unique uvs, so a texture atlas can be used (no swapping of textures or shader branches).
As always there are many way to sink a cat (or skin, both are mean if you ask me.. did I just invent a new saying by accident... I hope so).
Clipping against frustum planes is what you want, and the near plane is the most problematic one.
Do a search for triangle clipping against the near plane maybe.
Basically when a triangle intersects the near plane (either one or two of its vertices are behind the near plane) you will need to clip it and create either one or two new triangles to replace the old one.
If two points cross behind the plane will need to calculate two new points that lie on the plane using interpolation and create a new triangle using the vertex in the frustum and your two new verts. If one vertex is behind it, you will again need to make two new vertices on the plane but you will have to create two new triangles to account for the two vertices still left inside the frustum. Hope that made some kind of sense.
Deffo still some kinks to work out but you can avoid some self shadowing issues by checking normals, if you get the normal at each sample and dot it against the main normal you can use that value to determine influence. Normals that are essentially pointing in the same direction means the surface should not be casting shadow... if the normals are facing away then they should have more shadowing. You can get the normals from the depth map or if you have a normal pass available, use that.
It also looks like pixels from very different depths are having influence when they shouldn't, you need to sort out that cut-off/threshold.
You want my super duper top tip for building post processing effects?
Do it on the CPU first! It is so easy to render out some test maps (depth, normal etc...) then use the IDE of your choice to smash out the screen space effects on software.
No need for triangle rasterization just loop through all the pixels and boom.
Why do this you ask? Well being able to debug EVERYTHING at EVERY stage is very very handy! It isn't suited to every situation but I was able to build and SSAO in Flash with actionscript in a lunch break no less to prove my point:
It is not perfect but I was able to test things out and tweak variables incredibly easy, once it was working I just ported it to the GPU one step at a time and just checked it matched the results I was expecting as I went though!
It will help you spot issues very fast indeed, and if you get it working in software you then have a clear understanding of what you need to mimic on the GPU.
Also note the self shadowing on the banana where it shouldn't be, this is because this was a depth only approximation with no normal reconstruction.
There are programs out there that try and generate them for you but how well they work for you.. will be hard to know until you start experimenting.
One approach is to render your own billboards for your complex meshes and swap out to using those at a some distance threshold. To avoid popping when doing this you can fade the billboard once it is fully there hide the original mesh, This method can work well for things like trees (they are repeated often and usually not too much diversity of models.
For a simple 3D game with a level mesh and some obstacles I did a "roll your own" collision system.
Because I had already written the rendering engine I had code for matrices vectors and planes already.
All the levels were less then 15,000 polys so I wrote a super quick function to store them in a 32x32 grid structure.
So to test a sphere against it results in about 20-30 triangles being tested in a sphere vs triangle routine.
Very little code and adequately satisfying results. It was easy to integrate and very very fast.
BUT - and there is usually a but - this was only suitable for the situation and will not suit most use cases.
It was a simple game with only very basic physics, gravity/collisions/orient-to-surface, that kinda thing, no full blown rigid body dynamics. So no proper rotations, inertia, mass or any of that stuff.
If I had more bodies in the levels with more realistic physics then and engine would have been the way to go. It would have been overkill for me though as it was a simple environment.
So it does truly depend on the situation, including what resources you already have available and you knowledge level.