What is the best approach for scene ray-casting? I want to use it for calculating shoot target (decals positions), A.I. (visibility), picking (moving units accross the map) and possible ray-casting (some effects). There could also be some other use cases in the future.

I was thinking about an octree, but I am not quite sure what to store inside. Storing single triangles in leafs will lead to the huge waste of space (ID of geometry, ID of triangle plus the tree itself). Yes, I can have a simplified models (so less triangles), but this will work for A.I., but not for obtaining shoot target position for decals rendering and picking can also be inaccurate.

For only picking, I can use depth buffer, but it is also not quite precise for positions further from near plane. I was looking into Unity solution and they have Physics.Raycast. However, I am not able to find whats behind.

Usually you have some association between your collision geometry and units (e.g. a simple user pointer). Then you simply cast into the physics scene and get the unit back from this. Are you using a physics engine already and need help how to integrate this?

A third party physics engine seems the best solution here otherwise you could simply create your own set of bounding volumes; at least Box and Sphere/Cone and do a simple geometric Line/Box, Line/Circle or Line/Cone intersection test testing those objects that are on-screen

Depends if you are using a physics engine or not.

If you are not, start off simple. Fire a ray from some position (mouse position on the screen or is it an object in game etc.) and test for an intersection with an AABB or Sphere as these are very easy to implement. Once you have that working you, I would then start to think about how you could use something like an octree or other types of ray intersections.

I have already implemented bounding primitives / ray intersection. The problem I have is, how to detect actual triangle. Calculate it brute-force (even if I have the object from AABB or other primitive-ray intersection), is quite slow. With bounding geometry, I am unable to get precise points on surface.

How precise should you results be? Normaly a physics mesh is way more low poly than a visible rendered mesh. As I wrote in another post we have used collision detection based on a kind of quadtree, a collision tree sampling down the areas of the model to achive a seek cost of O(log(n)) for acurate collision hits

Ideally precise... I am using it in an "editor" like system, where I need to pick the actual triangle and position within it. I would also like to try ray-tracing on my scene, so I need precise ray intersections.

I am currently trying to implement it based on the outputsfrom deferred renderer (depth G-buffer and added triangle ID buffer) with a combination of octree (quadtree)

Ideally you would have some kind of broadphase (e.g. a dynamic AABB tree) which stores all the *unit* AABBs. Dynamic because you can select units and move them around. Once you have determined the unit you are hitting you need to decent further based on your current selection mode (e.g. vertex, edge, or face). Usually you have some kind of static AABB tree or a kd-tree for these operations.

Why would you need triangle selection in your editor? If you want to support basic modelling operations you need to check for a good mesh data structure (e.g. a relaxed half-edge data structure) and a dynamic structure which can be updated. You can also build the acceleration structures on the fly. In an editor you only need what it is called interactive performance. This is usually defined as the time an artists waits before he opens his web browser or email client. This is a very short time, but still much more than 16ms :)

The decals are kind of a fuzzy issue. If you have low poly collision models the results might not look good enough. In that case you need a way to also cast against the render geometry. Maybe it is a viable assumption to only cast against the static world geometry in which case you could have a very efficient kd-tree for these kind of casts,