meanwhile I found a suitable way:
- Get 3D coordinates of billboard sprite edges (glad to be using only rectangles!)
- Create plane according to a discussion at Google.
- Find the intersection point of the current ray with this plane.
- Determine if the intersection point resides within the rectangle.
I wonder if it is my fault or if the given hints on of Google discussion are incomplete. I think you need at least 4 dot products to determine if the intersection point is within a rectangular area.