When you say you won't use engines such as Unity or Unreal, is it because you would prefer to learn to code with a language rather than using a game engine?
If you want to make a 3D game, they will certainly do much of the heavy lifting for you.
I think most people will recommend you start small and build from there.
So far, I have coded Hangman, Deal or No Deal, a remake of a Commodore64 Graphic Adventure game and now I'm making a Zombie racing game. None of these even involve physics yet :)
I would encourage you to make smaller goals to begin with and go forward from there.