If I use the above way to do pathfinding,
the agent will be wagging at its tail when moving towards the destination....
Is Dubins more costly and octile less expensive?

I try to run unsmoothed counterparts,
If the path cost is using the Manhattan and the heuristics is using the Manhattan as well.
Or Both using Octile,
it is very stable.

But Dubins is a strange creature?
Especially 3D ones...

Yes, Dubins are more expensive than octile. I used a dubin for the heuristic as well, just without looking at the obstacles. You're going to get strange results mixing octile and dubin. Dubins' take into account facing and turn radius, so you'd get cases where the octile would think it's cheap, but for a Dubin, it would be either expensive or impossible to reach.