Sometimes when I want to open less node in the opened list,

I would give the higher g cost node a higher rank, but the result is less nodes get opened,

the problem is that the agent is unable to find alternative path.

This problem is indeed a problem.

Thanks

Jack

Not finding a path, means you have exhausted the open list completely. (Would be worth checking that, probably.)
The reason for that is because the A* algorithm will continue expanding open nodes until a) found the destination, or b) running out of nodes to expand.

So if you are indeed running out of open nodes while you see a solution, you must be missing some nodes that should have been in there. If you can point out one, set a break point on the neighbour that should add it, and find out why the missing node is not added.