I have started programming in SFML , created three mini projects , I also know the basics of Qt, so I am planning to switch from SFML to Qt5, I am happy with SFML  I am planning to switch only because I could do something in the Qt framework with fun games,   are there any drawbacks I should be aware of Qt5 in game development field ?   or Should I stay with SFML ?  Thanks!

This is really about what decisions you want to make.  The community can't really help you with the decision, and it is you who needs to live with the choice.

If SFML works for you, you already know it and that is an advantage. You can get more done quickly with a tool you already know.

You don't know Qt5, so do some research to see if it also does what you want. Learn it and use it if you want to, but it comes with the cost of a learning curve.

From there, look at the options and build a decision grid.  Figure out which is more beneficial to you, and choose that.

Qt is a technology that has little direct relevance to games - so if someone comes here and asks "should I switch to it" the answer can't be much better than "if you want to".

SFML is fine for games and if it works it's not clear why someone would abandon it. If however there was a specific project in mind, and we were told about that, it might be possible to give some advice on what technology to use (and that would probably not be Qt).