1. The quality of the artwork makes a big difference in the graphic quality of a game. A really advanced engine with bad artwork can look bad, and a really simple engine with great artwork can look great. If you're after specific effects or capabilities you should make sure the engine supports them, but if you just think the screenshots look bad keep in mind you may be able to get better results by using better quality models and textures.

Keep in mind also however that better quality artwork can sometimes cost more money, which you seem to be unwilling to spend.

2. Game engines are rarely simple to use, and if you're finding the ones that meet your desires to be too difficult you may need to work on improving your programming skills, or invest in buying something that provides simpler tools such as FPS Creator.