This forum is really strange to come back to when you've been frequenting game art sites. Entirely different attitude...
I know a handful of ex-architects making a reasonably happy living off being environment artists or level designers. They like their jobs, but the nature of the industry is that it's quite difficult to weather.
A teacher of mine (who left the school last year to go work at 2K Australia) said switching to games was one of the better decisions he'd made, even with the depressing reality of getting cut at the end of big projects and working on so many games that just never come out. See, the reason he'd wanted to be an architect in the first place was that he wanted to make buildings, interesting ones. He didn't realise how much hard work would go into doing things he really didn't care about. Then he discovered environment art, and to him, it was all of the things in Architecture that he wanted to do, but were unviable and not what you ended up spending all your time on. He could cojure up any crazy design he wanted, no matter how improbable, and it would actually be viable for the project. That was about when he jumped ship and took up a mid-level environment role at a reasonably sized studio.
Thats not to say that any of this is applicable to you, but to give you a real world example of this kind of situation working out.
But especially if you have a family, it's important to warn that the industry doesn't always treat it's employ well; the example of getting cut every time a big project is finished and similar are uncomfortably common stories.
Games may actually hold the ideal job for you, but it's best if you make it a passion project. Don't quit your day job, but spend your free time exploring jobs in the industry you could see yourself doing eg, environment artist, technical artist, level designer... If you really enjoy it (the task, not the idea of making games), and you want to persue it (meaning, you love it so much you wouldn't mind slaving away on it day in and day out), THEN consider it. You say you like games? Keep in mind that if you're serious about wanting to make them, you may very well not have that much time to actually play them. And making them isn't playing them. Kinda sucks the fun out of a lot of the ones you do play, too.
Plenty of people make it just fine in the industry, but it's a combination of luck and passion. You may not want to rely on luck when you have a family to think of, but if you like it enough, it's certainly possible to give it a go with all that fire in your belly and the hard work that comes along with it. Just actually figure out if you like it that much, first. Because you may find it holds the same issues as your previous career choices.
Good luck!