I would not recommend a graphic design degree if you want a job in games, unless you want to work for a publisher creating promotional material.
Look for a degree in animation, media design, multimedia arts, which are basically different names for the same thing.
Drawing is going to be a helpful skill to learn in any case, though you wouldn't have to be really great at it to be a graphic designer, where your job is to put together various visual elements, such as shape, text, photographs and illustrations in appealing ways.
3D is often compared more to sculpting than drawing, so you might get away with you lack of drawing skills.
A good art school will make you take drawing lessons no matter which major you choose, and by forcing you to work on these skills it is inevitable you will learn and improve.
At the end of the day, your portfolio, networking skills and attitude will take you further than a fancy piece of paper a school's name on it, and no one will ask for your grades.