if you want to really understand physics well, calculus is extremely important. In non calc based physics, you would merely using prederived formulas, where as calc based physics teaches you where to derive the formulas and allows you to do muchmore interesting work. I would say that if you are interested in doing physics for games, IMHO you must learn calculus and the physics that is based on it.

By the way, calc isnt that hard. It is just very different from the math you will be used to by the time you take it. If your algebra and trig are not that strong, I would recommend brushing up on them first.