I don't think of game design as having ideas. In fact, most successful game designers I've seen aren't the ones with the big ideas. Rather, it's really a matter of saying YES/NO to these ideas, and shaping the product accordingly so that it remains coherent.
Tossing feature ideas around is easy, but defining the minimum viable product (MVP) is significantly harder. Likewise, defining extra features not by "how cool they sound" but by "which objectives they meet" requires more thought/skills.
I tend to think that game design is harder, because, as a developer, when the requirements are clear, it doesn't take that long to figure out how we're going to architecture the work (by not that long, I mean, relative to the project's scope obviously).
Game design is much more subjective and requires to own a vision, and sell the idea. It is cross-functional and requires many different skillsets.
As a matter of fact, most of the problems developers tend to have comes from unclear design.