You always have the flight controller, also in a normal radio controlled drone, it's not possible to fly it without one.
The system for a normal RC drone would look like this:
Human -> RC Transmitter ===over the air====> RC Receiver -> Flight Controller -> Speed Controller(s) -> Motor(s).
At the risk of hijacking the thread:
What I wanted to achieve was: To write a computer vision program which allows a PC with a pair of cameras to control an RC vehicle as well as a human with a set of eyes. While I did find the computer vision algorithms I was looking for, I ended up giving up, beause it seemed too hard to emulate the RC. I assumed [falsely] that it would be as easy as emulating a TV remote control. My main problem, was that I couldn't find any USB rc controller.