This stackoverflow thread seems fairly straightforward:
1. Configure h264.
2. Alloc a picture.
3. Convert frame from RGB to YUV.
4. Send YUV frame to h264.
I did exactly this using what is now libav for MPEG/H.264 encoding 5+ years ago. What encoder are you using, @amtri, x264?