There are many "soft" limits.

There's the limit on an underlying Ethernet/ATM/whatever transmission packet; these vary between 48 (ATM cells) and 9000 (Ethernet Jumbo frames) or so. Upper layers will fragment across these. If you send a bigger datagram, the IP layer will fragment across this. (Unless Dont Fragment is set.)

There's the limit on UDP datagrams, where a single datagram is limited to 65535 bytes. RakNet will fragment across these -- if you send something bigger, it will send many UDP datagrams.

RakNet adds some internal book-keeping, so in practice, you will get multiple UDP datagrams on payloads less than 65535 bytes. RakNet will also fragment its own messages down to the MTU (the internal limitation of IP datagrams.)

So, no, I don't think there is a FIXED limit to the upper size, but there are of course practical limits to what will work in some particular networking situation.

The Raspberry Pi can compress to H264 using its GPU. It's one of the few things it's good at ?

The samples for the video capture/compress software are somewhat ... hard-coded to their tasks, but you can extract the "start the camera" and "run the hardware compressor" bits from the code and re-use in your own program. It's mainly an exercise in figuring out which bits you really need, and which large chunks you should cut out.