Howdy,

• I have an ARM Cortex A9 CPU, the NXP iMX6, dual core,
• on which runs a bare bones Linux (kernel 4.1.44).
• I'm trying to receive data over its Gigabit ethernet interface.
• goal throughput is ~ 40 MByte/s (320 Mbit/s)
• my default UDP payload is ~ 1350 bytes, I've experimented with sizes from 300 to 16k Bytes
• the ARM board is connected, via one Gigabit switch, to my PC
• nothing else on the switch
• on the PC runs the UDP client, sending data continuously
• the ARM runs the server which binds a socket to a port and then basically only does
  • while(true) { poll( socket, ...); recv( socket, ...); }
  • there is some other stuff going on like loop time jitter histogram and crude data integrity check, but commenting that out yields no difference in the problematic behavior

I.e. I'm using the socket library as available on Linux.
Now, according to what I've read, poll(..) blocks (in a non-busy manner) until there is data. I use it because it has timeout functionality.

So from that, I should not be seeing the near maxing out of the CPU (90..99 % for my process) that I unfortunately do. Unless that throughput really is too much for that ARM CPU.
But I doubt that, as iperf (2.0.5) eats "only" 50..56% CPU at ~ 40MB/s datarate. I'm not sure what exactly it does, though, looked briefly at the code and saw some fiddling with raw sockets, not sure I want to go that route... I have hardly any experience with network stuff.

Also, there was this strange effect of high packet loss, despite this short connection, while a second PC as UDP receiver has no packet loss.
I'd get it if the CPU were at 100% all the time, but it's slightly below - and iperf much less, which still can get like 2..4% loss.

I then found some Linux settings: net.core.rmem_max and net.core.rmem_default, which I set to 8 MB each via sysctl, instead of some KB it had prior.
Then the packet loss went to zero (still around 90+ % CPU load).
Today I tried to replicate this, put I have packet loss again. I had also put the cheap plastic switch somewhere else, now I'm suspecting that that thing may be unreliable... yet to test.
Can switches be a problem source like that? 

Any ideas about why the CPU load is so high?
Or, in other words, am I doing this totally wrong? What would a proper implementation of relatively high speed UDP reception look like?

Regards,
- UB
 

Hey, thanks for the reply!
50% CPU, now that you're asking, I didn't pay attention to that detail, lol! I stared at the top of "htop", perhaps it was only 1 core. It was shown with my process at the top, eating most of it, though.
I have to look into "time", and kernel level profilers sounds interesting, too, but probably quite involved to set up?
Cache coherency or rather its lack... there rings a bell, I think that problem is there.

While it does seem very plausible that my CPU just hasn't got enough horse power, I still wonder what iperf is doing better (if not exactly impressively), and why that one day, it seemed consistent zero packet loss after I set 8 MB buffers (4MB was barely not enough, there seemed to be a roughly proportionate effect).

I will try out what happens with only one system call!

Edit:
Ok, I tried it with only receive, without poll, and checking for timeout in a low frequency thread. That reduced the CPU load at best slighty. I wonder why it's not always the same - sometimes the core0 is, given my goal data rate, at 95..99% and then there is no packet loss, but when it gets to 100%, not surprisingly, there is.
Core1 is mostly < 2% busy.
I disabled the LXDE desktop completely to see whether the barely enough CPU (99%) would be more repeatable then, but it's not.

I'll look into whether all parts of my gear supports jumbo frames, to reduce the number of packets/sec I get...
=> Too bad, the iMX6 won't go above 1500 MTU.

Another edit:
I tried 2 things now, the second is semi-successful.
1) had a NVIDIA TK1 board in the drawer - unlike my iMX6 device, that thing allowed me to set the MTU to 4000, no 9000 but hey... my Windows PC also, and the switch maker also claims my switch support it.
But... that ran much worse than the iMX6, for some funny reason, and I saw kernel mesages about no memory for gluing packets together or something...
Anyway:
2) Next to setting rmem_max and rmem_default to 8MB (instead some KB defaults), I also set the CPU core affinity of my process to core1. You mentioned that probably all those interrupts run on core0 only, so I thought, how about doing whatever else possible on the other core - and bingo:
Now I have 30...55% CPU core load for both cores, no maxing out anymore, and no packet loss - unless I use the default buffer values instead my magical 8MB.

I guess getting the 4x core version then would leave me to actually do some things besides fetching the data
Don't know yet if that's a real option.