I use UDP in both directions.

I've tested at least 20 different places, including coffee shops, libraries, restaurants, 3g/4g cellular and never had a problem with firewalls/router incompatibility. Maybe my sample size is too small, I'd really like to have about 200 different places tested before calling it a day (that's what Beta is for), but that has been my experience.

While I haven't hit any router issues yet, I certainly have seen differences in packet loss and latency. A couple Starbucks location are the worse: they average about 15% packet loss, 1200ms ping.

Can I expect to lose potential players if I use UDP from server to client or can I accept that virtually 100% of routers out there will be able to properly deliver the packets back to the client.

Not 100%, but a very high percentage. There is a Wikipedia article that covers the four major types of NAT traversal.

For almost all NAT classes, if the NAT-protected client sends an outgoing message to public address, then for a brief (configurable) time that machine can send messages back on the same address:port pair. It does not matter if this is TCP or UDP for most devices.

Some strict environments, such as certain corporate and government systems, will lock down NAT traversal. Also some badly-configured devices will prevent NAT traversal when it is wanted.

Most consumer and corporate NAT devices will let the communications go through. Just make sure they are always on the same address:port pair and the connection is kept active.

On the few that it won't work those people should already know about the restrictions.