What's the art style of the game? If its sprite-like, one option would be to ship assets for the lower of each bracket (that is, 480x320 and 1024x768), and then upscale rather than downscale, perhaps using an algorithm like Scale2x, or give the user the option to specify which of several upscaling algorithms they like best. You could also, optionally and perhaps for cost, offer "HD" DLC for those with the higher-resolution screens.
For the Retina 5, I'd either expand the viewfield or letter-box as appropriate -- and perhaps slide UI elements outwards so as to cover less of the playable screen.
If text and UI feature heavily in your game, you might also consider providing just UI and text at native resolution and scale for all devices, and rendering them as such. This actually goes quite a way towards "fooling" a user into not noticing the low-resolution so much -- Back in the day, Unreal running on the software renderer would render the game at 320x240 and the UI at 640x480; you never noticed the low resolution while you were in the action. Rendering UI at full resolution can also serve as a subtle visual queue for what's UI and what's not -- for example, as a hint regarding UI elements that might respond to touch, though if the entire screen responds to touch this may not work in your favor.