You're looking for something less expensive than what, exactly? Digital distribution via App Stores and Steam (approximately 30% 'distribution' fee?)
While I do think that 30% is too high (quick plug for the Windows Store: They drop to 20% after you make 50K USD), its important to realize that you're getting a lot more than just distribution on those channels. Firstly, you get the benefit of their platform having an in-built audience who already has their payment details in the system; do not underestimate the benefit of impulse purchasing -- You'll lose a lot of sales merely for requiring a would-be customer to fill in their details in yet another payment system. You also get the benefit of their reputation -- people know and trust that Valve and Apple have policies to ensure there's no malware hitching a ride on their purchases, and that other policies generally require the software to be of reasonable quality.
You, on your own, don't have those benefits. You can use established payment processors like PayPal or Amazon payments but still some people won't be in whatever systems you use. Its a non-trivial amount of work to support each payment system you decide to use, too, and to manage the flow of money between them and deal with all the different receipt formats. But even if you solve those problems, the bigger issue is the trust problem -- there's really nothing you can do to get people to trust you, other than to build a strong reputation for integrity of time -- lots of time. Too much time to help you be successful in the near-term.
Between USB stick or CD/DVD, there's not a lot of difference. Discs will be cheaper, but you need to be able to do 1000 qty orders before you get to a proper, pressed disc (they way mass-market discs are produced) with printed art -- any less and your disks will come from a wall of DVD burners and have a glossy sticker pressed on. When I looked at these costs years ago, it cost about 2-3 USD per unit at a quantity of 2000 -- that was a CMYK-printed, pressed DVD, with an 8-page instruction manual (CMYK cover, B/W inside), inside a DVD case with CMYK sleeve insert, all shrinkwrapped. It meant laying out 4-6 thousand bucks up front, with the possibility of not selling them all, and needing to sell 400-600 units at $10 just to break even. You can get 4GB USB sticks pre-loaded with your software in similar quantities at prices near the higher end of disc production, and you can even choose from some standard body designs and get your logo or artwork on them. For more, you can get custom bodies injection-molded for your USB stick, and/or have them packaged in a nice tin or box.
[EDIT] Looking at the link Servant provided, it seems the cost of disk production has gone down a bit -- I can configure a similar production run to what I described for about $3000. USB sticks are far more expensive at over $10000 for 2000 units in reasonable packaging, but I've seen cheaper.
At this point, I would not really consider anything other than digital ditribution, for the bulk of sales. I'd consider a physical release mostly as a sort of special perk to those who might want such a thing, but only if I were able to charge a price that made financial sense for what would probably be an order for less than 1000 units. You really have to have quite a popular game to support a physical release.