This is something I have kind of avoided until now because I just did not want to bother with it, but how do I test what is the minimum android version my app should support in regards of performance? Until now I have just set minimum at the lowest and the target at the highest and hoped for the best. I understand that some players on the very low end might not be able to even play the game though because the performance will be so bad so that is really not so optimal for me.

I assume I should just go out and buy one android device of roughly each version, like one android device that is 2.3, one that is 3.0, one that is 4.0 etc. But are even devices of the same version of the same performance? Say I buy an android phone that is 4.0, can I count on it having roughly the same performance as other devices and tablets that are also 4.0?

It is all very confusing and I don't know how I can test out my app to get the performance right. Is there some kind of list of devices you should test on, on any given point in time to get the best ratio between userbase/performance to target? I assume the big companies, like Rovio and Supercell etc must have some cut of point where they decide what performance to target to maximice their userbase but at the same time not sacrifice all the eyecandy because they need to keep performance down.

Maybe even some recommendations on a list of devices to buy to test on?

How do you test and what is the lowest version you target?

Android 2.x and 3.x are so old that I don't feel like it is worthwhile to even mess with them. A couple of our older games at work will start target 2.3 but the Google Play breakdown is something like 97%+ Android 4 and above. Performance of the games just isn't very good since the devices are always low spec and it is just a drain on our support to have to deal with them. That is why we made a blanket rule just to support Android 4+ and iOS 6 (soon to be iOS 7).

As far as testing, it is a work place so we have the money and people to have a range of devices. We probably have 10-12 going from Android 2.3 to 5.1 (phones and tablets) and several manufactures. Games in development and beta testing will get run though a bunch of them to see if the game works alright but once the game is out then generally just the high end devices so QA can quickly do what they need to do and a few spot checks with older devices for major new features. Android is so fragmented that it is pretty much impossible to test on everything. You can pay for a service to test your game but they aren't always cheap and they won't catch everything either (but will have a better chance then you). There are some software emulators you can use that can simulate different hardware and Android versions but they don't really work that well in my experience. For a broke indie I'd say just get a Android 4/5 tablet and call it a day. The crash reporting for Google Play is pretty good (we've fixed more then one issue we couldn't repo from the crash logs) so it can give you a chance to fix things you couldn't test.

Useful numbers.

Gingerbread (2.3.x) was the first major wave of devices. Currently it has about 4% install base. Most of these are devices are three years old or older. In normal real life terms that isn't very long, but the mobile space has been rapidly changing.

Honeycomb (3.x) was the first major round of Android's Tablet-specific system. Most tablet vendors updated to Ice Cream Sandwich (4.0) so this doesn't show up on the stats.

Ice Cream Sandwich (4.0.x) has about 4% install base. It includes many early tablets, and phones made from 2011-2012.

Jelly Bean has three variants, (4.1.x, 4.2.x, 4.3) starting with phones and tablets from early 2012. With 12% market share itself, API level 16 targets about 92% of all Android devices in use. This is what many businesses choose for their target today.

You might also select an older (lower number) API version if you want to target older devices, or chose a newer API (higher number) depending on your personal needs and beliefs. If you only old an old Gingerbread phone, target that. If you only own a new Lollipop device and don't care to target older devices, or if you want to exploit some new feature, target that instead.

I think you are confusing minimum version with performance. The Android version you chose will limit what API you have access to, so unless you are using the latest and greatest( at which point you have no choice but to use the lowest version that support those features ), then you have to do the research and see what features you are using and the lowest API level that support that feature. Features introduced in a particular API level are generally available in higher API level unless the feature becomes deprecated, so targeting a lower API-level does NOT mean your app won't run on later version.

Do you really have to worry about what build API you use? I am using v 21 now and it works on my 2.* something mobile so all API:s seems to work on all devices?

Are android version, for example 4.0 really tied to performance also though? Can't two phones that both use 4.0 have vastly different performance? It is really hard to know what device to buy.. even if I buy a couple of 4.0 devices to target that version to test and release my game on, I don't know if they are representing the general performance of 4.0?

Do you really have to worry about what build API you use? I am using v 21 now and it works on my 2.* something mobile so all API:s seems to work on all devices?

Are android version, for example 4.0 really tied to performance also though? Can't two phones that both use 4.0 have vastly different performance? It is really hard to know what device to buy.. even if I buy a couple of 4.0 devices to target that version to test and release my game on, I don't know if they are representing the general performance of 4.0?

Isn't it this why the android emulator is created for? (little offtopic, i've written the sentence correctly? I'm not a native speaker so I have some doubts :/)

With emulator not only you can test any API, but the performance offered by different models, you can select RAM, screen size, and more.

Initially API level as they already told you is not related with performance (I guess google made some improvements on performance but I seriously doubt that's the main reason for an API change). you can have a mobile with a 4 gb of RAM and android 2.3, while a low end mobile with android 5.0, it doesn't matter, so yes, two mobiles with same API can have different performance, it depends on their hardware.

Different API should have different functions, some new class and another improvements, if you want to develop a game, frob give you probably the best link you can ever have, look which API is the most used and develop for it, it doesn't have any sense being limited for a level 10 API and below when it's only installed on the 4% of mobiles.

No, the emulator doesn't match real-world performance.

In practice, even real-world devices don't have the same performance. One person can have their phone loaded up with crapware, the other lean and mean, and they'll have very different performance. Beyond that, there are cheap phones and expensive phones, high end and low end, and phones in beteween.

We've found API level is a reasonably good indicator. A 3-year-old phone is going to be far worse performance GENERALLY than a 1-year-old phone. Specific performance needs to be measured on actual devices, preferably with some in-game performance counters.