I discovered what I think is a massive blunder in calculation of average performance values that tech review sites have being making.
The problem is caused because we use “frames per second” to measure performance. In fact, FPS is a derived value. The original measurement is the time to render an individual frame and we invert this to get the frames per second. For instance, if a game runs at 30 FPS, that means the time to render an individual frame is 33.333 ms.
Now, it’s pretty elemental math rule that you cannot add inverse values:

1/x + 1/y does not equal to 1/(x+y)

This means it’s meaningless to add “frame per second” values.
So, if you want to get an average value of performance for a number games you must use the render time, NOT frames per second to calculate the average value.
For instance, supposing a GPU card got 10 fps game X and 40 FPS in game Y, the average value is 16 FPS, not 25 FPS.
10 FPS = 100 ms to render one frame.
40 FPS = 25 ms to render one frame.
-> Total time to render one frame from both games = 100ms + 25ms = 125 ms
-> average time to render one frame = 125ms/2 = 62.5 ms
-> average FPS = 1/62.5 ms = 16 FPS

Now, take look at the review of Sandy Bridge processor on Anandtech: http://www.anandtech...ile-landscape/5

You see on that page he gets the average FPS for the six games just by totaling the FPS and dividing by six.
The figures in the last graphic are hence all incorrect.

Thoughts?

Yes, it is commonly known that FPS is not a linear metric.

For programmers, comparing execution times for e.g. different algorithms or unoptimized and optimized versions of the same code using FPS is a catastrophic idea, due to the problems you have discovered. That's why proper profilers display absolute "time taken" values, e.g. like this.

For end-users, comparing identical code on different hardware using FPS is not that bad, since it still allows "x is n% faster than y" comparisons, and FPS is a good value to express the subjective aspect of performance in, e.g. I regard 60fps being the bar to beat, since that corresponds to the vsync rate of almost all modern displays.

I think that 1/x + 1/y == (x+y) / (xy) ... and that you're pretty good at observing the fine details.

The definition of the arithmetic mean is the same whether you're using FPS or seconds per frame. You would still just add all of the numbers. You're proposing using the harmonic mean to describe the center of FPS rates.

First of all, neither one is mathematically correct and there are other measures of center such as the median and geometric mean. However, each measure of center has different behavior when it comes to values that deviate far from the rest of the values.

If you're trying find the average performance of a graphics card, then each game should be given equal weight. Also, if the card has exceptionally low performance on one game, that sample should be given less weight because this would indicate that the game has an optimization problem with that card or in general.

To compare the harmonic and arithmetic means, assume that two different cards had these FPS rates for a sample of 3 games:
60, 55, 30
150, 150, 28

The 3rd game can be assumed to be badly optimized and the second card is obviously much faster than the first one.

Harmonic means:
44.0
61.2 (40% more than the first card)

Arithmetic means:
48.3
109.3 (127% more than the first card)

If the harmonic mean was used to compare the two cards, then slow games would have more weight than fast ones. This isn't what we want since slow games are generally poorly optimized. In this example (where the 3rd game was bottlenecked by something other than the graphics card), most people would consider 127% to be more representative of the 2nd card's speed relative to the 1st one, rather than 40%, so the arithmetic mean would be preferred over the harmonic one.

Thanks for your answer, sundersoft.
It's interesting that you mention harmonic mean, because if you look at the wikipedia definintion it says the following:
"In certain situations, especially many situations involving rates and ratios, the harmonic mean provides the truest average."
FPS is most certainly a rate, so that would back up my argument (basically, getting the harmonic mean of the FPS is the same as getting the arithmetic mean of the render time).
Regarding your other point, you certainly can't assume that all games that get bad scores are because of "optimisation errors". Some GPUs perform badly with shader heavy games, for instance. But seeing as we don't have the source code, we really don't know.

It's interesting that you mention harmonic mean, because if you look at the wikipedia definintion it says the following:
"In certain situations, especially many situations involving rates and ratios, the harmonic mean provides the truest average."
FPS is most certainly a rate, so that would back up my argument (basically, getting the harmonic mean of the FPS is the same as getting the arithmetic mean of the render time).

I'm not going to argue whether it is right or wrong, but just providing a point of view and something to think about.

Yes, the frame rate is a rate and the harmonic mean will give you the average of the rate. But if the frame rate itself is not what is truly and ultimately interesting, then it really is no more right or wrong than any other averaging (within reasons of course). If they consider the frame rate sort of like a performance index instead, then it isn't really a rate anymore but just another performance value.

The difference only matters if you get technical about the actual meaning of the values. The example you made about rates and how you should average frame times instead of frame rates, or equivalently calculate the harmonic mean instead of the arithmetic mean of the frame rates, is good if the average frame rate is what you're really after. If you just use the frame rate as a general performance index, which is reasonable, then the arithmetic mean also provides a reasonable average result.

The arithmetic and harmonic mean differ in how outliers bias the result. In sundersoft's example, an unusually slow game resulted in a harmonic mean that (in my oppinion) resulted in a too low difference between the mean values. In his example, I do agree that the second card performed more like 127% better than 40% better since the third game's bad performance was an outlier. The reverse relationships for the arithmetic and harmonic mean would have been true if the outlier was an exceptionally fast game instead.

Unlessthere is a technical reason the actual rate is what you really want the mean of, then neither mean is really wrong. They just have different properties and you have to select the one with the desired properties. As in sundersoft's, and my extended, example, do you want an exceptionally fast or slow game to bias the result?

Thanks for your answer, sundersoft.
It's interesting that you mention harmonic mean, because if you look at the wikipedia definintion it says the following:
"In certain situations, especially many situations involving rates and ratios, the harmonic mean provides the truest average."
FPS is most certainly a rate, so that would back up my argument (basically, getting the harmonic mean of the FPS is the same as getting the arithmetic mean of the render time).

This means that, if you ran one game for 100 frames, stopped it, ran the second one for 100 frames, stopped it, etc, then your overall average frame rate would be given by the harmonic mean. This is not very useful.

Regarding your other point, you certainly can't assume that all games that get bad scores are because of "optimisation errors". Some GPUs perform badly with shader heavy games, for instance. But seeing as we don't have the source code, we really don't know.

That was an example to prove that the data is plausible. If you agree that the data is plausible, then you need some way to measure each GPU's overall performance taking into account only the data. You can use either the harmonic or arithmetic means for this purpose, or some other measure of center. In my example, a game that was not significantly affected by the choice of graphics card caused the percent increase in performance of the second card based on arithmetic mean to change from 161% (if the 3rd game is removed from the data) to 127%, whereas the percent increase associated with the harmonic mean would change from 161% to 40%.

Most people would consider the harmonic mean to perform worse in this case, but this is subjective. You also need to decide how likely it is to have a slow game that is not affected by choice of graphics card, and whether the harmonic mean has any other differences with respect to the arithmetic mean that would redeem it if you consider its behavior in this case bad. Both of these considerations are pretty subjective.

I don't agree that methods of calculation means are subjective. The correct one should be used depending on the nature of the data.
But your other point is good. Some games are easier to render than others and so a GPU doing well in those games gives a disportionate higher mean. However, there is an easy solution for that - normalise the results beforehand. That can be done by using the "geometric mean": http://en.wikipedia.org/wiki/Geometric_mean. As the geometric mean does not use addition the problem I mentioned originally is avoided, so basically it solves both problems.

I don't agree that methods of calculation means are subjective. The correct one should be used depending on the nature of the data.

I didn't really (mean to) say that the method to calculate the mean is subjective, but that the interpretation of the meaning of the data is subjective. As I was talking about in my post, if it is a rate you're measuring then the harmonic mean is the correct way to get the average of the rate. But if you don't consider the frame rate an actual rate for presentation and conclusion purpose, but rather a generic performance index, then it is not necessarily a rate you want anymore. That is what's subjective, whether the interpretation of the data and the desired result really is considered a rate or not.