Sign in to follow this  
Nicholas Kong

What gets a game to pass certification by a publisher?

Recommended Posts

Nicholas Kong    1535
It seems weird or more appropriately baffling how games with bugs, low frame rate and poor gameplay pass certification in which then the game distributed to the consumers? Am I missing something about certification by the publisher?

Share this post


Link to post
Share on other sites
Nicholas Kong    1535

Also to be quite frank, the console makers are often willing to look the other way when a big company asks for some ... flexibility in hitting the requirements.

 

flexibility as in being lenient with the rules? Is it necessarily so that the game can hit the deadline? I'm guessing since the publisher is publishing the game, they obviously want the game to be shipped so they have return on investment?

 

May you give me one example of a worst case that happened to a game but the game still got shipped? If specific information cannot be disclosed, I can understand. 

Edited by warnexus

Share this post


Link to post
Share on other sites
Pink Horror    2459

flexibility as in being lenient with the rules? Is it necessarily so that the game can hit the deadline? I'm guessing since the publisher is publishing the game, they obviously want the game to be shipped so they have return on investment?


I believe that flexibility is earned in a similar way to how the Mafia earns flexibility from the police.

Share this post


Link to post
Share on other sites
Stainless    1875

In the network case you mentioned, Call Of Duty should fail. Yet it is published. 

 

Can you imagine Microsoft refusing to certify COD ?  smile.png  Don't get me wrong, as far as I am concerned they should.

 

As far as I am concerned, certification is a good thing. It can be annoying, but no more annoying than bug reports you get from internal QA.

 

I had one many years ago...

 

BUG              : Game crashes

Actions          : Press these 5 keys with your left hand, these 5 keys with your right hand, and press the spacebar with your nose

Repeatability : 100%

 

The bug fix was "Don't fecking do it"

 

There are times when certification has to be "massaged". In my experience it is always when the test case that fails was badly designed.

 

For example we had massive problems getting a JVM certified by Sun. The test  case was the garbage collector. The Java garbage collector is crap, it has a known bug in it that means it will eventually fail. The test case exercised the garbage collector and had to run for 10 hours. 

 

This was fine for a normal JVM, but ours ran the test 147 times faster than the original Sun JVM. This meant we had to run for the equivalent of 1470 hours or 2 months. After between 9 hours 47 minutes and 9 hours 49 minutes, our JVM crashed.

 

We eventually managed to get Sun to accept that it was the test case that was at fault and we got our certification.

Share this post


Link to post
Share on other sites
Nicholas Kong    1535


The bug fix was "Don't fecking do it"

 

Oh boy...

 

Still it is important to fix the bugs, very important to the customers. I can understand the game needing to meet the deadline and fixing the bugs might comprise a lot of time in other things that still need to be worked on. It still should be fixed.

Share this post


Link to post
Share on other sites
Pink Horror    2459

Still it is important to fix the bugs, very important to the customers. I can understand the game needing to meet the deadline and fixing the bugs might comprise a lot of time in other things that still need to be worked on. It still should be fixed.


I know the feeling. I've worked on some input bugs that were most easily reproduced by slamming a bunch of buttons at a certain time, and it's easy to say the users deserve to break the game if that's what they're trying to do, but those have been tip-of-the-iceberg style bugs that have revealed issues such as race conditions in the underlying input system. It's easy to complain about QA doing stupid things, but they're only bugs because programmers did stupid things.

I know too many engineers who get something up to about 95% working properly when they're supposed to be finishing tasks and then get tons of praise from management (a) for "completing" work and then (b) fixing piles of bugs later. Of course, not all of them get fixed, and we ship games with these sorts of crashes, while they laugh at stupid QA.

Share this post


Link to post
Share on other sites
Stainless    1875

 


The bug fix was "Don't fecking do it"

 

Oh boy...

 

Still it is important to fix the bugs, very important to the customers. I can understand the game needing to meet the deadline and fixing the bugs might comprise a lot of time in other things that still need to be worked on. It still should be fixed.

 

 

Was a hardware bug, turned out that the same technique crashed ALL games 

 

He had found a way of dropping the voltage at the 6510 in the keyboard to the point where the chip crashed. This has a chain effect of sending a NMI back to the 68000 and triggering a hard reboot.

 

Nothing I could have done about it.

Share this post


Link to post
Share on other sites
Stainless    1875


I know too many engineers who get something up to about 95% working properly when they're supposed to be finishing tasks and then get tons of praise from management (a) for "completing" work and then (b) fixing piles of bugs later. Of course, not all of them get fixed, and we ship games with these sorts of crashes, while they laugh at stupid QA.

 

Yes I've seen that too, but I have also seen QA totally melt down.

 

One guy didn't like the game, so he just didn't test it. He spent an hour a day fiddling with it, then just went on to doing something else. He was supposed to test against all bios versions, he tested against one. Game went out with a huge bug caused by a bios change that had not been applied to my development machine.

 

The same guy nearly caused me to cover mount a demo on 1 million magazines, with a virus.

 

He used my machine at night to play pirated games. I didn't take it very well when I found out. I swear I didn't know that wall was only plaster board.

Share this post


Link to post
Share on other sites
frob    44908

The bug fix was "Don't fecking do it"


Oh boy...

Still it is important to fix the bugs, very important to the customers. I can understand the game needing to meet the deadline and fixing the bugs might comprise a lot of time in other things that still need to be worked on. It still should be fixed.
There are two classes of bugs in that category.

One is "Don't Do That', or DDT bugs. My personal favorite DDT bug was to pause the game, alt-tab out, run the uninstaller, skip the prompts about the game still running, then alt-tab back in the game. It crashes. DDT. Another of my most favorite was when the tester pressed down on the PS2 until the disc ground to a stop during reading, repeating five or ten times until a read error appeared. Thanks for that. DDT. (We joked for weeks asking for permission to grind discs in our very expensive devkits...)

The other is "Known Shippable", or KS. These are always a little troubling, but cannot be helped. These include one-off bugs; the game crashed once doing this, we don't know why and we cannot reproduce it. They also include little annoyances that we can live with; when a player is wearing this clothing combination, is running and turning left, some of their clothing polygons clip through each other.

As the deadline approaches, the number of KS bugs rapidly increases. In the days right before submitting to certification almost every new bug goes straight to KS status; if they haven't reported it with three months of testing and it isn't a crash bug, we can probably live with it.

Share this post


Link to post
Share on other sites
Stainless    1875

I hate crash once bugs. You spend ages trying to recreate them, add loads of debug code to try and track it down and you never see it again.

 

There has to be a reason.

 

When I worked at Panasonic we had one of those, we created a special build with loads of trace information in it. Then everyone in the company took a handful of phones and went somewhere. Like the true coder I am, I went to the pub. smile.png Ever 3 minutes we dialled the speaking clock on all handsets. We made so many calls, Vodafone cut us off.

 

Shut down all the companies phones. 

 

Eventually someone got a crash, actually at their home. A team with loads of test kit jumped in a van and camped in his front room until the problem was found.

 

That's the sort of massive effort required to find bugs sometimes.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this