• entries
    111
  • comments
    278
  • views
    155515

I Believe in Static Checking II: Tests and Bugs

Sign in to follow this  
Promit

207 views

Ok, I slipped schedule slightly. Just think of me like VG Cats -- the update days are just a formality.

I Believe in Static Checking II: Tests and Bugs

First, I'll address a comment on the last entry:
Quote:
It's not that simple. You need a static type system or some other form of syntactic analysis if you want to verify your code is free of deadlock, for example. Dynamic typing can't do that.
The trick is that static analysis can be run dynamically any time you want (barring limitations like needing source code). And a lot of these static analysis tools don't and can't make any assertions about the code, anyway. They merely attempt to detect obvious or common mistakes in code. They can't make an assertion as strong as "this will not deadlock" except for extremely trivial situations. They're still very useful tools, of course.

When it comes to application bugs, there are three basic points in time that a mistake can be detected:
  1. Build time
  2. Test time
  3. Runtime
Not all languages/environments incorporate a proper build time, but generally speaking this is anything that involves tools that look at and process the source code (or the output binary) without doing any actual execution. Test time is where special test code is run in order to check the correctness of the code, and includes both unit testing and regression testing. (Lots of people run this step as part of their build, but it is basically a separate phase.) Lastly, we've got runtime, which is when somebody is actually using your application, whether that someone is you, QA, or a customer/client.

When it comes to bugs, we don't want to find them at runtime if we can help it. If a bug pops up at runtime, it's far too late, and in many cases, may be quite difficult or tedious to reproduce and debug. Additionally, these bugs need to be triggered by a user; they will not make themselves readily known (except for severe mistakes). Worst of all, these are the bugs that are likely to slip out to customers. Finding bugs during the build or test stages is far safer and quicker. There's far less danger of an old bug silently reappearing unnoticed, and the problems are more readily reproduced. And when they're connected to the build system, you can pinpoint with considerable accuracy what it was that caused a test to break.

Ok, so now it's fairly obvious that post-build testing is a great idea. Why is it relevant though? As I mentioned in my last entry, testing is necessary for reliably findings places where constraints break, regardless of whether you have some static checking. Tests aren't perfect, though. They only check the code that they actually hit, and they only check that code for the values they're programmed to. Getting 100% coverage from your tests is near impossible, let alone running through all of the possible states that can occur for the code. For complex situations (for example, checking that your DAG based scene graph is correctly collecting and sorting all objects to be rendered, where correctly sorted is loosely defined), simply writing the code carefully and then doing QA and debugging is likely to be cheaper and easier than any testing method. (Unfortunately, this seems to come up in games a lot.)

Despite all those problems, testing gives us a lot of opportunities to sanity check code and prevent old bugs from reappearing. The truth is, tests are for all practical purposes nothing more than customized static checks. Whether or not our code is being run (in the case of tests) or simply read and processed (in the case of static analysis tools) is just an implementation detail. Suppose Valgrind could sandbox compile and run pieces of your code in order to check thread safety. Would that really change anything? Of course not.

So with this altered definition of static checking, it'd be pretty hard not to believe in it; not doing static checking would amount to developing software by accident. Don't worry, though -- I'm not about to leave you on such a pathetic cop out. I do believe in proper static checking and static typing, regardless of what kind of test framework you've got running post-build. In other words, I prefer to catch constraint mistakes at build time rather than test time. The reasons for that preference will be the subject of the next entry.
Sign in to follow this  


3 Comments


Recommended Comments

Quote:
Original post by Promit
They can't make an assertion as strong as "this will not deadlock" except for extremely trivial situations.
Are you talking just about "the majority of static analysis tools," or are you talking about static analysis tools as a whole? If the latter, you're wrong. Tools like FDR are very much capable of analysing code to find deadlocks.

Share this comment


Link to comment
The type system I linked to earlier also proves that programs are free of deadlock (in all cases - terms which contain deadlocks are ill-typed, and ill-typed terms are not programs).

My objection was to "All this talk of static and dynamic typing is just blind dancing around this simple statement of a rather obvious fact," because it makes it sound like the checking abilities of a dynamically typed/untyped language like untyped lambda calculus, Scheme or Python are equal to the checking abilities of a typed language, just delayed. And it isn't that simple.

Share this comment


Link to comment
Quote:
Original post by Rebooted
The type system I linked to earlier also proves that programs are free of deadlock (in all cases - terms which contain deadlocks are ill-typed, and ill-typed terms are not programs).

My objection was to "All this talk of static and dynamic typing is just blind dancing around this simple statement of a rather obvious fact," because it makes it sound like the checking abilities of a dynamically typed/untyped language like untyped lambda calculus, Scheme or Python are equal to the checking abilities of a typed language, just delayed. And it isn't that simple.


Without cross-checking with the previous post, wasn't his point simply that everything you can check statically, can also be checked dynamically, at runtime. And not everything that can be checked at runtime, can also be checked statically?

Of course you're right, when the language is cleverly designed to allow stuff like this, then yes, you can detect deadlocks in non-trivial cases. But in the general case? Across languages? No matter how complex your static analysis tools, I'd hate to see you try to reliably detect deadlocks in C++ [wink]

Of course, in languages with sensible type systems static tools can make a good number of useful assertions about your code. And of course, it's pretty much impossible to do this statically in untyped languages.
But that wasn't his point, was it? His point was simply that you can always delay the checks until runtime, so you'll always be able to make at least as many assertions, and catch at least as many errors, at runtime as you can statically at compile-time.

Share this comment


Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now