This is even more true for hardware, from what I know many hardware designers are in bad terms with programmers because they insist on completely ignoring to keep reserved bits untouched (that's another thing, if you just want to change a flag, make sure you only modify that bit rather than overwriting the entire value).

Then again don't assume that just because you document it as "must be zero" programmers will pay attention. There will always be somebody who notices that those bits do nothing and then will use them for its own purposes... Of course it's even worse when two programmers notice it separately and decide to reuse those bits with different purposes in each case - enjoy the debugging nightmare.

Errors are too easy to ignore - it might be better to just and the flag word with a mask of the valid bits.

That won't solve the problem if the library is dynamically linked, though, as soon as you'll roll out a version of the library which actually uses those reserved bits the crappy application will still be using the bits incorrectly and you're back to square one.

Yes, and moreover, if any of the "reserved" bits are set, the function should return an error.

Errors are too easy to ignore - it might be better to just and the flag word with a mask of the valid bits.

Tell that to Microsoft. A large amount of Windows programs do stuff that they should never do in the first place, and then if a new version of Windows comes along that breaks those programs relying on undefined behavior, the users will blame Microsoft, not the developers of those programs. There's a reason why Windows has such a ridiculous amount of weird stuff for the sake of backwards compatibility.

That said: a function that ends up in an error if you use reserved flags should be the kind of issue that pop ups immediately, right? I mean, the function would even refuse to work, that alone would prevent the buggy code from even working in the first place, which would force developers to not use reserved flags, period. OK, in some situations it may not be obvious and get ignored, but it should reduce the amount of cases considerably.

Simple solution: Don't just return an error, fail violently.

sometimes this is good and sometimes it is bad, and sometimes it helps find errors, or results in the app failing for stupid reasons.

case where it was good (maybe):

encoding JPEG files where the DC coefficient Huffman table had more than 16 entires;

"Windows Photo Viewer" was like "nope, not going to open this...", turns out this case is essentially reserved for some later (not widely used or supported) JPEG extensions, namely the ability to scale the quantization table on a block-per-block basis (and photo viewer cared how many entries were in the table, rather than which values were actually being used).

cases where it was not as good:

getting an error box whenever a LoadLibrary or GetProcAddress call failed, in a case where I really did want it to fail quietly;

I then ended up having to resort to using SEH to catch an exception, so that the code could quietly return a NULL on failure (it was being used to probe for optional libraries and features);

the event which originally prompted me to look into developing a custom script language in the first place (a bit over a decade ago): code within a library (IIRC, Guile) being hard-coded to call "abort()" on the first sign of trouble (*1).

*1: at the time, this seemed like damn near the worst possible solution to the problem, but it wasn't until later on that I discovered that the intention was that people would then use "signal()" with "SIGABRT" and "longjmp()" to implement a makeshift exception handler...

for the most part for C code, I have tended to prefer the "return with an error status" strategy. a lot of code basically ended up using a special pointer value (which I called "UNDEFINED"), generally with an error-status variable hidden in the background, so that it would be possible to spot the combination of UNDEFINED and an error status, and handle it as appropriate.

a better strategy in retrospect would have probably been to make UNDEFINED be a value range, where the type of condition that raised the error could be encoded within the value (with the use of an error variable, it is possible for multiple piece of code to set the status in the course of the same error, making it harder to detect the initial or "most important" error condition).

granted, initially, this was before I had the idea of encoding information inside pointer value-ranges either (tagged references and pointers were seen as separate, and tagged references would often actually encode the "index" of a heap object rather than the "address" of the object, and pointers were generally treated as raw pointers and generally assumed to normally point to accessible memory objects, ...). over the years, a lot of this would change...