Good luck, Krohm.

How can a backup system be so complex that it takes several days to write an evaluation for one of its components

Basically because only a minor part of the options are orthogonal. The functionality matrix has a lot of invalid configurations, most of which are blindly accepted and don't bail out until runtime so every minor reconfiguration must validated. Sometimes the service would never reboot requiring the host to be power cycled.

Just FYI, there's a basic component - no wait, it's part of the super-duper package - which has an option "to optimize performance". I haven't measured any performance improvement but for some reason it breaks pretty much every successive component. I still haven't figured out how to use that specific option!

This basically presumes that as it is, it doesn't even work 99% of the time!

This is incorrect. The bug is deterministic thus when given the wrong input it fails 100% of the time - I've written a procedure to cause it and so far, it never failed to reproduce. When the "workaround" is implemented the data will never be in the configuration causing the bug but there sure is a small percentage of cases in which the workaround cannot be applied. I think I have chosen words poorly here.

It's like selling a car where the brakes and the safety belts hopefully work 90% of the time.

Yes, it is. I bet this is the reason for which the official "expert" is not spending any effort on promoting The program.

your boss might also need to factor in who the consumer will see as liable for their data if it does fail miserably 1% of the time

I don't get your concern. Before I started looking at this no one even had the idea this didn't work. I don't know who my boss will consider for the damage but sure I know who the customer will sue.

Talking about shitstorms, one (perhaps the most important) customer called back. Apparently the program misplaced something like 5000 files, which had to be re-catalogued manually. Upon scrutiny, it appeared the program was misconfigured since day one, a thing that doesn't take me by surprise since the UI is a total disaster and it's easier to get an invalid config rather than one valid. Let alone the one wanted.

I suspect this raised considerable doubts since it appeared obvious config was not thoroughly validated.

Anyway, I guess I can draw a line now.

Last day of work I rushed to produce the last document, which I swiftly delivered to the Technical Director, let's call him that way. Of course, there was quite some... things... in this as well, but at least no chance of data destruction.

A couple of hours later, I get called by the CEO to meet him in his office, toghether with the Technical Director, who I have been reporting since day one. Notably, the "official" expert was missing. The CEO appeared very worried the program didn't perform and told me he asked their 1st party hardware vendor for clarification but the higher business decisions were clearly stated: the program must be pushed to high-profile customers. He asked if I could modify the program to get the rid of the issues. Of course I told him this would likely be illegal and a major technical feat so I suggested to profile their customers and provide an alternative solution. To not contradict higher-rank decisions, the alternative solution would be considered "low end" while the program would officially be the hi-end, feature-complete, top-quality solution.

At first, the proposal seemed to have been taken very well. It seems like he has been looking in that direction for quite a while and starting drawing a system where... frankly put, everyone would win. Honestly, this thing got me quite excited! I'm inclined to think this would be business as business should really be in 2013.

Nonetheless, it appears the company cannot afford "logistically speaking" to embark on such a project, therefore they will call me "when the situation stabilizes in terms of workforce".

On the pro side, I won't have to deal with the asbestos slabs.

edit: small clarification on enterprise-y strategy.

He asked if I could modify the program to get the rid of the issues. Of course I told him this would likely be illegal and a major technical feat

It's not clear from the preceding why "of course" it would be illegal to modify the program. But anyway, you appear to have done the right thing and come out in one piece.

My wording is probably flaky here. I told him it's "likely [to] be illegal" because that's what happens when hacking is involved - we didn't have access to the source and even acquiring the reseller license turned out to be an involved process. The "of course" applies to what I've said because I believe it's safer for everyone saying programs by default cannot be modified rather than vice-versa, not to the fact that "programs cannot be modified".