Last week we took a look at the bailout process for HP 3000 patches. By "bailout" I mean the green light to release a patch from beta testing to the 3000 community. More than 80 patches across three versions of the MPE/iX OS are in beta. Getting them out is a matter of having them tested. But tested how much?
I posed the question to HP labs member Jeff Vance late last week, who assured me that the issue was being looked into. While I wait for HP's answer, I looked up the last reply I got from last summer. Ross McDonald, chief of the 3000 labs, said the number of test reports needed, well, it depends. On how complex the patch is, what the scope of the change covers.
Quite awhile back (late summer of '04), the 3000 lab members gathered at an OpenMPE meeting during the final HP World. I could sense the frustration, even back then, on how little testing the user community could offer for the labs' enhancement work.
2004's HP comments during that OpenMPE meeting led me to believe that fewer customers might now be required to test patches -- fewer than, say, back in 2001, before the market got busy migrating their 3000s. How many are enough? Until I receive an update — I have asked how many of the 80 "jailed" patches have at least some beta report — I'll let McDonald speak for HP, in his 2005 reply.
As with many things, the real answer is “it depends.” The goal is to deliver solutions to the customers in a timely fashion without causing other side effects. It requires engineering judgment to determine the right balance. They engineer needs to consider several factors when deciding when to move a patch to General Release (GR) status. Some of these factors are:
- How many customers that have tested the patch?
- How was the patch tested (in test environment or heavily used production environment)?
- How many systems was the patch tested on?
- How long was the patch tested?
- Does the patch solve the problem (or provide the functionality) is was designed for?
- Does the patch introduce any side effects?
- Does the code that was change have a history of introducing side effects?
- How complex was the code change?
- How urgent/critical of a problem is this solving? (we may need to take more risk when solving a very critical problem)
- How often does the problem occur?
As you can see, number of customers testing a patch is only one of many factors that go into determining when to GR a patch. We rely on the judgment of the engineers involved when deciding when to GR a given patch.