The Cleanroom Development Process
- It doesn't matter how many logical flaws are in a piece of code...
if they never show up at run time, the client will have a high-quality
product.
- Similarly, if a program has only one logical flaw, but that
flaw causes an error the first time the client runs the program,
the client will consider the product to be of poor quality.
- Ergo... Quality is in the eyes of the client, not the engineer
Developed by Harlan Mills (then with IBM), Cleanroom seeks
an analogy in software development with the processes of
hardware production.
The process depends on strict separation of code creation and
testing tasks, and development of accurate profiles of expected
client usage of a product.
No unit testing... verification instead
In Cleanroom, no unit testing is done. Instead, units are formally
verified using Mills' own functional techniques. Mills verification
is a cookbook approach to denotational semantics, treating
programs as mappings from input domains to output domains.
Unit programmers work with functional specs, and when code is
developed they verify it manually with functional Mills proofs
(we did some of these in COMP 204). Unit programmers are given
parsers for the source language, but they do not generate and run
any target code.
Integration testing on random inputs
After units are verified they are sent to a separate team for
compilation and assembly into a system, and then for integration
testing. Unit developers do not do any integration testing;
they are completely separated from running their code.
When an integrated system is assembled, testing is done by drawing
data at random from a probability distribution of most likely
inputs. These statistical profiles are developed by the system
integration team in conjunction with the client. The client is
interviewed and asked how the product is going to be used in
practice.
Quality is in the eyes of the client
Since the data profiles reflect input probilities for actual
(expected) client use, the errors found during random testing are
then the errors most likely to occur had the client actually
run the program in the expected usage environment.
If you find the errors most likely to occur when a client uses the
program, then the client is less likely to find errors when testing
is complete, and the client is more likely to conclude that the
program is well-built.
The process is obviously sensitive to errors in the usage profiles.
The success of random testing is dependent on the accuracy of the
statistical data distributions developed with the client.