The Role of Motherhood in the Pop Art of System Programing

Peter G. Neumann
Bell Telephone Laboratories, Incorporated
Murray Hill, New Jersey

Summary: Numerous papers and conference talks have recently been devoted to the affirmation or reaffirmation of various common-sense principles of computer program design and implementation, particularly with respect to operating systems ad to large subsystems such as language translators. These principles are nevertheless little observed in practice, often to the detriment of the resulting systems. This paper attempts to summarize the most significant principles, to evaluate their applicability in the real world of large multi-access systems, and to assess how they can be used more effectively.

Introduction

Many principles are widely quoted by which computer operating systems and subsystems might advantageously be designed and implemented.[1] Most of these principles have gained wide acceptance - at least in principle. The situation has been somewhat different in practice, however, with numerous potentially productive principles being widely ignored, or else only superficially applied. Thus system programming unfortunately remains an art and not a science, although its creations are often far from aesthetic. It is this situation which the present paper attempts to analyze.

Motherhood

In what follows, the term operating system refers primarily to large operating systems with interactive and noninteractive capabilities. Subsystem refers to software systems such as language translators, context editors, data management facilities and user application packages. System (without specification) refers to operating systems and subsystems collectively. The fundamental system principles under consideration may be summarized here simply by invoking (among others) the following words:

• automatedness
• availability
• convenience
• debuggability
• documentedness
• efficiency
• evolvability
• flexibility
• forgivingness
• generality
• maintainability
• modularity
• monitorability
• portability
• reliability
• simplicity
• uniformity

You may observe the intent of these words by trying to answer the following questions for each of the various systems with which you are familiar.

Automatedness: What kind of mechanized aids do you employ for designing, implementing, documenting, and using systems? How easy is system generation?

Availability: Do you have continuous service in an interactive environment? Does a given functional capability remain compatibly available for long periods of time?

Convenience: Are your systems easy to learn and easy to use?

Debuggability: How hard is it to debug a complex program or group of programs? Is the system and/or the language hindering you? Are your system debugging tools adequate?

Documentedness: Is the user interface correctly documented? Is the system itself correctly documented? Is this documentation available? Is it understandable?

Efficiency: How much capacity is lost due to bad software, bad hardware, or the mismatch between them?

Evolvability: Can you easily add more capacity or new devices to the system? Can system data base layouts or calling sequences be changed during implementation without catastrophe? Can the system be modified on-line?

Flexibility: Can you replace modules for your own use without affecting other users (e.g., private commands or special input/output)? Can you redirect input and output streams without coding changes? Can the system be dynamically reconfigured?

Forgivingness: Can files be recovered after accidental deletion? Can a directory be easily clobbered by mis-typing a command?

Generality: Is the system capable of both interactive and noninteractive service? Can large problems be run without special overlays or storage management by the user?

Maintainability: How easily can a new system programmer become effective as a maintainer? Can the system be maintained by a small crew of people not involved in the implementation?

Modularity: Can a given capability be functionally altered by making changes only to a few programs and data bases?

Monitorability: Can you easily find out where the inefficiencies of a system are localized (or distributed), both in programs and in data structures?

Portability: Can you move an operating system to nonidentical hardware without substantial recoding? Can you move a subsystem to a different operating system with essentially no recoding? Does the system design facilitate the importing as well as the exporting of software?

Reliability: Does the operating system avoid losing files and jobs? If it crashes, does it recover gracefully? Is the protection of files reliable in a shared-file environment?

Simplicity: Is the logical structure of the system simple? Is the user interface simple? Can the system be implemented with a relatively small crew?

Uniformity: Does the file system look the same to interactive and noninteractive users? Does the command structure have inherent symmetry with respect to names, arguments, conventions, defaults, etc.?

Note that modularity, simplicity, and uniformity are primary principles, in that high uniformity, simplicity, and meaningful functional modularity aid in achieving many of the other principles. Note also that some of the principles are mutually antagonistic, requiring trade-off of one for the other. However, quantitative results concerning these trade-offs are badly needed.

Problems

In particular, the principles of uniformity, evolvability, portability and convenience are most flagrantly avoided. There seem to be many reasons for the avoidance of the principles. The following all appear to be at least partially relevant.

1. We don't write. The principles are badly formulated; papers containing them are badly written, badly motivated or inconclusive, or else never written at all. Many such papers tell how a system should be or is going to be developed; few papers analyze carefully and conclusively the results of a system development.
2. We don't read. Very few system developers are familiar with work done outside of their own project.
3. We profit neither from our mistakes nor from our successes. After a success, we tend to go out and make a new collection of mistakes (such as trying to build a grand design after a small system).
4. Almost all successes occur on a small scale.
5. We tend to repeat the mistakes of others.
6. Those with experience are often unavailable for succeeding projects. (They are dejected, ejected, or promoted [executed?].)
7. The system principles cannot be applied to the desired system environment for technical reasons.
8. The principles are technically applicable, but are difficult to apply because of the working environment, e.g., the inadequate or overadequate number of people involved, their quality and talents, their organization, their development tools, unsuitable hardware, tight schedules, unattainable goals, etc.
9. The principles are economically unsound. This argument has often been used (erroneously) to rule out the use of many of the principles other than efficiency. (See below.)
10. The principles if applied at all are applied without sense. Moderation is obviously essential. (Excessive modularity, generality, and reliability can be particularly expensive).
11. Programmers as well as managers tend to be set in their ways and reluctant to deviate from previous experience, for example by refusing to use a higher-level implementation language. According to one computer savant, programmers are typically "ignorant, stupid, lazy (though on the surface full of activity), unscientific, undisciplined, and often illiterate."

In most cases these reasons (where applicable) are independent of the actual principles. This leads to the next statement.

People are of course our most important resource. However, big software efforts tend to be unstable, partly because of a shortage of automated development tools, partly because of the problems of coordinating the designers, programmers, debuggers, documenters, and maintainers, especially if there is little continuity of personnel from one phase to the next. Close interaction among these phases is essential.

A paradox thus arises: since planners are not omniscient, they must strive for evolvability; however, easy evolvability is often possible only if the planners are omniscient.

Although convenience is probably the most important principle to the users of a system, existing systems demonstrate that human engineering of systems is of little importance to designers, implementers, and marketing organizations. A user should not have to worry about myriad idiosyncrasies in languages and command structures, or about detailed memory management problems such as fitting a large program into 5000 words of storage. Conventions should be simple, commands should be uniform, diagnostics should be helpful. We have been breeding a race of programmers who are forced to worry about trivia which are semantically irrelevant but syntactically critical. R. W. Hamming's "specialization breeds triviality" and Buckminster Fuller's "specialization breeds extinction" are both to the point.

A possible way of achieving a system with better human interfaces is to force system developers to use the system itself for its own development. However, system programmers appear to be better able than ordinary users to tolerate miserable interfaces, especially their own.

The Cost Metric

The fundamental principles are often abandoned due to the apparent additional overhead they incur. However, they should be evaluated in terms of the entire effect. The effectiveness of a system depends not only on the efficiency of its software but also on the usefulness of the totality of users, as well as on many factors in the hardware and software developments. An extremely oversimplified picture of the overall cost is provided by the invoice of Table 1. The user costs in the table are those for a class of users, possibly all users of a computation center or a single user, depending on the desired scope of the cost evaluation. Each of the hidden system costs is dependent on the choice of the user class in the same way, with the cost prorated somehow out of the industry-wide hidden system costs for the system(s) in question. The effective total cost is then the sum of the direct user costs and the prorated hidden system costs. This resulting total must be compared with the value returned to the user class in order to assess the effectiveness of the system for that class of users. In this way, it is possible to justify rationally such increases in capability which enable otherwise impossible jobs to be run.

To illustrate this, consider the controversial principle of generality. Traditionally "general-purpose" conjures up an image of wastefulness, while special-purpose brings to mind an image of intrinsic efficiency. However, in systems and languages alike there is a principle of efficiency which says that the infrequent cases should pay a premium. That is, the capability which most users normally require should not include much overhead associated with the ability to handle the infrequent cases. If this is enforced, the price of generality may be offset by the increased utility. (However help is needed here from language designers and machine designers.)

Many examples of the importance of the choice of the cost metric are visible today. As an illustration, consider the controversy over batch-processing versus time-sharing. Although CPU usage may be greater in a time-sharing environment, human productivity also seems greater.[2] It is clear that other factors than that CPU usage must be assessed to determine which environment is more effective for a given application. Hopefully both environments are compatibly available in the same system (the principle of uniformity) so that each user can strike his own balance.

The Use of Higher-Level Implementation Languages

Another illustration of the choice of the cost metric involves the use of higher-level languages in the development of various systems, most spectacularly of operating systems. F. J. Corbató[3] has cited various reasons for the use of a higher-level language (e.g., a compiler language) throughout the implementation of an operating system. In essence the tasks of coding, debugging, integration, learning, maintaining and upgrading the system are much simpler if the implementation language has fewer source statements, if it masks the machine-language idiosyncrasies of the hardware, and if it is in a real sense self-documenting. The management of the project is simplified by such an implementation language, since then accurate estimates as to the size and speed of the system are available earlier. Furthermore, it is often possible to get some prototype version of the system working much more quickly (albeit possibly inefficiently) than when written exclusively in machine code. Overall, the use of such a language contributes greatly to the achievement of many of the principles.

There are of course some difficulties in using a higher-level language for system implementation. The language usually affects the design approach by enforcing peculiarities of the language on the entire system. It also provides system programmers with idiosyncrasies whose employment serves the same self-defeating purpose as machine-language curiosities; in the case of the higher-level language, however, this is usually the result of sloppy language design.

Efficiency is at first glance also a considerable obstacle. In most systems, however, much of the overhead is attributable to just a few modules. Suppose for example that 90% of the system overhead is spent in a small subset of the system code. Suppose further that straightforward handcoding (i.e., handcoding into assembly language) of a module originally written in a particular higher-level language results in a three-to-one speedup of that particular module. Thus handcoding of the entire system would give a factor of three reduction in total system overhead, viz., the optimum for a given design. Hand-coding just the smaller subset with 90% duty cycle gives a factor of two and one half reduction, resulting in a system with only 20% more overhead than in the hand-coded equivalent. If the subset is in fact small (say 10% of the object code), this is a very effective way to develop a system. However, handcoding is recommended only as a last resort, since it reduces subsequent evolvability and portability. Experience shows that much greater factors of performance improvement can be obtained by strategy changes and module tuning, especially when facilitated by the evolvability of higher-level source code. (It is even conceivable that the resulting system could eventually be more effective than an equivalent machine-coded system, if the evolvability is used constructively.)

Examples exist in Multics of modules originally developed an maintained in machine code which grew too complex to maintain and evolve easily. Upon recoding in a higher-level language, these modules actually had shorter execution times than before. (This is due to better understanding, established design, and the ease of working with smaller source code, among others.) Such examples can be contrasted with the complexity of maintaining and upgrading a large machine-coded package.

The larger size of the object code resulting from a higher-level language still remains an obstacle today, with code costs and paging (or swapping) overhead still quite significant. However, this obstacle should be largely eliminated by the expected advent of cheap, fast, directly executable read-only memories (e.g., optical) which should radically alter the situation. With such stores, the extra cost attributable to the implementation language is largely amortizable, especially with handcoding of the high duty-cycle subset.

Higher-level languages are being increasingly used for large system and subsystem implementations. Although they are far from ideal, Burroughs' extended ALGOL, JOVIAL, Lawrence Radiation's LRLTRAN, NSA Algol are examples, along with BCPL and subsets of PL/I (Multics and now SABRE). Following the above principles, such languages should be easy to learn and easy to use; programs written in them should be easy to write, to debug, to integrate with other programs, to maintain, to monitor, and to be subsequently compiled and executed under a different operating system, possibly even on different hardware.

Conclusions

At the risk of belaboring the obvious, or of falling victim to the problems in metamotherhood discussed above, I have attempted to assess some of the problems confronting the software field. Although it would be possible to flesh out this paper with many real horror stories, that does not appear to be necessary. To users with any sophistication, the state of the software art appears ludicrous and does not appear to be improving. Thus I hope that this paper will provide some useful yardsticks by which to measure software.

Although there are a few applicable theoretical results and a scattering of useful practical experience, many of the real system problems are nontechnical. For better or worse, commercial vendors appear to be dominated by short-term considerations. On the other hand, desirable long-term goals are usually not attainable as a succession of short-term goals unless the former are clearly supported throughout. This is particularly true of the stated principles. One message of this paper is that doing it right pays off in the long run, if not also in the short run.

References

1. In the interests of brevity, references are skeletal. Relevant papers are included among the Multics papers, FJCC 1965, pp. 185-247; the Princeton Reprogramming Conference, CACM, December 1965, particularly M. I. Halpern, pp. 782-5; the San Dimas Conference, CACM, March 1966, particularly J. Gosden, pp. 189-190; AFIPS and IFIP galore; S. Michaelson, How to Succeed in Software, IFIP 1968; Software Engineering, NATO Science Committee, Brussels, Belgium, January 1969, see for example, M. D. McIlroy, Mass produced software components; Program transferability study group report (G. H. Healy et al.), private draft; F. J. Corbató, Sensitive issues in the design of multi-use systems, MIT MAC-M-383, December 12, 1968. See also the papers by Mealy, by Needham and Hartley, and by Poole and Waite at this symposium.
2. M. M. Gold, Time-sharing and batch-processing: an experimental comparison of their values in a problem-solving situation, CACM, May 1969, pp. 249-259. See also H. Sackman, Time-sharing versus batch-processing: the experimental evidence, SJCC 1968, pp. 1-10.
3. F. J. Corbató, PL/I as a tool for system programming. Datamation, May 1969, pp. 68, 73-76.

ASSOCIATION FOR COMPUTING MACHINERY
SECOND SYMPOSIUM ON OPERATING SYSTEM PRINCIPLES
October 20-22 1969

"This material is presented to ensure dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder."