Mythical Man-Month walkthrough, Chapter 1: The Tar Pit

Home

News: Mythical Man-Month walkthrough, Chapter 1: The Tar Pit

  1. Fred Brooks' "The Mythical Man-Month" is a classic in the field of computing literature, because it addresses the impact of complexity in systems implementation. That said, much of the work in the last ten years in Java has been focused on creating modular components with the focus of removing complexity. The same could be said of architectural designs such as SOA and REST, for that matter. Therefore, it's worth asking if The Mythical Man-Month isn't as relevant as it was. The book addresses the implementation of OS/360, the operating system for IBM's mainframe of yesteryear. OS/360 involved a mountain – literally – of effort. Millions of lines of code, seven months of architectural analysis, one hundred and fifty programmers, and budget constraints are factored in. What I'd like to do here is consider, a small bit at a time, the lessons that enterprise Java developers can pull from this seminal work in today's environment, given that the methods of today can be quite different, to say the least. The breakdown will be broken up into many pieces, because Mr. Brooks goes into a lot of detail in a lot of topics, and it's not fair to summarize the book in the interest of brevity, because many of the lessons even possible – if only by contrast – would be lost. The edition being considered is Addison-Wesley's 20th anniversary edition, 18th printing. This printing includes four additional essays in addition to the text contained in the original printing. The first chapter is called "The Tar Pit," referring to the programming systems product and practice. He points out that we regularly read accounts of a few garage programmers who've put out a system that exceeds the efforts of large, well-run teams. The question posed is: why, then, do we not replace large teams – no matter how well-run – with two and a half men coding away in garages? The answer is that the small systems don't really compare with the products that require large teams. Writing a wiki, with all due respect to wiki authors, isn't that hard. (Your Humble Editor put one together in five hours from scratch, for example, and there are certainly people who can do even better.) Tiny systems tend not to include in-depth analysis, requirements analysis, documentation, or even maintenance in some cases. This is where the age of the book appears, of course. Developers who use agile methods will say that heavy requirements analysis tends to yield lots of stuff users simply aren't going to need, and an overemphasis on design locks one in to a design that may not be appropriate as requirements change. That said, though, the shift of design away from complexity is an illusion – we, as Java programmers, just rely on canned complexity. We don't write an ESB, as Brooks' team would have had to do; we use one that someone else has written. The complexity is still there, just not localized, and the issues present haven't gone away... we just deal with them differently than monolithic designs would have had to. The title is fascinating. If you're unfamiliar with what a tar pit is, consider the La Brea Tar Pits, considered one of the world's best repositories for fossilized vertebrates. Animals would wander on top of a tar pit (which was hidden through dust, leaves, or other such detritus), and become unable to escape. Predators would be attracted by the struggles of a trapped animal, and become ensnared themselves. That's very much like common architectures today, where most of the application structure is trivial, and the last 20% of the application takes forever to work through (or escape.) In many ways, the "tar pit" describes much about our field, and symbolizes where the endless drives for componentization, software-as-services, enterprise service buses, service this, service that, service the-other. The use of services is designed to reduce complexity, by offering clear (and hopefully sane) interfaces, that mandate simple requirements. That said, life tends to not be so simple, so the cure – simple services – tends to be mangled by the addition of more complexity. Thus, we have the "Simple Object Access Protocol," which used to be the acronym behind SOAP... which gets WS-I, WS-Transaction, SAML, and many other protocols on top of it, all in the attempt to provide capabilities to the "simple" protocol. (This is where the WS-"Death Star" moniker comes from, as a way of referring to every WS protocol.) In the end, we have stopped pretending SOAP is "simple" and now SOAP stands for... SOAP. Nothing else. No "simple," no "object." It's just a specification that looks like it should mean something. SOAP is an example of the "tar pit" in microcosm: something that looked attractively simple, yet turned into a trap. The first chapter of The Mythical Man-Month is a throwing down of a glove: expressing the need for large teams, in comparison to the perceived effectiveness of small teams. Small teams can have great success with small projects, Brooks says, but that doesn't translate to large projects. Maybe the focus on enterprise capabilities and applications by TSS is a mistake; the current trend seems to be yet another attempt to create small components that can be tied together to create a 'large project' out of small building blocks built by small, elegant teams. We'll continue stepping through this book to glean some lessons and discussions from it. Feel free to let us know what you think; extra perspectives are good.

    Threaded Messages (28)

  2. I started to read that book a few years ago (same edition), but it felt dated to me and so I stopped. Besides that, it seems to me that Fred Brooks does not cover so many topics of good project management. Tom de Marco's book "Peopleware: Productive Projects and Teams" and "Death March" by Edward Yourdon were more interesting to me, frankly said.
  3. I started to read that book a few years ago (same edition), but it felt dated to me and so I stopped.

    Besides that, it seems to me that Fred Brooks does not cover so many topics of good project management.

    Tom de Marco's book "Peopleware: Productive Projects and Teams" and "Death March" by Edward Yourdon were more interesting to me, frankly said.
    DeMarco and Lister's "PeopleWare" is a fantastic book - I'm going to be rereading it to see if I can find lessons for the TSS audience as well. However, I don't think Brooks' work is too dated. Sure, it's from 1975 or so, but part of avoiding the mistakes of the past is learning about the mistakes of the past, and Brooks has a lot of good ideas in there as well. They may not be "cool" or "hip" based on what the Agile people say, and certainly "classic" development processes aren't always as much fun, but I think there's a lot to take from it. Who knows? Maybe you're right, and we can strip TMMM of its' "classic work" label, or maybe there's something to be taken from Big Iron systems work, even for applications that don't seem directly related.
  4. What? I don't remember what was contained within the book as much anymore but do know that I loved his writing style and esp. the stories about how some of the projects had "issues". Anything with "peopleware" in the title sounds much less compelling. but maybe that's just me...
    I started to read that book a few years ago (same edition), but it felt dated to me and so I stopped.

    Besides that, it seems to me that Fred Brooks does not cover so many topics of good project management.

    Tom de Marco's book "Peopleware: Productive Projects and Teams" and "Death March" by Edward Yourdon were more interesting to me, frankly said.
  5. I started to read that book a few years ago (same edition), but it felt dated to me and so I stopped.
    In a way, the fact that it is outdated is a good thing. It's easy to become convinced that things are completely different now and that it no longer applies. But this is an illusion. Scratch beneath the surface and it's clear that very few of Brooks lessons have been learned. For the same reason "Extraordinary Popular Delusions and the Madness of Crowds" is a great book to read even though it's over 160 years old especially if you plan on investing your hard earned money in the next hot thing.
  6. There is much to learn from the book[ Go to top ]

    Like most people I know of my generation (I started programming in 1968) reread this book every few years. I think it remains as relevant and valuable as the day it was written. Readers of the Mythical Man Month might also be interested in two documents written about 6 years previously. The reports on the 1968 and 1969 NATO Summer Schools on Software Engineering Techniques are freely downloadable here: http://homepages.cs.ncl.ac.uk/brian.randell/NATO/. I reread the second report every few years as well. I think that these three documents are trying to understand the problem of large scale software development rather than proposing solutions. Because the problem is unchanged over time then the value of the work is unchanged.
  7. You're right. Things have TOTALLY changed. Why, a bunch of salespeople came in last week and cooked up an enterprise application right in front of me in a thirty minute meeting. It is clear to me that software has improved so much that I've fired my developers since this software will allow my business analysts to build software automatically! Seriously though, anyone who says the Mythical Man Month doesn't apply to modern times...hasn't done any real software development.
  8. It is clear to me that software has improved so much that I've fired my developers since this software will allow my business analysts to build software automatically!
    That's not funny. The poor suckers that keep buy these products should be pitied, not mocked. You wouldn't make fun of a Grandma that gave her life savings to a 'Nigerian exile', would you? And those that repeatedly fall for this are clearly suffering from some sort of mental defect and will surely be a protected class in the near future.
  9. It is clear to me that software has improved so much that I've fired my developers since this software will allow my business analysts to build software automatically!


    That's not funny. The poor suckers that keep buy these products should be pitied, not mocked. You wouldn't make fun of a Grandma that gave her life savings to a 'Nigerian exile', would you? And those that repeatedly fall for this are clearly suffering from some sort of mental defect and will surely be a protected class in the near future.
    That's funny. I thought they already fired all the business analysts already because the manager of the users can just explain everything directly to the programmers. The new fangled tool was supposed to let middle managers build applications that enable them to fire all their actual works.
  10. That's not funny. The poor suckers that keep buy these products should be pitied, not mocked.
    No, mate, they should not be pitied, they should be fired! Most of the time they are the "business analysts" or "managers" who get extremely well payed, because there business is so complex (like, say devising phone plans and bundling them with phones, which *is* fairly complex, or at least can be) and their work secures the competitive advantage of their company (or should). Yet they happily believe, that there is a product out there that out of the box or with very little customization can support the processes that they are payed so well to figure out! Pitied! Hah!
  11. Willful misreading?[ Go to top ]

    Brooks Law - the maxim for which the book is most famous - states (more or less) that "adding more people to a late project makes it finish even later" is based on an understanding that communications overhead between members of a large team consumes too much time for practical projects. Even the title of the book strongly suggests the idea that calculating a project schedule based on man-months and then increasing the "man" component to compress the schedule is a serious mistake. Complex problems have intrinsic "think time" - an observation consistent with the agile view of incremental and iterative progress based on continuous delivery of releases of useful working systems. I'd say that this, coupled with the chapter on "surgical Teams" makes Mythical Man Month, in total, a fairly strong argument FOR small agile teams. I don't have my copy in front of me, but my recollection is that the first chapter emphasizes the complexity of systems software - not the need for large teams.
  12. Re: Willful misreading?[ Go to top ]

    Brooks Law - the maxim for which the book is most famous - states (more or less) that "adding more people to a late project makes it finish even later" is based on an understanding that communications overhead between members of a large team consumes too much time for practical projects.

    Even the title of the book strongly suggests the idea that calculating a project schedule based on man-months and then increasing the "man" component to compress the schedule is a serious mistake. Complex problems have intrinsic "think time" - an observation consistent with the agile view of incremental and iterative progress based on continuous delivery of releases of useful working systems.

    I'd say that this, coupled with the chapter on "surgical Teams" makes Mythical Man Month, in total, a fairly strong argument FOR small agile teams.

    I don't have my copy in front of me, but my recollection is that the first chapter emphasizes the complexity of systems software - not the need for large teams.
    Simon, that's an excellent point. You're right - he's saying that that complexity has requirements, more than team size is bad. He's simply saying that small teams aren't always an option. Surgical teams, honestly, was the inspiration for walking through the book in the first place... but don't tell anyone this! That's chapter three, and I've still got chapter two to finish writing up...
  13. Re: Willful misreading?[ Go to top ]

    Brooks Law - the maxim for which the book is most famous - states (more or less) that "adding more people to a late project makes it finish even later" is based on an understanding that communications overhead between members of a large team consumes too much time for practical projects.

    Even the title of the book strongly suggests the idea that calculating a project schedule based on man-months and then increasing the "man" component to compress the schedule is a serious mistake. Complex problems have intrinsic "think time" - an observation consistent with the agile view of incremental and iterative progress based on continuous delivery of releases of useful working systems.

    I'd say that this, coupled with the chapter on "surgical Teams" makes Mythical Man Month, in total, a fairly strong argument FOR small agile teams.

    I don't have my copy in front of me, but my recollection is that the first chapter emphasizes the complexity of systems software - not the need for large teams.
    Simon, that's an excellent point. You're right - he's saying that that complexity has requirements, more than team size is bad. He's simply saying that small teams aren't always an option.

    Surgical teams, honestly, was the inspiration for walking through the book in the first place... but don't tell anyone this! That's chapter three, and I've still got chapter two to finish writing up...
    Surgical teams in relation to xp pair programming. That could spawn some interesting discussion.
  14. Re: Willful misreading?[ Go to top ]

    Complex problems have intrinsic "think time" - an observation consistent with the agile view of incremental and iterative progress based on continuous delivery of releases of useful working systems.
    I agree whole heartedly that complex problems have an inherent "think time." But the impression I've received from members of the Agile community is that think time is to be discouraged because thinking doesn't produce usable software.
  15. Great Idea[ Go to top ]

    Reviewing the timeless lessons of the Mythical Man-Month is a great idea. After reading your entry, I am motivated to find my old copy and re-read it, which I tend to do every 5 years or so. While parts of the book are dated, I am always struck by how much of his analysis is still completely relevant, 30+ years after publishing. My copy is about 20 years old so I should probably buy a new one to read the additional essays. I look forward to your future posts on this.
  16. The question posed is: why, then, do we not replace large teams – no matter how well-run – with two and a half men coding away garages?

    The answer is that the small systems don't really compare with the products that require large teams.
    This isn't something that Brooks addresses in MMM but to me this is the primary argument for building systems in a modular and composable manner. It allows a large system to be developed as a number of small programs. Of course, this is easier said than done. Note that I didn't mention reuse. That is a secondary or even tertiary benefit. But it seems like whenever someone explains the benefit of OO (for example. other techniques provide similar benefits) they almost always start with reuse.
  17. Maybe I am too young to understand all of that book. I liked his idea of architectural integrity. People need to see the big picture, which they can't if for example the components which each person designs and implements are too small sized.
  18. Requirements are a Tar Pit[ Go to top ]

    Requirements have been the tar pit on larger projects for my organizations. Particularly critical is when one requirement has implications on another. These implications are not always obvious until time has passed and the entire scope of the problem has been thought out. However, designs follow closely on the heals of requirements gathering so that an "absorption" of the requirements isn't complete. Your body could digest every gram of food you eat, but it's how much your body absorbs that influences your health. As an analogy you could say that we "digested" the requirements but did not "absorb" them. Absorption is really the lurking key phase that's overlooked - and tough to estimate when it will happen. Until the problem is in one's head (to borrow from Paul Graham), requirements can lead to designs that are inappropriate. Enter the tar pit!
  19. Indeed![ Go to top ]

    Requirements have been the tar pit on larger projects for my organizations. Particularly critical is when one requirement has implications on another. These implications are not always obvious until time has passed and the entire scope of the problem has been thought out. However, designs follow closely on the heals of requirements gathering so that an "absorption" of the requirements isn't complete.
    Very well said. IMHO, another interesting idiosyncrasy of requirements is the matryoshka effect: one requirement has a little other requirement enclosed within itself. That makes this new requirement invisible to the crew at the moment of definition, and so on.
  20. Re: Indeed![ Go to top ]


    Very well said. IMHO, another interesting idiosyncrasy of requirements is the matryoshka effect: one requirement has a little other requirement enclosed within itself. That makes this new requirement invisible to the crew at the moment of definition, and so on.
    Yes! The matryoshka effect is more insideous than what I described because it's often not seen until a project is far along. IMO, solving the requirements tar pit is pre-requesite for moving large systems development to a more advanced stage. Unfortunately, I don't see a solution.
  21. Requirements tools[ Go to top ]

    The problem with requirements is often that they just get put into Word documents. WORD IS NOT A REQUIREMENTS TOOL! You need to be able to express dependencies, track and broadcast changes, and links or relations to other artifacts (like... code!) in ways that Word simply is not designed to do. If you need to express the requirements as a Word document it should be generated by the real requirements tool. You know you have a requirements tracking system in place when you can go into your code editor and see what requirement a certain piece of code implements, or you change a requirement and get a report on which other requirements and code you need to check for consequences of the requirement change. And before you ask: No, we don't have such a system in place either.
  22. Re: Requirements tools[ Go to top ]

    The problem with requirements is often that they just get put into Word documents. WORD IS NOT A REQUIREMENTS TOOL! You need to be able to express dependencies, track and broadcast changes, and links or relations to other artifacts (like... code!) in ways that Word simply is not designed to do. If you need to express the requirements as a Word document it should be generated by the real requirements tool.
    Are there any good Open Source requirements tools out there? The problem is the Word docs and PPTs are all necessary deliverables. I thought I was being really clever once and used some XSLT to translate a Use Case Model into a big hyperlinked requirements document in HTML. I liked it. No one else really understood it. It was missing the glue and organization that make a good requirements document comprehendable. After all, 90% of writing requirements is setting their context - not making a big long list of things the system has to do.
  23. Re: Requirements tools[ Go to top ]

    The problem is the Word docs and PPTs are all necessary deliverables. I thought I was being really clever once and used some XSLT to translate a Use Case Model into a big hyperlinked requirements document in HTML. I liked it. No one else really understood it. It was missing the glue and organization that make a good requirements document comprehendable. After all, 90% of writing requirements is setting their context - not making a big long list of things the system has to do.
    How long ago was that? The reason I ask is that Word now (theoretically) supports XML including DocBook at least to some degree. I've been thinking a lot lately about the inadequacies of Word (or any other standard word-processing application) for technical documentation. What I would like to have is a way to build templates that act more like forms. In other words, present a structure of what the document should provide without having to copy documents or write boiler-plat documents with sections to fill in. DocBook is a decent starting point because it already has a large eco-system of stylesheets and tools for creating documents such as PDFs.
  24. Re: Requirements tools[ Go to top ]

    How long ago was that? The reason I ask is that Word now (theoretically) supports XML including DocBook at least to some degree.
    Four or five years ago. On a subsequent project I tried to switch to MS Word XML but the effort kind of faded because really it was just a tool for me.
    I've been thinking a lot lately about the inadequacies of Word (or any other standard word-processing application) for technical documentation.
    One of the challenges with requirements is that they really straddle the fence between true technical documentation and "business" oriented documentation. A lot of the contents are heavily structured but also a lot are not.
  25. Re: Requirements tools[ Go to top ]

    One of the challenges with requirements is that they really straddle the fence between true technical documentation and "business" oriented documentation. A lot of the contents are heavily structured but also a lot are not.
    I agree. This is why I think XML and tools like DocBook can be useful. You can have sections that are structured and sections that are not in the same document. You can add unstructured elements to structured ones and vice-versa. Also, I'm not strictly speaking of requirements and I hadn't considered that much before you mentioned it. What I see a lot of now is that we have a 'template' for certain kinds of designs with required sections and a lot of boilerplate that is very similar from one document to the next but not always the same. Everyone copies some other document to start and invariably something from the previous document is left in the new document. And I personally am getting tired of fighting with Word to get the bullets to work right and with screwed up tables of contents and crazy page numbering etc. One of our standard documments requires a table and I always end up spending at least a half-an-hour trying to get everything on the page in a font that's large enough to read. Some of this is caused by inadequacies in Word but for the most part, I think it's just that Word is not really designed for technical documentation. A lot of it's features just get in the way. I need spell check but Word doesn't understand camel-case (tell me if I'm wrong here, please) for example. The cool thing about DocBook is that you can have your document and it's structure separated from the formatting. The problem is that there's a (rather silly IMO) dogma in the DocBook community that WSYIWYG is useless and that you should write in XML. Of course I'll drop down to XML when I need to, but a lack of tools is holding DocBook back, if you ask me.
  26. Re: Requirements tools[ Go to top ]

    How long ago was that? The reason I ask is that Word now (theoretically) supports XML including DocBook at least to some degree.


    Four or five years ago. On a subsequent project I tried to switch to MS Word XML but the effort kind of faded because really it was just a tool for me.
    I took a little time to play around with using XML word documents. It's worthless to me. Totally not what I am looking for. I've used XMLMind (a DocBook editor) in the past and while it was helpful, it didn't quite cut it for me. There may be software for working with DocBook that is better but is not free. If anyone has used any tools for WSYIWYG DocBook editing please share your experiences.
  27. Identifying Big Game[ Go to top ]

    The first chapter of The Mythical Man-Month is a throwing down of a glove: expressing the need for large teams, in comparison to the perceived effectiveness of small teams. Small teams can have great success with small projects, Brooks says, but that doesn't translate to large projects.
    One of the challenges is you have to be able to distinguish between a large project and a small project. You also have to be able to distinguish complexity that is a consequence of size or organizational effects from inherent complexity. On a project that is simply large you can successfully substitute warm bodies for real experts because they can just treat the problem as "make 5000 CRUD screens and a few hundred reports." But if the complexity stems from the algorithms, performance requirements, or other aspects that require a deep understanding then a billion typewriters just won't cut it.
  28. I've been meaning to do a blog on The Tar Pit for a long time now. Thanks for the inspiration to actually sit down and do it. http://erikengbrecht.blogspot.com/2007/09/tar-pit.html
  29. The reason Brooks' book remains relevant because all of its core truths are about human beings and complexity, not about technology. Large organizations can accomplish bigger and more complex tasks than small ones but they are less efficient at a work unit level in doing so. That is because the number of pairs of end points (people) communicating is much larger and because no one person can fit the entire problem and its solution in their mind at one time. Politics, emotions, and personal agendas all add to the friction. Yet at some point it all needs to come together. That requires trust and cooperation. That is damn difficult whether you are in IT or conducting a war. In economics there is a theory of "xefficiency". That is, organizations are not inherently profit maximizers. The individuals who make up the organization may or may not be profit maximizers (some are risk adverse) and their notion of "profit" is personal and may not be monetary. The challenge is to insure that the incentive's people have lead them to behaviors that maximize the organization's profits; however that has been defined. So our biggest chanllenges in IT are not about technologies. They are about managing people and complexity. It is not in our power to change that. We can only hope to manage it.