Interview with Ken Arnold on Designing Distributed Systems


News: Interview with Ken Arnold on Designing Distributed Systems

  1. has published an interview with Ken Arnold on designing distributed systems. In this interview,
    Arnold claims that failure should be your number one concern in distributed systems design, that transactions as a way to deal with failure have overhead (though containers like J2EE can make them simpler to use), that idempotency can be simple with less overhead, and that state is hell and should be avoided.

    Read Interview with Ken Arnold on Designing Distributed Systems.

    Here's an excerpt:

    Failure is the defining difference between distributed and local programming, so you have to design distributed systems with the expectation of failure. Imagine asking people, "If the probability of something happening is one in ten to the thirteenth, how often would it happen?" Your natural human sense would be to answer, "Never." That is an infinitely large number in human terms. But if you ask a physicist, she would say, "All the time. In a cubic foot of air, those things happen all the time." When you design distributed systems, you have to say, "Failure happens all the time." So when you design, you design for failure. It is your number one concern.

    Threaded Messages (10)

  2. State is Hell. So what?[ Go to top ]

    I find his argument on state rather fallacious:

    " Anyone who has dealt with any large scale, high availability, or high performance piece of the problem knows that state is hell because they've lived with that hell. So the question is, "Why have more hell than you need to have?" You have to try and avoid it. Avoid. Avoid. Avoid."

    Just because something is difficult to deal with does not mean to avoid it at all costs as he suggests. If saving state is the appropriate solution to your problem (as it would be for building any reasonable website), then you must save state.

    I mean, transaction management is very difficult. Object to relational mapping can be very tricky. Should we not use these solutions?

    For that matter, complicated business processes can be very, very difficult to implement properly. Should we avoid projects with hard requirements, because they are "hell"?

  3. State is Hell. So what?[ Go to top ]

    His point is not to treat state lightly and if all possible centralize the management of state in a subsystem. I believe it serves as a good design guideline. The other side of the story is that since there are many technology companies who have products that support state management (for relational and file-based data as least), it may not be always totally necessary to avoid state management, when a proven solution works for the problem at hand.

  4. State is Hell. So what?[ Go to top ]

    I think by "avoid state" Ken doesn't mean "never have state" or "don't even build the system if it is going to have state." He just means avoid state to the extent possible, because the existence of state in a component of a distributed system means I have to worry about all kinds of failure issues. So if you can avoid state completely, do so. If you can't, try to minimize the number of components that do maintain state so that you minimize the failure worries.

  5. State is Hell. So what?[ Go to top ]

    Dave, read a bit further. He goes on to say:

    "So I follow the rule: make everything you can stateless. If there is one piece of the system you can't make stateless—it has to have state—to the extent possible make it hold all the state. Have as few stateful components as you can. "

    So he isn't saying "avoid state at all costs".
  6. I think his intent is to communicate that functionality should not be compromised, but that up-front efforts to simplify the problem area (i.e. removing unecessary complications of maintaining multiple areas of state usually by determining stateless methods for accomplishing the same result) and working to contain that which is necessary in the smallest possible footprint will greatly improve the end result. I would hazzard to guess that most web apps with significant state capabilities are also subject to failures, inconsistent behavior and have lowered efficiencies even when very bright and capable people have designed them.
  7. Sometimes state can't be avoided. Let's say you are doing an on-line shipment and need to fill out a customs document with 50 fields. If your connection drops after you've typed in 40 of them, wouldn't you want to have them reapppear when you log back into the website? The only way for this to happen is for the webserver, ejb server, or something to maintain the state so you can resume. Routing the user back to the same physical box is not very elegant or robust. I've done this with EJB servers implementing EntityBeans. The trick is to get a caching EJB server that does on-the-wire cache synchronization across the cluster. At the time (the late 90s) Persistence, Luna, Gemstone, BEA and maybe a few others did this. Not an easy task, but that's how J2EE vendors earn their money. (Or in the case of JBoss....get the bragging rights?)

    Also an interesting discussion on idempotentcy. I'm not sure I would go that route when doing distributed transactions, though. For a pub/sub system I did once, it was a great design though.

    I think this paper is right on target, except for the blatant "with Jini you can do whatever...." comments.

  8. So what can't you can't do with Jini???
  9. See, failure happens... What I meant was:

    What *can't* you do with Jini?
  10. I'm not trying to bash jini. It just brings the level of the article down a bit when there's obvious salesmanship going on. Maybe it's my academic background creeping in. No big deal though. It's to be expected. :)

  11. There is ALWAYS state.....[ Go to top ]

    >>Sometimes state can't be avoided

    State can't be avoided.

    There is always state in a system.
    Its a question of where you store it.

    You can store it in the database, you can keep it on the client or you can kep it in the middle tier (or a mix of all 3). Where you choose to keep it depends on how much you have and how much you have to move it around.

    There are no one-line answers to these sorts of questions like "avoid state".