ThoughtWorker Blogs

I'm thinking of using agile software development - but should I use Lean software development instead?

This question is one I've run into a few times recently. It's not a question I can answer quickly as the question is based on a false premise about the relationship between lean and agile. So first I need to go into some history to help explain that relationship.

"Lean" fundamentally refers an approach in the manufacturing world that was originally developed by Toyota in the 1950's. At this time Japanese industry was recovering from the ravages of the second world war. This approach, often called the Toyota Production System is mostly credited to Taiichi Ohno, although he was influenced by various western thinkers - particularly Deming. The Toyota Production System became well known in the rest of the world in the 1990's when westerners started writing books to explain why the Japanese were beating the US at so many industries. The western writers called this approach Lean Manufacturing. Although Japanese industry in general has run into stickier times since then, Toyota continues to outperform most western auto companies.

Agile software development is an umbrella term for several software development methods (including Extreme Programming and Scrum) that were developed in the 1990s. These methods share a common philosophy which was described as values and principles in the Manifesto for Agile Software Development. (My essay "The New Methodology" goes into this in more depth.)

There was a connection between lean manufacturing and agile software from the beginning in that many of the developers of the various agile methods were influenced by the ideas of lean manufacturing. This connection was made more explicit by Mary and Tom Poppendieck. Mary had worked in a manufacturing plant that had adopted lean manufacturing and her husband Tom is an experienced software developer. They have written a couple of books on the application of Lean ideas in the software world. When people talk about Lean Software they are usually referring to the ideas in these books, although others have been making similar links.

Lean manufacturing and agile software methods have a very similar philosophy. Both place a lot of stress on adaptive planning and a people focused approach. As a result lean's ideas fit in very well with the agile software story. Mary and Tom have both been very active in the agile community - indeed I'd credit Mary with an important role in forming the Agile Alliance. (Like me, she was a founding board member of the Agile Alliance, but she was far more effective in it than I was.)

I've already mentioned that lean manufacturing was an influence on agilists from the beginning, and in the last few years more ideas have appeared in the agile world with a clear lean manufacturing heritage. These range from concrete techniques like Value Stream Maps, to articulations of existing agile concepts such as the Last Responsible Moment for making design decisions. The idea of thinking of analysis and design documentation as inventory came from the Poppendiecks. Several agilists I know emphasize the importance of reducing cycle time - another strongly lean-influenced idea. My colleague Richard Durnall has a nice summary of how lean knowledge can blend in with agile thinking.

A particularly strong appeal to many people about lean ideas is that they provide a way of explaining agile software development to people - particularly senior people both within IT and senior customers. This explanation ranges from a crude appeal to emulate Toyota to detailed discussions of how lean manufacturing's benefits translate to the ideas in agile software development.

So as you can see, lean and agile are deeply intertwined in the software world. You can't really talk about them being alternatives, if you are doing agile you are doing lean and vice-versa. Agile was always meant as a very broad concept, a core set of values and principles that was shared by processes that look superficially different. You don't do agile or lean you do agile and lean. The only question is how explicitly you use ideas that draw directly from lean manufacturing.

The Poppendiecks didn't introduce lean as a separate idea, nor did they introduce lean as a published process in the style of Scrum or XP. Rather they introduced lean as a set of thinking tools that could easily blend in with any agile approach. I think of lean as a strand of thinking within the agile community, like a pattern in a rich carpet.

There is a distinct lean software community, as in a mailing list calling itself lean and people who label themselves as lean thinkers. But this is no different to the fact that there are also strong XP, Scrum, and other communities. Most people in these communities consider themselves part of the broader agile movement and many people are active in more than one of these agile communities. The whole point of coining the word 'agile' comes from a recognition that we share a core set of values and principles and this common core means what we have in common is greater than our differences.

One of the biggest issues with media websites is dealing with high amounts of traffic. Media is all about getting eyeballs, but if you get too many hits at once, slow performance can cause problems and damage your reputation. This problem is exacerbated by the bursty nature of this web traffic. You can be cruising along at a manageable rate, then get hit with a big news story which causes a big spike. One of our clients have seen spikes of two orders of magnitude in a matter of a couple of minutes.

The general solution in computing to speed up access to the same information is to use caches. If you keep requesting my home page the web server will build up a cache in memory so repeated requests avoid touching the disk.

It's easy to keep a cache for my website, because this page, like my entire site, is entirely static. Most media sites, however, contain a lot of dynamic content. You might not think there's much business logic on your average newspaper website, but once you start looking at advertising links, related stories, special features and the like, things get a good bit more interesting. A travel story to France might link to articles on french food, and advertising that knows that a web browser in Canada is interested in a holiday in the Loire Valley. Personalization makes this even worse, my personalized preferences should generate a personalized feature list on heavy red wines. Such logic is complex in its own right, it makes for a lot of computation with each request, and crucially it ruins most caching strategies.

The way to deal with this is to divide a page up into segments where each segment has a similar determination of freshness. The article on Loire travel can be relatively static, changing only to correct errors. A related article list which feeds off tags for "France" and "Loire" will change more often, but maybe only every few days. If we arrange this properly a request for a page with these two items may be able to gather everything from caches.

The most common way of doing this that I've seen is to form caches on the web server and assemble the page segments when the page gets hit. Tools like Sitemesh are a good option for this approach. As you write the page for 18th century loire delights, you include call-outs for sections like related articles. When you get the actual web request the web server takes the page and assembles the page from the separate pieces. Much of this can be cached in the web server, which avoids hitting the back-end domain logic and database.

An interesting possibility is to go even further and use the many caches that exist in the web itself. Most calls for this web page don't even reach my web server since my page gets cached many times along the way. If you build a web page dynamically and assemble it on the server, you have to take the hit to deliver the page. An alternative is to assemble the page on the client and then draw each segment from its own URL. Each segment could be cached in different places with different caching policies.

How might this work? We might store the static article content as XHTML at an URL like http://gallifreyTimes/travel/18-century-loire-delights. Inside that file we want to insert some related articles by looking up articles tagged with "loire" and "france". In the static page we put in a simple "a" tag.

 <a class = "relatedLinks" href = "relatedLinks/france+loire">Related Links</a> 

In the header for the static page we link it to some javascript in a separate library file. When we download the Loire article the javascript runs and scans the article for elements with the right class: in this case an "a" element with the "relatedLinks" class. (The behaviour library is a good way to do this.) When it finds the element it uses the information in the element to synthesize an URL for that segment. In this case it would use what's in the element's href attribute to come up with an URL like http://gallifreyTimes/relatedArticles/france+loire. Once it's got that URL it then gets the content and uses it to replace the original "a" element. Since the related articles list is handled through an URL, other gets on that URL cause caches through the Internet to warm up, so there's a good chance that retrieving the page may never cause a hit on the original server.

This technique of using Javascript to replace a placeholder element with more content is a form of Progressive Enhancement. The descriptions I've found for Progressive Enhancement focus on adding features for accessibility with limited browsers. This example also has that benefit. If I browse the page with a browser that has no javascript, I'll get a useful link. The general idea behind Progressive Enhancement is that the basic page served is useful on basic browsers, then we use techniques such as javascript to add in more fancy features.

In the context of caching, the value is that each progressive enhancement weaves in a lump of HTML with different freshness rules. The original page is static, the related links change daily, but both can be cached independently and weaved together. I can do all sorts of additional elements, as long as I take care to keep segment the page by the freshness rules. So I could include a personalized weather forecast based on the user's profile to every page by having the javascript pick up the user id from the http session, using it to construct an URL like http://gallifreyTimes/personalWeather/martinfowler, retrieving the content (which would often be cached on my hard drive) and weaving it into the page.