Martin Fowler's Bliki

ObservedRequirement

2008-09-16T19:12:00-04:00

Here's one of my favorite software development quotes:

Requirements are the things that you should discover before starting to build your product. Discovering the requirements during construction, or worse, when you client starts using your product, is so expensive and so inefficient, that we will assume that no right-thinking person would do it, and will not mention it again.

--Suzanne and James Robertson

It's the opening paragraph of the first edition of their book "Mastering the Requirements Process". As regular readers might guess, my liking isn't connected to agreement. I like this quote because it sums up the waterfall value system to requirements (indeed the word 'requirement' is inherently waterfallish).

Agile methods violate this underlying assumption by intending to discover the 'requirements' during construction and after delivery. But even this cavalier disregard of the above sage advice is nothing compared to what many leading web sites do these days. These sites explore requirements by observing what the users do on their sites and using that information to generate ideas for new features along the following lines:

Look at what people are trying to do with the site and provide easier ways for them to do it.
Look at where people are abandoning doing something, and look for ways to fix whatever was frustrating them.
Build a new feature and see if people use it.
Build an experimental feature and make it available to a subset of the user base. Not just can you see if they like it, you can also assess how much load it puts on your servers.

To support this kind of analysis, you need to add user logging behavior into the application and build some tools to analyze these logs. Much logging appears for free in web applications, which I suspect was major impetus for people starting to do this. But the logging, and analysis, can go further as it's added to the application.

I haven't found much advice out there on the web on how to do this, and I don't hear much discussion about doing this in practice. Like many things it requires a concentrated effort to spend the time to build monitoring capability and then to use it to probe how to improve the software. Furthermore it's a mighty step away from how the traditional software process, even for agile projects.

But there is a huge potential here. Everyone knows how big the difference is between what people say they want and what people actually need and use. By watching what people actually do with your application, you can find out what actually happens with the software - which can give you much more direct information than other sources. As a result I think more teams should consider adding this approach to their toolkit.

EvolutionarySOA

2008-09-12T09:29:00-04:00

Can SOA be done with an agile approach?

I don't delve too much into the cluttered world of SOA (ServiceOrientedAmbiguity), but I get this question often enough (in some form or other) to be worth a pontification.

When I first came across agile software development (in the form of Extreme Programming) the most troubling aspect for me was its approach to software design. Like many I'd become used to the notion that software should be designed before it was programmed, while XP seemed to encourage an almost wilful embracing of design ignorance. In 2000 I was asked to give the closing keynote at the first Agile/XP conference, and in order to gather my thoughts I ended up writing Is Design Dead? - an essay that examined the fate of design in an agile world.

I'm still quite proud of that essay and I think it's well worth reading today - but for this bliki entry I'll summarize. I talked about two driving approaches to software design: planned and evolutionary. Planned design works out the design of software in one phase and builds (programs) it afterwards. In this case changing the design is hard once you've begun construction. Evolutionary design assumes regular change of the design even once you've done significant programming. I concluded that the practices of XP provided a disciplined approach to evolutionary design, thus making it much more practical than people had realized. This change did not remove software design (it is not dead) but did significantly change how we think about design.

The argument for planned design in an SOA context is that we are building webs of interconnected, loosely coupled systems. In this situation each system is making its services available as a PublishedInterface to the whole enterprise. Published interfaces are hard to change, therefore you need planned design to get them right so you don't have to change them. Planned design is also a reaction to the chaotic system interconnections that people see in most organizations. This chaos is the result of poor design, so the feeling is that better planned design will prevent this happening in future.

Evolutionary design is an essential aspect of agile methods. One of the key foundations of agile methods is the desire to handle, indeed welcome, change. Planned design assumes change is hard, and thus tries to predict where it occurs. If changes occur within the predicted boundaries then it's easy, but if it falls outside those boundaries you're out of luck. Agile thinking assumes unpredictable change is inevitable and tries to enable it in a controlled way.

So as I look at SOA, or any other design context, the fundamental question I ask is "is change predictable?" Only if change is predictable is a planned design approach valid. My sense is that if predictability is hard within the context of a single application, it's doubly hard across an enterprise. If we use planned design in a unpredictable context we find that however good the plans are, they will be undermined by the unpredictable changes, leading to the same mess we are currently in. Usually, however, the mess is worse because a there is significant sunk cost in a planned design, this can easily reduce the business value that an SOA effort is intended to produce, particularly if time-to-market is important.

As a result I think we have to bite the bullet and figure out how to do evolutionary design in this loosely connected context. After all the whole of point of loose coupling is to make change easier. At the center of this is thinking about contracts in terms of change, with such ideas as Consumer Driven Contracts.

This direction leads to things like Jim Webber's notion of Guerilla SOA. This builds up SOA using small steps directed by producing business value. Since producing business value is the whole point, this offers a path to producing a much better return on investment. It's certainly an approach our clients appreciate.

Can evolutionary design scale to SOA sizes? In my view we have an existence proof at a much larger scale than even a big SOA effort - the web itself. The web is built on very loose coupling and lots of unpredictable changes. It is, in many ways, a mess - but it's a mess that works, delivering real value to lots of people every day.

DslQandA

2008-09-09T18:33:00-04:00

I was asked to put together a discussion of DSLs for non-technical types. Maybe I've been reading too much Stephen O'Grady, but I felt an irresistible urge to do it in a Q and A manner. So here it comes.

What is a Domain Specific Language?

A Domain Specific Language (DSL) is a computer programming language of limited expressiveness focused on a particular domain. Most languages you hear of are General Purpose Languages, which can handle most things you run into during a software project. Each DSL can only handle one specific aspect of a system.

So you wouldn't write a whole project in a DSL?

No. Most projects will use one general purpose language and several DSLs

Are they a new idea?

Not at all. DSLs have been used extensively in Unix circles since the early days of that system. The lisp community often talks of creating DSLs in lisp and then using the DSLs to implement the logic. Most IT projects use several DSLs - you might of heard of things like CSS, SQL, regular expressions and the like.

So why are they getting a lot of noise now?

Probably because of Ruby and Rails. Ruby as a language has many features that make it easy to develop DSLs and the people who got involved in the Ruby community have been familiar with this approach from elsewhere, so they took advantage of these features. In particular Rails uses several DSLs which have played a big role in making it so easy to use. This in turn encouraged more people to take up these ideas.

Another reason is that many Java and C# systems need to have some of their behavior defined in a more dynamic way. This led to complex XML files that are difficult to comprehend, which in turn led to people exploring DSLs again.

So DSLs can be used with languages other than Ruby?

Yes, as I indicated DSLs have been around for much longer than Ruby has. Ruby has an unobtrusive syntax and meta-programming features that make it easier to create more elegant internal DSLs than languages like C# and Java. But there are useful internal DSLs in Java and C#.

What's the distinction between internal and external DSLs?

An internal DSL is just a particular idiom of writing code in the host language. So a Ruby internal DSL is Ruby code, just written in particular style which gives a more language-like feel. As such they are often called Fluent Interfaces or Embedded DSLs. An external DSL is a completely separate language that is parsed into data that the host language can understand.

Why are people interested in DSLs?

I see DSLs as having two main benefits. The most common benefit is that they make certain kinds of code easier to comprehend, which makes it much easier to modify, thus improving programmer productivity. This is worthwhile all on its own and is relatively easy to achieve.

The most interesting benefit, however, is that a well designed DSL can be understandable by business people, allowing them to directly comprehend the code that implements their business rules.

So is this the hook - business people write the rules themselves?

In general I don't think so. It's a lot of work to make an environment that allows business people to write their own rules. You have to make a comfortable editing tool, debugging tools, testing tools, and so on. You get most of the benefit of business facing DSLs by doing enough to allow business people to be able to read the rules. They can then review them for accuracy, talk about them with the developers and draft changes for developers to implement properly. Getting DSLs to be business readable is far less effort than business writable, but yields most of the benefits. There are times where it's worth making the effort to make the DSLs business-writable, but it's a more advanced goal.

Do you need special (ie expensive) tools?

In general, no. Internal DSLs just use the regular facilities of the programming language that you are using anyway. External DSLs do require you to use some special tools - but these are open source and are very mature. The biggest problem with these tools is that most developers aren't familiar with them and believe they are harder to use than they really are (a problem exacerbated by poor documentation).

There are exceptions on the horizon, however. These are a class of tools that I call a LanguageWorkbench. These tools allow you to define DSLs more easily, and also provide sophisticated editors for them. Tools like this make it more feasible to make business-writable DSLs.

So is this a repeat of the dream of developing software without programming (or programmers)?

That was the intent of COBOL, and I don't think there's any reason to think that DSLs will succeed where COBOL (and so many others failed). What I think is important is that DSLs allow business people and developers to collaborate more effectively because they can talk about a common set of precise rules that are the executable code.

When should I consider making a DSL?

When you are looking at an aspect of system with rich business rules or work-flow. A well-written DSL should allow customers to understand the rules by which the system works.

Isn't this going to lead to a cacophony of languages that people will find hard to learn?

We already have a cacophony of frameworks that programmers have to learn. That's the inevitable consequence of reusable software, which is the only way we can get a handle on all the things software has to do these days. In essence a DSL is nothing more than a fancy facade over a framework. As a result they contribute little complexity over what is already there. Indeed a good DSL should make things better by making these frameworks easier to use.

But won't people create lots of bad DSLs?

Of course, just like people create bad frameworks. But again I'd argue that bad DSLs don't do much additional harm compared to the cost of bad frameworks.

UpcomingTalks

2008-08-05T14:39:00-04:00

Neal Ford and Rebecca Parsons will be working with me again on a DSL tutorial at JAOO and QCon San Francisco.

The two biggest event I have coming up are two more conferences run by JAOO. The first is the mother-ship conference in Århus, which has become my favorite conference of the last few years. Later I'll be at QCon San Francisco. At both conferences I'll be giving a tutorial on Domain Specific Languages with Neal Ford and Rebecca. In addition I'm giving a shorter talk in the regular JAOO session on the patterns for internal DSLs, working directly on the topics that I've recently got into a coherent state for the book.

On a different note, I'll be talking at Steve McConnell's Construx Executive Summit. I find it rather odd to be invited to this as it's a conference aimed at executives and that's not my usual audience. I'll be talking about why software design is important and to get good design - which boils down to getting good designers and how to get and keep those.

DslBookRoadmap

2008-08-04T11:34:00-04:00

I've hit a significant, if purely internal, milestone in the DSL book recently. I also find that people regularly ask me what the status is of the book is. So it seems like a good moment to post a note about where I am with the book and where I see things going.

The most common question I get is "when will the book be out?" The way things look at the moment, I'd estimate it appearing some time in 2010. That's an estimate, with all the usual hedges that apply to such things.

To help see how the current state meshes in with that estimate, I'll describe the general progress of how books work, at least for me. The first phase is writing the First Review Draft. This means writing all the material in the book, to the level that I can submit it to peer review. For my larger books (eg Refactoring, P of EAA) this takes about 12-18 months. This book is taking longer, I've already been at it for nearly two years, and I suspect there's another year to go. Once I have the First Review Draft, it goes out to review. This takes time for people to read it, get their comments to me, and for me to modify the text based on the review. I usually do two rounds of review - and it takes around 6 months for that to happen. Once I'm done with formal review the book is a Final Draft. At this point it goes off for production - which includes copy-edit, indexing, layout, printing, etc. That takes another 6 months. So I estimate that the book will appear about a year after I get a First Review Draft.

Not all authors work this way, some write a few chapters and sent them for review while they write a few more chapters. I use the approach I do because it was how I did my first book and it seemed to work very well.

To describe the current state of the book, I need to talk a bit about its structure. The final structure of the book will be a DuplexBook, but while I'm working on it I tend to think of it in terms of a set of subject areas, where each subject area will end up as a set of narratives and topics within the duplex structure.

My recent milestone was completing my first coherent pass at the internal DSL subject area. By this I mean I now have what I think is a coherent draft of the that subject area. It will still need more work before I get to the First Review Draft, but reaching a coherent draft is a big point for me because it means I have reached a point where it's looking in good shape. Here's my current list of subject areas and their status:

Introduction: coherent (this is the first three narratives: An Introductory Example, Using Domain Specific Languages, and Implementing DSLs).
External DSLs: incoherent
Internal DSLs: coherent
Code Generation: coherent, but with some important bits missing
Alternative Computational Models: incoherent
Error Diagnostics: open.
Language Workbenches: open.

I've been working since March on getting the internal DSL section into a coherent state. I suspect it will take me a similar amount of time to get external DSLs into coherence, and a similar time again to get alternative computational models into coherence. This is why I think I'm at least a year off First Review Draft. These are also the next two subject areas that I'm going to work on, in that order.

As well as those two subject areas to make coherent, I also have some holes to fill and a couple of elephants to deal with. The holes are topics within subject areas that aren't really complete yet, although the overall structure is reasonable. Code generation is a good example of that. I'm pretty happy with what I have, and it is coherent. But I need to address at least one topic (the Generation Gap pattern) before I can really say that subject area is done. These holes don't take a huge amount of time to fill and I tend to work on them as the mood takes me - often I like to take a break in the middle of a major topic for something like this.

The two elephants are the open topics of error diagnostics and language workbenches. These are elephants because I don't know how I'm going to tackle them, or indeed if I'm going to put much effort into tackling them. It may be that I'll do little more than a skimpy overview chapter, or I may spend six months each on them. I'm deliberately putting off those decisions while I work on the next two subject areas.

Of course this road-map could all change. One of the reasons I don't like doing incremental review (eg sending off the internal DSL section for a formal review now) is that working on one section can cause me to completely rethink another. This has already happened. I did a substantial chunk of work on the internal DSL section late in 2007 and then worked for a while on what is now the Computational Network and Dependency Network patterns at the end of 2007. Working on those patterns made me realize my internal DSL thinking was all wrong and I thus needed to rewrite that whole section. This kind of thing is common with a topic area like this where there is no pre-existing structure. (This is different to a book on an existing technology, eg UML Distilled, where the thing I'm describing already has an understood structure before I start.)

Having said that, if you're interested in what I'm doing I think I'm at the point where you can take a good look at the introductory and internal DSL sections and it should make sense. I'd certainly be happy for feedback on those sections. Just beware that there are cross links between those and other sections that may still get worked on.

I'll update this page when I hit a major milestone or major re-plan. If you want more granular updates I have a more detailed feed. If you want to send me comments on work so far, please do so. I have a mailing list for the book, I'd prefer all comments to go to that mailing list so that there's an opportunity for discussion. So if you do send me comments, please let me know if you're happy with them going on the list or not and whether you wish to join the list.

MDSDandDSL

2008-07-14T17:10:00-04:00

What is the connection between ModelDrivenSoftwareDevelopment (MDSD) and DomainSpecificLanguages (DSLs)?

It's pretty common to see the term "DSL" crop up in the context of MDSD. Indeed some MDSD people seem to think that DSLs only exist within the MDSD world. I've been writing a lot on DSLs recently for my book, but so far I haven't really touched on the MDSD angle much Instead I've concentrated on DSLs role in more conventional programming. DSLs exist in both the textual language and MDSD worlds and play pretty much the same role for both.

In an MDSD context DSLs are again a language targeted at a specific kind of problem as opposed to general purpose languages such as the UML. As a result they can have the same kind of relationship: build a system in the general purpose modeling language and use DSLs for various specific aspects. Since MDSD hasn't caught on that much, however, you also see a different approach where modeling DSLs are used in the context of a traditional language environment. Here you might use several modeling DSLs that generate Java code to be combined in a Java project. In this case there's no general purpose MDSD model around - you use MDSD for each DSL relatively independently.

In order to use model-oriented DSLs you need a different, RepositoryBasedCode, approach to tooling. This introduces quite a few pragmatic issues as the general support environment for such tools is less established. In order to define your own DSLs you need more specialized tooling - something I call a Language Workbench.

DSLs seem to have a proportionately higher emphasis in the MDSD world than they do in the mainstream programming world. Cynics think this is a result of the MDSD community desperately searching for a way to remain relevant, fans of MDSD regard it as a sign of MDSD's superior sophistication. I think this is mainly due to the fact that the MDSD community is smaller and has far less in the form of established practice.

A particularly visible sub-community of MDSD is centered around ModelDrivenArchitecture (MDA). I'm not much of a fan of MDA in particular, but am particularly skeptical of MDA DSLs.

There is much that model-oriented DSLs share with textual DSLs. I put a lot of emphasis with textual DSLs in basing work around a Semantic Model. MDSD, as its name indicates, is very much about driving a system from that kind of a model. A difference is that most MDSD people assume that you'll want to generate code from that model rather than executing the model directly.

As I write this, I'm not sure how much I'm going to cover language workbenches in my book. Certainly I'll at least discuss the overall concept behind them, but the coverage may not be that deep. This will be partly due to the large amount of material I seem to be generating on textual DSLs and partly due to the fact that language workbenches are much newer and thus more volatile and less mature.

ModelDrivenSoftwareDevelopment

2008-07-14T17:10:00-04:00

Model Driven Software Development (MDSD) is a style of software development that considers itself as an alternative to the traditional style of programming. The approach centers itself on building models of a software system. These models are typically made manifest through diagrammatic design notations - the UML is one option. The idea is that you use these diagrams, to specify your system to a modeling tool and then you generate code in a conventional programming language.

The MDSD vision evolved from the development of graphical design notations and CASE tools. Proponents of these techniques saw graphical design notations as a way to raise the abstraction level above programming languages - thus improving development productivity. While these techniques and tools never caught on too far, the basic core ideas still live on and there is an ongoing community of people still developing them.

Although I've been involved, to some extent, in MDSD for most of my career, I'm rather skeptical of its future. Most fans of MDSD base their enthusiasm on the basis that models are ipso facto a higher level abstraction than programming languages. I don't agree with that argument - sometimes graphical notations can be a better abstraction, but not always - it depends on the specific cases. Furthermore To use MDSD you need tools that support RepositoryBasedCode, and these tools currently introduce a number of pragmatic issues in tooling - of which source control is the canonical example.

MDSD is surrounded by a terminological mess. One particular vision of MDSD is ModelDrivenArchitecture (MDA) which is an OMG initiative based on the UML. Many people in the MDSD community, however, don't think that MDA or UML is the right vision for MDSD. For a long time I would hear people talking about Model Driven Development (MDD) as the general concept and MDA as the OMG's specific vision. However the OMG has trademarks on several "Model Driven *" and "Model Based *" phrases - including MDD. As a consequence people have to be careful about what model driven phrase they use. I'm using MDSD as that is the title of a useful book on the topic.

IncrementalMigration

2008-07-07T17:12:00-04:00

Like any profession, software development has it's share of oft-forgotten activities that are usually ignored but have a habit of biting back at just the wrong moment. One of these is data migration.

Most new software projects involve data that's lived somewhere else and now needs to be moved into the new system once it's live. A system replacement might have to move all the old data, new functionality may lead to data being loaded from some other system.

It's common to not take this task very seriously. After all, it's just reading some data, munging it a bit, and loading into the new system. Furthermore the code only ever has to be run once, so there's no point making particularly fast or pretty. Once the migration is done the code can be safely chucked away.

And of course there's no need to worry about it till the end of the project since you only want to run the migration just before the new system goes live.

I have a high opinion of my readers, if only for their taste in software writing, so I'm sure I can see the wistful smiles. Data migration often looks easy from the safety of whiteboard abstractions, but is usually full of nasty details to trip you up.

You may suspect that the existing data is somewhat messy, but everyone is usually taken aback at how dirty the data really is. As a result the whole exercise is often far more complicated than it ought to be.
Because it's single use, throw-away code people don't tend to put much design effort into migration code since they assume it's below the DesignPayoffLine. That assumption is often wrong, especially with the previous bullet point.
Doing an activity that balloons into something harder than you think is never fun, but when you leave it till close to the ship date you're offering trouble a big signing bonus.

There's a soundbite I like to use in an agile context: if it hurts do it more often. Its surface illogicality makes it memorable, and there's a real truth in there. Many difficult activities can be made much more straightforward by doing them more frequently. XPers are particularly well known for applying this principle to testing, integration, design, and planning - so it shouldn't surprise anyone to see it applied to data migration.

I first saw this done by my colleague Josh Mackenzie on a moderately sized project (dozen developers for one year) with two failed attempts in its recent past. He decided he would migrate data with every two-week iteration. Each iteration the team figured out what data they needed to add to support the new functionality that was being built and updated the data migration system to migrate that extra data from the live system.

As is often the case with these things it ended up being much less impossible than people feared and the resulting reduction of risk and stress made it a worthwhile choice. They appreciated the obvious benefits, which boiled down to a distinct lack of hasty panic close to going live.

The most interesting benefit, however, was the one they didn't expect. Incremental migration made a significant improvement in communication with the domain experts. Usually when you want to talk about use cases with domain experts, you make up some pretend scenario. By using incremental data migration the team got into the habit of using real examples, which were much easier for the domain experts to relate to. Furthermore when the development made builds available for the domain experts to look at, it included a copy of the live data. As a result the domain experts could investigate how the new system worked with tricky cases they had run into recently. Particularly juicy predicaments could easily be copied over into the test environment.

Even without the improved communication it's worth the effort to do incremental migration. If you do, be prepared to take advantage of the opportunity to use real data to talk to domain experts.

AgileVersusLean

2008-06-26T09:21:00-04:00

I'm thinking of using agile software development - but should I use Lean software development instead?

This question is one I've run into a few times recently. It's not a question I can answer quickly as the question is based on a false premise about the relationship between lean and agile. So first I need to go into some history to help explain that relationship.

"Lean" fundamentally refers an approach in the manufacturing world that was originally developed by Toyota in the 1950's. At this time Japanese industry was recovering from the ravages of the second world war. This approach, often called the Toyota Production System is mostly credited to Taiichi Ohno, although he was influenced by various western thinkers - particularly Deming. The Toyota Production System became well known in the rest of the world in the 1990's when westerners started writing books to explain why the Japanese were beating the US at so many industries. The western writers called this approach Lean Manufacturing. Although Japanese industry in general has run into stickier times since then, Toyota continues to outperform most western auto companies.

Agile software development is an umbrella term for several software development methods (including Extreme Programming and Scrum) that were developed in the 1990s. These methods share a common philosophy which was described as values and principles in the Manifesto for Agile Software Development. (My essay "The New Methodology" goes into this in more depth.)

There was a connection between lean manufacturing and agile software from the beginning in that many of the developers of the various agile methods were influenced by the ideas of lean manufacturing. This connection was made more explicit by Mary and Tom Poppendieck. Mary had worked in a manufacturing plant that had adopted lean manufacturing and her husband Tom is an experienced software developer. They have written a couple of books on the application of Lean ideas in the software world. When people talk about Lean Software they are usually referring to the ideas in these books, although others have been making similar links.

Lean manufacturing and agile software methods have a very similar philosophy. Both place a lot of stress on adaptive planning and a people focused approach. As a result lean's ideas fit in very well with the agile software story. Mary and Tom have both been very active in the agile community - indeed I'd credit Mary with an important role in forming the Agile Alliance. (Like me, she was a founding board member of the Agile Alliance, but she was far more effective in it than I was.)

I've already mentioned that lean manufacturing was an influence on agilists from the beginning, and in the last few years more ideas have appeared in the agile world with a clear lean manufacturing heritage. These range from concrete techniques like Value Stream Maps, to articulations of existing agile concepts such as the Last Responsible Moment for making design decisions. The idea of thinking of analysis and design documentation as inventory came from the Poppendiecks. Several agilists I know emphasize the importance of reducing cycle time - another strongly lean-influenced idea. My colleague Richard Durnall has a nice summary of how lean knowledge can blend in with agile thinking.

A particularly strong appeal to many people about lean ideas is that they provide a way of explaining agile software development to people - particularly senior people both within IT and senior customers. This explanation ranges from a crude appeal to emulate Toyota to detailed discussions of how lean manufacturing's benefits translate to the ideas in agile software development.

So as you can see, lean and agile are deeply intertwined in the software world. You can't really talk about them being alternatives, if you are doing agile you are doing lean and vice-versa. Agile was always meant as a very broad concept, a core set of values and principles that was shared by processes that look superficially different. You don't do agile or lean you do agile and lean. The only question is how explicitly you use ideas that draw directly from lean manufacturing.

The Poppendiecks didn't introduce lean as a separate idea, nor did they introduce lean as a published process in the style of Scrum or XP. Rather they introduced lean as a set of thinking tools that could easily blend in with any agile approach. I think of lean as a strand of thinking within the agile community, like a pattern in a rich carpet.

There is a distinct lean software community, as in a mailing list calling itself lean and people who label themselves as lean thinkers. But this is no different to the fact that there are also strong XP, Scrum, and other communities. Most people in these communities consider themselves part of the broader agile movement and many people are active in more than one of these agile communities. The whole point of coining the word 'agile' comes from a recognition that we share a core set of values and principles and this common core means what we have in common is greater than our differences.

SegmentationByFreshness

2008-06-24T18:19:00-04:00

One of the biggest issues with media websites is dealing with high amounts of traffic. Media is all about getting eyeballs, but if you get too many hits at once, slow performance can cause problems and damage your reputation. This problem is exacerbated by the bursty nature of this web traffic. You can be cruising along at a manageable rate, then get hit with a big news story which causes a big spike. One of our clients have seen spikes of two orders of magnitude in a matter of a couple of minutes.

The general solution in computing to speed up access to the same information is to use caches. If you keep requesting my home page the web server will build up a cache in memory so repeated requests avoid touching the disk.

It's easy to keep a cache for my website, because this page, like my entire site, is entirely static. Most media sites, however, contain a lot of dynamic content. You might not think there's much business logic on your average newspaper website, but once you start looking at advertising links, related stories, special features and the like, things get a good bit more interesting. A travel story to France might link to articles on french food, and advertising that knows that a web browser in Canada is interested in a holiday in the Loire Valley. Personalization makes this even worse, my personalized preferences should generate a personalized feature list on heavy red wines. Such logic is complex in its own right, it makes for a lot of computation with each request, and crucially it ruins most caching strategies.

The way to deal with this is to divide a page up into segments where each segment has a similar determination of freshness. The article on Loire travel can be relatively static, changing only to correct errors. A related article list which feeds off tags for "France" and "Loire" will change more often, but maybe only every few days. If we arrange this properly a request for a page with these two items may be able to gather everything from caches.

The most common way of doing this that I've seen is to form caches on the web server and assemble the page segments when the page gets hit. Tools like Sitemesh are a good option for this approach. As you write the page for 18th century loire delights, you include call-outs for sections like related articles. When you get the actual web request the web server takes the page and assembles the page from the separate pieces. Much of this can be cached in the web server, which avoids hitting the back-end domain logic and database.

An interesting possibility is to go even further and use the many caches that exist in the web itself. Most calls for this web page don't even reach my web server since my page gets cached many times along the way. If you build a web page dynamically and assemble it on the server, you have to take the hit to deliver the page. An alternative is to assemble the page on the client and then draw each segment from its own URL. Each segment could be cached in different places with different caching policies.

How might this work? We might store the static article content as XHTML at an URL like http://gallifreyTimes/travel/18-century-loire-delights. Inside that file we want to insert some related articles by looking up articles tagged with "loire" and "france". In the static page we put in a simple "a" tag.

  <a class = "relatedLinks" href = "relatedLinks/france+loire">Related Links</a>

In the header for the static page we link it to some javascript in a separate library file. When we download the Loire article the javascript runs and scans the article for elements with the right class: in this case an "a" element with the "relatedLinks" class. (The behavior library is a good way to do this.) When it finds the element it uses the information in the element to synthesize an URL for that segment. In this case it would use what's in the element's href attribute to come up with an URL like http://gallifreyTimes/relatedArticles/france+loire. Once it's got that URL it then gets the content and uses it to replace the original "a" element. Since the related articles list is handled through an URL, other gets on that URL cause caches through the Internet to warm up, so there's a good chance that retrieving the page may never cause a hit on the original server.

This technique of using Javascript to replace a placeholder element with more content is a form of Progressive Enhancement. The descriptions I've found for Progressive Enhancement focus on adding features for accessibility with limited browsers. This example also has that benefit. If I browse the page with a browser that has no javascript, I'll get a useful link. The general idea behind Progressive Enhancement is that the basic page served is useful on basic browsers, then we use techniques such as javascript to add in more fancy features.

In the context of caching, the value is that each progressive enhancement weaves in a lump of HTML with different freshness rules. The original page is static, the related links change daily, but both can be cached independently and weaved together. I can do all sorts of additional elements, as long as I take care to keep segment the page by the freshness rules. So I could include a personalized weather forecast based on the user's profile to every page by having the javascript pick up the user id from the http session, using it to construct an URL like http://gallifreyTimes/personalWeather/martinfowler, retrieving the content (which would often be cached on my hard drive) and weaving it into the page.