Martin Fowler

Texas Speaking Events Rescheduled

2010-02-06T12:34:00-05:00

The family medical issue has been resolved happily, so I’m free to go back on the road. We’ve thus rescheduled the events I was supposed to do last month in Texas.

On February 23rd I’ll be speaking at DFW Scrum in Dallas.
On February 25th ThoughtWorks is organizing a technology forum in Austin.

As is usual for me, I haven’t planned exactly what I’ll talk about yet, but it’ll revolve around my usual topics of software design and agile methods.

ConversationalStories

2010-02-04T15:42:00-05:00

Here's a common misconception about agile methods. It centers on the way user stories are created and flow through the development activity. The misconception is that the product owner (or business analysts) creates user stories and then put them in front of developers to implement. The notion is that this is a flow from product owner to development, with the product owner responsible for determining what needs to be done and the developers how to do it.

A justification for this approach is that this separates the responsibilities along the lines of competence. The product owner knows the business, what the software is for, and thus what needs to be done. The developers know technology and know how to do things, so they can figure out how to realize the demands of the product owner.

This notion of product owners coming up with DecreedStories is a profound misunderstanding of the way agile development should work. When we were brainstorming names at Snowbird, I remember Kent suggesting "conversational". This emphasized the fact that the heart of our thinking was of an on-going conversation between customers and developers about how a development project should proceed.

In terms of coming up with stories, what this means is that they are always something to be refined through conversation - and that developers should play an active role in helping that definition.

spotting inconsistencies and gaps between the stories
using technical knowledge to come up with new stories that seem to fit the product owner's vision
seeing alternative stories that would be cheaper to build given the technological landscape
split stories to make them easier to plan or implement

This is the Negotiable principle in Bill Wake's INVEST test for stories. Any member of an agile team can create stories and suggest modifications. It may be that just a few members of a team gravitate to writing most of the stories. That's up to the team's self-organization as to how they want that to happen. But everyone should be engaged in coming up and refining stories. (This involvement is in addition to the develpers' responsibility to estimate stories.)

The product owner does have a special responsibility. In the end the product owner is the final decider on stories, particularly their prioritization. This reflects the fact that the product owner should be the best person to judge that slippery attribute of business value. But having a final decision maker should never stop others from participating, and should not lead people astray into a decreed model of stories.

DslBookRoadmap

2010-01-18T09:50:00-05:00

Time for another update on my DSL book's progress, since I've not been writing anything else recently.

I had my first round of technical review late in 2009 and have been incorporating comments into the current drafts. Progress on this has gone well, in large part because travel is light this time of the year. I'm also integrating my book production process into that of Pearson's.

The next visible targets are a second round of technical review and the launching of a roughcut. We're hoping to get these going in the next couple of months. The roughcut will also allow people other than official reviewers the chance to throw rocks at the text.

After that the material will be gradually readied for production. We're going to use a much more incremental process than I've used before, which will be both good and interesting. My sense at the moment that we'll see a physical book on bookshelves by the final quarter of 2010. It's currently looking at around 500 pages total in a DuplexBook split 150/350

The material currently on my web-site was last updated in June. While I've done quite a lot of detailed work on the book since, the broad structure is pretty similar, so the website gives a reasonably good picture of the scope of content.

Apologies for Canceling Texas Speaking Events

2010-01-15T20:00:00-05:00

I’m afraid I’ve had to cancel my speaking events in Dallas and Austin next week due to a family medical problem. As I write this, it’s not clear how serious the problem is going to be, but there is a good chance that I won’t be able to travel to Texas next week. As a result we felt it was best to cancel the events, while we still have a few days notice. We do intent to reschedule as soon the as dust settles. My Texas ThoughtWorkers are very keen to have me come out and do these talks, so we want to do them as soon as we reasonably can.

My apologies for this, and I hope you understand. In particular I want to thank the various collaborators in organizing these events for being very understanding under the awkward circumstances.

TechnicalDebtQuadrant

2009-10-14T10:03:00-04:00

There's been a few posts over the last couple of months about TechnicalDebt that's raised the question of what kinds of design flaws should or shouldn't be classified as Technical Debt.

A good example of this is Uncle Bob's post saying a mess is not a debt. His argument is that messy code, produced by people who are ignorant of good design practices, shouldn't be a debt. Technical Debt should be reserved for cases when people have made a considered decision to adopt a design strategy that isn't sustainable in the longer term, but yields a short term benefit, such as making a release. The point is that the debt yields value sooner, but needs to be paid off as soon as possible.

To my mind, the question of whether a design flaw is or isn't debt is the wrong question. Technical Debt is a metaphor, so the real question is whether or not the debt metaphor is helpful about thinking about how to deal with design problems, and how to communicate that thinking. A particular benefit of the debt metaphor is that it's very handy for communicating to non-technical people.

I think that the debt metaphor works well in both cases - the difference is in nature of the debt. A mess is a reckless debt which results in crippling interest payments or a long period of paying down the principal. We have a few projects where we've taken over a code base with a high debt and found the metaphor very useful in discussing with client management how to deal with it.

The debt metaphor reminds us about the choices we can make with design flaws. The prudent debt to reach a release may not be worth paying down if the interest payments are sufficiently small - such as if it were in a rarely touched part of the code-base.

So the useful distinction isn't between debt or non-debt, but between prudent and reckless debt.

There's another interesting distinction in the example I just outlined. Not just is there a difference between prudent and reckless debt, there's also a difference between deliberate and inadvertent debt. The prudent debt example is deliberate because the team knows they are taking on a debt, and thus puts some thought as to whether the payoff for an earlier release is greater than the costs of paying it off. A team ignorant of design practices is taking on its reckless debt without even realizing how much hock it's getting into.

Reckless debt may not be inadvertent. A team may know about good design practices, even be capable of practicing them, but decide to go "quick and dirty" because they think they can't afford the time required to write clean code. I agree with Uncle Bob that this is usually a reckless debt, because people underestimate where the DesignPayoffLine is. The whole point of good design and clean code is to make you go faster - if it didn't people like Uncle Bob, Kent Beck, and Ward Cunningham wouldn't be spending time talking about it.

Diving debt into reckless/prudent and deliberate/inadvertent implies a quadrant, and I've only discussed three cells. So is there such a thing as prudent-inadvertent debt? Although such a thing sounds odd, I believe that it is - and it's not just common but inevitable for teams that are excellent designers.

I was chatting with a colleague recently about a project he'd just rolled off from. The project that delivered valuable software, the client was happy, and the code was clean. But he wasn't happy with the code. He felt the team had done a good job, but now they realize what the design ought to have been.

I hear this all the time from the best developers. The point is that while you're programming, you are learning. It's often the case that it can take a year of programming on a project before you understand what the best design approach should have been. Perhaps one should plan projects to spend a year building a system that you throw away and rebuild, as Fred Brooks suggested, but that's a tricky plan to sell. Instead what you find is that the moment you realize what the design should have been, you also realize that you have an inadvertent debt. This is the kind of debt that Ward talked about in his video.

The decision of paying the interest versus paying down the principal still applies, so the metaphor is still helpful for this case. However a problem with using the debt metaphor for this is that I can't conceive of a parallel with taking on a prudent-inadvertent financial debt. As a result I would think it would be difficult to explain to managers why this debt appeared. My view is this kind of debt is inevitable and thus should be expected. Even the best teams will have debt to deal with as a project goes on - even more reason not to recklessly overload it with crummy code.

FeatureBranch

2009-09-03T09:45:00-04:00

With the rise of Distributed Version Control Systems (DVCS) such as git and Mercurial, I've seen more conversations about strategies for branching and merging and how they fit in with Continuous Integration (CI). There's a bit of confusion here, particularly on the practice of feature branching and how it fits in with CI.

Simple (isolated) Feature Branch

The basic idea of a feature branch is that when you start work on a feature (or story if you prefer that term) you take a branch of the repository to work on that feature. In a DVCS, you'll do this in your personal repository, but the same kind of thing works in a centralized VCS too.

I'm going to illustrate this with a series of diagrams. I have a shared project mainline, colored blue, and two developers, colored purple and green (since the developers names are Reverend Green and Professor Plum).

I'm using labeled colored boxes (eg P1 and P2) to represent local commits on the branch. Arrows between branches represent merges between branches, the boxes are colored orange to make them stand out. In this case there are updates, say a couple of bug-fixes, applied to the mainline (presumably by Mrs Peacock). When these happen our developers merge them into their work. To give this a sense of time, I'll assume we're looking at a few days work here, with each developer committing to their local branch roughly once a day.

In order to ensure things are working properly, they can run builds and tests on their branch. Indeed for this article I'll assume that each commit and merge comes with an automated build and test on the branch it's on.

The advantage of feature branching is that each developer can work on their own feature and be isolated from changes going on elsewhere. They can pull in changes from the mainline at their own pace, ensuring they don't break the flow of their feature. Furthermore it allows the team to choose its features for release. If Reverend Green takes too long, we can release with just Professor Plum's changes. Or we may want to delay Professor Plum's feature, perhaps because we are uncertain that the feature works the way we want to release it. In this case we just tell the professor to not merge his changes into mainline until we are ready for the feature. This is called cherry-picking, the team decides which features to merge in before release.

Attractive though that picture looks, there can be trouble ahead.

Although our developers can develop their features in isolation, at some point their work does have to be integrated. In this case Professor Plum easily updates the mainline with his own changes. There's no merge here because he's already incorporated the mainline changes into his own branch (there will be a build). Things are however not so simple for Reverend Green, he needs to merge all of his changes (G1-6) with all of Professor Plum's (P1-5).

(At this point many users of DVCSs may feel I'm missing something as this is a simple, perhaps simplistic view of feature branching. I'll get to a more involved scheme later.)

I've made this a big merge box as it's a scary merge. It may be just fine, the developers may have been working on completely separate parts of the code base with no interaction, in which case the merge will go smoothly. But they may be working on bits that do interact, in which case here lye dragons.

The dragons can come in many forms, and tooling can help slay some of them. The most of obvious dragon is the complexity of merging the source code and dealing with conflicts as developers edit the same files. Modern DVCSs actually handle this rather well, indeed somewhat magically. Git has quite the reputation for dealing with complicated merges. So much so that the textual issues of merging are much better than they used to be - indeed I'll go so far as to discount textual conflicts for the purposes of this article.

The problem I worry more about is a semantic conflict. A simple example of this is that if Professor Plum changes the name of a method that Reverend Green's code calls. Refactoring tools allow you to rename a method safely, but only on your code base. So if G1-6 contain new code that calls foo, Professor Plum can't tell in his code base as he doesn't have it. You only find out on the big merge.

A function rename is a relatively obvious case of a semantic conflict. In practice they can be much more subtle. Tests are the key to discovering them, but the more code there is to merge the more likely you'll have conflicts and the harder it is to fix them. It's the risk of conflicts, particularly semantic conflicts, that make big merges scary.

This fear of big merges also acts as a deterrent to refactoring. Keeping code clean is constant effort, to do it well it requires everyone to keep an eye out for cruft and fix it wherever they see it. However this kind of refactoring on a feature branch is awkward because it makes the Big Scary Merge worse. The result we see is that teams using feature branches shy away from refactoring which leads to uglier code bases.

Continuous Integration

It's these problems that Continuous Integration was designed to solve. With Continuous Integration my diagram looks like this.

There's a lot more merging going on here, but merging is one of those things that's much easier to do frequently and small rather than rarely and large. As a result if Professor Plum is changing some code that Reverend Green relies on, the Reverend will find it early, such as when he merges in P1-2. At that point he's only got to modify G1-2 to work with the changes, rather than G1-6.

CI is effective at removing the problem of big merges, but it's also a vital communication mechanism. In this scenario the potential conflict will actually appear when Professor Plum merges G1 and realizes that Reverend Green is actively building on Plum's libraries. At this point Professor Plum can go and find Reverend Green and they can discuss how their two features interact. It may be that Professor Plum's feature requires some changes that don't mesh well with Reverend Green's changes. By looking at both their features they can come up with a better design that affects both their work-streams. With the isolated feature branches our developers don't discover this till late, probably too late to do much about it. Communication is one of the key factors in software development and one of CI's most important features is that it facilitates human communication.

It's important to note that, most of the time, feature branching like this is a different approach to CI. One of the principles of CI is that everyone commits to the mainline every day. So unless feature branches only last less than a day, running a feature branch is a different animal to CI. I've heard people say they are doing CI because they are running builds, perhaps using a CI server, on every branch with every commit. That's continuous building, and a Good Thing, but there's no integration, so it's not CI.

Promiscuous Integration

Earlier I said parenthetically that there are other ways of doing feature branching. Say Professor Plum and Reverend Green take tea together early in the cycle. While chatting they discover they are working on features that interact. At this point they may choose to integrate with each other directly, like this.

With this approach they only push to the mainline at the end, as before. But they merge frequently with each other, so this avoids the Big Scary Merge. The point here is that the primary issue with the isolated feature branching scheme is its isolation. When you isolate the feature branches, there is a risk of a nasty conflict growing without you realizing it. Then the isolation is an illusion, and will be shattered painfully sooner or later.

So is this more ad-hoc integration a form of CI or a different animal entirely? I think it is a different animal, again a key point of CI is everyone integrates to the mainline every day. Integrating across feature branches, which I shall call promiscuous integration (PI), doesn't involve or even need a mainline. I think this difference is important.

I see CI as primarily giving birth to a release candidate at each commit. The job of the CI system and deployment process is to disprove the production-readiness of a release candidate. This model relies on the need to have some mainline that represents the current shared, most up to date picture of complete.

--Dave Farley

Promiscuous Integration vs Continuous Integration

So if it's different is PI better than CI, or more realistically under what circumstances is PI better than CI?

With CI, you lose the ability to use the VCS to do cherry picking. Every developer is touching mainline, so all features grow in the mainline. With CI, the mainline must always be healthy, so in theory (and often in practice) you can safely release after any commit. Having a half built feature or a feature you'd rather not release yet won't damage the other functionality of the software, but may require some masking if you don't want it to be visible in the user-interface. This can be as simple as not including a menu item in the UI to trigger the feature.

PI can provide some middle ground here. It allows Reverend Green the choice of when to incorporate Professor Plum's changes. If Professor Plum makes some core API changes in P2, then Reverend Green can import P1-2 but leave the others until Professor Plum's feature is put onto the release.

One worry with all this picking and choosing is that PI makes it really hard to keep track of who has what in their branch. In practice, it seems tooling pretty much solves this problem. DVCSs keep a clear track of changes and their origins and can figure out that when Professor Plum pulls G3 he already has G2 but doesn't have B2. I may have made mistakes drawing the diagram by hand, but tools do keep track of these things well.

On the whole, however, I don't think cherry-picking with the VCS is a good idea.

Feature Branching is a poor man's modular architecture, instead of building systems with the ability to easy swap in and out features at runtime/deploytime they couple themselves to the source control providing this mechanism through manual merging.

--Dan Bodart

I much prefer designing the software in such a way that makes it easy to enable or disable features through configuration changes. My colleague Paul Hammant calls this Branch by Abstraction. This requires you to put some thought into what needs to be modularized and how to control that variation, but we've found the result to be far less messy that relying on the VCS.

The main thing that makes me nervous about PI is the influence on human communication. With CI the mainline acts as a communication point. Even if Professor Plum and Reverend Green never talk, they will discover the nascent conflict - within a day of it forming. With PI they have to notice they are working on interacting code. An up-to-date mainline also makes it easy for someone to be sure they are integrating with everyone, they don't have to poke around to find out who is doing what - so less chance of some changes being hidden until a late integration.

PI arose out of open-source work, and it could be that the less intensive tempo of open-source could be a factor here. In a full time job, you work several hours a day on a project. This makes it easier for features to be worked in priority. With an open source project people often put in a hour here, and the next hour a few days later. A feature may take one developer quite a while to complete while other developers with more time are able to get features into a releasable state earlier. In this situation cherry picking can be more important.

It's important to realize that the tools you use are largely independent of the integration strategy you use. Although many people associate DVCSs with feature branching, they can be used with CI. All you need to do is mark one branch on one repository as the mainline. If everyone pulls and pushes to that every day, then you have a CI mainline. Indeed with a disciplined team, I would usually prefer to use a DVCS on a CI project than a centralized one. With a less disciplined team I would worry that a DVCS would nudge people towards long lived branches, while a centralized VCS and a reluctance to branch nudges them towards frequent mainline commits. Paul Hammant may be right: "I wonder though, if a team should not be adept with trunk-based development before they move to distributed."

UpcomingTalks

2009-08-13T10:30:00-04:00

Summer is coming to a close, and there's a bunch of conferences coming up in the next few months..

Before the summer does end, I'll be going to Agile 2009 in Chicago. I'm not speaking at the conference, but I will be hanging around the corridors and helping host some ThoughtWorks activities. We've long had an office in Chicago and we have a lot of speakers at the conference.

In October I'll be in Europe and as usual at this time of year I'll be at JAOO. Again Rebecca Parsons and Neal Ford will be joining me for our all-day DSL tutorial. We've worked a bit on rejigging it as my book gets more stable and Rebecca and I learn from Neal's presentation skills. I'll also be giving a talk on the role of patterns in software development skills.

Rebecca and Neal will be joining me for our DSL tutorial at JAOO

I have to fly out of Aarhus pretty quickly because the next week I'll be doing a keynote for Microsoft's Patterns and Practices Summit. I haven't exactly decided what my talk will cover yet, and indeed it may be ExtemporarySpeaking.

In November I return to the Left Coast to visit Vancouver, one of my favorite cities, to speak at Agile Vancouver.

My final booking so far is to return to QCon San Francisco, where I'll be giving my talk on Three Years of Ruby, that I gave at QCons in London, Beijing, and Tokyo earlier this year.

DigitalSLR

2009-08-07T15:30:00-04:00

Like many geeks I'm into photography. We geeks like photography because it provides the veneer of an artistic endeavor while allowing us to indulge in lots of technical details and spend money on expensive toys. A friend recently asked about my camera buying decisions, a question that prompted me to write them down. I got my first digital SLR a year ago. Before that I had owned a film SLR for many years, but started using digital cameras around 2000. I found the convenience of digital to be compelling and stopped using the film camera. I toyed with getting a digital SLR in 2004, but instead decided on a high end fixed lens camera - the Minolta A1. I enjoyed using it, but it conked out late in 2007. I considered a similar kind of camera, something like a Canon S5, but decided to bite the SLR bullet.

My first decision, and a critical one, was which system to buy. This is the critical decision as it's difficult (ie expensive) to reverse. Once you pick your system, you'll then commit money to it by buying lenses and the cost of switching is more than a dabbler like me can go with. I felt that the best choice was to go with the big two - Canon or Nikon. The choice between them was pretty much arbitrary, I ended up choosing Canon because a friend we occasionally vacation with has a Canon. A trifling distinction, but really the choice between the two wasn't a big one.

I'm still reasonably happy with it. One misgiving is that the technological advantage seems to have tipped in Nikon's favor over the last year, at least according to the blogs I read, but it's a tight race and Canon could well come back. I've also been recently intrigued by the new Micro Four Thirds format. Early days (and not around last year) but the small size and weight are very important to me.

With Canon as the choice, the next step was the initial choice of body and lenses. My approach was to get pretty much the cheapest body I could (the Digital Rebel XTI) because I'd rather spend more money on lenses than on the body. The whole point of SLRs is to have good lenses, so I'd rather concentrate my limited dollars there. Cameras also get upgraded much more frequently, so I'm likely to upgrade the camera in a few years while lenses stay current for much longer.

So which lenses? I forgoed the kit lens and got the camera body-only. As my main lens I went with a mega-zoom, the Sigma 18-200. Serious photographers will, probably rightly, turn their noses up at this lens. But I'm a dabbler. Most of my photos will only be seen on my screensaver or on a web page. A few get printed for a wall of our house, but only on a regular letter size printer. So I doubt that I'd appreciate the difference of a higher quality lens. Furthermore I can shoot within its limits. Reviews suggest that if you stick to f9, the quality stays pretty good. Since I'm mostly using it outside during the day, that limitation is easy to live with. As a result I tend to set my camera to aperture priority with f9, and that covers most of my shots.

The advantages of a single mega-zoom are considerable to me. Most of my photographs are taken while I'm doing something else, often with others around. I don't go out much to just shoot. In that situation even changing lenses can be a significant deterrent to getting a shot. Furthermore size and weight are a big deal when I'm travelling. While the lens isn't exactly svelte, it's much more compact that the alternative ways of getting that kind of zoom range. A final bonus is that it's image stabilized, which allows me to use it for static interiors.

The mega-zoom stays on my camera most of the time, but it wasn't the only lens I got with my camera. I also picked up the f1.8 50mm. This is an easy lens to get, very cheap, very light, very small but produces great quality. Since it's the equivalent of a 80mm on 35mm film, it's ideal for portrait photos - particularly with the f1.8 aperture. I use it a lot for shooting people in low light conditions.

I toyed with other lenses, but I wanted to get used to those two before I plonked money on any more.

After a few months with the camera I turned my eyes to a tripod. There are varying views on the net about tripods, some feel you should only use them if you really have to, some that you should use them whenever you can. I do like having one around, particularly for crepuscular shooting. I had a cheap and crummy silk tripod, but Duncan's blog persuaded me that I should get something better. I didn't go for his preferred Gitzos (beyond my budget) but I did get a light Induro tripod, together with a Really Right Stuff head and fast release clamp.

I went for the lightest setup I could get, as I wanted something that I'd actually be prepared to carry around and my camera/lens combos aren't particularly heavy. The fast release clamp was important as I'm someone who like to move around when shooting and such a clamp makes a big difference. In hindsight I wish I'd paid the extra for an L clamp, as I do find it frustrating to futz with the head when switching orientations.

It was only a month or so more before I went for another lens. A trip out to Colorado and Utah was the trigger to think about something wider than the 18-200 would go. I considered the Tokina 11-16 and the Canon 10-22, going for the latter due to it being lighter. It's a fun lens to use, allowing a few different things than what my regular lens provides. In particular what's interesting to work with is the huge depth of field you can get with an ultra wide: at 10mm you can easily get everything from a foot to infinity.

This is probably a reasonable moment to talk about filters. There's a good bit of discussion on the net about whether putting on a UV filter is worthwhile. I decided to get one for the 18-200 as it's on my camera so much, but not to get ones for my other lenses as I use them much less and am prepared to be more careful when those lenses are on the camera. For the mega-zoom I also picked up a polarizing filter, which I carry around with me all the time, but frequently forget to use.

The other issue that obsesses camera people is how to carry all this stuff. All things being equal, I like weight on my waist. So I went for a waist belt (from Tamrac, due to the double belt layout) and a Think Tank holster. I like the Think Tank's ability to extend when I have the hood on my lens. The only problem is that there are plenty of occasions when a waist belt isn't an option. The holster comes with a shoulder strap, which is fine, but I usually want the 10-22 as well. Cindy came to the rescue, sewing some straps onto the side of the holster so I can attach a lens pouch.

To keep track of my photos, and to do some post-processing, I got a copy of Apple's Aperture. (It seemed a toss up between Aperture and Lightroom.) I find it works well, better than sticking with iPhoto.

The latest lens I added to my collection is the Canon f2 100mm. I got this for shooting indoors, particularly at conferences for shooting someone on stage. In those situations I need more reach than the 50mm, but I still want a really fast aperture at a price and weight that's rather less than the serious zooms. So far I've only used the 100mm a couple of times, but have been very happy with it.

That burst of buying isn't something I expect to maintain. The quartet of lenses I have is pretty suited to my needs. There are some more I'm eyeing. The Canon 100-400mm zoom would be great for wildlife shots, but frankly we're rarely in the situation where I'd use it, so it's hard to justify its high cost. A different situation that regularly tickles my mind is cases where I'm primarily at a conference (so have the 50 and 100mm) but don't want to lug the 18-200 and want to have something wider. I could take the 10-22, but that leaves a gap and is less light than I'd like. Ironically this suggests the (now updated) kit zoom which is cheap and light. The primes less than 50mm are either too heavy, too expensive, or seem to have less quality than the kit zoom.

If you're curious, here are the results, (because there just aren't enough holiday snaps on the web.)

SelfInitializingFake

2009-08-04T14:47:00-04:00

One of the classic cases for using a TestDouble is when you call a remote service. Remote services are usually slow and often unreliable, so using a double is a good way to make your tests faster and more stable.

When you're querying a remote service, you need to find a way to load the expected data into your double. One way to do this is to use what I'm dubbing a self-initializing fake. The basic plan is simple. The first time you call the fake it passes the call onto the actual remote service, and as it returns the data it takes and saves a copy. Further calls just return the copy. In a sense this is like a cache, but with the important difference that there is no attempt to handle cache invalidation, which is handy as that's one of the TwoHardThings.

I've called this a fake, as that seems the closest fit from the various varieties of test doubles. The other reasonable alternative is a stub, but the distinction here is that a stub needs setting up when you build the fixture, while fakes are autonomous.

The interesting thing about a self-initializing fake is how you deal with situations where the remote service changes it's response.

One time I saw this approach was with a database controlled by another application. In this case the data did change, frequently. This is unhelpful for tests, because automated tests rely on getting the same answers to the same questions. But usually tests don't care whether the data is up to date or not, so saving an old value worked just fine.

I ran into this again recently while chatting with my colleague Josh Price. In his case the remote data was supposedly static, but occasionally there were changes, which would imply that the system he was developing needed to change - usually to handle formatting issues. In this case he had a special test suite that would get all self-initializing fakes to call the remote service and check that they returned the same value that was saved.

In this case early stages of their build pipeline ran against the fake, and the last (slowest) stage ran against the service itself. One interesting problem was that the remote service required some unimportant parameters which changed from call to call but didn't alter the results. These were stripped out of the URL when the fake looked the values up from the store.

(Thanks to Josh Price, Darren Cotterill, and Gerard Meszaros for their help with this piece.)

ComposedRegex

2009-07-24T14:33:00-04:00

One of the most powerful tools in writing maintainable code is break large methods into well-named smaller methods - a technique Kent Beck refers to as the Composed Method pattern.

People can read your programs much more quickly and accurately if they can understand them in detail, then chunk those details into higher level structures.

--Kent Beck

What works for methods often works for other things as well. One area that I've run into a couple of times where people fail to do this is with regular expressions.

Let's say you have a file full of rules for scoring frequent sleeper points for a hotel chain. The rules all look rather like:

score 400 for 2 nights at Minas Tirith Airport

We need to pull out the points (400) the number of nights (2) and the hotel name (Minas Tirith Airport) for each of these rows.

This is an obvious task for a regex, and I'm sure right now you're thinking - oh yes we need:

      const string pattern = @"^score\s+(\d+)\s+for\s+(\d+)\s+nights?\s+at\s+(.*)";

Then our three values just pop out of the groups.

I don't know whether or not you're comfortable in understanding how that regex works and whether it's correct. If you're like me you have to look at a regex like this and carefully figure out what it's saying. I often find myself counting parentheses so I can see where the groups line up (not actually that hard in this case, but I've seen plenty of others where it's tougher).

You may have read advice to take a pattern like this and to comment it. (Often needs a switch when you turn it into a regex.) That way you can write it like this.

    protected override string GetPattern() {
      const string pattern =
        @"^score
        \s+  
        (\d+)          # points
        \s+
        for
        \s+
        (\d+)          # number of nights
        \s+
        night
        s?             #optional plural
        \s+
        at
        \s+
        (.*)           # hotel name
        ";
  
      return pattern;
    }
  }

This is easier to follow, but comments never quite satisfy me. Occasionally I've been accused of saying comments are bad, and that you shouldn't use them. This is wrong, in both senses. Comments are not bad - but there are often better options. I always try to write code that doesn't need comments, usually by good naming and structure. (I can't always succeed, but I feel I do more often than not.)

People often don't try to structure regexs, but I find it useful. Here's one way of doing this one.

    const string scoreKeyword = @"^score\s+";
    const string numberOfPoints = @"(\d+)";
    const string forKeyword = @"\s+for\s+";
    const string numberOfNights = @"(\d+)";
    const string nightsAtKeyword = @"\s+nights?\s+at\s+";
    const string hotelName = @"(.*)";

    const string pattern =  scoreKeyword + numberOfPoints +
      forKeyword + numberOfNights + nightsAtKeyword + hotelName;

I've broken down the pattern into logical chunks and put them together again at the end. I can now look at that final expression and understand the basic chunks of the expression, diving into the regex for each one to see the details.

Here another alternative that seeks to separate the whitespace to make the actual regexs look more like tokens.

    const string space = @"\s+";
    const string start = "^";
    const string numberOfPoints = @"(\d+)";
    const string numberOfNights = @"(\d+)";
    const string nightsAtKeyword = @"nights?\s+at";
    const string hotelName = @"(.*)";

    const string pattern =  start + "score" + space + numberOfPoints + space +
      "for" + space + numberOfNights + space + nightsAtKeyword + 
       space + hotelName;

I find this makes the individual tokens a bit clearer, but all those space variables makes the overall structure harder to follow. So I prefer the previous one.

But this does raise a question. All of the elements are separated by space, and putting in lots of space variables or \s+ in the patterns feels wet. The nice thing about breaking out the regexs into sub-strings is that I can now use the programming logic to come up with abstractions that suit my particular purpose better. I can write a method that will take sub strings and join them up with whitespace.

    private String composePattern(params String[] arg) {
      return "^" + String.Join(@"\s+", arg);
    }

Using this method, I then have.

    const string numberOfPoints = @"(\d+)";
    const string numberOfNights = @"(\d+)";
    const string hotelName = @"(.*)";

    const string pattern =  composePattern("score", numberOfPoints, 
      "for", numberOfNights, "nights?", "at", hotelName);

You may not use exactly any of these alternative yourself, but I do urge you to think about how to make regular expressions clearer. Code should not need to be figured out, it should be just read.

Updates

In this discussion I've made the elements for the composed regexs be local variables. An variation is to take commonly used regex elements and use them more widely. This can be handy to use common regexs that are needed in lots of places. My colleague Carlos Villela comments that one thing to watch out for is if these fragments are not well-formed, ie having an opening parenthesis that's closed in another fragment. This can be tricky to debug. I've not felt the need to do it, so haven't run into this problem.

A few people mentioned using fluent interfaces (internal DSLs) as an more readable alternative to regexs. I see this as a separate thing. Regexs don't bother me if they are small, indeed I prefer a small regex to an equivalent fluent interface. It's the composition that counts, which you can do with either technique.

Some others mentioned named capture groups. Like comments, I find these are better than the raw regex, but still find a composed structure more readable. The point of composition is that it breaks the overall regex into small pieces that are easier to understand.

MercurialSquashCommit

2009-07-09T15:53:00-04:00

I've recently had a bit of a fiddle squashing some commits with Mercurial, so thought it was worth a post in case anyone else is looking to do this. I don't know whether this is the best procedure, but it seemed to work pretty well for me.

hg clone base working
# tip of base is revision 73
cd working
# do work, committing on the way
cd ..
hg clone working squash
cd squash
hg qimport -r 74:tip
hg qgoto 74.diff
hg qfold $(hg qunapp)
hg qfinish -a
cd ../base
hg pull ../squash

The basic task I was doing was some fairly severe moving around of files and folders. I wanted to do this in several steps to checkpoint my work as I went, but I wanted a single commit in the version history. (I gather git does this more easily with rebase.) Making a single commit makes it easier to understand what happened - particularly since moving files tends to complicate looking at repository logs. Moving files also complicates the process - a couple of times I ended up with a procedure that didn't work because it lost the ability to track the moves - I want to be able to go hg log -f and see when and what the original commits were before the move.

To begin I needed to enable the mq extension (mercurial queues) and set my diffs to git style. Git style diffs help to track file moves properly.

# in ~/.hgrc
[extensions]
mq=

[diff]
git=true

When using Mercurial in this way, it seems the general way of working is to have multiple repositories. Mercurial encourages different repositories where other systems, eg git or svn, would use different branches. People argue about this, but it's the Mercurial way of working. For this example I had 'base' as my original repos.

My first step was to clone base into a working repos.

hg clone base working

At this point the tip of base (and working) was revision 73. I did the file moves, with several checkpoint revisions as I went.

cd working
hg mv foo1 newdir/foo1
.. more hg mv ..
hg ci -m "moving around"
.. more hg mv ..
hg ci -m "moving around"
.. more hg mv and hg ci..
cd ..

By the time I was done the last revision was 80.

To squash them down into a single commit I cloned another repos.

hg clone working squash

It's important to clone at this point because I was about to edit history, so wanted to keep the original history handy until I knew it had worked. I now moved into there.

cd squash

Now I turned all the commits I'd done for the revisions into patches for the mercurial patch queue mechanism.

hg qimport -r 74:tip

I made the first change the current patch

hg qgoto 74.diff

I squashed all the patches together into a single patch

hg qfold $(hg qunapp)

The commit message for this folded patch would be all the individual commit messages linked together. I wanted a single message for my clean commit.

hg qrefresh -m "reorganized files"

I then turned the patch into a regular commit.

hg qfinish -a

I now had a single commit with all that work. I looked through it to see that everything was sane, in particular testing hg log -f on some moved files to ensure the history was still there. Once I was convinced all was well, I pulled the single changeset into the base repos.

 
cd ../base
hg pull ../squash

It's interesting to see how the attention on version control system has changed over the years. Early on the primary and only purpose was audit - to be able to safely go back to older revision - mainly to diagnose problems. Then attention switched to how they enabled collaboration between people. This didn't replace the need for audit, but built on top of it. Now there's more attention to using them to provide a narrative of how a code base changes - hence the desire for history rewriting commands like this. Again this need is built on top of the other two, but introduces new capabilities and new tensions.

My thanks to my colleague Chris Turner for his help and I also found this page very useful.

Android

2009-07-06T19:18:00-04:00

One of the side benefits of speaking at the Google IO conference last month was that I got a new phone - the HTC Magic android phone that Google gave to all attendees. I was actually in the market for changing my phone to something like this, so it came at a good time. Here's my impressions after carrying it around for a month or so.

My previous phone was a Nokia E61. I liked the E61 as a phone, but found it's web browser to be slow and unreliable and it that, together with the relatively small screen, was beginning to bug me - hence the desire for something else. Naturally I considered an iPhone, but although the company phone plan that I use is AT&T, it isn't possible to use the iPhone on it and I didn't fancy the hassle of sorting out a new phone plan. I tried a Blackberry storm for a few days, but (how about this for irony) the email was no good for me. Blackberries copy every email that comes into the email account, so it doesn't work well for an IMAP account with server-side filters - which is how I use my gmail account.

The short statement is that I do like the htc magic android.

The Good

Physically the device works very well for me. It's small, light, and fits well in my hand. The screen is bright, making web browsing is much nicer than with the Nokia.
Battery life seems reasonable, a day or two with my usual usage.
The app market seems to have a fair few useful things, I've downloaded a bunch of little apps which have seemed handy.
Video play works well. I've watched some TED videos and transcoded some other video using Handbrake which played well on the screen.
I like that I can upgrade the memory using a micro-SD card. It came with 2GB, and I'm upgrading to 8GB since it's pretty cheap.
I use gmail and Google calendar and the phone syncs nicely with those.
The phone charges via a mini-USB connection. One less charger to have to carry around.
I read one of the prags's books using an ebook reader and it worked pretty well.

The Bad

My biggest irritant so far is that it makes it hard to browse local HTML pages. It doesn't support file:// URLs. This is a big issue for me as I often copy static HTML files to my phone for reference purposes. There is a work-around, but it's kludgy.
Like every other calendar app on the planet, Google calendar suffers from TimeZoneUncertainty. This is a big issue with a phone that you want to change time zone as you travel.
I miss the Nokia's keyboard when typing. The soft keyboard just doesn't work as well.
While the touch navigation works pretty well, I'm sure I'd prefer the iPhone's multi-touch gestures.

The Uncertain

I haven't tried writing an app for it. I'd like to experiment, but I'm not allowing myself any such fun until I get the book finished.