Enterprise Transforming Projects That Don't Kill the Enterprise

Project Chardonnay

Chardonnay-Corp is a well known manufacturer who produces expensive items (I have to be vague!). Increasingly much of their revenue comes from leasing these items rather than selling them. Leasing is an arcane and complicated business. As their leasing business grew, their computer systems needed to grow too. That's where we came in - to build what's referred to as a front end leasing system. A front-end leasing system handles all the stuff that occurs before you sign (or in leasing lingo, book) the lease. This includes credit checking the lessor, quoting the deal, origination, and producing the documents. Before we started most of these operations were semi-automated at best.

The system we've built would satisfy the dreams of many technologists. It uses all the latest Enterprise Java technology - EJB, JSP, and other TLAs. The important thing, of course, is what it's done to the business. It's allowed Chardonnay-Corp to consolidate its offices moving from several regional offices to a single office, reduced the amount of people (while retaining the amount of leases processed), brought more consistency to the process, and improved the cycle time - getting a quote used to take two days, now it takes 45 minutes. Also the wonder of the Internet has allowed Chardonnay-Corp's dealers to access the system directly for quotes, which helps make Chardonnay-Corp's products more attractive to customers

Project Grenache

Grenache-Corp is a major retailer, one of those that has a branch in every boring mall in America, not to mention every chique shopping street I've ever tried to elbow my way through. Like any retailer they are very concerned with their supply chain. They have to get goods to their affluent customers from their less affluent suppliers. They have warehousing complexities that are a nightmare. Every SKU has multiple different varieties.

Although I'll refer to Grenache as a single project, in fact it's a series of projects each of six-nine months in length carried out over the course of four years. Each project worked on a particular problem, such as hordes of containers arriving at a distribution center before Christmas and sitting there for days while people worked out what relationship the items in the containers had to items the retailers were actually expecting (the answer is “less than you'd think”). We built the systems using Forte technology built on a common relational database.

Project Tempranillo

Tempranillo-Corp rents out shipping containers - containers that vary from plain metal boxes to complex and cavernous refrigerators. A shipping company rents the containers, moves them around the world for a while, and then drops them off. An important role computers can play helping track where the containers are and when they are going to be picked up. Tempranillo-Corp's existing system wasn't Y2K compliant, and initially a new system was seen as the cheapest way to fix this. It didn't take long, however, before this evolved into an enterprise transforming system. We developed this in Forte (we used to do a lot of work with Forte).

Project St Emilion

St Emilion looks rather like Chardonnay - well at least the projects do. Big company, front end leasing system, Enterprise Java with bells on, there's a lot of overlaps. (And yes there was some reuse involved.) Differences include the fact that St Emilion-Corp's very expensive items are also technologically very advanced, so they become obsolete the moment they start manufacturing them. So a big benefit of St Emilion was allowing better control of inventory, which we did by tying into their ERP system. Also St Emilion-Corp's leasing business was less mature than Chardonnay-Corp's, so there was a greater benefit in putting together a consistent approach to quoting. Having salespeople put together deals on spreadsheets was flexible, but caused havoc in accounting and profit management.

Deliver Frequently

If there's any one thing that's struck me as a common difference between success and failure between these kinds of projects, it lies in frequent delivery. I've often ran into people who talk about the major enterprise transforming initiative they are currently working on. They speak with great enthusiasm about how well things are going now that they're two years into a five year program. Although I'm not a statistically valid sample, I find it significant that I've yet to meet this level of enthusiasm five years into a five year program. The best I usually get is “we would have succeeded if....”

The common theme of our successful projects is that of frequent delivery of working software throughout the life of the project. Indeed the growing theme is that the more recent the project, the more frequently we've tried to deliver. Grenache, the older project of the bunch, delivered every six to nine months. Although this was a grand plan to overhaul the supply chain, each project was a separate project, designed to be a viable business proposition all on its own. The informal tale was that typically these projects would pay for themselves twice within a year. Chardonnay is a more recent example, highlighting the trend to shorten the cycle. Here we deploy new software every two or three months.

Why is this important? For a start it's about building credibility. Businesses are not famous for being patient while they put a lot of money into an initiative. Even when you do get somebody who's prepared to sponsor a five year plan for overhauling the company, that person is not likely to last the whole five years. Often the big new initiative becomes a millstone around their neck sometime about year three. Unless somebody can show viability at that stage either the project is sacrificed, or the sponsor and the project find themselves without a home.

On the other hand deliveries build credibility, especially if you see benefits: reduced costs, increased timeliness - these count far more than hoped benefits. As people see the system in action they can see the value it brings, and also the changes it forces. This visibility also helps the credibility of the software developers. There's nothing worse than the business folks lamenting that while they hear confident project briefings, they don't actually know if the project is on track. Deployed software is tangible, and gives a clear statement of progress. Sometimes this isn't so good, if things are going slower than hoped. Yet even in this case I find it's worthwhile. If development slows down you can only hide it for so long. Eventually reality has a nasty habit of catching up with you. But if people see steady progress they are either more understanding of delay, or will cancel a project earlier. Either of which is better for the business than a long drawn out failure.

Frequent delivery also allows you to learn from your development. That sounds like an odd phrase to many, yet our experience is that when doing this kind of project, nobody really knows what the whole picture is. You can know where you're going if you've ploughed that route before, but enterprise transformation means going into a whole new world. One that's unpredictable for client and contractor alike. While at St Emilion we were able to bring our experience of front-end leasing software to bear (it was our fourth such system), the truth is that at that scale every business has enough different twists to be a new journey.

This is compounded by the fact that enterprise transforming projects often involve very new technology (I'll discuss why later). In this case the early deliveries should focus on allowing the technologists to learn about the technology and explore some of the high risk areas.These discoveries will be both good and bad, so make them early so you can adjust your plans accordingly.

There is an alternative to exploring new ground. That is to not try an enterprise transforming project until others have trod the ground before you. That way, of course, you have some idea of where to go. The problem with that, of course, is by the time you get to the party, all the food's gone.

As I've talked about this, I've talked about delivery in terms of delivery to the business, that of putting the software into production. However this notion can be extended further. Even within a deployment cycle of several months, the software developers can build the software in tighter cycles: as short as a few weeks. While you may not want to put each of these cycles into production you can still get many of the learning and tracking benefits with these development iterations. In the last year or so we've been using these kinds of short internal cycles on several of our projects, and as we see more success with this we are doing it more and more.

Expect Surprises

As I've alluded to earlier, enterprise transformation is not a predictable exercise. New business processes mean you're heading into territory that is only known vaguely, if at all. With these kinds of projects come surprises. Some these can be bad surprises, some of them good. Only by reacting to them can you avoid the damage of the bad ones, and maximize the gains of the good.

Tempranillo provided examples of both kinds. On the bad side of the coin there was the unpleasant discovery about repairs. When the project was initially planned, it was expected that the tracking of the repairs to containers would be a small part of the project. As the project went on we found out that about half the work was in tracking repairs. This caused a big increase the costs and timelines of the project. In this circumstance the only thing to do was to look at problem squarely and rework the plans.

I've seen the alternative, trying to carry on as if the original plan is still valid in order to avoid the failure of deviating from the plan. It leads to a divergence between the plan and reality. You can sustain this for a while, occasionally you may even get back on track due to other more pleasant surprises. But more often the divergence continues until the difference is so big that reality can't be ignored any longer. The fall out is usually very painful.

For the happier alternative, these can come in many ways. Greg, our project manager on Tempranillo, tells the story of the business folks looking at the fact that we now had a relational database for tracking the containers. Suddenly they realized that they were a simple SQL query away from finding all the companies that were being laggardly in doing, and paying for, repairs on containers. The access to this information was simple, yet saved Tempranillo-Corp a considerable amount of money. This benefit was simply overlooked in the project plans, yet turned out to be a significant benefit of the new system, worth a short detour to realize.

Tempranillo also had a bigger example of this. Tempranillo-Corp does a lot more than rent containers, and a new executive management decided to sell off parts of the container renting business. As they did this they discovered that the pieces to sell off were the unusual containers, such as the enormous fridges. The new system made it possible to easily separate these lines of business, which would have been a nightmare in the old system. Not just that, but it was not difficult to modify the system so that Tempranillo-Corp could sell off the business for the “tanks” (transport chemicals, food products, etc.) and still use the system to manage the containers for the buyer, thus getting money at both ends of the bargain. No amount of in depth requirements analysis could have predicted this change, because it was a deep business change.

Faced with this unpredictability, it's tempting to conclude that planning is hardly worth the effort. After all if plans are likely to be discarded part way though the project, what value do they have? But in reality we find that planning is all-important in these unpredictable projects. It's just that planning is done with different emphasis, in particular it becomes a continuous planning process, where plans are reworked at regular intervals. In this environment long term plans are both volatile and sketchy, but still help set directions. Short term plans are more stable, although still open to shifts when needed.

Chardonnay is an excellent example of this kind of process. While the team is working on the software for the current two month deployment, an analyst team works out what needs to be done for the next two months. I asked Scott, Chardonnay's project manager, what he knew of the tasks for six months ahead. “A list of bullet points, I don't know what most of them mean, and I don't think the customer does either.” Yet this project has continued to be a success for both Thoughtworks and Chardonnay-Corp.

Get High Level Executive Support

This is no surprise, or at least it shouldn't be. Big changes in business process need commitment from the top of the business management tree. Without it, nothing's going to happen. Transforming change is rarely welcome, but human beings naturally prefer the devil they know to the unknown future, however nice people tell them it'll be. Overcoming this natural resistance requires leadership, and plenty of it.

High level leadership is also needed to provide a vision for the future. This may not be a very detailed vision, but given the nature of these projects it doesn't need to be. What it does need is something to guide progress, even as plans are changing. This involves setting priorities, making hard choices about what to do and what not do. We've also found it's important to maintain a commitment to the frequent deliveries that I talked about earlier. Often there is the temptation to put off a deployment, to make time for some really neat features. But one neat feature leads to another, and soon you get led into the trap of not delivering and losing the credibility and learning advantages. Here it helps to have very frequent deployments. If Chardonnay has to skip a feature in the current deployment, it's only a two month wait to the next one. With that short time interval it's easier to make the hard decision of what not to do.

What happens if you can't get the high level business support you need. Our advice overwhelmingly is: don't bother with the project. That may sound a little harsh, especially if the project is one that you're convinced is good for the company. Yet there's only so much executive attention around, and without it enterprise transformation has about as much chance of success as putting your salary on 15 at Las Vegas.

Even if a project gets executive support, it's not at all certain that it'll keep it. At Grenache, the original leaders moved on to new pastures. That's not surprising - successful people like to move on to new challenges. The new leader fancied a different approach relying on ERP systems rather than custom software. Again this is a natural human behavior, new people often prefer to take a different approach to their predecessor even if the current set up is working. As a result the Grenache initiative has ended before revamping the whole supply chain. Frequent delivery is again a valuable technique. Even though the whole grand plan of Grenache hasn't been realized, since all the constituent projects were successful, and gave an excellent ROI, the business has benefited from the work so far. Not just is the money not wasted, it's delivered well on the investment.

Treat Business and Software Development as Partners

Perhaps one of the hardest things to do in an enterprise transforming project is to ensure there's a true partnership between the business and technology sides of the project. Of course we notice this because we are a separate company to our clients, but you often find this on an in-house project as well. Business people and technical people are different personalities with different aims and personality styles.

Even with an in-house technology group there is a strong desire to make the arrangement one of a fixed price contract. Here's what we need, go ahead and build it, tell us when you're done. But as I've indicated already, these kinds of project aren't like that - there's too many surprises involved. I agree completely that a fixed price contract would be better, if only it would work. However a fixed price contract depends on very well understood set of requirements. There are software projects that have these, but you don't find these in enterprise transforming projects. Hence the need for a different approach to planning - and a different approach to the business/technology relationship.

The partnership I'm talking about requires the kind of joint planning which I've talked earlier. One that expects that new requirements will appear, that problems both in business and technology will crop up. As soon as the two parties start battling over what they agreed to, the project is in serious trouble. In this kind of relationship you have to rely on trust, if that trust breaks down, then the project will go with it. Now that may happen, I've certainly seen cases where the technology supplier wasn't delivering the goods. In that case the project needs to be terminated, or at least put into mothballs until a new supplier's found.

So how do you find a supplier that you can trust? (I'll try to resist the urge to just say “call Thoughtworks”.) The obvious advice is to try to do the project in-house. That's odd advice for someone in a technology provider company to give, but at the risk of giving our sales VP an apoplexy I'll give it anyway. With an in-house team you take away a substantial piece of commercial tug-of-war. Indeed I used to believe that these kinds of projects should be done in house rather than with a technology company such as ourselves.

What changed my mind? (Other than the monthly pay check.) The problem is that many companies just don't have the resources to do this work in house. Technology companies can offer software professionals a greater range of interesting projects using leading edge technologies than most businesses. They also offer a free wheeling environment that's more geared to geeks like me. As a result of this they get better geeks. In enterprise transforming projects you usually need the best geeks you can get, so this is an important thing to bear in mind.This doesn't mean you have to give the project over entirely to a technology company. We do a lot of work that uses teams that are a combination of ThoughtWorkers and client staff - both Grenache and Tempranillo succeeded with this model.

It's important to realize that it's not just a bunch of smart geeks that you need. People who work well together are far more effective than a brighter group of people who don't work together. So don't look for a bunch of clever bodies, but for a team of people who have a common culture and have worked together before. It's also good to find managers and analysts who are also part of this culture and are used to working with this kind of group.

So if you need to shop outside, how do you select your technology provider for a project like this? One factor is the quality of their people. Make sure that they really do have the staff to carry the job out. Looking at resumes isn't usually enough. Often a good way is to find a project where they can send in some of their people to work on for a few months. During this time you can evaluate their skills and get to know their culture. Look at their previous projects. The business domain is less important here, although it certainly helps if they've done this business before. The key thing is to see if they've succeeded in working with a client in this partnership style. Checking references is the obvious and wise step. Look at their reaction to working with people from other technology providers. I've found that companies that seek to get rid of people from outside usually do so out of a well-deserved lack of confidence.

Of course once you select your technology provider, you have to monitor how things go. Ronald Reagen's famous quote “trust but verify” comes to mind. You need tangible signs of progress. A clear project plan with milestones that you can verify is the key here, and you shouldn't be surprised when I repeat my refrain of frequent deliveries. There's nothing better than working software to measure progress.

Also expect snags, and expect the technology provider to be up front about them. Tempranillo provides an excellent example of why this is important. As I've mentioned earlier we were four months away from delivery when it became clear that the repair part of Tempranillo was far more complex than anyone realized. The project had to slip several months. Being open about the situation was important, as was a client project manager who evaluated the situation and agreed with the assessment. While this meant the project was later than hoped for in giving full function, it was still a success and by using iterative delivery we were still able to ship partial but still useful software.

Notice that this example shows the importance of having a business project manager who is deeply involved with the project and is thus able to understand the issues that come up. While this isn't completely essential, we often manage without one, it certainly helps.

Of course when things go wrong they're occasionally our fault, after all we're not perfect. But again the key is to work to find a solution. On St Emilion we made the mistake of under estimating the complexity of the job and ended up putting in an initial team that weren't experienced enough. When we discovered this, the project was in difficulties. While we did a part by sending a squad of more experienced folks that were prepared to take emergency measures, it was also vital that our client also helped by re-planning the work. From the beginning we had been very open about the risks and that openness helped to develop the necessary trust. In the end St Emilion succeeded, but it took both of us to make some sacrifices to make it so.

Choose Technology for the Future

There is a particular problem with choosing technology for a project such as this. You can't make the decision on what is the best technology now, you have to evaluate what the best technology will be for when the project is in full flow, several years later. With technology changing as it has, that's a tricky choice.

Grenache and Chardonnay are interesting contrasts here. Grenache started in the mid 1990's and selected Forte. In many ways Forte was an excellent choice because it was a maturing platform for multi-tiered information systems. It has many features that more popular platforms don't have now. As it turned out, however, Forte got hit by the Java steamroller and is now a technology without a future. In 1995 this was impossible to predict (Java only started to appear that year) so it was only a poor decision in 20/20 hindsight.

Chardonnay's an example of the right choice. In 1997 enterprise Java technology was a risky choice. The tools and technologies were very immature, much less mature then Forte, or established client server tools like Powerbuilder and VB. Yet things panned out very well. The immaturity of both tools and knowledge about enterprise Java certainly made things slower than they could have been, but now enterprise Java has proven a stable platform with a future that has many companies behind it.

Not just is the platform choice one of tools in the longer term, it also affects the people you can get. Software developers tend to be intelligent, especially the better ones. As such they know that working with dying technology is a career limiting move. So they will gravitate away from technologies that seem to have a doubtful future. It got harder and harder to find people prepared to work on Grenache, even before the Java world really began to catch up technologically.

In the face of this, it becomes important to pick an integration strategy rather than a full technological platform. In Grenache this was apparent from the beginning since the projects that made it up were done separately. For Grenache this platform was a relational database. Every project was expected to work with a common relational database that acted as the shared base of the data. As a result the different projects could operate with a fair degree of independence. Such an approach also gives Grenache a way out of the Forte problem, future development could be done in Java, although Forte is still needed to modify the existing system.

In our enterprise Java we've made extensive use of using XML for communication between processing units as well as working more and more with the Java platform. As well as the good prognosis for Java we see XML as something that also has the potential to be around for a long time. XML receives a fair amount of hype, but it has an important place in enterprise development.

Although it's slightly tangential to this theme, this is a good moment to throw in some advice about that most-touted benefit of object-oriented platforms: reuse. Over the years I've run into plenty of people who've talked about building a foundation of reusable frameworks on which they intend to build the next generation of applications. I've run into hardly any successes. That's even not taking into account the problems when technology platforms disappear. The problem is that it's very hard to build frameworks from scratch. Grenache has a typical example: a sophisticated security framework that took a long time to build and ended up being more complicated than it needed to be.

For a while now my view, and that of quite a few other frameworkers, is that frameworks need to be evolved as part of the applications that use them. Build a couple of applications and then harvest a framework from the commonalities. The resulting framework is then designed for known needs rather than speculation. I need to warn you that this isn't easy, indeed it's no easier than building an initial foundation - the only difference is that it's much more likely to work.

People are The Critical Success Factor

What really makes a project like this succeed is the people who are doing and how they work together. In the end this is the single most important factor. Technology choices, methodologies, business volatility, all these play second fiddle. A good group of people who work well together will find ways to surmount the insurmountable problems, while a poor team won't try to get over a six foot wall. So if you're responsible for such a project the most important thing to do is get good people, keep good people, and ensure they work together well.

Getting and keeping good people is primarily about giving them the responsibility to carry out their job properly. I've heard a lot talked about empowerment and new approaches to managing people, but in practice this rarely seems to get beyond providing free coffee.

What people find difficult is actually stepping back and allowing a team to make the choices about what they know about, and to ensure the people who make decisions are the ones who have knowledge to make them. Technologists need to be trusted to make the technical decisions, business experts need to be trusted to make the business decisions. The role of management is often to ensure that the right people make the right decisions.

The other key role of management is to ensure that there is a rich and regular flow of information. Look for organizational and physical barriers to communication and remove them. A lot of this lies in simple things. Make sure people who work together sit closely to each other. Make sure there are plenty of meeting rooms for collaborative work. Invest in team building activities. A team that gels is worth a lot of effort to create

Particularly important here is regular contact between business experts and the development team. Grenache followed the practice of assigning a business expert full time to the team for the duration of the project. Chardonnay has a large group of Thoughtworks leasing experts who work with Chardonnay-Corp's business experts, identifying key supervisors who know the hows and whys of Chardonnay-Corp's business process.. What's needed is regular face to face contact, typically you'll find daily interaction between business people and developers. This is often ignored to a project's cost as communications and trust breaks down. The business side has to put in people to the project if they want it to be a success.

The key test of whether this arrangement is working is the reaction to bad news. Inevitably there will come times when the developers on the ground will run into problems, often those that will hit the schedule. If the management's reaction is to hide things or to tell everyone to just “work harder”, then you haven't got the kind of management that will work effectively. An honest reaction to problems and an openness to them is important both to work effectively with developers and to be a good partner to the business.

Keep Learning

Working in a complex and new ground inevitably leads to discoveries. As I've discussed above, the important thing to do with surprises is to learn from them. Indeed a key theme of a successful long term project is a notion of continuous learning. This learning is about the needs of the business, the technology we're working with, and process with which were doing the work.

One of the common discoveries of software development is the fact that people change their mind about requirements once they see early versions of a system. This is a natural effect because the presence of the system clarifies things for business people. As a result they learn about both their needs and about what technology can provide. One of the benefits of frequent deliveries is that it accelerates this learning process. The business people should use these deliveries, and their immersion in the development process, as a vehicle to learn more about what would be valuable as the software develops. Both Chardonnay and St Emilion gained a great deal from the short development cycles to help the business folks guide the project to deliver the most valuable functionality

Continuous learning also applies to technology. Since enterprise transforming projects often use newer technologies, learning is essential to make the best use of technology. When technologies are new or evolving rapidly (and what isn't these days) many of the best ways of using the technology are not clear early on. Also there's bound to be some hidden gotchas that also don't appear early on. A good team looks at the technology it's using and alters their approach as the project continues to use the technology in the best way. St Emilion and Chardonnay were a couple of the earliest serious projects using enterprise Java. Both teams learned a great deal about how enterprise Java should be used and evolved the system to take advantage of that learning, and learning from other Thoughtworks projects and flood of articles on using Java.

Often the easiest form of learning to miss is the importance of learning about the development process. At regular intervals it's important to look at how the team is working and for the team to think about how to improve. St Emilion had a number of key retrospectives where both the development team and the client team reviewed the process and decided how to improve. These process checks were essential in turning the project around from a project in trouble to a banner success. On the other hand Grenache became a victim of its own success, and later projects did not evolve enough to take advantage of its own experiences, which has led to less effectiveness in the last year or so.