For the last month or so I've been hanging out in our UK office, catching up with various UK ThoughtWorkers. I was intending to visit some of our client projects, but just catching up with people in and around the office has kept me pretty busy (it's also wiped out any book writing progress, but that can wait till I get back home.)

As a physical office, our London office is probably our nicest. It's on the ninth floor just north of Covent Garden. As I write this I can see the roof of the British Museum on one side and the London Eye and Houses of Parliament on the other. The office is very open. There's half a dozen meeting rooms, but the rest is large open desks. A few permanent desk for operations people at the front and several open desks with no set spaces at the back for consultants who happen to be in for the day. The open plan encourages lots of communication, which is why we like it. It also helps reinforce our lack of hierarchy: Cyndi, the UK managing director, sits at an open desk spot just like everyone else.

The London office has gone through lots of changes in the few years it's been around. For its first couple of years it was dominated by one or two very large projects. Now we have a dozen clients on the board with various smallish teams doing a variety of different things. Most of the work is in the London area, which is nice because it allows people to get together relatively well - the British reputation for drinking is alive and well here. We do get some work out of town, currently there's a project starting up in Dorset, but these are more the exception - so travel isn't the bugbear that it is in the US.

For me it's a big difference to life at home. At home I'm pretty much on my own, other than email and occasional phone calls, all day. For contact I have to fly, which I do every other week on average. (Flying every other week feels like such a luxury compared to my earlier 'every week' travel schedule.) Here in London there's lots of people to talk to right here. This also effects our evenings as we've been going out with friends nearly every night, while we are such stay-at-homes in Boston.

The UK office is brewing a number of interesting things. Two to mention are our automated deployment work and QuickStart. Historically our deployment approach has been a handover to the client's operations group. In London we're building up quite a team of really sharp deployment people who can work with development project teams from the very start of a project. This allows us to consider deployment issues from the very beginning and also build up automated and continuous deployment techniques.

At the other side of development is our QuickStart effort - short intense system envisioning activities intended to scope out projects before we start thinking about delivery plans. The idea with QuickStart is to get people used to working in a collaborative manner from the beginning to give people a feel of how a full delivery project will work out. The sessions are quite intense and focus on surfacing inconsistent viewpoints so they can be discussed and resolved early on. There's a big emphasis on using highly visual techniques - probably stemming from the fact that the leading perpetrators, Luke Barret and Marc McNeil, have a background in UI and interaction design.

Although it's not a conscious initiative, another thing I've noticed in the London office is the heavy use of retrospectives. I've long been a fan of retrospectives, if nothing else because I've been so frustrated with clients' inability to learn from their own experience. I haven't been terribly successful in pushing retrospectives at ThoughtWorks in the US, but in London Tim Mackinnon has done a really good job of spreading them around. We now use them on nearly every project as well as on our internal activities. It helps that Tim is a terrific facilitator, and he's steadily teaching some more people to run retrospectives too. I'm hoping I can get him to write about some of this, including his technique of 'Futurespectives'.

ThoughtWorks has come to China. It's been a long-held ambition for several people to open a China office. Roy has always held it as part of RoysSocialExperiment. In addition Xiao Guo, who's given me so many good experiences and ideas in software development, has long wanted to start ThoughtWorks in China.

We looked at several options as to where to open an office. We settled on Xi'an, the ancient capital of China. Although it's not seen as much of an economic center as Beijing and Shanghai, it's a sizable city with a large student population and a rapidly growing economy. The folks at the software park were particularly welcoming, in particular wanting to learn about our ideas for agile development and help spread their usage.

Our intention with ThoughtWorks China is to have the office set up to sell into the Chinese economy (in contrast to India, which is used for offshore development). China's economy is growing very fast, which provides a lot of opportunity for us in the long term. It's also a place with a lot of desire for technological experiment. One of our hopes is that we'll be able to work with technologies in China before people in the west are ready to work with them.

I was out in China a few months ago (before my accident). It was the usual difficult trip - I'm finding travel and speaking less and less fun as I get older. I'm also reluctant to say I can learn too much from such a limited experience. But I do have a few thoughts, most of which are indirect from those on the ground.

• Software isn't valued much yet. The attitude is that software is something that comes with hardware - rather like it was in the west a couple of decades ago.
• People want things built fast - which means there's a lot of package development. That's alright as far as it goes, but there's going to be a lot of integration work to make these fit together. There's also opportunity for firms to get a competitive advantage by using custom software in strategic areas - that's a market we're particularly interested in.
• There is some desire to copy India with its use of CMM, but there's also a realization that that kind of approach isn't necessarily the best. I see China's big strength as the fact that it's a more diverse economy which means that the Chinese software industry can mostly work onshore. Competing with India for off-shoring ends up playing on a field where India is naturally strong. So (possibly except Japan) the big opportunity is for customers inside China.
• The 'great firewall' was a pain. I missed the BBC and couldn't read many blogs. With information such a driving force these days, these kinds of barriers are only going to hurt.

I must also apologise to those who came to meet me in Shanghai. Sadly I caught a stomach bug and spent that day in hospital. I hope to get to Shanghai on my next trip.

It's still an open question as to what kind of operation we can have in China - particularly since software is not yet valued as much as it should be. I look at this as a long term exercise. It will take time to figure out where the opportunities are, and if we aren't there we won't be able to find them.

One of the big themes in ThoughtWorks is to encourage a diverse range of people in all parts of the company. (In this context we mean diversity in terms of such things as gender, race, sexual orientation, and the like.) We want to be a company where historically disadvantaged groups such as women and non-whites can feel comfortable and get just as many opportunities as the traditional WASPish leaders. Roy, being a notable mongrel, obviously cares about this diversity.

My purpose in writing about ThoughtWorks isn't to sell the company (honest!) but to explore both the good and bad of what we do. There's a big difference between the lofty goals of RoysSocialExperiment and our gritty realities, and diversity is a good example.

One of my happiest moments, shortly after joining, was when Cindy came along with me to a company retreat. At one point a ThoughtWorker came up introduced himself, and his (male) partner. Cindy was delighted to find that not just was he comfortable with being open with his homosexuality (something she couldn't imagine at the engineering company she was currently employed by) but even more so - nobody batted an eyelid. True acceptance comes not when something is merely accepted, but when people don't notice it as unusual any more.

That's an illustration of a bright spot in ThoughtWorks - a degree of acceptance of diversity that's much wider than most places I've seen in my consultant's life. Sadly it doesn't take much to see darker spots. A recent survey found that a mere 15% of ThoughtWorks consultants were female, look at our global management team and nearly everyone is a white male - these facts are embarrassing to report.

Although we're not here to play the numbers, these observations are enough to tell us that things aren't right and we need to see some change. We next have to understand what the causes of this lack of actual diversity are, and what we should do about it.

Certainly one of the foundations of all this is attitude. If people expect and support a white male club, this is inherently self-perpetuating. Mostly I think there is a positive attitude to diversity here, but there are certainly exceptions, which I hope are only islands. At a gathering in February a workshop session exposed some nasty undercurrents; I heard depressing stories of sexism that I never wanted to hear at ThoughtWorks.

I'm not sure how to deal with this. Naturally any obviously nasty cases need to dealt with, but much more is difficult to assess. I take the view that it's rude to say hurtful things about someones gender or race (and indeed much else). However I can be as guilty as anyone else if I say something that seems harmless to me that isn't to someone else. Gaining sensitivity on what people find hurtful is important, but not easy. I also don't want a climate where everyone feels suffocated by political correctness.

More pernicious is sub-conscious attitude issues. A good example of this is how blind auditions have effected diversity within classical orchestras. For a long time it was felt that there were plausible reasons why women couldn't be as good as men in many orchestral roles, such as weaker lungs for brass instruments. Major US orchestras have done auditions behind a screen for many years and there's strong evidence that this is why there are more women in leading orchestras today. Some of this change is due to conscious sexism, but it's also likely that unconscious prejudice was at work - how can a small woman get a good loud sound from a French Horn?

I like to think of myself as without a prejudicial bone in my body, but when I do I remember this story. I was visiting a branch of a large multinational in South Carolina. A black man, somewhat shabbily dressed, slouched into the meeting room. My mind immediately classified him as a cleaner. Within a few seconds he introduced himself as the Vice President of technology. Although nobody knew of my pre-conscious blunder, I've often pondered since why I made it. Was it his way of walking - more of a slouch than a confident stride? Was it because I was aware of being in the South? I can't escape a sure feeling that wouldn't have made this classification if he was white. Now I could excuse myself by saying that this occurred ten years ago, but frankly I have little confidence that I wouldn't repeat this error today.

So even if our conscious mind frees itself from prejudice, our sub-conscious is there to trip us up. Is that part of the reason for our difficulties with diversity? I neither know nor do I know how to deal with it. I do belive there is an inherent inclination to encouage people who are 'like us' and it needs a conscious effort to get away from that.

A particular challenge in attitude that we face as a services company is that our environment is heavily affected by our clients. Even if we are able to solve our issues, clients often can bring their own problems.

To deal with this, I need to relate another favorite story, this time a recent one. A prospective client of ours gave a talk that included some thoughts on race that seemed to come out of 1930's Alabama. Just about everyone was offended, but the best reaction was from a young black analyst which I'll paraphrase as: "When he says things like that I don't want to walk away from this job in anger. Instead I want to work with this guy, to be in his face with my abilities and professionalism. People like this won't change if no one is there to show them differently. People need to step up and be change agents even when its uncomfortable."

Other questions still remain, even if the attitude gets sorted out. That 15% figure sounds grotty when you compare it to women being 50% of the population, but go to a geek conference and 15% seems rather high. Whatever the reasons, software development isn't exactly full of females - so can we really suffer too much angst if we have the same ratio as the rest of the industry?

As soon as I point this out, I feel I can't duck the obvious question of nature or nurture. Are women under-represented in geekdom because they tend to be wired differently, or does society push them out? My view is that we don't know the answer to this question yet - therefore we ought to operate on the assumption that women have every bit as much potential as men. Simple fairness, not to mention the sad example of hundreds of years of institutional discrimination, should lead us to this conclusion.

So it follows that, in my view, we should be feeling the angst. I want ThoughtWorks to take a leading role in bringing out the talents of groups that have been traditionally been under-served, by our industry and wider society. I want people to be struck by ThoughtWorks having more women, more ethnic minorities than other places, particularly in leadership roles. The under-representation of minorities in ThoughtWorks is an embarrassment to us, but the larger under-representation in the industry is equally an embarrassment to the industry.

How do we achieve this kind of re-balancing? One thing I don't approve of is changing the standards - accepting a lower quality standard for females, for instance. Such an approach is counter-productive. But there are other ways. We can more aggressively pursue places where we can find women employees. We can do things in the recruiting process that make it easier to find and attract minorities, even if the resulting hiring standards are the same.

Another important factor is helping those we have be role-models both within ThoughtWorks and in a wider society. To some extent that does place an extra burden on minorities within ThoughtWorks, but the long term cure to both our, and the industry's, diversity problems is to make it clear that people of any background can succeed on their merits. Connected with this is to be more determined to provide mentoring, a recent discussion about the dearth of women in open-source considered mentoring to be one of the more promising ways of improving the situation.

But all this is still pretty limited. We still have a long way to do before we look the way we should. As in so many things, my biggest comfort is that I'm not the one who has to come up with the answers. The great benefit of our hiring model is that we have lots of brighter people than me who can think about these problems and how to solve them. But until then, I have to confess that diversity is one of ThoughtWorks's current failures.

As a company grows, you have to worry more about how it's led and who's responsible for choosing the leaders. Most companies have owners (shareholders) and they ultimately select the executive management. Executives then make most decisions for the company (or at least they like to think they do).

In case you haven't noticed, we are trying to be different. Although Roy currently is the majority owner, he's keen to devolve decision making away from himself as much as he can - and intends to change the company so it's truly employee controlled. Quite what employee controlled means for international company is still an open question. But there's no interest in becoming a public company.

The idea of pushing decision making out through the company is fine, but how do we make it work in practice? We have lots of bright people with an opinion on how the company should be run. Trouble is most of them are working for clients. Most ThoughtWorkers are quite busy enough doing client work to have much energy to think about the company's operations and strategy. So we have an operational management team that concentrates on that.

But there lies the problem. How do we avoid this operational management group turning into a traditional executive group who are distant from the day to day delivery issues? There is already more of a 'us and them' mentality developing than I would like - (although like most things, it's better than most places I've seen.)

A goodly part of the problem is that most delivery folks, in particular technical folks, aren't really interested in the operational management issues. They're interested in the project they're on, and on technological issues generally. That's quite enough to keep the brain full. Questions such as the balancing act between hiring and demand, doing the resource management dance, finding and keeping clients, watching the balance sheet - these just aren't interesting.

I must admit I'm guilty as anyone at this. When I joined Roy gave my carte blanche to crash any meeting I wanted to. But even when he's dragged me to operational committee meetings I have to confess I have little interest in the issues they are discussing. I know they are important - it's just that I'd rather have someone else worrying about them. Who knows I might be good at operational management (though I doubt it), but even so it doesn't excite me the way my regular work does - and I have precious little time to do that.

I'm not in favor of pushing geeks into ops against their will, or even their inclination. I've always felt we should get people to do what they do best and aim for well-rounded teams rather than getting people to work on their weak areas to try and round out individuals. So it's good that those with a liking and talent for ops do the operational management. But since their decisions have a big impact on delivery people, a painful gap appears. Delivery people complain about ops people making decisions that mess up their life (and the ThoughtWorks promise) and ops people complain about delivery people not understanding the business realities.

So by now I hope you grasp the problem (I doubt it's an unfamiliar one) and are gasping for the solution.

So am I.

In a smaller organization there's more personal contact, so that help alleviate the gap. Certainly we try to do that, but it's very tough to scale. Roy has an unbelievable ability to network with lots of people, but he has his limits. The other leaders don't share that skill but bring other vital ones to the table (like the ability to organize things). Being a social network hub can't be mandatory for operational management, even if it could scale.

One pond we dipped our toes into was the idea of a council that acts as a channel to help this communication. The idea is that we form a council from the delivery (and the support) sides of the company that meets regularly to air issues and provide the communication channel that both sides need. Maybe this council could evolve into a strategic leadership group. But so far that initiative has frankly fizzled. We haven't given up on it, but like most of these initiatives it's an expensive business (in all sorts of ways) to pull our top delivery people away from clients for a week.

Another thing we are trying to do is encourage rotation around operational management, so that anyone who goes into operational management only spends a few years there and then returns to delivery work. This would also bring out the point that operational managers aren't 'higher' than delivery people - just different. Roy's doing this at the moment, spending time in a regular delivery role.

But these techniques so far are just experiments. We are still trying to find techniques that work to bridge this very difficult gap.

When we talk about ThoughtWorks, we mostly talk about us as a software application development company. We also talk a bit about our values and how we are trying to be a different kind of company to most corporations. But all this is dancing around the point - fundamentally ThoughtWorks isn't about being a company.

ThoughtWorks is really Roy's Social Experiment.

Roy Singham is our founder, and in many ways he started ThoughtWorks to see if it was possible to create the kind of company he wanted to make, and make that company last for many years. Many of the elements of that company were things that people told him couldn't be done:

• You can't have a company that entirely consists of high ability people, you need a mix of less able people that the high ability people leverage.
• Intellectuals aren't interested in making money, so a company built around them won't stay viable.
• It's a harsh world where nice guys finish last - so you can't afford to be nice to employees and customers without an ulterior business reason.
• High ability people can't collaborate effectively, they intellectualize and self-destruct.
• Large companies need a strong management structure to avoid falling apart.
• Intellectuals must be run by B students since intellectuals are idealist and only greedy B students are pragmatic enough to make real decisions
• Doing things for the long term doesn't work.
• Being transparent about economics and operations is bad internally, worse externally and certainly won't scale.
• Don't reveal your weaknesses, especially to outsiders.
• The purpose of being international is to take advantage of people in weaker countries.
• Don't give production people powers that can be abused and hurt the company.
• Culture is secondary - it cannot be a sustainable advantage - you need a superior business model

In lots of ways of ThoughtWorks is a reaction to this kind of common sense. It employs only high ability intellectuals, aiming to have only those with a high degree of personal integrity. It's a rapidly growing international company with a determinedly flat and dynamic structure. Rather than have procedures, we try to give people on the ground information and rough principles and let them make decisions themselves. Much of Roy's thinking is strongly influenced by Dee Hock - the creator of Visa.

So while many companies are primarily defined by a business model, ThoughtWorks is primarily defined by our social model. Our belief is that an organization with the right social model can jump business models. This is increasingly important because business models don't last as long now since everything is changing so much faster.

I won't claim we consistently manage to live up to the ideals we set ourselves, but I do think there is a common desire to be the kind of company Roy dreams of - at least on his good days.

Roy may have started this experiment, but I've become as fascinated by it as anyone else. After all it's directly in line with my fundamental assumption that PeopleMatterMost in software development. Can you really make a company that can be enjoyable to work in and still make money? A place where you can work with highly capable people from all over the world without the bureaucracy and politics that dominates so many organizations? Can you win and still be nice? It's hard to write about this without sounding trite and rosy - but it's this unusual mixture that made me step away from the comfortable life of a successful consultant to be part of this experiment.

There are a few things that I've come to think are fundamental to how I see software development. If I had to pick one as my key to software development it's that the critical element in a software development effort are the people you have doing the work. The productivity of the best developers is far more than the average, much more than the difference in salaries. Therefore the most important part of getting cost effective software development is to hire the best team you can, even if the individual cost of the developers is much higher than the average. A few high ability (and expensive) people will be much more productive than many low ability (cheap) developers. That productivity difference means that a few high ability people will produce software more cheaply even if they cost more on a daily rate.

A lot of people say this, but few people seem to do it. Many just give up saying "you can't have a team of A players" as if there's some law of nature that prevents it. I don't agree with this. I do think it's hard to build such a team (let alone a whole company, which is what ThoughtWorks is trying to do), but the benefits are worth the trouble.

Why is it so hard? For a start you have to create an environment where high ability geeks are comfortable. This involves doing many things that don't make sense to many accountants and managers - which is why I came to believe that IT organizations inherently can't do it. It's this fundamental assumption that made me join ThoughtWorks. Here I think there is a company that's prepared to do the hard stuff to make this kind of software development environment work.

Recently something else has struck me. Most writers, including myself, talk about this stuff and stress the ability of the people is really important. While that's true it misses out the fact that it's not just about ability - it's also about collaborativeness. One of the things that makes ThoughtWorks such a good place is that everyone is so thoroughly pleasant to work with. That's no accident - ThoughtWorks does a lot to foster and encourage that in its culture. That's not easy to do and it's yet another reason why I think most big companies find it so hard to build this kind of organization internally.

It's interesting that this has become more conscious to me recently. I remember clearly a decision I made early in career, just as I went independent. I decided then that I wouldn't work with unpleasant people, however capable they might be. I wouldn't fight them, just avoid them. I decided that whatever the advantages it would bring to hang around with able jerks, it just wasn't worth the hassle. I've never regretted that decision, but it kind of floated deep into my axiomatic base and I lost sight of it. Even when I joined ThoughtWorks I didn't consciously notice that as part of my reason to join - focusing again on the ability issue. Yet it was clearly part of the reason for joining and for some reason I've now become much more aware of it.

Whenever I talk like this, I think it's important to remind everyone that ThoughtWorks still has a way to go before it becomes the kind of organization I think it should be. I'm here because it's not enough just to hold an opinion on the right kind of software organization - I believe I also have to help to try to build it.