Malware on Thoughtworks Websites

Summary

Over the last couple of weeks our I.S. team has been dealing with a nasty bit of malware on our websites. It surfaces as a hidden iframe on a page. This page then redirects to other servers and downloads a bunch of corrupted windows binaries to infect the visitor. It's hard to spot as it doesn't do this very often, so you can visit our site many times without ever running into it. Furthermore most malware scanners aren't able to detect the problem yet. As far as we know only visitors using Windows are affected, we don't know of any attack affecting Unix based systems such Macs or Linux.

We currently have replaced all our active sites with static sites that we believe are clean. However this also means we aren't able to update our sites or use their active features, which is a considerable problem for our business. I am able to update martinfowler.com, since it is a static site by design (although I wasn't able to update it for a week or so).

I'm writing this to provide information that I hope will be useful to people facing similar problems. I should stress, however, that I'm not a security expert, so bear that in mind as you read.

Our Discovery and Response

Our first sign that something might be up was in early January when we received a report that Google's crawlers had detected malware on our site. We immediately investigated, found no problem, and Google crawled again and declared us clean. At that point we were worried, but unsure what to do next.

Google flagged us again at the end of January, and we went back into investigation. The fact that the malware pops up so rarely makes it very difficult to figure out what sites were affected and to what extent. We switched our main site (thoughtworks.com) over to a static site. Our studios site didn't seem to show any signs of infection so we kept that up and monitored it. I posted a note to my site letting people know what we knew.

Late last week we got firm reports that martinfowler.com was infected. This was particularly worrisome since martinfowler.com is a purely static site - it just serves files off the file system - so that implied the infection was inside the webserver. We moved martinfowler.com to a more secure site, which since it's a static site, is relatively easy. However it did mean I couldn't access the site to update it for several days.

Over that weekend (Feb 12-13) we set up new versions of our dynamic sites on a clean servers. We monitored them and were depressed to discover malware appearing on thoughtworks.com again, so we reverted to the static site. Studios was still in the clear. (We later found out that this indication of malware was a false positive.)

While we were doing this we also engaged a specialized security company. They helped dig into more about what was going on. On Feb 16 we saw evidence that the studios site was infected and so we switched that to a static site.

On martinfowler.com we set up a new server that would give me update access, so I'm able to provide the document you're now reading. It's completely separate from the compromised machines, but we are monitoring it.

How the malware attacks visitors to the site

From what we've seen so far, the malware appears as a hidden iframe on a webpage. This loads a java applet from another site. This applet then uses a newly discovered exploit in the JVM for Windows to write various binaries. The ones we've seen affected are AAA9.exe, conhost.exe, csrss.exe, dwm.exe, and ms0cfg32.exe. Virus checkers are only just now beginning to pick the signature of the virus involved in the infection.

This is the only payload we currently know about, but there may have been other payloads in the past.

Scanning to see if you have been infected

Currently it seems few malware scanners detect the known payload. We do know of one free online scanner which seems to reliably detect the infection. So if you've visited our site since December on a windows machine, you may want to give this try.

How the malware got into our servers

We don't know, and it's likely we'll never know. It seems that key binaries have been altered, including apache and ssh. We've isolated the compromised servers so that they will only be used for investigative purposes. From our investigations we believe the infection began on December 27 last year.

What's next

We're carrying on with our investigations, in conjunction with the security specialists. We think we have the infection contained, but we thought that before. So we'll have to monitor the situation to see if there's any new signs of trouble.

We also have to rebuild our dynamic sites and redeploy them. This is more tricky because there are libraries involved that need to be checked. Our last attempt to deploy into a clean environment didn't work, so we need to learn a bit more about what is going on.

If you have any questions or concerns about this, feel free to email me, although I cannot guarantee a rapid response. I will update this page as I learn more and announce updates via twitter and my atom feed.