Resilient infrastructure
"The biggest change has been the move from a dedicated control server to a more distributed model," says David Emm, senior security researcher for Kaspersky's Global Research and Analysis Team. This, he says, really started in earnest with the Storm worm, which used multiple points in the network that could perform the Command and Control (CnC) server function.
This has evolved over time - most notably with Conficker - to the point where botnets have a resilience that would be the envy of many corporate sysadmins. Conficker, for example, generated huge lists of domain names that the agent on the infected machine would check for instructions and updates. The botnet operators needed only to register a few of these to remain in control. "That meant it was a completely moving target," says Emm.
The ability of botnet operators to manage their systems is perhaps best illustrated by the Koobface worm. Kaspersky noted that cleaning operations had steadily reduced the number of CnC servers dropping from 107 to 71 over the space of a couple of weeks. Then, in 48 hours, the number doubled to 142.1 This was widely interpreted as the botnet operators making adjustments, like any good system administrator would.
The decline in the number of Koobface command and control servers was dramatically reversed over a 48-hour period.
Arguably the weakest point for the botnet is the necessary communication between the agent and a CnC server: this is where investigators can intervene to either gather intelligence. However, botnet agents deploy multiple anti-forensics techniques to reduce this risk. They look for known IP addresses belonging to research companies, or if they're running in the kind of environment used by such organisations (eg, VMware), and then refuse to run.
The code also has unpredictability designed into it: Conficker, for example, simply fails to replicate 10 per cent of the time. Not only the malware code but its communications with CnC servers have become increasingly encrypted. And some or all of the communications may use common protocols, such as HTTP, to blend in with legitimate network traffic. All of this is meant to make analysis and investigation difficult, and it works
Tracking down servers
Nevertheless, if a botnet agent can communicate with a CnC server then - theoretically - an investigator can too. Therefore the blackhats have become more sophisticated here too.
"The Gumblar botnet pushed the envelope by introducing a division of labour," says Emm, "with some machines responsible for injection of code, others handling redirection of requests, and so on. It created a kind of automated botnet, built on the theft of FTP credentials which helped it grow faster. All the managers of Gumblar need to do is just adjust or update the central code to get the botnet to do what they want it to do, whether it's spam delivery, the installation of fake AV software, etc."
The infrastructure and (for want of a better term) ‘business' strategies of the botnet operators have become more sophisticated in other ways, too. On a technical level, we've seen the development of techniques such as IP and domain fluxing. In the former, the IP address in the A record for a given Fully Qualified Domain Name (FQDN) is switched rapidly between hundreds or even thousands of addresses (using short Time-To-Live, or TTL, values), which the botnet operators register and deregister at an equally rapid rate. The double-flux technique also fluxes the IP addresses of DNS servers (by altering NS records). Domain fluxing is the opposite; here, a single IP address is used for a constantly changing range of domain names. Lately, this technique has been linked with domain generation algorithms, in which bot agents generate FQDNs according to a built-in algorithm. The botnet operators simply have to register a few of these names to maintain contact with the agents.
Multiple campaigns
Botnet operators - at least the more organised ones - further complicate matters by running several operations simultaneously.
"The general trend recently among the professional botnet operators is that they run multiple campaigns," says Gunter Ollman, VP of research at Damballa. "Traditionally, one botnet equalled one botnet operator group but now one group may have multiple botnets and multiple botnet-building campaigns going on simultaneously."
This means that, if one botnet goes down, they can switch to another. They also typically used a hierarchical CnC infrastructure at the first layer. A fairly typical, medium-sized botnet of maybe 10,000-50,000 agents will connect to, say, a couple of dozen CnC servers that may be geographically dispersed. But those servers in turn report up to a handful of master servers."While the first-tier CnC servers are generally unique to a particular campaign," says Ollman, "the higher tiers of servers tend to get recycled among campaigns."
Who's hosting these servers? Typically, says Ollman, the first-tier CnC servers will be owned by the kind of commercial hosting provider that normal organisations and individuals use. The explosion of ISPs and hosting services, offering online sign-up and instant configuration, makes it an easy matter to create a server in a matter of moments. What's more, many hosting providers offer colocation services, with servers conveniently scattered around the world. Upper-tier servers may be operated by the blackhats themselves. Because they exist behind the first-tier layer, they are usually hard to discover.
The professional botnet operators have become so adept at running these infrastructures that they now offer hosting services to other botnet operators. Their services might include everything from pure content delivery - eg, for drive-by downloads - to fast-flux and double-flux CnC server infrastructures.
Taking down botnets
The dreams of vigilante sysadmins aside, taking down botnets is largely a legal matter.
"The best way of closing down botnets is to focus on the botnet operators, as opposed to closing down the infrastructure," says Ollman.
Professional botnet operators will choose, for their first-tier servers, so-called ‘bulletproof' or ‘takedown resistant' hosting services. These are genuine and legitimate companies which have simply decided to adopt the attitude that they will ignore any requests for information or assistance that don't originate from in-country law enforcement. And while co-operation between various countries' law enforcement agencies is improving, it's still a long way from perfect, and tends to move slowly.
There's also a catch. The distributed and reconfigurable nature of botnet infrastructures means that you need to take down all of the first-tier CnC servers in order to knock the botnet offline. A professional botnet might have, say, 50 first-tier CnC servers, with only 20 or so actually live at any one time. Hit 19 of those and the operators bring up the others, while the one you missed pushes out new configuration files to the agents on the infected machines.
"Even if you could get all of them, most of these guys are running multiple botnets," says Ollman, "so they'd just switch. Losing 10,000 or 20,000 machines isn't that much of a hit, and they'd still have all the infrastructure - the distribution points and infection vectors - for rebuilding that botnet."
To see this effect in action, take a look at some of the recent takedown attempts, which met with various degrees of success.
Waledac
In tackling the Waledac botnet, Microsoft's ‘Operation b49' took a two-pronged approach - technical and legal.2 According to Microsoft, this action is its new model for combating the botnet menace, something the company has dubbed Project MARS (Microsoft Active Response for Security). The company obtained a restraining order that shut down 277 domains and the order required VeriSign to cut them off at the domain registry level. It also took what it called "additional technical countermeasures to downgrade much of the remaining peer-to-peer command and control communication within the botnet".
On the face of it, it had some success. As many as 90,000 zombied machines could no longer communicate with CnC servers (although they were still infected). But even Microsoft is being cautious about claiming a victory. There's some evidence that Waledac is having trouble making new infections but that it hasn't disappeared.
Mariposa
There was a somewhat greater success with the Mariposa botnet, but probably because its operators' professional standards fell far below those of the major criminal gangs (they were, in fact, dubbed as rank amateurs by most concerned). The botnet was said to control 13 million computers across 190 countries. Co-operation between anti-malware industry researchers, who formed the Mariposa Working Group, and law enforcement led to several arrests in Spain. The Mariposa Working Group assumed control of the communications channels used by Mariposa. But that was followed immediately by a DDoS attack against Defence Intelligence, the company that first indentified the botnet and a key member of the Mariposa Working Group. This attack was serious enough to temporarily cripple a major ISP.
Zeus and Troyak-AS
Troyak-AS is a hosting service based in Kazakhstan that was alleged to be home to a large number of ZeuS-based botnet CnC servers. ZeuS has been gaining quite a reputation as a major menace. In early March, Trend Micro reported it was seeing 300 unique samples a day. Many were being distributed via the Avalanche fast-flux botnet, responsible for large amounts of spam.3 The approach here was to cut off Troyak-AS at source, by removing its upstream Internet connection. It worked for a while - after the axe fell, on March 9, the number of active ZeuS CnC servers fell from 249 to 181, but the number has risen and fallen several times since then.4 It seems that Troyak-AS may have had as many as four ISPs serving it, and was able to find others. It's had at least three since the initial intervention.
Of course, Troyak-AS's service suppliers were probably unaware of any link to botnets, and this raises issues of governance and responsibility which have yet to be addressed. There is some talk of tier-1 providers installing technology to watch for dubious traffic, but this has not so far progressed beyond the concept stage.
Protecting yourself
For end users, the best protection against the depredations of botnets is not to get infected in the first place. But as Ollman says: "If only that actually worked."
The average Windows user stands a 20-30 per cent chance of getting infected in any given year. From a corporate perspective, how do you deal with this?
Increasingly, the solution is to treat a compromised machine like any other helpdesk issue. The traditional approach has been to run clean-up tools, and even, perhaps, full forensic analysis in an attempt to track or identify the bad guys. That can all take time, increasing the risk of further infection and damage to the corporate network. Instead, as soon as you've identified a compromised machine, just re-image it and leave the job of tackling botnets to the specialists.
References
1. Kaspersky Lab Discovers Koobface Worm Doubles its Number of Command and Control Servers in 48 Hours http://www.kaspersky.co.uk/news?id=207576049
2. ‘What we know (and learned) from the Waledac takedown', Microsoft Malware Protection Center, Threat Research & Response Blog
3. ‘ZeuS: A Persistent Criminal Enterprise', Trend Micro, March 2010 http://us.trendmicro.com/imperia/md/content/us/trendwatch/researchandanalysis/zeusapersistentcriminalenterprise.pdf
4. See the ZeuS tracker at abuse.ch https://zeustracker.abuse.ch/statistic.php
