English
The Internet threat alert status is currently normal. At present, no major epidemics or other serious incidents have been recorded by Kaspersky Lab’s monitoring service. Internet threat level: 1

Botnet Shutdown Success Story: How Kaspersky Lab Disabled the Hlux/Kelihos Botnet

Tillmann Werner
Kaspersky Lab Expert
Posted September 28, 19:53  GMT
Tags: Botnets
1.2
 

Earlier this week, Microsoft released an announcement about the disruption of a dangerous botnet that was responsible for spam messages, theft of sensitive financial information, pump-and-dump stock scams and distributed denial-of-service attacks.

Kaspersky Lab played a critical role in this botnet takedown initiative, leading the way to reverse-engineer the bot malware, crack the communication protocol and develop tools to attack the peer-to-peer infrastructure. We worked closely with Microsoft’s Digital Crimes Unit (DCU), sharing the relevant information and providing them with access to our live botnet tracking system.

A key part of this effort is the sinkholing of the botnet. It’s important to understand that the botnet still exists – but it’s being controlled by Kaspersky Lab. In tandem with Microsoft’s move to the U.S. court system to disable the domains, we started to sinkhole the botnet. Right now we have 3,000 hosts connecting to our sinkhole every minute. This post describes the inner workings of the botnet and the work we did to prevent it from further operation.

Let's start with some technical background: Kelihos is Microsoft's name for what Kaspersky calls Hlux. Hlux is a peer-to-peer botnet with an architecture similar to the one used for the Waledac botnet. It consists of layers of different kinds of nodes: controllers, routers and workers. Controllers are machines presumably operated by the gang behind the botnet. They distribute commands to the bots and supervise the peer-to-peer network's dynamic structure. Routers are infected machines with public IP addresses. They run the bot in router mode, host proxy services, participate in a fast-flux collective, and so on. Finally, workers are infected machines that do not run in router mode, simply put. They are used for sending out spam, collecting email addresses, sniffing user credentials from the network stream, etc. A sketch of the layered architecture is shown below with a top tier of four controllers and worker nodes displayed in green.



Figure 1: Architecture of the Hlux botnet

Worker Nodes

Many computers that can be infected with malware do not have a direct connection to the Internet. They are hidden behind gateways, proxies or devices that perform network address translation. Consequently, these machines cannot be accessed from the outside unless special technical measures are taken. This is a problem for bots that organize infected machines in peer-to-peer networks as that requires hosting services that other computers can connect to. On the other hand, these machines provide a lot of computing power and network bandwidth. A machine that runs the Hlux bot would check if it can be reached from the outside and if not, put itself in the worker mode of operation. Workers maintain a list of peers (other infected machines with public IP addresses) and request jobs from them. A job contains things like instructions to send out spam or to participate in denial-of-service attacks. It may also tell the bot to download an update and replace itself with the new version.

Router Nodes

Routers form some kind of backbone layer in the Hlux botnet. Each router maintains a peer list that contains information about other peers, just like worker nodes. At the same time, each router acts as an HTTP proxy that tunnels incoming connections to one of the Controllers. Routers may also execute jobs, but their main purpose is to provide the proxy layer in front of the controllers.

Controllers

The controller nodes are the top visible layer of the botnet. Controllers host a nginx HTTP server and serve job messages. They do not take part in the peer-to-peer network and thus never show up in the peer lists. There are usually six of them, spread pairwise over different IP ranges in different countries. Each two IP addresses of a pair share an SSH RSA key, so it is likely that there is really only one box behind each address pair. From time to time some of the controllers are replaced with new ones. Right before the botnet was taken out, the list contained the following entries:

193.105.134.189
193.105.134.190
195.88.191.55
195.88.191.57
89.46.251.158
89.46.251.160

The Peer-to-Peer Network

Every bot keeps up to 500 peer records in a local peer list. This list is stored in the Windows registry under HKEY_CURRENT_USER\Software\Google together with other configuration details. When a bot starts on a freshly infected machine for the first time, it initializes its peer list with some hard-coded addresses contained in the executable. The latest bot version came with a total of 176 entries. The local peer list is updated with peer information received from other hosts. Whenever a bot connects to a router node, it sends up to 250 entries from its current peer list, and the remote peer send 250 of his entries back. By exchanging peer lists, the addresses of currently active router nodes are propagated throughout the botnet. A peer record stores the information shown in the following example:

m_ip: 41.212.81.2
m_live_time: 22639 seconds
m_last_active_time: 2011-09-08 11:24:26 GMT
m_listening_port: 80
m_client_id: cbd47c00-f240-4c2b-9131-ceea5f4b7f67

The peer-to-peer architecture implemented by Hlux has the advantage of being very resilient against takedown attempts. The dynamic structure allows for fast reactions if irregularities are observed. When a bot wants to request jobs, it never connects directly to a controller, no matter if it is running in worker or router mode. A job request is always sent through another router node. So, even if all controller nodes go off-line, the peer-to-peer layer remains alive and provides a means to announce and propagate a new set of controllers.

The Fast-Flux Service Network

The Hlux botnet also serves several fast-flux domains that are announced in the domain name system with a TTL value of 0 in order to prevent caching. A query for one of the domains returns a single IP address that belongs to an infected machine. The fast-flux domains provide a fall-back channel that can be used by bots to regain access to the botnet if all peers in their local list are unreachable. Each bot version contains an individual hard-coded fall-back domain. Microsoft unregistered these domains and effectively decommissioned the fall-back channel. Here is the set of DNS names that were active before the takedown – in case you want to keep an eye on your DNS resolver. If you see machines asking for one of them, they are likely infected with Hlux and should be taken care of.

hellohello123.com
magdali.com
restonal.com
editial.com
gratima.com
partric.com
wargalo.com
wormetal.com
bevvyky.com
earplat.com
metapli.com

The botnet further used hundreds of sub-domains of ce.ms and cz.cc that can be registered without a fee. But these were only used to distribute updates and not as a backup link to the botnet.

Counteractions

A bot that can join the peer-to-peer network won't ever resolve any of the fall-back domains – it does not have to. In fact, our botnet monitor has not logged a single attempt to access the backup channel during the seven months it was operated as at least one other peer has always been reachable.

The communication for bootstrapping and receiving commands uses a special custom protocol that implements a structured message format, encryption, compression and serialization. The bot code includes a protocol dispatcher to route incoming messages (bootstrap messages, jobs, SOCKS communication) to the appropriate functions while serving everything on a single port. We reverse engineered this protocol and created some tools for decoding botnet traffic. Being able to track bootstrapping and job messages for a intentionally infected machine provided a view of what was happening with the botnet, when updates were distributed, what architectural changes were undertaken and also to some extend how many infected machines participate in the botnet.



Figure 2: Hits on the sinkhole per minute

This Monday, we started to propagate a special peer address. Very soon, this address became the most prevalent one in the botnet, resulting in the bots talking to our machine, and to our machine only. Experts call such an action sinkholing – bots communicate with a sinkhole instead of its real controllers. At the same time, we distributed a specially crafted list of job servers to replace the original one with the addresses mentioned before and prevent the bots from requesting commands. From this point on, the botnet could not be commanded anymore. And since we have the bots communicating with our machine now, we can do some data mining and track infections per country, for example. So far, we have counted 49,007 different IP addresses. Kaspersky works with Internet service providers to inform the network owners about the infections.



Figure 3: Sinkholed IP addresses per country

What now?

The main question is now: what is next? We obviously cannot sinkhole Hlux forever. The current measures are a temporary solution, but they do not ultimately solve the problem, because the only real solution would be a cleanup of the infected machines. We expect that the number of machines hitting our sinkhole will slowly lower over time as computers get cleaned and reinstalled. Microsoft said their Malware Protection Center has added the bot to their Malicious Software Removal Tool. Given the spread of their tool this should have an immediate impact on infection numbers. However, in the last 16 hours we have still observed 22,693 unique IP addresses. We hope that this number is going to be much lower soon.

Interestingly, there is one other theoretical option to ultimately get rid of Hlux: we know how the bot's update process works. We could use this knowledge and issue our own update that removes the infections and terminates itself. However, this would be illegal in most countries and will thus remain theory.

Nevertheless, we are interested in your opinion. Please tell us what you think and choose from the options below. Feel free to comment on this post as well.

How should Kaspersky proceed with the Hlux/Kelihos Botnet?


View results

Acknowledgement

Running a botnet sinkhole that is capable of dealing with a large number of connections requires appropriate resources. We would like to thank SURFnet for their support of the operation, and especially for providing a perfect infrastructure to run the sinkhole.


9 comments

Oldest first
Threaded view
 

Tatyana Nikitina

2011 Sep 29, 17:22
0
 

Cheers for LK!

Nice work, and thanks for illustrative review.

Reply    

Tiago R. Zacchello

2011 Sep 29, 20:09
0
 

Awesome.

We'll keep trusting in you. Good job.

Reply    

Benjamin Wright

2011 Sep 29, 20:26
0
 

Legality

I admire Microsoft's and Kasperksy's work here. I wish to know much more about the legal analysis of the work. I have read the blog announcement by Microsoft.

In the article above, Mr. Werner says, "This Monday, we started to propagate a special peer address." Then he further says, "We could ... issue our own update that removes the infections and terminates itself. However, this would be illegal in most countries..." Why is propagating a peer address legal while issuing an update illegal? Where can I find a discussion of the legality of the technical measures used in this take down?
Thank you

--Benjamin Wright
Attorney
Dallas, Texas
SANS Institute Instructor on the Law of Data Security and Investigations
http://legal-beagle.typepad.com

Reply    

Dallarki

2011 Oct 03, 16:41
0
 

Re: Legality

I feel propagating a new peer address to machines is not (or should not be) illegal, however issuing an update IS. In this case it is an antivirus company working together with Microsoft to get rid of something that's clearly known to do nothing but damage. The machines connect to their (Kaspersky's) machine and "follow" the controller machine's lead. Issuing an update is actually sending something to the clients which in my opinion is indeed illegal. However this is why i feel antivirus company's should be able to arrange "jurisdictional immunity" together with the government or law enforcing agency's.

Good job Kaspersky, i enjoyed reading the blog!

Reply    

Tarakanov

2011 Oct 04, 13:32
0
 

Re: Legality

Push an update means remote launching of arbitrary code on other's side without agreement. Push a special peer address means change of behaviour of existing malware. I'd say if you know that there is killer hiding in some house and you have intercepted a letter with photo of victim and send fake letter to killer with order - do nothing, wait - this is more or less acceptable. But if you'll send letter with anthrax spores to eliminate killer that also could cause poisoning the house where innocent people live - this is different story.

Reply    

Benjamin Wright

2011 Oct 04, 19:18
0
 

Re: Re: Legality

I'm just a lawyer; I have no technical credentials. So I will ask a dumb technical question: When a well-meaning party like Kaspersky/Microsoft propagates and promotes a new peer address, is there a 100% guarantee that this action will cause no harm to infected machines?

Please understand, I support and salute the efforts of Kaspersky and Microsoft. The reason for my questions is that I'm trying to understand the belief that Kaspersky and Microsoft are forbidden by law from taking more aggressive steps against this heinous botnet. --Ben

Reply    

Tillmann Werner

2011 Oct 05, 18:17
1
 

Re: Legality

Ben, the botnet is technically very similar to a file sharing network: nobody will give a guarantee that offering files does not break anything, because they can't. Or even more generally put, nobody knows what effect the packets cause that they send through the Internet ecosystem.

However, when we are talking about running code on other people's computers without their consent, that's obviously a totally different story.

Reply    

Benjamin Wright

2011 Oct 07, 04:17
0
 

Re: Re: Legality

Tillmann: Thank you for the reply. I argue there is tolerance in the law for socially-responsible actions like taking down an evil botnet.

I argue there is tolerance in law for propagating the new peer within the botnet. Further, there is no clear, bright-line legal distinction between propagating the new peer and pushing an update. Both actions arguably access computers, without explicit authority from the owners, with the risk causing harm.

However both actions, when executed carefully by responsible and capable parties (like Microsoft and Kaspersky), would be intended and designed to cause much good. I argue that the amount of good caused by an action carries much weight in determining whether the action is legal.

But if my arguments are wrong, I am very interested to hear why they are wrong. I am open to changing my mind.

[This is just a public discussion and not legal advice for any particular situation. If the reader needs legal advice, he should consult his own lawyer.]

Edited by Benjamin Wright, 2011 Oct 07, 07:27

Reply    

NeoSys

2011 Sep 29, 23:15
1
 

Keep up the good work Kasper !!

I am always amazed by the quality or your research, technical analysts and written articles. We all believe is a bot free world and your are part of this change to happen.

Thanks ,

NeoSys
Kaspersky Certified

Reply    
If you would like to comment on this article you must first
login


Bookmark and Share
Share

Analysis

Blog