Cracking Custom Encryption – An Intuitive Approach to Uncovering Malware’s Protected Data

WorkShop Duration: 2 Hours.

As banking fraud researchers, we take part in a never-ending chase after new configurations of banking malware. We strive to have the upper hand, by figuring out where the configurations are hidden and how they are encrypted. It can be quite thrilling, when a new version of the malware is released, encryption had changed, and the configuration must be decrypted before time runs out. We’d like to share this thrill, and teach useful skills that may come in handy when dealing with a variety of custom encryption algorithms used by malware authors. In many cases, cracking an encryption requires advanced skills in math and reverse engineering. But quite often malware authors create custom algorithm for data formatting and encryption, which can be overcome using a more intuitive skillset and methods. A great example is the encryption used by Dridex, which we shall use as a case study. In this workshop, lecturing will be kept to the necessary minimum and the major part of it will be dedicated to a hands-on guided process of analyzing raw encrypted data. We shall study the way it is encrypted, eventually formulating a simple method of decryption. Participants will gain an understanding of the process of researching an actual encryption method, acquire basic tools for addressing encrypted data of unknown format and enjoy the thrill of a live challenge.

Participants’ software:
A hex editor and a scripting platform such as python or ruby

Participants’ knowledge:
very basic programing knowledge – Hexadecimal digits and basic operations such as XOR. Basic scripting capabilities are not necessary but are highly recommended.

Outline of the workshop:
Introduction: First, we shall acquaint the participants with the world of banking malware, which will be our environment and case study for the workshop. We shall provide basic knowledge about Dridex itself and general banking fraud methods, along with a detailed explanation about the role of configurations, challenges researchers face at retrieving them and the corresponding challenges malware faces in protecting themselves.

Cryptographic warm-up challenges: Before getting started with the challenge itself, the participants will face a short series of basic cryptographic challenges, which we will use to teach and exemplify different methods of approach to encrypted data.

Getting started – introduction to the raw materials:
Participants will be provided with two files. The first containing a raw, encrypted configuration and the other containing the fully decrypted raw text version of the same configuration. Do note that this not a full Dridex configuration, but an excerpt of one, about 400 bytes long, adjusted to this workshop in terms of length and complexity. We shall make sure that all participants have successfully opened the encrypted data in a hex editor, raw text in a text editor, and that they understand the task. Participants will start by analyzing data without any hints, looking for patterns in the code, using naked eye observation, entropy calculation, length comparison between raw text and encrypted one, or any other method they deem fit. The first stage is purposely unguided, giving the participant a share of some healthy researcher’s frustration.

Main challenge – guided session:
In this section of hands-on challenge, participants will crack the unique encryption method by themselves with our guidance. As in the introduction stage, Participants will go through the process of finding patterns in the encrypted binary data, such as existence of data blocks delimiters and making smart assumptions about their role in the format. They will match this against the known plain text form of the configuration, to try and verify their assumptions. Every 15 minutes we shall call the attention of participants and provide pieces of information about the encryption, which have been extracted via reverse engineering of Dridex’s decryption functions. These “clues” will be given as Ida screenshots, along with very simple pseudo code explaining what we see in the screenshot. The main purpose of these clues is to guide participant in the process of cracking the configuration. These clues provide participants with information that helps them move forward. For example, some of the clues inform them as to how the nodes partitioning works in the binary code, what is the specific encryption method used (circular DWORD XOR), where the keys are hidden, how node binary flags are represented and so on. During our workshop we shall walk paticipants through a process, revealing clues according to the general pace and progress. It is noteworthy to say that these clues also provide participants with an understanding of the role reverse engineering plays in figuring out how Trojans encrypt their data. Plus, participants who are proficient in reverse engineering will be able to receive these clues in form of specially crafted IDB files with relevant code, and enjoy extra fun and challenge. Besides the hints, we shall constantly assist participants by discussing progress they have made and parts of the encryption format that still puzzle them. We shall do so by helping them ask the right questions instead of providing them with the answers. We will teach basic methods of binary data analysis, as well as help them quickly overcome technical issues that are not core to the workshop, and that they should not waste time on.

Summary: At this stage, all participants should have made significant progress in cracking the configuration. Some will likely be very close to a complete solution, and perhaps a few will have completed. We shall fully explain the format of encryption, revealing the complete solution, and answer participants. At this point, participants will have cracked the encryption pretty much by themselves, experienced research work and are ready to protect people of the world from this dire threat.

Final closure:
At this stage participants will have time to review their personal progress in view of the full solution, and finish decryption scripts they have been working on, while we are available for questions and assistance.

Making it fun for pros:
As briefly mentioned above, should there be any participants who are experienced with this kind of work, we will provide an IDB file with the code that decrypts the configuration. These will serve as replacement for the regular clues, and provide reverse engineering challenge to the workshop. The IDB will have named functions and other hints, to focus them on the relevant parts and help them not wander around the code too much.

Pavel Asinovsky

Pavel Asinovsky is a malware researcher at IBM Trusteer for the past year and a half. Prior to that Pavel worked as a malware researcher for F5 networks and as a malware analyst at RSA-EMC. Pavel has a wide experience and interest in malware analysis.

Magal Baz

Magal Baz is a malware researcher at IBM Trusteer, and has eight years of experience as a security researcher and team leader in the cyber security industry. Magal has a keen interest in network security, reverse engineering and malware analysis.