A Reverse Engineering Lab for Introductory Computer Security Students

We describe the result of my senior project completed at Cal Poly during Spring Quarter 2017 under the advisement of Prof. Zachary Peterson. The goal of this project was to develop a set of introductory and scaffolded lab exercises for Cal Poly’s Intro to Computer Security students that introduces concepts of malware analysis and reverse engineering. We use ransomware thematically throughout the lab to provide some fun contex and relevance to the assignment.

Background

To try to identify what malware analysis and reverse engineering learning objectives to include in the lab, I read through Malware Analyst’s Cookbook by Michael Hale Ligh, Steven Adair, Blake Hartstein, and Matthew Richard (2011) and Reversing: Secrets of Reverse Engineering by Eldad Eilam (2005). I also worked through the now infamous CMU Bomb Lab, which greatly influenced the structure of this lab. The CMU Bomb Lab, written by Randal E. Bryant and David R. O’Hallaron, can be found here.

The books largely focus on the use of more advanced tools and techniques, and were deemed inappropraite for students new the domain. However, by learning about and working through the exercises that utilized such tools and techniques, I was able to better determine what sort of activities would serve as a useful introduction to the concepts.

My lab utilizes basic software development tools that students may already be familiar with (e.g. GDB, objdump) to analyze malware, eschewing more advanced tools with steeper learning curves (e.g. IDA Pro, WinDbg, etc.). This decision also allows us to eliminate any complex configuration or expensive licensing necessary for the lab. Additionally, being comfortable with GDB is a good groundwork for being able to effectively use more advanced tools and help develop core software engineering competencies.

I concluded that a good introductory lab to malware analysis and reverse engineering would get students comfortable with looking through disassembled code and able to recognize basic patterns that can be present in real, albeit simplistic, malware. From personal experiencce, once those two objectives had been met, I was able to successfully move forward with learning more advanced techniques. The hope is that after completing this lab, students will have a foundation for furthering their ability to analyze malware, if they are so inclined.

Lab Overview

This lab introduces basic concepts of reverse engineering malware and familiarizes students with looking through disassembled code without reference to the original program. By completing this lab, the goal is for students to:

• become familiar with tools available for malware analysis (the focus in the lab write-up linked below is GDB),
• be able to recognize patterns in disassembled code that may be present in malware, and
• develop basic reverse engineering skills.

The lab is structured as a series of “ransomware attacks”. The premise is that each attack has been run and encrypted some very important files that need to be decrypted. Provided with the lab are executable files and encrypted files, with every attack having one of each (e.g. the first attack has the attack1 executable and an encrypted file attack1_enc.jpg). In order to decrypt the files, the attack executables must be analyzed to determine the input necessary to trigger decryption.

Each attack is an iteration on a common theme: the attack executable contains the method(s) that were used to encrypt the file, and upon inspection of the disassembled code it can be determined what needs to be done in order to decrypt the file. The first two attacks require inspection of the disassembled code in order to find the correct input to provide when the executable is run, while the third attack requires the student to write their own program to decrypt the file based on their analysis of how the file was encrypted.

The deliverables from this lab would be the decrypted files and the ability of the student to describe the flaw in the program that allowed them to decrypt the file.

##Lab Materials The lab write up is avaialbe here

The source for each attack can be found here

There are three C files, one for each attack, attack1.c attack2.c attack3.c. Each can encrypt a file by running it like so: attackX [<input_filename>] [<encrypted_filename]. To prompt for a key to decrypt, run an executable with no arguments.

Also provided is attack3_sample_decrypt.c, which is an example of a program that could be used to decrypt the file encrypted by attack3.

make will compile each attack and use the resulting executable to encrypt the corresponding original file. Each executable and encrypted file will be placed in the lab_files directory.

You can find pre-compiled x86 binaries and victim files here.

Possible Adaptations/Additions

• The file extension has been left on the encrypted files, but it could be removed in order to provide an added layer of difficulty.
• Initially it was planned to include an attack that deals with networking in some way. The idea was to require students to spoof a packet of some sort in order to decrypt their file. This was not included because I ran out of time to figure out the logistics of such an attack, but it could be a good addition to this lab.