Today, MalwareTech posted a beginner’s RE challenge on Twitter, and I thought it would be fun to go through it and write up how to do it with IDA and Python.

The ZIP file contained two files, README.txt and shellcode1.exe (which is the challenge file).

README.txt contains the basic rules of the challenge, stating that it’s meant to be done statically with IDA and not with a debugger. It also tells us that running the binary will output an MD5 of the flag.

To confirm this, I run the exe file to see what happens. (Please keep in mind that you should never execute code that you don’t trust on your system. I ran the file inside of a VM.)


As we can see, it opens up a message box called “We’ve been compromised!” and includes the hash of the flag, just as the README said it would.

I now open up IDA PRO to take a better look at the binary. As soon as it opens, I look at the function window and see that they are almost all related to MD5, which makes sense given the size of the binary and our knowledge that it utilizes MD5 to hash the flag.


I then take a look at the imports to get a sense of what Windows libraries are being used to increase the binary’s functionality. As you can see below, the binary has the ability to call a MessageBox, allocate space in memory, allocate space on the heap, and get the length of a string.


Then I go back to the start function of the binary and look at the assembly. Taking a look at the MessageBox call at the bottom of the function, I can see that the offset a2bBBb is used in the MD5 function digestString before it is passed onto MessageBox. This tells me that offset a2bBBb is the flag I’m looking for. In order to track it easily, I rename the offset to flag and continue on into the code.


Knowing where the flag is located, I move back up to the top of the function and see that it allocates some space on the heap, adding a pointer to the flag into the space and then adding the length of the flag onto the heap as well.


The next section of code allocates a 13 byte section of memory,  and copies whatever is at unk_404068 into the memory before calling it.


Going to the unk location, I see that IDA just reads it as bytes. To fix this, I press C on my keyboard, and IDA PRO turns it into the following readable assembly. This code takes the location of the flag, which was held inside the heap allocated earlier in the code, and then goes through every byte executing a ROL by 5.


This shellcode is a decoder, which decodes the flag in memory. The decoded flag is then returned to be used with MD5 to print out the message. In order to replicate this, I took a ROL python script I found and tweaked it to fit my purpose.

from __future__ import print_function # PEP 3105

# max bits > 0 == width of the value in bits (e.g., int_16 -> 16)

# Rotate left: 0b1001 --> 0b0011
rol = lambda val, r_bits, max_bits: \
(val << r_bits%max_bits) & (2**max_bits-1) | \
((val & (2**max_bits-1)) >> (max_bits-(r_bits%max_bits)))

# Rotate right: 0b1001 --> 0b1100
ror = lambda val, r_bits, max_bits: \
((val & (2**max_bits-1)) >> r_bits%max_bits) | \
(val << (max_bits-(r_bits%max_bits)) & (2**max_bits-1))

max_bits = 8 # For fun, try 2, 17 or other arbitrary (positive!) values

shellcode = ["Insert encoded Flag hex here"]

final = ""
for i in shellcode:
value = "0x" + i
value = int(value,16)
#print (value)
newval = rol(value, 5, max_bits)
print("0x%08x << 0x05 --> 0x%08x" % (value, newval))
final = final + chr(newval)


With this script ready, I copied out the bytes referenced by the flag location and pasted them into the shellcode variable. All that’s left is to run the script and receive the final flag.