Challenge RE #24

This challenge it's more interesting. This time we have a binary file with the following description

Challenge Description

Here is the simplest possible calculator program, a simplified version of bc UNIX utility:

123+456
(unsigned) dec: 579 hex: 0x243  bin: 1001000011

120/10
(unsigned) dec: 12 hex: 0xC  bin: 1100

-10
(unsigned) dec: 4294967286 hex: 0xFFFFFFF6  bin: 11111111111111111111111111110110
(signed) dec: -10 hex: -0xA  bin: -1010

-10+20
(unsigned) dec: 10 hex: 0xA  bin: 1010

It is known that it supports 4 arithmetic operations and negative numbers can be denoted with a minus before the number. But it's also known that there are at least 7 undocumented features. Try to find them all.

In order to approach this problem, we need to take a look at the assembly code of this binary. For this you have several tools to extract the disassembled code, here is a list of them

1. Ida free
2. Ghidra
3. radare2

For this challenge, I'll just use Ida because is the one I'm more familiar with. The great advantage of these tools is that they even draw you through the flow of the program, and even disassemble the code to C/C++ code. I'm not going to disassemble anything, but I'm not mad for them to draw me some arrow flows, that's sick, good stuff.

Analysis

Exploring the binary functionalities

Let's start our reversing process running the binary, as always don't do this if you are running with malware. Let's run this program

./calc_x64

1
(unsigned) dec: 1 hex: 0x1  bin: 1
1+1
(unsigned) dec: 2 hex: 0x2  bin: 10
1*1
(unsigned) dec: 1 hex: 0x1  bin: 1
1/1
(unsigned) dec: 1 hex: 0x1  bin: 1
1-1
(unsigned) dec: 0 hex: 0x0  bin: 
1**1
Error: unexpected token. Exiting.

As you see in this terminal output I explored the program running basic arithmetic operations, +, -, *, /. I also tried power in the Python style, and we received an error. The output in all cases, except the error one, is in the format (unsigned) dec: <number in decimal representation> hex: <number in hex representation> bin: <number in bin representation>. In all the cases we tried with positive numbers, let's try with negative ones

-1
(unsigned) dec: 4294967295 hex: 0xFFFFFFFF  bin: 11111111111111111111111111111111
(signed) dec: -1 hex: -0x1  bin: -1
-1 * 2
(unsigned) dec: 4294967294 hex: 0xFFFFFFFE  bin: 11111111111111111111111111111110
(signed) dec: -2 hex: -0x2  bin: -10
-1 + 2
(unsigned) dec: 1 hex: 0x1  bin: 1
-1 / 2
(unsigned) dec: 0 hex: 0x0  bin: 
-1 - 2
(unsigned) dec: 4294967293 hex: 0xFFFFFFFD  bin: 11111111111111111111111111111101
(signed) dec: -3 hex: -0x3  bin: -11

In this case, we can see that the calculator can handle signed numbers as well. Ok, everything is as expected. These are 4 arithmetics operations mentioned in the description of the challenge, we are missing 3 more. One thing we can try is using bitwise operators and see what we get.

1 ^ 2
(unsigned) dec: 3 hex: 0x3  bin: 11
1 & 2
Error: bad character. Exiting.

Here we see the calculator supports XOR with ^ operator, but does not support AND with &. We have 5 operations, we are still missing 2 more. I also tried | but we received an error output as well. The error for this case it's quite different, has the following output junk after expression, which might mean that we are just using badly this operator. For now, this exploration it's quite good we will need to look at the assembly code to find out the other two.

Reading the code

Let's start our analysis with the main thread of the program. Right away you can notice that we have a call to sub_400A3B, which we are going to look at a bit later. Followed by printings of a text with the following format, using the result of this subroutine to populate the formatted text. This is the formatted text (unsigned) dec: %u hex: 0x%X bin: <number in binary representation>\n. This matches with our previous exploration of the program. Here is the code snippet for the assembly code

; int __fastcall main(int, char **, char **)
main proc near
; __unwind {
push    rbx
call    sub_400A3B

loc_400696:
call    sub_400AEE ;; ???
;; PREPARING FOR PRINTING RESULT OF sub_400AEE on different formats
;; decimal, and hexadecimal
mov     esi, offset aUnsignedDecUHe ; "(unsigned) dec: %u hex: 0x%X "
mov     ecx, eax ;; result from sub_400AEE
mov     edx, eax
mov     ebx, eax
mov     edi, 1
xor     eax, eax
call    ___printf_chk

;; printing just 'bin: '
mov     esi, offset aBin ; " bin: "
mov     edi, 1
xor     eax, eax
call    ___printf_chk

mov     edi, ebx
call    sub_40082D

Now I personally don't want to read the whole assembly code, I just want to find out these missing operations. One smart approach would be to look in Ida for the signs of operations we already know. For example, let's look for the parsing of character +. In this way, we might identify other symbols as well in the code. The ASCII value for + it's 0x2b, another way to represent this is 2bh which is the way Ida represents the hexadecimal numbers. To make the search you can press Alt+T or go to Search > Text, another way could be that if you previously disassembled the binary to a file you can use grep to search in the file where the disassembled code. When we made that search Ida added a comment with the value '+', this is done in the subroutine sub_400C07. Let's analyze that particular function.

sub_400C07 proc near
; __unwind {
push    rbx
xor     eax, eax
call    sub_400BBB

loc_400C0F:
mov     ebx, eax

loc_400C11:
cmp     cs:dword_602490, 1
mov     al, cs:byte_602090
jnz     short loc_400C69

loc_400C20:
cmp     al, 2Ah ; '*'
jz      short loc_400C43
cmp     al, 2Fh ; '/'
jz      short loc_400C54
cmp     al, 25h ; '%'
jnz     short loc_400C20
call    advance
xor     eax, eax
call    sub_400BBB
mov     esi, eax
mov     eax, ebx
cdq
idiv    esi
mov     ebx, edx
jmp     short loc_400C11

;; ... method continues

Right away in loc_400C20 we can see that apart from * and /, operations that we already know, there's as well %. Let's rerun our calc_x64 to see what this operation gives us, possibly just the remainder between two numbers, but let's check.

5 % 2
(unsigned) dec: 1 hex: 0x1  bin: 1
4 % 2
(unsigned) dec: 0 hex: 0x0  bin: 0

Nice, so this is another supported operation, at this point, we have the +, -, *, /, ^ and %, just one operation it's remaining. Now that we know Ida commented to us these special signs it would be smart on our part to make a smarter search, in this case, let's look for the following regular expression:

; '.{1}'

With this, we are asking Ida to look for comments that are enclosed in '' and have exactly one character.

This search gave us several results, we only need our extra operation in order to solve the puzzle so I'm going to look right for that. Doing so, I spotted the following character ~, so this could be our last operation, which could be the negation.

~ 1
(unsigned) dec: 4294967294 hex: 0xFFFFFFFE  bin: 11111111111111111111111111111110
(signed) dec: -2 hex: -0x2  bin: -10

Yep!! Now we have all the operations, +, -, *, /, ^, % and ~ are our seven arithmetics operations.

Conclusion

I think the lesson to learn here is to focus on your goal try to find what you want and not analyze every single line of code.