Lua Virtualization Part 4: Devirtualizing Luraph
Table of Contents
This is part 4 of a 5 part series about Lua Virtualization:
- Part 1: The Internals of the Lua VM
- Part 2: Obfuscation techniques
- Part 3: Taking a Look at Luraph
- Part 4: Devirtualizing Luraph
- Part 5: Actually Devirtualizing Luraph
Rather than redoing all of the work from scratch, I’ve decided to continue working with the same file for now. We’re still operating on the Hello World script. The float-handling deobfuscation has been fixed, and in this post we’ll document pre-VM execution, partially address the crash from the previous post, and begin digging into the outer VM layer—the Deserialization VM.
Pre VM Execution!
Before diving into the execution flow of the Real VM, it’s important to understand what occurs beforehand. Execution begins in the pre-VM stage, which is responsible for deserializing the raw data blob (the long string starting with LPH}). This data is then passed into the Deserialization VM, which reconstructs the instruction stream for the Real VM while enforcing multiple anti-tamper mechanisms. Some of which remains undocumented.
Finally, execution transitions into the Post-VM stage, responsible for error handling and processing the function’s return value(s).
In short, the execution looks like this:
Below is a visual representation of the Pre-VM deserialization process:
The pre-VM stage begins by normalizing and decompressing the blob, producing a format suitable for deserialization. The process starts with parsing the constants, followed by the function prototypes (embedded functions). Each prototype’s instruction stream is parsed via Get Next Function Instructions, which as the name suggests: extracts the instruction set for the next embedded function or prototype.
Once all prototypes have been processed, the globals used as an interface between these routines are cleared. Finally, the index of the entrypoint is read from the prototype table, marking the function from which execution will begin.
The Deserialized Output
Now that we understand how the pre-VM stage operates, we can examine its output. This stage produces a table containing the following fields:
local Stk_size = EXECUTION_DATA[6]
local line_debugging_data = EXECUTION_DATA[11]
local Insts = EXECUTION_DATA[3]
local REG_B = EXECUTION_DATA[4]
local REG_A = EXECUTION_DATA[5]
local decrypted_constants = EXECUTION_DATA[8]
local constants = EXECUTION_DATA[9]
local function_prototypes = EXECUTION_DATA[7]
local REG_C = EXECUTION_DATA[10]
EXECUTION_DATA[1] and EXECUTION_DATA[2] remain unidentified.
Of particular interest is the constants table. Unfortunately, it is encrypted for consumption by the Real VM, rendering it unusable in its raw form. The decrypted_constants table, however, is exactly what its name implies: it stores the constants after they have been decrypted at runtime. This is an important observation, as it gives us a concrete anchor for locating the constant decryption routine—one that we can eventually lift from Luraph’s custom opcodes back into native Lua.
The most interesting is constants, unfortunately it’s encrypted for the Real VM, making it useless for now. decrypted_constants is exactly what its name implies, it’s where the decrypted constants is stored once decrypted in runtime, this is useful knowledge as it’ll help us locate the decryption function.
The Insts table, withholds all the Instructions relative to the VIP (Virtual IP), same goes for REG_A, REG_B, and REG_C, they’re all relative to the VIP, and withholds all the register indexes.
If we look at how both the Deserialization VM and the Real VM are invoked:
local deserialized_execution_data = h_funcs["VM"](
Deserialize(),
h_funcs["upvalues"]
)(
Deserialize,
Self.nil_self_index,
h_funcs["get_self_index"],
execute_first_arg,
h_funcs["gFloat"],
h_funcs["gBits8"],
h_funcs["gBits32"],
Self.control_flow_tbl_for_first_vm,
nil,
h_funcs["VM"]
)
local ret = unpack({
h_funcs["VM"](deserialized_execution_data, h_funcs["upvalues"])
})
return ret
We’ll notice that the Deserialization VM produces the same type of output of deserialized_execution_data as the Deserialize function. Making it easier for us later to figure out how the Deserialization VM works, while we know the expected output.
We can see that the Deserialization VM produces an output (deserialized_execution_data) that is structurally identical to the output of the Deserialize function itself. This symmetry helps with reversing the Deserialization VM: we already know the exact shape and semantics of the expected output, which acts as a strong reference point.
Resolving the crash
Picking up from the last post—this was the issue we couldn’t solve.
After some testing, it became clear that the crash only occurred when calling print, regardless of the arguments passed. The fact that only print triggered the crash pointed toward the following behavior:
tostring = function(variable)
return "Hijacked"
end
print("Hello World!") -- Will output "Hijacked"
Internally, print invokes tostring on its arguments. By hooking tostring, you implicitly and silently hook print as well. One way to work around this is to use io.write, which performs no formatting and behaves similarly to printf in C.
With that realization, I implemented a custom tostring replacement for VM analysis:
local function custom_tostring(value, indent)
indent = (indent or 1)
local type = type(value)
local str = ""
if type == "nil" then
str = "nil"
elseif type == "boolean" then
str = value and "true" or "false"
elseif type == "number" then
str = string.format("%g", value)
elseif type == "string" then
str = '"' .. value:gsub('"', '\\"') .. '"'
elseif type == "table" then
str = "{\n"
for k, v in pairs(value) do
str = str .. string_rep(" ", indent)
str = str .. "[" .. custom_tostring(k, indent + 1) .. "] = " .. custom_tostring(v, indent + 1) .. "\n"
end
str = str .. string_rep(" ", indent - 1)
str = str .. "}"
elseif type == "function" then
str = get_var_name(value, 3 + indent) -- for base padding
else
str = "<" .. type .. ">"
end
return str
end
This mitigated the crash almost entirely—but not completely.
After some time, I discovered the missing piece: strings have metatables.
debug.setmetatable("", {
__tostring = function()
return "Hijacked"
end
})
print("Hello World!") -- Will output "Hijacked"
Some times goes, and i eventually discover that strings have metatables:
print(setmetatable) -- function: 0x5c9f8f944760
print(debug.setmetatable) -- function: 0x5c9f8f94ab20
Another way to achieve the same result is via direct access to the string metatable:
getmetatable("").__tostring = function ()
return "Hijacked"
end
print("Hello World!") -- Will output "Hijacked"
At the time, this wasn’t immediately useful. Attempts to hook getmetatable, debug.getmetatable, and debug.setmetatable showed nothing in Luraph, but the behavior of this strongly suggested that this mechanism was being abused.
It wasn’t until much later after @beeatomic sent an email describing a POC bypass, we’ve managed to progress.
The Bypass
The bypass itself is trivial: during constant parsing, every constant is checked, and if its value equals "__tostring", it is overwritten.
if IS_FLOAT then
data = h_funcs["gFloat"]()
elseif IS_BOOL then
data = (h_funcs["gBits8"]() == 1)
elseif IS_STRING then
data = h_funcs["gString"]()
else
data = h_funcs["gInt"]()
end;
if data == "__tostring" then data = "" end -- Bypass
I implemented this directly inside the Deserialize function. The reasoning is simple: this is the earliest point at which constants are plaintext, before the Deserialization VM has a chance use them.
This bypass only affects the Deserialization VM, and even then it’s only a partial fix. However, when combined with io.write (to avoid print) and the custom tostring implementation, it is sufficient to analysis and prevent the crash.
Digging into the Deserialization VM
With the bypass in place and knowing that the Deserialization VM produces the same structural output as the pre-VM stage we can begin dissecting the Deserialization VM itself.
Inspecting the Deserialization VM
My approach to inspecting the Deserialization VM was straightforward: add logging to every opcode:
-- OP_MUL
io.write(("[VIP=%d, Enum=%d] Stk[%d] = Stk[%d] * Stk[%d]\n"):format(VIP, Enum, REG_B[VIP], REG_C[VIP], REG_A[VIP]))
Stk[REG_B[VIP]] = Stk[REG_C[VIP]] * Stk[REG_A[VIP]]
This reveals roughly 20,000 instructions executed before the Real VM is invoked. Which itself executes around 5,000 instructions. For a high-level overview and interactive inspection of instruction sequences, I used ZeroBrane Studio which works well as a Lua debugger in this context.
What We Know
The inner workings of OP_CLOSURE
At first glance you might think the following opcode is a return:
-- OP_RETURN
return true, REG_C[VIP], 0
However, this is not a normal return. It is a function dispatcher.
There is no explicit function call here. Instead, the VM signals that execution should continue in another closure by returning metadata, rather than nesting another VM instance. This design avoids the traditional VM-nesting model (IronBrew-style), where the parent VM remains alive while the child VM executes. In LPH’s design, the enclosing VM is stopped before dispatching the next one, ensuring that only one VM exists at any given time, significantly reducing memory usage.
This becomes obvious when we examine what happens immediately after the return. The VM itself is executed inside a pcall, which captures both errors and structured return values:
local success, err_and_is_closure, closure_index, ARGS_FLAG = pcall(function()
-- Handeling of the VM
end)
Afterward, the dispatcher logic handles both errors and continuation:
if not (success) then
-- Handeling of any errors
else
if err_and_is_closure then
local NO_ARGS = 1
if ARGS_FLAG ~= NO_ARGS then
return Stk[closure_index](h_funcs["safe_tbl_unpack"](Stk, Top, closure_index + 1))
else
return Stk[closure_index]()
end
else
if (closure_index) then
return h_funcs["safe_tbl_unpack"](Stk, ARGS_FLAG, closure_index)
end
end
end
Given the return:
-- OP_RETURN
return true, REG_C[VIP], 0
We end up with:
success = true
err_and_is_closure = true
closure_index = REG_C[VIP]
ARGS_FLAG = 0
Since ARGS_FLAG is 0, arguments are present, which results in:
return Stk[closure_index](h_funcs["safe_tbl_unpack"](Stk, Top, closure_index + 1))
In other words, the VM cleanly exits and immediately transfers control to the next closure, passing arguments via the stack. Without ever nesting another VM instance.
Second hidden way of executing closure functions!
There is another, less obvious way closures are invoked.
The function_prototypes table has a metatable that triggers VM execution when accessed. This same trick is used in arithmetic opcode handlers:
-- OP_MOD
Stk[REG_C[VIP]] = Stk[REG_B[VIP]] % Stk[REG_A[VIP]]
If both operands involved in the modulus operation are tables, Lua will invoke the __mod metamethod. LPH exploits this behavior to silently execute code during what appears to be a harmless arithmetic operation, effectively hiding VM dispatch inside metatable callbacks.
Register shuffling
According to standard Lua VM documentation, MUL behaves as follows:
MUL A B C R(A) := RK(B) * RK(C)
However, Luraph intentionally deviates from this convention:
-- OP_MUL
Stk[REG_B[VIP]] = Stk[REG_C[VIP]] * Stk[REG_A[VIP]]
Here, registers A and B are swapped relative to the documented behavior. While this isn’t particularly difficult to deal with, it must be accounted for during opcode lifting.
In Next Blog …
In the next post, we’ll analyze the execution logic we’ve dumped and, hopefully, develop a solid understanding of how Luraph’s Deserialization VM operates. Bringing us one step closer to fully devirtualizing the code.
For those following along, the almost fully reversed Lua file I’m currently working on is available here
Another Bonus
In the last post, I shared a method for recovering function names dynamically. This approach is significantly more robust. It works for all variables, not just functions:
local function find_name(value, level)
level = (level or 2) -- 2 => caller of the function that calls find_name
for i = 1, math.huge do
local name, v = debug.getlocal(level, i)
if not name then break end
if v == value and name ~= "(*temporary)" then return name, "local" end
end
local info = debug.getinfo(level, "f")
if info and info.func then
local func = info.func
for i = 1, math.huge do
local name, v = debug.getupvalue(func, i)
if not name then break end
if v == value then return name, "upvalue" end
end
end
for k, v in pairs(_G) do
if v == value then return k, "global" end
end
for k, v in pairs(string) do
if v == value then return k, "string" end
end
for k, v in pairs(debug) do
if v == value then return k, "debug" end
end
for k, v in pairs(math) do
if v == value then return k, "math" end
end
for k, v in pairs(table) do
if v == value then return k, "table" end
end
return nil
end
local function get_var_name(x)
local name, kind = find_name(x, 3) -- 3 => skip one more stack level if wrapper
if name then
return name, kind
else
return "NO NAME FOUND"
end
end
local foo = "bar"
print(get_var_name(foo)) -- Outputs: "foo local"