← Back to Security Research Portal (ccsimplyspolit.github.io)

Union Type Confusion (CWE-843) & Out-of-Order Deserialization in Embeddable Scripting VMs

0x00 // Tagged Unions in Bytecode & Closure Deserializers

Embeddable scripting virtual machines written in C (such as Lua, Gravity, QuickJS, and Janet) frequently represent function prototypes using a compact tagged union. A single tag field (e.g., EXEC_TYPE_NATIVE, EXEC_TYPE_INTERNAL, EXEC_TYPE_SPECIAL) selects whether the inline union holds:

typedef struct {
    vm_exec_type tag; // Discriminator: NATIVE (0), INTERNAL (1), BRIDGED (2), SPECIAL (3)
    union {
        // Variant A: tag == EXEC_TYPE_NATIVE (0)
        struct {
            struct { size_t n, m; vm_value_t *p; } cpool; // offsets +0, +8, +16
            struct { size_t n, m; vm_value_t *p; } pvalue;
            struct { size_t n, m; vm_value_t *p; } pname;
            uint32_t  ninsts;
            uint32_t *bytecode;
            uint32_t *lineno;
        };
        // Variant B: tag == EXEC_TYPE_INTERNAL (1)
        vm_c_internal internal; // C function pointer at union offset +0

        // Variant C: tag == EXEC_TYPE_SPECIAL (3)
        struct {
            uint16_t index;       // offset +0
            void    *special[2];  // getter/setter closure pointers at offsets +8, +16
        };
    };
} vm_function_t;

0x01 // The Out-of-Order Key Deserialization Hazard

When a VM serializes a function prototype to JSON or a key-value stream, it typically writes "tag" first, followed by variant-specific keys (such as "pool" and "bytecode" for EXEC_TYPE_NATIVE, or "$get" and "$set" for EXEC_TYPE_SPECIAL).

However, a deserializer that iterates over untrusted JSON keys in input order starts with a default discriminator (f->tag = EXEC_TYPE_NATIVE). If an attacker places "pool": [1] before "tag": 3 (EXEC_TYPE_SPECIAL):

Union Memory Layout on x86_64:
+-----------------------+-----------------------+-----------------------+
| Offset +0 (8 bytes)   | Offset +8 (8 bytes)   | Offset +16 (8 bytes)  |
+-----------------------+-----------------------+-----------------------+
| EXEC_TYPE_NATIVE (0):                                                 |
| cpool.n = 0x1         | cpool.m = 0x8         | cpool.p = 0x602000... |
+-----------------------+-----------------------+-----------------------+
| EXEC_TYPE_SPECIAL (3) [after late "tag": 3 overwrite]:                |
| index   = 0x0001      | special[0] = 0x8 (!)  | special[1] = cpool.p  |
+-----------------------+-----------------------+-----------------------+

Because the late "tag": 3 assignment changes the union discriminator without verifying that no EXEC_TYPE_NATIVE fields were already populated, f->special[0] aliases the integer capacity cpool.m (0x8) and f->special[1] aliases the raw vm_value_t * array cpool.p. During garbage-collector marking (vm_gc_mark_function()) or property getter invocation, the VM dereferences 0x8 as a vm_closure_t * pointer.

0x02 // Defensive Invariant Enforcement

To eliminate union type confusion in key-value deserializers:

// 1. Whitelist only serializable union tags (reject EXEC_TYPE_INTERNAL / C function pointers)
if (tag != EXEC_TYPE_NATIVE && tag != EXEC_TYPE_SPECIAL) goto abort_load;

// 2. Lock the active union variant once any variant-specific field has been populated
if (native_fields_used && tag != EXEC_TYPE_NATIVE) goto abort_load;

// 3. Decouple auxiliary debug array lengths (lineno) from executable bytecode length (ninsts)
//    and enforce order-independent post-loop consistency before returning `f`:
uint32_t nlineno = 0;
f->lineno = vm_bytecode_deserialize(ptr, len, &nlineno);
if (!f->lineno) goto abort_load;
lineno_parsed = true;
...
// Post-loop invariant (independent of whether "lineno" or "bytecode" appeared first in JSON):
if (lineno_parsed && (!bytecode_parsed || nlineno < f->ninsts)) goto abort_load;

0x03 // Signed LEB128 Count Casts & Placeholder Reference Cycles in Binary VM Loaders

In binary bytecode loaders that ingest untrusted module blobs, three additional structural checks are required to prevent integer sign-extension wraps and uninitialized reference dereferences:

// 1. Reject unsigned LEB128 values > INT32_MAX when reading signed section counts:
uint32_t raw_count = 0;
if (read_uleb128(reader, &raw_count) < 0 || raw_count > INT32_MAX) {
    return VM_ERR_INVALID_BYTECODE;
}

// 2. Verify remaining bytecode buffer length (bc_len - pos >= instr_len) before
//    performing multi-byte operand endianness or atom table fixups.

// 3. Reject self-referential object table lookups where table[idx] is still a NULL
//    placeholder during recursive container deserialization.