Merge pull request #1662 from xwings/dev

refine ARCHITECTURE file

Author: xwings

ARCHITECTURE.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # Arch — CPU architecture layer
@@ -14,9 +14,9 @@ exceptions below). No roadmap milestone applies; maturity-based status.
 
 ## Status
 
-`done` — all ten architectures are exercised by the CI suites; CPU model
-selection is covered by `tests/test_cpu_models.py` (observed:
-`Ran 7 tests … OK`).
+`done` — all ten architectures are exercised by the CI suites; the CPU
+model enums are checked against Unicorn's constants by
+`tests/test_cpu_models.py` (observed: `Ran 7 tests … OK`).
 
 ## Code Structure
 
@@ -52,10 +52,12 @@ Python; root rules apply. Local patterns:
 
 ## Design and Invariants
 
-- `QlArch` creates the `Uc` lazily as a cached property (`qiling/arch/arch.py:34`)
-  and exposes `regs` (`:42`), `stack_push/stack_pop` (`:52`/`:66`),
-  `save/restore` via `UcContext` (`:108`/`:112`), `disassembler` (`:117`),
-  and `assembler` (`:125`). Everything above arch must go through these.
+- `QlArch` declares `uc` as an abstract property that each concrete arch
+  builds lazily as a `cached_property` (`qiling/arch/x86.py:27`); the base
+  (`qiling/arch/arch.py:32-34`) exposes `regs` (`:42`),
+  `stack_push/stack_pop` (`:52`/`:66`), `save/restore` via `UcContext`
+  (`:108`/`:112`), `disassembler` (`:117`), and `assembler` (`:125`).
+  Everything above arch must go through these.
 - `ql.uc` is a proxy to `arch.uc` (`qiling/core.py:479`); there is exactly
   one Unicorn instance per `Qiling` (the multi-Unicorn threading idea in
   `TODO.md:462-488` is a proposal only).
@@ -66,9 +68,10 @@ Python; root rules apply. Local patterns:
   base for the GDT manager's constructor annotation. Do not add further
   upward imports; see [os-baremetal.md](os-baremetal.md) for the multitask
   contract.
-- CPU models are selected by the `cputype` kwarg and validated by
-  `select_arch`; a model belongs to exactly one enum in
-  `qiling/arch/models.py`.
+- CPU models come in through the `cputype` kwarg; `select_arch` forwards
+  the value unvalidated (`qiling/utils.py:377`) and the arch class applies
+  it with `ctl_set_cpu_model` (`qiling/arch/arm.py:47`); a model belongs
+  to exactly one enum in `qiling/arch/models.py`.
 - Endianness and thumb are constructor inputs for ARM/MIPS only
   (`qiling/utils.py:379-386`).
 

ARCHITECTURE/arch.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # CLI — qltool and qltui
@@ -44,8 +44,8 @@ argparse actions mapping lowercase names to `QL_ARCH`/`QL_OS`/`QL_ENDIAN`/
   `--arch`, `--os`, `--endian`, `--thumb`, `--format asm|hex|bin`),
   `examples`, `qltui` (`qiling/cli.py:196-217`); common flags cover
   verbosity, `--env` (pickled dict), `--gdb`, `--qdb`, `--rr`,
-  `--profile`, `--filter`, `--log-file`, `--log-plain`, `--root`,
-  `--debug-stop`, `--multithread`, `--timeout`, `--coverage-file`,
+  `--profile`, `--filter`, `--log-file`, `--log-plain`, `--no-console`,
+  `--root`, `--debug-stop`, `--multithread`, `--timeout`, `--coverage-file`,
   `--coverage-format`, `--json`, `--libcache` (`:225-242`).
 - `handle_run`/`handle_code` return the kwargs dict; `Qiling(**ql_args)`
   at `qiling/cli.py:276` is the single construction point, followed by

ARCHITECTURE/cli.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # Core — the Qiling facade and plumbing
@@ -42,7 +42,7 @@ Python only; root toolchain and style rules apply (see
 patterns to follow:
 
 - Circular-import avoidance with `TYPE_CHECKING` guards and string
-  annotations (`qiling/core.py:1-33`, `qiling/log.py:15-19`).
+  annotations (`qiling/core.py:1-33`, `qiling/log.py:19-20`).
 - Components are reached through read-only properties (`mem`, `arch`,
   `loader`, `os`, `hw`, `log`; `qiling/core.py:203-244`); setters exist only
   for `verbose`, `debugger`, `filter`, `debug_stop`.
@@ -70,12 +70,13 @@ patterns to follow:
   `QL_HOOK_BLOCK` set (`qiling/const.py:77`) stops remaining hooks
   (`qiling/core_hooks.py:186`, `:209`). Address hooks are keyed per address
   (`hook_address`, `:550`). `begin=1, end=0` means "whole address space".
-- **Exceptions raised inside hooks** are captured by the OS layer into
-  `ql.internal_exception` and re-raised after `uc.emu_start` returns
+- **Exceptions raised inside hooks** are captured by the hook wrapper
+  (`qiling/core_hooks.py:141-144`) into `ql._internal_exception`, which
+  `emu_start` resets beforehand and re-raises after `uc.emu_start` returns
   (`qiling/core.py:763`, `:773-774`), because Unicorn cannot propagate
   Python exceptions through its C callbacks.
 - **Emulation state** is tracked in `QL_STATE` (`qiling/const.py:67`) around
-  `emu_start` (`qiling/core.py:768-771`); `QlOs.call` refuses to move `pc`
+  `emu_start` (`qiling/core.py:765-770`); `QlOs.call` refuses to move `pc`
   once stopped to work around a Unicorn bug (`qiling/os/os.py:215-221`).
 - **Stop guard**: when `stop=QL_STOP.*` is requested, a trap page is mapped
   at or above `0x9000000` (`qiling/core.py:525`) and the loader's

ARCHITECTURE/core.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # Debugger — GDB server and Qdb
@@ -67,8 +67,10 @@ Python; root rules apply. Local patterns:
   (`qiling/debugger/gdb/xmlregs.py:56-137`, `qiling/debugger/gdb/gdb.py:488`).
 - **Qdb** installs a breakpoint hook and drives `ql.emu_start` per
   step; branch predictors compute the next pc for `step_over`; `rr` mode
-  snapshots full state per step in `SnapshotManager`
-  (`qiling/debugger/qdb/utils.py:260`) for `do_backward`.
+  takes a full `ql.save()` before and after each stepping command and
+  keeps only the diff as a `SnapshotManager` layer
+  (`qiling/debugger/qdb/utils.py:260`, decorator `:275-297`) for
+  `do_backward`.
 - **State access** goes through `ql.arch.regs`, `ql.mem`, and
   `hook_address`; the one concrete-type import is `QlProcFS`, used to
   serve `/proc/self/maps` to the client (`qiling/debugger/gdb/gdb.py:38`,
@@ -138,5 +140,7 @@ cd tests && python3 test_qdb.py   # pass = "Ran 6 tests … OK", exit 0
 
 - Qdb has no PowerPC support (`qiling/debugger/qdb/arch/`); GDB XML covers
   all ten arches.
-- Record/replay stores full state per step; memory-heavy on long runs.
+- Record/replay keeps one diff layer per step but performs two full
+  `ql.save()` calls per step (`qiling/debugger/qdb/utils.py:273`); slow
+  on long runs.
 - `TODO.md:628-636` lists bare `except:` blocks in `qiling/debugger/qdb/qdb.py`.

ARCHITECTURE/debugger.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # Extensions — optional tooling on top of the core
@@ -17,10 +17,10 @@ roadmap milestone applies; maturity-based status.
 
 ## Status
 
-`done` — the history coverage tracker is tested in CI (observed:
-`Ran 4 tests … OK`); pipes are exercised by many suites; AFL and r2 need
-optional extras (`fuzz`, `RE`) and are untested here; the IDA plugin needs
-IDA Pro and is untested in CI.
+`done` — the history coverage tracker has its own suite, run manually
+rather than by CI (observed: `Ran 4 tests … OK`); pipes are exercised by
+many suites; AFL and r2 need optional extras (`fuzz`, `RE`) and are
+untested here; the IDA plugin needs IDA Pro and is untested in CI.
 
 ## Code Structure
 
@@ -54,15 +54,15 @@ file with IDA-specific conventions; treat it as its own style domain.
 - Everything here consumes only public APIs of [core.md](core.md) (hooks,
   `mem`, `arch.regs`, `save/restore`) and [os-base.md](os-base.md) (heap,
   stdio, `set_syscall`/`set_api`). Nothing in `qiling/os`, `qiling/loader`,
-  or `qiling/arch` may import from here except the documented cases:
-  `qiling/cli.py:22-23` for coverage/report, `qiling/arch/utils.py:94`
-  lazy r2, and the multitask/mcu owners noted above.
+  or `qiling/arch` may import from here except the exceptions listed under
+  "Downward imports only" in the root System Design section.
 - `collect_coverage` activates hooks on enter and dumps on exit even on
   exceptions (`qiling/extensions/coverage/utils.py:48-63`); `qltool
   --coverage-format` lists `factory.formats` (`qiling/cli.py:240`).
 - The heap sanitizer replaces `ql.os.heap` with a compatible object that
-  surrounds chunks with canaries and detects double-free/UAF; it hooks
-  memory access to report faults (`qiling/extensions/sanitizers/heap.py:29`).
+  surrounds chunks with canaries and detects double-free/UAF; `alloc` and
+  `free` install memory hooks on the canaries to report faults
+  (`qiling/extensions/sanitizers/heap.py:97-107`, `:125`).
 - `pipe.py` streams implement the `ql_file` surface expected by the fd
   table so they can be assigned to `ql.os.stdin/stdout/stderr`.
 - AFL: `ql_afl_fuzz` wraps `unicornafl.uc_afl_fuzz` with input placement,
@@ -98,8 +98,9 @@ file with IDA-specific conventions; treat it as its own style domain.
   tenda_ac15,dlink_dir815,rt_n12_b1}` pair AFL with `pipe.py` and
   `set_syscall` ([os-posix.md](os-posix.md)) or MCU mode
   ([os-baremetal.md](os-baremetal.md)).
-- `pipe.py` is used by `tests/test_pe.py`, `test_windows_stdio.py`,
-  `test_riscv.py`, `test_windows_cpp_x86.py`, and `test_kernel_proxy.py`.
+- `pipe.py` is used by `tests/test_elf.py`, `test_pe.py`,
+  `test_windows_stdio.py`, `test_riscv.py`, `test_windows_cpp_x86.py`,
+  `test_windows_cpp_x8664.py`, and `test_kernel_proxy.py`.
 - The IDA plugin offers a front end comparable to [debugger.md](debugger.md).
 
 ## How to Test

ARCHITECTURE/extensions.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # HW — peripheral emulation for bare-metal targets
@@ -17,8 +17,8 @@ No roadmap milestone applies; maturity-based status.
 ## Status
 
 `done` — exercised by `tests/test_mcu.py` (observed: `Ran 18 tests … OK`)
-across GPIO, USART, EXTI, I2C, SPI, DMA, CRC, RTC, timers, and NVIC on
-STM32F103/F407/F411/F429, GD32VF103, and SAM3X8E firmware.
+across GPIO, USART, EXTI, I2C, SPI, DMA, CRC, ADC, watchdog, timers, and
+NVIC on STM32F103/F407/F411/F429, GD32VF103, and SAM3X8E firmware.
 
 ## Code Structure
 
@@ -40,10 +40,10 @@ Python; root rules apply. Local patterns (observed, enforced by
 - A peripheral is a class whose inner `Type(ctypes.Structure)` lists
   registers in datasheet order with offset comments
   (`qiling/hw/char/stm32f4xx_usart.py:12-49`); `__init__(ql, label,
-  **kwargs)` builds `self.instance` with reset values (`:52-58`).
+  **kwargs)` builds `self.instance` with reset values (`:51-58`).
 - Register access overrides `read/write` decorated with
   `@QlPeripheral.monitor()` and falls back to `raw_read/raw_write`
-  (`:60-70`, `qiling/hw/peripheral.py:162-172`).
+  (`:60-70`, `qiling/hw/peripheral.py:157-173`).
 - Class names are looked up by `struct` string through
   `ql_get_module_function('qiling.hw', struct)` (`qiling/hw/hw.py:81`), so
   every peripheral must be exported from `qiling/hw/__init__.py`.
@@ -70,8 +70,10 @@ Python; root rules apply. Local patterns (observed, enforced by
 - **User hooks**: `hook_read/hook_write` with `QL_INTERCEPT` stages
   (`qiling/hw/peripheral.py:34-46`); `watch()` enables verbose access logs.
 - **Snapshots**: `QlHwManager.save/restore` (`qiling/hw/hw.py:165-171`)
-  pickle peripheral instances; `QlPripheralHandler.__getstate__` strips
-  the manager reference (`:24`).
+  serialize each peripheral's ctypes register struct to bytes and back
+  (`qiling/hw/peripheral.py:258-262`); `QlPripheralHandler.__getstate__`
+  (`qiling/hw/hw.py:24`) strips the manager reference so the MMIO handler
+  can be pickled with the memory map.
 - Peripheral fidelity is demand-driven: registers behave as observed
   firmware needs, not per full datasheets (root deviations).
 

ARCHITECTURE/hw.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # Kernel proxy — hybrid syscall forwarding to a real Linux kernel
@@ -48,7 +48,7 @@ through the standard `logging` module, not `ql.log`, because it has no
 
 - **Integration is only `set_syscall`**: `forward_syscall` registers a
   CALL hook named `ql_syscall_<name>`
-  (`qiling/os/posix/kernel_proxy/__init__.py:136-166`, `:211`), so
+  (`qiling/os/posix/kernel_proxy/__init__.py:136-166`, `:210`), so
   user ENTER/EXIT hooks still fire around it and a later user CALL hook
   overrides it (`tests/test_kernel_proxy.py:186-257`).
 - **Two syscall tables**: guest numbers come from the guest arch table and
@@ -116,9 +116,10 @@ through the standard `logging` module, not `ql.log`, because it has no
 cd tests && python3 test_kernel_proxy.py   # Linux host; currently "Ran 21 tests … FAILED (errors=1)"
 ```
 
-- Expected after the Open Gaps fix: `Ran 21 tests … OK`. Every case builds
-  a `Qiling` on `examples/rootfs/x8664_linux/bin/x8664_hello` and invokes
-  the registered hook directly.
+- Expected after the Open Gaps fix: `Ran 21 tests … OK`. All cases except
+  the pure unit tests `test_ipc_roundtrip` and `test_ptr_size_callable`
+  build a `Qiling` on `examples/rootfs/x8664_linux/bin/x8664_hello` and
+  invoke the registered hook directly.
 - Non-Linux hosts skip the whole class (`tests/test_kernel_proxy.py:17`).
 - No end-to-end test runs a guest binary that actually issues a forwarded
   syscall (`TODO.md:251-259` lists the intended validation).
@@ -130,7 +131,7 @@ cd tests && python3 test_kernel_proxy.py   # Linux host; currently "Ran 21 tests
   instead, and add a message type to `ipc.py` if the wire format grows.
 - **Wire-format changes** must update both `ProxyClient` and
   `ProxyServer`, and the format comment at
-  `qiling/os/posix/kernel_proxy/ipc.py:40-52`.
+  `qiling/os/posix/kernel_proxy/ipc.py:40-53`.
 - **Security review** focuses on `_collect_buffers`/`_writeback_buffers`
   sizes (guest-controlled via `PtrOut(size=callable)`) and on which
   syscalls a harness chooses to forward; there is no allow-list.

ARCHITECTURE/kernel-proxy.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # Loader — binary format loaders
@@ -43,9 +43,10 @@ Python; root rules apply. Local patterns:
   segments are mapped through `ql.mem.map` with an `info` label naming the
   image so `QlMemoryManager.get_lib_base` can find it by basename
   (`qiling/os/memory.py:254`).
-- Loaders expose `entry_point`, `exit_point`/OS exit point, `images`, and
-  `skip_exit_check` (`qiling/loader/loader.py:27`); the ELF loader also sets
-  `elf_entry` and `is_driver` (`qiling/loader/elf.py:121`).
+- The base loader owns `images` and `skip_exit_check`
+  (`qiling/loader/loader.py:26-27`); concrete loaders set `entry_point`,
+  and the exit point lives on `ql.os`. The ELF loader also sets
+  `is_driver` (`qiling/loader/elf.py:121`) and `elf_entry` (`:221`).
 
 ## Design and Invariants
 

ARCHITECTURE/loader.md

@@ -1,5 +1,5 @@
 ---
-eatmycode_version: "1.1.0"
+eatmycode_version: "1.2.0"
 ---
 
 # OS bare-metal — MCU firmware and raw blobs
@@ -18,7 +18,9 @@ applies; maturity-based status.
 
 `done` — MCU covered by `tests/test_mcu.py` (observed: `Ran 18 tests … OK`,
 STM32F1/F4, GD32VF103, SAM3X8E firmware); BLOB by
-`test_blob.BlobTest.test_uboot_arm` (`OK`) and `tests/test_edl.py` (`OK`).
+`test_blob.BlobTest.test_uboot_arm` (`OK`). `tests/test_edl.py` is not
+BLOB coverage: its ELF is sniffed as Linux and runs under `QlOsLinux`
+([os-posix.md](os-posix.md)).
 `test_blob.BlobTest.test_blob_raw` errors on the pinned rootfs submodule
 (see Open Gaps).
 
@@ -53,8 +55,10 @@ memory manager.
   installed by the MCU loader (`qiling/loader/mcu.py:140`).
 - **Effective PC**: run loops read `arch.effective_pc` when present to
   keep the thumb bit consistent (`qiling/os/mcu/mcu.py:54-58`).
-- **BLOB**: `QlOsBlob.run` executes `entry_point` → `exit_point` with the
-  standard `emu_start` and nothing else (`qiling/os/blob/blob.py:43`).
+- **BLOB**: `QlOsBlob.run` resolves entry/exit overrides, creates a heap
+  only if the profile `[CODE]` section defines `heap_address`/`heap_size`,
+  then runs `entry_point` → `exit_point` with the standard `emu_start`
+  (`qiling/os/blob/blob.py:43-60`).
 - `ql.hw` exists only when `ql.baremetal` (`qiling/core.py:191`,
   `:357`); BLOB targets needing hardware must use MCU mode.
 
@@ -79,7 +83,8 @@ memory manager.
   `qiling/utils.py:419`).
 - Interrupt entry/exit uses [arch.md](arch.md) `QlArchCORTEX_M`; the
   Cortex-M arch constructs a `MultiTaskUnicorn` instead of a plain `Uc`
-  (`qiling/arch/cortex_m.py:22`), the one upward import in the arch layer.
+  (`qiling/arch/cortex_m.py:78`; the import at `:22` is one of the three
+  documented upward imports in the arch layer, see [arch.md](arch.md)).
 - Fuzzing MCU firmware (`examples/fuzzing/stm32f429/`) combines this mode
   with [extensions.md](extensions.md) AFL support.
 - Snapshots include hardware state when `save(hw=True)`
@@ -93,8 +98,6 @@ cd tests && python3 -m unittest test_blob.BlobTest.test_uboot_arm   # pass = "OK
 
 - MCU (also proves [hw.md](hw.md)): `cd tests && python3 test_mcu.py`
   — pass = `Ran 18 tests … OK`.
-- Qualcomm EDL blob: `cd tests && python3 test_edl.py` — pass =
-  `Ran 1 test … OK`.
 - `python3 test_blob.py` as a whole fails on a clean checkout; see Open Gaps.
 
 ## Review and Refactor Guide