Skip to content

Commit d9b4d9d

Browse files
xwingsxwings
authored
Merge pull request #1660 from xwings/dev
renew docs
2 parents d206aed + 8850fd2 commit d9b4d9d

14 files changed

Lines changed: 1823 additions & 470 deletions

File tree

‎.github/workflows/pythonpublish.yml‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,10 +48,15 @@ jobs:
4848
path: ${{ github.workspace }}/dist/*
4949
if-no-files-found: error
5050

51-
publish:
51+
publish-testpypi:
5252
needs: [build]
5353
runs-on: ubuntu-latest
54-
if: startsWith(github.ref, 'refs/tags/')
54+
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
55+
environment:
56+
name: testpypi
57+
url: https://test.pypi.org/p/qiling
58+
permissions:
59+
id-token: write
5560
steps:
5661
- uses: actions/download-artifact@v4
5762
with:
@@ -61,13 +66,23 @@ jobs:
6166
- name: Publish distribution 📦 to test PyPI
6267
uses: pypa/gh-action-pypi-publish@release/v1
6368
with:
64-
user: __token__
65-
password: ${{ secrets.testpypi_pass }}
6669
repository-url: https://test.pypi.org/legacy/
6770
skip-existing: true
6871

72+
publish:
73+
needs: [build, publish-testpypi]
74+
runs-on: ubuntu-latest
75+
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
76+
environment:
77+
name: pypi
78+
url: https://pypi.org/p/qiling
79+
permissions:
80+
id-token: write
81+
steps:
82+
- uses: actions/download-artifact@v4
83+
with:
84+
name: distributions
85+
path: dist
86+
6987
- name: Publish distribution 📦 to PyPI
7088
uses: pypa/gh-action-pypi-publish@release/v1
71-
with:
72-
user: __token__
73-
password: ${{ secrets.pypi_pass }}

‎ARCHITECTURE.md‎

Lines changed: 352 additions & 141 deletions
Large diffs are not rendered by default.

‎ARCHITECTURE/arch.md‎

Lines changed: 118 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,146 @@
1+
---
2+
eatmycode_version: "1.1.0"
3+
---
4+
15
# Arch — CPU architecture layer
26

37
## Goal
48

5-
Own everything CPU-specific: the Unicorn `Uc` instance, register access, stack
6-
primitives, disassembler/assembler, CPU models, and per-arch calling
7-
conventions. This is the bottom layer — everything else reads `ql.arch`; arch
8-
depends only on Unicorn/Capstone/Keystone. Mature released infrastructure;
9-
maturity-based status.
9+
Own everything CPU-specific: the Unicorn `Uc` instance, register access,
10+
stack primitives, disassembler/assembler, CPU models, and per-arch calling
11+
conventions. This is the bottom layer: every other module reads `ql.arch`;
12+
arch depends only on Unicorn/Capstone/Keystone (with the documented
13+
exceptions below). No roadmap milestone applies; maturity-based status.
1014

1115
## Status
1216

13-
`done` — all architectures exercised by the CI suite; CPU model selection
14-
covered by `tests/test_cpu_models.py`.
17+
`done` — all ten architectures are exercised by the CI suites; CPU model
18+
selection is covered by `tests/test_cpu_models.py` (observed:
19+
`Ran 7 tests … OK`).
1520

1621
## Code Structure
1722

1823
| File | Role |
1924
| ---- | ---- |
20-
| `qiling/arch/arch.py` | Abstract base `QlArch`: owns `uc`, `regs`, stack push/pop, save/restore, disassembler |
21-
| `qiling/arch/x86.py` | `QlArchIntel` base + `QlArchA8086`/`QlArchX86`/`QlArchX8664`, GDT/MSR wiring |
22-
| `qiling/arch/arm.py`, `arm64.py` | ARM/AArch64, thumb handling, coprocessor access |
23-
| `qiling/arch/cortex_m.py` | Cortex-M on top of ARM: NVIC-style interrupt entry/exit for MCU mode |
24-
| `qiling/arch/mips.py`, `riscv.py`, `riscv64.py`, `ppc.py` | Remaining architectures |
25-
| `qiling/arch/register.py` | `QlRegisterManager` — attribute-style register read/write |
25+
| `qiling/arch/arch.py` | Abstract base `QlArch`: owns `uc`, `regs`, stack push/pop, save/restore, disassembler/assembler |
26+
| `qiling/arch/x86.py`, `x86_utils.py`, `x86_const.py`, `msr.py` | `QlArchIntel` base + `QlArchA8086`/`QlArchX86`/`QlArchX8664`; GDT/segment setup; MSRs |
27+
| `qiling/arch/arm.py`, `arm_utils.py`, `arm_const.py`, `cpr.py` | ARM: thumb handling, coprocessor registers |
28+
| `qiling/arch/arm64.py`, `arm64_const.py`, `cpr64.py` | AArch64 |
29+
| `qiling/arch/cortex_m.py`, `cortex_m_const.py` | Cortex-M on top of ARM: `QlInterruptContext`, NVIC-style exception entry/exit for MCU mode; uses `MultiTaskUnicorn` |
30+
| `qiling/arch/mips.py`, `riscv.py`, `riscv64.py`, `ppc.py` (+ `*_const.py`) | Remaining architectures |
31+
| `qiling/arch/register.py` | `QlRegisterManager`: attribute-style register read/write |
2632
| `qiling/arch/models.py` | CPU model enums (`X86_CPU_MODEL` … `RISCV64_CPU_MODEL`) |
27-
| `qiling/arch/msr.py`, `cpr.py`, `cpr64.py` | x86 MSRs, ARM/ARM64 coprocessor registers |
28-
| `qiling/arch/utils.py` | `QlArchUtils`: disassembly output for verbose/trace modes |
29-
| `qiling/cc/__init__.py` + `intel.py`, `arm.py`, `mips.py`, `ppc.py`, `riscv.py` | Calling conventions (arg/retval marshalling) consumed by `qiling/os/fcall.py` |
33+
| `qiling/arch/utils.py` | `QlArchUtils` (disassembly output for verbose modes) and the `assembler()` factory |
34+
| `qiling/cc/__init__.py`, `intel.py`, `arm.py`, `mips.py`, `ppc.py`, `riscv.py` | Calling conventions (argument/return marshalling) consumed by `qiling/os/fcall.py` |
35+
36+
## Language and Conventions
37+
38+
Python; root rules apply. Local patterns:
39+
40+
- One class per architecture named `QlArch<ENUMNAME>` so `select_arch`
41+
can derive it (`qiling/utils.py:376-406`).
42+
- Register tables live in `*_const.py` as name→Unicorn-constant maps and are
43+
handed to `QlRegisterManager` (`qiling/arch/register.py:16`).
44+
- Calling conventions are small classes named after the ABI (`cdecl`,
45+
`stdcall`, `ms64`, `amd64`, `macosx64`, `aarch64`, `aarch32`, `mipso32`;
46+
`qiling/cc/intel.py:61-95`, `qiling/cc/arm.py:35-40`,
47+
`qiling/cc/mips.py:9`), all deriving from `QlCommonBaseCC`
48+
(`qiling/cc/__init__.py:110`).
49+
- `TODO.md:638-644` records that GDT/segment validation in
50+
`qiling/arch/x86_utils.py` uses `assert`; treat that as observed, not a
51+
convention to copy.
52+
53+
## Design and Invariants
54+
55+
- `QlArch` creates the `Uc` lazily as a cached property (`qiling/arch/arch.py:34`)
56+
and exposes `regs` (`:42`), `stack_push/stack_pop` (`:52`/`:66`),
57+
`save/restore` via `UcContext` (`:108`/`:112`), `disassembler` (`:117`),
58+
and `assembler` (`:125`). Everything above arch must go through these.
59+
- `ql.uc` is a proxy to `arch.uc` (`qiling/core.py:479`); there is exactly
60+
one Unicorn instance per `Qiling` (the multi-Unicorn threading idea in
61+
`TODO.md:462-488` is a proposal only).
62+
- **Layering exceptions**: `qiling/arch/cortex_m.py:22` imports
63+
`MultiTaskUnicorn` from `qiling/extensions/multitask.py`;
64+
`qiling/arch/utils.py:94` lazily imports the r2 extension for symbol
65+
names; `qiling/arch/x86_utils.py:10` imports `QlMemoryManager` from OS
66+
base for the GDT manager's constructor annotation. Do not add further
67+
upward imports; see [os-baremetal.md](os-baremetal.md) for the multitask
68+
contract.
69+
- CPU models are selected by the `cputype` kwarg and validated by
70+
`select_arch`; a model belongs to exactly one enum in
71+
`qiling/arch/models.py`.
72+
- Endianness and thumb are constructor inputs for ARM/MIPS only
73+
(`qiling/utils.py:379-386`).
3074

3175
## Key Types and Entry Points
3276

33-
- `qiling/arch/arch.py:22` - `QlArch(ABC)` - cached properties `uc` (`:34`), `regs` (`:42`), `stack_push/stack_pop` (`:52`/`:66`), `save/restore` via UcContext (`:108`/`:112`), `disassembler` (`:117`).
34-
- `qiling/arch/register.py:11` - `QlRegisterManager` - `ql.arch.regs.rax`-style access, backed by per-arch `*_const.py` tables.
35-
- `qiling/arch/x86.py:22,53,79,111` - `QlArchIntel` / `QlArchA8086` / `QlArchX86` / `QlArchX8664`.
36-
- `qiling/arch/cortex_m.py:67` - `QlArchCORTEX_M(QlArchARM)` - plus `QlInterruptContext` (`:25`) for exception entry/exit in MCU mode.
37-
- `qiling/arch/models.py` - CPU model enums selected via the `cputype` kwarg (resolved in `select_arch`, `qiling/utils.py:376`).
38-
- `qiling/cc/__init__.py:9` - `QlCC` - abstract calling convention; `QlCommonBaseCC` (`:110`); e.g. `qiling/cc/intel.py` defines `cdecl`/`stdcall`/`ms64`/`macosx64`.
77+
- `qiling/arch/arch.py:22` - `QlArch(ABC)` - base class; see properties
78+
above.
79+
- `qiling/arch/register.py:11` - `QlRegisterManager` - `ql.arch.regs.rax`
80+
style access (`__getattr__` `:35`, `__setattr__` `:44`), plus
81+
`read/write` by name or Unicorn id (`:53`/`:62`).
82+
- `qiling/arch/x86.py:22,53,79,111` - `QlArchIntel` / `QlArchA8086` /
83+
`QlArchX86` / `QlArchX8664`.
84+
- `qiling/arch/cortex_m.py:67` - `QlArchCORTEX_M(QlArchARM)` -
85+
`interrupt_handler` (`:146`) consults `ql.hw.nvic` and enters the handler
86+
inside `QlInterruptContext` (`:25`).
87+
- `qiling/arch/utils.py:106` - `assembler(arch, endianness, is_thumb)` -
88+
Keystone factory used by `qltool code --format asm`.
89+
- `qiling/cc/__init__.py:9` - `QlCC` - abstract calling convention
90+
(`getRawParam`, `setReturnValue`, …); `QlCommonBaseCC` (`:110`).
91+
- `qiling/utils.py:376` - `select_arch(archtype, cputype, endian, thumb)` -
92+
the only construction path (`qiling/core.py:154`).
3993

4094
## Interactions
4195

42-
- Instantiated first by [core.md](core.md) (`qiling/core.py:154`); `Qiling.uc` proxies `arch.uc` (`qiling/core.py:479`).
43-
- [loader.md](loader.md) and the OS layers use `arch.regs` and stack primitives to set up entry state.
44-
- `qiling/cc/` is consumed by `QlFunctionCall` in [os-base.md](os-base.md) for API argument marshalling.
45-
- [debugger.md](debugger.md) reads/writes registers through this layer.
96+
- Constructed first by [core.md](core.md); `QlCoreStructs`/`QlCoreHooks`
97+
are initialized from `arch.endian`/`arch.bits`/`arch.uc`
98+
(`qiling/core.py:157-158`).
99+
- [loader.md](loader.md) and the OS layers set initial register/stack state
100+
through `arch.regs` and the stack primitives.
101+
- `qiling/cc/` is consumed by `QlFunctionCall` ([os-base.md](os-base.md))
102+
and by the Windows fcall selector (`qiling/os/windows/windows.py:41-65`).
103+
- POSIX syscall ABIs are a separate table in
104+
`qiling/os/posix/syscall/abi/` ([os-posix.md](os-posix.md)), not here.
105+
- [debugger.md](debugger.md) reads/writes registers through this layer and
106+
ships per-arch GDB target XML.
107+
- [hw.md](hw.md) NVIC peripherals call `arch.interrupt_handler`
108+
(`qiling/hw/intc/cm_nvic.py:55`, `:127`).
46109

47110
## How to Test
48111

49112
```sh
50-
cd tests && python3 test_cpu_models.py # pass = unittest "OK", exit code 0
113+
cd tests && python3 test_cpu_models.py # pass = "Ran 7 tests … OK", exit 0
51114
```
52115

53-
- Broader arch coverage comes for free from `test_shellcode.py` (5 archs) and the per-OS suites.
116+
- Broader coverage comes from `tests/test_shellcode.py` (five archs) and the
117+
per-OS suites; RISC-V has `tests/test_riscv.py` (`Ran 4 tests … OK`).
118+
- There is no unit test for `qiling/cc/`; it is proven through Windows API
119+
calls (Windows host) and UEFI (`tests/test_uefi.py`).
120+
121+
## Review and Refactor Guide
122+
123+
- **New architecture**: add `qiling/arch/<name>.py` with `QlArch<ENUM>`,
124+
a `*_const.py` register table, a `qiling/cc/` convention, a
125+
`qiling/os/posix/syscall/abi/` ABI, a GDB XML directory, and the enum in
126+
`qiling/const.py:15`; then extend `select_arch`.
127+
- **Register changes** affect `qiling/debugger/gdb/xml/<arch>/` and
128+
`qiling/debugger/qdb/arch/`; run `tests/test_debugger.py` and
129+
`tests/test_qdb.py`.
130+
- **Do not** put OS-specific state (segment selectors for Windows, TLS) in
131+
arch classes; the OS layer owns that (`qiling/os/windows/windows.py:132`).
132+
- Improvement candidate (proposal): move the thumb-bit fixup out of
133+
`Qiling.emu_start` (`qiling/core.py:759`) into `QlArchARM` once Unicorn
134+
reflects thumb mode in `cpsr` at init. Success check:
135+
`tests/test_shellcode.py::test_linux_arm_thumb` and `tests/test_qdb.py`
136+
stay green.
54137

55138
## Open Gaps / Roadmap
56139

57-
- PowerPC and RISC-V have fewer OS-level tests than x86/ARM/MIPS (no dedicated POSIX suite beyond `tests/test_riscv.py`).
58-
- Thumb state handling has a known fixup in `Qiling.emu_start` (`qiling/core.py:743`) rather than in the arch layer itself.
140+
- PowerPC has no OS-level test suite beyond CPU model selection; RISC-V has
141+
four Linux tests.
142+
- 8086 GDB stop replies use the wrong register names (FIXME at
143+
`qiling/debugger/gdb/gdb.py:242`).
144+
- `TODO.md:655-657` notes the 32-bit GDT data segment is built with
145+
`QL_X86_A_PRIV_0` (`qiling/arch/x86_utils.py:167`); the 64-bit manager
146+
already uses ring 3 (`:200`, `:215`).

‎ARCHITECTURE/cli.md‎

Lines changed: 82 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,55 +1,113 @@
1+
---
2+
eatmycode_version: "1.1.0"
3+
---
4+
15
# CLI — qltool and qltui
26

37
## Goal
48

59
Give users a no-code way to run emulations: `qltool run` executes a binary
610
against a rootfs, `qltool code` runs shellcode (hex/asm/bin), `qltool
711
examples` prints usage samples, and `qltool qltui` launches an interactive
8-
TUI that gathers the same options. Mature released infrastructure;
9-
maturity-based status.
12+
TUI that gathers the same options. Owns argument parsing and the mapping
13+
from flags to `Qiling` kwargs; it must not implement emulation behavior.
14+
No roadmap milestone applies; maturity-based status.
1015

1116
## Status
1217

13-
`done` — covered by `tests/test_qltool.py`, which shells out to `qltool` for
14-
run/code subcommands and coverage output. `InstalledQltool_Test` checks the
15-
installed command, shellcode exit status, bundled profiles, and TUI import
16-
outside the checkout.
18+
`done` — `tests/test_qltool.py` (observed: `Ran 8 tests … OK` once the
19+
package is installed) shells out to `qltool` for run/code subcommands,
20+
coverage, JSON, and log filtering; `InstalledQltool_Test` checks the
21+
installed console script, shellcode exit status, bundled profiles, and TUI
22+
import outside the checkout.
1723

1824
## Code Structure
1925

2026
| File | Role |
2127
| ---- | ---- |
22-
| `qltool` | Checkout launcher for `qiling.cli.run` |
23-
| `qiling/cli.py` | Argparse CLI; installed as `qltool`, builds kwargs and drives `Qiling` |
24-
| `qltui.py` | questionary/pyfx/termcolor TUI; collects options, returned to qltool |
28+
| `qltool` | Checkout launcher: `from qiling.cli import run` |
29+
| `qiling/cli.py` | Argparse CLI; installed as the `qltool` console script (`pyproject.toml:33-34`); builds kwargs and drives `Qiling` |
30+
| `qltui.py` | questionary/pyfx/termcolor TUI; collects options and returns them to `qltool` |
31+
32+
## Language and Conventions
33+
34+
Python; root rules apply. Enum-valued flags use `__make_enum_arg`
35+
argparse actions mapping lowercase names to `QL_ARCH`/`QL_OS`/`QL_ENDIAN`/
36+
`QL_VERBOSE` (`qiling/cli.py:59-75`). Errors surface as argparse errors or
37+
`Qiling` exceptions; the process exits with `ql.os.exit_code`
38+
(`qiling/cli.py:321`). `qltui.py` is the only module that imports
39+
`questionary`, `pyfx`, and `termcolor` (`pyproject.toml:48-50`).
40+
41+
## Design and Invariants
42+
43+
- Subcommands: `run` (`-f`, `--rootfs`, `--args …`), `code` (`-f`/`-i`,
44+
`--arch`, `--os`, `--endian`, `--thumb`, `--format asm|hex|bin`),
45+
`examples`, `qltui` (`qiling/cli.py:196-217`); common flags cover
46+
verbosity, `--env` (pickled dict), `--gdb`, `--qdb`, `--rr`,
47+
`--profile`, `--filter`, `--log-file`, `--log-plain`, `--root`,
48+
`--debug-stop`, `--multithread`, `--timeout`, `--coverage-file`,
49+
`--coverage-format`, `--json`, `--libcache` (`:225-242`).
50+
- `handle_run`/`handle_code` return the kwargs dict; `Qiling(**ql_args)`
51+
at `qiling/cli.py:276` is the single construction point, followed by
52+
optional Qdb (`:279`), gdbserver (`:285`), coverage-wrapped `ql.run()`
53+
(`:306-310`), JSON report (`:312`), and exit (`:321`).
54+
- `code --format asm` assembles with Keystone via
55+
`qiling.arch.utils.assembler` (`:104`).
56+
- The installed script and the checkout launcher must behave identically;
57+
`InstalledQltool_Test` runs with an empty `PYTHONPATH` from a temp
58+
directory to prove profiles ship in the wheel (`tests/test_qltool.py:55-72`).
2559

2660
## Key Types and Entry Points
2761

28-
- `qiling/cli.py:189` - `run()` - argparse setup with subcommands `run`, `code`, `examples`, `qltui`; enum-mapping actions translate `--arch/--os/--endian/--verbose` strings to `QL_ARCH`/`QL_OS` enums (`qiling/cli.py:59-75`).
29-
- `qiling/cli.py:129` - `handle_run(options)` - builds `{'argv': [file]+args, 'rootfs': ...}`.
30-
- `qiling/cli.py:78` - `handle_code(options)` - reads hex/asm/bin shellcode, assembling asm via `qiling.arch.utils.assembler` (`qiling/cli.py:104`).
31-
- `qiling/cli.py:276` - `ql = Qiling(**ql_args)` - the single construction point; then optional Qdb (`:279`), gdbserver (`:285`), coverage-wrapped `ql.run()` (`:306-310`), JSON report (`:312`), exit with `ql.os.exit_code` (`:321`).
62+
- `qiling/cli.py:189` - `run()` - argparse setup and dispatch.
63+
- `qiling/cli.py:129` - `handle_run(options)` - builds
64+
`{'argv': [file]+args, 'rootfs': …}`.
65+
- `qiling/cli.py:78` - `handle_code(options)` - reads hex/asm/bin shellcode.
66+
- `qiling/cli.py:59` - `__make_enum_arg(enum_rmap, aliases)` - argparse
67+
action factory.
68+
- `qltui.py` - TUI entry invoked by the `qltui` subcommand.
3269

3370
## Interactions
3471

3572
- Thin client of [core.md](core.md): constructs `Qiling` and calls `run()`.
36-
- Attaches [debugger.md](debugger.md) via `--gdb host:port` / `--qdb [--rr]`.
37-
- Uses [extensions.md](extensions.md) for `--coverage-file` (drcov) and `--json` report output.
38-
- `qltool examples` mirrors scripts documented in `examples/README.md`.
73+
- Attaches [debugger.md](debugger.md) via `--gdb [HOST:PORT]` /
74+
`--qdb [--rr]`.
75+
- Uses [extensions.md](extensions.md) for coverage (`cov_utils.factory`)
76+
and the JSON report.
77+
- Packaging: `pyproject.toml:33-34` defines the console script; the PyPI
78+
workflow runs `InstalledQltool_Test` against the built wheel
79+
(`.github/workflows/pythonpublish.yml:41-44`).
3980

4081
## How to Test
4182

4283
```sh
43-
python3 -m pip install .
44-
cd tests && python3 test_qltool.py # pass = unittest "OK", exit 0
84+
python3 -m pip install -e .
85+
cd tests && python3 test_qltool.py # pass = "Ran 8 tests … OK", exit 0
4586
```
4687

47-
- Packaging regression checks, from the repository root after installing
48-
the built wheel: `python3 -I tests/test_qltool.py InstalledQltool_Test -v`.
49-
- Manual smoke test: `./qltool run -f examples/rootfs/x8664_linux/bin/x8664_hello --rootfs examples/rootfs/x8664_linux` — pass = prints `Hello, World!`.
88+
- `Qltool_Test` uses the checkout launcher `../qltool`;
89+
`InstalledQltool_Test` needs the console script on the interpreter's
90+
scripts path (hence the install step). Without it the two installed
91+
cases error with `FileNotFoundError: …/bin/qltool`.
92+
- Manual smoke test from the repository root (pass = prints
93+
`Hello, World!`):
94+
95+
```sh
96+
./qltool run -f examples/rootfs/x8664_linux/bin/x8664_hello --rootfs examples/rootfs/x8664_linux
97+
```
98+
99+
## Review and Refactor Guide
100+
101+
- **New flag**: add it to the common or subcommand parser, thread it into
102+
`ql_args` or the post-construction block, mirror it in `qltui.py`, and
103+
add a `Qltool_Test` case.
104+
- **Do not** add emulation logic here; new behavior belongs to the owning
105+
module and is only surfaced by a flag.
106+
- Keep `qltool` (checkout) a two-line launcher so installed and checkout
107+
behavior cannot diverge.
50108

51109
## Open Gaps / Roadmap
52110

53-
- `qltui.py` has an installed import smoke test; its interactive flows are
54-
not covered by automated tests.
55-
- `qltool` predates subcommand-style config files; complex setups (fs mappers, custom hooks) still require the Python API.
111+
- `qltui.py` has only an import smoke test; its interactive flows are not
112+
covered.
113+
- Complex setups (fs mappers, custom hooks) still require the Python API.

0 commit comments

Comments
 (0)