[WIP] XMB (PS3 UI) Support

[elad335]

In this pull request I implement and reverse-engineer the operating-system interfaces needed for making XMB (the PS3 User Interface) be able to launch games.

What already works:

• Launching disc games: you need to mount the disc image using the PRCS3 UI:
  In order to mount the disc directory you need to use "Insert Disc" button while the XMB runs. Point it to to the directory you normally laucnh the game from.

• Launching trial/demo games.

• PSN Games wth license: to be supported, you need to dump a few files from your console:

1. ACT.DAT file - located at /dev_hdd0/home/[[USER ID]/exdata/act.dat
   Place it in the user-equivalent path in RPCS3.
2. RIF file, if it exists, in the same placement as RAP files.
3. You need to get the console IDPS and set it on RPCS3.

This is a big step in supporting fully accurate PS3 emulation. XMB is not only an interface, it also communicates with the game code. You see, PS3 was made in the modern consoles era, many security features have been placed so game code depend a lot on the hosting environment for obtaining and saving a lot of information. This is being done to protect the PS3 files frrom malicious behaviour. This is slower than obtaining the information dirrectly by the game process and induces dependencies.
Previously, RPCS3 skipped these operations and manipulated information in HLE, breaking dependecy chains and CPU thread scheduler activity load. This reduced compatibility of many games and is likely the cause of many instabilities today. RPCS3 had implemented many, many hacks around in the past for game compatibility.
One of which dependencies is known to flix #10954 for example, this dependency is currently being brought to HLE by @Yahfz.
We plan to move away frrom hacks, and improve game compatibility in the most accurate way. By allowing games to communicate with XMB.

Remaining tasks:

• making lv2_process object compatible with savestates (need to be integrated with ppu_load_self function, similar to how lv2_prx and lv2_overlay are done)
• Integrating UI to disc insert/eject buttons directly to disc games.
• Placing SPU cache of each process in a unique file, instead of being global. I have done a similar workaround for PPU cache already.
• Reversing why PS button press during ingame is not detected by VSH.
• RSX GPU cache surfaces compatibity & optimization. Currently the RSX interface does not support process memory distinguation which can cause cache collitions. Affecting performance and grahics. (cc @kd-11)
• checking sys_storage_async_send_device_command values against realhw with disc being inserted and ejected. Currently the values there are there to please VSH from the disassembly.
• Support laucnhing licensed PSN games with RAP only. (a clever hack is likely needed)
• Support game termiantion and relaunching with VSH.
• Fixing audio. In addition to supporting LLE cellAudio for the first time.
• Fixing closing games.
• Fixing games with exitspawn support.

2026-08-31_22-59-54.mp4

[AhayriSG]

Is there a command-line argument to boot the VSH/XMB and insert/mount a specified disc image at the same time?

right now, I'm just using this batch script to launch the XMB:
start "" "rpcs3.exe" --config "%~dp0portable\config\custom_configs\config_vsh.yml" "%~dp0portable\dev_flash\vsh\module\vsh.self"

it would be fantastic if it supports a cleaner, more direct syntax such as:
rpcs3.exe -vsh -mount="path/game.iso"

Something like Dolphin does with its Wii system menu : dolphin.exe -b -n 0000000100000002 -C Dolphin.Core.DefaultISO="path/game.iso"

This would be super helpful for frontend users and others who rely on custom batch scripts for automation etc.

[kd-11]

I have to ask - what is going on here?

[elad335]

GCM_context is now a typedef of lv2_rsx_context and not a type by itself, somehow that makes the function definition compile only in global namespace. I didn't change spacing to avoid altering git history.

[kd-11]

So it's a compiler issue workaround? Or a C++ standards thing?

[elad335]

Now that I think about it makes sense. rsx::GCM_context is a type, the namespace the function was declared is the global namespace (lv2_rsx_context struct) so that is where it should be defined.

[kd-11]

The core of the RSX changes (getting rid of rsx::method_registers abuse) is correct. Obviously it shouldn't be accessing REGS(0) but that will be covered at a later time.
I have been splitting up the RSX emulation in the background very slowly according to the general plan in #15424 but this work doesn't conflict with that effort yet.