Skip to content

Commit c80f8d3

Browse files
jcfrjcfr
committed
DOC: Update README and DeveloperGuide to document OpenXR backend support
In the README: - Revise the top-level and Setup sections to reflect OpenXR support. - Update existing headset setup instructions to cover both OpenXR and OpenVR setups, with OpenXR being enabled by default. Added a note at the end of the section explaining how to enable OpenVR support. - Add new sections for "Hololens 2" and "Meta Quest" headsets - Add entries for "Supported headsets" and "Supported XR modality" to each headset section - Add "How to use hand interaction" section - Update the Contributors section to acknowledge @LucasGandel and @sankhesh. In the DeveloperGuide: - Remove obsolete build instructions. - Update Python snippet. - Add references to the documentation of associated VTK modules. - Add the "Mapping of Controller Action to VTK event" section.
1 parent d3bdb81 commit c80f8d3

File tree

2 files changed

+196
-43
lines changed

2 files changed

+196
-43
lines changed

Diff for: ‎DeveloperGuide.md

+82-24
Original file line numberDiff line numberDiff line change
@@ -4,20 +4,79 @@ This Slicer extension is in active development. The API may change from version
44

55
## Build instructions
66

7-
- Build the extension against the newly built Slicer with Qt5 and VTK9 enabled.
8-
- To start Slicer from a build tree and ensure the extension is properly loaded, considering using the ``--launcher-additional-settings`` option:
7+
- Build the extension against the newly built Slicer.
8+
- To start Slicer from a build tree and ensure the extension is properly loaded, consider running the `SlicerWithVirtualReality` launcher. For more details, see [here](https://slicer.readthedocs.io/en/latest/developer_guide/extensions.html#run-slicer-with-your-custom-modules).
99

10-
```
11-
./Slicer.exe --launcher-additional-settings C:\path\to\SlicerVirtualReality-build\inner-build\AdditionalLauncherSettings.ini --additional-module-paths C:\path\to\SlicerVirtualReality-build\inner-build\
12-
```
10+
## Mapping of Controller Action to VTK event
1311

14-
Alternatively, to avoid the need to specify additional command-line parameters, virtual reality extension can be set up in Slicer by:
15-
- Copy these files to (SlicerVirtualReality-binary)\inner-build\lib\Slicer-4.9\qt-loadable-modules\Release:
16-
- (SlicerVirtualReality-binary)\bin\Release\vtkRenderingOpenVR-9.0.dll
17-
- (SlicerVirtualReality-binary)\OpenVR\bin\win64\openvr_api.dll
18-
- Add (SlicerVirtualReality-binary)\inner-build\lib\Slicer-4.9\qt-loadable-modules\Release folder to additional module paths in Slicer application settings.
12+
The mapping process consists of two main steps:
1913

20-
Note that specifying the top-level build directory of the extension ensures that Slicer find all types of modules.
14+
1. Parsing the `vtk_open<vr|xr>_actions.json` action manifest file to link controller-specific interaction paths with generic event paths. This file references controller-specific binding files, usually named `vtk_open<vr|xr>_binding_<vendor_name>.json`, where each controller interaction path is associated with a VTK-specific event path.
15+
16+
2. Assigning a VTK event path to either a VTK event or a `std::function`. This association of a VTK event path involving a single controller with a VTK event is carried out in `vtkOpen<VR|XR>InteractorStyle::SetupActions()`.
17+
18+
### Action Manifest File
19+
20+
The controller interaction paths are specific to each backend:
21+
22+
* For OpenVR: Refer to the [List of common controller types](https://github.com/ValveSoftware/openvr/wiki/List-of-common-controller-types) and the [SteamVR Input Guide](https://github.com/OpenVR-Advanced-Settings/OpenVR-AdvancedSettings/blob/master/docs/SteamVRInputGuide.md).
23+
* For OpenXR: Refer to the [Reserved Paths](https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#semantic-path-reserved) and the [Interaction Profile Paths](https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#semantic-path-interaction-profiles).
24+
25+
As of [Slicer@c7fe8657c](https://github.com/Slicer/Slicer/commit/c7fe8657c6a4bc0666685349b3222ff3c1b4fa02), the provided `vtk_open<vr|xr>_actions.json` and `vtk_open<vr|xr>_binding_<vendor_name>.json` files in the `vtkRenderingOpenVR` and `vtkRenderingOpenXR` VTK modules are as follow:
26+
27+
| | OpenVR | OpenXR |
28+
| ------------------------------- | ------------------------------------------------ | -------------------------------------------------------- |
29+
| Action manifest | [url][vtk_openvr_actions_json_url] | [url][vtk_openxr_actions_json_url] |
30+
| - HP Motion Controller | [url][vtk_openvr_binding_hpmotioncontroller_url] | [url][vtk_openxr_binding_hp_mixed_reality_url] |
31+
| - HTC Vive Controller | [url][vtk_openvr_binding_vive_controller_url] | [url][vtk_openxr_binding_htc_vive_controller_url] |
32+
| - Microsoft Hand Interaction | | [url][vtk_openxr_binding_microsoft_hand_interaction_url] |
33+
| - Oculus Touch | [url][vtk_openvr_binding_oculus_touch_url] | [url][vtk_openxr_binding_oculus_touch_url] |
34+
| - Valve Knuckles | [url][vtk_openvr_binding_knuckles_url] | [url][vtk_openxr_binding_knuckles_url] |
35+
| - Khronos Simple Controller[^1] | | [url][vtk_openxr_binding_khr_simple_url] |
36+
37+
[^1]: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#_khronos_simple_controller_profile
38+
39+
These files serve as essential references for mapping controller actions to VTK events.
40+
41+
<!-- vtkRenderingOpenVR -->
42+
[vtk_openvr_actions_json_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenVR/vtk_openvr_actions.json
43+
[vtk_openvr_binding_hpmotioncontroller_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenVR/vtk_openvr_binding_hpmotioncontroller.json
44+
[vtk_openvr_binding_vive_controller_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenVR/vtk_openvr_binding_vive_controller.json
45+
[vtk_openvr_binding_knuckles_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenVR/vtk_openvr_binding_knuckles.json
46+
[vtk_openvr_binding_oculus_touch_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenVR/vtk_openvr_binding_oculus_touch.json
47+
48+
<!-- vtkRenderingOpenXR -->
49+
[vtk_openxr_actions_json_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_actions.json
50+
[vtk_openxr_binding_hp_mixed_reality_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_binding_hp_mixed_reality.json
51+
[vtk_openxr_binding_htc_vive_controller_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_binding_htc_vive_controller.json
52+
[vtk_openxr_binding_microsoft_hand_interaction_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_binding_microsoft_hand_interaction.json
53+
[vtk_openxr_binding_oculus_touch_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_binding_oculus_touch_controller.json
54+
[vtk_openxr_binding_knuckles_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_binding_knuckles.json
55+
[vtk_openxr_binding_khr_simple_url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtk_openxr_binding_khr_simple_controller.json
56+
57+
### Mapping of VTK event path
58+
59+
The association of VTK event paths to VTK events hardcoded in each VTK modules is as follow:
60+
61+
* For OpenVR, refer to [vtkOpenVRInteractorStyle::SetupActions()][vtkOpenVRInteractorStyle-url]
62+
* For OpenXR, refer to [vtkOpenXRInteractorStyle::SetupActions()][vtkOpenXRInteractorStyle-url]
63+
64+
[vtkOpenVRInteractorStyle-url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtkOpenVRInteractorStyle.cxx
65+
[vtkOpenXRInteractorStyle-url]: https://github.com/Slicer/VTK/blob/slicer-v9.2.20230607-1ff325c54-2/Rendering/OpenXR/vtkOpenXRInteractorStyle.cxx
66+
67+
### Complex Gesture Support
68+
69+
Recognition of complex gesture events commences when the two controller buttons mapped to the ComplexGesture action are pressed.
70+
71+
The SlicerVirtualReality implements its own heuristic by specializing the `HandleComplexGestureEvents()` and `RecognizeComplexGesture()` in the [vtkVirtualRealityComplexGestureRecognizer][vtkVirtualRealityComplexGestureRecognizer-url] class.
72+
73+
[vtkVirtualRealityComplexGestureRecognizer-url]: https://github.com/KitwareMedical/SlicerVirtualReality/blob/master/VirtualReality/MRMLDM/vtkVirtualRealityComplexGestureRecognizer.cxx
74+
75+
Limitations:
76+
77+
* The selected controller buttons are exclusively mapped to the ComplexGesture action and cannot be associated with a regular action.
78+
79+
* To workaround an OpenVR specific [limitation](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/10778), each button expected to be involved in the complex gesture needs to be respectively associated with `/actions/vtk/in/ComplexGestureAction` and `/actions/vtk/in/ComplexGestureAction_Event2`.
2180

2281
## Useful Python Snippets
2382

@@ -28,20 +87,14 @@ Activate virtual reality view:
2887
import logging
2988
import slicer
3089

31-
def isVRInitialized():
32-
"""Determine if VR has been initialized
33-
"""
90+
def isXRBackendInitialized():
91+
"""Determine if XR backend has been initialized."""
3492
vrLogic = slicer.modules.virtualreality.logic()
35-
if (vrLogic is None
36-
or vrLogic.GetVirtualRealityViewNode() is None
37-
or not vrLogic.GetVirtualRealityViewNode().GetVisibility()
38-
or not vrLogic.GetVirtualRealityViewNode().GetActive()):
39-
return False
40-
return True
93+
return vrLogic.GetVirtualRealityActive() if vrLogic else False
4194

4295
def vrCamera():
4396
# Get VR module widget
44-
if not isVRInitialized():
97+
if not isXRBackendInitialized():
4598
return None
4699
# Get VR camera
47100
vrViewWidget = slicer.modules.virtualreality.viewWidget()
@@ -54,16 +107,15 @@ def vrCamera():
54107
return rendererCollection.GetItemAsObject(0).GetActiveCamera()
55108

56109

57-
assert isVRInitialized() is False
110+
assert isXRBackendInitialized() is False
58111
assert vrCamera() is None
59112

60113
vrLogic = slicer.modules.virtualreality.logic()
61114
vrLogic.SetVirtualRealityActive(True)
62115

63-
assert isVRInitialized() is True
116+
assert isXRBackendInitialized() is True
64117
assert vrCamera() is not None
65118

66-
67119
```
68120

69121
Set virtual reality view background color to black:
@@ -85,3 +137,9 @@ nodeLocked.SetSelectable(0)
85137
nodeMovable.SetSelectable(1)
86138

87139
```
140+
141+
## Related VTK modules
142+
143+
* [VTK::RenderingOpenXR](https://docs.vtk.org/en/latest/modules/vtk-modules/Rendering/OpenXR/README.html)
144+
* [VTK::RenderingOpenXRRemoting](https://docs.vtk.org/en/latest/modules/vtk-modules/Rendering/OpenXRRemoting/README.html)
145+
* [VTK::RenderingOpenVR](https://docs.vtk.org/en/latest/modules/vtk-modules/Rendering/OpenVR/README.html)

0 commit comments

Comments
 (0)