Skip to content

Commit d9d559d

Browse files
tracygardnertracygardner
committed
Initial documentation on screen reader support
AI use: Intial structure and formatting using Sonnet 4.6 based on detailed instructions. Lots of manual editing and writing.
1 parent d5c04fd commit d9d559d

1 file changed

Lines changed: 163 additions & 0 deletions

File tree

‎docs/screen-reader.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
# Screen reader accessibility in Flock XR
2+
3+
## Overview
4+
5+
Flock XR has an intial prototype of built-in screen reader support, enabling users who rely on assistive technology to navigate 3D worlds, interact with objects, and build programs. The implementation uses standard ARIA live regions and works with major screen readers including Windows Narrator, NVDA, JAWS, and VoiceOver.
6+
7+
This functionality is available in the [development version of Flock XR](https://flipcomputing.github.io/flock/) which is under active development. If you're interested in a more stable version for trying out the features then do get in touch.
8+
9+
Flock XR also has initial screen reader support for creating project using Blockly blocks using Blockly v13. Additional work is ongoing to integrate this functionality into Flock XR so that creating projects is fully screen reader accessible.
10+
11+
Flock XR supports spatial audio so sounds can be attached to meshes in 3D space and the volume and direction changes based on the proximity of the player to the source.
12+
13+
The 2D UI componentsthat can be used in Flock XR are accessible by using tab and then interacting. This can be used to build text-based programs that are screen reader accessible.
14+
15+
Many more features are on our roadmap, subject to funding, making Flock XR a viable coding environment for blind and visually impaired learners.
16+
17+
## Using Flock XR projects
18+
19+
### How it works
20+
21+
When a Flock XR world loads, the screen reader announces a brief introduction describing the scene and available controls. After that, objects and events are announced on demand via keyboard shortcuts, or automatically as they happen during a running program.
22+
23+
### Keyboard commands
24+
25+
These shortcuts work when the 3D canvas is focused.
26+
27+
- **Ctrl+H** — Repeat the help text and keyboard instructions
28+
- **Ctrl+I** — Read a summary of the current scene and nearby objects
29+
- **Ctrl+J** — Describe the nearest object — name, direction, distance, and any text or interaction hint
30+
- **Ctrl+K** — Describe the object directly ahead (within a 45° forward cone)
31+
32+
#### Movement and navigation
33+
34+
- **W or Z** — Move forward (Z is an AZERTY layout alias)
35+
- **S** — Move backward
36+
- **A or Q** — Move left (Q is an AZERTY layout alias)
37+
- **D** — Move right
38+
- **Mouse** — Look around
39+
- **E** — Interact with the nearest interactable object
40+
41+
### Gamepad / controller support
42+
43+
A gamepad or game controller can be used in place of, or alongside, the keyboard. Standard PS4-compatible controllers (and most other modern controllers) are supported.
44+
45+
#### Accessibility controls — D-pad
46+
47+
On controllers with a left analogue stick, the D-pad is reserved for accessibility commands. The directions correspond to the positions of the I, J, and K keys on a keyboard — I is above J, and K is below it:
48+
49+
- **D-pad Up** — Scene summary (Ctrl+I)
50+
- **D-pad Left** — Nearest object (Ctrl+J)
51+
- **D-pad Down** — Object ahead (Ctrl+K)
52+
53+
#### Movement and action buttons
54+
55+
- **Left stick** — Move forward / backward / left / right
56+
- **Right stick** — Look around (camera)
57+
- **L1 / R1 shoulder buttons** — Turn left / turn right
58+
- **Circle (PS4)** — Interact with nearest object (same as E key)
59+
- **Cross / X (PS4)** — General action (space bar equivalent)
60+
- **Triangle (PS4)** — Action / camera up
61+
- **Square (PS4)** — Action / camera down
62+
63+
On controllers without a left analogue stick the D-pad controls movement instead of accessibility commands which are still available from the keyboard.
64+
65+
### Scene and object descriptions
66+
67+
#### Scene summary — Ctrl+I
68+
69+
Gives an overview of the environment and the main objects in the scene, sorted by distance from your character. Includes any on-screen instruction text set by the program.
70+
71+
#### Nearest object — Ctrl+J
72+
73+
Announces the closest non-environment object, including:
74+
75+
- Object name
76+
- Direction relative to where you are facing (e.g. "in front of you, to your right")
77+
- Distance ("very close", "nearby", "a short distance away", "further away", "far away")
78+
- Any text or say text associated with the object
79+
- An interaction hint if the object is interactable
80+
81+
Distance is measured from the nearest surface of the object's bounding box, so large objects are described as close when you are standing next to them.
82+
83+
#### Facing object — Ctrl+K
84+
85+
Identifies the object directly ahead within a 45° cone in front of your character or the camera, measured along the ground plane. Reports direction (Forward, Left, or Right) and distance. Objects you are standing on or inside are excluded from this check.
86+
87+
### Interacting with objects
88+
89+
When an object is interactable, this is included in the nearest object announcement (Ctrl+J). Press **E** (or Circle on a PS4 controller) to interact. The screen reader announces the object name, any text it shows, and any interaction hint.
90+
91+
The visual interact indicator appears when an interactable object is within approximately 4 metres of your character. Objects further away are not highlighted.
92+
93+
### 2D UI components
94+
95+
Flock XR programs can include 2D UI components such as text input fields, sliders, and buttons.
96+
97+
- **Tab** from the canvas to move focus into the UI controls.
98+
- **Shift+Tab** moves backwards, returning to the canvas at either end.
99+
- Sliders are exposed as range inputs and can be adjusted with the arrow keys.
100+
- Text inputs are full HTML text fields — screen readers stay in forms mode and keypresses go directly to the field.
101+
- Buttons can be activated with Enter or Space.
102+
103+
The tab order of UI controls reflects the order in which they are added.
104+
105+
## Creating projects with Blockly
106+
107+
Blockly v13 is adding screen reader support for creating projects using a screen reader. Flock XR has partial support for this and we're hoping to add full support as soon as we can by fully integrating with Flock XR blockly blocks.
108+
109+
Screen reader support builds on keyboard controls support for Blockly. Keyboard navigation is currently being added to the Flock XR UI and Blockly blocks.
110+
111+
### Sound blocks
112+
113+
Flock XR has a range of sound blocks including longer themes, sound effects and musical note blocks. We intend to improve these blocks and make them blocks fully accessible to screen reader users.
114+
115+
Sounds can be played from everywhere in the scene or attached to an object to provide spatial audio. Spatial audio means that a sound comes from a particular position in 3D space and is heard relative to the player.
116+
117+
This feature can be used to add interest and orientation for users who are blind or visually impaired.
118+
119+
### Say blocks and print blocks
120+
121+
Flock XR programs can display text to the player using **say** and **print** blocks.
122+
123+
- **Print blocks** — text is live-announced by the screen reader when it appears on screen.
124+
- **Say blocks with a duration** — text is live-announced immediately when the say block fires, and is also included in the nearest object description (Ctrl+J) for as long as it is displayed.
125+
- **Say with duration 0** (persistent say) — the text is not live-announced, but is stored as a permanent description for the object and becomes part of the nearest object description when you use Ctrl+J. This is useful for setting a standing description that players can discover when they approach.
126+
127+
Flock XR also has speech blocks that can generate speech from text.
128+
129+
### Text projects
130+
The 2D controls mean that Flock XR can be used to build text-based programs — games and interactive stories that work through text output and input without relying on the 3D scene visually. Using print blocks, say blocks, and UI input blocks, it is possible to create projects that are usable by screen reader users.
131+
132+
## Future
133+
There's so much we can do in Flock XR to allow the creation and exploration of 3D spatial worlds using a screen reader.
134+
135+
This includes adding sonification to help users navigate, adding improved music composition and spatial audio support and adding haptic feedback through low cost games controllers. And probably lots of things we haven't thought of yet.
136+
137+
Flock XR enables young people to have creative expression. It can also be used to increase awareness of screen reader use so that young people can themselves create projects that are accessible to others.
138+
139+
Spatial computing is imcreasingly important in society and industry with some key capabilities that are relevant to users who are blind or visually impaired. Flock XR has the potential to offer virtual experiences that enable users to practice skills such as tech-supported 3D navigation.
140+
141+
## Credits
142+
143+
Screen reader support for creating Blockly projects in Flock XR is possible through [Blockly's keyboard navigation and screen reader support](https://www.blockly.com/accessibility). Tracy (creator and maintainer of Flock XR played a key role in this work through the [micro:bit Educational Foundation](https://microbit.org/accessibility/microsoft-makecode/)), working closely with a youth panel.
144+
145+
Support for adding keyboard controls support to Flock XR is ongoing and supported by [NLnet](https://nlnet.nl/project/FlockXR-a11y-mobile-UX/).
146+
147+
The initial screen reader accessibility prototype for Flock XR was developed by **Esther Mbugua**, a final year Computer Science student at the University of Sheffield.
148+
149+
- [Esther's GitHub](https://github.com/essymbugua)
150+
- [Esther's LinkedIn](https://uk.linkedin.com/in/esther-mbugua-513b9a25b)
151+
152+
The work has been extended and further developed since the initial prototype and we would like to take it much further.
153+
154+
We continually use the [RNIB Gaming Devkit](https://github.com/RNIB-MediaAndCulture/Gaming_Devkit/blob/main/Devkit.md) as a useful reference for accessible game development.
155+
156+
## Supporting the work
157+
158+
Flock XR is open-source and accessibility work is ongoing. We are actively looking for sponsorship to continue developing screen reader support, improve the Blockly editor experience, and make Flock XR more widely accessible.
159+
160+
We're really excited about what is possible here, but we're going to need support to complete this work to a high standard including working closely with users who have a deep understanding of how this capability needs to work.
161+
162+
- [Sponsor Flock XR on GitHub Sponsors](https://github.com/sponsors/flipcomputing)
163+
- [Get in touch via flockxr.com](https://flipcomputing.com/contact/)

0 commit comments

Comments
 (0)