Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable Pico face tracking lipsync mode #2738

Closed
wants to merge 1 commit into from
Closed

Disable Pico face tracking lipsync mode #2738

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
use openxr::{self as xr, sys};

const TRACKING_MODE_FACE_BIT: u64 = 0x00000008;
const TRACKING_MODE_FACE_LIPSYNC: u64 = 0x00002000;

Check warning on line 5 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-latest 

warning: constant `TRACKING_MODE_FACE_LIPSYNC` is never used
 --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:5:7
  |
5 | const TRACKING_MODE_FACE_LIPSYNC: u64 = 0x00002000;
  |       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default

Check warning on line 5 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-windows 

warning: constant `TRACKING_MODE_FACE_LIPSYNC` is never used
 --> alvr\client_openxr\src\extra_extensions\face_tracking_pico.rs:5:7
  |
5 | const TRACKING_MODE_FACE_LIPSYNC: u64 = 0x00002000;
  |       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default

Check warning on line 5 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-old 

warning: constant `TRACKING_MODE_FACE_LIPSYNC` is never used
 --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:5:7
  |
5 | const TRACKING_MODE_FACE_LIPSYNC: u64 = 0x00002000;
  |       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(dead_code)]` on by default
const TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES: u64 = 0x00000100;

Check warning on line 6 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-latest 

warning: constant `TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES` is never used
 --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:6:7
  |
6 | const TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES: u64 = 0x00000100;
  |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Check warning on line 6 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-windows 

warning: constant `TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES` is never used
 --> alvr\client_openxr\src\extra_extensions\face_tracking_pico.rs:6:7
  |
6 | const TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES: u64 = 0x00000100;
  |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Check warning on line 6 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-old 

warning: constant `TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES` is never used
 --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:6:7
  |
6 | const TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES: u64 = 0x00000100;
  |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

#[repr(C)]
struct FaceTrackingDataPICO {
Expand Down Expand Up @@ -38,7 +38,7 @@
}

impl FaceTrackerPico {
pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {

Check warning on line 41 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-latest 

warning: unused variable: `audio`
  --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:41:58
   |
41 |     pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {
   |                                                          ^^^^^ help: if this is intentional, prefix it with an underscore: `_audio`

Check warning on line 41 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-latest 

warning: unused variable: `visual`
  --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:41:44
   |
41 |     pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {
   |                                            ^^^^^^ help: if this is intentional, prefix it with an underscore: `_visual`
   |
   = note: `#[warn(unused_variables)]` on by default

Check warning on line 41 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-windows 

warning: unused variable: `audio`
  --> alvr\client_openxr\src\extra_extensions\face_tracking_pico.rs:41:58
   |
41 |     pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {
   |                                                          ^^^^^ help: if this is intentional, prefix it with an underscore: `_audio`

Check warning on line 41 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-windows 

warning: unused variable: `visual`
  --> alvr\client_openxr\src\extra_extensions\face_tracking_pico.rs:41:44
   |
41 |     pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {
   |                                            ^^^^^^ help: if this is intentional, prefix it with an underscore: `_visual`
   |
   = note: `#[warn(unused_variables)]` on by default

Check warning on line 41 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-old 

warning: unused variable: `audio`
  --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:41:58
   |
41 |     pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {
   |                                                          ^^^^^ help: if this is intentional, prefix it with an underscore: `_audio`

Check warning on line 41 in alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs

View workflow job for this annotation

GitHub Actions / check-linux-old 

warning: unused variable: `visual`
  --> alvr/client_openxr/src/extra_extensions/face_tracking_pico.rs:41:44
   |
41 |     pub fn new<G>(session: xr::Session<G>, visual: bool, audio: bool) -> xr::Result<Self> {
   |                                            ^^^^^^ help: if this is intentional, prefix it with an underscore: `_visual`
   |
   = note: `#[warn(unused_variables)]` on by default
session
.instance()
.exts()
Expand All @@ -52,12 +52,7 @@

let mut tracking_flags = 0;

if visual {
tracking_flags |= TRACKING_MODE_FACE_BIT;
}
if audio {
tracking_flags |= TRACKING_MODE_FACE_LIPSYNC | TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES;
}
tracking_flags |= TRACKING_MODE_FACE_BIT;
Copy link
Contributor

@curoviyxru curoviyxru Mar 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are bools in constructors for that purpose. IMO this should not be just removed but implemented as switches in dashboard for both FaceTracker2Fb and FaceTrackingPico. See curoviyxru@b6aa99f.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. instead of removing audio input at all, we should only remove the flag TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should only remove the flag TRACKING_MODE_FACE_LIPSYNC_BLEND_SHAPES

but it matches FaceTracker2Fb's audio face tracking, doesn't it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meta has VIDEO and AUDIO flags, which control the sources of tracking. the change we should do here is to not generate visemes/blend shapes, which regards tracking output, not input

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is to not generate visemes/blend shapes

but what does AUDIO flag on Meta do then if not generating blend shapes based on audio input?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right. I think there is something we are missing, Pico API might be a bit confusing. What we want is to maybe allow audio source but not generate visemes. You could look up how to actually do this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allow audio source but not generate visemes

I'm afraid that there is no such thing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then it's a matter of sensible defaults. If generating visemes is more resource intensive, probably we should disable the audio source by default. Furthermore we should not send the visemes array range, and we should add another rule on the server-side to detect the Pico face tracking without visemes. As for VRCFaceTracking, i'm afraid we can't do much unless we make a breaking change. It should be fine to just send zeroes for the visemes range


Ok(Self {
session: session.into_any_graphics(),
Expand Down
Loading