Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for an external signature server #4772

Merged
merged 11 commits into from
Aug 13, 2024

Conversation

SamantazFox
Copy link
Member

@SamantazFox SamantazFox commented Jul 3, 2024

Closes #4649

This PR adds support for inv_sig_helper, which offloads the player fetching, function extraction and signature parsing, which in turn allows to use the web client to watch videos.

Side note: I had to update the crystal overrides because the stdlib changed quite a while ago (See crystal-lang/crystal#11049) and it was required to properly use TCP/unix sockets.


TODO:

  • Only activate if something like inv_sig_helper_url: XX is filled. This way, this functionality is only enabled when the user needs it. And the user can specify a TCP endpoint in case of Docker, for example.
  • Additionally, if this is not activated, ANDROID_TEST_SUITE will still be used. (As long as it still works fine for "unblocked IP addresses") This will be addressed when this and Add ability to set po_token and visitor data #4789 will be merged

@syeopite syeopite self-requested a review July 3, 2024 21:21
Copy link
Member

@syeopite syeopite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall I don't really see anything wrong with the logic here. I just have a couple questions to make sure I understand everything correctly.

Also, what do you think about splitting the signature communication logic into its own shard?

src/invidious/helpers/signatures.cr Outdated Show resolved Hide resolved
src/invidious/helpers/sig_helper.cr Show resolved Hide resolved
Copy link
Member

@syeopite syeopite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some Ameba nitpicks. Feel free to ignore.

src/invidious/videos.cr Outdated Show resolved Hide resolved
src/invidious/helpers/sig_helper.cr Show resolved Hide resolved
src/invidious/helpers/sig_helper.cr Show resolved Hide resolved
src/invidious/helpers/sig_helper.cr Show resolved Hide resolved
"referer" => "https://www.youtube.com/watch?v=#{video_id}",
} of String => String | Int64

if {"WEB", "TVHTML5"}.any? { |s| client_config.name.starts_with? s }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[C] Naming/BlockParameterName: Disallowed block parameter name found

Can you pick a more descriptive name than s here?

src/invidious/helpers/sig_helper.cr Outdated Show resolved Hide resolved
src/invidious/helpers/signatures.cr Show resolved Hide resolved
@SamantazFox
Copy link
Member Author

Also, what do you think about splitting the signature communication logic into its own shard?

Why though? Sound harder to maintain and I don't see anyone using this code except us.
The connection multiplexing code might actually be a good candidate for a shard, but I don't think the code is versatile enough in its current state.

@syeopite
Copy link
Member

syeopite commented Jul 6, 2024

Ah right. I did not think about the additional maintenance burden.

I requested it to potentially open the opportunity for other YouTube projects within the Crystal ecosystem though the possibility of that is pretty low...

Let's just keep it within Invidious.

@SamantazFox
Copy link
Member Author

Important note: with the current state of the PR, invidious can't start if the socket is not present. I will add some logic to fallback to another client in that case.

@unixfox
Copy link
Member

unixfox commented Jul 12, 2024

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?

I'm thinking about docker, where each service has its separate private IP address.

@techmetx11
Copy link
Contributor

techmetx11 commented Jul 12, 2024

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?

I'm thinking about docker, where each service has its separate private IP address.

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock)
I could change it to let it listen on UNIX or TCP

@unixfox
Copy link
Member

unixfox commented Jul 12, 2024

@techmetx11

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?
I'm thinking about docker, where each service has its separate private IP address.

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock) I could change it to let it listen on UNIX or TCP

Ah, yes that would be a great improvement because we have people in the community that run Invidious over multiple servers, and you can only communicate through TCP if you have multiple servers, a UNIX socket won't work.

I'm not sure if the sig parameter can be shared between multiple public IP addresses or not, though.

@techmetx11
Copy link
Contributor

techmetx11 commented Jul 12, 2024

@techmetx11

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?
I'm thinking about docker, where each service has its separate private IP address.

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock) I could change it to let it listen on UNIX or TCP

Ah, yes that would be a great improvement because we have people in the community that run Invidious over multiple servers, and you can only communicate through TCP if you have multiple servers, a UNIX socket won't work.

That would make the current design, of Invidious sending a FORCE_UPDATE opcode to the server, a bit network-intensive. I would have to make the signature server have its own scheduler

I'm not sure if the sig parameter can be shared between multiple public IP addresses or not, though.

The JS code doesn't require network access, so I would hope so

@unixfox
Copy link
Member

unixfox commented Jul 12, 2024

Continuing the discussion about signature helper server here: iv-org/inv_sig_helper#1

@unixfox
Copy link
Member

unixfox commented Jul 21, 2024

There will be a need to add the support for TCP/IP, as such that it can be used from multiple invidious processes.

Issue opened on inv_sig_helper side: iv-org/inv_sig_helper#2

@999eagle
Copy link

i've been running this and #4789 on iv.catgirl.cloud for a while now to circumvent the login requirements, but occasionally invidious workers fail to start correctly after their hourly restart due to waiting indefinitely for the sig helper to force update the player. i've opened iv-org/inv_sig_helper#4 for the underlying issue here

@unixfox
Copy link
Member

unixfox commented Jul 26, 2024

Missing feature:

  • Only activate if something like inv_sig_helper_url: XX is filled. This way, this functionality is only enabled when the user needs it. And the user can specify a TCP endpoint in case of Docker, for example.
  • Additionally, if this is not activated, ANDROID_TEST_SUITE will still be used. (As long as it still works fine for "unblocked IP addresses")

syeopite added a commit to syeopite/invidious that referenced this pull request Jul 27, 2024
src/invidious/helpers/sig_helper.cr Outdated Show resolved Hide resolved
src/invidious/helpers/signatures.cr Outdated Show resolved Hide resolved
@SamantazFox SamantazFox force-pushed the sig_helper branch 2 times, most recently from 17ef3f8 to 3592858 Compare August 7, 2024 21:18
@MMaster
Copy link

MMaster commented Aug 9, 2024

To also write it here:
Video caching in invidious also caches and reuses data from player so if the player changes the video that is pulled from cache will cause 403s to be returned for videoplayback requests.

@SamantazFox
Copy link
Member Author

To also write it here: Video caching in invidious also caches and reuses data from player so if the player changes the video that is pulled from cache will cause 403s to be returned for videoplayback requests.

Yeah, I should rewrite the URLs during parsing (i.e before caching) not during use.

@SamantazFox SamantazFox added ready and removed blocked require something else first labels Aug 13, 2024
@SamantazFox SamantazFox merged commit 2d7869b into iv-org:master Aug 13, 2024
6 of 8 checks passed
@SamantazFox SamantazFox deleted the sig_helper branch August 17, 2024 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] Add decryption of the sig parameter for bypassing throttling
7 participants