[Discussion] Multiple Invidious process using the same signature helper server

[unixfox]

@unixfox:

I don't see the way to configure to which address the inv_sig_helper is listening on. Is it done on purpose or an oversight?
I'm thinking about docker, where each service has its separate private IP address.

@techmetx11:

Currently, it's listening on a UNIX socket (aka. on a path, by default: /tmp/inv_sig_helper.sock) I could change it to let it listen on UNIX or TCP

@unixfox:

Ah, yes that would be a great improvement because we have people in the community that run Invidious over multiple servers, and you can only communicate through TCP if you have multiple servers, a UNIX socket won't work.

@techmetx11

That would make the current design, of Invidious sending a FORCE_UPDATE opcode to the server, a bit network-intensive. I would have to make the signature server have its own

[unixfox]

The issue is that we have many people that are running multiple Invidious processes at the same time, often the processes are even restarted very often. In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.

So currently that is not feasible to have multiple Invidious process for the same signature helper?

[techmetx11]

The issue is that we have many people that are running multiple Invidious processes at the same time, often the processes are even restarted very often. In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.

So currently that is not feasible to have multiple Invidious process for the same signature helper?

I didn't make it clear, the server CAN handle multiple processes and connections, but if each invidious process tries to update the player in the server with FORCE_UPDATE, it will send more requests to YouTube than it should (the server has to download a video's page to check the current player ID)

[techmetx11]

I would have to make the server have its own scheduler for updating the player, rather than for Invidious to handle it, so that it'll reduce load on YouTube's servers.
Or have a ratelimit for the FORCE_UPDATE opcode

[unixfox]

The issue is that we have many people that are running multiple Invidious processes at the same time, often the processes are even restarted very often. In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.
So currently that is not feasible to have multiple Invidious process for the same signature helper?

You misunderstood me, the server CAN handle multiple processes and connections, but if each invidious process tries to update the player in the server with FORCE_UPDATE, it will send more requests to YouTube than it should (the server has to download a video's page to check the current player ID)

ok! to be fair people that are running multiple invidious processes are already using a lot of bandwidth. it's public instances that are major users of this.

public instances are already transferring a lot of data between postgres and invidious anyway. same goes for the video streams.

if it's like 10MB/s constant for me it's fine. I'm not sure what amount of data you would think? maybe I can do some testing.

[techmetx11]

The issue is that we have many people that are running multiple Invidious processes at the same time, often the processes are even restarted very often. In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.
So currently that is not feasible to have multiple Invidious process for the same signature helper?

You misunderstood me, the server CAN handle multiple processes and connections, but if each invidious process tries to update the player in the server with FORCE_UPDATE, it will send more requests to YouTube than it should (the server has to download a video's page to check the current player ID)

ok! to be fair people that are running multiple invidious processes are already using a lot of bandwidth. it's public instances that are major users of this.

public instances are already transferring a lot of data between postgres and invidious anyway. same goes for the video streams.

if it's like 10MB/s constant for me it's fine. I'm not sure what amount of data you would think? maybe I can do some testing.

If the server has an outdated player ID: it'll download the YouTube video page that it has hardcoded right now ("Me at the zoo") and the base.js inside it (total: ~3.4MB)

If the server already has an updated player ID: it'll only download the YouTube video page (total: ~1MB)

Multiply that by the amount of processes doing FORCE_UPDATE on their own scheduler

[TheFrenchGhosty]

@unixfox

In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.

That's what docker networks are for, or just have a different hostname configured per signer (and bind each invidious process to a single signer)

Or am I missing something?

[techmetx11]

Worst case if there are 4 processes doing FORCE_UPDATE and the server has a outdated player ID: the first request will update the server's player and then the rest will make it request the video page more (which will make it download ~6.4MB)

Best case if there are 4 processes doing FORCE_UPDATE and the server already has an updated player ID: all the requests will make it request the video page more (which will make it download ~4MB)

As i can see from the invidious implementation, it'll try to update every minute I think.
So, what would you think about that?

[techmetx11]

@unixfox

In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.

That's what docker networks are for, or just have a different hostname configured per signer (and bind each invidious process to a single signer)

Or am I missing something?

That would make the bandwidth usage worse, and besides the program is written to handle multiple connections and requests at once

[unixfox]

Worst case if there are 4 processes doing FORCE_UPDATE and the server has a outdated player ID: the first request will update the server's player and then the rest will make it request the video page more (which will make it download ~6.4MB)

Best case if there are 4 processes doing FORCE_UPDATE and the server already has an updated player ID: all the requests will make it request the video page more (which will make it download ~4MB)

As i can see from the invidious implementation, it'll try to update every minute I think. So, what would you think about that?

Thank you for the insights, I now understand much more. (All of this is quite new to me, @SamantazFox knows a lot more than me 😃)

I think it's great for me every minute! Maybe we can wait a little more to have the feedback from @SamantazFox

[SamantazFox]

I don't think that doing many force-update from different processes is a problem. We can always change the timing to two minutes (or more) inside invidious!

Though getting track of that timestamp inside inv_sig_helper would be nice! @techmetx11 can you do that?

[techmetx11]

I don't think that doing many force-update from different processes is a problem. We can always change the timing to two minutes (or more) inside invidious!

Though getting track of that timestamp inside inv_sig_helper would be nice! @techmetx11 can you do that?

Yes, I can. Thank you

[techmetx11]

I don't think that doing many force-update from different processes is a problem. We can always change the timing to two minutes (or more) inside invidious!

Though getting track of that timestamp inside inv_sig_helper would be nice! @techmetx11 can you do that?

A new opcode has been implemented (PLAYER_UPDATE_TIMESTAMP) that can do this now.

The issue is that we have many people that are running multiple Invidious processes at the same time, often the processes are even restarted very often. In a Docker context, it's difficult to bind each Invidious process to one individual signature server when you are running multiple Invidious processes.

So currently that is not feasible to have multiple Invidious process for the same signature helper?

I just implemented TCP/IP support, maybe it'll help with this a little.