add steps about sig_helper and potoken + depreciate gluetun guide

[unixfox]

Update 21/09/2024: This technique doesn't work anymore, read the announcement here: iv-org/invidious#4734 (comment)

---

The installation guide (Docker, Linux and MacOS) has been reflected on how to setup the new tools that will solve the error message "This helps protect our community.".

Please read the rendered markdown here for updating your Invidious installation: https://github.com/iv-org/documentation/blob/sig_helper_po_token/docs/installation.md. If you are using Docker, you need to switch to the Docker tag :master, like so:

image: quay.io/invidious/invidious:master

Because the code is not yet ready for a stable release. It's better to switch back to :latest after this pull request has been merged.

If you have any bug regarding sig_helper or generating a potoken, please open a GitHub issue on the Invidious repository. Make sure this is not a miss-configuration issue on your side, if you are unsure, please ask your question on Matrix or IRC.

If you have previously setup inv_sig_helper manually, please use the updated docker-compose one because this tool runs untrusted code from Google. You don't want to get anything nasty from Google themselves.

---

Also, this pull request delist gluetun which is not a recommended workaround anymore. Since this is ultimately delaying the ban until YouTube detects the need to block your public IP address.

Fixes #580

[fox34]

Can confirm, the solution is working fine for now. I will watch my instance for the next days and report any issues. Thanks a lot for the hard work, very appreciated!

[animegrafmays]

confirming it is functional on youchu.be

[bugmaschine]

Once again amazing work!

[TheFrenchGhosty]

Other than those minor changes, it's all good.

Tested and working

[ravenblackdusk]

My private instance is working now with these changes. I'll tell you if something goes wrong.

[KokaKiwi]

My semi-private instance is now working with these changes, thanks so much for the work done! 😄
Will post here if any problem arise :)

[manfromdownunder]

Why is gluetun no longer needed? If you are regularly changing the visitor_data and po_token while behind a VPN, then doesn't that make the source ip and metadata less useful as 1000's of users would be coming from the same IP with also their own visitor_data and po_token?

[kalipso-cyber]

Can also confirm this is working beautifully so far. Thank you very much for your hard work!

[mas1701]

I updated my semi-private instance to the current Git version including po_token/visitor data and sig_helper and so far it works beautifully again. Thanks a lot to all who made this possible.

EDIT by unixfox: don't advertise your instance here

[goodtube4u]

This worked perfectly on my locally hosted instance. Great work all!

[accessiblepixel]

I've followed the instructions with my self-hosted private instance, which was again blocked today (after having a workaround-working for a while).

I'm using portainer for various reasons (I'm looking to get away) so some of the steps I had to modify and do slightly differently - but it works perfectly so far.

Excellent work Invidious Team. Thank you! Very much appreciated.

Things I had to do:

Original notes by the author

The ``init-invidious-db.sh`` file I had to use an absolute path not a relative, and the inv_sig_helper my invidious didn't find it through the name, so I had to manually specify the IP. I also had to add the inv_sig_helper to my invidious 'network'.

Included is the full config I have.

version: "2.4"
services:
  postgres:
    image: postgres:10
    restart: unless-stopped
    networks:
      - invidious
    volumes:
      - postgresdata:/var/lib/postgresql/data
      - ./config/sql:/config/sql
      - /root/invidious/docker/init-invidious-db.sh:/docker-entrypoint-initdb.d/init-invidious-db.sh
    environment:
      POSTGRES_DB: invidious
      POSTGRES_USER: kemal
      POSTGRES_PASSWORD: kemal
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB"]
  invidious:
    image: quay.io/invidious/invidious:master
    restart: unless-stopped
    networks:
      - invidious
    mem_limit: 1024M
    cpus: 0.5

    environment:
      INVIDIOUS_CONFIG: |
        channel_threads: 1
        check_tables: true
        feed_threads: 1
        check_tables: true
        signature_server: 172.19.0.3:12999
        visitor_data: GENERATED_FROM_TRUSTED_TOKEN_GENERATOR
        po_token: GENERATED_FROM_TRUSTED_TOKEN_GENERATOR
        db:
          dbname: invidious
          user: kemal
          password: kemal
          host: postgres
          port: 5432
        full_refresh: false
        https_only: true
        domain: my-domain.tld
        hmac_key: SET_AS_DOCUMENTATION
        external_port: 443
    depends_on:
      - postgres

  inv_sig_helper:
    image: quay.io/invidious/inv-sig-helper:latest
    command: ["--tcp", "0.0.0.0:12999"]
    environment:
      - RUST_LOG=info
    restart: unless-stopped
    cap_drop:
      - ALL
    read_only: true
    security_opt:
      - no-new-privileges:true
    networks:
      - invidious


volumes:
  postgresdata:

networks:
  invidious:

Edit: Typoed my copy of my config, and missed something else I did to fix ;)

Remarks by unixfox: Not recommended using a fixed IP address because in a docker network, containers can change their private IP address when recreated so the setup will be broken at some point.

[animegrafmays]

confirming it is functional on youchu.be

looks like this method works for about two days. restarting the signature helper and rotating the resolved trusted session tokens resolves it. I may just write a bash script to handle this on a crontab. error from the signature helper before restarting:

Successfully fetched player
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }

[unixfox]

confirming it is functional on youchu.be

looks like this method works for about two days. restarting the signature helper and rotating the resolved trusted session tokens resolves it. I may just write a bash script to handle this on a crontab. error from the signature helper before restarting:

Successfully fetched player
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }

iv-org/inv_sig_helper#12

[techmetx11]

confirming it is functional on youchu.be

looks like this method works for about two days. restarting the signature helper and rotating the resolved trusted session tokens resolves it. I may just write a bash script to handle this on a crontab. error from the signature helper before restarting:

Successfully fetched player
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
I/O error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }

If this is inv_sig_helper running in TCP, this looks like whatever client (Invidious or something else in the middle) on the other end of the connection forcefully reset the connection. Are you running inv_sig_helper in a seperate server?
(note that invidious doesn't yet respawn the connection when it unexpectedly terminates like this)

[animegrafmays]

same problem as @fjcloud

2024-08-24 09:53:01 UTC [warn] i18n: Missing translation key "This helps protect our community. Learn more"
2024-08-24 09:53:01 UTC [info] 500 GET /watch?v=MoUPcVt4GUA 34.03ms
2024-08-24 09:53:11 UTC [info] 200 GET /api/v1/trending 390.2ms```

running a public instance google eventually rate limits the IPs. for us its about 2 days of 5-6000 unique visitors. I stop the invidious service, rotate the IPv6 address, generate a new token and restart the signature service and it works again for 2 days. does not fail always starts with google rate limiting the IP and rotating and running through the process always results in a positive service for 2 days

[unixfox]

@fjcloud @Happyfeet01 try to generate the tokens potoken and visitordata on another separate IP address to see if it's better. Like your home IP address.

We have had some feedback about this working for some people.

[unixfox]

same problem as @fjcloud

2024-08-24 09:53:01 UTC [warn] i18n: Missing translation key "This helps protect our community. Learn more"
2024-08-24 09:53:01 UTC [info] 500 GET /watch?v=MoUPcVt4GUA 34.03ms
2024-08-24 09:53:11 UTC [info] 200 GET /api/v1/trending 390.2ms```

running a public instance google eventually rate limits the IPs. for us its about 2 days of 5-6000 unique visitors. I stop the invidious service, rotate the IPv6 address, generate a new token and restart the signature service and it works again for 2 days. does not fail always starts with google rate limiting the IP and rotating and running through the process always results in a positive service for 2 days

You are most likely running into another problem if you have running a public instance.

Make sure to still have smart ipv6 rotator running. And make sure to block any bot usage, we have had some report from a public instance owner of rate limits by youtube. Message related to "browse endpoint rate limit exhausted"

It was caused by bots spamming the instance.

[souxd]

inv_sig_helper-1  | Fetching player JS URL: https://www.youtube.com/s/player/a87a9450/p
layer_ias.vflset/en_US/base.js
inv_sig_helper-1  | nsig function ending did not work: =\s*function(\([\w]+\)\{\s*var\s
+[\w\s]+=[\w\.\s]+?\.call\s*\([\w\s$]+?,[\(\)\",\s]+\)[\S\s]*?\}\s*return [\w\.\s$]+?\.
call\s*\([\w\s$]+?\s*,[\(\)\",\s]+\)\s*\}\s*;)
inv_sig_helper-1  | sig code: var dCa;var WK={YL:function(a,b){var c=a[0];a[0]=a[b%a.le
ngth];a[b%a.length]=c},
inv_sig_helper-1  | KY:function(a){a.reverse()},
inv_sig_helper-1  | qf:function(a,b){a.splice(0,b)}};dCa=function(a){a=a.split("");WK.q
f(a,1);WK.YL(a,43);WK.qf(a,3);WK.YL(a,23);WK.KY(a,60);WK.YL(a,3);WK.KY(a,11);WK.YL(a,22
);return a.join("")}
inv_sig_helper-1  | Successfully fetched player

...

invidious-1       | 2024-08-25 12:54:43 UTC [info] 200 GET /vi/o-cBA2DFLVs/mqdefault.jpg 584.28ms
invidious-1       | 2024-08-25 12:54:43 UTC [info] 200 GET /vi/JOCqCs6sl8k/mqdefault.jpg 377.65ms
invidious-1       | 2024-08-25 12:54:44 UTC [error] get_video: y92IoM_ERcg : This helps protect our community. Learn more
invidious-1       | 2024-08-25 12:54:44 UTC [warn] i18n: Missing translation key "This helps protect our community. Learn more"
invidious-1       | 2024-08-25 12:54:44 UTC [info] 500 GET /watch?v=y92IoM_ERcg 107.65ms
invidious-1       | 2024-08-25 12:54:44 UTC [info] 200 GET /vi/VhiyrpX09LU/hqdefault.jpg 435.87ms
invidious-1       | 2024-08-25 12:54:44 UTC [info] 200 GET /favicon-32x32.png?v=eda7444 19.44µs
invidious-1       | 2024-08-25 12:54:48 UTC [info] 200 GET /api/v1/comments/jNQXAC9IVRw 601.45ms

i still have the same issue. I've tried to generate the visitor_data and po_token both on the machine and my home but it doesnt work

[cu]

Working for me on my private instance.

[WardPearce]

Has been working mostly fine for around a week & a half for me on https://invidious.materialio.us. Recently I've had issues with videos not loading but rotating the visitor_data & po_token fixes that.

[auralglow]

Also working for me on my semi-private instance. If you need a way to rotate the po_token and visitor_data automatically, here is a way:

Create a new script and take note of the path (in this example: /opt/script/refresh-inv-token.sh)

#!/bin/sh

tokens=$(docker run quay.io/invidious/youtube-trusted-session-generator)
po_token=$(echo "$tokens" | grep po_token | cut -d ' ' -f 2)
visitor_data=$(echo "$tokens" | grep visitor_data | cut -d ' ' -f 2)
sed -i 's/po_token.*/po_token: '"$po_token"'/g' <path_docker_compose> && \
sed -i 's/visitor_data.*/visitor_data: '"$visitor_data"'/g' <path_docker_compose>

docker compose --project-directory <path_docker_compose_folder> up -d && docker container prune -f

Replace the <path_docker_compose> with the path of the docker-compose.yml file and <path_docker_compose_folder> with the path of the folder where the docker-compose.yml file is located.

Give it execution permissions:
chmod +x /opt/script/refresh-inv-token.sh

Modify the crontab, to make the script run periodically with crontab -e and add this line:

00 * * * * /opt/script/refresh-inv-token.sh

This line makes the script run every hour at minute 0. If you wish to make the script run once every 2 hours or more, replace the first * with */X
Example for an once every 3 hours run

00 */3 * * * /opt/script/refresh-inv-token.sh

EDIT: I just saw there was a similar method posted on here: #580 (comment)
Sorry about that :3

[outis151]

Doesn't work for me on a private instance. No videos load unless I comment out signature_server in INVIDIOUS_CONFIG. This is without using visitor_data and po_token since my instance has never been blocked yet. It is unclear to me whether inv_sig_helper is required regardless of whether the instance is blocked.

inv_sig_helper output

Fetching player
Fetching player JS URL: https://www.youtube.com/s/player/57c75fa4/player_ias.vflset/en_US/base.js
nsig function ending did not work: =\s*function(\([\w]+\)\{\s*var\s+[\w\s]+=[\w\.\s]+?\.call\s*\([\w\s$]+?,[\(\)\",\s]+\)[\S\s]*?\}\s*return [\w\.\s$]+?\.call\s*\([\w\s$]+?\s*,[\(\)\",\s]+\)\s*\}\s*;)
sig code: var bCa;var TK={dQ:function(a,b){a.splice(0,b)},
l8:function(a){a.reverse()},
QF:function(a,b){var c=a[0];a[0]=a[b%a.length];a[b%a.length]=c}};bCa=function(a){a=a.split("");TK.dQ(a,3);TK.QF(a,41);TK.l8(a,43);TK.dQ(a,2);TK.QF(a,27);TK.dQ(a,2);return a.join("")}
Successfully fetched player

[auralglow]

Doesn't work for me on a private instance. No videos load unless I comment out signature_server in INVIDIOUS_CONFIG. This is without using visitor_data and po_token since my instance has never been blocked yet. It is unclear to me whether inv_sig_helper is required regardless of whether the instance is blocked.

inv_sig_helper output

Fetching player
Fetching player JS URL: https://www.youtube.com/s/player/57c75fa4/player_ias.vflset/en_US/base.js
nsig function ending did not work: =\s*function(\([\w]+\)\{\s*var\s+[\w\s]+=[\w\.\s]+?\.call\s*\([\w\s$]+?,[\(\)\",\s]+\)[\S\s]*?\}\s*return [\w\.\s$]+?\.call\s*\([\w\s$]+?\s*,[\(\)\",\s]+\)\s*\}\s*;)
sig code: var bCa;var TK={dQ:function(a,b){a.splice(0,b)},
l8:function(a){a.reverse()},
QF:function(a,b){var c=a[0];a[0]=a[b%a.length];a[b%a.length]=c}};bCa=function(a){a=a.split("");TK.dQ(a,3);TK.QF(a,41);TK.l8(a,43);TK.dQ(a,2);TK.QF(a,27);TK.dQ(a,2);return a.join("")}
Successfully fetched player

For me, the po_token/visitor_data and the signature server parameter go hand in hand to resolve the YouTube-side checks. So, if you use inv_sig_helper, you also need to configure the po_token and the visitor_data parameters.

[doc75]

Working fine on my personal instance (tested with both latest and master images). Many thanks.
I took inspiration from the docker-compose containing both invidious and inv-sig-helper (cf. commit content of this PR)

[outis151]

I can confirm inv_sig_helper works but only when the po_token and the visitor_data parameters are used as well. Maybe it should be mentioned that they are not optional and without them no videos play (even when the instance is not blocked and the instance plays videos fine without setting a signature server )

[Handrail9]

Will delete if this is just noise but I want to confirm this is working for me as well, although I'm noticing my db will exit more frequently than usual. It is also working with Materialious.
EDIT: I said i made a script to generate the tokens automatically but it didn't work as expected and i dont know how to fix it.
EDIT 2: I got it working! :) Use your favorite method of automatically running scripts and change the paths for your usecase and it should be good to go to regen keys however often you want. Make sure to make an .env file and make two variables for visitor data and po token.

#!/bin/bash
# Run the Docker command and capture the output
cd /opt/invidious
output=$(docker run --rm quay.io/invidious/youtube-trusted-session-generator)

# Extract visitor_data and po_token using grep and awk

visitor_data=$(echo "$output" | grep -oP 'visitor_data: \K[^ ]+')
po_token=$(echo "$output" | grep -oP 'po_token: \K[^ ]+')

# Define the path to the .env file

env_file="/opt/invidious/.env"

# Check if the extraction was successful

if [[ -z "$visitor_data" || -z "$po_token" ]]; then
    echo "Failed to extract visitor_data or po_token."
    exit 1
fi

# Update the .env file
# Use sed to replace the existing values or add them if they don't exist
sed -i "s/^VISTOR_DATA=.*/VISTOR_DATA=$visitor_data/" "$env_file" || echo "VISTOR_DATA=$visitor_data" >> "$env_file"
sed -i "s/^PO_TOKEN=.*/PO_TOKEN=$po_token/" "$env_file" || echo "PO_TOKEN=$po_token" >> "$env_file"
#echo "Updated .env file with new visitor_data and po_token."
docker compose down invidious inv_sig_helper invidious-db
docker compose up -d

[Ex0danify]

After Working fine a few weeks using inv_sig_helper and the po/visitor token, today I was again greeted by the "This helps protect our community. Learn more ".

I tried refreshing the tokens, now I can watch a video after refreshing a bunch of times, but most of the time it takes a few refreshes. Maybe they blocked my IP somehow (using a static IPv4 that i can not change).

[doc75]

After Working fine a few weeks using inv_sig_helper and the po/visitor token, today I was again greeted by the "This helps protect our community. Learn more ".

I tried refreshing the tokens, now I can watch a video after refreshing a bunch of times, but most of the time it takes a few refreshes. Maybe they blocked my IP somehow (using a static IPv4 that i can not change).

Same for me since today. No more video will load even with po_token and visitor-data change ;-(

[animegrafmays]

this is also broken for us after rotating ipv6 and switching ipv6 /64 to a previously unused /64

[mas1701]

Unfortunately, we seem to be back at step 1: This will help to protect our community.

Several public instances seem to be affected as well as my own. I already tried refreshing tokens and switch to alternate IPv4 or IPv6 addresses using Source NAT, nothing helps.

(At first, reloading the browser window several times helped, but it no longer works now. It seems it was only a temporary solution while YouTube was still in the process to roll-out the latest changes to their servers.)

[unixfox]

A new fix has been delivered in the master docker image: iv-org/invidious#4928

Which fixes the latest issue related to This helps protect our community.. Please try if it fixes the issue for you.

Reminder:

image: quay.io/invidious/invidious:master

[animegrafmays]

just merged the most recent commit and it's functional on youchu.be

additionally, it seems as an aside that all videos now say "Shared September 19, 2024" instead of their actual posting date

[syeopite]

Yep unfortunately the workaround means that we cannot parse the video's published time (along with some other info) the way we currently do as that data isn't returned there anymore.

[mas1701]

With new update it worked for some hours, now I'm getting:
"Sign in to confirm you're not a bot."

[animegrafmays]

With new update it worked for some hours, now I'm getting: "Sign in to confirm you're not a bot."

this is also happening since about 6PM EST. have rotated addresses within the ipv6 /64, assigned a new one via tunnelbroker and forced ipv4 to test all of which result in the same response "Sign in to confirm you're not a bot."

[unixfox]

Yes unfortunately they patch all the current workarounds.

I have made an announcement: iv-org/invidious#4734 (comment)

Join our matrix or IRC to discuss more. Don't go offtopic here thank you.