New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Update robots.txt to disallow crawling of non-latest docs #506

Closed

nzakas wants to merge 1 commit into main from docs-next-head-disallow

Member

nzakas commented Jan 2, 2024

Prerequisites checklist

I have read the contributing guidelines.

What is the purpose of this pull request?

What changes did you make? (Give an overview)

To ensure that only the latest version docs are crawled, disallowed crawling of /docs/next/ and /docs/head/.

I also removed the old disallow when we were testing new.eslint.org.

Related Issues

Is there anything you'd like reviewers to focus on?


          chore: Update robots.txt to disallow crawling of non-latest docs

18fb09e

eslint-github-bot bot added the chore label

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for hi-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/hi-eslint/deploys/65945dc23b5e75000872f0c9
😎 Deploy Preview	https://deploy-preview-506--hi-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for new-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/new-eslint/deploys/65945dc2ef46750008532918
😎 Deploy Preview	https://deploy-preview-506--new-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for ja-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/ja-eslint/deploys/65945dc2a537e40008b97560
😎 Deploy Preview	https://deploy-preview-506--ja-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for zh-hans-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/zh-hans-eslint/deploys/65945dc2b4b4e50008cc7c5e
😎 Deploy Preview	https://deploy-preview-506--zh-hans-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for es-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/es-eslint/deploys/65945dc2a6f44e0008c74be4
😎 Deploy Preview	https://deploy-preview-506--es-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for fr-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/fr-eslint/deploys/65945dc2bae3b3000837b26c
😎 Deploy Preview	https://deploy-preview-506--fr-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for de-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/de-eslint/deploys/65945dc20410d8000853f242
😎 Deploy Preview	https://deploy-preview-506--de-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

netlify bot commented Jan 2, 2024 •

edited

Loading

✅ Deploy Preview for pt-br-eslint ready!

Name	Link
🔨 Latest commit	`18fb09e`
🔍 Latest deploy log	https://app.netlify.com/sites/pt-br-eslint/deploys/65945dc21340140008fd5767
😎 Deploy Preview	https://deploy-preview-506--pt-br-eslint.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

mdjermanovic reviewed

View reviewed changes

Member

mdjermanovic left a comment

I'm not sure if this is the right approach. We already have <meta name="robots" content="noindex"> on those pages, and some articles say that one should not be used with the other.

https://developers.google.com/search/docs/crawling-indexing/block-indexing

Important: For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler. If the page is blocked by a robots.txt file or the crawler can't access the page, the crawler will never see the noindex rule, and the page can still appear in search results, for example if other pages link to it.

Might be good to ask for a SEO advice?

Member Author

nzakas commented Jan 2, 2024

Ah, totally forgot about the meta tag. Nevermind. :)

nzakas closed this

nzakas deleted the docs-next-head-disallow branch

January 2, 2024 20:13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels