How does dub handle malicious (usually phishing) links and avoid getting flagged by search engines?

[SneakySensei]

Loving the codebase and I'm learning so many new things from dub. Thanks @steven-tey for actively updating this with best practices 🙌🏽

I've made a similar link shortener in the past, complete with analytics, and password protection. But soon after deploying to prod, I kept getting spammed by malicious users who create phishing links using my project and as a result, my domain kept getting flagged by Google. My assumption was that the flagging algorithm was misinterpreting the redirect to the target URL flagging the wrong domain (the shortener service). Here's a screenshot of the exact notices I used to get. Eventually, I had to take my project off prod.

How does dub avoid this? I'm sure there's something in place that's allowing Google and other search engines to better detect the actual malicious URL. But I'm not able to infer that from the code.

[matytyma]

In my opinion, dub just does not let search engines discover the page by setting up noindex
🔗 https://developers.google.com/search/docs/crawling-indexing/block-indexing

[SneakySensei]

In my opinion, dub just does not let search engines discover the page by setting up noindex 🔗 https://developers.google.com/search/docs/crawling-indexing/block-indexing

I've checked the whole project and couldn't find a noindex meta tag anywhere. Additionally, the LinkMiddleware which is responsible for the redirect, has just this:

Is the proxy page responsible for this? Or the fact that NextResponse.redirect() produces a 307 (I was using 302) redirect?

[matytyma]

Hmm, idk then 😄

[gabefletch]

One way that Dub deals with phishing links is by actively removing ones reported over at dub.co/abuse.

Dub's Terms of Service (dub.co/terms) also state that it is the user's responsibility for their use of the service and any content posted or transmitted through it. The user is prohibited by the Terms from using Dub for any malicious purpose, including but not limited to: phishing or scam websites, pornography or adult content, betting or gambling, copyright infringement.

As far as I'm aware, dub.sh links are not indexed by search engines. This is pretty standard practice for link shortening services to avoid the obvious issues that would arise if they were indexed.