User-Agent

[roniemartinez]

• Set a value for Dude User-Agent instead of using the default values on each parser backend (e.g.: pydude/{version} (+https://github.com/roniemartinez/dude))
• Add option to override the User-Agent
• For Playwright and Pyppeteer and Selenium, the User-Agent should include the original (Chromium, Firefox, Webkit) with the Dude User-Agent inserted into the string

[ghost]

@roniemartinez I would like to start with this issue. But I could not completely understand the structure of the code base. Is there any document or architecture diagram to start with?

[roniemartinez]

@FluffyDietEngine All of the scrapers are using ScraperAbstract https://github.com/roniemartinez/dude/blob/master/dude/base.py#L433

For this feature, you also need to modify the following:

1. Playwright's launch():

   dude/dude/playwright_scraper.py

   Line 170 in 6d846aa

   browser = p[browser_type].launch(headless=headless, proxy=proxy, **launch_kwargs)

2. Selenium's Webdriver options:

   dude/dude/optional/selenium_scraper.py

   Line 242 in 6d846aa

   def _get_driver(self, browser_type: str, headless: bool) -> WebDriver:

3. httpx Client (lxml, parsel and bs4 are on different places):

   dude/dude/optional/lxml_scraper.py

   Line 69 in 6d846aa

   with httpx.Client(

[ghost]

@FluffyDietEngine All of the scrapers are using ScraperAbstract https://github.com/roniemartinez/dude/blob/master/dude/base.py#L433

For this feature, you also need to modify the following:

1. Playwright's launch():

   dude/dude/playwright_scraper.py

   Line 170 in 6d846aa

   browser = p[browser_type].launch(headless=headless, proxy=proxy, **launch_kwargs)

2. Selenium's Webdriver options:

   dude/dude/optional/selenium_scraper.py

   Line 242 in 6d846aa

   def _get_driver(self, browser_type: str, headless: bool) -> WebDriver:

3. httpx Client (lxml, parsel and bs4 are on different places):

   dude/dude/optional/lxml_scraper.py

   Line 69 in 6d846aa

   with httpx.Client(

Will start working on it. Please assign the task to me. Will reach out if any help needed.

[ghost]

@roniemartinez I am going through the code. Firstly, hats off. The thought process for the ScraperAbstract class is so inspiring. While working on this issue, I came across few questions and thoughts. I would like to have your opinion on them.

1. Can I have the user agent as a direct attribute to the ScraperAbstract class?
2. I am able to find that the useragent is being used to validate whether it is allowed for the given domain. But what if ignore_robots_txt has been enabled, Where should we implement the user-agents then?
3. I am thinking of using fake-useragent library for the implementation of user agents. Any thoughts?

[roniemartinez]

Hi @FluffyDietEngine thank you for working on this feature.

1. Can I have the user agent as a direct attribute to the ScraperAbstract class?

I am also leaning towards this since there should only be one default user agent but of course this can be overriden by the derived classes. However, if a user wants to replace it with custom user agent, it should be replaceable from terminal args or from run() function.

2. I am able to find that the useragent is being used to validate whether it is allowed for the given domain. But what if ignore_robots_txt has been enabled, Where should we implement the user-agents then?

I think the answer from 1 is still true in this scenario.

3. I am thinking of using fake-useragent library for the implementation of user agents. Any thoughts?

It's a nice suggestion but I think it might bloat the library if this is a builtin feature. I believe, this can be used directly in a code by feeding the generated user agent to run(). An example and additional documentation might be enough to guide users.

[ghost]

@roniemartinez Apologies for the delayed response. Been busy with some work. I want to resume here. And I am seeing pretty much of changes and kind of lost in the middle. Should I make the changes at run method from Scraper class at https://github.com/roniemartinez/dude/blob/master/dude/scraper.py ?

[roniemartinez]

@FluffyDietEngine No worries, also been busy. Yes, that should be the right place though you also have to duplicate this into the derived classes, too. Thanks for looking into it.