Web Scrapping using python Scrapy & Selenium

A versatile web scraper to extract data including texts, images, videos, and audios from different websites, including those with dynamic content loading.

Technologies

1. Python.
2. Selenium.
3. Scrapy.
4. Pycharm IDE.

How to Run

1. Install and configure python.
2. Install scrapy & selenium using the following command:

pip install scrapy selenium

3. Open terminal in folder WebScrapping.
4. Use the following command to run scrapy spider spider1.py:

scrapy crawl spider1

5. Run spider2.py or spider3.py using the following command:

python spider2.py

6. spider2.py & spider3.py are selenium spiders.
7. Must change the path in code where scraped data will be stored.