If you're not familiar with Scrapy or how it works, we recommend researching the Scrapy documentation (https://docs.scrapy.org/en/latest/) to continue development with this tool.
To get started with Scrapy, you will first need to install it using the methods provided in its documentation. Check here for more information
Once you get Scrapy up and running, if you have not yet, make sure that you create your project folder:
scrapy startproject yourprojectname
When project directory is setup, you can deploy our middleware:
- Open Terminal window.
- Navigate to the main directory of your project folder using
cd yourprojectname - Download our proxy middleware using the following command:
curl https://raw.githubusercontent.com/Decodo/Scrapy-Middleware/master/decodo_auth.py > decodo_auth.py - You should now see your project folder populated with Decodo_auth.py file.
To start using our middleware for proxy authentication, you'll need to configure settings for our proxy authentication.
Doing so is very simple:
- Using file manager, navigate to your project folder, you should see settings.py file located at the bottom of the directory.
- Edit the settings.py file using an editor of your choice and add the following properties at the bottom:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
'yourprojectname.decodo_auth.ProxyMiddleware': 100,
}
Decodo_USER = 'username' ## Decodo Username (Sub-user)
Decodo_PASSWORD = 'password' ## Password for your user
Decodo_ENDPOINT = 'gate.decodo.com' ## Endpoint you'd like to use
Decodo_PORT = '7000' ## Port of the endpoint you are using.
- In
DOWNLOADER_MIDDLEWARESchangeyourprojectnameline to the name of your project.
- Make sure that you enter your account details as well as proxy details within quotation marks ('').
- Save the file.
Once all that is done, all of your spiders will be going through our proxies. If you are not sure how to set up a spider, take a look here
Email - [email protected]
Live chat 24/7