Price Comparison Pro provides multiple scraping types. Below is a guide explaining when to use each. You’ll find the Scrape Type on your Price Comparison Pro -> Price Comparison Pro settings page:
Once you have websites added, you’ll see the Scrape Type drop down where you can configure your scraping type.
Standard
Standard is the default approach to scraping. It fetches the HTML using your server. This should be your default choice as it’s the cheapest, i.e. Free. Only change from this to another approach if it is not working for some reason.
Reasons it may not work include:
- The website you are targeting produces different output per country and does not give you the data you need because of the location of your server – in this case any of the proxy solutions would help
- Your web server has been blocked by the target website. In this case, either the proxy solutions or either of the Scraper API solutions will help
- The target page uses Ajax to populate prices on the page. In this case, you need to simulate a browser and run the JavaScript that puts the prices on the page. Phantom JS, Phantom JS Proxied or Scraper API Rendered will help in these cases.
Proxied
The proxied scrape still uses your server, but loads the data through a proxy. This can be useful to simulate traffic coming from a specific country – e.g. if the website produces country-specific prices.
It can also be useful to avoid bot-blockers. If you scrape 1000 prices per day from your own server, most websites will end up blocking you. If you use 20 proxies, you can reduce this to 50 prices being scraped per proxy IP address.
Phantom JS
This is our own service which uses PhantomJS to simulate a web browser and generate HTML after JavaScript has run. It’s useful for scraping prices from websites which use Ajax to insert prices on the page. This is commonly used by ticketing websites.
Phantom JS Proxied
This is the same as Phantom JS except the scrapes will be rotated through the proxies you configure.
Scraper API
This 3rd party company do automatic proxy-rotation and offer very affordable prices. They have a free plan too for 1000 scrapes per month. They will avoid your website scrapes being stopped by bot-blockers.
Scraper API Rendered
This is the Scraper API version of our own Phantom JS service.
