- Cyber Monday Deals Live – grab a bargain before the Scalability Pro licensing model changes! - November 21, 2024
- Early access to new free plugin which speeds up imports even more! - October 14, 2024
- ChatGPT really sucks at SQL - August 9, 2024
Many of you have been asking for updates to our popular plugin, Price Comparison Pro. Today, there’s a new release which adds most of the functionality you’re looking for.
Table of Contents
Improved Debugging & Logging
In this major upgrade for Price Comparison Pro, we’ve massively improved the debugging features available in the plugin. So – you can set up your sites, get them scraping prices or images, then if there are any failures you will be able to see them and diagnose them easily.
In addition to the above useful logging and debugging info, you can test scrapes using different selectors and different scraping techniques.
Verbose Logging
If you enable WP_DEBUG then Price Comparison Pro will output detailed info to your debug.log file explaining each part of the scraping and data gathering process.
Background Scraping
We’ve added the ability to scrape in bulk and to rescrape prices and images at whatever schedule you like.
Our background scraping does a round-robin style form of scraping – rather than scraping 10 things from the same domain, it will rotate through different domains to reduce the likelihood of your server being banned.
You can scrape and rescrape everything with ease and scraping is then handled by WordPress cron.
Datafeedr Integration
We were already quite tightly integrated with Datafeedr, but now we’ve made it even easier for Datafeedr users to make the best use of Price Comparison Pro.
For example, datafeeds frequently provide poor quality images. Now you can bulk copy the Datafeedr direct_url field and then bulk scrape images and/or prices. That means, your affiliate store doesn’t need to use the grainy, tiny affiliate images – now you can have the large hires images available directly from the merchant’s stores.
External Images gallery compatibility
In conjunction with External Images gaining ‘gallery’ compatibility, Price Comparison Pro integrates seamlessly. That means, you don’t need to download any of these images – you can use Price Comparison Pro to scrape the image URLs and External Images to display the gallery on your product pages without any extra storage space used on your server.
HTML Caching
We’ve added proper HTML caching to reduce the hits your server makes against the websites you’re scraping. This is especially important when you’re setting up your CSS/Regex/XPath expressions to grab data elements.
ScraperAPI Integration
We’ve added an alternative means of scraping. You can still use your own server, or proxied requests, or our own scraping service. But now you can also use the Scraper API service – one of the best things they provide is their ‘rendered’ scraping. This means they use a browser fetch the page and run JavaScript on the page. This makes it possible to scrape data from difficult to scrape sources, like ticketing services or other sites which don’t add prices or images to the page until the JavaScript runs.
You can still use our own scraping service for this too which also runs JavaScript and renders the page using a (headless) browser.
Upcoming Features
Here’s a quick list of the stuff that never made it into this upgrade, but which is coming soon:
- Price History – we will show price history to your users, helping them figure out if now is a good time to buy
- Different views – we will add multiple different views to give you options for displaying prices and price history
- Price Alerts – one of the major issues with affiliate marketing is that users are in the ‘research’ phase. Price alerts will make it easy for your users to add their email address and receive an alert to come back to your page once the price drops to a certain level, massively increasing the amount of affiliate revenue you earn
- Centralised scraping database – we are building a repository of selectors for different domains which will mean you can get set up and running immediately without any configuration
Summary
This upgrade adds a lot of new features, probably the most important being far easier testing and debugging. The Datafeedr integration makes it far easier to start scraping images for your existing affiliate store and the background scraping lets you process all this stuff in the background, rather than waiting for users to visit your product detail pages.
