Scraping queue not processed
Lucas V. asked 4 years ago
Hi,
When trying to setup the first product to scrape prices for, I'm getting stuck because whatever I do, the scraping queue doesn't seem to get processed.
I checked the price comparison settings
- Removed https etc.
- Used Selector Gadget for the right CSS selector
- Used Scraper API to check if it is my server or something
Whatever I do, the queue doesn't seem to get processed, no log is created, no errors appear. The scraping urls remain in queue, and it keeps trying until I close the page.
Can you help me out?
Thanks!
Lucas V. answered 4 years ago
Just installed the WP Crontrol plugin, but I don’t see any error messages from them either. Just nothing seem to happen, although the cron is running every minute, as far as I can see. Pretty clueless to be honest..
Dave Hilditch Staff answered 4 years ago
For anyone who experiences this issue, please check the following:
- Confirm you can scrape pages normally, e.g. on a woocommerce product detail page or a page where you have placed the comparison shortcode
- You can view debug from the JavaScript console – there is a lot of extra debug gets printed to the console when you click ‘Check Prices’ manually in the comparison table
- If the issue persists, please raise a question and provide wp-admin access so I can view what’s going on.
If you don’t provide wp-admin access, it’s v hard to diagnose issues. You can provide access to a dev or test server, but I’ll most likely need access to debug.
Lucas (OP), if this is still persisting, can you please update your question and provide creds in the hidden area please.
Dave H. Staff replied 4 years ago For extra debug info, if you enable debug logging then any queue processing or scraper processing errors will get output there: https://www.superspeedyplugins.com/2020/09/02/how-to-enable-debug-log-to-collect-wordpress-error-messages/
Dave Hilditch Staff answered 4 years ago
It’s not finding the config at all – in the domain, you have included a forward slash / at the end of the domain name.
https://i.imgur.com/7jGSQye.png
Instead of http://www.urbanebikes.nl/ it should be http://www.urbanebikes.nl (for example) or even just urbanebikes.nl
I’ve created a bug for this, I’ll fix it so others don’t fall foul of this.
Next thing I had to alter – you didn’t have a price for this example product and you didn’t have an external URL set – without those, WooCommerce will not display the add to cart button and you had Price Comparison Pro configured to display before the add to cart button – so nothing was displaying!
Then when I view the page, I see an error of ‘decimal symbol not defined’ – this is related to the config not being picked up because you entered the / on the domain name:
https://i.imgur.com/N4jPmNg.png
I’ve created a bug report for the default config missing the decimal symbol here:
https://www.superspeedyplugins.com/idea/fix-default-config-to-include-a-decimal-symbol/
For now, I altered your domain config to remove the / and then refreshed the page and I see a price on one of the products (the other is 404ing to the URL you gave). The price is 4.75 which for a motorbike is incredibly cheap!
https://i.imgur.com/FdJqKSK.png
The reason for this is because in your (now working) config, you have , entered instead of . for the decimal separator. The page you are scraping uses commas for thousands and periods for decimals, so you should enter a period here.
Once I’d changed the config, I see the correctly scraped price appearing:
https://i.imgur.com/FXyGUot.png
Sorry it took me so very long to help you with this. I’ve hired 2 people to help me now.
Dave H. Staff replied 4 years ago That’s great news! Thanks for letting me know
Lucas V. replied 4 years ago Thanks Dave for helping me out. Hopefully this helps others as well. With this, I’ve setup other products as well with correct scraping.
I’ve added this bug to the current bug fix sprint here – https://trello.com/c/FooPbLkM/134-bug-and-perf-roundupany-plugin-this-sprint-includes-small-bugs-and-perf-improvements-from-any-plugins-other-than-fww
Very disappointed not to hear from you @Dave H. for almost a month now, leaving the plugin unusable at the moment. I feel a bit scammed, especially given the license fees paid. When could we expect a response to help us solve the problem?
@dave H. Wondering when you get back to us, because the problem is definitely still there, leaving the plugin unusable at the moment.
been having the same issue described here for a while too…
Just installed the WP Crontrol plugin, but I don’t see any error messages from them either. Just nothing seem to happen, although the cron is running every minute, as far as I can see. Pretty clueless to be honest..
Do you have your WP Cron running? If you install the WP Crontrol plugin or something similar you can see your cron jobs in the queue and any error messages from them.