WPI Scrapping Service
Mark M. asked 8 years ago
I have purchased this service as recommended by one of your support clerks after explaining what I was looking at achieving. The following website www.luluwebstore.com is what I am having issues with when grabbing the pricing from the product page; it is giving me the following currency AED (United Arab Emirates), where I want it to give me the BHD (Bahraini) currency.
Correct me if I am wrong, but I Believe that website relies on the country selection input to be able to display the correct currency and not by accessing the website using a proxy?
Any clarification on how I can proceed with this using the Scrapping Service is appreciated...
Regards,
Vasil Y. answered 8 years ago
Hey Marvin. If you select Bahrain as the country and then grab the correct CSS selector from the price you wish to scrape, and use the correct Proxy from the settings page, it should work without a problem. Let me try and walk you through the process:
First of all, please make sure you have the correct CSS selector for the price selected for the particular product you wish to scrape.
Second, go to the Price Comparison Pro settings page in your WP admin area and scroll all the way down until you see "Proxy and Scraper Configuration".
Under this, you will see the following:
"Choose the option you wish to use below. If in doubt, Standard Scraping is what you want. If the scraped page is using Ajax to load prices (you can check by viewing the source of the page and if you can’t find the price, it’s using ajax) then you’ll need to use our Scraping Service. If you wish to simulate scraping from a different country, or you wish to rotate the scraper IP address to avoid detection then you’ll have to use proxies."
The thing which you need in particular is this line:
" If you wish to simulate scraping from a different country, or you wish to rotate the scraper IP address to avoid detection then you’ll have to use proxies."
You will see four options right under this, you can tick each one of them in order to use a different method for scraping. Try the following:
Basic proxied – Uses random proxy from specified proxy list to request page. (You must have at least one specified proxy in your list)
Basic PhantomJS Parsing – Loads data after page has been parsed and javascript executed on Price Comparison Pro’s servers.
Advanced PhantomJS Proxied Pre-parsing – Uses random proxy from specified proxy list to request before loading in PhantomJS as above. (You must have at least one specified proxy in your list)
One of those three settings should work to grab the price in Bahrain currency.
You can test them out one by one and see which one will display the price, but always make sure you grab the CSS selector from the BAHRAIN version of the website.
Mark M. replied 8 years ago Reminder!
Mark M. replied 8 years ago I am grabbing the CSS Selector for Bahrain, the Basic Proxy Does not yield any results and keeps showing “checking…” only and the Basic PhantomJS Parsing yields the same results as the standard scrapping which is with the AED Currency and not BHD.
Vasil Y. replied 8 years ago The key provided (API key), after you bought the subscription, is all that you need. Please try using “Basic Proxy” or “Basic PhantomJS Parsing” and see if you can fetch the price that you need? Technically what proxy you use shouldn’t matter, as we are using a phantom JS server for the PCP.
Mark M. replied 8 years ago Alright, this clears it up a bit, but my dilemma now is that i do not have any proxy sources to use for Bahrain, do you have suggestions for any?
Also is their any place where i can track the requests i have used for the scrapping service purchased?
Vasil Y. replied 8 years ago P.S – if none of this works, please contact me on my personal email – vasil.yordanov1@gmail.com with an admin access information to your website, and I will see what might be wrong with your configuration.
Mark M. answered 8 years ago
Reminder….
Vasil Y. replied 8 years ago I just received your email after you forwarded it again. We are getting to working on the issue asap, expect a solution within a day. Sorry for the wait.
Mark M. replied 8 years ago i haven’t received any emails!
Vasil Y. replied 8 years ago Reminder about what? The issue was resolved with an email to the customer. We are awaiting an answer with the requested information which we have not yet received.
Mark M. replied 8 years ago Reminder!!!!
Mark M. replied 8 years ago reminder
Mark M. replied 8 years ago sent
Vasil Y. replied 8 years ago Since nothing else seems to be working, I will need admin access to your website in order to determine what’s wrong with your configuration. Please share it on my personal email – vasil.yordanov1@gmail.com – and i will have a look.
Vasil Y. answered 8 years ago
Hey there, here is the direct answer from the dev who had a look at your issue:
" I don’t think there is a direct solution for the lulu webstore site. The problem is that the site is always delivered first with the AED currency set. This is rendered on the lulu web store server. When you select a language, it stores a cookie on your browser and then reloads the site. The site can then get the stored cookie and render the desired currency.
Generally speaking, it is possible to get a url that links to a product with a desired language / currency setting. Unfortunately the luluwebstore doesn’t work like this – the url for AED is the same as BD. As far as I can tell, PCP doesn’t support setting cookies in the request header at the moment.
The only workaround I can suggest just now would be just to fetch the AED price, and inject a script that performs a conversion to BD – this wouldn’t be ideal, as BD prices might not follow an exact or static conversion formula, but it might be close enough. It wouldn’t take me long to write a script to do this and insert it into the customer’s site, if they think this approach would be acceptable."
Please let us know if you would like us to create the script mentioned here, as this would be the only work around at the moment.
Kind regards.
Dave Hilditch replied 7 years ago Yes sorry – it’s implemented – run your updates. Let me know if any issues. There’s an article with all the changes here: https://www.superspeedyplugins.com/2017/04/28/major-new-upgrade-price-comparison-pro-released/
Mark M. replied 7 years ago any news regarding the last comment/implementation?
Mark M. replied 8 years ago Great, Any ETA i should be looking at?
Jared Varley replied 8 years ago Hi Moka,
I am implementing a new setting into the Price Comparison Pro settings page, to enable the input of a cookies.txt formatted file (see: http://www.cookiecentral.com/faq/#3.5). If you then use any of the scraper PhantomJS based methods while using the plugin it will then automatically load this file for all requests.
We will let you know when the new version is ready for use.
Thanks,
Jed.
Dave Hilditch replied 8 years ago Yes that sounds right. I’m bringing Jed here to comment as he’s inside price comparison pro code right now working on an upgrade.
Mark M. replied 8 years ago Sounds Promising, i believe i should be able to grab the site-config cookie details which includes the location id for that website through the chrome browser, just to check i was able to access the detail = Content: {“LocationId”:”13241″,”DeliverySlot”:null,”DM”:”H”}.
Does this sound right?
Dave Hilditch replied 8 years ago Hi – my apologies – I had not read this properly. I see the issue you are describing – and the solution, to allow users to pass a cookie along with the scrape. If we modify the code to give you the option to add a cookie when scraping would you be able to identify the cookie name and value needed to enter into the scraper config?
Mark M. replied 8 years ago but still no update on the issue i have encountered?
Dave Hilditch replied 8 years ago There was an update to Price Comparison Pro released last week including a number of fixes. There is another update due out end of this week which will add an extra selector to identify SALE prices.
Dave Hilditch replied 8 years ago There was an update to Price Comparison Pro released last week including a number of fixes. There is another update due out end of this week which will add an extra selector to identify SALE prices.
Mark M. replied 8 years ago Anything new regarding the PCP implementation?
Vasil Y. replied 8 years ago Hey there. I will direct your request for the feature to the dev team and see what can be done about it. Meanwhile, I will refund your 3.99 service later today, you don’t need to provide us with any information. 🙂
Mark M. replied 8 years ago The script will not be ideal at all, that particular business has stores in Multiple Countries. The AED currency is meant for products sold in the country of Dubai and the BHD currency is meant for Bahrain which are two different countries / Pricing for any particular product… Please get back to me if there will be any implementation / ETA on being able to have PCP Support cookies as mentioned (if possible at all).
Additionally i’d like to get a refund on the Purchased Scrapping Service. Please get back to me on what i need to do from my end to make it happen.
Dave Hilditch Staff answered 7 years ago
This has been implemented – an article discussing the changes was released here:
https://www.superspeedyplugins.com/2017/04/28/major-new-upgrade-price-comparison-pro-released/
To use cookies, upgrade your Price Comparison Pro and paste the cookies you wish to use into the little cookie text box.
Mark M. replied 7 years ago Kind reminder…
Mark M. replied 7 years ago thanks
Dave Hilditch replied 7 years ago Yes that’s correct. If you’re having this many issues, there must be something wrong with your config. I’ll be able to look at the email you sent shortly – I’m in the middle of a release just now.
Mark M. replied 7 years ago not sure if i am understanding it correct, but the update woocommerce price option should reflect the lowest price from PCP in the woocommerce item and update its price as well?
Mark M. replied 7 years ago noted, mail sent
Dave Hilditch replied 7 years ago Hi – for woocommerce price/images option, you need to enter an images selector. Visit your settings page, and choose to save prices and/or images. For the cookie option, paste in your cookies you would like to persist in the standard format as per the link I provided.
If you need further help – email me on support@superspeedyplugins.com and provide login details so I can see your configuration and provide a URL where I can see an example of your price comparison in action.
Mark M. replied 7 years ago alright, i have subscribed for that service, is there a write up on the cookie option, i cannot seem to be able to get it to work. also the update woocommerce price/images option. does woocommerce have to be setup in a specific way or am i misunderstanding this feature?
Dave Hilditch replied 7 years ago Currently, you need to use the scraper service to use the cookies file. Next release I’ll add the cookies to the direct-scrape code. Sorry, I thought it was fully implemented, but just checked and it’s only implemented on scraper service. There’s a free trial of scraper service.
Mark M. replied 7 years ago do i need any additional services to be able to use the cookies option?
Dave Hilditch replied 7 years ago Hi Moka – we’re using the standard cookie file format. The easiest way to generate your cookies.txt file is to visit the page you need to scrape, generate the cookie, then copy it from the browser e.g. using this guide: http://stackoverflow.com/questions/21919156/how-do-i-copy-cookies-from-chrome
Mark M. replied 7 years ago and please elaborate on how the cookies.txt file should be setup?
Mark M. replied 7 years ago also reading through the update, i believe i would have to be subscribed to the PhantomJS Scraping Service to be able to use the cookies? can you please clarify on this?
Mark M. replied 7 years ago that’s great news, is there anywhere i can test it out before repurchasing it again? as i have already received a refund because if this.
