- Discord Performance Community Server and some new server rules - November 27, 2024
- Calling for early access test files for Super Speedy Imports - November 27, 2024
- Cyber Monday Deals Live – grab a bargain before the Scalability Pro licensing model changes! - November 21, 2024
Since Christmas, I’ve been working on a plugin called Super Speedy Imports. This plugin is coming along nicely with some clients already using it in production.
Since the inception of my business, I’ve been working on optimising WordPress in various ways and speeding up imports has always been a concern that I’ve tried to (successfully!) speed up through my Scalability Pro plugin.
However, ultimately, a client came to me at Christmas with a challenge to be able to import 1 million items inside a day, or at least to be able to update the prices and stock levels inside a day.
“Inside a day??!” I thought? That’s too generous.
Table of Contents
Why other import plugins are slow
Every import plugin I’ve seen solves WordPress imports by reading your CSV files one line at a time and importing each item one by one. This works, but it’s an incredibly slow approach with many actions being triggered after each item import causing recounts and all kinds of slowness.
The client in question reported that the import plugin we was using was STILL running to import 1 million products after a week! So – a new approach and a new architecture was required.
Super Speedy Imports to the rescue
I architected Super Speedy Imports to import your CSV file to a staging table and then run ‘massage’ functions on this staging table to modify the data however required. It then runs various stages to fetch from the database any existing post IDs, existing terms and existing images. Once the table has been fully massaged, all that’s left is to run some big SQL statements, for example 1 SQL statement to update all existing postmeta, 1 SQL statement to update all existing wp_posts, 1 SQL statement to insert any missing posts etc.
While this works extremely well (we’re currently at 1 million products being imported in 100 minutes) it introduces another problem when it comes to image import.
Super Speedy Imports caused my server to get banned from image providers
It turns out that trying to import images as quickly as happens inside Super Speedy Imports caused my server to get banned from the image CDNs. They see 1000 requests per second and then either throttle traffic to and from my IP or ban my server completely!
I guess it’s quite nice for my server to get banned because I made the imports far too fast!
So, I needed a solution to divorce the image retrieval from the import. I already have my External Images plugin, but this requires you to have a separate workflow for your images where they’re uploaded to some external source. I wanted a solution for everyone else who still wants to have their images inside WordPress.
Delayed Image Import plugin to the rescue
I made a nice simple plugin that I’ve submitted to wordpress.org. With everything happening at wordpress.org right now, I’m sure it might get into their repo by Christmas, but in the meantime any customers of Super Speedy Plugins can get this plugin for free.
The way it works is that you place your featured image URL into a postmeta field called delayed_image_url and you put your gallery images (if importing products) into the delayed_gallery_image_urls field. This makes the import incredibly fast since it’s not fetching images, it’s just inserting to or updating wp_postmeta.
Then, as soon as the post/product/CPT is accessed – either through wp-admin or the front-end, in any way, e.g. even if it’s in an archive list – at this point, the image import happens.
The results of this are phenomenal and actually really help all existing import plugins out there already since image import is one of the parts that slows down your imports the most.
So, regardless of which import plugin you are using, if you are importing images then my new Delayed Image Import plugin will speed that whole process up.
It has another nice side effect too – only items which are actually used or fetched will have their images downloaded, so if you have a massive drop shipping or affiliate site then this will reduce your storage requirements to only store images for products people are accessing.
Release Date for Super Speedy Imports
While Delayed Image Import speeds up your existing imports, you’re still going to see a phenomenal speed boost from using Super Speedy Imports. If, in the past, you’ve been limiting the size of your site because of the slowness of imports then that problem is now in the past.
I plan to release Super Speedy Imports to the public for Cyber Monday weekend, so that’s about 6 or 7 weeks away. If you want access to this plugin earlier than this, I am allowing early access for those who sponsor development of the plugin. In return for sponsoring development of the plugin, you will either get a 1-site, 5-site or unlimited site license for life – depending on the level of your sponsorship.
If you are interested in this, contact me through Discord.
Get access to the Delayed Image Import plugin today!
If you are an existing customer, you can get immediate access to my new Delayed Image Import plugin. In your account area, I’ve added a new Early Access tab where you can find out about any early access plugins we currently have available.
All you need to do to gain access is complete a short imports-related survey to help me develop the plugins further.
Remember, it works with ALL existing import plugins and any import technique you are currently using will be improved with this free plugin.
Complete the survey now for immediate access to Delayed Image Imports
Summary
I’m not going to stop until every performance problem in WordPress has been solved. I’m looking to make a big splash this Cyber Monday with my new Super Speedy Imports plugin, but in the meantime all customers get access to the Delayed Image Import plugin which arose out of the creation of Super Speedy Imports because I made that plugin so fast I got banned from image provider CDNs!
