The way FacetWP works is pretty poor to be honest and it’s no wonder it’s slow if you look at their underlying code – they basically load your entire database into an object – this works really well and fast below 5000 products or so, but above that it’s very difficult to make that code pattern work fast.

I recommend you take a look at my Faster Woo Widgets plugin to handle your filters instead. The way it has been coded makes it scale to millions of products with sub-second filtering.

Can you also edit your question and explain more about your 50,000 terms in your taxonomy? That’s quite a lot – should still be fast – the thing that’s difficult to optimise in this area would be if you have 50,000 taxonomies.