I am also aware that Elasticsearch is mullered. That might need to be rebuilt from the ground up, and ES indexing jobs are backing up Sidekiq which is leading to memory pressure.
Who knew that keeping a Rails app running was such hard work! 😅🥵
We're now running out my basement. People's home feed are rebuilding on a new, empty Redis. The old one had its Sidekiq queue drained; apart from the home feed thing this should be a seamless cut over and way faster than it's been since Friday.
We are still sending HTTP from France to Canada, which is not ideal. So, I'll look at getting a VM in North America to be our proxy in order to avoid these oceanic voyages.
Aaand now a DigitalOcean droplet in Toronto is providing us with a stable IP address and fronting the whole show (and autorenewing TLS certs, hopefully).
Those of us in North America should have the best ttw experience so far. My European friends: sorry, but also the server should be broken less often so that's also nice. ❤️
#admin intensifies: I've got a new, less phone company, router. I'll need to take some downtime tomorrow to do the swap, including moving a fibre optic SFP into a switch and hoping for the best. I can always rollback if something goes wrong. This will hopefully be the last scheduled downtime for ttw for a bit.
@selfsame it's on my list. The old index was corrupted so I need to start from scratch, which took a few days to index last time. That said, we're on better servers now, so maybe it will be quicker.