I am also aware that Elasticsearch is mullered. That might need to be rebuilt from the ground up, and ES indexing jobs are backing up Sidekiq which is leading to memory pressure.
Who knew that keeping a Rails app running was such hard work! 😅🥵
We're now running out my basement. People's home feed are rebuilding on a new, empty Redis. The old one had its Sidekiq queue drained; apart from the home feed thing this should be a seamless cut over and way faster than it's been since Friday.
We are still sending HTTP from France to Canada, which is not ideal. So, I'll look at getting a VM in North America to be our proxy in order to avoid these oceanic voyages.
Aaand now a DigitalOcean droplet in Toronto is providing us with a stable IP address and fronting the whole show (and autorenewing TLS certs, hopefully).
Those of us in North America should have the best ttw experience so far. My European friends: sorry, but also the server should be broken less often so that's also nice. ❤️
#admin intensifies: I've got a new, less phone company, router. I'll need to take some downtime tomorrow to do the swap, including moving a fibre optic SFP into a switch and hoping for the best. I can always rollback if something goes wrong. This will hopefully be the last scheduled downtime for ttw for a bit.
@insom will we get search back?
@selfsame it's on my list. The old index was corrupted so I need to start from scratch, which took a few days to index last time. That said, we're on better servers now, so maybe it will be quicker.
@selfsame several days later: it's still going.
Okay, it's looking pretty good. Thanks Docker! But I am also having electricians in tomorrow so just in case they need to knock the whole panel off, I'll wait until the middle of the week to actually move ttw over. I hope y'all can put up with the sluggish speed until then -- I know it's annoying 🙏