We were so close! (or Epic Fail, Part Deux)

After we’d scraped our Tech Director, Stefan Antonowicz, off of the ceiling of the office on Tuesday night, he composed this message on a tear-stained paper towel before going home to subject himself to a scourging and a numbing whiskey IV. Little does he know that we have a frat house-style paddle line in store for him today. I think cream pies might be involved, too. —Gareth

This time, everything booted up as expected and scaled nicely. I sat back in my chair, well pleased with myself, until I got an IM from the sysad helping me out:

“Error establishing a database connection”

“Uh oh,” I thought. But we persevered, or attempted to — I hoped that this was just a result of all the pages attempting to generate their cache files, but after ten minutes or so, I saw that the load on the database server was unusually high. The error message was being thrown once every handful of connect attempts (your mileage may have varied, of course). Not OK. We rolled back the DNS changes and went back to make some tweaks.

My understanding at this point is that the AWS cloud instance for the database wasn’t beefy enough. We’re reconfiguring things to run on a bigger virtual machine, and I’m taking a look at our benchmarking application to see why it didn’t pick up the db load during testing. We’re looking to try again next week or two once we have the evening resources available again and get a few other projects off our plate.

Anyway, we’re looking at this as a partial victory — the application seemed to hold up well under the load, and when they couldn’t reliably connect to the database, they started spawning new instances (exactly what we wanted). Not fully back to the drawing board — let’s call it back to the tweaking board. You never know when boingboing or slashdot will strike, after all, and I want to be sure everyone is getting the best experience possible.

Thanks for bearing with us. Third time’s a charm, right guys?

Discuss this article with the rest of the community on our Discord server!

Gareth Branwyn is a freelance writer and the former Editorial Director of Maker Media. He is the author or editor of over a dozen books on technology, DIY, and geek culture. He is currently a contributor to Boing Boing, Wink Books, and Wink Fun. His free weekly-ish maker tips newsletter can be found at garstipsandtools.com.

View more articles by Gareth Branwyn