After we’d scraped our Tech Director, Stefan Antonowicz, off of the ceiling of the office on Tuesday night, he composed this message on a tear-stained paper towel before going home to subject himself to a scourging and a numbing whiskey IV. Little does he know that we have a frat house-style paddle line in store for him today. I think cream pies might be involved, too. —Gareth
This time, everything booted up as expected and scaled nicely. I sat back in my chair, well pleased with myself, until I got an IM from the sysad helping me out:
“Error establishing a database connection”
“Uh oh,” I thought. But we persevered, or attempted to — I hoped that this was just a result of all the pages attempting to generate their cache files, but after ten minutes or so, I saw that the load on the database server was unusually high. The error message was being thrown once every handful of connect attempts (your mileage may have varied, of course). Not OK. We rolled back the DNS changes and went back to make some tweaks.
My understanding at this point is that the AWS cloud instance for the database wasn’t beefy enough. We’re reconfiguring things to run on a bigger virtual machine, and I’m taking a look at our benchmarking application to see why it didn’t pick up the db load during testing. We’re looking to try again next week or two once we have the evening resources available again and get a few other projects off our plate.
Anyway, we’re looking at this as a partial victory — the application seemed to hold up well under the load, and when they couldn’t reliably connect to the database, they started spawning new instances (exactly what we wanted). Not fully back to the drawing board — let’s call it back to the tweaking board. You never know when boingboing or slashdot will strike, after all, and I want to be sure everyone is getting the best experience possible.
Thanks for bearing with us. Third time’s a charm, right guys?
ADVERTISEMENT
