After we’d scraped our Tech Director, Stefan Antonowicz, off of the ceiling of the office on Tuesday night, he composed this message on a tear-stained paper towel before going home to subject himself to a scourging and a numbing whiskey IV. Little does he know that we have a frat house-style paddle line in store for him today. I think cream pies might be involved, too. —Gareth
This time, everything booted up as expected and scaled nicely. I sat back in my chair, well pleased with myself, until I got an IM from the sysad helping me out:
“Error establishing a database connection”
“Uh oh,” I thought. But we persevered, or attempted to — I hoped that this was just a result of all the pages attempting to generate their cache files, but after ten minutes or so, I saw that the load on the database server was unusually high. The error message was being thrown once every handful of connect attempts (your mileage may have varied, of course). Not OK. We rolled back the DNS changes and went back to make some tweaks.
My understanding at this point is that the AWS cloud instance for the database wasn’t beefy enough. We’re reconfiguring things to run on a bigger virtual machine, and I’m taking a look at our benchmarking application to see why it didn’t pick up the db load during testing. We’re looking to try again next week or two once we have the evening resources available again and get a few other projects off our plate.
Anyway, we’re looking at this as a partial victory — the application seemed to hold up well under the load, and when they couldn’t reliably connect to the database, they started spawning new instances (exactly what we wanted). Not fully back to the drawing board — let’s call it back to the tweaking board. You never know when boingboing or slashdot will strike, after all, and I want to be sure everyone is getting the best experience possible.
Thanks for bearing with us. Third time’s a charm, right guys?
6 thoughts on “We were so close! (or Epic Fail, Part Deux)”
Comments are closed.
ADVERTISEMENT
Join Make: Community Today
Thanks for the updates. It’s encouraging to see how all this is working and the victories and defeats you face and the fact you still move forward. Thanks for keeping us readers in the loop.
Could you imagine if huge corporations did stuff like this?
Standard Car Dealer Update: Just finished testing out the new fly by wire and break by wire system. Car flys well, brakes not so good. Will have to get a new test wall on the next go around.
Standard Food Manufacturer Update: Just finished putting the final touches on the new energy bar. Was really heavy with chocolate bits and made our test subjects gag. Fell apart when lifted up to your mouth. Will try adding more xanthan gum to get to stick together better and acquire new test subjects.
As a DBA I have seen this before. People decide they are going to virtualize an environment and then skimp on resources for the database server. It does not work. Unless your database server is very lightly loaded, you need to give it virtual resources to match the original physical environment, plus some overhead for the virtual system itself. Don’t try to pinch pennies there.
I can’t remember if I read it on the make blog, or somewhere else, but there was a guy who was winning races with a frame made of steel tubing. It was much heavier than the carbon fiber or titanium alloys his competitors were using, but because it was far easier to repair and modify, he won.
So, glad that you’re making progress, moving to cloud services is a bit more difficult than it looks to be sure.
Hope your new system is robust, but also internally visible to you, so you can weld some steel tubing where it’s needed.
-RG