Practical Technology

We ran into an issue when trying to load test our 'new' production environment - the ELB IP addresses change as it silently auto-scales.  And since you're throwing load at it, of course those IPs will change! When we started the tests, we had updated the hosts file on the master/slave Jmeter nodes, but of course at some point during the day the IPs changed and we got a pile of timeouts.  After much gnashing of teeth, we found the DNS Cache Manager in Jmeter, and figured there were two solutions: Use the DNS Cache Manager to point to a custom DNS server (that we set up), and have that DNS server deal with keeping the IPs up to date. Write a cron job to run on each node that updates the hosts file, then use the DNS Cache Manager to 'clear cache on each iteration' We elected to do #2 (since our Jmeter infra is built/destroyed a lot, it didn't make sense to have another server in the mix), and here's what we ended up with. Terraform applies userdata

And I've learned a few things... Take detailed notes of the discussion - you can use these to refresh memories, provide contrast, use for discussion points, etc Take care to note the action items, the expected outcomes, the action item owners, and update notes on this as time goes on (if you're involved with the teams) If you are doing it remotely, remember that GoToMeeting has a 6 camera limit - just because someone's face isn't on the screen doesn't mean they aren't there!  Keep everyone involved! Don't let newbies off the hook - they tend to have good insight as a relative outsider If a topic is voted all one colour, get people to talk about why they voted that way Try to ensure that everyone is there, especially if the team has a track record of appreciating the time spent doing this Note outliers/contextual changes on the results doc i.e. A team recently took over a function.  Their lack of visibility into that function had previously led the

Over the last while we've had the opportunity to put our new Jmeter learnings to work. Bug came up that was only evident under load - we were able to reproduce it in our dev environments!  The dev ran Jmeter off his laptop, and it was enough load to generate the bug. I think I mentioned last time about how the simple act of mapping out a Jmeter script revealed excess calls to our middleware - tickets were created to address this. QA has used it to help draw out issues with a new production environment, but... ...the other day they ran out of steam on their laptops.  So we got to come back to the Terraform/Jmeter setup we built a few months back.  Thankfully everything still worked, and we were quickly (15m) on our feet with 1 master and 6 slaves (c4.large) raising heck. This is where I will talk about the lessons we learned today... Terraform is amazing and was totally worth the time investment If you are testing a cold production environment - ASK ABOUT HOSTS FILE