Skip to main content

Posts

Jmeter-Terraform - Dealing with AWS ELB IP changes during load testing

We ran into an issue when trying to load test our 'new' production environment - the ELB IP addresses change as it silently auto-scales.  And since you're throwing load at it, of course those IPs will change!

When we started the tests, we had updated the hosts file on the master/slave Jmeter nodes, but of course at some point during the day the IPs changed and we got a pile of timeouts.  After much gnashing of teeth, we found the DNS Cache Manager in Jmeter, and figured there were two solutions:

Use the DNS Cache Manager to point to a custom DNS server (that we set up), and have that DNS server deal with keeping the IPs up to date.Write a cron job to run on each node that updates the hosts file, then use the DNS Cache Manager to 'clear cache on each iteration' We elected to do #2 (since our Jmeter infra is built/destroyed a lot, it didn't make sense to have another server in the mix), and here's what we ended up with.
Terraform applies userdata to each EC2 i…
Recent posts

Another round of team health checks is upon us

And I've learned a few things...

Take detailed notes of the discussion - you can use these to refresh memories, provide contrast, use for discussion points, etcTake care to note the action items, the expected outcomes, the action item owners, and update notes on this as time goes on (if you're involved with the teams)If you are doing it remotely, remember that GoToMeeting has a 6 camera limit - just because someone's face isn't on the screen doesn't mean they aren't there!  Keep everyone involved!Don't let newbies off the hook - they tend to have good insight as a relative outsiderIf a topic is voted all one colour, get people to talk about why they voted that wayTry to ensure that everyone is there, especially if the team has a track record of appreciating the time spent doing thisNote outliers/contextual changes on the results doci.e. A team recently took over a function.  Their lack of visibility into that function had previously led them to vote 'yel…

Terraform/Jmeter performance testing - practical experience

Over the last while we've had the opportunity to put our new Jmeter learnings to work.

Bug came up that was only evident under load - we were able to reproduce it in our dev environments!  The dev ran Jmeter off his laptop, and it was enough load to generate the bug.I think I mentioned last time about how the simple act of mapping out a Jmeter script revealed excess calls to our middleware - tickets were created to address this.QA has used it to help draw out issues with a new production environment, but... ...the other day they ran out of steam on their laptops.  So we got to come back to the Terraform/Jmeter setup we built a few months back.  Thankfully everything still worked, and we were quickly (15m) on our feet with 1 master and 6 slaves (c4.large) raising heck.
This is where I will talk about the lessons we learned today... Terraform is amazing and was totally worth the time investmentIf you are testing a cold production environment - ASK ABOUT HOSTS FILE CHANGES!!  We hammer…

Strategyzer webinar learnings: Designing experiments that matter

Strategyzer put on a free webinar to discuss how to design business experiments that matter, with David Bland & Alex Osterwalder providing the discussion.  I stumbled on these guys after being recommended the book 'Business Model Generation' (http://cuddletech.com/?p=910), and subsequently reading the follow-up book 'Value Proposition Design' (my present CTO is encouraging VPD practices) - plus using many of their resources.

(Will add a link to the recording here when it arrives...)
Big lessons
When companies start seeing failure as success, culture is moving.e.g. I am so glad we learned there was nothing in this idea; we saved millions of investment dollarsThe ramifications of incentivizing output instead of outcomeCurrent software development methods reward output (deliver by a certain date, deliver a feature or product) instead of rewarding the outcomeThis leads to focusing on building instead of learningLearn / Build / Measure loopStart with learn!  Almost every…

Rambletron - Patterns for software development orgs?

Lessons I have learned so far; the naivete of youth allows me to write this down...the wisdom of hindsight allows future me to laugh.  :)
Building portfolio vs. delivering value This is first because it's ultimately the most important, and hardest, part to get right.
Suspect that things like this are part of it:
Business Model Generation/Value Proposition Design concepts being appliedi.e. think hard about what you are doing and whyContinuous improvement as a cultural centerpieceSenior leadership defines the big picture, explicitly calling out continuous improvement as a necessary fact of life is keyTeaching all levels to apply BMG/VPD concepts to projectsReally just the act of thinking about something vs. just doing it because you're sure it's the best idea I cannot comment on how you actually solve/execute on this, but I feel very strongly that this is core to sustainable success & innovation.
Systems thinking Books like 'Thinking in Systems' and 'The Fifth…

So what would you say you do here? Release management?

I recently was asked to give a presentation on 'release' - audience: entire company.  Putting that together gave me much time to ruminate on 'what value are you delivering?'.

Note:  I'm writing this as a mental exercise; working on clarifying my purpose.  Much of this is naive (and rambling) philosophy.

While part of the operations department, I have been unofficially tasked with 'release management' for our dev/qa teams.  What 'release management' constitutes, I would posit, depends on your perspective of 'people, process, tools'.  Personally I fall under the 'people, process, tools - in that order' camp, so a good chunk of my focus has been on helping teams 'level up'.  Delivering value to customers (vs. products/features) and product ownership are also high on my priority list.
FundamentalsHelp teams grow by addressing/highlighting risk gaps (e.g. missing non-functional requirements)Help teams understand by facilitating he…

Mobile performance testing using Jmeter & Terraform - getting started

Everyone has performance and scalability on their minds these days, so I am working on a project to bring more visibility to our bottlenecks.  The project will center around simple 'user scenarios':We send a huge batch of push notifications, and a subset of users will open the app.  That simple action actually does a bunch of stuff, like authentication, refreshing feeds, and so on.We make a change that dumps auth, and every user has to log in fresh.We add an endpoint as part of a marketing effort (say, a news page), and a push notification lets users know this new thing is a thing.  The endpoint gets 100k hits in 5m.  Does it survive? In all these scenarios you are dealing with read-only traffic, and nothing super fancy to put together.
Nobody here really knows much about performance testing (aside from previous experiences discovering that mass effort was not worth it, which I agree with), so I wanted to make a point of not turning this into a behemoth.  What we really wanted…