Practical Technology

I'll use (this|these) blog entr(y|ies) as a brain dump for a new proof-of-concept project I'm working on.  Long story short, we investigated build/deploy automation and concluded config mgmt was where it was at (after extensive due diligence). We have some hurdles to overcome: Everything build/deploy 'process' today is manual, frankly just due to bad original implementation that was never corrected We have a LOT of VS2010 setup projects - no longer a thing in VS2013 (2012 actually, and the MS plugin for '13 to support them apparently is buggy) We don't use software versioning (yes, it's boggling - we use the build version) We have both Windows services and web apps/svcs We use BizTalk 'proof of concepts' tend to be adopted as production tools, so I have to build it right the first time The initial gist (hm...should this be on github?) of this is straightforward.  Jenkins will build from TFS2010, but instead of bad MSIs, we get raw filedu...

More fun with Logstash, Nagios, and IIS logging.  We have all of our environment website logs dump into Logstash via NXlog, and stuff is tagged/outputted. The below gives us a passive monitor that CRITs on 400/500 codes and OKs on 200/300 codes.  Not sure how this'll really work out, but the monitor should get flooded with enough 400/500 during a serious outage to cause a notification to go out.  We'll have to test that.  This is working like a charm at the moment, though! 20-filters_tagging.conf <snipped> if [SourceName] == "IIS" {         if [s-ip] =~ /^192.168.(\d{1,3}).(\d{1,3})/ {                 grok {                         match => ["sc-status", "[2,3,4,5]\d\d"]                         add_tag => ["nagios_check_iislog","UAT","IIS"]        ...

This took way longer than it should have to get going...so here's a config and brain dump... Why? You want to have a central place to analyze Windows Event/IIS/local application logs, alert off specific events, alert off specific situations.  You don't have the budget for a boxed solution.  You want pretty graphs.  You don't particularly care about individual server states.  (see rationale below - although you certainly have all the tools here to care, I haven't provided that configuration) How? ELK stack, OMD, NXlog agent, and Rsyslog.  The premise here is as follows: Event generated on server into EventLog NXlog ships to Logstash input Logstash filter adds fields and tags to specified events Logstash output sends to a passive Nagios service via the Nagios NSCA output The passive service on Nagios (Check_MK c/o OMD) does its thing w. alerting OMD Open Monitoring Distribution, but the real point here is Check_MK (IIRC Icinga uses this...). ...

One of our lines of inquiry for using the ELK stack was monitoring/alerting.  The best way is to use Logstash to directly send to Nagios, however this proved to be more difficult than anticipated, so the next best route was utilizing the simplicity of the Elasticsearch query via curl to get an alertable value. I found one other person doing this, but didn't have the heart to ask for their script!  Decided to spend some time figuring it out for myself and share the results.  If we truly can't get Logstash output working, then we'll come back to this with a full Nagios service script.  This gives you everything but the Nagios service... curl -XGET 'http://kibana.domain.com:9200/_count' -d '{"query": {"filtered":{"query":{"match":{"EventID": "4624"}},"filter":{"range": {"@timestamp": {"gt": "now-1m"}} }} }}' 2>/dev/null | egrep -o '"coun...

Let me preface this with a disclaimer: I am not the creator/inventor of any of these ideas, this is just a combination of Google and the context of our environment.  For all I know people have been doing this for ages...it's new to me! Our context Our 'prod' environment is in a managed hosting context for various reasons (compliance, past decisions, etc).  This means that from a 'traditional' monitoring point of view we are covered.  Anything beyond that we're on our own.  Also, we are 95% Windows. What is monitored today: Host/svc, host/svc, ports alive, HTTP GET = XYZ, is CPU crazy, is RAM maxed, etc... Picking up what's left:  Logs! Centrally aggregate all logs and query that pile of data to get alerts.  i.e. the services/applications we run (written in-house) dump stuff to the event log, among other places. What are our goals? Get visibility into what our svs/apps are doing - know (be alerted) that svcX is dumping errors right now, not a wee...

To keep it short... Swap went well Managed to snap some key coolant piece was the only issue at the time, decided to do the coolant re-route Also managed to incorrectly assume that the clutch would not need re-adjusting because...hey I haven't changed clutch components or drained the fluid or anything... The above caused me to run for 2 weeks on a misadjusted clutch, so the shiny FM Stage1 clutch is now pretty much toast - we played with the adjustment afterwards to try and salvage it...no luck I've decided to get off the 'because racecar' bandwagon - ordered the OEM-level value clutch, and I'll get the 1.8 flywheel I have resurfaced The de-powered steering thing has become a saga...continue to have 'stiction' issues Had the bushings done (IL Motorsports rubber ones), only the mechanic didn't replace the rear bushings due to some miscommunication.  The good news is that alignments are now holding, the bad news is that 1 week after the fronts w...

Well, some pretty fantastic exhaust leaks have cropped up sooner than I would have hoped, so the decision point came to this:  Put in a stainless 1.6 system, or advance the plans to do a 1.8 swap.  Seems kinda wacky, but moving to a 1.8 means I have a much larger selection of parts, and more importantly used parts, to choose from. Work has already started on the '96 exhaust system I picked up - been cleaning, grinding, wire brushing, and even welding! (someone decided a self-tapping screw was an acceptable way to plug a hole in the cat)  I also learned a new trick to get studs out of a (rusty) cat (alytic converter). The plan is to use a full 94-97 exhaust system painted with POR-20 - a small expenditure that should help keep the rust away.  The engine will most likely also be 94-97, but JDM sourced (JDM for availability, not JDM yo) (if they have an NB1 engine for not much more, will get that).  I may end up getting a transmission as well, depending on what...