Logstash configuration update - outputs and concluding note




Config files like this:
  • 90-outputs_start.conf
  • 91-outputs_elasticsearch.conf
  • 92-outputs_nagios.conf
  • 93-outputs_graphite.conf
  • 99-outputs_end.conf


Outputs
90-outputs_start
output {
Orrrrrganization.

91-outputs_elasticsearch
# Old embedded ES
#elasticsearch {
#               host => "127.0.0.1"
#       #cluster => "elasticsearch"
#       }
# New ES cluster
elasticsearch {
        host => "es01.domain.com"
        cluster => "site.elk.elasticsearch"
        }
Just a simple ES output - note that I left in the old embedded config stuff for some reason.  Another reason to get these things into Git!!

92-outputs_nagios
# This sends nagios output only if the service and status fields
# are populated
if [nagios_service] =~ /./ and [nagios_status] =~ /./ {
        if "nagios_check_eventlog_biztalk" in [tags] {
                nagios_nsca {
                                        host => "omdbox.domain.com"
                                        port => 5667
                                        send_nsca_config => "/etc/nagios/send_nsca.cfg"
                                        nagios_host => "%{Hostname}"
                                        nagios_service => "%{nagios_service}"
                                        nagios_status => "%{nagios_status}"
                                        message_format => "%{EventTime} %{EventID} %{SourceName}"
                }
        } #end nagios eventlog_biztalk
        if "nagios_check_iis" in [tags] {
                nagios_nsca {
                                        host => "omdbox.domain.com"
                                        port => 5667
                                        send_nsca_config => "/etc/nagios/send_nsca.cfg"
                                        nagios_host => "%{hostname}"
                                        nagios_service => "%{nagios_service}"
                                        nagios_status => "%{nagios_status}"
                                        message_format => "%{EventTime} %{cs-method} %{cs-uri-stem} %{sc-status}"
                }
        } #end nagios iislogs
}# End nagios service/status
This is where we send to Nagios, and the reason for separate nsca outputs?  message_format is type-specific.  For example, Event Logs don't have cs-method, so I assume it'll fail.  (note that the EV section of this is awaiting testing, but should work - the IIS section already works)

As a side note - these dump into OMD/Check_MK.  I've configured passive services to match the Nagios service fields.

93-outputs_graphite
# Output metrics to graphite
if [SourceName] == "IIS" {
        graphite {
                host => "graphite.domain.com"
                metrics => [
                        "IIS.cs-bytes", "%{cs-bytes}",
                        "IIS.cs-method", "%{cs-method}",
                        "IIS.sc-bytes", "%{sc-bytes}",
                        "IIS.sc-status", "%{sc-status}",
                        "IIS.sc-substatus", "%{sc-substatus}",
                        "IIS.time-taken", "%{time-taken}"
                ]
        }
}
Exporting metric data from Logstash to Graphite.  I don't know if there is any coherence to this (I suspect there isn't), but it was testing, and proved it works.


99-outputs_end
} # End output section
No surprises here.  End!

Conclusion...

There you have it: How we use Logstash in a Windows environment.  Hope this helps someone get started!






Comments

  1. As a quick update to this, one of the Logstash-users contributors pointed out some improvements we could make: https://groups.google.com/forum/#!topic/logstash-users/EAb2jQqexr4

    "The reason you use tag_on_failure is due to the fact you call grok sometimes when you know it won't match.

    Take for example your nagios tagging, You call one grok followed by another inside the "if [SourceName] == "BizTalk Server" {" only expecting one to match. To avoid the need to have it try a grok when its going to fail you can extract the decider ahead of time. For example if you have extracted the SeverityValue already then you would just do "if [SeverityValue] == '[4]'" before the first grok. Of course as the grok is not really extracting anything but only tagging the event you could do away with the grok completely and just use mutate add_tag.

    Similarly in your iss tagging if you extra sc_status as an int, you can do (if [sc_status] >= 200 and [sc_status] < 299) mutate, it as an int will make it easier to use in elasticsearch for metrics or ranges too.

    If you end up doing high volume log parsing avoiding regex's that are going to fail can help with performance."

    ReplyDelete

Post a Comment

Popular posts from this blog

DFSR - eventid 4312 - replication just won't work

Fixing duplicate SPNs (service principal name)

Logstash to Nagios - alerting based on Windows Event ID