Logstash

Contents

  1. Logstash
    1. what is logstash
    2. how to install
      1. install requirement
      2. start it
      3. Install plugins
      4. testing plugins
      5. Removing plugin
    3. TEST
      1. test elasticsearch
      2. test logstash
      3. syslog test
      4. test stdin&stdout
      5. Use Logstash Config File
      6. server
      7. Starting server
    4. Troubleshooting

what is logstash

Logstash is an open source tool for collecting and managing log files. It鈥檚 part of an open-source stack which includes ElasticSearch for indexing and searching through data and Kibana for charting and visualizing data. Together they form a powerful Log management solution.

how to install

install requirement

What is elasticsearch What is curator

  1. Like a museum curator manages the exhibits and collections on display, Elasticsearch Curator helps you curate, or manage your Elasticsearch indices.
  2. Curator performs many operations on your Elasticsearch indices, from delete to snapshot to shard allocation routing.

What is kibana 

emerge -av app-misc/elasticsearch www-apps/kibana-bin \
app-admin/logstash-forwarder app-admin/logstash-bin \
dev-python/click dev-python/elasticsearch-py elasticsearch-curator

brightmoon ~ # /etc/init.d/elasticsearch start
elasticsearch          | * /etc/elasticsearch/elasticsearch.in.sh must be
copied into place
elasticsearch          | * ERROR: elasticsearch failed to start

bzip2 -dv /usr/share/doc/elasticsearch-1.5.0/examples/elasticsearch.in.sh.bz2
bzip2 -dv /usr/share/doc/elasticsearch-1.5.0/examples/logging.yml.bz2 
bzip2 -dv /usr/share/doc/elasticsearch-1.5.0/examples/elasticsearch.yml.bz2

cp -vi /usr/share/doc/elasticsearch-1.5.0/examples/elasticsearch.in.sh /etc/elasticsearch/elasticsearch.in.sh
cp -vi /usr/share/doc/elasticsearch-1.5.0/examples/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
cp -vi /usr/share/doc/elasticsearch-1.5.0/examples/logging.yml /etc/elasticsearch/

start it

brightmoon ~ # /etc/init.d/elasticsearch start 

elasticsearch          | * Starting elasticsearch ...
elasticsearch          | * /var/lib/elasticsearch: correcting mode
elasticsearch          | * /var/lib/elasticsearch: correcting owner
elasticsearch          | * /var/log/elasticsearch: correcting mode
elasticsearch          | * /var/log/elasticsearch: correcting owner
elasticsearch          | * /run/elasticsearch: creating directory
elasticsearch          | * /run/elasticsearch: correcting owner
elasticsearch          | * /var/lib/elasticsearch/_default: creating directory
elasticsearch          | * /var/lib/elasticsearch/_default: correcting owner
elasticsearch          | * /var/log/elasticsearch/_default: creating directory
elasticsearch          | * /var/log/elasticsearch/_default: correcting
owner [ ok ]csearch          |

Install plugins

http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-plugins.html#analysis-plugins 

cd /usr/share/elasticsearch
brightmoon elasticsearch # bin/plugin --install mobz/elasticsearch-head
-> Installing mobz/elasticsearch-head...
Trying https://github.com/mobz/elasticsearch-head/archive/master.zip...
Downloading
..................................................................................................................................................................................................................................................................................................................................................................................DONE
Installed mobz/elasticsearch-head into /usr/share/elasticsearch/plugins/head
Identified as a _site plugin, moving to _site structure ...

bin/plugin --install lukas-vlcek/bigdesk --verbose
bin/plugin --install elasticsearch/elasticsearch-analysis-smartcn/2.5.0 --verbose
bin/plugin --install elasticsearch/elasticsearch-analysis-icu/2.5.0
bin/plugin --install karmi/elasticsearch-paramedic
bin/plugin install elasticsearch/elasticsearch-lang-javascript/2.5.0
bin/plugin install elasticsearch/elasticsearch-lang-python/2.5.0
bin/plugin -u https://github.com/NLPchina/elasticsearch-sql/releases/download/1.3.2/elasticsearch-sql-1.3.2.zip --install sql
bin/plugin --install royrusso/elasticsearch-HQ
bin/plugin --install andrewvc/elastic-hammer
bin/plugin --install polyfractal/elasticsearch-inquisitor
bin/plugin --install xyu/elasticsearch-whatson/0.1.3
bin/plugin --install polyfractal/elasticsearch-segmentspy
bin/plugin --install info.johtani/elasticsearch-extended-analyze/1.5.0  #will be removed
bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/1.6.0
bin/plugin -url https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip -install elasticsearch-jetty-1.2.1

testing plugins

http://localhost:9200/_plugin/sql/

http://localhost:9200/_plugin/HQ/

http://localhost:9200/_plugin/elastic-hammer/

http://localhost:9200/_plugin/paramedic/index.html

http://localhost:9200/_plugin/inquisitor

http://localhost:9200/_plugin/whatson/

http://localhost:9200/_plugin/segmentspy/

Removing plugin

bin/plugin --remove <pluginname>

TEST

test elasticsearch

# curl http://localhost:9200/_cat/plugins?v
 name            component version type url               
 Captain Germany head      NA      s    /_plugin/head/    
 Captain Germany bigdesk   NA      s    /_plugin/bigdesk/
[[http://localhost:9200/_plugin/head/]]
[[http://localhost:9200/_plugin/bigdes/]]

test logstash

logstash -e 'input { stdin { } } output { elasticsearch { host => localhost }}' http://localhost:9200/_search?pretty 

# curl 'http://localhost:9200/_search?pretty'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 11,
    "successful" : 11,
    "failed" : 0
  },
  "hits" : {
    "total" : 9,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : ".kibana",
      "_type" : "config",
      "_id" : "4.0.2",
      "_score" : 1.0,
      "_source":{"buildNum":6004,"defaultIndex":"logstash-*"}
    }, {
      "_index" : ".kibana",
      "_type" : "index-pattern",
      "_id" : "logstash-*",
      "_score" : 1.0,
      "_source":{"title":"logstash-*","timeFieldName":"@timestamp","customFormats":"{}","fields":"[{\"type\":\"string\",\"indexed\":false,\"analyzed\":false,\"name\":\"_index\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":true,\"analyzed\":false,\"name\":\"_type\",\"count\":0,\"scripted\":false},{\"type\":\"geo_point\",\"indexed\":true,\"analyzed\":false,\"doc_values\":false,\"name\":\"geoip.location\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":true,\"analyzed\":false,\"doc_values\":false,\"name\":\"@version\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":false,\"analyzed\":false,\"name\":\"_source\",\"count\":2,\"scripted\":false},{\"type\":\"string\",\"indexed\":false,\"analyzed\":false,\"name\":\"_id\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":true,\"analyzed\":false,\"doc_values\":false,\"name\":\"message.raw\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":true,\"analyzed\":false,\"doc_values\":false,\"name\":\"host.raw\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"name\":\"message\",\"count\":0,\"scripted\":false},{\"type\":\"date\",\"indexed\":true,\"analyzed\":false,\"doc_values\":false,\"name\":\"@timestamp\",\"count\":0,\"scripted\":false},{\"type\":\"string\",\"indexed\":true,\"analyzed\":true,\"doc_values\":false,\"name\":\"host\",\"count\":0,\"scripted\":false}]"}
    }, {
      "_index" : "logstash-2015.04.23",
      "_type" : "logs",
      "_id" : "3nfCwlzmRXK79m5qpWVxNw",
      "_score" : 1.0,
      "_source":{"message":"help","@version":"1","@timestamp":"2015-04-23T08:42:34.052Z","host":"brightmoon"}
    }, {
      "_index" : "logstash-2015.04.23",
      "_type" : "logs",
      "_id" : "gJtDgbKZQgOunIr5vWe2lQ",
      "_score" : 1.0,
      "_source":{"message":"lkjfasd","@version":"1","@timestamp":"2015-04-23T08:42:31.515Z","host":"brightmoon"}
    }, {
      "_index" : "logstash-2015.04.23",
      "_type" : "logs",
      "_id" : "dEwfXHmNTpaiXd_QRcA7Ag",
      "_score" : 1.0,
      "_source":{"message":"z","@version":"1","@timestamp":"2015-04-23T08:42:35.496Z","host":"brightmoon"}
    }, {
      "_index" : "logstash-2015.04.23",
      "_type" : "logs",
      "_id" : "Yu59_SXfRrG-FvPeVhwUNA",
      "_score" : 1.0,
      "_source":{"message":"s","@version":"1","@timestamp":"2015-04-23T08:42:36.388Z","host":"brightmoon"}
    }, {
      "_index" : "logstash-2015.04.24",
      "_type" : "logs",
      "_id" : "nw1boXLoQvi0chZb93LfYQ",
      "_score" : 1.0,
      "_source":{"message":"help","@version":"1","@timestamp":"2015-04-24T01:03:04.278Z","host":"brightmoon"}
    }, {
      "_index" : "logstash-2015.04.24",
      "_type" : "logs",
      "_id" : "drmhv6j9SeWEOiVuDzmzqA",
      "_score" : 1.0,
      "_source":{"message":"help","@version":"1","@timestamp":"2015-04-24T01:03:02.729Z","host":"brightmoon"}
    }, {
      "_index" : "logstash-2015.04.24",
      "_type" : "logs",
      "_id" : "91YTHiYfSSqEWvh45690LQ",
      "_score" : 1.0,
      "_source":{"message":"Good morning","@version":"1","@timestamp":"2015-04-24T01:03:20.239Z","host":"brightmoon"}
    } ]
  }
}

{
  "took" : 35,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 4,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

syslog test

cat > /etc/logstash/conf.d/logstash-syslog.conf << "EOF"
input {
  tcp {
    port => 5000
    type => syslog
  }
  udp {
    port => 5000
    type => syslog
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}
EOF

logstash -f /etc/logstash/conf.d/logstash-syslog.conf

http://localhost:5000

test stdin&stdout

logstash --verbose -e 'input { stdin { } } output { stdout { codec => rubydebug } }' 

hello
Pipeline started {:level=>:info}
{
       "message" => "hello",
      "@version" => "1",
    "@timestamp" => "2015-04-23T08:06:19.270Z",
          "host" => "brightmoon"
}

Use Logstash Config File

vi logstash-simple.conf 

input { stdin { } } 
output { 
elasticsearch { host => localhost } 
stdout { codec => rubydebug } 
}

logstash -f logstash-simple.conf

server

cat server.conf 

input {
  redis {
    host => "127.0.0.1"
    type => "redis"
    data_type => "list"
    key => "logstash"
  }
}
output {
  stdout { }
  elasticsearch {
    cluster => "localhost"
  }
}

java -jar /opt/logstash/ agent -v -f /etc/logstash/conf.d/server.conf --log /var/log/logstash/server.log

Starting server

/etc/init.d/logstash start 

logstash               | * Checking your configuration ...
logstash               |Sending logstash logs to
/var/log/logstash/logstash.log.
logstash               |Using milestone 2 input plugin 'redis'. This plugin
should be stable, but if you see strange behavior, please let us know! For more
information on plugin milestones, see
http://logstash.net/docs/1.4.2/plugin-milestones {:level=>:warn}
logstash               |Configuration OK
logstash               |Configuration
OK
[ ok ]sh               |
logstash               | * /run/logstash: creating directory
logstash               | * /var/log/logstash: correcting mode
logstash               | * /var/log/logstash/logstash.log: correcting mode
logstash               | * Starting
logstash ...
[ ok ]sh               |

/etc/init.d/kibana start

http://127.0.0.1:5601

Troubleshooting

elasticsearch-kopf is a plugin for elasticsearch, not logstash. You'll need to download elasticsearch separately, depending on your version of logstash and then run ./bin/plugin -install lmenezes/elasticsearch-kopf there. Take a look here: http://www.elasticsearch.org/overview/elkdownloads/

References http://www.slashroot.in/logstash-tutorial-linux-central-logging-server logstash setup http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash https://groups.google.com/forum/#!topic/logstash-users/e3z8iD5PXnw https://github.com/medcl/elasticsearch-rtf

désert/Logstash (last edited 2015-04-24 07:08:52 by merlyn)