logstash generic output filename

By : Philipp
Source: Stackoverflow.com
Question!

I'd like to collect the logs of multiple servers on one logstash node. As output I'd like to store one file per server. In the logs, I got a "source_host" field that indicates which server has produced the log.

As output, I'd like to get a bunch of files named by "source_host". The source hosts change frequently, so I'd need a generic configuration

e.g. logs originating from server "foo" should be saved in /logs/foo and logs from server "bar" in /logs/bar

I tried the config like this, but the file gets named "%{source_host}". When using %{host}, the file gets the hostname of the collecting server.

output{
  file {
    path => "/tmp/%{source_host}"
  }
}
By : Philipp


Answers

My configuration:

input {
  tcp {
    port => 5544
    codec => json_lines
  }
}

output{
  file {
    path => "/tmp/%{source_host}"
  }
}

outputs to a file /tmp/foo with your sample log.

echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"foo","message":"testmsg"}' | nc localhost 5544

EDIT: Here are my test results:

pancake$ echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"foo","message":"testmsg"}' | nc localhost 5544
pancake$ cat /tmp/foo
{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z","source_host":"foo","message":"testmsg","port":56716}
pancake$ echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"bar","message":"testmsg"}' | nc localhost 5544
pancake$ echo '{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z", "source_host":"bar","message":"one more message!"}' | nc localhost 5544
pancake$ cat /tmp/bar
{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z","source_host":"bar","message":"testmsg","port":56717}
{"version":"debug","host":"devel","level":5,"@version":"1","@timestamp":"2016-09-15T10:41:00.549Z","source_host":"bar","message":"one more message!","port":56718}

EDIT 2: Ooh, I just thought of something. You said earlier that you aren't using any filters, right? You need to use a filter of some kind or another, otherwise the field source_host won't exist. If you have codec => json_lines (because your logs are JSON) in your input block, as I do in my example, it will parse your JSON into key value pairs. If you have no filter or codec, the entire body of the log will be stored in the message field, unmodified. Try adding an input codec and see if that helps.

By : pancake


You can debug your mongo queries in multiple ways.

Start with your index usage using below command :
db.images.aggregate( [ { $indexStats: { } } ] )

If this don't give you any useful information, then check the execution plan of the slow queries using : db.setProfilingLevel(2)
db.system.profile.find().pretty()

db.system.profile will give you complete profile of your queries.



C++11 solution:

#include <type_traits>

template<typename From, typename To>
To map(From e) {
    return static_cast<To>(
        static_cast<typename std::underlying_type<To>::type>(
        static_cast<typename std::underlying_type<From>::type>(e)));
}

This casting cascade is very explicit and supports enum classes.

For older C++ versions and for enums without class, static_cast<Enum2>(e) would suffice.

Edit:

With template specialization, you can use map without specifying any types explicitly:

enum class Enum1: int {A, B, C, D};
enum class Enum2: char {A1, B1, C1, D1};

template<typename T>
struct target_enum {
};

template<>
struct target_enum<Enum1> {
    typedef Enum2 type;
};

template<typename From>
typename target_enum<From>::type map(From e) {
    typedef typename target_enum<From>::type To;
    return static_cast<To>(
        static_cast<typename std::underlying_type<To>::type>(
        static_cast<typename std::underlying_type<From>::type>(e)));
}

You can then call map(Enum1::A). Also works with simple, non-class enums.

By : flyx


This video can help you solving your question :)
By: admin