1
Ich möchte CSV in ElasticSearch laden. Ich habe die Schritte wie in diesem Blog beschrieben verfolgt: http://blog.webkid.io/visualize-datasets-with-elk/Logstash erstellt keinen Index für den CSV-Dateityp
Ich verwende elasticsearch 2.3.1, kibana-4.5.0-linux-x64, logstash-2.3.1. Ich habe auch die sincedb-Datei gelöscht, die sich in meinem Verzeichnis/home/ec2-user/befand. Unten finden Sie die logstash-csv.conf Datei:
input {
file {
path => "/etc/logstash/data.csv"
type => "core2"
start_position => "beginning"
}
}
filter {
csv {
separator => ","
columns => ["Date","Open","High","Low","Close","Volume","Adj Close"]
}
mutate {convert => ["High", "float"]}
mutate {convert => ["Open", "float"]}
mutate {convert => ["Low", "float"]}
mutate {convert => ["Close", "float"]}
mutate {convert => ["Volume", "float"]}
}
output {
elasticsearch {
action => "index"
hosts => "localhost:9200"
index => "stock"
workers => 1
}
stdout {}
}
Unten ist das, was ich bekomme, wenn ich curl 'localhost:9200/_cat/indices?v' auf der Kommandozeile geben:
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibi 1 1 4 0 20.1kb 20.1kb
yellow open .kibana 1 1 2 0 8.3kb 8.3kb
Warum wird der Index nicht erstellt zu werden? Bereitstellung auch mit --verbose Option unterhalb der Ausgabe:
starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/etc/logstash/data.csv"], :level=>:info}
No sincedb_path set, generating one based on the file path {:sincedb_path=>"/home/ec2-user/.sincedb_07be1b1b016d7137746afa513c111dad", :path=>["/etc/logstash/data.csv"], :level=>:info}
Using mapping template from {:path=>nil, :level=>:info}
Attempting to install template {:manage_template=>{"template"=>"logstash-*", "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "omit_norms"=>true}, "dynamic_templates"=>[{"message_field"=>{"match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"string", "index"=>"analyzed", "omit_norms"=>true, "fielddata"=>{"format"=>"disabled"}, "fields"=>{"raw"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true, "ignore_above"=>256}}}}}, {"float_fields"=>{"match"=>"*", "match_mapping_type"=>"float", "mapping"=>{"type"=>"float", "doc_values"=>true}}}, {"double_fields"=>{"match"=>"*", "match_mapping_type"=>"double", "mapping"=>{"type"=>"double", "doc_values"=>true}}}, {"byte_fields"=>{"match"=>"*", "match_mapping_type"=>"byte", "mapping"=>{"type"=>"byte", "doc_values"=>true}}}, {"short_fields"=>{"match"=>"*", "match_mapping_type"=>"short", "mapping"=>{"type"=>"short", "doc_values"=>true}}}, {"integer_fields"=>{"match"=>"*", "match_mapping_type"=>"integer", "mapping"=>{"type"=>"integer", "doc_values"=>true}}}, {"long_fields"=>{"match"=>"*", "match_mapping_type"=>"long", "mapping"=>{"type"=>"long", "doc_values"=>true}}}, {"date_fields"=>{"match"=>"*", "match_mapping_type"=>"date", "mapping"=>{"type"=>"date", "doc_values"=>true}}}, {"geo_point_fields"=>{"match"=>"*", "match_mapping_type"=>"geo_point", "mapping"=>{"type"=>"geo_point", "doc_values"=>true}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "doc_values"=>true}, "@version"=>{"type"=>"string", "index"=>"not_analyzed", "doc_values"=>true}, "geoip"=>{"type"=>"object", "dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip", "doc_values"=>true}, "location"=>{"type"=>"geo_point", "doc_values"=>true}, "latitude"=>{"type"=>"float", "doc_values"=>true}, "longitude"=>{"type"=>"float", "doc_values"=>true}}}}}}}, :level=>:info}
New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["localhost:9200"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started
Im Folgenden sind die Nachrichten, die ich bekomme, wenn Elasticsearch gestartet wird:
[2016-04-20 09:22:02,734][INFO ][node ] [Centurion] version[2.3.1], pid[7301], build[bd98092/2016-04-04T12:25:05Z]
[2016-04-20 09:22:02,735][INFO ][node ] [Centurion] initializing ...
[2016-04-20 09:22:03,394][INFO ][plugins ] [Centurion] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-04-20 09:22:03,416][INFO ][env ] [Centurion] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [31.2gb], net total_space [49gb], spins? [no], types [ext4]
[2016-04-20 09:22:03,416][INFO ][env ] [Centurion] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-04-20 09:22:03,416][WARN ][env ] [Centurion] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-04-20 09:22:05,313][INFO ][node ] [Centurion] initialized
[2016-04-20 09:22:05,313][INFO ][node ] [Centurion] starting ...
[2016-04-20 09:22:05,458][INFO ][transport ] [Centurion] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2016-04-20 09:22:05,464][INFO ][discovery ] [Centurion] elasticsearch/fSRusanNRgeBN9D38nXVFg
[2016-04-20 09:22:08,511][INFO ][cluster.service ] [Centurion] new_master {Centurion}{fSRusanNRgeBN9D38nXVFg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-04-20 09:22:08,526][INFO ][http ] [Centurion] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2016-04-20 09:22:08,526][INFO ][node ] [Centurion] started
[2016-04-20 09:22:08,588][INFO ][gateway ] [Centurion] recovered [2] indices into cluster_state
[2016-04-20 09:22:08,981][INFO ][cluster.routing.allocation] [Centurion] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibi][0]] ...]).