Fluentd nicht Protokolle in Amazon s3 schreiben

Ich versuche, das fluentd-s3-Plugin noch im Moment zu testen es nicht meine Protokolle in s3 Eimer posten. Ich laufe alles auf ubuntu xenial, nachdem ich flüssig mit td-Agent installiert habe. Das Folgende ist die /var/log/td-agent/td-agent.log bei befindet Protokolldatei td-agentFluentd nicht Protokolle in Amazon s3 schreiben

2016-09-23 09:16:18 -0300 [info]: reading config file path="/etc/td-agent/td-agent.conf" 
2016-09-23 09:16:18 -0300 [info]: starting fluentd-0.12.26 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-kinesis' version '1.1.1' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-mongo' version '0.7.13' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-s3' version '0.6.8' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-scribe' version '0.10.14' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td' version '0.10.28' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2' 
2016-09-23 09:16:18 -0300 [info]: gem 'fluentd' version '0.12.26' 
2016-09-23 09:16:18 -0300 [info]: adding match pattern="nginx.log" type="s3" 
2016-09-23 09:16:18 -0300 [info]: adding source type="tail" 
2016-09-23 09:16:18 -0300 [info]: using configuration file: <ROOT> 
    <source> 
    type tail 
    format nginx 
    path /var/log/nginx/access.log 
    pos_file /var/log/td-agent/nginx-access.pos 
    tag nginx.log 
    </source> 
    <match nginx.log> 
    @type s3 
    s3_bucket kd.creatives 
    aws_key_id xxxxxx 
    aws_sec_key xxxxxx 
    s3_region us-west-2 
    path logs/ 
    buffer_path /var/log/td-agent/s3 
    time_slice_format %Y%m%d%H%M 
    utc 
    format_json true 
    include_time_key true 
    buffer_chunk_limit 256m 
    </match> 
</ROOT> 
2016-09-23 09:16:18 -0300 [warn]: parameter 'format_json' in <match nginx.log> 
    @type s3 
    s3_bucket kd.creatives 
    aws_key_id xxxxxx 
    aws_sec_key xxxxxx 
    s3_region us-west-2 
    path logs/ 
    buffer_path /var/log/td-agent/s3 
    time_slice_format %Y%m%d%H%M 
    utc 
    format_json true 
    include_time_key true 
    buffer_chunk_limit 256m 
</match> is not used. 
2016-09-23 09:16:20 -0300 [info]: following tail of /var/log/nginx/access.log 
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\"" 
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""

Auch hier ist mein td-agent.conf Datei

# Centralized nginx logs into Amazon s3 
<source> 
    type tail 
    format nginx 
    path /var/log/nginx/access.log 
    pos_file /var/log/td-agent/nginx-access.pos 
    tag nginx.log 
</source> 

<match nginx.log> 
    @type s3 
    s3_bucket xxxxxxx 
    aws_key_id xxxxxxxxxxxxxxxx 
    aws_sec_key xxxxxxxxxxxxxxxxxxxxxxxxxx 
    s3_region us-west-2 

    path logs/ 
    buffer_path /var/log/td-agent/s3 
    time_slice_format %Y%m%d%H%M 

    utc 

    format_json true 
    include_time_key true 
    buffer_chunk_limit 256m 
</match>

Ich würde wirklich jede Antwort schätzen, warum fluentd nicht die Protokolle in s3 Eimer posten wird.

Quelle

2016-09-23 francotestori

Es sieht in_tail fehlgeschlagen, Ihr Nginx-Protokoll zu analysieren. Also, fluentd hat nicht alle Protokolle an s3 gesendet. format nginx behandelt das kombinierte Protokollformat als Standard.

Ich denke, dass Sie benutzerdefinierte Format verwenden müssen, um Ihr Nginx-Protokoll zu analysieren.

<source> 
    type tail 
    format /^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" "(?<forwarder>[^\"]*)")?/ 
    time_format %d/%b/%Y:%H:%M:%S %z 
    path /var/log/nginx/access.log 
    pos_file /var/log/td-agent/nginx-access.pos 
    tag nginx.log 
</source>

Ref.-Nr. http://docs.fluentd.org/articles/in_tail

Sie können Ihre eigene Regex von fluentd-ui testen. http://docs.fluentd.org/articles/fluentd-ui

Quelle

2016-09-24 11:34:29 torut

Danke für Ihre Antwort. Ich habe gerade mit fluentd angefangen und jetzt habe ich mich an das Log-Regex-Format gewöhnt (wie Ruby). – francotestori

Fluentd nicht Protokolle in Amazon s3 schreiben

Antwort

Verwandte Themen