Elasticsearch im Docker Container - "konnte nicht in das Datenverzeichnis schreiben" nach einem einfachen Neustart

Question

vor einigen Wochen habe ich einen Elk Stack (elasticsearch, logstash, kibana) erstellt, um die Last von Logfiles besser zu bewältigen.

Es hat alles perfekt funktioniert. Heute habe ich einige neue Patterns in Logstash aufgerufen und aus irgendeinem Grund habe ich über docker-compose down && docker-compose up -d neu gestartet.

Jetzt startet elasticsearch nicht mehr.

root@xyz:/srv/elk# docker-compose logs elasticsearch 
Attaching to elk_elasticsearch_1 
elasticsearch_1 | [2017-07-01T07:34:36,859][INFO ][o.e.n.Node    ] [lw-e01] initializing ... 
elasticsearch_1 | [2017-07-01T07:34:36,999][INFO ][o.e.e.NodeEnvironment ] [lw-e01] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/HDD-ELK)]], net usable_space [19.1gb], net total_space [49gb], spins? [possibly], types [ext4] 
elasticsearch_1 | [2017-07-01T07:34:36,999][INFO ][o.e.e.NodeEnvironment ] [lw-e01] heap size [3.9gb], compressed ordinary object pointers [true] 
elasticsearch_1 | [2017-07-01T07:34:37,635][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [lw-e01] uncaught exception in thread [main] 
elasticsearch_1 | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to created node environment 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 | Caused by: java.lang.IllegalStateException: Failed to created node environment 
elasticsearch_1 |  at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  ... 6 more 
elasticsearch_1 | Caused by: java.io.IOException: failed to write in data directory [/usr/share/elasticsearch/data/nodes/0/indices/a94kXbSER2CE97qdPhgVLA/_state] write permission is required 
elasticsearch_1 |  at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1075) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1047) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:277) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.node.Node.<init>(Node.java:262) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  ... 6 more 
elasticsearch_1 | Caused by: java.nio.file.FileAlreadyExistsException: /usr/share/elasticsearch/data/nodes/0/indices/a94kXbSER2CE97qdPhgVLA/_state/.es_temp_file 
elasticsearch_1 |  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88) ~[?:?] 
elasticsearch_1 |  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?] 
elasticsearch_1 |  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?] 
elasticsearch_1 |  at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?] 
elasticsearch_1 |  at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_131] 
elasticsearch_1 |  at java.nio.file.Files.createFile(Files.java:632) ~[?:1.8.0_131] 
elasticsearch_1 |  at org.elasticsearch.env.NodeEnvironment.tryWriteTempFile(NodeEnvironment.java:1072) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.env.NodeEnvironment.assertCanWrite(NodeEnvironment.java:1047) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:277) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.node.Node.<init>(Node.java:262) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.node.Node.<init>(Node.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.4.0.jar:5.4.0] 
elasticsearch_1 |  ... 6 more 

OK es sieht aus wie eine einfache Berechtigungen, Problem, sondern auch nach einem chown -R 1000.1000 elasticsearch/ stürzt (AND gesetzt anderen Eigentümern).

Das Setup: Ich richte einen Server mit einem LVM für das Docker-Compose-Projekt ein. In der docker-compose.yml beschrieb ich die drei Dienste.

version: '3' 

services: 
    elasticsearch: 
    image: my/elasticsearch/image:5.4.0 
    volumes: 
     - ./elasticsearch/data:/usr/share/elasticsearch/data 
     - ./elasticsearch/config:/usr/share/elasticsearch/config 
     - /etc/localtime:/etc/localtime:ro 
    environment: 
     ES_JAVA_OPTS: "-Xmx4g -Xms1g" 
    ulimits: 
     memlock: 
     soft: -1 
     hard: -1 
    networks: 
     - nginx_net 

    logstash: 
    image: my/logstash/image:5.4.0 
    command: ["logstash", "-f", "/etc/logstash.conf"] 
    volumes: 
     - ./logstash.conf:/etc/logstash.conf:ro 
     - ./logstash.yml:/etc/logstash/logstash.yml:ro 
     - ./GeoDb/GeoLite2-City.mmdb:/GeoLite2-City.mmdb:ro 
     - ./patterns:/etc/logstash/patterns:ro 
     - /etc/localtime:/etc/localtime:ro 
    ports: 
     - "5044:5044" 
    environment: 
     LS_JAVA_OPTS: "-Xmx1g -Xms512m" 
    depends_on: 
     - elasticsearch 
    networks: 
     - nginx_net 

    kibana: 
    image: my/kibana/image:5.4.0 
    volumes: 
     - ./kibana/config/:/usr/share/kibana/config 
     - ./kibana/config/kibana.yml:/etc/kibana/kibana.yml 
     - /etc/localtime:/etc/localtime:ro 
    depends_on: 
     - elasticsearch 
    networks: 
     - nginx_net 

networks: 
    nginx_net: 
    external: true 

Wie Sie sehen können, habe ich die offiziellen Bilder nicht direkt verwenden, I (im Moment) XPack installieren auch alle drei suchen Bilder wie dieses

FROM elasticsearch:5.4.0 

RUN bin/elasticsearch-plugin install x-pack --batch 

The scond, was ich anders machen ist, dass ich keine benannten Volumes verwendet habe. Das liegt daran, dass ich gerne einen Ordner habe, der das gesamte Projekt enthält, auch besser für mein LVM-Management.

root@xyz:/srv/elk# ls -l 
insgesamt 43488 
-rw-r--r-- 1 root root  1514 Jul 1 09:34 docker-compose.yml 
drwxr-xr-x 4 1000 1000  4096 Mai 18 17:43 elasticsearch 
drwxr-xr-x 3 root root  4096 Mai 21 12:49 GeoDb 
-rw-r--r-- 1 root root 25398754 Mai 21 12:49 GeoLite2-City.tar.gz 
-rw-r--r-- 1 root root 19074950 Mai 21 12:03 GeoLiteCity.dat 
drwxr-xr-x 3 root root  4096 Mai 14 16:20 kibana 
-rw-r--r-- 1 root root  5523 Jul 1 09:02 logstash.conf 
-rw-r--r-- 1 root root  4708 Jun 3 11:25 logstash.yml 
drwx------ 2 root root 16384 Mai 17 23:40 lost+found 
drwxr-xr-x 2 root root  4096 Jun 7 22:08 patterns 
-rwxr-xr-x 1 root root  168 Mai 21 12:49 update-geoip.sh 
root@xyz:/srv/elk# du -hs elasticsearch/ 
28G  elasticsearch/ 

Ich las über Plugins wie local-persist namens Volumen zu verwenden, sondern auch den DIR angeben, die Dateien zu speichern. Aber ich habe auch gelesen, dass docker empfiehlt, keine Plugins in der Produktion zu verwenden.

Ich wäre ziemlich glücklich für irgendeine Idee/Verbindung.

Answer 1

OK einfach: run (in meinem Fall) rm elasticsearch/data/nodes/0/indices/a94kXbSER2CE97qdPhgVLA/_s‌​tate/.es_temp_file in Hauptordner von Docker-compose Projekt hat mir geholfen EL wieder von vorn anfangen ...

Um die genau Pfad Blick auf die wie oben java.nio.file.FileAlreadyExistsException

Answer 2

herauszufinden Antwort funktionierte nicht für mich. Es endete mit einem Speicherproblem mit Docker. Die Speicherzuweisung von Boosting Docker löste es.