Julia arbeitet nicht zum Laden von großen Daten in Docker Container

Question

Ich habe ein Julia-Programm, das 8 GB Daten lädt. Es funktioniert gut in meiner lokalen Maschine.

Aber wenn ich es in Docker Container versuche es nicht Daten laden und Bus Fehler geben. Es funktioniert gut mit kleinen Daten wie 20 MB in Docker Container.

Dockerfile

FROM ubuntu:16.04 

WORKDIR /julia 

RUN apt-get -y update 

RUN apt-get -y install unzip 

RUN apt-get -y install cmake 

RUN apt-get -y install clang 

RUN apt-get -y install wget 

RUN cd /tmp/ 

RUN wget "https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz" 

RUN tar -xzvf julia-0.5.0-linux-x86_64.tar.gz 

RUN mv julia-3c9d75391c/ ~/julia 

ENV PATH="/root/julia/bin:${PATH}" 

RUN julia --eval 'Pkg.add("JSON")' 

RUN julia --eval 'Pkg.add("HttpServer")' 

RUN julia --eval 'Pkg.add("URIParser")' 

RUN julia --eval 'Pkg.clone("https://github.com/deep-compute/AdaGram.jl.git")' 

RUN julia --eval 'Pkg.build("AdaGram")' 

CMD ["julia", "/tmp/adagram_server.jl", "80", "/julia/full.embed"] 

Docker-compose.yml

version: "3.1" 
services: 
    julia: 
     image: ramidavalapati/julia:v-1 
     volumes: 
      - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed 
     ports: 
      - 8080:80 
     command: ["sleep", "1h"] 

Weiter Ich tue

sudo docker-compose up 
sudo docker exec -it $(sudo docker-compose ps -q julia) bash 
root@3748d5958f77:/julia# julia 
julia> using AdaGram 
julia> AdaGram.load_model("/julia/full.embed") 

Er ror

signal (7): Bus error 
while loading no file, in expression starting on line 0 
macro expansion at ./cartesian.jl:62 [inlined] 
macro expansion at ./multidimensional.jl:429 [inlined] 
_unsafe_batchsetindex! at ./multidimensional.jl:421 
_setindex! at ./multidimensional.jl:370 [inlined] 
setindex! at ./abstractarray.jl:832 [inlined] 
#9 at /root/.julia/v0.5/AdaGram/src/AdaGram.jl:64 
#600 at ./multi.jl:1030 
run_work_thunk at ./multi.jl:1001 
run_work_thunk at ./multi.jl:1010 [inlined] 
#597 at ./event.jl:68 
unknown function (ip: 0x7fe1822db16f) 
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:189 [inlined] 
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1942 
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1392 [inlined] 
start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:253 
unknown function (ip: 0xffffffffffffffff) 
Allocations: 9661042 (Pool: 9659980; Big: 1062); GC: 16 
Bus error (core dumped) 

Docker Version

Client: 
Version:  17.09.0-ce 
API version: 1.32 
Go version: go1.8.3 
Git commit: afdb6d4 
Built:  Tue Sep 26 22:42:18 2017 
OS/Arch:  linux/amd64 

Server: 
Version:  17.09.0-ce 
API version: 1.32 (minimum version 1.12) 
Go version: go1.8.3 
Git commit: afdb6d4 
Built:  Tue Sep 26 22:40:56 2017 
OS/Arch:  linux/amd64 
Experimental: false 

Docker info

Containers: 24 
Running: 0 
Paused: 0 
Stopped: 24 
Images: 24 
Server Version: 17.09.0-ce 
Storage Driver: overlay2 
Backing Filesystem: extfs 
Supports d_type: true 
Native Overlay Diff: true 
Logging Driver: json-file 
Cgroup Driver: cgroupfs 
Plugins: 
Volume: local 
Network: bridge host macvlan null overlay 
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog 
Swarm: active 
NodeID: jlkmigmtyjhz6yvi3zuvkobu7 
Is Manager: true 
ClusterID: rqt03ulgvvnym235m1qm8vd17 
Managers: 4 
Nodes: 15 
Orchestration: 
    Task History Retention Limit: 5 
Raft: 
    Snapshot Interval: 10000 
    Number of Old Snapshots to Retain: 0 
    Heartbeat Tick: 1 
    Election Tick: 3 
Dispatcher: 
    Heartbeat Period: 5 seconds 
CA Configuration: 
    Expiry Duration: 3 months 
    Force Rotate: 0 
Autolock Managers: false 
Root Rotation In Progress: false 
Node Address: X.X.X.X 
    Manager Addresses: 
    X.X.X.X:2377 
    X.X.X.X:2377 
    X.X.X.X:2377 
    X.X.X.X:2377 
Runtimes: runc 
Default Runtime: runc 
Init Binary: docker-init 
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0 
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64 
init version: 949e6fa 
Security Options: 
    apparmor 
    seccomp 
    Profile: default 
Kernel Version: 4.10.0-35-generic 
Operating System: Ubuntu 16.04.3 LTS 
OSType: linux 
Architecture: x86_64 
CPUs: 12 
Total Memory: 251.8GiB 
Name: ram 
ID: 3OGG:275C:Q3IW:O4HX:DPLP:DPI3:5TIT:AG5J:EDMK:7NK3:L4UZ:BTQH 
Docker Root Dir: /var/lib/docker 
Debug Mode (client): false 
Debug Mode (server): false 
Username: ramidavalapati 
Registry: https://index.docker.io/v1/ 
Experimental: false 
Insecure Registries: 
    127.0.0.0/8 
Live Restore Enabled: false 

WARNING: No swap limit support 

Vielen Dank im Voraus für jede Art von Hilfe

link to 40MBfull.embed file

Answer 1

Das Problem ist, es gibt keinen ausreichenden gemeinsamen Speicher im Docker-Container (Standard ist 64 MB).

Das Problem wurde gelöst, indem die Option --shm-size beim Ausführen von Docker-Image angegeben wurde.

Docker-compose Datei

version: "3.1" 
services: 
    julia: 
     image: ramidavalapati/julia:v-1 
     shm_size: 1g 
     volumes: 
      - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed 
     ports: 
      - 8080:80 
     command: ["sleep", "1h"] 

Wenn wir in swarm mode arbeiten wollen, müssen wir gemeinsam genutzten Speicher in Volumenabschnitt beziehen.

version: "3.3" 
services: 
    julia: 
     image: ramidavalapati/julia:v-1 
     volumes: 
      - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed 
      - /dev/shm:/dev/shm 
     ports: 
      - 8080:80 
     command: ["sleep", "1h"] 

Hier Container verwendet Speicher des Host-geteilt, in dem er ausgeführt wird.