Ich versuche, eine Datenbank aus einer 4,5 GB mysqldump Datei in eine AWS Aurora DB-Instanz zu importieren. Es gibt ~ 80 Tabellen in meiner DB und die größte Tabelle hat ~ 13 Millionen Zeilen (der Rest ist viel kleiner). Meine Speicherabbilddatei hat mehrwertige Einfügungen, die mysqldump auf 64MB jeweils gekappt hat, gemäß dem Max_allowed_packet, das ich in my.cnf angegeben habe.AWS Aurora Speicherfehler mit großem Import
hatte ich verschiedene Probleme zunächst mit dem Import, aber ich war in der Lage diejenigen zu beheben, indem
max_allowed_packet = 1073741824 (1 GB)
wait_timeout = 10800 (3 Stunden)
dh Optionen Aurora Parametergruppe Einstellung net_read_timeout = 10800
net_write_timeout = 10800
interactive_timeout = 10800
Zuerst war meine Aurora-Instanz ein db.t2.small (2GB RAM), aber als ich den Import z. mysql -u ... mydb < dump.sql von einer EC2-Instanz (m3.medium, 4 GB RAM) ist der Prozess nach 1 Minute Laufzeit fehlgeschlagen. Das RDS-Protokoll sagt mir, dass ein Speicherfehler aufgetreten ist. Ich stieß die Aurora-Instanz bis zu einem db.t2.medium (4GB RAM), aber der Prozess schlug nach ungefähr ~ 20 Minuten mit dem gleichen Typ von Speichermangel fehl.
Ich möchte nicht zum nächsten Instanztyp (15GB RAM) springen, aber auf jeden Fall macht es keinen Sinn, dass ich müsste. Ich habe die gleiche mysqldump Datei regelmäßig in einen lokalen MySQL Server auf der M3.medium EC2 Instanz importiert, die ich benutze, und ich hatte nie irgendwelche Probleme. Es dauert ~ 40 Minuten, um zu importieren.
Hier ist das Aurora-Fehlerprotokoll des letzten Import Ich habe versucht:
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 101748 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: no num success: 0 system KB: 4050724 available KB: 111688 low-threshold KB: 202536 recovery time: 11 num declined query: 0 num killed query: 0 num killed connection: 0
OOM crash avoidance result: success: yes num success: 1 system KB: 4050724 available KB: 529464 low-threshold KB: 202536 recovery time: 24 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 200956 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 2 system KB: 4050724 available KB: 556020 low-threshold KB: 202536 recovery time: 5 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 170392 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 3 system KB: 4050724 available KB: 554108 low-threshold KB: 202536 recovery time: 7 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 194900 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 4 system KB: 4050724 available KB: 554340 low-threshold KB: 202536 recovery time: 8 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 198780 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: no num success: 4 system KB: 4050724 available KB: 133160 low-threshold KB: 202536 recovery time: 11 num declined query: 0 num killed query: 0 num killed connection: 0
OOM crash avoidance result: success: yes num success: 5 system KB: 4050724 available KB: 556540 low-threshold KB: 202536 recovery time: 25 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 170224 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
OOM crash avoidance result: success: yes num success: 6 system KB: 4050724 available KB: 579368 low-threshold KB: 202536 recovery time: 1 num declined query: 0 num killed query: 0 num killed connection: 0
Available memory is low. Trying to avoid OOM crash: system KB: 4050724 available KB: 175612 low-threshold KB: 202536 decline query: no tune caches: no kill query: no kill connection: no
<jemalloc>: Error in mmap(): err: 12, msg: Cannot allocate memory
<jemalloc>: Error in malloc(): out of memory
<jemalloc>: System-wide: MemTotal: 4050724kb, MemFree: 137440kb, Buffers: 20428kb, Cached: 62340kb, Active: 2188968kb, Dirty: 204kb, Inactive: 37228kb, Mapped: 41592kb
<jemalloc>: terminating process due to out of resources
10:58:03 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=6
max_threads=90
thread_count=5
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63814 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x2ab59d623000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 2ab524642c08 thread_stack 0x40000
/rdsdbbin/oscar/bin/mysqld(my_print_stacktrace+0x2c)[0x9897ec]
/rdsdbbin/oscar/bin/mysqld(handle_fatal_signal+0x491)[0x6f0651]
/lib64/libpthread.so.0(+0xf5b0)[0x2ab5165405b0]
/lib64/libc.so.6(gsignal+0x39)[0x2ab51925cbe9]
/lib64/libc.so.6(abort+0x148)[0x2ab51925dfe8]
/rdsdbbin/oscar/lib/libjemalloc.so(malloc+0x1226)[0x2ab5160f62e6]
/rdsdbbin/oscar/bin/mysqld(my_malloc+0x25)[0x986525]
/rdsdbbin/oscar/bin/mysqld(alloc_root+0x8f)[0x9824bf]
/rdsdbbin/oscar/bin/mysqld[0x5b57a5]
/rdsdbbin/oscar/bin/mysqld(_Z10MYSQLparsePv+0xc1ce)[0x826ade]
/rdsdbbin/oscar/bin/mysqld(_Z9parse_sqlP3THDP12Parser_stateP19Object_creation_ctx+0xb5)[0x772175]
/rdsdbbin/oscar/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0xf2)[0x772502]
/rdsdbbin/oscar/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xf43)[0x774003]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer7consumeEjj+0xd3)[0x803963]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer5startEv+0x98)[0x803a98]
/rdsdbbin/oscar/bin/mysqld(_ZN22OscarSchedulerConsumer11drain_queueEPv+0x6a)[0x803cda]
/lib64/libpthread.so.0(+0x7f18)[0x2ab516538f18]
/lib64/libc.so.6(clone+0x6d)[0x2ab51930bb2d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (2ab5ad400010): INSERT INTO `Documents` VALUES (478555572,150317,1321817,1,9,627609600,0,60,5471267,0,639014400,''),(478555571,150317,1321816,1,1,623980800,0,60,0,0,623980800,''),(478555575,150318,1321820,1,1,623980800,0,60,0,0,623980800,'')
Connection ID (thread ID): 6
Status: NOT_KILLED
Da meine EC2-Instanz mit 4 GB RAM den Import OK verarbeiten kann, sicher muss dies ein Konfigurationsproblem sein. Gibt es andere Parametergruppenoptionen, die ich ändern kann?
Ich habe auch versucht Binärlogging deaktivieren, indem Sie die binlog_format Parameter OFF in meiner DB-Cluster-Parametergruppe (gemäß den Anweisungen here) und dem Neustart der Instanz einstellen, aber wenn ich die Abfrage select @@binlog_format laufen bekomme ich das Ergebnis STATEMENT.