0
Ich habe eine MySQL-Tabelle (rangeTest), die über 10 Millionen Zeilen von Daten hat, die ich ständig in einer Web-App abfragen muss und daher sehr schnell sein muss. Jede Zeile deckt einen Ganzzahlbereich ab, wie in den Spalten rangeStartInt und rangeEndInt beschrieben. Keiner der Integer-Bereiche überschneidet sich von Zeile zu Zeile und der gleichzeitige Durchlauf zwischen den Zeilen. So zum Beispiel würde eine Serie von 5 Zeilen wie folgt (sie sind die ersten beiden Spalten in den Daten unten):Abfrage einer partitionierten MySQL DB mit <= And > = In Where Klausel
3758080000,3758084095,1835841,1835841,,0,0,,37.5112,126.9741,200
3758084096,3758085119,1835841,1835841,,0,0,,37.5112,126.9741,200
3758085120,3758086143,1841811,1835841,,0,0,,35.1547,126.9156,1
3758086144,3758088191,1835841,1835841,,0,0,,37.5112,126.9741,200
3758088192,3758088447,8314434,1643084,,0,0,,1.2000,104.0000,100
Jede Abfrage gibt eine einzelne Zeile basierend auf einer ganze Zahl. Beispielsweise ein ganzzahliger Wert 3758085130 würde ergeben folgende Abfrage:
SELECT * FROM rangeTest WHERE rangeStartInt <= 3758085130 AND rangeEndInt >= 3758085130
Und diese zurückkehren würde die folgenden einreihige:
3758085120,3758086143,1841811,1835841,,0,0,,35.1547,126.9156,1
In un partitionierten Form jede Abfrage auf der rangeTest Tabelle dauert etwa 8 Sekunden . Ich habe daher RANGE Aufteilung wie folgt hinzugefügt:
CREATE TABLE `rangeTest` (
`rangeStartInt` bigint(11) NOT NULL,
`rangeEndInt` bigint(11) NOT NULL,
`a` bigint(11) NOT NULL,
`b` varchar(11) COLLATE utf8_unicode_ci DEFAULT NULL,
`c` varchar(11) COLLATE utf8_unicode_ci DEFAULT NULL,
`d` varchar(1) COLLATE utf8_unicode_ci DEFAULT NULL,
`e` varchar(1) COLLATE utf8_unicode_ci DEFAULT NULL,
`f` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`g` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`h` varchar(10) COLLATE utf8_unicode_ci DEFAULT NULL,
`i` varchar(5) COLLATE utf8_unicode_ci DEFAULT NULL,
PRIMARY KEY (`rangeStartInt`,`rangeEndInt`,`a`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
PARTITION BY RANGE COLUMNS(rangeStartInt,rangeEndInt)
(
PARTITION p0 VALUES LESS THAN (8000000,8000000) ENGINE = InnoDB,
PARTITION p1 VALUES LESS THAN (16000000,16000000) ENGINE = InnoDB,
PARTITION p2 VALUES LESS THAN (24000000,24000000) ENGINE = InnoDB,
PARTITION p3 VALUES LESS THAN (32000000,32000000) ENGINE = InnoDB,
PARTITION p4 VALUES LESS THAN (40000000,40000000) ENGINE = InnoDB,
PARTITION p5 VALUES LESS THAN (48000000,48000000) ENGINE = InnoDB,
PARTITION p6 VALUES LESS THAN (56000000,56000000) ENGINE = InnoDB,
PARTITION p7 VALUES LESS THAN (64000000,64000000) ENGINE = InnoDB,
PARTITION p8 VALUES LESS THAN (72000000,72000000) ENGINE = InnoDB,
PARTITION p9 VALUES LESS THAN (80000000,80000000) ENGINE = InnoDB,
PARTITION p10 VALUES LESS THAN (88000000,88000000) ENGINE = InnoDB,
PARTITION p11 VALUES LESS THAN (96000000,96000000) ENGINE = InnoDB,
PARTITION p12 VALUES LESS THAN (104000000,104000000) ENGINE = InnoDB,
PARTITION p13 VALUES LESS THAN (112000000,112000000) ENGINE = InnoDB,
PARTITION p14 VALUES LESS THAN (120000000,120000000) ENGINE = InnoDB,
PARTITION p15 VALUES LESS THAN (128000000,128000000) ENGINE = InnoDB,
PARTITION p16 VALUES LESS THAN (136000000,136000000) ENGINE = InnoDB,
PARTITION p17 VALUES LESS THAN (144000000,144000000) ENGINE = InnoDB,
PARTITION p18 VALUES LESS THAN (152000000,152000000) ENGINE = InnoDB,
PARTITION p19 VALUES LESS THAN (160000000,160000000) ENGINE = InnoDB,
PARTITION p20 VALUES LESS THAN (168000000,168000000) ENGINE = InnoDB,
PARTITION p21 VALUES LESS THAN (176000000,176000000) ENGINE = InnoDB,
PARTITION p22 VALUES LESS THAN (184000000,184000000) ENGINE = InnoDB,
PARTITION p23 VALUES LESS THAN (192000000,192000000) ENGINE = InnoDB,
PARTITION p24 VALUES LESS THAN (200000000,200000000) ENGINE = InnoDB,
PARTITION p25 VALUES LESS THAN (208000000,208000000) ENGINE = InnoDB,
PARTITION p26 VALUES LESS THAN (216000000,216000000) ENGINE = InnoDB,
PARTITION p27 VALUES LESS THAN (224000000,224000000) ENGINE = InnoDB,
PARTITION p28 VALUES LESS THAN (232000000,232000000) ENGINE = InnoDB,
PARTITION p29 VALUES LESS THAN (240000000,240000000) ENGINE = InnoDB,
PARTITION p30 VALUES LESS THAN (248000000,248000000) ENGINE = InnoDB,
PARTITION p31 VALUES LESS THAN (256000000,256000000) ENGINE = InnoDB,
PARTITION p32 VALUES LESS THAN (264000000,264000000) ENGINE = InnoDB,
PARTITION p33 VALUES LESS THAN (272000000,272000000) ENGINE = InnoDB,
PARTITION p34 VALUES LESS THAN (280000000,280000000) ENGINE = InnoDB,
PARTITION p35 VALUES LESS THAN (288000000,288000000) ENGINE = InnoDB,
PARTITION p36 VALUES LESS THAN (296000000,296000000) ENGINE = InnoDB,
PARTITION p37 VALUES LESS THAN (304000000,304000000) ENGINE = InnoDB,
PARTITION p38 VALUES LESS THAN (312000000,312000000) ENGINE = InnoDB,
PARTITION p39 VALUES LESS THAN (320000000,320000000) ENGINE = InnoDB,
PARTITION p40 VALUES LESS THAN (328000000,328000000) ENGINE = InnoDB,
PARTITION p41 VALUES LESS THAN (336000000,336000000) ENGINE = InnoDB,
PARTITION p42 VALUES LESS THAN (344000000,344000000) ENGINE = InnoDB,
PARTITION p43 VALUES LESS THAN (352000000,352000000) ENGINE = InnoDB,
PARTITION p44 VALUES LESS THAN (360000000,360000000) ENGINE = InnoDB,
PARTITION p45 VALUES LESS THAN (368000000,368000000) ENGINE = InnoDB,
PARTITION p46 VALUES LESS THAN (376000000,376000000) ENGINE = InnoDB,
PARTITION p47 VALUES LESS THAN (384000000,384000000) ENGINE = InnoDB,
PARTITION p48 VALUES LESS THAN (392000000,392000000) ENGINE = InnoDB,
PARTITION p49 VALUES LESS THAN (400000000,400000000) ENGINE = InnoDB,
PARTITION p50 VALUES LESS THAN (408000000,408000000) ENGINE = InnoDB,
PARTITION p51 VALUES LESS THAN (416000000,416000000) ENGINE = InnoDB,
PARTITION p52 VALUES LESS THAN (424000000,424000000) ENGINE = InnoDB,
PARTITION p53 VALUES LESS THAN (432000000,432000000) ENGINE = InnoDB,
PARTITION p54 VALUES LESS THAN (440000000,440000000) ENGINE = InnoDB,
PARTITION p55 VALUES LESS THAN (448000000,448000000) ENGINE = InnoDB,
PARTITION p56 VALUES LESS THAN (456000000,456000000) ENGINE = InnoDB,
PARTITION p57 VALUES LESS THAN (464000000,464000000) ENGINE = InnoDB,
PARTITION p58 VALUES LESS THAN (472000000,472000000) ENGINE = InnoDB,
PARTITION p59 VALUES LESS THAN (480000000,480000000) ENGINE = InnoDB,
PARTITION p60 VALUES LESS THAN (488000000,488000000) ENGINE = InnoDB,
PARTITION p61 VALUES LESS THAN (496000000,496000000) ENGINE = InnoDB,
PARTITION p62 VALUES LESS THAN (504000000,504000000) ENGINE = InnoDB,
PARTITION p63 VALUES LESS THAN (512000000,512000000) ENGINE = InnoDB,
PARTITION p64 VALUES LESS THAN (520000000,520000000) ENGINE = InnoDB,
PARTITION p65 VALUES LESS THAN (528000000,528000000) ENGINE = InnoDB,
PARTITION p66 VALUES LESS THAN (536000000,536000000) ENGINE = InnoDB,
PARTITION p67 VALUES LESS THAN (544000000,544000000) ENGINE = InnoDB,
PARTITION p68 VALUES LESS THAN (552000000,552000000) ENGINE = InnoDB,
PARTITION p69 VALUES LESS THAN (560000000,560000000) ENGINE = InnoDB,
PARTITION p70 VALUES LESS THAN (568000000,568000000) ENGINE = InnoDB,
PARTITION p71 VALUES LESS THAN (576000000,576000000) ENGINE = InnoDB,
PARTITION p72 VALUES LESS THAN (584000000,584000000) ENGINE = InnoDB,
PARTITION p73 VALUES LESS THAN (592000000,592000000) ENGINE = InnoDB,
PARTITION p74 VALUES LESS THAN (600000000,600000000) ENGINE = InnoDB,
PARTITION p75 VALUES LESS THAN (608000000,608000000) ENGINE = InnoDB,
PARTITION p76 VALUES LESS THAN (616000000,616000000) ENGINE = InnoDB,
PARTITION p77 VALUES LESS THAN (624000000,624000000) ENGINE = InnoDB,
PARTITION p78 VALUES LESS THAN (632000000,632000000) ENGINE = InnoDB,
PARTITION p79 VALUES LESS THAN (640000000,640000000) ENGINE = InnoDB,
PARTITION p80 VALUES LESS THAN (648000000,648000000) ENGINE = InnoDB,
PARTITION p81 VALUES LESS THAN (656000000,656000000) ENGINE = InnoDB,
PARTITION p82 VALUES LESS THAN (664000000,664000000) ENGINE = InnoDB,
PARTITION p83 VALUES LESS THAN (672000000,672000000) ENGINE = InnoDB,
PARTITION p84 VALUES LESS THAN (680000000,680000000) ENGINE = InnoDB,
PARTITION p85 VALUES LESS THAN (688000000,688000000) ENGINE = InnoDB,
PARTITION p86 VALUES LESS THAN (696000000,696000000) ENGINE = InnoDB,
PARTITION p87 VALUES LESS THAN (704000000,704000000) ENGINE = InnoDB,
PARTITION p88 VALUES LESS THAN (712000000,712000000) ENGINE = InnoDB,
PARTITION p89 VALUES LESS THAN (720000000,720000000) ENGINE = InnoDB,
PARTITION p90 VALUES LESS THAN (728000000,728000000) ENGINE = InnoDB,
PARTITION p91 VALUES LESS THAN (736000000,736000000) ENGINE = InnoDB,
PARTITION p92 VALUES LESS THAN (744000000,744000000) ENGINE = InnoDB,
PARTITION p93 VALUES LESS THAN (752000000,752000000) ENGINE = InnoDB,
PARTITION p94 VALUES LESS THAN (760000000,760000000) ENGINE = InnoDB,
PARTITION p95 VALUES LESS THAN (768000000,768000000) ENGINE = InnoDB,
PARTITION p96 VALUES LESS THAN (776000000,776000000) ENGINE = InnoDB,
PARTITION p97 VALUES LESS THAN (784000000,784000000) ENGINE = InnoDB,
PARTITION p98 VALUES LESS THAN (792000000,792000000) ENGINE = InnoDB,
PARTITION p99 VALUES LESS THAN (800000000,800000000) ENGINE = InnoDB,
PARTITION p100 VALUES LESS THAN (808000000,808000000) ENGINE = InnoDB,
PARTITION p101 VALUES LESS THAN (816000000,816000000) ENGINE = InnoDB,
PARTITION p102 VALUES LESS THAN (824000000,824000000) ENGINE = InnoDB,
PARTITION p103 VALUES LESS THAN (832000000,832000000) ENGINE = InnoDB,
PARTITION p104 VALUES LESS THAN (840000000,840000000) ENGINE = InnoDB,
PARTITION p105 VALUES LESS THAN (848000000,848000000) ENGINE = InnoDB,
PARTITION p106 VALUES LESS THAN (856000000,856000000) ENGINE = InnoDB,
PARTITION p107 VALUES LESS THAN (864000000,864000000) ENGINE = InnoDB,
PARTITION p108 VALUES LESS THAN (872000000,872000000) ENGINE = InnoDB,
PARTITION p109 VALUES LESS THAN (880000000,880000000) ENGINE = InnoDB,
...
PARTITION p497 VALUES LESS THAN (3984000000,3984000000) ENGINE = InnoDB,
PARTITION p498 VALUES LESS THAN (3992000000,3992000000) ENGINE = InnoDB
)
zu 500 Partitionen über den Tisch zu erzeugen. Dies funktioniert für alle explizite SELECT Suche nach INT-Wert fein wie folgt:
explain partitions select * from rangeTest where rangeStartInt = 3758084096 AND rangeEndInt = 3758085119;
+----+-------------+------------------------------+------------+------+---------------+---------+---------+-------------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------------------------+------------+------+---------------+---------+---------+-------------+------+----------+-------+
| 1 | SIMPLE | rangeTest | p469 | ref | PRIMARY | PRIMARY | 16 | const,const | 1 | 100.00 | NULL |
+----+-------------+------------------------------+------------+------+---------------+---------+---------+-------------+------+----------+-------+
Mit der entsprechenden Partition verwendet wird und die entsprechende Zeile zurückgegeben wird in der Nähe von 0.007s. Allerdings, wenn ich die weniger als und mehr als Betreiber in die Mischung der
explain partitions select * from rangeTest where rangeStartInt <= 3758084096 AND rangeEndInt >= 3758085119;
+----+-------------+------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+---------------+---------+---------+------+---------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+---------------+---------+---------+------+---------+----------+-------------+
| 1 | SIMPLE | rangeTest | p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44,p45,p46,p47,p48,p49,p50,p51,p52,p53,p54,p55,p56,p57,p58,p59,p60,p61,p62,p63,p64,p65,p66,p67,p68,p69,p70,p71,p72,p73,p74,p75,p76,p77,p78,p79,p80,p81,p82,p83,p84,p85,p86,p87,p88,p89,p90,p91,p92,p93,p94,p95,p96,p97,p98,p99,p100,p101,p102,p103,p104,p105,p106,p107,p108,p109,p110,p111,p112,p113,p114,p115,p116,p117,p118,p119,p120,p121,p122,p123,p124,p125,p126,p127,p128,p129,p130,p131,p132,p133,p134,p135,p136,p137,p138,p139,p140,p141,p142,p143,p144,p145,p146,p147,p148,p149,p150,p151,p152,p153,p154,p155,p156,p157,p158,p159,p160,p161,p162,p163,p164,p165,p166,p167,p168,p169,p170,p171,p172,p173,p174,p175,p176,p177,p178,p179,p180,p181,p182,p183,p184,p185,p186,p187,p188,p189,p190,p191,p192,p193,p194,p195,p196,p197,p198,p199,p200,p201,p202,p203,p204,p205,p206,p207,p208,p209,p210,p211,p212,p213,p214,p215,p216,p217,p218,p219,p220,p221,p222,p223,p224,p225,p226,p227,p228,p229,p230,p231,p232,p233,p234,p235,p236,p237,p238,p239,p240,p241,p242,p243,p244,p245,p246,p247,p248,p249,p250,p251,p252,p253,p254,p255,p256,p257,p258,p259,p260,p261,p262,p263,p264,p265,p266,p267,p268,p269,p270,p271,p272,p273,p274,p275,p276,p277,p278,p279,p280,p281,p282,p283,p284,p285,p286,p287,p288,p289,p290,p291,p292,p293,p294,p295,p296,p297,p298,p299,p300,p301,p302,p303,p304,p305,p306,p307,p308,p309,p310,p311,p312,p313,p314,p315,p316,p317,p318,p319,p320,p321,p322,p323,p324,p325,p326,p327,p328,p329,p330,p331,p332,p333,p334,p335,p336,p337,p338,p339,p340,p341,p342,p343,p344,p345,p346,p347,p348,p349,p350,p351,p352,p353,p354,p355,p356,p357,p358,p359,p360,p361,p362,p363,p364,p365,p366,p367,p368,p369,p370,p371,p372,p373,p374,p375,p376,p377,p378,p379,p380,p381,p382,p383,p384,p385,p386,p387,p388,p389,p390,p391,p392,p393,p394,p395,p396,p397,p398,p399,p400,p401,p402,p403,p404,p405,p406,p407,p408,p409,p410,p411,p412,p413,p414,p415,p416,p417,p418,p419,p420,p421,p422,p423,p424,p425,p426,p427,p428,p429,p430,p431,p432,p433,p434,p435,p436,p437,p438,p439,p440,p441,p442,p443,p444,p445,p446,p447,p448,p449,p450,p451,p452,p453,p454,p455,p456,p457,p458,p459,p460,p461,p462,p463,p464,p465,p466,p467,p468,p469 | range | PRIMARY | PRIMARY | 8 | NULL | 5048679 | 33.33 | Using where |
+----+-------------+------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+---------------+---------+---------+------+---------+----------+-------------+
Ein Add bin ich zurück zu 8s für die Abfrage. Also meine Frage ist, wie vermeide ich das? Ich denke, ich ein Array in meiner Anwendung (Python) müssen die, welche Partition ich immer nur mit einem Antrag für diese spezielle Partition trifft die DB müssen abzufragen und dann berechnet. Dies funktioniert und bringt die Abfragezeit um 0,008 Sekunden wieder herunter, aber es scheint ziemlich klobig zu sein, da die Tabelle regelmäßig über CSV-Importe aktualisiert wird und sich daher die oberen Endbereiche ändern können, was Änderungen am Python-Array erfordert und Fehler vermeidet .
Bin ich auf dem richtigen Weg, wenn ich die Partitionsfilterung aus MySQL nehme und diese im Python-Code behandle? Oder gibt es in MySQL etwas, das mir für "Größer als" und "Kleiner" fehlt als Abfragen für partitionierte Tabellen?
Oder ist es möglich wieder die Partitionsbereiche von MySQl in einer Abfrage abzurufen, die ich dann in der Python-Code verwenden können, die entsprechende Partition zu berechnen, die in der Abfrage zu benutzen?
Danke.