Wie optimiert man eine Hive-Abfrage mit mehreren Zählungen (distinct ...) auf überlappende Perioden?

Question

Ich habe eine große Abfrage-Code Absatz hive wie dies unter:

select 
count(distinct case when click_day between ${hiveconf:dt_180} and ${hiveconf:dt_end} and recommend_flag=1 then productid else null end) as unique_hk_products_cnt_180d, 
count(distinct case when click_day between ${hiveconf:dt_90} and ${hiveconf:dt_end} and recommend_flag=1 then productid else null end) as unique_hk_products_cnt_90d, 
count(distinct case when click_day between ${hiveconf:dt_30} and ${hiveconf:dt_end} and recommend_flag=1 then productid else null end) as unique_hk_products_cnt_30d, 
count(distinct case when click_day between ${hiveconf:dt_15} and ${hiveconf:dt_end} and recommend_flag=1 then productid else null end) as unique_hk_products_cnt_15d, 
count(distinct case when click_day between ${hiveconf:dt_7} and ${hiveconf:dt_end} and recommend_flag=1 then productid else null end) as unique_hk_products_cnt_7d 
from mytable ; 

der einzige Unterschied zwischen diesen Feldern die Anzahl der Tage ist, die die Länge des Zeitfensters darstellt. Das macht meine Abfrage sehr groß und es ist schwierig, Fehler zu machen.

dt_15 ist nur ein String-Variable definiert vor:

set dt_15 = CONCAT(SUBSTRING(date_sub(current_date,15), 1, 4), SUBSTRING(date_sub(current_date,15), 6, 2), SUBSTRING(date_sub(current_date,15), 9, 2)); 

Alle Jungs mir einfacher zu rekonstruieren können helfen? wie die Verwendung von Schleifen zu Produktfeldern in einer neuen Tabelle?

Danke.

Answer 1

Versuchen Sie, diese

select count (case when click_day between ${hiveconf:dt_180} and ${hiveconf:dt_end} then productid end) as unique_hk_products_cnt_180d 
     ,count (case when click_day between ${hiveconf:dt_90} and ${hiveconf:dt_end} then productid end) as unique_hk_products_cnt_90d 
     ,count (case when click_day between ${hiveconf:dt_30} and ${hiveconf:dt_end} then productid end) as unique_hk_products_cnt_30d 
     ,count (case when click_day between ${hiveconf:dt_15} and ${hiveconf:dt_end} then productid end) as unique_hk_products_cnt_15d 
     ,count (case when click_day between ${hiveconf:dt_7} and ${hiveconf:dt_end} then productid end) as unique_hk_products_cnt_7d 

from (select click_day,recommend_flag,productid 
       ,row_number() over 
       (
        partition by productid 
        order by  click_day desc  
       ) as rn 

     from mytable 

     where click_day between ${hiveconf:dt_180} and ${hiveconf:dt_end} 
      and recommend_flag=1 
     ) t 

where rn = 1 

P. S.
Gründe, warum Sie Ihre Daten in einem nicht standardmäßigen Format speichern?

Answer 2

Try this: Verwendung build-in-Datum Funktion

set dt_15 = from_unixtime(unix_timestamp(date_sub(current_date,15),'yyyy-mm-dd'),'yyyymmdd') 

für Wert dieser Einstellung wird concat und Teilzeichenfolge Betrieb entfernen.

select 
count(case when click_day between ${hiveconf:dt_180} and ${hiveconf:dt_end} then productid else null end) as unique_hk_products_cnt_180d, 
count(case when click_day between ${hiveconf:dt_90} and ${hiveconf:dt_end} then productid else null end) as unique_hk_products_cnt_90d, 
count(case when click_day between ${hiveconf:dt_30} and ${hiveconf:dt_end} then productid else null end) as unique_hk_products_cnt_30d, 
count(case when click_day between ${hiveconf:dt_15} and ${hiveconf:dt_end} then productid else null end) as unique_hk_products_cnt_15d, 
count(case when click_day between ${hiveconf:dt_7} and ${hiveconf:dt_end} then productid else null end) as unique_hk_products_cnt_7d 
from (select distinct click_day,productid where recommend_flag = 1) tmp ; 

Dies reduziert die Eingangslautstärke. Sie können auch die click_day < dt_end setzen, wenn es für alle Spalten gleich ist und entfernen Sie die zwischen.