BigQuery: Nummer kann nicht in wissenschaftlicher Notation gespeichert werden

Question

Beim Speichern von BigQuery-Daten zwischen 22:00 und 5:00 UTC treten Probleme auf. Das Problem begann um Mitternacht am 7. Februar 2017 und ist noch heute nicht zu Ende.

Jeden Tag in diesem Zeitraum haben wir eine große Anzahl von Fehlern beim Einfügen von Daten. Die Anzahl der Ausfälle nimmt drastisch ab, bis sie in den nächsten Stunden verschwindet.

Das Datenformat, das wir einfügen, ist immer den ganzen Tag gleich.

Dies ist einer der „n“ Fehler, die wir in der Antwort des Einsatzes Anfrage gefunden:

{"debugInfo":"generic::invalid_argument: Cannot convert value to integer (bad value):8.3163161541138022e+17","location":"id","message":"Cannot convert value to integer (bad value):8.3163161541138022e+17","reason":"invalid"} 

Es scheint, das Problem zu Zahlen mit wissenschaftlicher Notation verwandt ist. Wie oben erwähnt, führen wir Speichervorgänge mit dieser Art von Zahlen den ganzen Tag, aber wir stoßen auf das Problem nur in einem Zeitrahmen des Tages seit dem 7. Februar 2017

Dies ist ein Ausdruck der Zeile (TableDataInsertAllRequest.Rows) verwandt zu dem obigen Fehler.

{insertId=831631615411380224, json={in_reply_to_status_id_str=null, in_reply_to_status_id=null, created_at_timestamp=2017-02-14 22:30:02, created_a 
t=Tue Feb 14 22:30:02 +0000 2017, in_reply_to_user_id_str=null, source=<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>, retweeted_status={in_reply_to_status_id_str=null, in_reply_to_status_id=null, cr 
eated_at=Tue Feb 14 15:56:07 +0000 2017, in_reply_to_user_id_str=null, source=<a href="http://www.keyakizaka46.com/" rel="nofollow">bot</a>, retweet_count=2.0, retweeted=false, geo=null, filter_level=low, in_reply_to_screen_name 
=null, is_quote_status=false, id_str=831532483447488513, in_reply_to_user_id=null, favorite_count=25.0, id=8.3153248344748851E17, text=2017/02/15 00:48, place=null, lang=ja, favorited=fal 
se, possibly_sensitive=false, coordinates=null, truncated=false, entities={urls=[{display_url=google.com/g0lSa9, indices=[31.0, 54.0], expanded_url=https://google.com/g0lSa9, url=https://twitter.com/W2Vpuwrq1F}], hashtags=[], user_mentions=[], symbol 
s=[]}, contributors=null, user={utc_offset=null, friends_count=5.0, profile_image_url_https=https://pbs.twimg.com/profile_images/665843786015113217/126-R8pG_normal.jpg, listed_count=91.0, profile_background_image_url=http://abs.twimg.c 
om/images/themes/theme1/bg.png, default_profile_image=false, favourites_count=0.0, description=abc, created_at=Sat Nov 14 
10:18:54 +0000 2015, is_translator=false, profile_background_image_url_https=https://abs.twimg.com/images/themes/theme1/bg.png, protected=false, screen_name=keyaki_blog, id_str=4233385392, profile_link_color=1DA1F2, id=4.233385392E9, g 
eo_enabled=false, profile_background_color=C0DEED, lang=ja, profile_sidebar_border_color=C0DEED, profile_text_color=333333, verified=false, profile_image_url=http://pbs.twimg.com/profile_images/665843786015113217/126-R8pG_normal.jpg, t 
ime_zone=null, url=http://www.keyakizaka46.com/mob/news/diarShw.php?site=k46o&ima=2757&cd=member, contributors_enabled=false, profile_background_tile=false, profile_banner_url=https://pbs.twimg.com/profile_banners/4233385392/1447584504 
, statuses_count=22077.0, follow_request_sent=null, followers_count=11669.0, profile_use_background_image=true, default_profile=true, following=null, name=bot, location=tokyo, profile_sidebar_fill_color=DDEEF6, notifications=null}}, retweet_count=0.0, retweeted=false, geo=null, filter_level=low, in_reply_to_screen_name=null, is_quote_status=false, id_str=831631615411380224, in_reply_to_user_id=null, favorite_count=0.0, id=8.3163161541138022E+17, text=RT @keyaki_blog: 2017/02/15 00:48 https://twitter.co/W2Vpuwrq1F, place=null, lang=ja, favorited=false, possibly_sensitive=false, coordinates=null, truncated=false, timestamp_ms=1487111402739, entities={urls=[{display_url=google.com/g0lSa9, indices=[48.0, 71.0], expanded_url=https://google.com/g0lSa9, url=https://twitter.co/W2Vpuwrq1F}], hashtags=[], user_mentions=[{indices=[3.0, 15.0], screen_name=keyaki_blog, id_str=4233385392, name=bot, id=4.233385392E9}], symbols=[]}, process_instance_id=user-twitter-listener-st-1, contributors=null, user={utc_offset=-28800.0, friends_count=328.0, profile_image_url_https=https://pbs.twimg.com/profile_images/601593430666412032/PQioR336_normal.jpg, listed_count=26.0, profile_background_image_url=http://abs.twimg.com/images/themes/theme1/bg.png, default_profile_image=false, favourites_count=49710.0, description=null, created_at=Fri May 22 01:28:04 +0000 2015, is_translator=false, profile_background_image_url_https=https://abs.twimg.com/images/themes/theme1/bg.png, protected=false, screen_name=tonton_gash1, id_str=3222825788, profile_link_color=1DA1F2, id=3.222825788E9, geo_enabled=true, profile_background_color=C0DEED, lang=ja, profile_sidebar_border_color=C0DEED, profile_text_color=333333, verified=false, profile_image_url=http://pbs.twimg.com/profile_images/601593430666412032/PQioR336_normal.jpg, time_zone=Pacific Time (US & Canada), url=null, contributors_enabled=false, profile_background_tile=false, statuses_count=113901.0, follow_request_sent=null, followers_count=404.0, profile_use_background_image=true, default_profile=true, following=null, name=bb@Onlooker, location=aaa, profile_sidebar_fill_color=DDEEF6, notifications=null}}} 
14/02/2017 22:30:33 [ERROR] [95] [saveBulk] [Thread: Thread-1] ERROR BigQueryDataSaver: {insertId=831631616057425921, json={quoted_status={in_reply_to_status_id_str=null, in_reply_to_status_id=null, coordinates=null, created_at=Tue Feb 14 22:17:50 +0000 2017, truncated=false, in_reply_to_user_id_str=null, source=<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>, retweet_count=0.0, retweeted=false, geo=null, filter_level=low, in_reply_to_screen_name=null, is_quote_status=false, entities={urls=[], hashtags=[], user_mentions=[], symbols=[]}, id_str=831628543281025024, in_reply_to_user_id=null, favorite_count=0.0, id=8.3162854328102502E17, text=kaitlyn's baby pictures?, place=null, contributors=null, lang=en, user={utc_offset=-18000.0, friends_count=505.0, profile_image_url_https=https://pbs.twimg.com/profile_images/813191388967960577/Q6Wh4HyC_normal.jpg, listed_count=36.0, profile_background_image_url=http://pbs.twimg.com/profile_background_images/720373579259351041/mPcBv5UA.jpg, default_profile_image=false, favourites_count=48248.0, description=▽ liar, monster, snappy dresser ▽ → kaitlyn // 18 // she/her // antifa // studies history + sociology in scotland // #FreePalestine #AllBlackLivesMatter, created_at=Sun Apr 26 16:05:35 +0000 2009, is_translator=false, profile_background_image_url_https=https://pbs.twimg.com/profile_background_images/720373579259351041/mPcBv5UA.jpg, protected=false, screen_name=ghostcommie, id_str=35495052, profile_link_color=F5ABB5, id=3.5495052E7, geo_enabled=false, profile_background_color=89C9FA, lang=en, profile_sidebar_border_color=FFFFFF, profile_text_color=CE7834, verified=false, profile_image_url=http://pbs.twimg.com/profile_images/813191388967960577/Q6Wh4HyC_normal.jpg, time_zone=Quito, url=null, contributors_enabled=false, profile_background_tile=false, profile_banner_url=https://pbs.twimg.com/profile_banners/35495052/1471561647, statuses_count=59269.0, follow_request_sent=null, followers_count=836.0, profile_use_background_image=true, default_profile=false, following=null, name=a nearby galaxy, location=a few light years away, profile_sidebar_fill_color=78C0A8, notifications=null}, favorited=false}, in_reply_to_status_id_str=null, in_reply_to_status_id=null, created_at_timestamp=2017-02-14 22:30:02, created_at=Tue Feb 14 22:30:02 +0000 2017, in_reply_to_user_id_str=null, source=<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>, quoted_status_id=8.3162854328102502E+17, retweet_count=0.0, retweeted=false, geo=null, filter_level=low, in_reply_to_screen_name=null, is_quote_status=true, id_str=831631616057425921, in_reply_to_user_id=null, favorite_count=0.0, id=8.3163161605742592E+17, text=The people have spoken https://twitter.co/8yPASyGyww, place=null, lang=en, favorited=false, possibly_sensitive=false, coordinates=null, truncated=false, timestamp_ms=1487111402893, entities={urls=[{display_url=twitter.com/ghostcommie/st…, indices=[23.0, 46.0], expanded_url=https://twitter.com/ghostcommie/status/831628543281025024, url=https://twitter.co/8yPASyGyww}], hashtags=[], user_mentions=[], symbols=[]}, process_instance_id=user-twitter-listener-st-1, display_text_range=[0.0, 22.0], quoted_status_id_str=831628543281025024, contributors=null, user={utc_offset=-18000.0, friends_count=505.0, profile_image_url_https=https://pbs.twimg.com/profile_images/813191388967960577/Q6Wh4HyC_normal.jpg, listed_count=36.0, profile_background_image_url=http://pbs.twimg.com/profile_background_images/720373579259351041/mPcBv5UA.jpg, default_profile_image=false, favourites_count=48248.0, description=▽ liar, monster, snappy dresser ▽ → kaitlyn // 18 // she/her // antifa // studies history + sociology in scotland // #FreePalestine #AllBlackLivesMatter, created_at=Sun Apr 26 16:05:35 +0000 2009, is_translator=false, profile_background_image_url_https=https://pbs.twimg.com/profile_background_images/720373579259351041/mPcBv5UA.jpg, protected=false, screen_name=ghostcommie, id_str=35495052, profile_link_color=F5ABB5, id=3.5495052E7, geo_enabled=false, profile_background_color=89C9FA, lang=en, profile_sidebar_border_color=FFFFFF, profile_text_color=CE7834, verified=false, profile_image_url=http://pbs.twimg.com/profile_images/813191388967960577/Q6Wh4HyC_normal.jpg, time_zone=Quito, url=null, contributors_enabled=false, profile_background_tile=false, profile_banner_url=https://pbs.twimg.com/profile_banners/35495052/1471561647, statuses_count=59270.0, follow_request_sent=null, followers_count=836.0, profile_use_background_image=true, default_profile=false, following=null, name=a nearby galaxy, location=a few light years away, profile_sidebar_fill_color=78C0A8, notifications=null}}} 

Answer 1

Ich habe ein wichtiges Update: Ich habe ein Problem in einem anderen Projekt auch auf Google Cloud. In diesem Projekt tritt das Problem 24h auf (nicht nur zwischen 22:00 und 5:00 UTC).

Wenn wir dieselbe Software auf einem Entwicklungscomputer in derselben BigQuery-Tabelle testen, tritt das Problem nicht auf.

Also haben wir die Software auf einer GCE-Maschine ausprobiert, aber in einer anderen Zone: europe-west1-c statt uns-central1-a. In diesem Bereich treten die Probleme nicht auf!

Hinweis: Die vorherige Nachricht bezieht sich auf ein anderes Projekt. Bei diesem Projekt passiert das Problem immer noch jeden Tag, aber nur zwischen 22:00 und 5:00 UTC. In diesem Projekt haben wir bereits Maschinen in Europa-West1-c.

Dies ist ein sehr seltsames Verhalten.