Dataflow pipline "Kontakt mit dem Dienst verloren"

Question

Ich habe Probleme mit einer Apache Beam Pipline in Google Cloud Dataflow.

Die Pipeline ist einfach: JSON aus GCS lesen, Text aus einigen verschachtelten Feldern extrahieren, zurück in GCS schreiben.

Es funktioniert gut beim Testen mit einer kleineren Teilmenge von Eingabedateien, aber wenn ich es auf den vollständigen Datensatz ausführen, erhalte ich den folgenden Fehler (nach dem Durchlaufen rund 260M Elemente).

Irgendwie ist der "Arbeiter schließlich verlor den Kontakt mit dem Service"

(8662a188e74dae87): Workflow failed. Causes: (95e9c3f710c71bc2): S04:ReadFromTextWithFilename/Read+FlatMap(extract_text_from_raw)+RemoveLineBreaks+FormatText+WriteText/Write/WriteImpl/WriteBundles/Do+WriteText/Write/WriteImpl/Pair+WriteText/Write/WriteImpl/WindowInto(WindowIntoFn)+WriteText/Write/WriteImpl/GroupByKey/Reify+WriteText/Write/WriteImpl/GroupByKey/Write failed., (da6389e4b594e34b): A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on: 
    extract-tags-150110997000-07261602-0a01-harness-jzcn, 
    extract-tags-150110997000-07261602-0a01-harness-828c, 
    extract-tags-150110997000-07261602-0a01-harness-3w45, 
    extract-tags-150110997000-07261602-0a01-harness-zn6v 

Der Stacktrace eine Failed to update work status/Progress reporting thread got error Fehler zeigt:

Exception in worker loop: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 776, in run deferred_exception_details=deferred_exception_details) File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 629, in do_work exception_details=exception_details) File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 168, in wrapper return fun(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 490, in report_completion_status exception_details=exception_details) File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 298, in report_status work_executor=self._work_executor) File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/workerapiclient.py", line 333, in report_status self._client.projects_locations_jobs_workItems.ReportStatus(request)) File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 467, in ReportStatus config, request, global_params=global_params) File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 723, in _RunMethod return self.ProcessHttpResponse(method_config, http_response, request) File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 729, in ProcessHttpResponse self.__ProcessHttpResponse(method_config, http_response, request)) File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 600, in __ProcessHttpResponse http_response.request_url, method_config, request) HttpError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/qollaboration-live/locations/us-central1/jobs/2017-07-26_16_02_36-1885237888618334364/workItems:reportStatus?alt=json>: response: <{'status': '400', 'content-length': '360', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 26 Jul 2017 23:54:12 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{ "error": { "code": 400, "message": "(7f8a0ec09d20c3a3): Failed to publish the result of the work update. Causes: (7f8a0ec09d20cd48): Failed to update work status. Causes: (afa1cd74b2e65619): Failed to update work status., (afa1cd74b2e65caa): Work \"6306998912537661254\" not leased (or the lease was lost).", "status": "INVALID_ARGUMENT" } } > 

Und schließlich:

HttpError: HttpError accessing <https://dataflow.googleapis.com/v1b3/projects/[projectid-redacted]/locations/us-central1/jobs/2017-07-26_18_28_43-10867107563808864085/workItems:reportStatus?alt=json>: response: <{'status': '400', 'content-length': '358', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Thu, 27 Jul 2017 02:00:10 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 'application/json; charset=UTF-8'}>, content <{ "error": { "code": 400, "message": "(5845363977e915c1): Failed to publish the result of the work update. Causes: (5845363977e913a8): Failed to update work status. Causes: (44379dfdb8c2b47): Failed to update work status., (44379dfdb8c2e88): Work \"9100669328839864782\" not leased (or the lease was lost).", "status": "INVALID_ARGUMENT" } } > 
at __ProcessHttpResponse (/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py:600) 
at ProcessHttpResponse (/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py:729) 
at _RunMethod (/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py:723) 
at ReportStatus (/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py:467) 
at report_status (/usr/local/lib/python2.7/dist-packages/dataflow_worker/workerapiclient.py:333) 
at report_status (/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py:298) 
at report_completion_status (/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py:490) 
at wrapper (/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py:168) 
at do_work (/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py:629) 
at run (/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py:776) 

Das sieht wie ein Fehler für den Datenfluss Internals für mich. Kann jemand bestätigen? Gibt es Workarounds?

Answer 1

Der HttpError erscheint normalerweise nach der Workflow ist fehlgeschlagen und ist Teil des Fehlers/Teardown-Prozesses.

Es sieht so aus, als ob in Ihrer Pipeline andere Fehler gemeldet wurden, z. Beachten Sie, dass wenn die gleichen Elemente 4 Mal fehlschlagen, wird dies als fehlgeschlagen markiert.

Versuchen Sie, den Abschnitt Stack Traces in der Benutzeroberfläche zu suchen, um die anderen Fehler und ihre Stack-Traces zu identifizieren. Da dies nur bei größeren Datasets auftritt, sollten Sie die Möglichkeit in Betracht ziehen, dass es sich um fehlerhafte Elemente handelt, die nur im größeren Dataset vorhanden sind.