Skip to content

GoogleCloudStorageDownloadOperator raises TypeError on Composer #2

@ricardolsmendes

Description

@ricardolsmendes

The GoogleCloudStorageDownloadOperator (used by the CompareXComMapsOperator in wordcount_test.py) works with Python 2.7 but raises a TypeError: Object of type 'bytes' is not JSON serializable when running in Python 3.6.

The error is easily reproducible in the below Cloud Composer Environment:

  1. Composer version: 1.16.8
  2. Airflow version: 1.10.15
  3. Python version: 3.6

Error details:

Traceback (most recent call last)
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 985, in _run_raw_tas
    result = task_copy.execute(context=context
  File "/usr/local/lib/airflow/airflow/contrib/operators/gcs_download_operator.py", line 96, in execut
    context['ti'].xcom_push(key=self.store_to_xcom_key, value=file_bytes
  File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 1516, in xcom_pus
    execution_date=execution_date or self.execution_date
  File "/usr/local/lib/airflow/airflow/utils/db.py", line 74, in wrappe
    return func(*args, **kwargs
  File "/usr/local/lib/airflow/airflow/models/xcom.py", line 101, in se
    value = XCom.serialize_value(value
  File "/usr/local/lib/airflow/airflow/models/xcom.py", line 224, in serialize_valu
    return json.dumps(value).encode('UTF-8'
  File "/opt/python3.6/lib/python3.6/json/__init__.py", line 231, in dump
    return _default_encoder.encode(obj
  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 199, in encod
    chunks = self.iterencode(o, _one_shot=True
  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 257, in iterencod
    return _iterencode(o, 0
  File "/opt/python3.6/lib/python3.6/json/encoder.py", line 180, in defaul
    o.__class__.__name__
TypeError: Object of type 'bytes' is not JSON serializabl

GoogleCloudStorageDownloadOperator is deprecated and probably needs to be replaced in order to fix the test workflow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions