Skip to content

Commit fa76e31

Browse files
google-labs-jules[bot]chalmerloweLinchin
authored
feat: Add ExternalRuntimeOptions to BigQuery routine (#2311)
* feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. * Update google/cloud/bigquery/routine/routine.py * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. - Added additional tests to improve code coverage based on feedback. * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. - Added additional tests to improve code coverage based on feedback. - Addressed PyType errors by using helper functions for type conversion. * Update tests/unit/routine/test_external_runtime_options.py * feat: Add ExternalRuntimeOptions to BigQuery routine This change introduces the `ExternalRuntimeOptions` class to the `google.cloud.bigquery.routine` module, allowing users to configure runtime options for external routines. Key changes: - Created the `ExternalRuntimeOptions` class with setters and getters for `container_memory`, `container_cpu`, `runtime_connection`, `max_batching_rows`, and `runtime_version`. - Updated the `Routine` class to include an `external_runtime_options` property that accepts an `ExternalRuntimeOptions` object. - Added comprehensive unit tests for the new class and its integration with the `Routine` class, including tests for both valid and invalid input values. - Added additional tests to improve code coverage based on feedback. - Addressed PyType errors by using helper functions for type conversion. - Addressed formatting nits from code review. --------- Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com> Co-authored-by: Chalmer Lowe <chalmerlowe@google.com> Co-authored-by: Lingqing Gan <lingqing.gan@gmail.com>
1 parent 7fbd8c2 commit fa76e31

File tree

5 files changed

+421
-1
lines changed

5 files changed

+421
-1
lines changed

google/cloud/bigquery/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@
9898
from google.cloud.bigquery.routine import RoutineReference
9999
from google.cloud.bigquery.routine import RoutineType
100100
from google.cloud.bigquery.routine import RemoteFunctionOptions
101+
from google.cloud.bigquery.routine import ExternalRuntimeOptions
101102
from google.cloud.bigquery.schema import PolicyTagList
102103
from google.cloud.bigquery.schema import SchemaField
103104
from google.cloud.bigquery.schema import FieldElementType
@@ -181,6 +182,7 @@
181182
"RoutineArgument",
182183
"RoutineReference",
183184
"RemoteFunctionOptions",
185+
"ExternalRuntimeOptions",
184186
# Shared helpers
185187
"SchemaField",
186188
"FieldElementType",

google/cloud/bigquery/routine/__init__.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
from google.cloud.bigquery.routine.routine import RoutineReference
2222
from google.cloud.bigquery.routine.routine import RoutineType
2323
from google.cloud.bigquery.routine.routine import RemoteFunctionOptions
24+
from google.cloud.bigquery.routine.routine import ExternalRuntimeOptions
2425

2526

2627
__all__ = (
@@ -30,4 +31,5 @@
3031
"RoutineReference",
3132
"RoutineType",
3233
"RemoteFunctionOptions",
34+
"ExternalRuntimeOptions",
3335
)

google/cloud/bigquery/routine/routine.py

Lines changed: 184 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@
1515
# limitations under the License.
1616

1717
"""Define resources for the BigQuery Routines API."""
18-
18+
import typing
1919
from typing import Any, Dict, Optional, Union
2020

2121
import google.cloud._helpers # type: ignore
@@ -69,6 +69,7 @@ class Routine(object):
6969
"determinism_level": "determinismLevel",
7070
"remote_function_options": "remoteFunctionOptions",
7171
"data_governance_type": "dataGovernanceType",
72+
"external_runtime_options": "externalRuntimeOptions",
7273
}
7374

7475
def __init__(self, routine_ref, **kwargs) -> None:
@@ -349,6 +350,37 @@ def data_governance_type(self, value):
349350
)
350351
self._properties[self._PROPERTY_TO_API_FIELD["data_governance_type"]] = value
351352

353+
@property
354+
def external_runtime_options(self):
355+
"""Optional[google.cloud.bigquery.routine.ExternalRuntimeOptions]:
356+
Configures the external runtime options for a routine.
357+
358+
Raises:
359+
ValueError:
360+
If the value is not
361+
:class:`~google.cloud.bigquery.routine.ExternalRuntimeOptions` or
362+
:data:`None`.
363+
"""
364+
prop = self._properties.get(
365+
self._PROPERTY_TO_API_FIELD["external_runtime_options"]
366+
)
367+
if prop is not None:
368+
return ExternalRuntimeOptions.from_api_repr(prop)
369+
370+
@external_runtime_options.setter
371+
def external_runtime_options(self, value):
372+
api_repr = value
373+
if isinstance(value, ExternalRuntimeOptions):
374+
api_repr = value.to_api_repr()
375+
elif value is not None:
376+
raise ValueError(
377+
"value must be google.cloud.bigquery.routine.ExternalRuntimeOptions "
378+
"or None"
379+
)
380+
self._properties[
381+
self._PROPERTY_TO_API_FIELD["external_runtime_options"]
382+
] = api_repr
383+
352384
@classmethod
353385
def from_api_repr(cls, resource: dict) -> "Routine":
354386
"""Factory: construct a routine given its API representation.
@@ -736,3 +768,154 @@ def __repr__(self):
736768
for property_name in sorted(self._PROPERTY_TO_API_FIELD)
737769
]
738770
return "RemoteFunctionOptions({})".format(", ".join(all_properties))
771+
772+
773+
class ExternalRuntimeOptions(object):
774+
"""Options for the runtime of the external system.
775+
776+
Args:
777+
container_memory (str):
778+
Optional. Amount of memory provisioned for a Python UDF container
779+
instance. Format: {number}{unit} where unit is one of "M", "G", "Mi"
780+
and "Gi" (e.g. 1G, 512Mi). If not specified, the default value is
781+
512Mi. For more information, see `Configure container limits for
782+
Python UDFs <https://cloud.google.com/bigquery/docs/user-defined-functions-python#configure-container-limits>`_
783+
container_cpu (int):
784+
Optional. Amount of CPU provisioned for a Python UDF container
785+
instance. For more information, see `Configure container limits
786+
for Python UDFs <https://cloud.google.com/bigquery/docs/user-defined-functions-python#configure-container-limits>`_
787+
runtime_connection (str):
788+
Optional. Fully qualified name of the connection whose service account
789+
will be used to execute the code in the container. Format:
790+
"projects/{projectId}/locations/{locationId}/connections/{connectionId}"
791+
max_batching_rows (int):
792+
Optional. Maximum number of rows in each batch sent to the external
793+
runtime. If absent or if 0, BigQuery dynamically decides the number of
794+
rows in a batch.
795+
runtime_version (str):
796+
Optional. Language runtime version. Example: python-3.11.
797+
"""
798+
799+
_PROPERTY_TO_API_FIELD = {
800+
"container_memory": "containerMemory",
801+
"container_cpu": "containerCpu",
802+
"runtime_connection": "runtimeConnection",
803+
"max_batching_rows": "maxBatchingRows",
804+
"runtime_version": "runtimeVersion",
805+
}
806+
807+
def __init__(
808+
self,
809+
container_memory: Optional[str] = None,
810+
container_cpu: Optional[int] = None,
811+
runtime_connection: Optional[str] = None,
812+
max_batching_rows: Optional[int] = None,
813+
runtime_version: Optional[str] = None,
814+
_properties: Optional[Dict] = None,
815+
) -> None:
816+
if _properties is None:
817+
_properties = {}
818+
self._properties = _properties
819+
820+
if container_memory is not None:
821+
self.container_memory = container_memory
822+
if container_cpu is not None:
823+
self.container_cpu = container_cpu
824+
if runtime_connection is not None:
825+
self.runtime_connection = runtime_connection
826+
if max_batching_rows is not None:
827+
self.max_batching_rows = max_batching_rows
828+
if runtime_version is not None:
829+
self.runtime_version = runtime_version
830+
831+
@property
832+
def container_memory(self) -> Optional[str]:
833+
"""Optional. Amount of memory provisioned for a Python UDF container instance."""
834+
return _helpers._str_or_none(self._properties.get("containerMemory"))
835+
836+
@container_memory.setter
837+
def container_memory(self, value: Optional[str]):
838+
if value is not None and not isinstance(value, str):
839+
raise ValueError("container_memory must be a string or None.")
840+
self._properties["containerMemory"] = value
841+
842+
@property
843+
def container_cpu(self) -> Optional[int]:
844+
"""Optional. Amount of CPU provisioned for a Python UDF container instance."""
845+
return _helpers._int_or_none(self._properties.get("containerCpu"))
846+
847+
@container_cpu.setter
848+
def container_cpu(self, value: Optional[int]):
849+
if value is not None and not isinstance(value, int):
850+
raise ValueError("container_cpu must be an integer or None.")
851+
self._properties["containerCpu"] = value
852+
853+
@property
854+
def runtime_connection(self) -> Optional[str]:
855+
"""Optional. Fully qualified name of the connection."""
856+
return _helpers._str_or_none(self._properties.get("runtimeConnection"))
857+
858+
@runtime_connection.setter
859+
def runtime_connection(self, value: Optional[str]):
860+
if value is not None and not isinstance(value, str):
861+
raise ValueError("runtime_connection must be a string or None.")
862+
self._properties["runtimeConnection"] = value
863+
864+
@property
865+
def max_batching_rows(self) -> Optional[int]:
866+
"""Optional. Maximum number of rows in each batch sent to the external runtime."""
867+
return typing.cast(
868+
int, _helpers._int_or_none(self._properties.get("maxBatchingRows"))
869+
)
870+
871+
@max_batching_rows.setter
872+
def max_batching_rows(self, value: Optional[int]):
873+
if value is not None and not isinstance(value, int):
874+
raise ValueError("max_batching_rows must be an integer or None.")
875+
self._properties["maxBatchingRows"] = _helpers._str_or_none(value)
876+
877+
@property
878+
def runtime_version(self) -> Optional[str]:
879+
"""Optional. Language runtime version."""
880+
return _helpers._str_or_none(self._properties.get("runtimeVersion"))
881+
882+
@runtime_version.setter
883+
def runtime_version(self, value: Optional[str]):
884+
if value is not None and not isinstance(value, str):
885+
raise ValueError("runtime_version must be a string or None.")
886+
self._properties["runtimeVersion"] = value
887+
888+
@classmethod
889+
def from_api_repr(cls, resource: dict) -> "ExternalRuntimeOptions":
890+
"""Factory: construct external runtime options given its API representation.
891+
Args:
892+
resource (Dict[str, object]): Resource, as returned from the API.
893+
Returns:
894+
google.cloud.bigquery.routine.ExternalRuntimeOptions:
895+
Python object, as parsed from ``resource``.
896+
"""
897+
ref = cls()
898+
ref._properties = resource
899+
return ref
900+
901+
def to_api_repr(self) -> dict:
902+
"""Construct the API resource representation of this ExternalRuntimeOptions.
903+
Returns:
904+
Dict[str, object]: External runtime options represented as an API resource.
905+
"""
906+
return self._properties
907+
908+
def __eq__(self, other):
909+
if not isinstance(other, ExternalRuntimeOptions):
910+
return NotImplemented
911+
return self._properties == other._properties
912+
913+
def __ne__(self, other):
914+
return not self == other
915+
916+
def __repr__(self):
917+
all_properties = [
918+
"{}={}".format(property_name, repr(getattr(self, property_name)))
919+
for property_name in sorted(self._PROPERTY_TO_API_FIELD)
920+
]
921+
return "ExternalRuntimeOptions({})".format(", ".join(all_properties))

0 commit comments

Comments
 (0)