Skip to content

feat: allow to skip wrapping data into custom types #1490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alvarowolfx wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat: allow to skip wrapping data into custom types #1490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 50 additions & 13 deletions src/bigquery.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@
pageToken?: string;
wrapIntegers?: boolean | IntegerTypeCastOptions;
parseJSON?: boolean;
skipWrapCustomTypes?: boolean;
// Overrides default job creation mode set on the client.
jobCreationMode?: JobCreationMode;
};
Expand All @@ -141,6 +142,8 @@
export type QueryOptions = QueryResultsOptions;
export type QueryStreamOptions = {
wrapIntegers?: boolean | IntegerTypeCastOptions;
skipWrapBig?: boolean;
skipWrapBigQueryCustomTypes?: boolean;
parseJSON?: boolean;
};
export type DatasetResource = bigquery.IDataset & {
Expand Down Expand Up @@ -369,17 +372,17 @@
private _universeDomain: string;
private _defaultJobCreationMode: JobCreationMode;

createQueryStream(options?: Query | string): ResourceStream<RowMetadata> {

Check warning on line 375 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
// placeholder body, overwritten in constructor
return new ResourceStream<RowMetadata>({}, () => {});
}

getDatasetsStream(options?: GetDatasetsOptions): ResourceStream<Dataset> {

Check warning on line 380 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
// placeholder body, overwritten in constructor
return new ResourceStream<Dataset>({}, () => {});
}

getJobsStream(options?: GetJobsOptions): ResourceStream<Job> {

Check warning on line 385 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
// placeholder body, overwritten in constructor
return new ResourceStream<Job>({}, () => {});
}
Expand Down Expand Up @@ -579,14 +582,16 @@
* Please see {@link IntegerTypeCastOptions} for options descriptions.
* @param {array} options.selectedFields List of fields to return.
* If unspecified, all fields are returned.
* @param {array} options.parseJSON parse a 'JSON' field into a JSON object.
* @param {boolean} options.parseJSON parse a 'JSON' field into a JSON object.
* @param {boolean} options.skipWrapCustomTypes skip converting values into BigQuery custom types.
* @returns Fields using their matching names from the table's schema.
*/
static mergeSchemaWithRows_(
schema: TableSchema | TableField | undefined,
rows: TableRow[],
options: {
wrapIntegers: boolean | IntegerTypeCastOptions;
skipWrapCustomTypes?: boolean;
Copy link
Contributor

@leahecole leahecole Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit - can this be the last option? I know order isn't necessarily preserved but I feel like we should have these options in the order in which we have added them

Copy link
Contributor

@leahecole leahecole Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

additional comment on this line - Any reason why we can't call this wrapCustomTypes? I know it would flip the boolean logic later on, but it would be in line with the wrapIntegers naming scheme

selectedFields?: string[];
parseJSON?: boolean;
},
Expand Down Expand Up @@ -1429,6 +1434,7 @@
* @param {boolean} [options.wrapIntegers] Optionally wrap INT64 in BigQueryInt
* or custom INT64 value type.
* @param {boolean} [options.parseJSON] Optionally parse JSON as a JSON Object.
* @param {boolean} [options.skipWrapCustomTypes] Optionally skip wrapping values into BigQuery Custom Types.
* @param {object|array} [options.params] Option to provide query prarameters.
* @param {JobCallback} [callback] The callback function.
* @param {?error} callback.err An error returned while making this request.
Expand Down Expand Up @@ -1589,7 +1595,7 @@
const parameterMode = is.array(params) ? 'positional' : 'named';
const queryParameters: bigquery.IQueryParameter[] = [];
if (parameterMode === 'named') {
const namedParams = params as {[param: string]: any};

Check warning on line 1598 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

Unexpected any. Specify a different type
for (const namedParameter of Object.getOwnPropertyNames(namedParams)) {
const value = namedParams[namedParameter];
let queryParameter;
Expand Down Expand Up @@ -2202,6 +2208,7 @@
? {
wrapIntegers: query.wrapIntegers,
parseJSON: query.parseJSON,
skipWrapCustomTypes: query.skipWrapCustomTypes,
}
: {};
const callback =
Expand Down Expand Up @@ -2237,10 +2244,11 @@

options = extend({job}, queryOpts, options);
if (res && res.jobComplete) {
let rows: any = [];

Check warning on line 2247 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

Unexpected any. Specify a different type
if (res.schema && res.rows) {
rows = BigQuery.mergeSchemaWithRows_(res.schema, res.rows, {
wrapIntegers: options.wrapIntegers || false,
skipWrapCustomTypes: options.skipWrapCustomTypes,
parseJSON: options.parseJSON,
});
delete res.rows;
Expand Down Expand Up @@ -2408,13 +2416,21 @@
return;
}

const {location, maxResults, pageToken, wrapIntegers, parseJSON} = query;
const {
location,
maxResults,
pageToken,
wrapIntegers,
parseJSON,
skipWrapCustomTypes,
} = query;

const opts = {
location,
maxResults,
pageToken,
wrapIntegers,
skipWrapCustomTypes,
parseJSON,
autoPaginate: false,
};
Expand All @@ -2423,6 +2439,7 @@
delete query.maxResults;
delete query.pageToken;
delete query.wrapIntegers;
delete query.skipWrapCustomTypes;
delete query.parseJSON;

this.query(query, opts, callback);
Expand Down Expand Up @@ -2462,14 +2479,18 @@
value: any,
options: {
wrapIntegers: boolean | IntegerTypeCastOptions;
skipWrapCustomTypes?: boolean;
Copy link
Contributor

@leahecole leahecole Jul 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same note about order of options

selectedFields?: string[];
parseJSON?: boolean;
},
) {
if (is.null(value)) {
return value;
}

let {skipWrapCustomTypes, wrapIntegers, parseJSON} = options;

Check failure on line 2490 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'parseJSON' is never reassigned. Use 'const' instead

Check failure on line 2490 in src/bigquery.ts

View workflow job for this annotation

GitHub Actions / lint 

'skipWrapCustomTypes' is never reassigned. Use 'const' instead
if (skipWrapCustomTypes) {
wrapIntegers = false;
}
switch (schemaField.type) {
case 'BOOLEAN':
case 'BOOL': {
Expand All @@ -2487,7 +2508,6 @@
}
case 'INTEGER':
case 'INT64': {
const {wrapIntegers} = options;
value = wrapIntegers
? typeof wrapIntegers === 'object'
? BigQuery.int(
Expand All @@ -2498,12 +2518,12 @@
: Number(value);
break;
}
case 'BIGNUMERIC':
case 'NUMERIC': {
value = new Big(value);
break;
}
case 'BIGNUMERIC': {
value = new Big(value);
if (skipWrapCustomTypes) {
value = value.toFixed();
}
break;
}
case 'RECORD': {
Expand All @@ -2512,36 +2532,52 @@
}
case 'DATE': {
value = BigQuery.date(value);
if (skipWrapCustomTypes) {
value = value.value;
}
break;
}
case 'DATETIME': {
value = BigQuery.datetime(value);
if (skipWrapCustomTypes) {
value = value.value;
}
break;
}
case 'TIME': {
value = BigQuery.time(value);
if (skipWrapCustomTypes) {
value = value.value;
}
break;
}
case 'TIMESTAMP': {
const pd = new PreciseDate();
pd.setFullTime(PreciseDate.parseFull(BigInt(value) * BigInt(1000)));
value = BigQuery.timestamp(pd);
if (skipWrapCustomTypes) {
value = value.value;
}
break;
}
case 'GEOGRAPHY': {
value = BigQuery.geography(value);
if (skipWrapCustomTypes) {
value = value.value;
}
break;
}
case 'JSON': {
const {parseJSON} = options;
value = parseJSON ? JSON.parse(value) : value;
break;
}
case 'RANGE': {
value = BigQueryRange.fromSchemaValue_(
value,
schemaField.rangeElementType!.type!,
);
value = skipWrapCustomTypes
? value
: BigQueryRange.fromSchemaValue_(
value,
schemaField.rangeElementType!.type!,
);
break;
}
default:
Expand Down Expand Up @@ -2645,6 +2681,7 @@
}
return convertSchemaFieldValue({type: elementType}, value, {
wrapIntegers: false,
skipWrapCustomTypes: false,
});
};
return BigQuery.range(
Expand Down
6 changes: 6 additions & 0 deletions src/job.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,13 @@
export type QueryResultsOptions = {
job?: Job;
wrapIntegers?: boolean | IntegerTypeCastOptions;
skipWrapCustomTypes?: boolean;
parseJSON?: boolean;
} & PagedRequest<bigquery.jobs.IGetQueryResultsParams> & {
/**
* internal properties
*/
_cachedRows?: any[];

Check warning on line 59 in src/job.ts

View workflow job for this annotation

GitHub Actions / lint 

Unexpected any. Specify a different type
_cachedResponse?: bigquery.IQueryResponse;
};

Expand Down Expand Up @@ -133,7 +134,7 @@
bigQuery: BigQuery;
location?: string;
getQueryResultsStream(
options?: QueryResultsOptions,

Check warning on line 137 in src/job.ts

View workflow job for this annotation

GitHub Actions / lint 

'options' is defined but never used
): ResourceStream<RowMetadata> {
// placeholder body, overwritten in constructor
return new ResourceStream<RowMetadata>({}, () => {});
Expand Down Expand Up @@ -556,6 +557,10 @@

const wrapIntegers = qs.wrapIntegers ? qs.wrapIntegers : false;
delete qs.wrapIntegers;
const skipWrapCustomTypes = qs.skipWrapCustomTypes
? qs.skipWrapCustomTypes
: false;
delete qs.skipWrapCustomTypes;
const parseJSON = qs.parseJSON ? qs.parseJSON : false;
delete qs.parseJSON;

Expand Down Expand Up @@ -597,6 +602,7 @@
if (resp.schema && resp.rows) {
rows = BigQuery.mergeSchemaWithRows_(resp.schema, resp.rows, {
wrapIntegers,
skipWrapCustomTypes,
parseJSON,
});
}
Expand Down
6 changes: 6 additions & 0 deletions src/table.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ export type TableRowValue = string | TableRow;

export type GetRowsOptions = PagedRequest<bigquery.tabledata.IListParams> & {
wrapIntegers?: boolean | IntegerTypeCastOptions;
skipWrapCustomTypes?: boolean;
parseJSON?: boolean;
};

Expand Down Expand Up @@ -1851,6 +1852,10 @@ class Table extends ServiceObject {
typeof optionsOrCallback === 'function' ? optionsOrCallback : cb;
const wrapIntegers = options.wrapIntegers ? options.wrapIntegers : false;
delete options.wrapIntegers;
const skipWrapCustomTypes = options.skipWrapCustomTypes
? options.skipWrapCustomTypes
: false;
delete options.skipWrapCustomTypes;
const parseJSON = options.parseJSON ? options.parseJSON : false;
delete options.parseJSON;
const selectedFields = options.selectedFields
Expand All @@ -1868,6 +1873,7 @@ class Table extends ServiceObject {
}
rows = BigQuery.mergeSchemaWithRows_(this.metadata.schema, rows || [], {
wrapIntegers,
skipWrapCustomTypes,
selectedFields,
parseJSON,
});
Expand Down
Loading
Loading