Skip to content
This repository was archived by the owner on Sep 28, 2022. It is now read-only.
This repository was archived by the owner on Sep 28, 2022. It is now read-only.

GenericParser should be resilient to bad fields #53

@jaffee

Description

@jaffee

The generic parser currently returns an error any time it fails to parse a field's value, and stops processing for that record. In reality, there are lots of innocuous reasons why a certain field might fail to parse, and it doesn't indicate that the entire record is suspect.

We should have stat/log options for notifying and counting which fields are failing, why, and how often, but we should make a best effort to parse any record and return some data.

I don't think we need to go crazy with configurability, some count stats that use the field path and encapsulate what the error is (e.g. null value, unsupported type, etc.). pdk/ingest.go has an example of a simple stats interface, that we should probably extend throughout the codebase so that it can be configured at the top level.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions