Skip to content

LLM Observability: Losing all type hints when using PostHog OpenAI wrapper #371

@alexeichhorn

Description

@alexeichhorn

Following this tutorial posthog.com/docs/llm-analytics/installation/openai, we setup everything in our existing quite large codebase.

However, we noticed already on the first example (exactly copied from tutorial):

response = client.responses.create(
    model="gpt-4o-mini",
    input=[
        {"role": "user", "content": "Tell me a fun fact about hedgehogs"}
    ],
    posthog_distinct_id="user_123", # optional
    posthog_trace_id="trace_123", # optional
    posthog_properties={"conversation_id": "abc123", "paid": True}, # optional
    posthog_groups={"company": "company_id_in_your_db"},  # optional 
    posthog_privacy_mode=False # optional
)

Our type-checker (pyright) reports:
error: No overloads for "create" match the provided arguments Argument types: (Literal['gpt-4o-mini'], list[dict[str, str]], Literal['user_123'], Literal['trace_123'], dict[str, Unknown], dict[str, str], Literal[False]) (reportCallIssue

Removing all posthog_-prefixed fields makes everything work nicely again.
To fix this we now chose to use it as kwargs instead:

posthog_args = {
    "posthog_distinct_id": "user_123",
    "posthog_trace_id": "trace_123",
    "posthog_properties": {"conversation_id": "abc123", "paid": True},
    "posthog_groups": {"company": "company_id_in_your_db"},
    "posthog_privacy_mode": False,
}

response = self.sync_openai_client.responses.create(
    model="gpt-4o-mini",
    input=[{"role": "user", "content": "Tell me a fun fact about hedgehogs"}],
    **posthog_args,
)

This no longer gives a type error, but now the response parameter has its type complete erased. It's now a Unknown.
This is our current setup that we have running now. Everything "works", BUT since we have a large codebase, it's just a matter of time something changes from OpenAI library side or maybe our side and something breaks. Usually no problem thanks to type-checks. But now, we won't notice it, since all types are gone.
So, we walking currently on a very dangerous path.

Would appreciate any comment on if this is planned to be improved or fixed. Would be very valuable to us.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions