[SUPPORT] Guidance on token counts for structured output schemas

[wilfredjonathanjames]

It's easy to find methods to count tokens in messages, but so far I've had no luck finding guidance on the additional token count incurred when attaching a structured output schema.

Pure JSON string encoding the schema gives roughly twice as many tokens as actual usage, sometimes more, sometimes less, so I'm clearly missing something important.

[wilfredjonathanjames]

I've come back to this today and have figured it out

In typescript it is

  encoding.encode(
    JSON.stringify(
      omit(format.schema, ["required", "additionalProperties", "$schema"]),
    ),
  ).length + 9

Where format is AutoParseableTextFormat from openai/lib/parser.mjs
and omit is omit-deep

I've tested this with different lengths and the deltas, previously not linear, now match.

[wilfredjonathanjames]

If you are using o4-mini, subtract one token even if format is undefined, and one token if you are including a format

  // here 👇
  if (modelKey === "openai__o4_mini") { // use your own o4-mini identifier
    numTokens--
  }

  if (format) {
    numTokens +=
      encoding.encode(
        JSON.stringify(
          omit(structuredClone(format.schema), [
            "required",
            "additionalProperties",
            "$schema",
          ]),
        ),
      ).length + 9
    // here 👇
    if (modelKey === "openai__o4_mini") {
      numTokens--
    }
  }

I have tested this against long/short inputs, and long/short/undefined schemas on models [ 'openai__gpt_4o_mini', 'openai__gpt_4o', 'openai__o4_mini' ]

I'm sure there is more guidance needed for other models