Support `auto_identifiers` extension for Typst

[MarvinJWendt]

Explain the problem.
When converting from typst to markdown and including a table of contents there are no links to the headings.

Typst:

= Heading
#lorem(5)

== Subheading
#lorem(5)

== Subheading 2
#lorem(5)

Converting

pandoc typst.typ --toc -s -o markdown.md

Output markdown (wrong):

-   Heading
    -   Subheading
    -   Subheading 2

# Heading

Lorem ipsum dolor sit amet,

## Subheading

Lorem ipsum dolor sit amet,

## Subheading 2

Lorem ipsum dolor sit amet,

Removing the toc and converting the markdown to markdown again with --toc

pandoc markdown.md --toc -s -o markdown2.md

(correct, should have been the output of the first command)

-   [Heading](#heading){#toc-heading}
    -   [Subheading](#subheading){#toc-subheading}
    -   [Subheading 2](#subheading-2){#toc-subheading-2}

# Heading

Lorem ipsum dolor sit amet,

## Subheading

Lorem ipsum dolor sit amet,

## Subheading 2

Lorem ipsum dolor sit amet,

The same issue happens when using other markdown flavours (I tested with gfm).

Note: I am unsure if this only happens with typst to md, might happen with other formats to markdown too.

Pandoc version?

pandoc 3.6
Features: +server +lua
Scripting engine: Lua 5.4

[tarleb]

Pandoc creates links in the TOC only if it has something to link to, i.e., if the heading has an identifier. There are no labels in your typst input, so pandoc can't link to the headings.

However, Pandoc's Markdown reader supports the auto_identifiers extension, which automatically adds identifiers based on the heading text. That means that all headings have an ID and are linkable. Hence the difference.

Try with this input:

= Heading <heading>
#lorem(5)

== Subheading <subheading>
#lorem(5)

== Subheading 2 <subheading2>
#lorem(5)

[MarvinJWendt]

Makes sense! Is there are reason that the typst reader does not support the auto_identifiers extension?

[tarleb]

The readers generally try to stay close to the reference implementations, and Typst itself doesn't have any such feature (AFAIK).
_{For Markdown, it could be claimed that pandoc is the reference implementation, so it's easier to do whatever makes people's life easier…}

You can work around this by using a Lua filter to generate the id. E.g.,

function Header (h)
  if not h.identifier then
    h.identifier = pandoc.utils.stringify(h):gsub('%s', '-')
    return h
  end
end

The above doesn't account for possible duplicate IDs, but should do ok otherwise.

[tarleb]

I've added a new Lua function to make this easier and more robust. It exposes the function that's used internally by auto_identifiers.

local exts = {'gfm_auto_identifiers'}
local collect_identifiers_into = function (used)
  return function (elem)
    if elem.attr and elem.identifier ~= '' then
      used[elem.identifier] = true
    end
  end
end
function Pandoc (doc)
  -- collect all ids used within the document
  local used_ids = {}
  doc:walk {
    Inline = collect_identifiers_info(used_ids)
    Block = collect_identifiers_info(used_ids) 
  }
  -- now generate IDs for all headings that don't have an ID yet.
  return doc:walk {
    Header = function (h)
      if not h.identifier then
        local id = pandoc.structure.unique_identifier(h.content, used_ids, exts)
        h.identifier = id
        return h
      end
    end
  }
end

[jgm]

I'm not sure there's any good reason not to make auto_identifiers work for typst, and I can see it would be useful for some things.