Hangs, unbounded memory consumption when \def\@setfontsize present in LaTeX input

[mikolysz]

Pandoc hangs indefinitely and consumes an unbounded amount of memory when the following LaTeX command is present in an input file:

\def\foo{\@setfontsize\foo\@ixpt{10}}

For this bug to trigger, the \foo command must be used somewhere in the document.

The minimal reproducer for this bug is as follows:

%reproducer.tex
\documentclass{article}
\def\foo{\@setfontsize\foo\@ixpt{10}}
\begin{document}
\foo
test

\end{document}

The bug can be triggered with the following invocation of Pandoc:

Pandoc reproducer.tex -o reproducer.txt

This bug occurs independently of:

• The used font size, e.g. \ixpt, \xpt.
• The output format (tested with Markdown, txt)
• The command name
• Whether the command is defined directly or in a .sty file.
• Whether the command is used directly in the document, or included e.g. in an environment definition which is ten used.

This bug does NOT appear if:

• the \renewcommand command is used instead of \def.
• The \foo command is defined but never used.

I have tested Pandoc versions 3.7.0.2, 3.1.3, the issue appears in both.

In the wild, the problematic command is used as part of the AAI2026.sty file. Here's A real document using this.

[jgm]

Pandoc doesn't generally handle low-level TeX stuff well.
In this case, the problem is that pandoc is recursively expanding \foo within its own definition.

Does this even work with latex? I get an error with your reproducer:

! You can't use `\spacefactor' in vertical mode.

If I rearrange to test \foo, I get a different error more akin to a loop:

! TeX capacity exceeded, sorry [input stack size=10000].
\@->\spacefactor \@m 
                     {}
l.6 test \foo

[mikolysz]

You're right. For some reason, this only works in LaTeX if the \def\ is part of a .sty file (this is how it has been in the original document I linked, before I simplified it to the minimalistic example presented in the issue).

For example, have reproducer.tex as follows:

\documentclass{article}
\usepackage{foo}  

\begin{document}

\small
test


\end{document}

And then foo.sty as follows:

\def\small{\@setfontsize\small\@ixpt{10}}    % 9 point on 10

This works in LaTeX but hangs in Pandoc.

[jgm]

That's really strange, and it shows that there's something I don't understand about how LaTeX works. I would have to understand why putting it in a .sty file makes this difference in order to know how to fix this.

[tarleb]

The main difference between direct input and .sty files seems to be that @ is a letter by default in .sty files. So the above example can be simplified to

\documentclass{article}
\makeatletter
\def\small{\@setfontsize\small\@ixpt{10}}
\makeatother
\begin{document}
\small test
\end{document}

This works in LaTeX but fails in pandoc.

The command \@setfontsize is defined in latex.ltx as

\def\@setfontsize#1#2#3{\@nomath#1%
    \ifx\protect\@typeset@protect
      \let\@currsize#1%
    \fi
    \fontsize{#2}{#3}\selectfont}

Adding all that plus \def\@nomath#1{\relax} to the input file makes the problem go away, as it allows pandoc to produce output.