`lineno` makes `lua-widow-control` think there are many nonexistent pages in the output

[d125q]

Issue

lineno seems to trick lua-widow-control into thinking there are (many) nonexistent pages in the output. In the MWE below, lua-widow-control complains about stuff on page 65 whereas the output has only 2 pages.

Expected Behavior

I expected that lua-widow-control wouldn’t have this weird interaction with lineno. Unfortunately, I can’t say if this is just a false alarm or if it actually prevents lua-widow-control from doing its job. (It definitely seems to be doing some job still.)

Reproduction

\documentclass[a4paper,11pt]{article}

\usepackage[american]{babel}

\usepackage[debug=true]{lua-widow-control}

\usepackage{lineno}%
\linenumbers{}

\usepackage{lipsum}

\begin{document}
\lipsum{}
\end{document}

Log File

Running `LaTeX' on `lineno+lua-widow-control' with ``lualatex --jobname=lineno\+lua-widow-control  -file-line-error -shell-escape  --synctex=1 -interaction=nonstopmode lineno\+lua-widow-control.tex''
This is LuaHBTeX, Version 1.16.0 (TeX Live 2023) 
 system commands enabled.
(./lineno+lua-widow-control.tex
LaTeX2e <2022-11-01> patch level 1
 L3 programming layer <2023-03-30>
($TEXMFDIST/tex/latex/base/article.cls
Document Class: article 2022/07/02 v1.4n Standard LaTeX document class
($TEXMFDIST/tex/latex/base/size11.clo))
($TEXMFDIST/tex/generic/babel/babel.sty
($TEXMFDIST/tex/generic/babel/luababel.def)
($TEXMFDIST/tex/generic/babel/luababel.def)
($TEXMFDIST/tex/generic/babel-english/american.ldf
($TEXMFDIST/tex/generic/babel-english/english.ldf)))
($TEXMFDIST/tex/generic/babel/locale/en/babel-american.tex)
($TEXMFDIST/tex/lualatex/lua-widow-control/lua-widow-control.sty)
($TEXMFDIST/tex/latex/lineno/lineno.sty
($TEXMFDIST/tex/latex/etoolbox/etoolbox.sty)) ($TEXMFDIST/tex/latex/lipsum/lipsum.sty
($TEXMFDIST/tex/latex/l3packages/l3keys2e/l3keys2e.sty
($TEXMFDIST/tex/latex/l3kernel/expl3.sty
($TEXMFDIST/tex/latex/l3backend/l3backend-luatex.def)))
($TEXMFDIST/tex/latex/lipsum/lipsum.ltd.tex))
($TEXMFDIST/tex/latex/microtype/microtype.sty ($TEXMFDIST/tex/latex/graphics/keyval.sty)

($TEXMFDIST/tex/latex/microtype/microtype-luatex.def)
($TEXMFDIST/tex/latex/microtype/microtype.cfg)) (./lineno+lua-widow-control.aux)
($TEXMFDIST/tex/latex/base/ts1cmr.fd)
($TEXMFDIST/tex/latex/microtype/mt-LatinModernRoman.cfg)
($TEXMFDIST/tex/generic/hyph-utf8/loadhyph/loadhyph-la.tex UTF-8 Latin hyphenation patterns
($TEXMFDIST/tex/generic/hyph-utf8/patterns/tex/hyph-la.tex))
Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 1 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 11 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 13 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 19 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 21 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 29 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 31 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 36 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 38 on input line 13
 [1{$TEXMFSYSVAR/fonts/map/pdftex/updmap/pdftex.map}]
Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 48 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 50 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 56 on input line 13

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 58 on input line 14

Module lua-widow-control Warning: Widow/Orphan/Broken Hyphen NOT removed on page 65 on input line 14
 [2] (./lineno+lua-widow-control.aux))
 439 words of node memory still in use:
   4 hlist, 1 vlist, 1 rule, 4 glue, 3 kern, 1 glyph, 8 attribute, 48 glue_spec, 5 attribute_list, 1 write nodes
   avail lists: 1:5,2:542,3:181,4:111,5:329,6:10,7:7856,9:299,10:52,11:856
<$TEXMFDIST/fonts/opentype/public/lm/lmsans8-regular.otf><$TEXMFDIST/fonts/opentype/public/lm/lmroman10-regular.otf>
Output written on lineno+lua-widow-control.pdf (2 pages, 14287 bytes).
SyncTeX written on lineno+lua-widow-control.synctex.gz.
Transcript written on lineno+lua-widow-control.log.

TeX Output finished at Tue Apr  4 10:07:20

Format

LaTeX

Distribution

TeX Live

Distribution Version

2023

Lua-widow-control Version

v3.0.0

Other Relevant Packages and Versions

No response

Other

No response

[gucci-on-fleek]

This is definitely a bug. What's happening is that lineno is setting \interlinepenalty to a really small value (-100000) to force the output routine to run between every line. This prevents lua-widow-control from actually doing anything other than printing a bunch of false warnings.

This issue is much more complicated than it looks, so I don't have a solution quite yet. Hopefully I'll figure something a proper solution in the next few days; if not, I'll patch lua-widow-control to print a single warning and then disable itself if it detects lineno.

[gucci-on-fleek]

I've done some more testing and this looks really hard to fix. The two problems here are that (1) lineno messes with \outputpenalty, which confuses lua-widow-control, and (2) lineno calls the output routine after every line, which also confuses lua-widow-control. My initial testing shows that (1) isn't too hard to fix, but (2) is much more challenging.

lua-widow-control empties its paragraph cache each time that the output routine is called. But lineno calls the output routine with every line, so the paragraph cache is always empty and it can't find any paragraphs to expand. If I have lua-widow-control not empty the cache if it detects that the output routine was triggered by lineno, then the cache contains bogus entries from previous pages, and LuaTeX give lots of nasty errors if you try and expand a paragraph on a previous page.

I'll leave this issue open since I don't want to say that it's impossible to solve, but I also wouldn't expect a solution to be released any time soon.

[Udi-Fogiel]

@d125q I'm currently working on a package for line numbering in LuaTeX. Since it does not change any typesetting parameters, nor mess with the output routine it should have less problems. If you want to test it, you can find it here https://github.com/Udi-Fogiel/lua-lineno, but be ware that it is still in very early stage. Here is a minimal test code

\documentclass[a4paper,11pt]{article}

\usepackage[american]{babel}

\usepackage{lualineno}
\newcounter{lineno}
\deflineno
  {
    name = default
    preamble = {\stepcounter{lineno}}
    left = {\tiny\thelineno\kern.8em}
  }
\setlineno{default}

\usepackage[debug=true]{lua-widow-control}
\usepackage{lipsum}

\begin{document}
\lipsum
\end{document}

@gucci-on-fleek I've noticed that you use the pre_shipout_filter callback as well.
Are you doing any actual typesetting there? I just want to make sure we are
not competing against each other.

[gucci-on-fleek]

@Udi-Fogiel

I'm currently working on a package for line numbering in LuaTeX. Since it does not change any typesetting parameters, nor mess with the output routine it should have less problems. If you want to test it, you can find it here https://github.com/Udi-Fogiel/lua-lineno, but be ware that it is still in very early stage.

Ah neat, that looks quite nice.

I've noticed that you use the pre_shipout_filter callback as well. Are you doing any actual typesetting there? I just want to make sure we are not competing against each other.

Yes, but only to typeset the paragraph costs in draft mode, so it's fairly unimportant if that doesn't work. But I've tested it and it seems to be fine, regardless of which package is loaded first:

%% Setup
\documentclass{article}
\usepackage[textwidth=345pt, textheight=550pt, showframe]{geometry}

%% lualineno
\usepackage{lualineno}
\newcounter{lineno}
\deflineno{
    name = default
    preamble = {\stepcounter{lineno}}
    left = {\tiny\thelineno\kern.8em}
}
\setlineno{default}

%% lua-widow-control
\usepackage[draftoffset=\oddsidemargin + 1in]{lua-widow-control}

%% Filler content
\newcount\recurselevel
\long\def\dorecurse#1#2{%
    \recurselevel=0%
    \loop\ifnum\recurselevel<#1%
        \advance\recurselevel by 1\relax%
        #2%
    \repeat%
}

\def\test{
    \dorecurse{19}{Filler line filler line filler line. }

    \dorecurse{4}{\input{knuth}\par}

    \clearpage
}

%% Demonstration
\begin{document}
    \lwcsetup{disable} \test

    \lwcsetup{enable} \test

    \lwcsetup{draft} \test
\end{document}

But why are you using pre_shipout_filter for this? I'm sure that you've already looked into it, but pre_output_filter seems like it might be a better choice here. Actually, using contribute_filter might be even easier:

\documentclass[twocolumn]{article}
\setlength{\columnsep}{1cm}

\newcounter{lineno}
\def\printlinenumber{\llap{
    \stepcounter{lineno}%
    \tiny\thelineno%
    \quad%
}}

\usepackage{luacode}
\begin{luacode*}
    local hlist_id    = node.id("hlist")
    local line_subid  = table.swapped(node.subtypes("hlist"))["line"]
    local typeset_tok = token.create("printlinenumber")

    luatexbase.add_to_callback("contribute_filter", function(info)
        -- Only process boxes in the main vertical list
        if info ~= "box" or tex.nest.ptr ~= 0 then
            return
        end

        -- Only process lines
        local line = tex.nest[0].tail or {}
        if line.id ~= hlist_id or line.subtype ~= line_subid then
            return
        end

        -- Process the line
        tex.runtoks(function()
            -- Get a box holding the line number
            token.put_next(typeset_tok)
            local number_box = token.scan_list()

            -- Insert the box at the beginning of the line
            line.head = node.insert_before(line.head, line.head, number_box)
        end)
    end, "line_numbers")
\end{luacode*}

\usepackage{babel, lipsum}

\begin{document}
    \lipsum
\end{document}

This isn't compatible with lua-widow-control as-is, but that's probably fixable.

[Udi-Fogiel]

Ah neat, that looks quite nice.

Thanks!

Yes, but only to typeset the paragraph costs in draft mode, so it's fairly unimportant if that doesn't work. But I've tested it and it seems to be fine, regardless of which package is loaded first:

Good.

But why are you using pre_shipout_filter for this? I'm sure that you've already looked into it, but pre_output_filter seems like it might be a better choice here. Actually, using contribute_filter might be even easier:

The whole idea started only several days ago when I encountered this post, so the main thing I wanted to tackle is balanced columns. Since balancing is done just before shipout, you can't tell in which column the line will end up until shipout, so I did not even think about other callbacks (but i'm more than open to suggestions).

In the long run I'll probably want to add several options, for different algorithms. After thinking about it for several days I don't every will agree on what should be considered a line.

[Udi-Fogiel]

There are also people who wants to number footnotes lines
https://ctan.org/pkg/fnlineno
and equations and alignments and...

[gucci-on-fleek]

@Udi-Fogiel

But why are you using pre_shipout_filter for this? I'm sure that you've already looked into it, but pre_output_filter seems like it might be a better choice here. Actually, using contribute_filter might be even easier:

The whole idea started only several days ago when I encountered this post, so the main thing I wanted to tackle is balanced columns. Since balancing is done just before shipout, you can't tell in which column the line will end up until shipout, so I did not even think about other callbacks (but i'm more than open to suggestions).

Ahhh, that makes sense then. In that case, I agree that pre_shipout_filter is the best option. I apparently had the same issue with trying to figure out which side of the page a column is on last year (although I oddly have no memory of that), and I also used pre_shipout_filter:

lua-widow-control/source/lua-widow-control.lua

Lines 1200 to 1201 in 96523a2

	--- Called immediately before the page is shipped out so that we can get
	--- the costs on the correct side in multi-column layouts.

Actually, the more “fun” solution would be rewriting multicol in Lua. This would then let you have columns of uneven widths, even in the middle of the paragraph. Something like https://tex.stackexchange.com/a/695271/270600 would get you the changing column widths, and balancing the column heights from Lua shouldn't be that hard either. Although I can think of a hundred different edge cases with this, so probably easiest to just use multicol :)

In the long run I'll probably want to add several options, for different algorithms. After thinking about it for several days I don't every will agree on what should be considered a line.

There are also people who wants to number footnotes lines https://ctan.org/pkg/fnlineno and equations and alignments and...

Yeah, you'll probably need to make the package very configurable for this. At least it should be fairly easy to do this from Lua, where with TeX macros it would be borderline impossible.

[Udi-Fogiel]

Ahhh, that makes sense then. In that case, I agree that pre_shipout_filter is the best option. I apparently had the same issue with trying to figure out which side of the page a column is on last year (although I oddly have no memory of that), and I also used pre_shipout_filter:

lua-widow-control/source/lua-widow-control.lua

Lines 1200 to 1201 in 96523a2

	--- Called immediately before the page is shipped out so that we can get
	--- the costs on the correct side in multi-column layouts.

Hmmm, this code looks rather familiar :)
https://tex.stackexchange.com/questions/689505/automatic-vertical-skip-in-margin-notes-in-optex

Actually, the more “fun” solution would be rewriting multicol in Lua. This would then let you have columns of uneven widths, even in the middle of the paragraph. Something like https://tex.stackexchange.com/a/695271/270600 would get you the changing column widths, and balancing the column heights from Lua shouldn't be that hard either. Although I can think of a hundred different edge cases with this, so probably easiest to just use multicol :)

Just thinking about dealing with inserts in multiple columns gives me headache...

In the long run I'll probably want to add several options, for different algorithms. After thinking about it for several days I don't every will agree on what should be considered a line.

There are also people who wants to number footnotes lines https://ctan.org/pkg/fnlineno and equations and alignments and...

Yeah, you'll probably need to make the package very configurable for this. At least it should be fairly easy to do this from Lua, where with TeX macros it would be borderline impossible.

Yes, probably checking node's (sub)type covers several possible configurationwith few lines of code. What bothers me more now are "nested" lines. Especially in LaTeX, where tabular, minipage, array are all nested inside a vertical box inside a line or an equation.

I thought about using similar approach to your column detecting code and analyze the the content of the lines while recording the horizontal shift, then if the first thing in the line is, say, tabulr, consider the original line as "fake line" and do the numbering according to the tabular rows while shifting the boxed numbers according to the shift recorded. There are also "fake" alignment which contains only rules, maybe they should not be numbered.