rmk122--Next round on fonts and MCCS

[rmkaplan]

Many files have changed as I have rearchitected the font interfaces, introduced the Medley font-format files, and worked to normalize to the MCCS encoding of internal data.

This PR is a draft that doesn't yet include the flood of files. For starters, it just has the documentation files that describe the strategy and new and changed features. It will be helpful to get some initial reactions so I can make adjustments before pushing everything. The code and font files will come later.

[pamoroso]

On Linux Mint 22.1 Cinnamon this PR builds with no issues and I didn't notice anything unusual. I tested DInfo and NoteCards and the fonts look fine so far.

[rmkaplan]

I have uploaded all of the medleyformat fonts, and removed the draft status.

I expect/hope the only issue is that filenames with underscores and carets might not look the same in Medley vs Unix. I would like to get this tested and merged, to avoid confusion with the changes to the imagefile/hardcopy interfaces.

[pamoroso]

On Linux Mint 22.1 Cinnamon I tested up to commit d01503c and I haven't noticed anything unusual. Names containing ← or ↑ in TEdit look as expected on Linux, i.e. with _ and ^.

[rmkaplan]

I found that PRETTYFILEINDEX had its own compensation for underscore/arrow, at least for Interpress. Removed it.

[rmkaplan]

@MattHeffron , LOADUPFULLFONTS on internal/loadups/LOADUP-FULL has some special code for building the postscript fontcache. I wonder whether this should be done when POSTSCRIPTSTREAM is loaded, as part of POSTSCRIPT.INIT.

[rmkaplan]

I pushed a revised set of medleyformat fonts, plus a few code files with some glitch fixes. The new fonts are built in a sequence that first moves character information in legacy encodings (XCCS$, Altotext...) to MCCS positions in all fonts, then replaces slugs with information it may grab from other MCCS fonts (according to DISPLAYCHARCOERCIONS). Probably no difference from before for the standard encodings (which mostly had arrow/underscore shifts within a font), but I think the coercions for Palatino are now correct.

[pamoroso]

I tested up to commit b747e0f and I have nothing unusual to report other the the persistence of the Terminal font issue.

[rmkaplan]

I loaded VTCHAT, selected Chat/shell from the background menu. I see the left/right arrows when I type the keys they are assigned to (shift-6, shift hyphen), as I would expect, independent of the font. But the function keys don't work. I see a black string of f's for each of the keys, no matter which key of f4/46/f10 I press. The other function keys just flash the window.

[pamoroso]

I also see the left and up arrows when typing _ and ^. By function 10 and 6 do you mean the F10 and F6 keys?

[rmkaplan]

I investigated a bit further, a little progress.

The input characters are handled by the function CHAT.TYPEIN. It gets the codes from the keyboard (e.g. 0,255 for F-10, the MCCS underscore code) and puts it on the Chat OUTPUTSTREAM, a Unix-pty stream. That's a UTF-8 stream, with outcharfn UTF8.OUTCHARFN. That functions does the MCCS to Unicode coercion, mapping 0,255 to the Unicode underscore 0,137.

But CHAT.TYPEIN does a direct BOUT to the stream, so the translation doesn't happen. That's a case that I apparently missed when I did the external format transformation. With BOUT, the 0,255 gets put on the Unix stream directly, and presumably what I see as is the echo coming from the shell side--the shell doesn't like that character.

Changing the BOUT's to \OUTCHARFN improves things. The goes away but instead is replaced by the left-arrow glyph. My bet now is that the window's display stream is actually just being used for echoing from the shell side, and the echo that's being provided is the already converted 0,137 (the MCCS left arrrow). That is, the keyboards 0,255 is sent down as 0,137, and then send back for echoing as 0,137.

So that has to be translated the other way (UTOMCODE) before it is displayed in the new MCCS fonts. I'm still looking at that.

[nbriggs]

I think this is going to break the terminal emulation -- the CHAT code is processing the returned codes as ASCII bytes to emulate either (in the two common cases) a Datamedia DM2500 or a DEC VT100 terminal. At times, the characters received are going to be used as binary values for things like cursor positioning.

[nbriggs]

You should be able to download the ncurses examples from https://invisible-island.net/ncurses/ncurses-examples.html (untar the file, cd ncurses-examples-20211021, mkdir build, cd build, ../configure, make) then you'll find a collection of test/example/demo programs that make use of the "ncurses" package to drive ASCII graphic displays in the terminal.
Try, for example, ./xmas in a dm2500 chat, and in a vt100 chat.

[pamoroso]

Another useful resource is the VTTEST tool which "tests the compatibility (demonstrates the non-compatibility) of so-called “VT100-compatible” terminals"

[rmkaplan]

I think this is going to break the terminal emulation -- the CHAT code is processing the returned codes as ASCII bytes to emulate either (in the two common cases) a Datamedia DM2500 or a DEC VT100 terminal. At times, the characters received are going to be used as binary values for things like cursor positioning.

Is the problem that this interface doesn't use Unicode/utf-8, but something else (ISO8859/1 ?)

Should the MTOUCODE (or MCCS to ...) then be done explicitly in CHAT.TYPEIN, and then let it do the BOUTs as before on those translated codes? And likewise the reverse. So a code 0,255 from the keyboard would get translated to 0,137, and then bouted. CHAT.TYPEIN already has a guard that it will only bout 8 bit codes, otherwise it flashes the window (which is what I was seeing for the other function keys).

And effectively do the reverse sequence the other way around--get the code back, maybe interpret it according to the protocol, but then translate actual characters back to MCCS.

Essentially not rely on the stream's external format--or define a separate external format that does the desired mappings

[nbriggs]

There are a few things to consider

• there are CHAT.KEYACTIONS that you can adjust if you want -- standard keyaction entries, which take effect when the chat window has the focus
• characters sent down to the shell should probably be by default ASCII, so shift-6 ^ as 0x5E, shift-minus _ as 0x5F -- I'm not sure what a non-US terminal (or terminal emulator) would do... that's an extended (and future) investigation
• characters coming up from the shell need to get processed as raw bytes before they get any translation and before they're displayed, and any actions that they imply would be performed then. This, https://espterm.github.io/docs/VT100%20escape%20codes.html has a description of what you'd expect a VT100 terminal to respond to, so our CHAT architecture should be able to support an emulator that wants to implement them. Presumably characters that are actually going to be displayed would need (1) translation to MCCS where appropriate, (2) translation to graphic symbols that aren't part of ASCII/ISO-8859-1 when special-graphics mode has been selected.

[rmkaplan]

There are a few things to consider

• there are CHAT.KEYACTIONS that you can adjust if you want -- standard keyaction entries, which take effect when the chat window has the focus

WRT CHAT.KEYACTIONS, I think they would have to be defined to return the MCCS codes that will translate to whatever the user wants the shell to see, they would then get converted

• characters sent down to the shell should probably be by default ASCII, so shift-6 ^ as 0x5E, shift-minus _ as 0x5F -- I'm not sure what a non-US terminal (or terminal emulator) would do... that's an extended (and future) investigation

If the user types Function-10, the keyaction table will produce the MCCS code for underscore 0,255, the translation will take it to 0,137, and reverting to the previous BOUT will just send that code.

• characters coming up from the shell need to get processed as raw bytes before they get any translation and before they're displayed, and any actions that they imply would be performed then. This, https://espterm.github.io/docs/VT100%20escape%20codes.html has a description of what you'd expect a VT100 terminal to respond to, so our CHAT architecture should be able to support an emulator that wants to implement them. Presumably characters that are actually going to be displayed would need (1) translation to MCCS where appropriate, (2) translation to graphic symbols that aren't part of ASCII/ISO-8859-1 when special-graphics mode has been selected.

There is a function TERM.PRINTCHAR in CHATTERMINAL that seems to be the one that puts the character on the display, after all special interpretation has been done. Putting the inverse translator there produces the MCCS code that the MCCS fonts expect.

Assuming that the desired 8-bit codes are essentially Unicode charset 0/ISO8859-1, and putting MTOUCODE and UTOMCODE in one or two places seems to make it behave as expected, at least with respect to what shows up in the window with any of the new MCCS fonts: underscores when you type F-10, circumflex when you type F-6. Flashes the window when you type shift-6 (uparrow) or shift-hyphen (left arrow).

So this seems like an easy fix...except that back-space doesn't work as I would expect. The Medley level keyaction table produces code 8 (CHARCODE BS) for the Delete key. But when I type Delete in the chat window the \GETKEY in CHAT.TYPEIN returns 127 (CHARCODE DEL). This doesn't exist in MCCS, so MTOUCODE would convert it to an unassigned Unicode, and the window would flash. To get it to work I have to brute-force exclude (CHARCODE DEL) from the translation.

But, I'm not sure why the DEL is coming up, if this is the way backspace is supposed to be communicated. Or some sort of accident.

(One other note: in release and in this PR, there seems to be an off-by-one bug in the backspace. It shows up if you backspace over an underscore--the right most pixel remains black.)

[rmkaplan]

A second thought re (CHARCODE DEL): Another solution would be to simply add (CHARCODE DEL) to MCCS, with an identity mapping into Unicode/ISO. And forget about this glitch.

[pamoroso]

I also noticed some minor screen corruption when pressing Backspace in certain cases but couldn't reproduce it.

[rmkaplan]

Chat with MCCS fonts now works for me, for simple inputs (underscore) and backspace. I haven't done more elaborate testing.

Added DEL as an MCCS known character

[rmkaplan]

One other Chat improvement: I modernized its window, so you can drag it around by the title and reshape by the corners.

[nbriggs]

What do I need to put on CHAT.KEYACTIONS to make shift-6 send ASCII caret and shift-minus send underscore? There's no way I'm going to be able to change my touch typing to use different keystrokes between a regular terminal window and a CHAT window.

[rmkaplan]

Also on Chat, I didn't look at the back-space display glitch. And noticed that now and in the release, when it starts up there are some garbage characters on the screen. This is a snap from the release:

[nbriggs]

The garbage is because the shell it's bringing up thinks it's talking to an xterm-256color, but the CHAT terminal emulation is a dm2500 (by default) - xterm controls that the shell is sending show as garbage on a dm2500 terminal. I suspect that something tried to pass down to the shell info about the terminal emulator type, but it failed.

[rmkaplan]

If you want it to generally switch the bindings of shift-6/f-6 and shift-hyphen/f10 (but not shift-4/f-4), that's in the general keyaction table in LLKEY. If people think it's better that the characters align with the labels on current keyboards, then we can revert that change. I think it was discussed somewhere. You would then have to type f-10 to get the left-arrrow in create expressions--maybe dwim could be fixed to coerce _ to ←.

…

On Oct 11, 2025, at 1:44 PM, Nick Briggs ***@***.***> wrote: nbriggs left a comment (Interlisp/medley#2280) <#2280 (comment)> What do I need to put on CHAT.KEYACTIONS to make shift-6 send ASCII caret and shift-minus send underscore? There's no way I'm going to be able to change my touch typing to use different keystrokes between a regular terminal window and a CHAT window. — Reply to this email directly, view it on GitHub <#2280 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJN2YVC5JYHWYMNKTVT3XFTZDAVCNFSM6AAAAACF4OLO3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOJTGY2DOMBRHA>. You are receiving this because you were mentioned.

[rmkaplan]

Is that something to worry about?

…

On Oct 11, 2025, at 1:51 PM, Nick Briggs ***@***.***> wrote: nbriggs left a comment (Interlisp/medley#2280) <#2280 (comment)> The garbage is because the shell it's bringing up thinks it's talking to an xterm-256color, but the CHAT terminal emulation is a dm2500 (by default) - xterm controls that the shell is sending show as garbage on a dm2500 terminal. I suspect that something tried to pass down to the shell info about the terminal emulator type, but it failed. — Reply to this email directly, view it on GitHub <#2280 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQSTUJN2TBJ7CF34DVHBART3XFUURAVCNFSM6AAAAACF4OLO3CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGOJTGY2TANRRGY>. You are receiving this because you were mentioned.

[nbriggs]

Not generally, just in CHAT, which I think should be doable using the CHAT.KEYACTIONS, right? I'm fine with using the underscore key to get a left-arrow in an exec (or SEdit), I'm not sure how I feel about it typing file name patterns in a file browser, but when I'm typing at a Unix shell through a CHAT window it's got to be like every other Unix shell.

We should fix the way it tries to tell the shell about the terminal type. It works better if you have VTCHAT loaded.

[nbriggs]

We're actually sending the correct info for the terminal type - it gets set to dm2500 in the default chat (implementation in DMCHAT). I have a test for LDESHELL in my .zshrc so that I try to set a simpler prompt -- I guess I need to look at what DMCHAT.HANDLECHARACTER is doing with <esc>[?2004h --

1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 27
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 91
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 63
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 50
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 48
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 48
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 52
1 - Exit DMCHAT.HANDLECHARACTER =>
1 - Enter DMCHAT.HANDLECHARACTER:
          CHAR = 104
1 - Exit DMCHAT.HANDLECHARACTER =>

[nbriggs]

I was wrong about the source of the garbage -- it's the zsh line editor (zle) trying to say that we should "set bracketed paste mode". CSI here is <esc>[, not sure why they used mixed Pm/Ps in the description.

CSI ? Pm h
          DEC Private Mode Set (DECSET).
[...]
            Ps = 2 0 0 4  ⇒  Set bracketed paste mode, xterm.
[...]

This is what gets you the reverse-highlighted text (and no activation if there's a linefeed) when you paste into a terminal window.

We could fix up the DMCHAT code to parse the set/reset modes, but the easiest thing (on a mac, with zsh) is to
unset zle_bracketed_paste, except there are some subtleties (see below)

My ~/.zshrc contains (with a few other things)

ssh-add --apple-use-keychain -q
export OSTYPE=$OSTYPE
[ -v LDESHELL ] && source ~/.zshrc-medley

and my ~/.zshrc-medley now contains

export PROMPT="%n %*> "
zmodload zsh/zle   # load it so we can change parameters
unset zle_bracketed_paste

[pamoroso]

I tested up to commit ab5c281. Now pressing F10, F6, and F4 in a Terminal 10 Chat window yields _, ^, and $, respectively. However, the behavior of Backspace and Del is as described earlier, i.e. deleting characters doesn't work.