Mangled locales `pt_BR.utf8`, `hu_HU.utf8`, `pl_PL.utf8` in GoboLinux 017

[nuc1eon]

A user on IRC reported that in tty console certain non-ASCI characters were mangled:

We have figured out that this is affecting the locales pt_BR-UTF-8, hu_HU-UTF-8 and pl_PL-UTF-8, specifically when assigned to $LC_ALL.

Upon checking further, it turned out the locales pt_BR-UTF-8, hu_HU-UTF-8 and pl_PL-UTF-8 are wrongly generated as ISO-8859 locales, which can be verified on GoboLinux 017 by running:

LC_ALL=hu_HU.utf8 locale charmap

the output will be ISO-8859-2, but obviously it should be UTF-8. Other locales, like en_US.utf8
or de_DE.utf8 (in fact all other *.utf8 locales) will correctly output UTF-8.

The solution is, to run the following command on affected locales:

localedef -i pt_BR -f UTF-8 pt_BR.UTF-8

This regenerates these locales permanently as UTF-8, which can again be verified with our previous check.

Looking through our source code, the only instance where localedef and these specific locales are called is in:

GoboALFS/GoboALFS

Lines 218 to 228 in 08d24e9

	# Install extra locales. Please refer to 'localedef --list-archive' on a regular desktop machine
	# for a list of common locales.
	InChroot localedef -i pt_BR -f ISO-8859-1 pt_BR
	InChroot localedef -i pt_BR -f UTF-8 pt_BR
	InChroot localedef -i pl_PL -f ISO-8859-2 pl_PL
	InChroot localedef -i pl_PL -f UTF-8 pl_PL
	InChroot localedef -i hu_HU -f ISO-8859-2 hu_HU
	InChroot localedef -i hu_HU -f UTF-8 hu_HU
	InChroot localedef -i ru_RU -f KOI8-R ru_RU
	InChroot localedef -i ru_RU -f UTF-8 ru_RU

I have not tested this yet, but it appears that the localedef command localedef -i pt_BR -f UTF-8 pt_BR is faulty...
It should be instead: localedef -i pt_BR -f UTF-8 pt_BR.UTF-8 (basically was missing an appended UTF-8).

We see that the unaffected ru_RU is also called there, however this entry has been added after our last ISO was generated, specifically version 016.01 (017/017.01 have been generated only by editing the previous): fcbc64e
Therefore at the moment, ru_RU is still unaffected.

Also, weirdly de_DE is not even included in the above list, but works fine when selected in the setup 🤔

This should be properly investigated and tested, so I am posting this here for future reference.

For now this will be fixed "manually" in our next ISO (we don't use GoboALFS atm but regenerate ISOs based on the previous).