mksprite: add bleedfix option to mitigate halos for transparent sprites WIP

[phoboslab]

This adds an optional switch to mksprite to mitigate halos around sprites with transparency.

The idea is to "radiate" the sprite outwards. These newly filled pixels will have the r,g,b values of the edges of the sprite, while still maintaining full transparency (a = 0). Only fully transparent pixels are modified; all pixels with partial transparency are left as is.

One complication arises with paletted sprites: in most cases the palette is already completely filled (with either 16 or 256 colors), so we can't create any new colors. The only option is to modify the existing palette entry with the transparent color to a single "background" color. This background color is simply computed as the average of all colors of the sprite's edges.

For now, all paletted images are filled with a single "background" color. We could be smarter about this if we have some free palette entries to use: create k-means of each of the edge colors and use those to "radiate" outwards, same as with rgba16/32. I'm not yet sure if it's worth to implement this...

I'm still doing some testing, to make sure it works as intended. I just wanted some early feedback if

• this is wanted in libdragon and
• this PR is going in the right direction.

Also, I'm open for suggestions for a better name. -bleedfix is not that descriptive :]

Examples (alpha value set to 255 for illustrative purposes):

Source

RGBA16

CI4

IA8

[rasky]

Thanks! I think the PR is promising and yes we want the feature.

For paletted images, I suggest to copy the approach we have for paletted LOD calculation: expand to RGBA, do whatever you want, then quantize back using the same palette as before. You can copy the LOD palette code verbatim.

Notice the code handles also the case of multiple, alternative 16-color palettes, and that will need to be handled as well, by restoring the full original palette.

I think it's important to keep the number of affected pixels to the minimum, to not affect also ROM size (a sprite after this fix has more data than before, so by the definition it will increase ROM size after compression). I see you have a macro for the number of iterations and I would say 1 should be enough, unless my math is wrong.

I don't have good immediate ideas on the option name, we can revisit that later. I'm still pondering whether this should be an opt-in or an opt-out.

[phoboslab]

[...] expand to RGBA, do whatever you want, then quantize back using the same palette as before.

How would that help? If there's only a single transparent color in the palette and we don't change the palette, then... there's only a single transparent color in the palette :F

From what I can see, exoquant doesn't seem to have a "keep these N colors from the original palette, but extend the palette with K new colors" function!? So, instead:

1. keep the original palette of length N, calculate the number of free slots K
2. expand to rgba
3. run bleedfix_rgba(), while collecting all new colors created
4. use exoquant to reduce all new colors to len K(?)
5. create a new palette with the N original colors and K new ones
6. quantize to new palette

[phoboslab]

Above commit now makes use of free colors in the palette. All the edge colors are fed to exq to produce a new palette, that is then appended to the existing one. It's a bit unfortunate that I have to create this "image" with all edge colors, instead of feeding a histogram to exq, but it works just fine. The sprite silhouette is then expanded by 1px (same as RGBA16/32/GREY) and re-quantized with the fully filled palette.

Easiest way to check if this all worked: load the sprite in libdragon and draw without alphamasking/blending. Alternatively, write a debug PNG (-d), load it in Photoshop, change image mode to RGB, then select layer -> layer mask -> from transparency and finally delete the layer mask to reveal the transparent colors. Sigh.

[phoboslab]

Two more things:

1. we could skip the whole "quantize all edge colors using exq to squeeze them into the remaining palette space" if we have less unique edge colors than free palette space. I'm OK with always using exq. It's not time sensitive, less code, not much slower(?) and this special case probably is not that common.
2. spr->palette.used_colors might not be accurate after bleedfix_adjust_palette() anymore. It's always expanded to 16 or 256 colors, regardless of how many are actually used. Should I count used_colors again in case a (future) next step expects it to be accurate?

Edit: addressed both points. I'd call this PR ready.

[thekovic]

Is the purpose of this feature to prevent an outline around transparent sprites when texture filtering is enabled?

[phoboslab]

Yes