accessMode `visual` sometimes not inferred while jpg resources are included

[nicolascyr]

When using the inference mode, for some files, the accessMode visual is not inferred.

• it's not declared in the content.opf's schema:accessMode property
• the RWPM actually has image entries (with "type": "image/jpeg") in the resources list

Sample RWPMs:
https://assets.edenlivres.fr/medias/rwpm/4e0a3f0d53cbab7add7b4a7518e85d73c488b77d.json
https://assets.edenlivres.fr/medias/rwpm/970a82f9b9bd34e2ffa5992e87bca19bd574b99b.json
https://assets.edenlivres.fr/medias/rwpm/20c55e7bb989f405fb149ce4702f8d963544b3af.json
https://assets.edenlivres.fr/medias/rwpm/8933770ffe9a8abade9ea670e303ec7233b597ac.json

[HadrienGardeur]

@nicolascyr could you share these examples privately with us (Slack) so we can look into it? Thanks!

[chocolatkey]

@nicolascyr thanks for sharing. I've identifier the cause, you can follow along in this file's logic: https://github.com/readium/go-toolkit/blob/develop/pkg/streamer/a11y_infer.go#L15

The provided EPUB file already has accessibility metadata in the OPF:

<meta property="schema:accessMode">textual</meta>
<meta property="schema:accessModeSufficient">textual</meta>
<!-- other a11y meta here...-->

and the logic of the a11y inference currently includes the following:

• If a11y metadata was present in the source file, use it as the starting point for a11y inference
• If there are no current access modes in the current a11y metadata during inference, perform access mode inference
• If there are no current sufficient access modes in the current a11y metadata, perform sufficient access mode inference

This combo leads to no inference in the supplied EPUB. @HadrienGardeur as the inference code was not originally written by me, and you are far more familiar with the spec and intentions of the inference logic, could you let us know the next steps? Is this intended behavior or do we fix it and how?

[HadrienGardeur]

This combo leads to no inference in the supplied EPUB. @HadrienGardeur as the inference code was not originally written by me, and you are far more familiar with the spec and intentions of the inference logic, could you let us know the next steps? Is this intended behavior or do we fix it and how?

If I remember correctly, we followed this approach because we had less confidence in our ability to infer accessMode and accessModeSufficient than other properties.

For accessModeSufficient, we can further infer textual on its own now that we can finally inspect HTML files (which wasn't the case before), but other values (especially combinations) would be difficult to infer reliably.

I think that we can change the current rule that we have in place and allow the inference of this value, even when the publications already contains an accessModeSufficient property.

For accessMode, I think that it's fairly safe to infer textual when we detect text, but should we really infer visual when a publication contains for example a single logo on a copyright page? That's more controversial. That's why we initially decided to restrict this inference to publications who do not contain any accessMode.

Since this metadata has zero impact on what the user sees (based on the a11y display guide), is this really a showstopper for you @nicolascyr?

[nicolascyr]

I thought the current implementation was supposed to infer visual as soon as there was at least an image in the package, no matter what the image was : https://github.com/readium/cli/blob/develop/docs/manifest.md#list-of-inferred-metadata

So for the sample file, I was considering it as a bug, because there was an image file and visual was not inferred. However, I agree that ideally, visual should not be inferred with the mere presence of an image file.

It should be inferred (no matter the presence of a provided accessMode) only if there is at least an image that is not identified as the cover nor tagged as decorative in the package (which means, presence of images that contain meaningful information -- no matter if they have an alt text).

So this is not a showstopper, but I think it could be better to move towards this interpretation. Do you think this is possible with the new inspection possibilities?

This behavior would align the inferred metadata with the DAISY/Schema.org guidance, which clearly states that accessMode visual should not be triggered by covers or decorative images.

[HadrienGardeur]

It should be inferred (no matter the presence of a provided accessMode) only if there is at least an image that is not identified as the cover nor tagged as decorative in the package (which means, presence of images that contain meaningful information -- no matter if they have an alt text).

So this is not a showstopper, but I think it could be better to move towards this interpretation. Do you think this is possible with the new inspection possibilities?

We definitely can:

• the current version is already capable of detecting the cover
• the upcoming one will be able to identify decorative images and extract alt text

There's only one thing to keep in mind : we won't treat an empty alt tag as a clear indication that the image is decorative. We'll require either presentation or none as an ARIA role to do so, which could impact rather heavily how many images are identified as decorative.

[nicolascyr]

We definitely can:
the current version is already capable of detecting the cover
the upcoming one will be able to identify decorative images and extract alt text

Great news!

There's only one thing to keep in mind : we won't treat an empty alt tag as a clear indication that the image is decorative. We'll require either presentation or none as an ARIA role to do so, which could impact rather heavily how many images are identified as decorative.

That's fine.