bug: path conflicts with two single globs

[lczyk]

here is a short example based on libbinutils from 26.04:

package A:
/foo/bar/libopcodes-2.45.50-system.20251212.so
package B:
/foo/bar/libopcodes-2.45.50-arm64.so

we would like to glob the version (2.45.50) and the date string (20251212). the following globs cause conflict:

package A:
/foo/bar/libopcodes-*-system.*.so
package B:
/foo/bar/libopcodes-*-arm64.so

error: slices <slice-in-package-A> and <slice-in-package-B> conflict on /foo/bar/libopcodes-*-amd64.so and /foo/bar/libopcodes-*-system.*.so

when translating the globs to regex we get libopcodes[^\/]*-system\.[^\/]*\.so which matches only the intended expression: https://regex101.com/r/pYWots/1

• the libbinutils issue in chisel_releases: feat(26.04): change binutils paths to devel pattern chisel-releases#776

[lczyk]

ok. with a little bit of thought put into it, this is alas not a bug but a feature. here is a patter of substitutions of the globs in the above example which lead to the two globs being identical:

/foo/bar/libopcodes-*-system.*.so ( '*' -> '-arm64' )
/foo/bar/libopcodes-*-arm64.so

/foo/bar/libopcodes-*-system.-arm64.so
/foo/bar/libopcodes-*-arm64.so ( '*' -> 'baz-system' )

/foo/bar/libopcodes-*-system.-arm64.so ( '*' -> 'baz' )
/foo/bar/libopcodes-baz-system.-arm64.so

/foo/bar/libopcodes-baz-system.-arm64.so
/foo/bar/libopcodes-baz-system.-arm64.so

[lczyk]

one way this could be resolved is by allowing custom globs, for example something like this:

package: mypkg1
slices:
  myslice:
    contents:
      /file/f🎄obar:
        special-globs:
          🎄: '[^o]'

where the key is a singe character which replaced with the regex in the value.

here is a quick implementation, courtesy of Calude Sonet, just to see what code changes would need to happen for something like this: https://github.com/canonical/chisel/pull/260/files#diff-a055819a3b201a409d3d680cb93178b6562bb193a7571a66d1bc1943e45c9645R513-R528.

( obviously i'm not proposing to use a christmas tree! )

[upils]

Thanks for the investigation @lczyk!

The initially described behavior matches the test case in https://github.com/canonical/chisel/blob/main/internal/strdist/strdist_test.go#L60 (even if a single wildcard is used in both paths in the test). When validating the release, the archive is not used, so as you noticed the two paths could try to extract the same file.

Your suggestion seems very powerful but also very risky, because we would need to think of every possible corner cases with all the possible regex combinations. As demonstrated above, even basic wildcards can be tricky to reason about, so expanding that to the whole regex "grammar" seems to be opening the Pandora box, even if the initial idea is to be more precise.

A safer path forward may be to think about supporting another wildcard.

[lczyk]

Your suggestion seems very powerful but also very risky...

Yep. This was just an idea to start thinking from / see what it would take to implement something like it. I'm also very lukewarm to the idea of just dropping an arbitrary regex in as a glob. Maybe some way of specifying custom globs of the form "anything except for ..."? something like:

package: mypkg1
slices:
  myslice:
    contents:
      /file/f🎄obar:
        special-globs:
          🎄:
            except-for: [o, p, q]
          🎅:
            except-for: [/]
          🍪:
            except-for: [/, .]

in the example above 🎅 is equivalent to normal *. i'm imagining this would be simpler to implement using the current logic for * since it would only need generalization to 1) symbols other than the path separator / and 2) more than one symbol.

then, the original issue could be resolved by:

package A:
/foo/bar/libopcodes-🍪-system.🍪.so
package B:
/foo/bar/libopcodes-*-arm64.so

aka adding . as an additional separator, therefore separating the 🍪 globs.

( the proposed names of the fields are completely ad-hoc. i would re-think what to call them in any actual implementation )

[upils]

This was just an idea

No worries with that! This was perfectly clear to me :).

Just to make sure I understand, would the issue not be resolved with:

package A:
/foo/bar/libopcodes-*-system.🍪.so
package B:
/foo/bar/libopcodes-*-arm64.so

instead? Or maybe both would work? Or maybe I misunderstood your proposition?

How do you know for sure that both paths would not still match the same file? If the real file path were /foo/bar/libopcodes-baz-system-arm64.so, both regexes would match, right? So I suspect this approach is influenced by some information chisel cannot rely on.

More generally: It feels like any solution that would rely on defining very specific rules at the contents level would be hard to scale. My gut feeling is that if we come up with another wildcard to match a specific set of characters, we will likely want to make it usable everywhere.

[lczyk]

Just to make sure I understand, would the issue not be resolved with:

... i think so, but i have not reasoned though the exact interaction of * being bound only on /, but 🍪 being bound on / and .`:

                         🍪 range
                             |
                             v                    
/foo/bar/libopcodes-*-system.🍪.so
         ^                       ^
         |_______ * range _______|

note that, in the case above, 🍪 also disallows / so cannot interrupt *'s range (it woudl be much harder to reason about if it could)

so i've opted out to making a bit of a stringer statement where i know it woudl work:

                        🍪2 range
                             |
                             v                    
/foo/bar/libopcodes-*-system.🍪.so
         ^                 ^
         |___ 🍪1 range ___|

---

How do you know for sure that both paths would not still match the same file?

i woudl do the same kind of reasoning which chisel is doing already: "can these two globs ever match?", but now in a slightly more general way which would allow for some globs to be made more restrictive than they already are. in the example above, /foo/bar/libopcodes-*-system.🍪.so only constrains the filename more than /foo/bar/libopcodes-*-system.*.so would.

My gut feeling is that if we come up with another wildcard to match a specific set of characters, we will likely want to make it usable everywhere.

hmmm... it feels like an overly specific solution. like, sure, in this case lets say we add 🍪 to chisel as a "builtin". it would solve this problem, but then we find that someone packaged /foo/bar/libopcodes-XXXX-system-YYYY-final.so and we're back to square 1. we could then define another glob for - but then /foo/bar/libopcodes-XXXX-system(YYYY).so comes along and () are an issue etc etc.

defining a custom glob with extra restrictions (the except-for) would solve all of the above.

if we wanted to be more restrictive we could say that we every glob defined in special-globs must be a strict subset of * (aka 🎅), and so we never have to deal with a hard case mentioned above (with 🍪 potentially containing /). i guess then we would also have to only allow only one special glob, else someone could create this kind of situation still.

[letFunny]

As demonstrated above, even basic wildcards can be tricky to reason about, so expanding that to the whole regex "grammar" seems to be opening the Pandora box, even if the initial idea is to be more precise.

In my opinion this is exactly the reason why we want to restrict the grammar. If we had arbitrary regexp (or something similarly expressive), it will be very hard for contributers to reason about two paths conflicting and how to solve the conflict while respecting the intent of both expressions. Today, it can be already hard to reason about * but it is still somewhat simple to guess why the glob was used, e.g. /foo/bar/libopcodes-*-system.*.so clearly tells you the parts of the file name that are considered irrelevant.

hmmm... it feels like an overly specific solution

About the solution being specific or not, that is the crux of the issue. We should not engineer a change in the grammar that will only ever be used a handful of occasions as that will require the users to learn about a more confusing and obscure UX. If the lack of expressiveness becomes a pervasive problem in the release then we can think carefully about the design and try to expand the grammar for it to be useful for the majority of cases without it being overly complex. If these happens for only a handful of cases I am more inclined to avoid changing the grammar for now and instead opt to try to work around it for the time being, that is until we gather more data or it becomes a burden to maintain it from the point of view of chisel-releases.

Lastly, I do like your proposal about having support for more regex globs like [^c]* though I think it should be done inline in the path without the extra substitution. Maybe even simpler would be allowing only character plus glob, so in your example the date could be expressed as \d* and the hyphens act as delimiters naturally.