Rv2 - generic resource tree + addons discussion issue

[trws]

We've had a few discussions about this, and after noodling on it for a while I thought it would help to get us talking to put together something we can poke at. Here's the example from the Rv1 RFC, with the hostname changed to disambiguate for later:

{
  "version": 1,
  "execution": {
    "R_lite": [
      {
        "rank": "19-22",
        "children": {
          "core": "0-47",
          "gpu": "0-7"
        }
      }
    ],
    "nodelist": [
      "fluke[186-189]"
    ],
    "starttime": 1676560542,
    "expiration": 1676562342
  }
}

I'm going to walk through a thought process here, so forgive me if this wanders a bit... My goals for the updated format:

1. Still reasonably easy for a human to write/read
2. Similar idset-based structure
3. Shared implementation usable by core and sched
4. Non-tree details can be ignored by core
5. The containment hierarchy, our main tree that contains all resource vertices in fluxion, is specified in a way core can read if it cares to
6. If possible:
   1. incremental parsing, because big JGFs are a problem because we can't there right now
   2. Ability to encode everything fluxion needs

Given that= I'm thinking we make the actual resource tree structure nested like we do for the resource request in jobspec, where there are requirements on the "node" typed entries that match the current requirements. Taking just what is currently the "R_lite" key, probably "R" now or "R_2%-skim" or something, I'm thinking something like this:

{
  "type": "node",
  "ids": "19-22",
  "children": [ // note array now
    {
      "type": "core",
      "ids": "0-47",
    },
    {
      "type": "gpu",
      "ids": "0-7",
    }
  ]
}

The main difference here, is that children is now an array of objects that describe arbitrary resource types and the "ranks" key is missing. Thought process is that the hardware of the machine is probably more or less stable, and the tree structure makes it easy to express "local" IDs at a level where ranks are global.

Here's another one with some more levels just to see what it does:

{
  "type": "cluster",
  "ids": "0",
  "children": [
    {
      "type": "rack",
      "ids": "0-5",
      "children": [
        {
          "type": "ssd",
          "ids": "0-15"
        },
        {
          "type": "node",
          "ids": "0-14",
          "children": [ // note array now
            {
              "type": "socket",
              "ids": "0",
              "children": [
              {
                "type": "core",
                "ids": "24-47",
                "ids_under": "node"
              },
              {
                "type": "gpu",
                "ids": "4-7",
                "ids_under": "node"
              }
             ]
            }
            {
              "type": "socket",
              "ids": "1",
              "children": [
              {
                "type": "core",
                "ids": "0-23",
                "ids_under": "node"
              },
              {
                "type": "gpu",
                "ids": "0-3",
                "ids_under": "node"
              }
             ]
            }
          ]
        }
      ]
    }
  ]
}

Now we have a system with 6 racks, each of which has 16 ssds and 15 nodes, each of which has two sockets with numbered sets of cores and GPUs. I'm still scratching my head on this a bit, the ID situation makes it easy to say which cores we want to allow and which we don't, but when it's used in this hierarchical way to expand out into a complete system it seems to run into problems. I threw in an "ids_under" key here that goes in a specific resource to say that the IDs are unique or applicable under a specific resource type which must exist above that level. Here using that for cores and GPUs, referring to the node through the socket. Could maybe also use that to have a global rank mapping, in here but I think that would get confusing fast. Ranks then would be applied like the nodelist is now, based on a mapping to generated resources in order. Now that I have this written out, chewing on if there's another way to break this up, but this is pretty close to how our resource key works in jobspec, in fact if it were "with" and "count"... yeah. Maybe worth making them similar.

To finish out the thought on the rest of it, I'm thinking we could make the full thing a json stream format, multiple consecutive objects rather than one, a canonical version of that is jsonl (just one obj per line). Slightly more annoying to parse, but much more efficient to produce and consume if it has to be large, and means we could easily tolerate having it broken up into files. So, something like this might be a complete one including edges for the fluxion graph:

{
  "version": 2,
  "type": "R",
  "resources": {
    "type": "node",
    "ids": "19-22",
    "children": [ // note array now
      {
        "type": "core",
        "ids": "0-47"
      },
      {
        "type": "gpu",
        "ids": "0-7"
      }
    ]
  }
}
{"type": "edge", "labels":"storage", "from": "node19", "to": "node20"}
... // other extra data

[grondo]

I like the direction this is going!

I threw in an "ids_under" key here that goes in a specific resource to say that the IDs are unique or applicable under a specific resource type which must exist above that level. Here using that for cores and GPUs, referring to the node through the socket.

Could that not be implied by the fact that the core and gpu types are children of node?

If there were multiple racks with ssds under cluster, would they need "ids_under" as well? I'm probably just missing the purpose of that field.

I'm just going to throw out a couple notes while I thought of them, just as extra information to chew on:

First, system instances are currently configured via a resource.config array of tables, e.g.

[[resource.config]]
hosts = "test[1-100]"
cores = "0-7"

[[resource.config]]
hosts = "test[1,2]"
gpus = "0-1"

As much as possible, we should make resource configuration just as simple for an Rv2 based system. Obviously if we're adding extra hierarchy it can't be just as simple, but just something to keep in mind.

Second, @sam-maloney brought up elsewhere the point that embedding slot information in R for jobs is almost necessary when the job request is moldable by specifying resource ranges instead of counts in the job request (since the scheduler effectively chooses the slot shape). Maybe this could be another line in the jsonl format for Rv2, or there is a clever way to embed it in the R object itself. Either way, it should be optional since a slot shape doesn't make sense for a cluster config.
(unless the "slot" for a Flux cluster/job is always "whole node"?)

[sam-maloney]

Oooh yes, I would love to see slot information directly encoded in the next Resource Set version! As @grondo already mentioned, this is extremely helpful for non-integer counts, where otherwise the core may have to 'guess' at the slot shapes that the scheduler chose. Additionally, I think it will also be important for canonical jobspec where multiple labeled slots can be specified, so having slot vertices directly in the resource tree would clearly encode which resources have been allocated to which slots.

[trws]

I threw in an "ids_under" key ...

Could that not be implied by the fact that the core and gpu types are children of node?

It could, but where things get strange is with that intervening "slot" vertex. The general rule I was thinking of was that the parent would be the base where IDs need to be unique unless otherwise specified. We could say that node is special and takes over everything under it, but it's an exception. Definitely doable, but worth calling out.

If there were multiple racks with ssds under cluster, would they need "ids_under" as well? I'm probably just missing the purpose of that field.
The racks are based under cluster, the SSDs under the rack. The reason the cores need it is that we want their identifiers to be consistent, and unique, across the whole node regardless of intervening vertices. Otherwise it would be something like two sets of core[0-23] under each of two sockets, which I think would make it hard to reason about.

I'm just going to throw out a couple notes while I thought of them, just as extra information to chew on:

First, system instances are currently configured via a resource.config array of tables, e.g.

[[resource.config]]
hosts = "test[1-100]"
cores = "0-7"

[[resource.config]]
hosts = "test[1,2]"
gpus = "0-1"
As much as possible, we should make resource configuration just as simple for an Rv2 based system. Obviously if we're adding extra hierarchy it can't be just as simple, but just something to keep in mind.

Good point. This would be expressible in toml, but yeah it wouldn't be that easy if you want the extra information in there.

Second, @sam-maloney brought up elsewhere the point that embedding slot information in R for jobs is almost necessary when the job request is moldable by specifying resource ranges instead of counts in the job request (since the scheduler effectively chooses the slot shape). Maybe this could be another line in the jsonl format for Rv2, or there is a clever way to embed it in the R object itself. Either way, it should be optional since a slot shape doesn't make sense for a cluster config. (unless the "slot" for a Flux cluster/job is always "whole node"?)

We could do either I think. The original concept for the slot was embedded in the resource DAG specification, so we could emit a type=slot level that contains the resources probably. I'll have to think about how exactly we would do that since it doesn't quite work to have it go through this kind of parent/child expansion, might need to rethink that just a bit to make it work. An example that would be especially valuable though would be a rabbit reservation in practice, doing a single slot so I don't have to come up with a multi-slot mapping solution on the fly:

{
  "version": 2,
  "type": "R",
  "resources": {
    "type": "slot",
    "ids": "0",
    "children": [
      {// this would be an exclusive node, need to figure out how we manage exclusivity I guess
        "type": "node",
        "ids": "19",
        "children": [
          {
            "type": "core",
            "ids": "0-23",
            "ids_under": "node"
          },
          {
            "type": "gpu",
            "ids": "0-3",
            "ids_under": "node"
          }
        ]
      },
      { // need globally unique identifier that's _stable_
        // going with resource path for now, maybe path/idset?
        "type": "ssd",
        "ids": "rack0/ssd1"
      }
    ]
  }
}

In writing this out, a couple extra things pop up as things to solve:

1. We need a way to have expansion or no expansion, or a way to map things together. This is so that we can deal with slots in a concrete spec. Otherwise one with two slots that lists ids=5-10 would imply we've allocated 5-10 twice, once to each slot, which doesn't make sense.
2. Exclusivity or resource splitting becomes an issue. I was thinking about this earlier this morning and considering if maybe we want to just make all resources pools for the sake of simplicity, and have them default to either a pool of 1 or something big like 1000 pieces (Kube does this for cores for example) so that we can just make everything exclusive and instead of worrying about that concept if it's the whole thing we allocate all 1000 units, and if it's not we allocate less. I think it makes implementation easier, not sure about teachability.
3. We need to work out the unique identifier issue. To be clear, I do not mean in the sense of fluxion/resource spec UUIDs, we may use those internally but they have no place here. I mean something that will let us declaratively reference every object in the tree from the extra properties or from other places in the spec. I'd mentioned using paths for this before, because it's something we're used to using in fluxion that works well (though it's a bit annoying to query right now). If we have some fun with combining paths and hostlists/idsets it might actually be somewhat ergonomic, say we allow cluster0/rack[0-1]/node[0-15] to refer to all 16 nodes in each of racks 0 and 1? I'm really not sure if I like that but it's possible. We could also require that each entity in the config has a globally unique ID added onto it, though it could be auto-generated by specifying a traversal order that generates them I guess... That would be a bit like the current nodelist connection, allow "core6751" to mean a core in a specific node, but it would be meaningless to a user. This one needs some thinking.

[milroy]

Based on the in-person meeting earlier in the week, I can address some of the discussion items by combining our ideas from that meeting.

1. We need a way to have expansion or no expansion, or a way to map things together.

@trws, @zekemorton, and I considered an approach that would aggregate children with identical subgraph structure but different ids or ranges of ids. The idea is to define the value corresponding to the "ids": key to be an array of idsets. Amplifying on the example above, we would have:

{
  "version": 2,
  "type": "R",
  "resources": {
    "type": "slot",
    "ids": ["0-1", "4", "9-10"],
    "children": [
      {
        "type": "node",
        "ids":  ["19-20", "24", "39-40"],
        "children": [
          {
            "type": "core",
            "ids": ["0-23", "24-47", "0-23"],
          },
          {
            "type": "gpu",
            "ids": ["0-3", "4-7", "0-3"],
          }
        ]
      }
    ]
  }
}

Where the length of "ids": arrays must be identical to ensure a correct expansion. @zekemorton further suggested templated subgraphs such as node, and referencing them rather than explicitly specifying them. That approach might make human readability difficult, but there could be ways to mitigate that difficulty.

2. [...] I was thinking about this earlier this morning and considering if maybe we want to just make all resources pools for the sake of simplicity [...]

That makes sense to me!

Second, @sam-maloney brought up elsewhere the point that embedding slot information in R for jobs is almost necessary when the job request is moldable by specifying resource ranges instead of counts in the job request (since the scheduler effectively chooses the slot shape).

I'll take this opportunity to mention a few ideas about malleable, moldable, evolving, and elastic jobs and resources by cherry picking some notions from this brainstorm: flux-framework/flux-core#6493 (comment).

1. For initial simplicity, we might want to consider the slot to be immutable and allow ranges of counts. We could support Flux-directed resource changes (for malleable and moldable jobs) by allowing mutations in integer multiples of a slot.
   • To support malleability and moldability, we need a notion of a "core" ("stencil", "hull" or some other term) to designate the minimum resource subgraph that Flux should not remove. That could be a key that takes an array of idsets as a value.
2. Could we support a "time" key whose value is a time interval when the subgraph is valid? For initial simplicity we could restrict the support to "slot".

Let me know what you think!