Config: allow customization of project pre_checkout and checkout step

[agjohnson]

What would we think of allowing for pre_checkout and/or checkout build job customization at the Project model level? Obviously it doesn't work to include a pre_checkout/checkout configuration item as we need to clone the repo in the first place, so this would be a model field instead.

A more general approach would be a project model field for a base configuration file, where configuration file rendering is merging the model field base configuration and repository configuration file? I believe some users have asked for this more directly.

The intent here is to accommodate projects that have very custom needs during cloning, like having long clone times and custom git clone commands to speed up checkout. There might be some commands that are useful to run pre_checkout as well.

But it's not important that this is a user facing field, much like build time limit. It's also not important that this be an extension of our config file concept necessarily. It could simply be a field for the custom command used to check out a project's repository. There might be value in a generic approach though.

Front conversations

[humitos]

I wouldn't complicate things too much here and just allow customers to customize only the "clone step" by support request. This can be done by adding a Project.clone_command and checking for this field at build time. I don't think we need more than that for now.

What would we think of allowing for pre_checkout and/or checkout build job customization at the Project model level? Obviously it doesn't work to include a pre_checkout/checkout configuration item as we need to clone the repo in the first place, so this would be a model field instead.

Eventually, we can allow pre_checkout/checkout command inside the YAML file by using the VCS API to download the .readthedocs.yaml file as the first step and run the normal build process.

[agjohnson]

This can be done by adding a Project.clone_command and checking for this field at build time. I don't think we need more than that for now.

Yeah that was the option I was noting above too.

we can allow pre_checkout/checkout command inside the YAML file by using the VCS API to download

Ah good point, I thought about the downloading the config file option too, but with a normal request that would only easily work for settings files that were publicly available. I didn't think about using the VCS API to grab the file, that's an interesting approach.

We could support per-version downloading of the config file too, and don't have to have the weird UX where project level setting only come from main branch.

[stsewd]

What command would they want to execute instead of the default clone? Our build process does rely on a couple of things (syncing branches, creating directories, having the repo cloned into a specific directory, SSH agent).

[agjohnson]

The linked conversation has an example of the git clone being used for a monorepo configuration.

[humitos]

I will give it a try to this with a POC in the next sprint and see if this is useful.

[humitos]

I created a POC for this at #12412 and I was able to build one of the projects from https://github.com/ROCm/rocm-libraries. The whole "checkout step" takes around ~90 seconds with my home connection, which I suppose it's acceptable.

Besides the custom Git checkout command, the project also needs to define some relative paths like a custom readthedocs_yaml_path, the config python.install.requirements and maybe others.

[humitos]

The whole "checkout step" takes around ~90 seconds with my home connection, which I suppose it's acceptable.

I ran these same commands from a build-default instance and it took 1m12s

real	1m12.535s
user	0m3.884s
sys	0m1.171s