Add initial attempt to keep working dir #12327
Conversation
For Enterprise users with dedicated builders, keeping the checkout around can make their build much faster, especially in the case of a monorepo. This is just a draft PR to start testing this concept.
|
If we need to go in this direction, we should recover the work we've done already. It caches the whole environment and upload it to S3, so other builders can re-use it: #6763 |
|
@humitos the issue is the download speed of the entire repo on disk, whether from GitHub or S3. The goal is to just keep it checked out on the server to avoid any download penalty for multi-GB repos. |
|
I see. Keeping the output directory in the builder will only help when the project runs another build in the same builder, tho. That's very unlikely unless we do something to force it. It's hard to make it with spot instances... |
|
@humitos as noted in the description, this is for enterprise folks with dedicated builders. |
| # Check for existing checkout and skip clone if it exists. | ||
| from readthedocs.projects.models import Feature | ||
|
|
||
| if self.project.has_feature(Feature.DONT_CLEAN_BUILD) and os.path.exists( |
There was a problem hiding this comment.
This looks like plenty to start with. We might eventually use something more specific to test the validity of the repository too, in case the clone is stuck in an unusable state. Something like rev-parse and supporting CLI arguments could probably fit here.
…ep-checkout-enterprise
|
Going to get this in to try and get in the next deploy so we can test it. |
``` In [3]: dict(Feature.FEATURES) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[3], line 1 ----> 1 dict(Feature.FEATURES) ValueError: dictionary update sequence element #2 has length 4; 2 is required ``` The object got malformed in #12327
``` In [3]: dict(Feature.FEATURES) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[3], line 1 ----> 1 dict(Feature.FEATURES) ValueError: dictionary update sequence element #2 has length 4; 2 is required ``` The object got malformed in #12327
``` In [3]: dict(Feature.FEATURES) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[3], line 1 ----> 1 dict(Feature.FEATURES) ValueError: dictionary update sequence element #2 has length 4; 2 is required ``` The object got malformed in #12327
``` In [3]: dict(Feature.FEATURES) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[3], line 1 ----> 1 dict(Feature.FEATURES) ValueError: dictionary update sequence element #2 has length 4; 2 is required ``` The object got malformed in #12327
For Enterprise users with dedicated builders,
keeping the checkout around can make their build much faster,
especially in the case of a monorepo.
This is just a draft PR to start testing this concept.
Locally: