From 1392d9b0bf996f5afa921d11523a2919a3080c88 Mon Sep 17 00:00:00 2001 From: Alexander Graul Date: Tue, 8 Apr 2025 11:45:16 +0200 Subject: [PATCH 1/4] use git notes --- accepted/0000-salt-new-maintaining-workflow.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index e20f5b9c..d5ce43e6 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -77,11 +77,12 @@ This is the place now where all those files will be maintained. #### Tracking upstream and downstream patches -This RFC proposes the usage of "obs_scm" service as the mechanism to pull the sources for the Salt package to build. This service will automatically produce a sources tarball according to a configured GitHub branch. +This RFC proposes the usage of "obs_scm" service as the mechanism to pull the sources for the Salt package to build. This service generates a sources tarball based on the content of a configured GitHub repository. -For the regular Salt maintenance, this means it won't be needed anymore to manually produce patch files to add them to the spec file, as the tarball now contains the updated sources (with the exception of EMBARGOED bugs, where patches are still needed as we cannot push any fix to public GitHub repositories). +Except for bugs under embargo, we won't use patches in the spec file. The generated tarball includes the latest changes we merged into our `openSUSE/release/` branch. +Bugs under embargo use a different workflow with manually created patches since we cannot push fixes for such bugs to our public Github repository. -To avoid losing the useful labeling of "PATCH-FIX_UPSTREAM" and "PATCH-FIX_OPENSUSE" (with a direct link to the origin PR on the spec file for each new patch we introduced into our Salt package), we will keep adding this information to the spec file on every new PR but this time without adding the patch itself, only the comment. +We will therefore not track bug fixes in the spec file anymore. The origin of a bug fix, i.e. whether it is backport from upstream or not, will be stored in a Git note attached to the commit of each bug fix. Git notes can be added after bug fixes are merged without modifying the Git history. #### Salt RPM changelogs From 5dc8da5e2be07bc14f1402ad255c0316d5567c4d Mon Sep 17 00:00:00 2001 From: Alexander Graul Date: Wed, 16 Apr 2025 13:10:34 +0200 Subject: [PATCH 2/4] Update section about RPM changelogs The previously shown bash script is only capable of adding new entries, we need something more sophisticated to change modify the latest entry as well (e.g. to add a bug report id) --- .../0000-salt-new-maintaining-workflow.md | 31 +++---------------- 1 file changed, 4 insertions(+), 27 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index d5ce43e6..425268a5 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -63,7 +63,7 @@ pkg/suse/html.tar.bz2 pkg/suse/salt-tmpfiles.d pkg/suse/transactional_update.conf pkg/suse/update-documentation.sh -pkg/suse/mkchlog.sh +pkg/suse/rpmchangelogs pkg/suse/_multibuild pkg/suse/salt.spec pkg/suse/changelogs/factory/salt.changes @@ -86,33 +86,10 @@ We will therefore not track bug fixes in the spec file anymore. The origin of a #### Salt RPM changelogs -As mentioned, the changelog files are now maintained in the `openSUSE/salt` GitHub repo, under `pkg/suse/changelogs/` directory. +As mentioned, the changelog files are now maintained in the `openSUSE/salt` GitHub repo, under `pkg/suse/changelogs/` directory. New changelog are part of pull requests to `openSUSE/salt`. +Our packaging artifacts will contain a `rpmchangelogs` Python script to easily `add`, `modify` and `remove` changelog entries for all changelogs at once. -Our packaging artifacts will contain a `mkchlog.sh`, which is a helper script to generate a changelog entry to all maintained changelog in one shot. Something like this: - -```bash -echo "Generating changelog entry for Salt package" -if ! osc vc _temp.changes; -then - exit 1; -fi - -echo "Update changelog files" -echo >> _temp.changes - -for i in $(ls changelogs/*/salt.changes); do - echo "$(cat _temp.changes $i)" > $i - git add $i -done - -rm _temp.changes -``` - -When creating a PR to `openSUSE/salt` the user must also include the corresponding changelog entry for all maintained changelog files. - -Similarly to the main Uyuni repository, we should add a GitHub action to warn the user in case no changelog entry is added in the PR. - -NOTE: I think it is better to decouple commit messages (focus on developers) from changelog entries (focus on users/customers), so I prefer to not use commit messages from "openSUSE/salt" to autogenerate the changelog entries but rather to manually write a meaningful changelog message to be included in your PR as part of your changes. Similarly to what we do in other Uyuni repositories. +We will use a Github status check to prevent us from merging pull requests without changelogs, like we already do in `uyuni-project/uyuni` ### OBS project structure From f080d717078f40f1f2154b60572e4fe1b206255c Mon Sep 17 00:00:00 2001 From: Alexander Graul Date: Wed, 21 May 2025 15:45:06 +0200 Subject: [PATCH 3/4] update salt extensions section --- .../0000-salt-new-maintaining-workflow.md | 40 ++++++++++++++++--- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/accepted/0000-salt-new-maintaining-workflow.md b/accepted/0000-salt-new-maintaining-workflow.md index 425268a5..0e9a68be 100644 --- a/accepted/0000-salt-new-maintaining-workflow.md +++ b/accepted/0000-salt-new-maintaining-workflow.md @@ -231,16 +231,46 @@ Feel free to open new PRs against `openSUSE/devel/master` to see this in action. ### Salt Extensions -#### Builtin extensions -The sources for the builtin Salt Extensions will be located together with the main Salt codebase at the `openSUSE/salt` GitHub repository. No new packages or subpackages will be created for these extensions as they will be part of the main `python3*-salt` package. +Salt extensions are collections of Salt modules that are maintained outside of the Salt repository. +They can be maintained by the upstream Salt Core team, by our openSUSE Salt maintainers or anyone +else. By convention, most Salt extensions are maintained in repositories in the +[salt-extensions](https://github.com/salt-extensions) Github organization. -If a fix is needed for any of the builtin extensions, workflow would be the same as for a code fix in the main Salt package. +#### Extensions Maintained by openSUSE + +- transactional\_update +- zypperpkg +- rebootmgr +- btrfs +- openscap +- libvirt_events +- virt +- snapper +- suse\_ip? + + +Each of these extensions are maintained in a separate Git repository, created with +[`salt-extension-migrate`](https://github.com/salt-extensions/salt-extension-migrate). This tool +keeps the Git log from the main Salt repository and is the standard tool to move Salt modules to new extensions. + +A) We maintain these extensions in the [salt-extensions](https://github.com/salt-extensions) Github +organization. That makes it easy to discover our Salt extensions and makes it easier for anyone to +contribute to our Salt extensions, just like any other existing Salt extension. The standard tooling +helps us with the configuration for publishing the documentation and the PyPI upload. + +B) We maintain these extensions in the [openSUSE](https://github.com/openSUSE) Github organization, +to make the ownership and responsibility clear. We manually configure PyPI upload and docs publishing. + +#### RPM Packages for Extensions + +Salt Extensions are like any other Python package. `py2pack` can be used to generate 80% of the spec +file for a Salt Extension. That makes it easy to add new Salt Extensions or drop old ones, e.g. when +they are moved back to the Salt core repository. #### Packaged Salt Extensions -For the Salt Extensions that are packaged separately from the main Salt package, we will create a separated GitHub repository where we will maintain these extensions. +For the Salt Extensions that are packaged separately from the main Salt package, we will create a separated GitHub repository for each of these extensions. -This "openSUSE/salt-extensions" repository will contain: - a common salt-extension spec file that will generate all RPM packages - The sources for each Salt Extension we package - The changelog files for each maintained codestream From 9fb4c25c6ff8a4a8ef01e3237b3f187bbf9cdfd0 Mon Sep 17 00:00:00 2001 From: Alexander Graul Date: Wed, 21 May 2025 15:44:28 +0200 Subject: [PATCH 4/4] WIP: alternative git workflow for Salt packaging --- accepted/0000-git-salt-packaging.md | 185 ++++++++++++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 accepted/0000-git-salt-packaging.md diff --git a/accepted/0000-git-salt-packaging.md b/accepted/0000-git-salt-packaging.md new file mode 100644 index 00000000..4545e087 --- /dev/null +++ b/accepted/0000-git-salt-packaging.md @@ -0,0 +1,185 @@ +- Feature Name: Git-based Salt Packaging +- Start Date: 2025-05-21 + +# Summary +[summary]: #summary + +Package Salt on src.opensuse.org, with one branch per code stream. + +# Motivation +[motivation]: #motivation + +We want to change our packaging process for Salt, starting with Salt 3008. This is the +first release we package after moving a lot of Salt modules to Salt Extensions. This +approach should meet the following goals: + +1. After merging a pull request at [openSUSE/salt](https://github.com/openSUSE/salt), a new package (RPM) is built automatically +2. Building Salt and Salt Extensions can be controlled from a single location (sources may be located elsewhere) +3. We're aligned with the new openSUSE Tumbleweed / SLE 16 workflow packaging +4. We're aligned with upstream's Salt Extension workflows for maintaining, documenting and publish + +Package sources for Tumbleweed and SLE 16 will be tracked in git repositories. +[OBS](https://openbuilservice.org) uses the sources from central git forges located at +https://src.opensuse.org and https://src.suse.de respectively. + + +## Requirements + +We maintain Salt RPMs in different code streams from the same sources. The code streams' +changelogs differ and release timings can be + +- Why are we doing this? +- What use cases does it support? +- What is the expected outcome? + +Describe the problem you are trying to solve, and its constraints, without coupling them too closely to the solution you have in mind. If this RFC is not accepted, the motivation can be used to develop alternative solutions. + +# Detailed design +[design]: #detailed-design + +## `openSUSE/Salt` on Github + + Build metadata (salt.spec, salt.changes, \_multibuild, …) is moved to a subdirectory in +`openSUSE/salt` on GitHub. This allow us to include packaging updates at the time we +create pull requests, e.g. we can include an appropriate changelog together with the +changes. + +Files moved to Salt repository: +- pkg/suse/README.SUSE +- pkg/suse/html.tar.bz2 ??? +- pkg/suse/salt-tmpfiles.d +- pkg/suse/transactional_update.conf +- pkg/suse/update-documentation.sh +- pkg/suse/rpmchangelogs +- pkg/suse/_multibuild +- pkg/suse/salt.spec +- pkg/suse/changelogs/factory.changes +- pkg/suse/changelogs/sles15sp2.changes +- pkg/suse/changelogs/sles15sp3.changes +- pkg/suse/changelogs/sles15sp4.changes +- pkg/suse/changelogs/sles15sp5.changes +- pkg/suse/changelogs/sles15sp6.changes +- pkg/suse/changelogs/sles15sp7.changes + +### RPM Changelogs + +New changelog entries should be part of pull requests. It easy for the code author to +write the user-facing changelog entry while she has all the required context available. + +Our changelogs differ between code streams. Most differences are due to different grouping +and entry dates, since we generally keep the package contents in sync. + +To help adding new changelog entries in pull requests and update them during rebases, we +add a new Python script `rpmchangelogs`. This script wraps `osc vc` to modify all +`*.changes` files at once. It can `add`, `modify`, and `remove` the latest changelog entry +in all changelogs when the entries are the same. + +We use a Github status check to prevent accidental pull requests merges without changelog entries. + +## "Package-Git" repository on src.{suse.de,opensuse.org} + +Package git: One repository with different branches (see below). The repository contains +openSUSE/salt as a Git submodule. + +Branches: +- `factory` (devel for Tumbleweed) +- `products` (why?) +- `testing` (devel for Manager / Uyuni) +- `sles15sp5` (code stream, src.suse.de) +- `sles15sp6` (code stream, src.suse.de) +- `sles15sp7` (code stream, src.suse.de) +- `next` (why?) + +### Packaging sources in salt repo + +``` text +.gitattributes # created with obs-git-init +.gitignore # created with obs-git-init +.gitmodules # contains Git submodule status +salt # Git submodule +README.SUSE # extracted from `salt` Git submodule +_multibuild # extracted from `salt` Git submodule +html.tar.bz2 # extracted from `salt` Git submodule +salt-tmpfiles.d # extracted from `salt` Git submodule +salt.spec # extracted from `salt` Git submodule +salt.changes # extracted from `salt` Git submodule,for the given branch +transactional_update.conf # extracted from `salt` Git submodule +update-documentation.sh # extracted from `salt` Git submodule +``` + +## "Project-Git" repository on src.{suse.de,opensuse.org} + +We use a single "Project-Git" repository, again with one branch per code stream. Packages +in this project are included as Git submodules, checked out at the corresponding branch. + +Per convention, the project git repository is located in the "salt" organisation and +called `_ObsPrj`. An example organisation with the same layout (except that it uses a +singular `master` branch in `_ObsPrj`) is [lua](https://src.opensuse.org/lua) + +### Packages + +``` text +salt +salt-ext-zypper +salt-ext-transactional_update +salt-ext-rebootmgr +... +``` + +## Build Service project on build.{suse.de,opensuse.org} + +Build Service projects configure build repositories via it's `meta`. The rest (including +`prjconf`) is maintained in the "Project-Git". + +## Update End-to-end Workflow + +When we merge a PR to a release/ branch in openSUSE/salt, a jenkins job updates the Package-Git repository. + +``` text +1. update git submodule +2. extract files (salt.spec, \_multibuild, …) +3. rename to salt.changes +4. commit +5. push +``` + +This is implemented with a Makefile, the Jenkins job just calls `make` with +required variables set. The same Makefile also defines targets to similarly update Salt +Extensions. + +[`workflow-direct`](https://src.opensuse.org/adamm/autogits) keeps the Project-Git +up-to-date with changes to the Package-Git repositories. + +## Salt Extensions + +Salt Extensions are packaged individually. Each salt extension is a typical +Python RPM, built with the standard `python-rpm-macros`, tracked in +"Package-Git" repositories. These package git repositories are included next to +Salt in the "Project-Git". + +Packaging sources are different from Salt since we do not control the upstream +repositories. Specfile and changelog are not stored in the extension source repos, instead +we keep them directly in the Package-Git. Since these are new packages, we don't have +diverging changelogs and can use a single branch. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we **not** do this? + + * obscure corner cases + * will it impact performance? + * what other parts of the product will be affected? + * will the solution be hard to maintain in the future? + +# Alternatives +[alternatives]: #alternatives + +- What other designs/options have been considered? +- What is the impact of not doing this? + +# Unresolved questions +[unresolved]: #unresolved-questions + +- What are the unknowns? +- What can happen if Murphy's law holds true?