Updated files for creating CESM cases under git(hub) control

[kdraeder]

The previous instructions applied to putting existing cases under git(hub) control.
These additions describe how to create a controlled CESM case in a way that keeps cases separate for cleanly working with multiple cases simultaneously.

[hkershaw-brown]

Hi @kdraeder
what level of review are you looking for here?
Do you want this readme to be used by other people (dart team, general cesm-dart users)?
Is this readme notes for your own future use?

Cheers,
Helen

[kdraeder]

The kind of review depends on who you think the right audience is.
I'll definitely refer to it for my own work.
It will be useful to the team if or when they need to deal with what I leave behind.
I wrote it with general cesm-dart users in mind, who don't already have a way to archive their experiment setups. If it looks useful for that, there should be a link to it somewhere, so that they can find it.

[hkershaw-brown]

Hi Kevin,

I put some notes on things I did not follow. In general I'd say it reads like record of what you have done to keep track of cases. I'm not sure if

I think in general for anyone putting their case directory under version control for a particular experiment, they would do the following, rather than

cd case_directory
git init
git add .
git commit
git push to their own github repo

I think if other people are going to use (commit to) DART_CASEs then you'll need some way to enforce unique names for the branches, or protect the branches.

[hkershaw-brown]

the GitHub limit is everything (all branches) in the repo.

[hkershaw-brown]

why would you delete an existing branch?
What if someone else made the case?

[hkershaw-brown]

this is for <your_github> rather than NCAR/DART_CASES

[hkershaw-brown]

Are you taking about push -u here?
You can push any branch to the origin.

[hkershaw-brown]

this is shell_scripts in
DART/modesl/cam-fv/shell_scripts ?

[hkershaw-brown]

why not add everything in the case?

[kdraeder]

Thanks for pointing out git init. I don't remember seeing that before.
This may turn into git tutorial for me . . .

One question that may determine the answers to a lot of others is whether there should be an archive
of experiments that are of interest to multiple people in DAReS (and ...?) and will be maintained independently of who's in DAReS.
If so, then I think it's helpful to have a standardized workflow spelled out.
If not, then we'll each archive the way we like and other people can figure it out later, if needed.

You started a thought with "I'm not sure if", which I'm guessing ends with something like "anyone else will want to do it this way". Entirely possible, and helpful to know, if you think that's the case.

So far it looks to me like git init would replace just my git checkout -b $CASE,
but without naming the branch. So the rest of the steps might still be useful.
If a bunch of experiments will be pushed to the same repository, I feel like they should be kept separate.
It's not clear how separate is enough.
Would it be good to have each in a separate directory but all be parts of the main branch?
My impression of that is that each new branch from main would then have all existing experiments in it.
I guess init would avoid that by making an empty branch, which can be named later. ?

Good point about enforcing the branch names. I'll look into that if the strategy I've proposed survives the review.

If this seems like it could be useful to anyone but me, I'll clarify the confusing parts.
Otherwise, I'll mostly leave it as is and we can wrap this up.

Thanks for working through it.

[hkershaw-brown]

I think my brain was farting when I made the comment, there are two sentences that just tail off.

The init would be for a new repo for each case. So ignore that if you are using a branch for each case, you have it correct for the branch for each case.

I think the thing I would be worried about would be multiple people using (or deleting the same branch). So long as people are aware of that when they make use of this CASES repo.

[kdraeder]

To add a bit more to my tutorial; does 'a new repo for each case' mean only a local repo,
which can have the same 'origin' as other local repos, or does it also mean a new repo on github?

I'll work on the protection for branches, and answer your other questions.

[hkershaw-brown]

new repo for each case, github and local. But ignore that for your method which is this one repo, with a branch for each case

[kdraeder]

I think if other people are going to use (commit to) DART_CASEs then you'll need some way to enforce unique names for the branches, or protect the branches.

I dug into the old style branch protection and the newer "rule sets" and eventually found what I think I want
in the branch protection (but not in rule sets):

Optionally, enable branch restrictions.
         Select Restrict who can push to matching branches.
         Optionally, to also restrict the creation of matching branches

But, of course, that option is not offered in the form that defines a new branch rule.
I sent a query about it through the github complaints button at the bottom of the branch protection page.

[kdraeder]

I've looked into the protection of branches some more, run some tests, and need someone to test some things
before I proceed. Here's the setup.

I cloned a repository onto /glade:
git clone -n git@github.com:kdraeder/Test_CASES.git /glade/work/raeder/Exp/Git_perms2_git

I put some files in what would be the CESM CASE directory; .../Git_perms2/example_file.

I (re)initialized it to impose the permissions I want to protect this "CESM assimilation CASE"
setenv GIT_DIR /glade/work/raeder/Exp/Git_perms2_git/.git
git init -b Git_perms2 --shared=0640
[It ignores this branch that I suggested that it use, maybe because is doesn't exist yet.]
This results in .git/{HEAD,index,maybe refs} getting the permissions I imposed,
but the Git_perms2 directory and none of its contents get those permissions.
Maybe because I imposed the permissions (by git init) after the directory and files were created.

I checked out the new branch, named the same as the CASE
git checkout -b Git_perms2
Then I added directory Git_perms2 to git, and committed.

So now I'm wondering; do the permission restrictions on HEAD and index prevent other people
from changing this branch locally? So I'd like someone to change
/glade/work/raeder/Exp/Git_perms2_git/Git_perms2/example_file and try to commit it.
I think that I've changed the permissions on the path to allow group members to edit that file.
Thanks

[hkershaw-brown]

Hi Kevin

I'm not sure what your goal is here.
Branch projections on github are for preventing un-authorized pushes to github.
File permissions are what you want to protect files on Derecho. The permissions are user, group, others unless you use ACLs.

You can't prevent someone cloning this repo and making changes in their local clone.

[kdraeder]

This is partly about understanding how git(hub) actually works. I've read a lot of confusing documentation and comments about it, so I'm figuring out relationships between local file permissions, local git permissions (init --shared), and github branch protections. So far it seems that local permissions have nothing to do with github branch protections and can't be propagated to there, but branch protections are not my focus at the moment.

It's partly about my wanting to set up 2 kinds of assimilations; cases where I don't want anyone else to be able to do work in the local directories or push to github, and cases where I do. An example of the latter is the Reanalysis, which should at least be easy for someone else in the group to continue if I weren't able to, and ideally would be run by more than 1 person, so that no one is responsible for it 100% of the time. During the Reanalysis we decided that changes to the run configuration should be approved by someone and archived on github. One or more people need permission to do each stage of this; change files, add and commit them to git, push them to github, approve them, and run the job.

The immediate goal here is that I've created a case in which I think that I've given permission to group members to change the files, but not commit them to git. I need someone else in the group to test that.
It should only take a couple minutes, and would help me finish this issue of developing a system to archive work.

[hkershaw-brown]

I cannot read any of your work directory:

ls -l /glade/work/raeder/Exp
ls: cannot access '/glade/work/raeder/Exp': Permission denied
hkershaw@derecho6:~$ ls -l /glade/work/raeder
ls: cannot access '/glade/work/raeder/cases': Permission denied
ls: cannot access '/glade/work/raeder/Models': Permission denied
ls: cannot access '/glade/work/raeder/group_git': Permission denied
ls: cannot access '/glade/work/raeder/Diag': Permission denied
ls: cannot access '/glade/work/raeder/Bf_ptmp': Permission denied
ls: cannot access '/glade/work/raeder/Exp': Permission denied
ls: cannot access '/glade/work/raeder/Git': Permission denied
total 0
d????????? ? ? ? ? ? Bf_ptmp
d????????? ? ? ? ? ? cases
d????????? ? ? ? ? ? Diag
d????????? ? ? ? ? ? Exp
d????????? ? ? ? ? ? Git
d????????? ? ? ? ? ? group_git
d????????? ? ? ? ? ? Models

[kdraeder]

This is partly about understanding how git(hub) actually works. I've read a lot of confusing documentation and comments about it, so I'm figuring out relationships between local file permissions, local git permissions (init --shared), and github branch protections. So far it seems that local permissions have nothing to do with github branch protections and can't be propagated to there, but branch protections are not my focus at the moment.

It's partly about my wanting to set up 2 kinds of assimilations; cases where I don't want anyone else to be able to do work in the local directories or push to github, and cases where I do. An example of the latter is the Reanalysis, which should at least be easy for someone else in the group to continue if I weren't able to, and ideally would be run by more than 1 person, so that no one is responsible for it 100% of the time. During the Reanalysis we decided that changes to the run configuration should be approved by someone and archived on github. One or more people need permission to do each stage of this; change files, add and commit them to git, push them to github, approve them, and run the job.

The immediate goal here is that I've created a case in which I think that I've given permission to group members to change the files, but not commit them to git. I need someone else in the group to test that.
It should only take a couple minutes, and would help me finish this issue of developing a system to archive work.

[2024-4-10 Oops, forgot to submit this comment weeks ago]