-
Notifications
You must be signed in to change notification settings - Fork 55
add autobuild on delta #3764
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
matthiasdiener
wants to merge
33
commits into
main
Choose a base branch
from
autobuild-delta
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
add autobuild on delta #3764
Changes from 30 commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
1d596d2
add autobuild on delta
matthiasdiener 42d18d9
add a test
matthiasdiener 45e1409
cancel concurrent
matthiasdiener e9e6ee0
on push
matthiasdiener 560764f
remove old ref
matthiasdiener 11a4013
actually run a build, more comments
matthiasdiener bbc78d5
slurm script to submit autobuild run
ericjbohm 6bd105a
switch to interactive
ericjbohm f0eb63f
switch to all-test
ericjbohm d914b09
slimmed down script
ericjbohm 6fddd49
use srun directly
matthiasdiener b33e7b3
bump nnodes, ntasks, mem
matthiasdiener ea44037
remove ntasks
matthiasdiener c1db5dc
proper production and error checking flags in build
ericjbohm b55b653
more verbosity in the script
ericjbohm 325221f
bump cpus-per-task
matthiasdiener 05a93a9
temporarily remove Github CI files
matthiasdiener f9b6235
nnodes=1
matthiasdiener 253c1f8
run the tests after compiling them
ericjbohm d93f40c
add delta sbatch as run dependency
matthiasdiener 5ee524b
switch back to sbatch approach
ericjbohm 0a62264
script to poll the jobqueue and then cat the results of the run
ericjbohm b982df5
batch version revised to use SLURM variables for filenames
ericjbohm 3596981
revised to call the jobmonitor for resource montoring
ericjbohm 88cc925
preserve output and status in latest.output and latest.status
ericjbohm e28a559
revise to check status file and report success/failure
ericjbohm 07be593
remove tabs that yaml hates
ericjbohm 92931ff
more whitespace syntax adherence
ericjbohm 7e08549
!fixup typo in result file name
ericjbohm 91297ae
Merge branch 'main' into autobuild-delta
matthiasdiener 1eb8e7f
disable normal CI
matthiasdiener 084cf33
cleanups, first attempt at pages upload
matthiasdiener 075364d
Merge branch 'main' into autobuild-delta
matthiasdiener File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
name: Autobuild | ||
|
||
on: | ||
schedule: | ||
# Run the build as part of a fixed nightly schedule | ||
- cron: '15 06 * * *' # UTC 6:15am, corresponds to 00:15 CST or 01:15 CDT | ||
push: | ||
paths: | ||
# Also run the build when this file gets modified as part of a PR | ||
- '.github/workflows/autobuild.yml' | ||
- '.github/workflows/delta-sbatch-slurm.sh' | ||
|
||
|
||
# Cancel in progress CI runs when a new run targeting the same PR or branch/tag is triggered. | ||
# https://stackoverflow.com/questions/66335225/how-to-cancel-previous-runs-in-the-pr-when-you-push-new-commitsupdate-the-curre | ||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.ref }} | ||
cancel-in-progress: true | ||
|
||
jobs: | ||
Delta: | ||
timeout-minutes: 60 | ||
|
||
runs-on: delta | ||
name: Delta mpi-linux-x86_64 # Could test various builds (e.g., MPI, UCX, ...) via a build matrix | ||
|
||
steps: | ||
- uses: actions/checkout@v3 | ||
- name: Host info | ||
run: | | ||
set -x | ||
echo "Running autobuild on delta" | ||
hostname | ||
uname -a | ||
lsb_release -a | ||
pwd | ||
- name: build | ||
run: | | ||
set -ex | ||
export target="mpi-linux-x86_64" | ||
.github/workflows/jobmonitor.sh .github/workflows/delta-sbatch-slurm.sh | ||
- name: results | ||
run: | | ||
if grep '0' result.latest | ||
then | ||
echo "Success" | ||
else | ||
echo "Failure" | ||
fi | ||
# should also https://github.com/marketplace/actions/send-email |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
#!/bin/bash -l | ||
#SBATCH -N 2 | ||
#SBATCH -n 64 | ||
#SBATCH -o %j.output | ||
#SBATCH -e %j.output | ||
#SBATCH -t 1:00:00 | ||
#SBATCH -J autobuild | ||
#SBATCH -p cpu | ||
#SBATCH -A mzu-delta-cpu | ||
#cd $indir | ||
set -x | ||
module load libfabric; module load cmake | ||
./build all-test $target --with-production --enable-error-checking -j64 -g | ||
# | ||
cd $target | ||
make -C tests test OPTS="$flags" TESTOPTS="$testopts" $maketestopts | ||
make -C examples test OPTS="$flags" TESTOPTS="$testopts" $maketestopts | ||
make -C benchmarks test OPTS="$flags" TESTOPTS="$testopts" $maketestopts | ||
# Save make exit status | ||
status=$? | ||
echo $status > ../$SLURM_JOBID.result |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,69 @@ | ||
#!/bin/bash | ||
|
||
export script=$1; | ||
queue_qsub=sbatch | ||
queue_kill=scancel | ||
queue_stat="squeue -j" | ||
|
||
End() { | ||
echo "autobuild> $queue_kill $jobid ..." | ||
$queue_kill $jobid | ||
exit $1 | ||
} | ||
echo "Submitting batch job for> $target OPTS=$flags" | ||
echo " using the command> $queue_qsub $script" | ||
chmod 755 $script | ||
while [ -z "$jobid" ] | ||
do | ||
$queue_qsub $script > .status.$$ 2>&1 | ||
if grep 'have no accounts' .status.$$ > /dev/null | ||
then | ||
echo "NO account for submitting batch job!" | ||
rm -f .status.$$ | ||
exit 1 | ||
fi | ||
jobid=`cat .status.$$ | tail -1 | awk '{print $4}'` | ||
rm -f .status.$$ | ||
done | ||
echo "Job enqueued under job ID $jobid" | ||
export output=$jobid.output | ||
export result=$jobid.result | ||
# kill job if interrupted | ||
trap 'End 1' 2 3 | ||
retry=0 | ||
# Wait for the job to complete, by checking its status | ||
while [ true ] | ||
do | ||
$queue_stat $jobid > tmp.$$ | ||
linecount=`wc -l tmp.$$ | awk '{print $1}' ` | ||
exitstatus=$? | ||
#if [[ $exitstatus != 0 || $linecount != 2 ]] | ||
if [[ $linecount != 2 ]] | ||
then | ||
# The job is done-- print its output | ||
rm tmp.$$ | ||
# When job hangs, result file does not exist | ||
test -f $result && status=`cat $result` || status=1 | ||
echo "==================================== OUTPUT (STDOUT & STDERR) ========================================" | ||
cat $output | ||
echo "======================================================================================================" | ||
if [[ "$status" != 0 ]]; | ||
then | ||
#print script | ||
echo "=============================================== SCRIPT ===============================================" | ||
cat $script | ||
echo "======================================================================================================" | ||
echo "=============================================== RESULT ===============================================" | ||
cat $result | ||
echo "======================================================================================================" | ||
fi | ||
# mv result and output to result.latest | ||
mv $result result.latest | ||
mv $output output.latest | ||
exit $status | ||
fi | ||
# The job is still queued or running-- print status and wait | ||
tail -1 tmp.$$ | ||
rm tmp.$$ | ||
sleep 60 | ||
done |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.