Skip to content

Commit cec22a0

Browse files
alexbenedictoalexbenedicto
committed
Add ci_developer_guide.rst
1 parent 8d5b2aa commit cec22a0

File tree

2 files changed

+328
-0
lines changed

2 files changed

+328
-0
lines changed

‎docs/ci_developer_guide.rst‎

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
======================================
2+
Continuous Integration Developer Guide
3+
======================================
4+
5+
This document explains the geosPythonPackages CI/CD system, how it works, and how developers should use it.
6+
7+
.. contents::
8+
:local:
9+
:depth: 3
10+
11+
Overview
12+
========
13+
14+
The geosPythonPackages repository uses a **mandatory two-stage CI system** that ensures all changes are compatible with both the geosPythonPackages ecosystem and the GEOS integration environment.
15+
16+
System Architecture
17+
===================
18+
19+
The CI system consists of multiple workflows that run in a specific sequence:
20+
21+
Workflow Structure
22+
------------------
23+
24+
.. code-block:: text
25+
26+
Pull Request Created
27+
├── Stage 1: Regular CI Tests (Automatic)
28+
│ ├── semantic_pull_request
29+
│ ├── build (matrix: Python 3.10, 3.11, 3.12 × 11 packages)
30+
│ └── check_geos_integration_required
31+
├── Stage 2: User Action Required
32+
│ └── Add "test-geos-integration" label
33+
└── Stage 3: GEOS Integration Tests
34+
├── test_geos_integration (reusable workflow)
35+
└── final_validation
36+
37+
Workflow Files
38+
--------------
39+
40+
* **python-package.yml**: Main CI orchestrator
41+
* **test_geos_integration.yml**: GEOS integration testing (reusable)
42+
* **doc-test.yml**: Documentation testing
43+
* **typing-check.yml**: Type checking validation
44+
45+
Stage 1: Regular CI Tests
46+
=========================
47+
48+
These tests run automatically on every PR and validate the geosPythonPackages code itself.
49+
50+
Semantic Pull Request Check
51+
---------------------------
52+
53+
**Job**: ``semantic_pull_request``
54+
55+
Validates that PR titles follow conventional commit semantics:
56+
57+
.. code-block:: text
58+
59+
feat: add new XML validation tool # Valid
60+
fix: correct mesh generation bug # Valid
61+
docs: update installation guide # Valid
62+
Updated some files # Invalid
63+
64+
Configuration:
65+
- Uses ``amannn/action-semantic-pull-request@v5.5.3``
66+
- Allows work-in-progress PRs
67+
- Scope is optional
68+
69+
Build Matrix Testing
70+
--------------------
71+
72+
**Job**: ``build``
73+
74+
Tests all packages across multiple Python versions:
75+
76+
**Matrix Dimensions:**
77+
- **Python versions**: 3.10, 3.11, 3.12
78+
- **Packages**: 11 packages (geos-ats, geos-geomechanics, geos-mesh, etc.)
79+
- **Dependencies**: Automatic handling of inter-package dependencies
80+
81+
**Test Steps:**
82+
1. Install package with dependencies
83+
2. Lint with yapf
84+
3. Run pytest (handles packages without tests)
85+
86+
**Key Features:**
87+
- Fail-fast disabled (tests all combinations)
88+
- Max parallel jobs: 3
89+
- Dependency resolution for complex packages
90+
91+
Label Requirement Check
92+
-----------------------
93+
94+
**Job**: ``check_geos_integration_required``
95+
96+
Determines if the mandatory ``test-geos-integration`` label is present.
97+
98+
.. important::
99+
This job **always runs** and outputs whether GEOS integration testing should proceed.
100+
101+
Stage 2: User Action Required
102+
=============================
103+
104+
After regular CI passes, users **must** add the ``test-geos-integration`` label to proceed.
105+
106+
Why This Step Exists
107+
--------------------
108+
109+
1. **Performance**: GEOS integration tests are resource-intensive
110+
2. **User Control**: Allows validation of regular changes first
111+
3. **Mandatory Validation**: Ensures ALL PRs are tested against GEOS
112+
4. **Clear Workflow**: Explicit user action required for final validation
113+
114+
How to Add the Label
115+
--------------------
116+
117+
**Via GitHub Web Interface:**
118+
119+
1. Navigate to your PR
120+
2. Click "Labels" in the right sidebar
121+
3. Select ``test-geos-integration``
122+
123+
**Via GitHub CLI:**
124+
125+
.. code-block:: bash
126+
127+
gh pr edit <PR_NUMBER> --add-label "test-geos-integration"
128+
129+
Stage 3: GEOS Integration Tests
130+
===============================
131+
132+
These tests run **only after** the label is added and validate integration with GEOS.
133+
134+
Integration Test Components
135+
---------------------------
136+
137+
**Repository Checkout:**
138+
- Clones geosPythonPackages (current PR branch)
139+
- Clones GEOS (develop branch)
140+
141+
**System Setup:**
142+
- Python 3.10, 3.11, 3.12 environment
143+
- CMake and build tools
144+
- System dependencies
145+
146+
**Integration Validation:**
147+
148+
1. **Python Environment Compatibility**
149+
150+
- Tests ``scripts/setupPythonEnvironment.bash`` from GEOS
151+
- Validates environment setup procedures
152+
153+
2. **CMake Integration**
154+
155+
.. code-block:: bash
156+
157+
cmake .. \
158+
-DGEOSX_BUILD_SHARED_LIBS=OFF \
159+
-DENABLE_GEOSX_PYTHON_TOOLS=ON \
160+
-DGEOS_PYTHON_PACKAGES_SOURCE="${GEOSPYTHONPACKAGES_ROOT}" \
161+
-DCMAKE_BUILD_TYPE=Release
162+
163+
3. **Build Target Testing**
164+
165+
- Verifies ``make geosx_python_tools`` target exists
166+
- Attempts to build Python tools (graceful failure handling)
167+
- Validates build system integration
168+
169+
4. **Tool Availability Verification**
170+
171+
- XML processing tools (preprocess_xml.py, format_xml.py)
172+
- ATS integration (run_geos_ats.py)
173+
- Package structure validation
174+
175+
5. **Script Integrity Checks**
176+
177+
- Syntax validation for installation scripts
178+
- Directory structure verification
179+
180+
Final Validation
181+
----------------
182+
183+
**Job**: ``final_validation``
184+
185+
**Critical Logic:**
186+
187+
.. code-block:: yaml
188+
189+
if GEOS integration was triggered:
190+
if GEOS tests passed:
191+
SUCCESS - PR can be merged
192+
else:
193+
FAIL - PR blocked from merging
194+
else:
195+
FAIL - Label is mandatory for ALL PRs
196+
197+
Developer Workflow
198+
==================
199+
200+
Complete PR Lifecycle
201+
---------------------
202+
203+
1. **Create Pull Request**
204+
205+
.. code-block:: bash
206+
207+
git checkout -b authorname/feature/my-new-feature
208+
# Make changes
209+
git commit -m "feat: add new functionality"
210+
git push origin authorname/feature/my-new-feature
211+
# Create PR via GitHub
212+
213+
2. **Monitor Regular CI**
214+
215+
- Wait for ``semantic_pull_request`` and ``build`` jobs to complete
216+
- Fix any failures in regular testing
217+
- **Do not add the label until regular CI passes**
218+
219+
3. **Add Integration Label**
220+
221+
Once regular CI is green:
222+
223+
1. Navigate to your PR
224+
2. Click "Labels" in the right sidebar
225+
3. Select ``test-geos-integration``
226+
227+
OR
228+
229+
.. code-block:: bash
230+
231+
gh pr edit <PR_NUMBER> --add-label "test-geos-integration"
232+
233+
4. **Monitor GEOS Integration**
234+
235+
- GEOS integration tests will automatically start
236+
- Monitor progress in the Actions tab
237+
238+
5. **Address Integration Issues**
239+
240+
If GEOS integration fails:
241+
- Review the integration test logs
242+
- Fix compatibility issues
243+
- Push updates (tests will re-run automatically)
244+
245+
6. **Merge When Ready**
246+
247+
PR can be merged when ALL status checks are green:
248+
- semantic_pull_request
249+
- build (all matrix combinations)
250+
- Check GEOS Integration Required
251+
- Test geosPythonPackages Integration with GEOS
252+
- Final CI Validation
253+
- Reviews addressed and approved
254+
255+
Troubleshooting
256+
===============
257+
258+
Common Issues and Solutions
259+
---------------------------
260+
261+
**Issue**: "Regular CI failing on build job"
262+
263+
**Solutions**:
264+
- Check package dependencies in matrix configuration
265+
- Verify yapf formatting: ``yapf -r --diff ./package-name --style .style.yapf``
266+
- Run tests locally: ``python -m pytest ./package-name``
267+
268+
**Issue**: "GEOS integration test not starting"
269+
270+
**Solutions**:
271+
- Verify ``test-geos-integration`` label is present
272+
- Ensure regular CI completed successfully first
273+
- Check workflow logs for prerequisite failures
274+
275+
**Issue**: "make geosx_python_tools failing in integration"
276+
277+
**Solutions**:
278+
- Review CMake configuration in integration logs
279+
- Check if new dependencies are needed
280+
- Verify package structure matches GEOS expectations
281+
- Consider if changes require GEOS-side updates
282+
283+
Debug Commands
284+
--------------
285+
286+
**Local Testing Simulation**:
287+
288+
.. code-block:: bash
289+
290+
# Test yapf formatting
291+
yapf -r --diff ./geos-xml-tools --style .style.yapf
292+
293+
# Test package installation
294+
python -m pip install ./geos-xml-tools[test]
295+
296+
# Run package tests
297+
python -m pytest ./geos-xml-tools
298+
299+
# Test installation script syntax
300+
bash -n install_packages.sh
301+
302+
**Integration Testing Locally**:
303+
304+
.. code-block:: bash
305+
306+
# Clone GEOS for local testing
307+
git clone https://github.com/GEOS-DEV/GEOS.git
308+
cd GEOS
309+
310+
# Test CMake configuration
311+
mkdir build && cd build
312+
cmake .. \
313+
-DENABLE_GEOSX_PYTHON_TOOLS=ON \
314+
-DGEOS_PYTHON_PACKAGES_SOURCE="/path/to/geosPythonPackages" \
315+
-DCMAKE_BUILD_TYPE=Release
316+
317+
# Test build target
318+
make help | grep geosx_python_tools
319+
make geosx_python_tools

‎docs/index.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,3 +99,12 @@ Packages
9999
geos-xml-viewer
100100

101101
pygeos-tools
102+
103+
104+
Developer Documentation
105+
-----------------------
106+
107+
.. toctree::
108+
:maxdepth: 2
109+
110+
ci_developer_guide

0 commit comments

Comments
 (0)