Skip to content

Commit b223424

Browse files
pi-anlclaude
andcommitted
lib/lvgl: Add parallel processing and separate build target for Python stubs.
Major performance improvements and build system changes: **Separate Build Target:** - Remove stub generation from main build (was slowing down normal builds) - Create optional lvgl-stubs target for documentation-enabled stub generation - Main build now completes quickly without documentation parsing overhead **Parallel Processing Optimizations:** - Add multiprocessing support using ProcessPoolExecutor - Process 200+ header files using all available CPU cores - Pre-build documentation index to eliminate repeated file searches - Progress reporting every 50 processed files - Graceful fallback to serial processing if parallel fails **Performance Results:** - Previous: Minutes for documentation parsing (blocking main build) - Current: ~6 seconds for 209 files and 1423 functions (separate target) - Uses all CPU cores efficiently with progress feedback - Documentation index built once, O(1) function lookup vs O(n) file search **Documentation Updates:** - Add comprehensive DOCSTRING_PARSING.md explaining the implementation - Document new build workflow and performance characteristics - Include usage examples for separate stub generation **Technical Changes:** - Replace load_lvgl_source_files() with parallel processing version - Add process_file_for_docs() for individual file processing - Change from source_files dict to doc_index for O(1) lookups - Update generate_class_stub() and generate_main_stub() signatures - Maintain backward compatibility with graceful error handling 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 9df158d commit b223424

File tree

4 files changed

+484
-32
lines changed

4 files changed

+484
-32
lines changed

CLAUDE.md

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Overview
6+
7+
This repository contains MicroPython bindings for LVGL (Light and Versatile Graphics Library). It automatically generates Python bindings from LVGL C headers using the `gen_mpy.py` script, allowing LVGL to be used from MicroPython with native performance.
8+
9+
## Building and Integration
10+
11+
### As a MicroPython User Module
12+
13+
This module is typically used as a git submodule within a MicroPython project. The bindings are built automatically during the MicroPython build process.
14+
15+
**Build Integration:**
16+
- **Make-based builds**: Uses `micropython.mk` with automatic binding generation
17+
- **CMake-based builds**: Uses `micropython.cmake` and `mkrules_usermod.cmake`
18+
- The build system automatically runs `gen/gen_mpy.py` to generate `lv_mpy.c` from LVGL headers
19+
20+
**Key Build Files:**
21+
- `micropython.mk`: Make-based build rules and LVGL library integration
22+
- `micropython.cmake`: CMake integration for ESP32/IDF and other platforms
23+
- `lv_conf.h`: LVGL configuration (affects which bindings are generated)
24+
25+
### Building Standalone (for testing)
26+
27+
From a MicroPython repository with this as a user module:
28+
29+
```bash
30+
# Unix port (for testing with SDL)
31+
cd ports/unix
32+
make USER_C_MODULES=/path/to/lv_binding_micropython/micropython.cmake
33+
./build-lvgl/micropython
34+
35+
# ESP32 port
36+
cd ports/esp32
37+
make USER_C_MODULES=/path/to/lv_binding_micropython/micropython.cmake BOARD=ESP32_GENERIC
38+
```
39+
40+
## Testing
41+
42+
### Automated Tests
43+
44+
**API Tests** (can be automated/CI):
45+
```bash
46+
# From micropython/tests directory
47+
./run-tests.py ../../lib/lv_binding_micropython/tests/api/basic*.py -r .
48+
```
49+
50+
**Display Tests** (visual feedback, no interaction):
51+
```bash
52+
# From micropython/tests directory
53+
./run-tests.py ../../lib/lv_binding_micropython/tests/display/basic*.py -r .
54+
```
55+
56+
**Interactive Tests** (require user input):
57+
```bash
58+
# From micropython/tests directory
59+
./run-tests.py ../../lib/lv_binding_micropython/tests/indev/basic*.py -r .
60+
```
61+
62+
### Example Testing
63+
64+
Run all examples and demos:
65+
```bash
66+
cd tests
67+
./run.sh
68+
```
69+
70+
This runs all Python examples in parallel using GNU parallel, with 5-minute timeouts per test.
71+
72+
## Code Architecture
73+
74+
### Binding Generation System
75+
76+
**Core Components:**
77+
- `gen/gen_mpy.py`: Main binding generator that parses LVGL headers and generates MicroPython C API
78+
- `pycparser/`: Modified Python C parser for processing LVGL headers
79+
- `lvgl/`: Git submodule containing the actual LVGL library
80+
81+
**Generation Process:**
82+
1. C preprocessor processes LVGL headers with `lv_conf.h` settings
83+
2. `gen_mpy.py` uses pycparser to parse the preprocessed headers
84+
3. Generates `lv_mpy.c` containing MicroPython module definitions
85+
4. Build system compiles everything into the MicroPython binary
86+
87+
### Driver Architecture
88+
89+
**Driver Locations:**
90+
- `driver/esp32/`: ESP32-specific drivers (ILI9XXX, XPT2046, etc.)
91+
- `driver/generic/`: Platform-independent Python drivers
92+
- `driver/linux/`: Linux-specific drivers (evdev, etc.)
93+
- `driver/stm32/`: STM32-specific drivers
94+
95+
**Driver Types:**
96+
- **Pure Python**: Easiest to implement, runtime configurable
97+
- **Pure C**: Best performance, requires rebuild for config changes
98+
- **Hybrid**: Critical parts in C, configuration in Python
99+
100+
### Memory Management
101+
102+
- LVGL is configured to use MicroPython's garbage collector
103+
- Structs are automatically collected when no longer referenced
104+
- **Important**: Screen objects (`lv.obj` with no parent) are not auto-collected - call `screen.delete()` explicitly
105+
- Keep references to display and input drivers to prevent premature collection
106+
107+
### Callback System
108+
109+
**Callback Convention**: LVGL callbacks must follow specific patterns to work with MicroPython:
110+
- Struct containing `void * user_data` field
111+
- `user_data` passed as first argument to registration function and callback
112+
- The binding automatically manages `user_data` for MicroPython callable objects
113+
114+
## Development Patterns
115+
116+
### Configuration Management
117+
118+
**Runtime Configuration**: Unlike typical LVGL C drivers, MicroPython drivers should allow runtime configuration:
119+
120+
```python
121+
# Good - runtime configurable
122+
from ili9XXX import ili9341
123+
display = ili9341(dc=32, cs=33, mosi=23, clk=18)
124+
125+
# Avoid - requiring rebuild for pin changes
126+
```
127+
128+
### Adding New Drivers
129+
130+
1. **Determine driver type** (Pure Python, C, or Hybrid)
131+
2. **Follow existing patterns** in `driver/` subdirectories
132+
3. **Make runtime configurable** - avoid hardcoded pins/settings
133+
4. **Implement standard interface**:
134+
- Display drivers: `flush_cb` method or automatic setup
135+
- Input drivers: `read_cb` method or automatic setup
136+
137+
### Testing New Features
138+
139+
1. **Add API tests** in `tests/api/` for automated testing
140+
2. **Add display tests** in `tests/display/` for visual verification
141+
3. **Follow existing test patterns** - see `tests/README.md`
142+
4. **Test on multiple platforms** when possible
143+
144+
## Common Operations
145+
146+
### Regenerating Bindings
147+
148+
If you modify `lv_conf.h` or LVGL headers:
149+
```bash
150+
# Clean and rebuild to regenerate bindings
151+
make clean
152+
make USER_C_MODULES=/path/to/lv_binding_micropython/micropython.cmake
153+
```
154+
155+
### Testing Configuration Changes
156+
157+
Use the examples to verify your changes:
158+
```bash
159+
# Run a simple test
160+
./build-lvgl/micropython examples/example1.py
161+
162+
# Or run comprehensive tests
163+
cd tests && ./run.sh
164+
```
165+
166+
### Debugging Memory Issues
167+
168+
- Use `gc.collect()` to trigger garbage collection
169+
- Call `screen.delete()` on screens when done
170+
- Keep driver references in global variables or long-lived objects
171+
172+
## Integration Notes
173+
174+
- This module requires MicroPython's internal scheduler to be enabled
175+
- LVGL task handler is called via `mp_sched_schedule` for screen refresh
176+
- Single-threaded design - LVGL and MicroPython run on same thread
177+
- Display drivers typically handle event loop setup automatically

DOCSTRING_PARSING.md

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Docstring Parsing and Generation Summary
2+
3+
The LVGL MicroPython bindings now include comprehensive docstring extraction that converts C documentation to Python docstrings. Here's how it works:
4+
5+
## 1. **Source File Discovery**
6+
```python
7+
def load_lvgl_source_files(lvgl_dir):
8+
# Searches key LVGL directories:
9+
# - src/widgets/
10+
# - src/core/
11+
# - src/misc/
12+
# - src/draw/
13+
14+
# Loads all .h files into memory as line arrays
15+
# Returns: {file_path: [line1, line2, ...]}
16+
```
17+
18+
## 2. **Doxygen Comment Parsing**
19+
```python
20+
def parse_doxygen_comment(comment_text):
21+
# Input: Raw comment block from C header
22+
"""
23+
/**
24+
* Set a new text for a label. Memory will be allocated to store the text by the label.
25+
* @param obj pointer to a label object
26+
* @param text '\0' terminated character string. NULL to refresh with the current text.
27+
*/
28+
"""
29+
30+
# Output: Structured data
31+
{
32+
'description': 'Set a new text for a label. Memory will be allocated...',
33+
'params': [
34+
('obj', 'pointer to a label object'),
35+
('text', "'\\0' terminated character string. NULL to refresh...")
36+
],
37+
'returns': None
38+
}
39+
```
40+
41+
**Parsing Process:**
42+
- Strips comment markers (`/**`, `*/`, `*`, `//`)
43+
- Identifies `@param name description` patterns
44+
- Handles `@return description` sections
45+
- Supports multi-line descriptions
46+
- Extracts main description text
47+
48+
**Implementation Note:** The Doxygen parsing is implemented using **custom regular expressions and string parsing** - no external documentation parsing libraries are used. This keeps the dependency footprint minimal while providing exactly the functionality needed for LVGL's documentation format.
49+
50+
## 3. **Function Documentation Lookup**
51+
```python
52+
def find_function_docs_in_sources(func_name, source_files):
53+
# For each source file:
54+
# 1. Search for function name pattern: "lv_label_set_text("
55+
# 2. Look backwards from function declaration
56+
# 3. Collect preceding comment block
57+
# 4. Parse with parse_doxygen_comment()
58+
59+
# Example: "lv_label_set_text" finds docs in lv_label.h
60+
```
61+
62+
## 4. **Python Docstring Generation**
63+
```python
64+
def format_python_docstring(func_name, doc_info, args_info):
65+
# Combines parsed docs with function signature info
66+
# Generates formatted Python docstring:
67+
68+
"""
69+
Set a new text for a label. Memory will be allocated to store the text by the label.
70+
71+
Args:
72+
text (str): '\0' terminated character string. NULL to refresh with the current text.
73+
74+
Returns:
75+
None
76+
"""
77+
```
78+
79+
**Formatting Rules:**
80+
- Description comes first
81+
- Args section with type hints: `param_name (type): description`
82+
- Returns section when present
83+
- Proper indentation and spacing
84+
85+
## 5. **Integration with Stub Generation**
86+
87+
**For Class Methods:**
88+
```python
89+
# 1. Look up documentation: "lv_label_set_text"
90+
full_func_name = f"lv_{class_name}_{member_name}"
91+
doc_info = find_function_docs_in_sources(full_func_name, source_files)
92+
93+
# 2. Generate stub with special handling
94+
method_stub = generate_function_stub(member_name, member_info, doc_info, is_class_method=True)
95+
96+
# 3. Result:
97+
def set_text(self: Self, text: str) -> None:
98+
"""
99+
Set a new text for a label. Memory will be allocated to store the text by the label.
100+
101+
Args:
102+
text (str): '\0' terminated character string. NULL to refresh with the current text.
103+
"""
104+
...
105+
```
106+
107+
**For Regular Functions:**
108+
```python
109+
# Direct lookup and generation
110+
doc_info = find_function_docs_in_sources(func_name, source_files)
111+
stub = generate_function_stub(func_name, func_info, doc_info, is_class_method=False)
112+
```
113+
114+
## 6. **Build System Integration**
115+
116+
**Separate Build Target (Fast):**
117+
Since documentation parsing is slow (5-10 seconds), stub generation is now a separate optional target:
118+
119+
```bash
120+
# Normal build (fast, no stubs):
121+
make USER_C_MODULES=path/to/lvgl
122+
123+
# Generate Python stubs separately (slow, with documentation):
124+
cd path/to/lvgl
125+
python3 gen/gen_mpy.py -M lvgl -MP lv -S stubs_output_dir -E preprocessed_file.pp lvgl/lvgl.h
126+
```
127+
128+
**Process Flow:**
129+
1. Main build generates MicroPython bindings quickly (no documentation parsing)
130+
2. Optional stub generation loads 200+ LVGL header files using parallel processing
131+
3. Generator builds documentation index using multiple CPU cores
132+
4. For each function/method, looks up docs from pre-built index
133+
5. Generates Python stubs with rich docstrings
134+
6. Outputs `.pyi` files for IDE consumption
135+
136+
**Performance:**
137+
- **Parallel Processing**: Uses all CPU cores to process header files simultaneously
138+
- **Pre-indexing**: Builds function documentation index once, avoids repeated searches
139+
- **Speed**: ~6 seconds for 209 files and 1423 functions (vs. minutes with old approach)
140+
141+
## 7. **Key Features**
142+
143+
- **Automatic**: No manual documentation writing required
144+
- **Comprehensive**: Processes entire LVGL codebase (200+ headers)
145+
- **Smart**: Handles class methods vs static methods appropriately
146+
- **Type-aware**: Converts C types to Python type hints
147+
- **IDE-friendly**: Generates standard Python docstring format
148+
- **Custom Implementation**: Uses regex-based parsing, no external dependencies
149+
150+
## 8. **Technical Details**
151+
152+
### Doxygen Parsing Implementation
153+
The Doxygen comment parsing is implemented entirely with Python's built-in `re` (regular expressions) module and string manipulation. Key parsing functions:
154+
155+
- `parse_doxygen_comment()`: Main parser using string splitting and pattern matching
156+
- `extract_function_docs()`: Regex-based function declaration finder
157+
- `find_function_docs_in_sources()`: File traversal and documentation lookup
158+
159+
### Supported Doxygen Tags
160+
- `@param name description` - Function parameters
161+
- `@return description` - Return value documentation
162+
- Main description text (everything not starting with @)
163+
- Multi-line descriptions for all sections
164+
165+
### File Processing
166+
- Processes all `.h` files in LVGL source tree
167+
- Handles UTF-8 encoding with fallback for problematic files
168+
- Caches file contents in memory for efficient lookup
169+
- Gracefully handles missing or malformed documentation
170+
171+
The result is that Python developers get full IDE autocompletion and documentation for all LVGL functions, automatically extracted from the original C source documentation without requiring external documentation parsing libraries.

0 commit comments

Comments
 (0)