Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/cicd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,16 @@ jobs:
ports:
- 9202:9202

redis:
image: redis:7-alpine
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379

strategy:
matrix:
python-version: [ "3.9", "3.10", "3.11", "3.12", "3.13"]
Expand Down Expand Up @@ -126,3 +136,6 @@ jobs:
DATABASE_REFRESH: true
ES_VERIFY_CERTS: false
BACKEND: ${{ matrix.backend == 'elasticsearch7' && 'elasticsearch' || matrix.backend == 'elasticsearch8' && 'elasticsearch' || 'opensearch' }}
REDIS_ENABLE: true
REDIS_HOST: localhost
REDIS_PORT: 6379
3 changes: 2 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ repos:
]
additional_dependencies: [
"types-attrs",
"types-requests"
"types-requests",
"types-redis"
]
- repo: https://github.com/PyCQA/pydocstyle
rev: 6.1.1
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
- Moved SFEOS Tools to its own repository at [Healy-Hyperspatial/sfeos-tools](https://github.com/Healy-Hyperspatial/sfeos-tools). The CLI package is now maintained separately. [#PR_NUMBER]
- CloudFerro logo to sponsors and supporters list [#485](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/485)
- Latest news section to README [#485](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/485)
- Added Redis caching configuration for navigation pagination support, enabling proper `prev` and `next` links in paginated responses. [#488](https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/pull/488)

### Changed

Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,4 @@ docs-image:
.PHONY: docs
docs: docs-image
docker compose -f compose.docs.yml \
run docs
run docs
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,31 @@ You can customize additional settings in your `.env` file:
> [!NOTE]
> The variables `ES_HOST`, `ES_PORT`, `ES_USE_SSL`, `ES_VERIFY_CERTS` and `ES_TIMEOUT` apply to both Elasticsearch and OpenSearch backends, so there is no need to rename the key names to `OS_` even if you're using OpenSearch.

**Redis for Navigation:**
These Redis configuration variables enable proper navigation functionality in STAC FastAPI. The Redis cache stores navigation state for paginated results, allowing the system to maintain previous page links using tokens. The configuration supports either Redis Sentinel or standalone Redis setups.

| Variable | Description | Default | Required |
|-------------------------------|----------------------------------------------------------------------------------------------|--------------------------|---------------------------------------------------------------------------------------------|
| **General** | | | |
| `REDIS_ENABLE` | Enables or disables Redis caching for navigation. Set to `true` to use Redis, or `false` to disable. | `false` | **Required** (determines whether Redis is used at all) |
| **Redis Sentinel** | | | |
| `REDIS_SENTINEL_HOSTS` | Comma-separated list of Redis Sentinel hostnames/IP addresses. | `""` | Conditional (required if using Sentinel) |
| `REDIS_SENTINEL_PORTS` | Comma-separated list of Redis Sentinel ports (must match order). | `"26379"` | Conditional (required if using Sentinel) |
| `REDIS_SENTINEL_MASTER_NAME` | Name of the Redis master node in Sentinel configuration. | `"master"` | Conditional (required if using Sentinel) |
| **Redis** | | | |
| `REDIS_HOST` | Redis server hostname or IP address for Redis configuration. | `""` | Conditional (required for standalone Redis) |
| `REDIS_PORT` | Redis server port for Redis configuration. | `6379` | Conditional (required for standalone Redis) |
| **Both** | | | |
| `REDIS_DB` | Redis database number to use for caching. | `0` (Sentinel) / `0` (Standalone) | Optional |
| `REDIS_MAX_CONNECTIONS` | Maximum number of connections in the Redis connection pool. | `10` | Optional |
| `REDIS_RETRY_TIMEOUT` | Enable retry on timeout for Redis operations. | `true` | Optional |
| `REDIS_DECODE_RESPONSES` | Automatically decode Redis responses to strings. | `true` | Optional |
| `REDIS_CLIENT_NAME` | Client name identifier for Redis connections. | `"stac-fastapi-app"` | Optional |
| `REDIS_HEALTH_CHECK_INTERVAL` | Interval in seconds for Redis health checks. | `30` | Optional |

> [!NOTE]
> Use either the Sentinel configuration (`REDIS_SENTINEL_HOSTS`, `REDIS_SENTINEL_PORTS`, `REDIS_SENTINEL_MASTER_NAME`) OR the Redis configuration (`REDIS_HOST`, `REDIS_PORT`), but not both.

## Datetime-Based Index Management

### Overview
Expand Down
19 changes: 19 additions & 0 deletions compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ services:
- BACKEND=elasticsearch
- DATABASE_REFRESH=true
- ENABLE_COLLECTIONS_SEARCH_ROUTE=true
- REDIS_ENABLE=true
- REDIS_HOST=redis
- REDIS_PORT=6379
ports:
- "8080:8080"
volumes:
Expand All @@ -31,6 +34,7 @@ services:
- ./esdata:/usr/share/elasticsearch/data
depends_on:
- elasticsearch
- redis
command:
bash -c "./scripts/wait-for-it-es.sh es-container:9200 && python -m stac_fastapi.elasticsearch.app"

Expand Down Expand Up @@ -58,6 +62,9 @@ services:
- BACKEND=opensearch
- STAC_FASTAPI_RATE_LIMIT=200/minute
- ENABLE_COLLECTIONS_SEARCH_ROUTE=true
- REDIS_ENABLE=true
- REDIS_HOST=redis
- REDIS_PORT=6379
ports:
- "8082:8082"
volumes:
Expand All @@ -66,6 +73,7 @@ services:
- ./osdata:/usr/share/opensearch/data
depends_on:
- opensearch
- redis
command:
bash -c "./scripts/wait-for-it-es.sh os-container:9202 && python -m stac_fastapi.opensearch.app"

Expand Down Expand Up @@ -96,3 +104,14 @@ services:
- ./opensearch/snapshots:/usr/share/opensearch/snapshots
ports:
- "9202:9202"

redis:
image: redis:7-alpine
hostname: redis
ports:
- "6379:6379"
volumes:
- redis_test_data:/data
command: redis-server
volumes:
redis_test_data:
2 changes: 2 additions & 0 deletions stac_fastapi/core/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"jsonschema~=4.0.0",
"slowapi~=0.1.9",
]
extra_reqs = {"redis": ["redis~=6.4.0"]}

setup(
name="stac_fastapi_core",
Expand All @@ -43,4 +44,5 @@
packages=find_namespace_packages(),
zip_safe=False,
install_requires=install_requires,
extras_require=extra_reqs,
)
41 changes: 39 additions & 2 deletions stac_fastapi/core/stac_fastapi/core/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,10 @@
from stac_fastapi.core.base_settings import ApiBaseSettings
from stac_fastapi.core.datetime_utils import format_datetime_range
from stac_fastapi.core.models.links import PagingLinks
from stac_fastapi.core.redis_utils import redis_pagination_links
from stac_fastapi.core.serializers import CollectionSerializer, ItemSerializer
from stac_fastapi.core.session import Session
from stac_fastapi.core.utilities import filter_fields
from stac_fastapi.core.utilities import filter_fields, get_bool_env
from stac_fastapi.extensions.core.transaction import AsyncBaseTransactionsClient
from stac_fastapi.extensions.core.transaction.request import (
PartialCollection,
Expand Down Expand Up @@ -273,6 +274,7 @@ async def all_collections(
A Collections object containing all the collections in the database and links to various resources.
"""
base_url = str(request.base_url)
redis_enable = get_bool_env("REDIS_ENABLE", default=False)

global_max_limit = (
int(os.getenv("STAC_GLOBAL_COLLECTION_MAX_LIMIT"))
Expand Down Expand Up @@ -428,6 +430,14 @@ async def all_collections(
},
]

if redis_enable:
await redis_pagination_links(
current_url=str(request.url),
token=token,
next_token=next_token,
links=links,
)

if next_token:
next_link = PagingLinks(next=next_token, request=request).link_next()
links.append(next_link)
Expand Down Expand Up @@ -772,8 +782,8 @@ async def post_search(
search_request.limit = limit

base_url = str(request.base_url)

search = self.database.make_search()
redis_enable = get_bool_env("REDIS_ENABLE", default=False)

if search_request.ids:
search = self.database.apply_ids_filter(
Expand Down Expand Up @@ -877,6 +887,33 @@ async def post_search(
]
links = await PagingLinks(request=request, next=next_token).get_links()

collection_links = []
if search_request.collections:
for collection_id in search_request.collections:
collection_links.extend(
[
{
"rel": "collection",
"type": "application/json",
"href": urljoin(base_url, f"collections/{collection_id}"),
},
{
"rel": "parent",
"type": "application/json",
"href": urljoin(base_url, f"collections/{collection_id}"),
},
]
)
links.extend(collection_links)

if redis_enable:
await redis_pagination_links(
current_url=str(request.url),
token=token_param,
next_token=next_token,
links=links,
)

return stac_types.ItemCollection(
type="FeatureCollection",
features=items,
Expand Down
Loading
Loading