Skip to content

Commit 7159648

Browse files
Update documentation (#980)
Co-authored-by: daniil-quix <133032822+daniil-quix@users.noreply.github.com>
1 parent 48d1ace commit 7159648

File tree

2 files changed

+80
-46
lines changed

2 files changed

+80
-46
lines changed

docs/api-reference/dataframe.md

Lines changed: 40 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -388,11 +388,11 @@ or False otherwise.
388388

389389
```python
390390
def to_topic(
391-
topic: Topic,
391+
topic: Union[Topic, Callable[[Any, Any, int, Any], Topic]],
392392
key: Optional[Callable[[Any], Any]] = None) -> "StreamingDataFrame"
393393
```
394394

395-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L670)
395+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L684)
396396

397397
Produce current value to a topic. You can optionally specify a new key.
398398

@@ -417,13 +417,30 @@ sdf = app.dataframe(input_topic)
417417
sdf = sdf.to_topic(output_topic_0)
418418
# does not require reassigning
419419
sdf.to_topic(output_topic_1, key=lambda data: data["a_field"])
420+
421+
# Dynamic topic selection based on message content
422+
def select_topic(value, key, timestamp, headers):
423+
if value.get("priority") == "high":
424+
return output_topic_0
425+
else:
426+
return output_topic_1
427+
428+
sdf = sdf.to_topic(select_topic)
420429
```
421430

422431

423432
<br>
424433
***Arguments:***
425434

426-
- `topic`: instance of `Topic`
435+
- `topic`: instance of `Topic` or a callable that returns a `Topic`.
436+
If a callable is provided, it will receive four arguments:
437+
value, key, timestamp, and headers of the current message.
438+
The callable must return a `Topic` object.
439+
**Important**: We recommend declaring all `Topic` instances before
440+
staring the application instead of creating them dynamically
441+
within the passed callback. Creating topics dynamically can lead
442+
to accidentally creating numerous topics and
443+
saturating the broker's partitions limits.
427444
- `key`: a callable to generate a new message key, optional.
428445
If passed, the return type of this callable must be serializable
429446
by `key_serializer` defined for this Topic object.
@@ -446,7 +463,7 @@ def set_timestamp(
446463
func: Callable[[Any, Any, int, Any], int]) -> "StreamingDataFrame"
447464
```
448465

449-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L715)
466+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L753)
450467

451468
Set a new timestamp based on the current message value and its metadata.
452469

@@ -499,7 +516,7 @@ def set_headers(
499516
) -> "StreamingDataFrame"
500517
```
501518

502-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L758)
519+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L796)
503520

504521
Set new message headers based on the current message value and metadata.
505522

@@ -548,7 +565,7 @@ a new StreamingDataFrame instance
548565
def print(pretty: bool = True, metadata: bool = False) -> "StreamingDataFrame"
549566
```
550567

551-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L809)
568+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L847)
552569

553570
Print out the current message value (and optionally, the message metadata) to
554571

@@ -611,7 +628,7 @@ def print_table(
611628
int]] = None) -> "StreamingDataFrame"
612629
```
613630

614-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L855)
631+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L893)
615632

616633
Print a table with the most recent records.
617634

@@ -704,7 +721,7 @@ sdf.print_table(size=5, title="Live Records", slowdown=1)
704721
def compose(sink: Optional[VoidExecutor] = None) -> dict[str, VoidExecutor]
705722
```
706723

707-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L971)
724+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1009)
708725

709726
Compose all functions of this StreamingDataFrame into one big closure.
710727

@@ -758,7 +775,7 @@ def test(value: Any,
758775
topic: Optional[Topic] = None) -> List[Any]
759776
```
760777

761-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1005)
778+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1043)
762779

763780
A shorthand to test `StreamingDataFrame` with provided value
764781

@@ -798,7 +815,7 @@ def tumbling_window(
798815
) -> TumblingTimeWindowDefinition
799816
```
800817

801-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1044)
818+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1082)
802819

803820
Create a time-based tumbling window transformation on this StreamingDataFrame.
804821

@@ -890,7 +907,7 @@ def tumbling_count_window(
890907
name: Optional[str] = None) -> TumblingCountWindowDefinition
891908
```
892909

893-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1133)
910+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1171)
894911

895912
Create a count-based tumbling window transformation on this StreamingDataFrame.
896913

@@ -963,7 +980,7 @@ def hopping_window(
963980
) -> HoppingTimeWindowDefinition
964981
```
965982

966-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1183)
983+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1221)
967984

968985
Create a time-based hopping window transformation on this StreamingDataFrame.
969986

@@ -1066,7 +1083,7 @@ def hopping_count_window(
10661083
name: Optional[str] = None) -> HoppingCountWindowDefinition
10671084
```
10681085

1069-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1286)
1086+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1324)
10701087

10711088
Create a count-based hopping window transformation on this StreamingDataFrame.
10721089

@@ -1144,7 +1161,7 @@ def sliding_window(
11441161
) -> SlidingTimeWindowDefinition
11451162
```
11461163

1147-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1343)
1164+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1381)
11481165

11491166
Create a time-based sliding window transformation on this StreamingDataFrame.
11501167

@@ -1242,7 +1259,7 @@ def sliding_count_window(
12421259
name: Optional[str] = None) -> SlidingCountWindowDefinition
12431260
```
12441261

1245-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1438)
1262+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1476)
12461263

12471264
Create a count-based sliding window transformation on this StreamingDataFrame.
12481265

@@ -1312,7 +1329,7 @@ sdf = (
13121329
def fill(*columns: str, **mapping: Any) -> "StreamingDataFrame"
13131330
```
13141331

1315-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1491)
1332+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1529)
13161333

13171334
Fill missing values in the message value with a constant value.
13181335

@@ -1369,7 +1386,7 @@ def drop(columns: Union[str, List[str]],
13691386
errors: Literal["ignore", "raise"] = "raise") -> "StreamingDataFrame"
13701387
```
13711388

1372-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1543)
1389+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1581)
13731390

13741391
Drop column(s) from the message value (value must support `del`, like a dict).
13751392

@@ -1413,7 +1430,7 @@ a new StreamingDataFrame instance
14131430
def sink(sink: BaseSink)
14141431
```
14151432

1416-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1587)
1433+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1625)
14171434

14181435
Sink the processed data to the specified destination.
14191436

@@ -1441,7 +1458,7 @@ operations, but branches can still be generated from its originating SDF.
14411458
def concat(other: "StreamingDataFrame") -> "StreamingDataFrame"
14421459
```
14431460

1444-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1625)
1461+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1663)
14451462

14461463
Concatenate two StreamingDataFrames together and return a new one.
14471464

@@ -1482,7 +1499,7 @@ def join_asof(right: "StreamingDataFrame",
14821499
name: Optional[str] = None) -> "StreamingDataFrame"
14831500
```
14841501

1485-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1661)
1502+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1699)
14861503

14871504
Join the left dataframe with the records of the right dataframe with
14881505

@@ -1565,7 +1582,7 @@ def join_interval(
15651582
forward_ms: Union[int, timedelta] = 0) -> "StreamingDataFrame"
15661583
```
15671584

1568-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1737)
1585+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1775)
15691586

15701587
Join the left dataframe with records from the right dataframe that fall within
15711588

@@ -1668,7 +1685,7 @@ def join_lookup(
16681685
) -> "StreamingDataFrame"
16691686
```
16701687

1671-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1842)
1688+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1880)
16721689

16731690
Note: This is an experimental feature, and its API is likely to change in the future.
16741691

@@ -1729,7 +1746,7 @@ sdf = sdf.join_lookup(lookup, fields)
17291746
def register_store(store_type: Optional[StoreTypes] = None) -> None
17301747
```
17311748

1732-
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1931)
1749+
[[VIEW SOURCE]](https://github.com/quixio/quix-streams/blob/main/quixstreams/dataframe/dataframe.py#L1969)
17331750

17341751
Register the default store for the current stream_id in StateStoreManager.
17351752

0 commit comments

Comments
 (0)