Skip to content

Commit f6114f6

Browse files
committed
adding examples and documentation
1 parent 9c934a5 commit f6114f6

File tree

1 file changed

+40
-5
lines changed

1 file changed

+40
-5
lines changed

README.md

Lines changed: 40 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,41 @@
11
# kafka-connect-elasticsearch-source
2-
Kafka Connect Elasticsearch Source: fetch data from elasting indices using scroll API, fetch only new data using an incremental / temporal field, dynamic schema generation with support of nested objects/ arrays.
2+
Kafka Connect Elasticsearch Source: fetch data from elasting indices using scroll API. Fetch only new data using a strictly incremental / temporal field.
3+
It supports dynamic schema and nested objects/ arrays.
34

4-
## Elasticsearch Configuration
5+
## Installation:
6+
7+
Download (https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/raw/master/target/kafka-connect-elastic-source-connect-0.1.jar) the jar and put into the connect classpath (e.g ``/usr/share/java/kafka-connect-elasticsearch`` ) or set ``plugin.path`` parameter appropriately.
8+
9+
## Example
10+
Using kafka connect in distributed way, a sample config file to fetch ``metric*`` indices and to produce output topics with ``es_`` prefix:
11+
12+
13+
```json
14+
{ "name": "elastic-source",
15+
"config": {"connector.class":"com.github.dariobalinzo.ElasticSourceConnector",
16+
"tasks.max": "1",
17+
"es.host" : "localhost",
18+
"es.port" : "9200",
19+
"index.prefix" : "metric",
20+
"topic.prefix" : "es_",
21+
"incrementing.field.name" : "@timestamp"
22+
}
23+
}
24+
```
25+
To start the connector we send the json config with post:
26+
```bash
27+
curl -X POST -H "Content-Type: application/json" --data @config.json http://localhost:8083/connectors | jq
28+
```
29+
30+
To check the status:
31+
```bash
32+
curl localhost:8083/connectors/elastic-source/status | jq
33+
```
34+
35+
36+
## Documentation
37+
38+
### Elasticsearch Configuration
539

640
``es.host``
741
ElasticSearch host
@@ -46,14 +80,14 @@ Kafka Connect Elasticsearch Source: fetch data from elasting indices using scrol
4680
* Importance: low
4781

4882
``index.prefix``
49-
List of indices to include in copying.
83+
Indices prefix to include in copying.
5084

51-
* Type: list
85+
* Type: string
5286
* Default: ""
5387
* Importance: medium
5488

5589

56-
## Connector Configuration
90+
### Connector Configuration
5791

5892
``poll.interval.ms``
5993
Frequency in ms to poll for new data in each index.
@@ -74,3 +108,4 @@ Kafka Connect Elasticsearch Source: fetch data from elasting indices using scrol
74108

75109
* Type: string
76110
* Importance: high
111+

0 commit comments

Comments
 (0)