Skip to content

Commit 06ffb9b

Browse files
akshaisarmaakshaisarma
committed
Updating for Bullet Core. Updating arch image. Updating examples
1 parent fa4791b commit 06ffb9b

File tree

7 files changed

+86
-45
lines changed

7 files changed

+86
-45
lines changed

‎docs/about/contributing.md

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,16 @@ Bullet is hosted under the [Yahoo Github Organization](https://github.com/yahoo)
88

99
## Future plans
1010

11-
Here is a list of features we are currently considering/working on. If the Status column is empty, we are still discussing how to prioritize/approach/break them down. They will be updated as they are solidified. Feel free to [contact us](contact.md) with any ideas/suggestions/PRs!
11+
Here is a list of features we are currently considering/working on. Feel free to [contact us](contact.md) with any ideas/suggestions/PRs for features mentioned here or anything else you think about!
1212

13-
This list is neither comprehensive nor in any particular order.
13+
This list is neither comprehensive nor in any particular order and lists some high level directions.
1414

15-
| Feature | Components | Description | Status |
16-
|------------------- | ----------- | ------------------------- | ------------- |
17-
| Pub-Sub Queue | BE, WS, UI | WS and BE talk through the pub/sub. Bullet Storm uses Storm DRPC for this, which is strictly request-response. This will let us work on other Stream Processors and support incremental updates through WebSockets or SSEs | |
18-
| Incremental updates| BE, WS, UI | Push results back to users as soon as they arrive. Monoidal operations implies additive, so progressive results can be streamed back. Micro-batching and other features come into play | |
19-
| SQL API | BE, WS | WS supports an endpoint that converts a SQL-like query into Bullet queries | |
20-
| LocalForage | UI | Migration to LocalForage to distance ourselves from the relatively small LocalStorage space | [#9](https://github.com/yahoo/bullet-ui/issues/9) |
21-
| UI Packaging | UI | Github releases and building from source are the only two options. Docker or something similar may be more apt | |
15+
| Feature | Components | Description | Status |
16+
|-------------------- | ----------- | ------------------------- | ------------- |
17+
| Pub-Sub Queue | BE, WS, UI | WS and BE talk through the pub/sub. Bullet Storm uses Storm DRPC for this, which is request-response. Using a pub/sub queue will let us implement Bullet on other Stream Processors, support incremental updates through WebSockets and more! | In Progress |
18+
| Incremental updates | BE, WS, UI | Push results back to users as soon as they arrive. Monoidal operations implies additive, so progressive results can be streamed back. Micro-batching and other features come into play | In Progress |
19+
| Security | WS, UI | The obvious enterprise security for locking down access to the data and the instance of Bullet. Considering SSL, Kerberos, LDAP etc. | Planning |
20+
| Bullet on X | BE | With the pub/sub feature, Bullet can be implemented on other Stream Processors like Spark Streaming, Flink, Kafka Streaming, Samza etc | Open |
21+
| SQL API | BE, WS | WS supports an endpoint that converts a SQL-like query into Bullet queries | Open |
22+
| LocalForage | UI | Migration to LocalForage to distance ourselves from the relatively small LocalStorage space | [#9](https://github.com/yahoo/bullet-ui/issues/9) |
23+
| UI Packaging | UI | Github releases and building from source are the only two options. Docker or something similar may be more apt | Open |

‎docs/about/releases.md

Lines changed: 52 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,47 @@ This sections gathers all the relevant releases of the three components of Bulle
44

55
Bullet is still in active development. We welcome all contributions. Feel free to raise any issues/questions/bugs and whatever else on the relevant issues section for each component. Please include as many details as you can.
66

7+
## Bullet Core
8+
9+
The core Bullet logic (a library) that can be used to implement Bullet on different Stream Processors (like Flink, Storm, Kafka Streaming etc.). This core library can also be reused in other Bullet components that wish to depend on core Bullet concepts. This actually lived inside the [Bullet Storm](#bullet-storm) package prior to version [0.5.0](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.5.0). Starting with 0.5.0, Bullet Storm only includes the logic to implement Bullet on Storm.
10+
11+
| | |
12+
| ------------------------- | --------------- |
13+
| **Repository** | [https://github.com/yahoo/bullet-core](https://github.com/yahoo/bullet-core) |
14+
| **Issues** | [https://github.com/yahoo/bullet-core/issues](https://github.com/yahoo/bullet-core/issues) |
15+
| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-core/all.svg)](https://github.com/yahoo/bullet-core/tags) |
16+
| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-core/images/download.svg)](https://bintray.com/yahoo/maven/bullet-core/_latestVersion) |
17+
| **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-core) |
18+
19+
### Releases
20+
21+
| Date | Release | Highlights |
22+
| ------------ | --------------------------------------------------------------------------------- | ---------- |
23+
| 2016-06-27 | [**0.1.2**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.1.2) | Changes to the BulletConfig interface previously used in Bullet Storm. Users now use BulletStormConfig instead but YAML config is the same |
24+
| 2016-06-27 | [**0.1.1**](https://github.com/yahoo/bullet-core/releases/tag/bullet-core-0.1.1) | First stable release containing the core of Bullet as a library including parsing, implementing queries, creating results, DataSketches etc |
25+
26+
## Bullet Record
27+
28+
The AVRO container that you need to convert your data into to be consumed by Bullet.
29+
30+
| | |
31+
| -------------------------- | --------------- |
32+
| **Repository** | [https://github.com/yahoo/bullet-record](https://github.com/yahoo/bullet-record) |
33+
| **Issues** | [https://github.com/yahoo/bullet-record/issues](https://github.com/yahoo/bullet-record/issues) |
34+
| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-record/all.svg)](https://github.com/yahoo/bullet-record/tags) |
35+
| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-record/images/download.svg)](https://bintray.com/yahoo/maven/bullet-record/_latestVersion) |
36+
| **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-record) |
37+
38+
### Releases
39+
40+
| Date | Release | Highlights |
41+
| ------------ | ------------------------------------------------------------------------------------ | ---------- |
42+
| 2017-04-17 | [**0.1.1**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.0) | Helper methods to remove, rename, check presence and count fields in the Record |
43+
| 2017-02-09 | [**0.1.0**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.0) | Map constructor |
44+
745
## Bullet Storm
846

9-
The implementation of Bullet on Storm. Due to major API changes between Storm <= 0.10 and Storm 1.0, Bullet Storm [builds two artifacts](../backend/setup-storm.md#older-storm-versions). The ```artifactId``` changes from ```bullet-storm``` (for 1.0+) to ```bullet-storm-0.10```.
10-
All releases include migration and testing of the code on *both* versions. Both versions are built simultaneously. Feature parity depends on what was new in Storm 1.0. For example, the Resource Aware Scheduler or RAS, is only present in Storm 1.0+. So, bullet-storm-0.10 removes
11-
certain CPU and memory related settings specific to RAS in its configuration. There are also minor changes to the Metrics API in Storm. In terms of Bullet itself, there should be no differences.
47+
The implementation of Bullet on Storm. Due to major API changes between Storm <= 0.10 and Storm 1.0, Bullet Storm [builds two artifacts](../backend/setup-storm.md#older-storm-versions). The ```artifactId``` changes from ```bullet-storm``` (for 1.0+) to ```bullet-storm-0.10```. All releases include migration and testing of the code on *both* versions. Both versions are built simultaneously. Feature parity depends on what was new in Storm 1.0. For example, the Resource Aware Scheduler or RAS, is only present in Storm 1.0+. So, bullet-storm-0.10 removes certain CPU and memory related settings specific to RAS in its configuration. There are also minor changes to the Metrics API in Storm. In terms of Bullet itself, there should be no differences.
1248

1349
!!! note "Future support"
1450

@@ -21,11 +57,13 @@ certain CPU and memory related settings specific to RAS in its configuration. Th
2157
| **Issues** | [https://github.com/yahoo/bullet-storm/issues](https://github.com/yahoo/bullet-storm/issues) |
2258
| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-storm/all.svg)](https://github.com/yahoo/bullet-storm/tags) |
2359
| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-storm/images/download.svg)](https://bintray.com/yahoo/maven/bullet-storm/_latestVersion) |
60+
| **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-storm) |
2461

2562
### Releases
2663

2764
| Date | Storm 1.0 | Storm 0.10 | Highlights |
2865
| ------------ | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------- |
66+
| 2017-06-27 | [**0.5.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.5.0) | [**0.5.0**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.5.0) | Pulled out Bullet Core. BulletConfig to BulletStormConfig |
2967
| 2017-06-09 | [**0.4.3**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.3) | [**0.4.3**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.3) | Adding rounding for DISTRIBUTION. Latency metric |
3068
| 2017-04-28 | [**0.4.2**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.2) | [**0.4.2**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.2) | Strict JSON output and fix for no data distributions |
3169
| 2017-04-26 | [**0.4.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.1) | [**0.4.1**](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.10-0.4.1) | Result Metadata Concept name mismatch fix |
@@ -40,19 +78,24 @@ certain CPU and memory related settings specific to RAS in its configuration. Th
4078

4179
The Web Service implementation that can serve a static schema from a file and talk to the Storm backend.
4280

43-
| | |
44-
| ------------------- | --------------- |
45-
| **Repository** | [https://github.com/yahoo/bullet-service](https://github.com/yahoo/bullet-service) |
46-
| **Issues** | [https://github.com/yahoo/bullet-service/issues](https://github.com/yahoo/bullet-service/issues) |
47-
| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-service/all.svg)](https://github.com/yahoo/bullet-service/tags) |
48-
| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-service/images/download.svg)](https://bintray.com/yahoo/maven/bullet-service/_latestVersion) |
81+
| | |
82+
| -------------------------- | --------------- |
83+
| **Repository** | [https://github.com/yahoo/bullet-service](https://github.com/yahoo/bullet-service) |
84+
| **Issues** | [https://github.com/yahoo/bullet-service/issues](https://github.com/yahoo/bullet-service/issues) |
85+
| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-service/all.svg)](https://github.com/yahoo/bullet-service/tags) |
86+
| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-service/images/download.svg)](https://bintray.com/yahoo/maven/bullet-service/_latestVersion) |
87+
| **Package Manager Setup** | [Setup for Maven, Gradle etc](https://bintray.com/bintray/jcenter?filterByPkgName=bullet-service) |
4988

5089
### Releases
5190

5291
| Date | Release | Highlights |
5392
| ------------ | -------------------------------------------------------------------------------------- | ---------- |
5493
| 2016-12-16 | [**0.0.1**](https://github.com/yahoo/bullet-service/releases/tag/bullet-service-0.0.1) | The first release with support for DRPC and the file-based schema |
5594

95+
!!! note "Want to directly download jars?"
96+
97+
Head over to the JCenter download page to [directly download all Bullet Storm, Core, Service, Record artifacts](http://jcenter.bintray.com/com/yahoo/bullet/).
98+
5699
## Bullet UI
57100

58101
The Bullet UI that lets you build, run, save and visualize results from Bullet.
@@ -74,21 +117,3 @@ The Bullet UI that lets you build, run, save and visualize results from Bullet.
74117
| 2016-05-02 | [**0.2.1**](https://github.com/yahoo/bullet-ui/releases/tag/v0.2.1) | Fixes a bug with a dependency that broke sorting the Filters |
75118
| 2016-05-01 | [**0.2.0**](https://github.com/yahoo/bullet-ui/releases/tag/v0.2.0) | Release for Top K and Distribution. Supports Bullet Storm 0.4.2+ |
76119
| 2016-02-21 | [**0.1.0**](https://github.com/yahoo/bullet-ui/releases/tag/v0.1.0) | The first release with support for all features included in Bullet Storm 0.2.1+ |
77-
78-
## Bullet Record
79-
80-
The AVRO container that you need to convert your data into to be consumed by Bullet.
81-
82-
| | |
83-
| ------------------- | --------------- |
84-
| **Repository** | [https://github.com/yahoo/bullet-record](https://github.com/yahoo/bullet-record) |
85-
| **Issues** | [https://github.com/yahoo/bullet-record/issues](https://github.com/yahoo/bullet-record/issues) |
86-
| **Last Tag** | [![Latest tag](https://img.shields.io/github/release/yahoo/bullet-record/all.svg)](https://github.com/yahoo/bullet-record/tags) |
87-
| **Latest Artifact** | [![Download](https://api.bintray.com/packages/yahoo/maven/bullet-record/images/download.svg)](https://bintray.com/yahoo/maven/bullet-record/_latestVersion) |
88-
89-
### Releases
90-
91-
| Date | Release | Highlights |
92-
| ------------ | ------------------------------------------------------------------------------------ | ---------- |
93-
| 2017-04-17 | [**0.1.1**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.0) | Helper methods to remove, rename, check presence and count fields in the Record |
94-
| 2017-02-09 | [**0.1.0**](https://github.com/yahoo/bullet-record/releases/tag/bullet-record-0.1.0) | Map constructor |

‎docs/img/overallarch.png

-17.8 KB
Loading

‎docs/index.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,10 @@ Bullet is used in production internally at Yahoo by having it sit on a subset of
3030

3131
This instance of Bullet also powers other use-cases such as letting analysts validate assumptions about data, product managers verify launches instantly, debug issues and outages, or simply explore and play around with the data.
3232

33+
!!! note "Blog post"
34+
35+
Here is a [link to our blog post](https://yahooeng.tumblr.com/post/161855616651/open-sourcing-bullet-yahoos-forward-looking) condensing most of this information if you want to take a look.
36+
3337
---
3438

3539
# Quick Start
@@ -145,7 +149,7 @@ The Web Service can be deployed with your favorite servlet container like [Jetty
145149

146150
In the case of Bullet on Storm, the Web Service and UI talk to the backend using [Storm DRPC](http://storm.apache.org/releases/1.0.0/Distributed-RPC.html).
147151

148-
## End-to-End Architecture
152+
## End-to-End Architecture on Storm
149153

150154
![Overall Storm Architecture](img/overallarch.png)
151155

‎docs/quick-start.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ This section gets you running a mock instance of Bullet to play around with. The
44

55
At the end of this section, you will have:
66

7-
* Setup the Bullet topology using a custom spout on [bullet-storm-0.4.3](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.4.3)
7+
* Setup the Bullet topology using a custom spout on [bullet-storm-0.5.0](https://github.com/yahoo/bullet-storm/releases/tag/bullet-storm-0.5.0)
88
* Setup the [Web Service](ws/setup.md) talking to the topology and serving a schema for your UI using [bullet-service-0.0.1](https://github.com/yahoo/bullet-service/releases/tag/bullet-service-0.0.1)
9-
* Setup the [UI](ui/setup.md) talking to the Web Service using [bullet-ui-0.3.1](https://github.com/yahoo/bullet-ui/releases/tag/v0.3.1)
9+
* Setup the [UI](ui/setup.md) talking to the Web Service using [bullet-ui-0.3.2](https://github.com/yahoo/bullet-ui/releases/tag/v0.3.2)
1010

1111
**Prerequisites**
1212

@@ -19,7 +19,7 @@ At the end of this section, you will have:
1919
Simply run:
2020

2121
```bash
22-
curl -sLo- https://raw.githubusercontent.com/yahoo/bullet-docs/v0.3.2/examples/install-all.sh | bash
22+
curl -sLo- https://raw.githubusercontent.com/yahoo/bullet-docs/v0.3.3/examples/install-all.sh | bash
2323
```
2424

2525
This will setup a local Storm cluster, a Bullet running on it, the Bullet Web Service and a Bullet UI for you. Once everything has launched, you should be able to go to the Bullet UI running locally at [http://localhost:8800](http://localhost:8800). You can then [**continue this guide from here**](#what-did-we-do).
@@ -114,7 +114,7 @@ cp $BULLET_EXAMPLES/storm/* $BULLET_HOME/backend/storm
114114

115115
!!! note "Settings"
116116

117-
Take a look at bullet_settings.yaml for the settings that are being overridden for this example. You can add or change settings as you like by referring to [bullet_defaults.yaml](https://github.com/yahoo/bullet-storm/blob/master/src/main/resources/bullet_defaults.yaml). In particular, we have [customized these settings](https://github.com/yahoo/bullet-docs/blob/master/examples/storm/src/main/resources/bullet_settings.yaml) that affect the Bullet queries you can run:
117+
Take a look at bullet_settings.yaml for the settings that are being overridden for this example. You can add or change settings as you like by referring to [core Bullet settings in bullet_defaults.yaml](https://github.com/yahoo/bullet-core/blob/master/src/main/resources/bullet_defaults.yaml) and [Storm settings in bullet_storm_defaults.yaml](https://github.com/yahoo/bullet-storm/blob/master/src/main/resources/bullet_storm_defaults.yaml). In particular, we have [customized these settings](https://github.com/yahoo/bullet-docs/blob/master/examples/storm/src/main/resources/bullet_settings.yaml) that affect the Bullet queries you can run:
118118

119119
```bullet.query.max.duration: 570000``` Longest query time can be 570s. The Storm cluster default DRPC timeout is 600s.
120120

0 commit comments

Comments
 (0)