Updating docs (#27)

akshaisarma · web-flow · commit 84fe89b9716b · 2018-09-06T18:00:46.000-07:00
diff --git a/docs/about/contributing.md b/docs/about/contributing.md
@@ -8,17 +8,4 @@ Bullet is hosted under the [Bullet Github Organization](https://github.com/bulle
 
 ## Future plans
 
-Here is a selected list of features we are currently considering/working on. Feel free to [contact us](contact.md) with any ideas/suggestions/PRs for features mentioned here or anything else you think about!
-
-This list is neither comprehensive nor in any particular order.
-
-| Feature             | Components  | Description               | Status        |
-|-------------------- | ----------- | ------------------------- | ------------- |
-| Bullet on X         | BE          | With the pub/sub feature, Bullet can be implemented on other Stream Processors like Flink, Kafka Streaming, Samza etc | Open |
-| SQL API             | BE, WS      | WS supports an endpoint that converts a SQL-like query into Bullet queries | In Progress |
-| More Windows        | BE          | We have implemented a few of the windows we wanted to support initially but there are still more we can add | Open |
-| More Aggregations   | BE, UI      | We can add more aggregations like Group By Count Distinct etc | Open |
-| Post Aggregations   | BE, UI      | Post aggregations once the aggregations are done is useful | Open |
-| Bullet on Beam      | BE          | Bullet can be implemented on [Apache Beam](https://beam.apache.org) as an alternative to implementing it on various Stream Processors | Open |
-| Security            | WS, UI      | The obvious enterprise security for locking down access to the data and the instance of Bullet. Considering SSL, Kerberos, LDAP etc. Ideally, without a database | Planning |
-| Packaging           | UI, BE, WS  | Github releases and building from source are the only two options for the UI. Docker images or the like for quick setup and to mix and match various pluggable components would be really useful | Open |
+Our issues are tracked through GitHub on the appropriate repo issues pages. You are welcome to take a look and contribute. Also, feel free to [contact us](contact.md) with any ideas/suggestions/PRs for features mentioned here or anything else you think about!
diff --git a/docs/apidocs/bullet-bql/0.1.2/index.html b/docs/apidocs/bullet-bql/0.1.2/index.html
@@ -0,0 +1 @@
+Replace me with the real documentation.
diff --git a/docs/apidocs/bullet-core/0.4.3/index.html b/docs/apidocs/bullet-core/0.4.3/index.html
@@ -0,0 +1 @@
+Replace me with the real documentation.
diff --git a/docs/apidocs/bullet-record/0.2.1/index.html b/docs/apidocs/bullet-record/0.2.1/index.html
@@ -0,0 +1 @@
+Replace me with the real documentation.
diff --git a/docs/apidocs/bullet-service/0.4.1/index.html b/docs/apidocs/bullet-service/0.4.1/index.html
@@ -0,0 +1 @@
+Replace me with the real documentation.
diff --git a/docs/releases.md b/docs/releases.md
@@ -4,6 +4,10 @@ This sections gathers all the relevant releases of the components of Bullet that
 
 Bullet is still in active development. We welcome all contributions. Feel free to raise any issues/questions/bugs and whatever else on the relevant issues section for each component. Please include as many details as you can.
 
+## API Documentation
+
+API (Java and Scala) docs can also be found for the releases below.
+
 ## Download
 
 For downloading any artifact listed below manually, you should preferably use the [**JCenter mirror here**](https://jcenter.bintray.com/com/yahoo/bullet/). For resolving artifacts in your build tool, follow the directions in each of the components' Package Manager Setup sections.
@@ -26,6 +30,7 @@ The core Bullet logic (a library) that can be used to implement Bullet on differ
 
 |    Date      |                                        Release                                        | Highlights | APIDocs |
 | ------------ | ------------------------------------------------------------------------------------- | ---------- | ------- |
+| 2018-09-05   | [**0.4.3**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.4.3)  | Sliding Windows, SIZEIS, CONTAINSKEY, CONTAINSVALUE, filtering against other fields | [JavaDocs](apidocs/bullet-core/0.4.3/index.html) |
 | 2018-06-26   | [**0.4.2**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.4.2)  | Fixes a bug with unclosed connections in the RESTPubSub | [JavaDocs](apidocs/bullet-core/0.4.2/index.html) |
 | 2018-06-22   | [**0.4.1**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.4.1)  | Added RESTPublisher HTTP Timeout Setting | |
 | 2018-06-18   | [**0.4.0**](https://github.com/bullet-db/bullet-core/releases/tag/bullet-core-0.4.0)  | Added support for Integer and Float data types, and configurable BulletRecordProvider class used to instantiate BulletRecords in bullet-core | |
@@ -66,8 +71,8 @@ The implementation of Bullet on Storm. Due to major API changes between Storm <=
 
 ### Releases
 
-|    Date      |                               Storm 1.0                                            |                                      Storm 0.10                                         | Highlights | APIDocs |
-| ------------ | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | ---------- | ------- |
+|    Date      |                            Storm 1.0                                        |                                 Storm 0.10                                  | Highlights | APIDocs |
+| ------------ | --------------------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------- | ------- |
 | 2018-06-18   | [**0.8.3**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.8.3) | [**0.8.3**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.8.3) | Using new bullet-record and bullet-core supporting Integer and Float data types | [JavaDocs](apidocs/bullet-storm/0.8.3/index.html) |
 | 2018-04-12   | [**0.8.2**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.8.2) | [**0.8.2**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.8.2) | Delaying query start in Join Bolt | |
 | 2018-04-04   | [**0.8.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.8.1) | [**0.8.1**](https://github.com/bullet-db/bullet-storm/releases/tag/bullet-storm-0.10-0.8.1) | Fixed bug in JoinBolt | |
@@ -127,6 +132,7 @@ The Web Service implementation that can serve a static schema from a file and ta
 
 |    Date      |                                      Release                                           | Highlights | APIDocs |
 | ------------ | -------------------------------------------------------------------------------------- | ---------- | ------- |
+| 2018-09-06   | [**0.4.1**](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.4.1) | Max Queries limit and bullet-bql 0.1.2 | [JavaDocs](apidocs/bullet-service/0.4.1/index.html) |
 | 2018-07-17   | [**0.4.0**](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.4.0) | Enhanced Web Service to support BQL queries | [JavaDocs](apidocs/bullet-service/0.4.0/index.html) |
 | 2018-06-25   | [**0.3.0**](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.3.0) | Upgrades to Netty-less Bullet Core for the RESTPubsub | |
 | 2018-06-14   | [**0.2.2**](https://github.com/bullet-db/bullet-service/releases/tag/bullet-service-0.2.2) | Adding settings to configure Websocket | |
@@ -181,6 +187,7 @@ The AVRO container that you need to convert your data into to be consumed by Bul
 
 |    Date      |                                  Release                                             | Highlights | APIDocs |
 | ------------ | ------------------------------------------------------------------------------------ | ---------- | ------- |
+| 2018-08-14   | [**0.2.1**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.2.1) | Supports List of Primitive types | [JavaDocs](apidocs/bullet-record/0.2.1/index.html) |
 | 2018-06-14   | [**0.2.0**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.2.0) | Makes BulletRecord pluggable, adds simple record and avro record implementations | [JavaDocs](apidocs/bullet-record/0.2.0/index.html) |
 | 2017-05-19   | [**0.1.2**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.1.2) | Reduces the memory footprint needed to serialize itself by a factor of 128 for small records | |
 | 2017-04-17   | [**0.1.1**](https://github.com/bullet-db/bullet-record/releases/tag/bullet-record-0.1.1) | Helper methods to remove, rename, check presence and count fields in the Record | |
@@ -223,5 +230,6 @@ A library facilitating the conversion from Bullet BQL queries to Bullet JSON que
 
 |    Date      |                                  Release                                             | Highlights | APIDocs |
 | ------------ | ------------------------------------------------------------------------------------ | ---------- | ------- |
+| 2018-09-06   | [**0.1.2**](https://github.com/bullet-db/bullet-bql/releases/tag/bullet-bql-0.1.2) | Supports CONTAINSKEY, CONTAINSVALUE, SIZEOF, comparing to other fields. Fixes some bugs | [JavaDocs](apidocs/bullet-bql/0.1.2/index.html) |
 | 2018-07-17   | [**0.1.1**](https://github.com/bullet-db/bullet-bql/releases/tag/bullet-bql-0.1.1) | Stops publishing fat jar and marks slf4j dependency provided | [JavaDocs](apidocs/bullet-bql/0.1.1/index.html) |
 | 2018-07-05   | [**0.1.0**](https://github.com/bullet-db/bullet-bql/releases/tag/bullet-bql-0.1.0) | First release | |
diff --git a/docs/ws/api-bql.md b/docs/ws/api-bql.md
@@ -4,14 +4,11 @@ This section gives a comprehensive overview of the Web Service API for launching
 
 For examples of BQL queries, see the [examples page](examples.md).
 
-BQL queries that are received by the Web Service will be detenced and automatically converted to
-[the JSON format](api-json.md) before being sent to the backend (which requires the basic JSON format). This converstion
-is done in the web service using [the bullet-bql library](../releases/#bullet-bql).
+BQL queries that are received by the Web Service will be detenced and automatically converted to [the JSON format](api-json.md) before being sent to the backend (which requires the basic JSON format). This conversion is done in the web service using [the bullet-bql library](../releases/#bullet-bql).
 
 ## Overview
 
-Bullet-BQL provides users with a friendly SQL-like API to submit queries to the Web Service instead of using the more
-cumbersome [JSON API](api-json.md).
+Bullet-BQL provides users with a friendly SQL-like API to submit queries to the Web Service instead of using the more cumbersome [JSON API](api-json.md).
 
 ## Statement Syntax
 
@@ -34,7 +31,9 @@ where `select_clause` is one of
     TOP ( ( Integer | Long ) ( , Integer | Long ) )? , reference_expr ( , reference_expr )? ) ( AS? ColumnReference )?
 
 
-`group_function` is one of `SUM(reference_expr)`, `MIN(reference_expr)`, `MAX(reference_expr)`, `AVG(reference_expr)` and `COUNT(*)`. `reference_expr` is one of ColumnReference and Dereference. `distribution_type` is one of `QUANTILE`, `FREQ` and `CUMFREQ`. The 1st number in `TOP` is K, and the 2nd number is an optional threshold.  The `input_mode` is one of
+`reference_expr` is one of `ColumnReference` or `Dereference`.
+
+and `group_function` is one of `SUM(reference_expr)`, `MIN(reference_expr)`, `MAX(reference_expr)`, `AVG(reference_expr)` and `COUNT(*)`. `reference_expr` is one of ColumnReference and Dereference. `distribution_type` is one of `QUANTILE`, `FREQ` and `CUMFREQ`. The 1st number in `TOP` is K, and the 2nd number is an optional threshold.  The `input_mode` is one of
 
     LINEAR, ( Integer | Long )                                              evenly spaced
     REGION, ( Integer | Long ), ( Integer | Long ), ( Integer | Long )      evenly spaced in a region
@@ -59,9 +58,14 @@ and `where_clause` is one of
     reference_expr NOT? BETWEEN value_expr AND value_expr
     reference_expr NOT? IN ( value_expr ( , value_expr )* )
     reference_expr NOT? LIKE ( value_expr ( , value_expr )* )
+    reference_expr NOT? CONTAINSKEY ( value_expr ( , value_expr )* )
+    reference_expr NOT? CONTAINSVALUE ( value_expr ( , value_expr )* )
     reference_expr ( = | <> | != | < | > | <= | >= ) value_expr
+    SIZEOF(reference_expr) ( = | <> | != ) value_expr
+    SIZEOF(reference_expr) NOT? IN ( value_expr ( , value_expr )* )
+    SIZEOF(reference_expr) NOT? DISTINCT FROM value_expr
 
-`value_expr` is one of Null, Boolean, Integer, Long, Double, Decimal and String.
+`value_expr` is one of Null, Boolean, Integer, Long, Double, Decimal, String or `reference_expr`.
 
 and `groupBy_clause` is one of
 
@@ -109,16 +113,14 @@ and `limit_clause` is one of
 
 * **String**: character string which can have escapes. Example: `'this is a string'`, `'this is ''another'' string'`.
 
-* **ColumnReference**: representation of a single column. Unquoted ColumnReference must start with a letter or `_`. Quoted ColumnReference can have escape. Example: `column_name`, `"#column""with""escape"`.
+* **ColumnReference**: representation of a single column. Unquoted ColumnReference must start with a letter or `_`. Example: `column_name`.
 
 * **Dereference**: representation of a column field. Example: `column_name.field_name`.
 
 * **All**: representation of all columns. Example: `*`. `column_name.*` is interpreted as `column_name`.
 
 ## Reserved Keywords
 
-Reserved keywords must be double quoted in order to be used as ColumnReference or Dereference.
-
 |      Keyword          |    SQL:2016     |   SQL-92      |
 | --------------------- | :-------------: | :-----------: |
 | `ALTER`               |     reserved    |   reserved    |
diff --git a/docs/ws/api-json.md b/docs/ws/api-json.md
@@ -21,7 +21,7 @@ Fields inside maps can be accessed using the '.' notation in queries. For exampl
 
 `myMap.key`
 
-will access the "key" field inside the "myMap" map. There is no support for accessing fields inside Lists or inside nested Maps as of yet. Only the entire object can be operated on for now.
+will access the "key" field inside the "myMap" map. There is no support for accessing fields inside Lists yet. Only the entire object can be operated on for now.
 
 The main constituents of a Bullet query listed above create the top level fields of the Bullet query:
 ```javascript
@@ -75,19 +75,22 @@ Note that the "filter" field in the query is a __list__ of as many filters as yo
 
 ### Relational Filters
 
-Relational filters allow you to specify conditions on a field, using a comparison operator and a list of values.
+Relational filters allow you to specify conditions on a field, using a comparison operator and a list of constant values or other fields.
 
 The current comparisons allowed in filters are:
 
-| Comparison | Meaning |
-| ---------- | ------- |
-| ==         | Equal to any value in values |
-| !=         | Not equal to any value in values |
-| <=         | Less than or equal to any value in values |
-| >=         | Greater than or equal to any value in values |
-| <          | Less than any value in values |
-| >          | Greater than any value in values |
-| RLIKE      | Matches using [Java Regex notation](http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html), any Regex value in values |
+| Comparison    | Meaning |
+| ------------- | ------- |
+| ==            | Equal to any value in values |
+| !=            | Not equal to any value in values |
+| <=            | Less than or equal to any value in values |
+| >=            | Greater than or equal to any value in values |
+| <             | Less than any value in values |
+| >             | Greater than any value in values |
+| RLIKE         | Matches using [Java Regex notation](http://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html), any Regex value in values |
+| SIZEIS        | If a map, list, or string has a given size |
+| CONTAINSKEY   | If a map field contains the given key. This map field can be a top level map field or a map field inside a list of maps |
+| CONTAINSVALUE | If a map field or a list field contains the given value. If the list contains maps instead of primitives, the values in the maps are used |
 
 Note: These operators are all typed based on the type of the __left hand side__ from the Bullet record. If the elements on the right hand side cannot be
 casted to the types on the LHS, those items will be ignored for the comparison.
@@ -96,7 +99,20 @@ The format for a Relational filter is:
 
 ```javascript
 {
-    "operation": "== | != | <= | >= | < | > | RLIKE"
+    "operation": "== | != | <= | >= | < | > | RLIKE | SIZEIS | CONTAINSKEY | CONTAINSVALUE"
+    "field": "record_field_name | map_field.subfield",
+    "values": [
+        { "kind": "VALUE", "value": "foo"},
+        { "kind": "FIELD", "value": "another_record_field_name"}
+    ]
+}
+```
+
+Note that you may specify ```VALUE``` or ```KIND``` currently for the ```kind``` key in the entries in the ```values``` field above, denoting the type of value this is. As a shortcut, you can also specify the following format for ```VALUE``` kind.
+
+```javascript
+{
+    "operation": "== | != | <= | >= | < | > | RLIKE | SIZEIS | CONTAINSKEY | CONTAINSVALUE"
     "field": "record_field_name | map_field.subfield",
     "values": [
         "string values",
@@ -107,6 +123,9 @@ The format for a Relational filter is:
     ]
 }
 ```
+
+You may __not__ mix and match both styles in the same filter.
+
 *Multiple top level relational filters behave as if they are ANDed together.* This is supported as a convenience to do a bunch of ```AND```ed relational filters without having to nest them in a logical clause.
 
 ## Projections
@@ -286,7 +305,7 @@ Note that the ```K``` in ```TOP K``` is specified using the ```size``` field in
 
 ## Window
 
-The "window" field is **optional** and allows you to instruct Bullet to return incremental results. For example you might want to return the COUNT of a field and return that count every 2 seconds. 
+The "window" field is **optional** and allows you to instruct Bullet to return incremental results. For example you might want to return the COUNT of a field and return that count every 2 seconds.
 
 If "window" is omitted Bullet will emit only a single result at the very end of the query.
 
@@ -351,15 +370,13 @@ The above example would be specified with the window:
 
 In this example the first window would include 3 records, the second would include 7 records, the third would include 10 records and the fourth would include 12 records.
 
-#### **Sliding "Reactive" Windows**
+#### **Sliding Windows**
 
-Sliding windows emit based on the arrival of an event, rather than after a certain period of time. In general sliding windows often do some aggregation on the previous X records, or on all records that arrived in the last X seconds.
-Bullet will support this functionality in the future, at this time Bullet only supports **Sliding Windows of size 1**, often referred to as "reactive" windows. It does not support sliding windows with an aggregation at this time.
-Effectively this query will simply return every event that matches the filters instantly to the user.
+Sliding windows emit based on the arrival of an event, rather than after a certain period of time. In general sliding windows often do some aggregation on the previous X records, or on all records that arrived in the last X seconds. At this time, Bullet only supports sliding windows on the previous X records. It does not support sliding windows with any aggregation but ```RAW``` at this time. If you set X to 1, the query will effectively return each record as they arrive to the user.
 
 ![Reactive Windows](../img/reactive.png)
 
-The above example would be specified with the window:
+The above example would be specified with the window (you may replace every and last with higher values):
 
 ```javascript
 "window": { "emit": { "type": "RECORD", "every": 1 },
diff --git a/docs/ws/examples.md b/docs/ws/examples.md

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1 @@`
	`1`	`+Replace me with the real documentation.`