Merge pull request #5 from paulgc/master

Project import generated by Copybara.

Author: paulgc17

README.md

@@ -1,4 +1,6 @@
-# TensorFlow Data Validation
+<!-- See: www.tensorflow.org/tfx/data_validation/ -->
+
+# TensorFlow Data Validation [![PyPI](https://img.shields.io/pypi/pyversions/tensorflow-data-validation.svg?style=plastic)](https://github.com/tensorflow/data-validation)
 
 *TensorFlow Data Validation* is a library for exploring and validating
 machine learning data. `tf.DataValidation` is designed to be highly scalable
@@ -18,18 +20,14 @@ TF Data Validation includes:
 *    An anomalies viewer so that you can see what features have anomalies and
      learn more in order to correct them.
 
-See the [examples](https://github.com/tensorflow/data-validation/tree/master/examples)
-to learn how to use TF Data Validation and the [quick guide to get started](https://github.com/tensorflow/data-validation/tree/master/g3doc/get_started.md).
+For instructions on using TF Data Validation, see the [get started guide](g3doc/get_started.md).
 
 Caution: `tf.DataValidation` may be backwards incompatible before version 1.0.
 
 ## Installing from PyPI
 
-Note: PyPI package is not available yet.
-
-The `tensorflow-data-validation`
-[PyPI package](https://pypi.org/project/tensorflow-data-validation/) is the
-recommended way to install `tf.DataValidation`:
+The recommended way to install TensorFlow Data Validation is using the
+[PyPI package](https://pypi.org/project/tensorflow-data-validation/):
 
 ```bash
 pip install tensorflow-data-validation
@@ -43,7 +41,7 @@ To compile and use TensorFlow Data Validation, you need to set up some prerequis
 
 #### Install NumPy
 
-If NumPy is not installed on your system, install it now following [these
+If NumPy is not installed on your system, install it now by following [these
 directions](https://www.scipy.org/scipylib/download.html).
 
 #### Install Bazel
@@ -62,39 +60,18 @@ Note that these instructions will install the latest master branch of TensorFlow
 Data Validation. If you want to install a specific branch (such as a release branch),
 pass `-b <branchname>` to the `git clone` command.
 
-### 3. Build
-
-TensorFlow Data Validation uses Bazel to build. Use Bazel commands to build individual
-targets or the entire source tree.
-
-To build the Python wrappers for the C++ modules, execute:
-
-```shell
-bazel build -c opt tensorflow_data_validation/anomalies:pywrap_tensorflow_data_validation
-```
-
-### 4. Copy over generated Python wrappers
-
-```shell
-cp bazel-bin/tensorflow_data_validation/anomalies/_pywrap_tensorflow_data_validation.so tensorflow_data_validation/anomalies/
-cp bazel-bin/tensorflow_data_validation/anomalies/pywrap_tensorflow_data_validation.py tensorflow_data_validation/anomalies/
-```
-
-### 5. Build the pip package
+### 3. Build and install pip package
 
-Run the following command to create a `.whl` file within `dist` directory.
+TensorFlow Data Validation uses Bazel to build and install the pip package from source:
 
 ```shell
-python setup.py bdist_wheel
+bazel run -c opt tensorflow_data_validation:pip_installer
 ```
 
-### 6. Install the pip package
-
-Invoke `pip install` to install that pip package. The filename of the `.whl` file depends on your platform.
-
-```shell
-pip install dist/*.whl
-```
+Note that the previous command performs a `pip install` in the current python
+environment. You can find the installed `.whl` file in the `dist`
+subdirectory. It is also possible to pass options to the executed `pip install`
+through the environment variable `TFDV_PIP_INSTALL_OPTIONS`.
 
 ## Dependencies
 
@@ -114,6 +91,14 @@ The following table shows the `tf.DataValidation` package versions that are
 compatible with each other. This is determined by our testing framework, but
 other *untested* combinations may also work.
 
-|tensorflow-data-validation                                                            |tensorflow    |apache-beam[gcp]|
-|--------------------------------------------------------------------------------------|--------------|----------------|
-|GitHub master |nightly (1.x) |2.5.0           |
+|tensorflow-data-validation |tensorflow    |apache-beam[gcp]|
+|---------------------------|--------------|----------------|
+|GitHub master              |nightly (1.x) |2.6.0           |
+|0.9.0                      |1.9           |2.6.0           |
+
+## Questions
+
+Please direct any questions about working with TF Data Validation to
+[Stack Overflow](https://stackoverflow.com) using the
+[tensorflow-data-validation](https://stackoverflow.com/questions/tagged/tensorflow-data-validation)
+tag.

RELEASE.md

@@ -0,0 +1,2 @@
+# Release 0.9.0
+ * Initial release of TensorFlow Data Validation.

WORKSPACE

@@ -11,8 +11,8 @@ load("//tensorflow_data_validation:repo.bzl", "tensorflow_http_archive")
 
 tensorflow_http_archive(
     name = "org_tensorflow",
-    sha256 = "31eddae9f4e08e20639b577acadd03cc7ffc177c8178a597196163678ba16bf6",
-    git_commit = "e6fb3baeed99543b3ffd22522c30b08c94835e15",
+    sha256 = "696c4906d6536ed8d9f8f13c4927d3ccf36dcf3e13bb352ab80cba6b1b9038d4",
+    git_commit = "25c197e02393bd44f50079945409009dd4d434f8",
 )
 
 # TensorFlow depends on "io_bazel_rules_closure" so we need this here.

examples/chicago_taxi/chicago_taxi_tfdv.ipynb

@@ -4588,7 +4588,7 @@ pickup_community_area,fare,trip_start_month,trip_start_hour,trip_start_day,trip_
 8,14.85,12,18,7,1417889700,41.899155613,-87.626210532,41.87866742,-87.671653621,3.5,,17031838100,Credit Card,,1680,28,3.0
 8,8.75,5,23,7,1463874300,41.899155613,-87.626210532,41.904935302,-87.649907226,1.8,,17031842200,Cash,,720,8,0.0
 56,27.05,9,20,3,1378241100,41.785998518,-87.750934289,,,0.7,,,Cash,Taxi Affiliation Services,1380,,0.0
-56,0.0,4,11,2,1398078000,41.785998518,-87.750934289,,,0.0,,,Unknown,Taxi Affiliation Services,,,0.0
+56,0.0,4,11,2,1398078000,41.785998518,-87.750934289,,,0.0,,,Unknown,Taxi Affiliation Services,700,,0.0
 56,60.25,4,13,4,1366206300,41.785998518,-87.750934289,,,0.0,,,Cash,Northwest Management LLC,4560,,0.0
 56,54.25,4,11,4,1398253500,41.785998518,-87.750934289,,,27.3,,,Credit Card,Taxi Affiliation Services,2100,,16.65
 56,31.0,7,10,6,1469182500,41.785998518,-87.750934289,,,11.3,,,Credit Card,KOAM Taxi Association,1560,,10.2

examples/chicago_taxi/data/eval.csv

@@ -0,0 +1,101 @@
+pickup_community_area,fare,trip_start_month,trip_start_hour,trip_start_day,trip_start_timestamp,pickup_latitude,pickup_longitude,dropoff_latitude,dropoff_longitude,trip_miles,pickup_census_tract,dropoff_census_tract,payment_type,company,trip_seconds,dropoff_community_area
+8,6.45,9,17,4,1441213200,41.892072635,-87.628874157,41.880994471,-87.632746489,0.9,,17031839100,Credit Card,Taxi Affiliation Services,540,32
+8,8.05,10,20,7,1414269000,41.899155613,-87.626210532,41.880994471,-87.632746489,0.0,,17031839100,Cash,Taxi Affiliation Services,840,32
+32,7.65,3,20,2,1395087300,41.880994471,-87.632746489,41.877406123,-87.621971652,1.9,,17031320400,Cash,,480,32
+8,36.05,6,17,7,1370713500,41.890922026,-87.618868355,41.97907082,-87.903039661,17.4,,17031980000,Cash,Dispatch Taxi Affiliation,2280,76
+32,4.45,4,12,3,1398775500,41.880994471,-87.632746489,41.880994471,-87.632746489,0.3,,17031839100,Cash,,240,32
+8,36.45,10,8,6,1413532800,41.892507781,-87.626214906,41.97907082,-87.903039661,17.5,,17031980000,Credit Card,,2040,76
+76,37.05,3,23,4,1394667000,41.97907082,-87.903039661,41.898331794,-87.620762865,0.0,,17031081300,Cash,Choice Taxi Association,1680,8
+28,6.25,10,9,5,1380791700,41.879255084,-87.642648998,41.880994471,-87.632746489,0.9,,17031839100,Cash,,480,32
+8,8.44,10,8,1,1382862600,41.899602111,-87.633308037,41.944226601,-87.655998182,2.8,,,Cash,,360,6
+32,8.85,7,19,5,1406834100,41.884987192,-87.620992913,41.851017824,-87.635091856,0.0,,17031841100,Cash,Taxi Affiliation Services,720,34
+28,7.25,11,20,4,1448481600,41.874005383,-87.66351755,41.874005383,-87.66351755,1.9,,,Cash,Taxi Affiliation Services,480,28
+8,10.5,4,1,7,1461979800,41.90749193,-87.63576009,41.884987192,-87.620992913,2.7,,17031320100,Credit Card,,720,32
+32,3.65,8,10,1,1407060900,41.880994471,-87.632746489,41.880994471,-87.632746489,0.0,,17031839100,Cash,,60,32
+32,12.5,10,19,4,1477511100,41.880994471,-87.632746489,41.899155613,-87.626210532,1.8,,17031081201,Credit Card,Dispatch Taxi Affiliation,1440,8
+8,10.45,5,18,2,1399919400,41.892042136,-87.63186395,41.884987192,-87.620992913,0.1,,17031320100,Cash,Taxi Affiliation Services,960,32
+28,4.65,2,16,3,1360686600,41.879255084,-87.642648998,,,0.65,,,Cash,,240,
+76,40.05,11,17,5,1384451100,41.97907082,-87.903039661,41.880994471,-87.632746489,1.0,,17031839100,Cash,Taxi Affiliation Services,4440,32
+28,3.25,11,12,6,1383308100,41.88528132,-87.6572332,41.88528132,-87.6572332,0.0,,17031833000,Cash,,0,28
+8,7.05,12,15,6,1417793400,41.900221297,-87.629105186,41.892072635,-87.628874157,1.0,,17031081600,Cash,Dispatch Taxi Affiliation,660,8
+32,5.05,1,19,3,1421781300,41.880994471,-87.632746489,41.879255084,-87.642648998,0.0,,17031281900,Credit Card,Blue Ribbon Taxi Association Inc.,300,28
+8,21.45,9,20,5,1441310400,41.899602111,-87.633308037,41.954027649,-87.763399032,0.5,,,Credit Card,Taxi Affiliation Services,1080,15
+3,7.85,5,22,7,1433023200,41.972437081,-87.671109526,41.946294536,-87.654298084,1.9,,17031832000,Cash,Dispatch Taxi Affiliation,600,6
+32,24.85,9,11,6,1443181500,41.880994471,-87.632746489,41.785998518,-87.750934289,11.0,,17031980100,Credit Card,,1440,56
+32,7.25,11,1,1,1446339600,41.877406123,-87.621971652,41.89321636,-87.63784421,0.0,,17031081800,Credit Card,Taxi Affiliation Services,540,8
+32,6.65,11,18,2,1383590700,41.880994471,-87.632746489,41.899155613,-87.626210532,1.1,,17031081201,Credit Card,Northwest Management LLC,540,8
+34,15.05,1,22,2,1420495200,41.842076117,-87.633973422,41.775928827,-87.666596265,6.1,,,Unknown,Taxi Affiliation Services,1200,67
+32,36.65,10,14,6,1445610600,41.880994471,-87.632746489,41.97907082,-87.903039661,17.3,,17031980000,Credit Card,Taxi Affiliation Services,2640,76
+8,5.45,3,19,1,1364758200,41.899155613,-87.626210532,41.907412816,-87.640901525,0.5,,17031080400,Credit Card,Dispatch Taxi Affiliation,300,8
+28,35.25,1,16,3,1420560900,41.885300022,-87.642808466,41.97907082,-87.903039661,17.1,,17031980000,Credit Card,Taxi Affiliation Services,1800,76
+8,5.65,5,20,6,1399060800,41.892507781,-87.626214906,41.880994471,-87.632746489,0.4,,17031839100,Cash,Dispatch Taxi Affiliation,480,32
+24,8.05,6,21,4,1433972700,41.901206994,-87.676355989,41.878865584,-87.625192142,2.13,,,Credit Card,,480,32
+28,5.65,1,11,3,1358249400,41.879255084,-87.642648998,41.877406123,-87.621971652,0.0,,17031320400,Cash,Dispatch Taxi Affiliation,360,32
+33,9.05,8,11,6,1438947900,41.859349715,-87.617358006,41.892042136,-87.63186395,2.3,,17031081700,Credit Card,,660,8
+32,7.25,10,15,5,1446131700,41.880994471,-87.632746489,41.89503345,-87.619710672,1.4,,17031081401,Cash,Choice Taxi Association,540,8
+6,6.25,10,2,2,1413167400,41.944226601,-87.655998182,41.922686284,-87.649488729,0.0,,,Cash,Taxi Affiliation Services,360,7
+8,7.0,2,9,1,1454835600,41.899602111,-87.633308037,41.899602111,-87.633308037,0.0,,,Cash,Choice Taxi Association,360,8
+28,4.65,5,19,4,1401303600,41.885300022,-87.642808466,41.879255084,-87.642648998,0.6,,17031281900,Cash,Taxi Affiliation Services,180,28
+7,3.25,6,20,3,1435090500,41.922082541,-87.634156093,,,0.03,,,Cash,,0,
+6,5.45,9,2,7,1443234600,41.944226601,-87.655998182,41.944226601,-87.655998182,0.0,,,Credit Card,Blue Ribbon Taxi Association Inc.,300,6
+8,6.0,6,17,7,1465062300,41.89321636,-87.63784421,41.877406123,-87.621971652,0.0,,17031320400,Cash,Blue Ribbon Taxi Association Inc.,300,32
+56,26.85,4,22,1,1365976800,41.785998518,-87.750934289,41.892072635,-87.628874157,0.0,,17031081600,Cash,Choice Taxi Association,1380,8
+32,7.85,2,13,6,1424437200,41.880994471,-87.632746489,41.898331794,-87.620762865,2.1,,17031081300,Cash,,540,8
+32,12.85,12,22,3,1450217700,41.878865584,-87.625192142,41.944226601,-87.655998182,4.7,,,Cash,,780,6
+76,28.65,6,13,5,1434029400,41.980264315,-87.913624596,41.944226601,-87.655998182,0.0,,,Credit Card,Taxi Affiliation Services,2100,6
+4,11.65,4,19,3,1396380600,41.975170943,-87.687515515,41.922686284,-87.649488729,0.2,,,Credit Card,Taxi Affiliation Services,1080,7
+8,8.05,5,11,3,1401188400,41.89321636,-87.63784421,41.880994471,-87.632746489,1.8,,17031839100,Cash,,660,32
+28,4.65,9,17,2,1378746900,41.885300022,-87.642808466,41.879255084,-87.642648998,0.0,,17031281900,Cash,Taxi Affiliation Services,240,28
+3,14.05,2,6,3,1360046700,41.96581197,-87.655878786,41.899602111,-87.633308037,0.0,,,Unknown,Taxi Affiliation Services,540,8
+13,3.25,1,15,7,1421509500,41.983636307,-87.723583185,41.983636307,-87.723583185,0.0,,,Cash,,0,13
+8,34.65,3,19,1,1396206000,41.890922026,-87.618868355,41.97907082,-87.903039661,17.4,,17031980000,Cash,,1380,76
+8,5.05,12,2,1,1450576800,41.902788048,-87.62614559,41.892507781,-87.626214906,0.0,,17031081500,Credit Card,Taxi Affiliation Services,300,8
+8,4.25,4,20,4,1398285900,41.892072635,-87.628874157,41.898331794,-87.620762865,0.0,,17031081300,Credit Card,Blue Ribbon Taxi Association Inc.,120,8
+77,11.45,9,14,4,1378908000,41.9867118,-87.663416405,41.944226601,-87.655998182,0.0,,,No Charge,Northwest Management LLC,540,6
+8,9.45,10,23,5,1382657400,41.892042136,-87.63186395,41.89830587,-87.653613982,0.0,,17031842300,Cash,Taxi Affiliation Services,720,24
+7,7.45,5,20,5,1400791500,41.914616286,-87.631717366,41.928967266,-87.656156831,1.6,,17031070400,Cash,,540,7
+32,16.25,3,20,2,1364244300,41.870607372,-87.622172937,41.928967266,-87.656156831,0.0,,17031070400,Cash,Choice Taxi Association,1440,7
+8,5.85,11,15,2,1416238200,41.892507781,-87.626214906,41.880994471,-87.632746489,1.2,,17031839100,Credit Card,Dispatch Taxi Affiliation,300,32
+8,5.5,1,10,2,1451901600,41.892507781,-87.626214906,41.89503345,-87.619710672,0.5,,17031081401,Cash,KOAM Taxi Association,300,8
+32,5.25,6,13,6,1435325400,41.880994471,-87.632746489,41.880994471,-87.632746489,0.5,,17031839100,Credit Card,KOAM Taxi Association,300,32
+6,5.45,7,1,7,1373677200,41.942691844,-87.651770507,41.936237179,-87.656411531,0.8,,17031062900,Credit Card,Choice Taxi Association,360,6
+32,4.65,8,17,6,1408727700,41.880994471,-87.632746489,41.892042136,-87.63186395,0.6,,17031081700,Cash,,180,8
+32,4.05,1,12,3,1390305600,41.884987192,-87.620992913,41.884987192,-87.620992913,0.4,,17031320100,Cash,,120,32
+6,10.75,8,1,1,1471138200,41.945282331,-87.661545096,41.936237179,-87.656411531,2.8,,17031062900,Credit Card,,780,6
+28,8.45,8,0,7,1438993800,41.885300022,-87.642808466,41.902788048,-87.62614559,2.1,,17031081202,Cash,Taxi Affiliation Services,720,8
+7,5.25,9,19,7,1442690100,41.914747305,-87.654007029,41.929046937,-87.651310877,0.9,,17031070300,Cash,Taxi Affiliation Services,300,7
+6,9.25,1,0,7,1422060300,41.944226601,-87.655998182,41.899602111,-87.633308037,0.1,,,Cash,Taxi Affiliation Services,480,8
+5,11.05,4,0,7,1398470400,41.947791586,-87.683834942,41.983636307,-87.723583185,0.2,,,Cash,Taxi Affiliation Services,840,13
+32,5.85,10,8,2,1382343300,41.880994471,-87.632746489,41.89503345,-87.619710672,1.1,,17031081401,Credit Card,,300,8
+32,13.05,8,23,4,1408576500,41.884987192,-87.620992913,41.942577185,-87.647078509,0.3,,17031062000,Cash,Taxi Affiliation Services,660,6
+33,15.45,9,20,3,1410899400,41.859349715,-87.617358006,41.93057857,-87.642206313,5.54,,17031070102,Cash,,900,7
+76,45.85,5,15,2,1401118200,41.97907082,-87.903039661,41.880994471,-87.632746489,19.7,,17031839100,Credit Card,Taxi Affiliation Services,4200,32
+32,17.05,3,18,4,1425492000,41.880994471,-87.632746489,41.938391258,-87.63857492,0.0,,17031063200,Credit Card,Taxi Affiliation Services,1440,6
+6,19.45,4,19,5,1366916400,41.944226601,-87.655998182,41.850266366,-87.667569312,7.2,,,Cash,,1680,31
+8,4.65,5,21,3,1431467100,41.892507781,-87.626214906,41.884987192,-87.620992913,0.59,,17031320100,Credit Card,,180,32
+32,7.25,8,11,3,1408448700,41.884987192,-87.620992913,41.879255084,-87.642648998,1.0,,17031281900,Cash,,660,28
+8,12.05,5,21,4,1401311700,41.891971508,-87.612945414,41.936237179,-87.656411531,0.2,,17031062900,Cash,Taxi Affiliation Services,720,6
+24,6.85,6,11,5,1372330800,41.901206994,-87.676355989,41.878865584,-87.625192142,0.0,,,Credit Card,Taxi Affiliation Services,420,32
+7,8.85,9,11,4,1378899000,41.914747305,-87.654007029,41.879255084,-87.642648998,2.5,,17031281900,Cash,3201 - CID Cab Co Inc,600,28
+32,6.25,3,15,5,1395933300,41.880994471,-87.632746489,41.89321636,-87.63784421,1.1,,17031081800,Credit Card,,420,8
+8,7.05,9,9,2,1410773400,41.89321636,-87.63784421,41.880994471,-87.632746489,1.3,,17031839100,Cash,KOAM Taxi Association,480,32
+7,8.65,2,13,7,1361022300,41.922082541,-87.634156093,41.890922026,-87.618868355,0.0,,17031081403,Cash,Taxi Affiliation Services,540,8
+28,12.65,9,20,7,1379189700,41.874005383,-87.66351755,41.92276062,-87.699155343,0.0,,,No Charge,Choice Taxi Association,1140,22
+7,3.25,5,18,4,1432144800,41.928967266,-87.656156831,41.928967266,-87.656156831,0.0,,17031070400,Cash,Blue Ribbon Taxi Association Inc.,60,7
+8,4.45,8,21,2,1439847900,41.898331794,-87.620762865,41.892507781,-87.626214906,0.5,,17031081500,Cash,Taxi Affiliation Services,240,8
+7,7.45,5,2,1,1431224100,41.914616286,-87.631717366,41.916005274,-87.675095116,2.2,,17031831000,Cash,Taxi Affiliation Services,420,22
+7,10.05,3,13,1,1394975700,41.914616286,-87.631717366,41.879255084,-87.642648998,3.0,,17031281900,Cash,Choice Taxi Association,840,28
+28,9.25,5,14,6,1400856300,41.874005383,-87.66351755,41.901206994,-87.676355989,2.4,,,Cash,Taxi Affiliation Services,720,24
+32,4.85,5,8,3,1400573700,41.880994471,-87.632746489,41.892507781,-87.626214906,0.7,,17031081500,Cash,Dispatch Taxi Affiliation,240,8
+8,4.05,10,2,2,1444615200,41.892072635,-87.628874157,41.89321636,-87.63784421,0.6,,17031081800,Cash,Dispatch Taxi Affiliation,120,8
+8,7.5,10,23,7,1475969400,41.904935302,-87.649907226,41.926811182,-87.642605247,1.6,,17031070103,Cash,Taxi Affiliation Services,540,7
+76,36.85,1,22,1,1422223200,41.97907082,-87.903039661,41.884987192,-87.620992913,18.1,,17031320100,Credit Card,,1560,32
+8,37.45,2,10,1,1393150500,41.898331794,-87.620762865,41.97907082,-87.903039661,0.0,,17031980000,Credit Card,Taxi Affiliation Services,1560,76
+32,6.0,12,23,6,1481325300,41.880994471,-87.632746489,41.880994471,-87.632746489,0.7,,17031839100,Cash,,360,32
+8,5.05,1,11,4,1389179700,41.900265687,-87.63210922,41.898331794,-87.620762865,0.06,,17031081300,Cash,,0,8
+8,12.85,8,22,1,1407708000,41.892042136,-87.63186395,41.943237122,-87.643470956,0.49,,17031061901,Cash,,660,6
+13,30.65,10,18,4,1413396000,41.983636307,-87.723583185,41.922686284,-87.649488729,0.6,,,Cash,Taxi Affiliation Services,3300,7
+32,10.45,1,7,5,1389252600,41.878865584,-87.625192142,41.874005383,-87.66351755,3.6,,,Cash,,600,28
+7,7.25,12,7,1,1387698300,41.922686284,-87.649488729,41.901206994,-87.676355989,0.1,,,Cash,Taxi Affiliation Services,420,24
+32,6.65,5,18,5,1401386400,41.880994471,-87.632746489,41.880994471,-87.632746489,0.61,,17031839100,Cash,,540,32
+32,10.5,1,1,3,1452561300,41.878865584,-87.625192142,41.922686284,-87.649488729,2.9,,,Cash,KOAM Taxi Association,600,7