Skip to content
This repository was archived by the owner on May 12, 2021. It is now read-only.

Conversation

@blrunner
Copy link
Contributor

@blrunner blrunner commented Apr 4, 2016

It depends on #846 and #953.

Here is my benchmark results as follows.

Configuration

  • EC2 instance type : c3.xlarge
  • Cluster: 1 master, 3 worker
  • Dataset: TPC-H (factor = 1)
  • Partition table schema
CREATE EXTERNAL TABLE lineitem_p (l_orderkey INT8, l_partkey INT8, l_suppkey INT8, l_linenumber INT8, l_quantity FLOAT8, l_extendedprice FLOAT8, l_discount FLOAT8, l_tax FLOAT8, l_returnflag TEXT, l_linestatus TEXT, l_commitdate text, l_receiptdate text, l_shipinstruct TEXT, l_shipmode TEXT, l_comment TEXT)
USING TEXT WITH ('text.delimiter'='|')
PARTITION BY COLUMN(l_shipdate text)
LOCATION 's3://Xyz';
  • Partition numbers of lineitem table: 2526 (each partitions includes just one file)

Queries

  • Q1: select * from lineitem_p limit 5;
  • Q2: select count(*) from lineitem_p;
  • Q3: select count(*) from lineitem_p where l_shipdate > '1994-09-25' and l_shipdate < '1994-10-10';

Query Execution Time

Query No Optimized Optimized Improvement
Q1 573.425 sec 4.228 sec 135.6x
Q2 653.175 sec 33.444 sec 19.5x
Q3 4.099 sec 2.429 sec 1.6x

Split Computation Time

Query No Optimized Optimized Improvement
Q1 572921 ms 2233 ms 256.5x
Q2 599437 ms 701 ms 855.1x
Q3 2537 ms 388 ms 6.5x

…into TAJO-1952

Conflicts:
	tajo-core-tests/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
…into TAJO-1952

Conflicts:
	tajo-plan/src/main/java/org/apache/tajo/plan/logical/PartitionedTableScanNode.java
blrunner added 28 commits April 22, 2016 15:16
…into TAJO-1952

Conflicts:
	tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
@blrunner
Copy link
Contributor Author

I'll reopen this PR after finishing #1020 and #1024.

@blrunner blrunner closed this May 16, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant