henryr

See screenshot

henryr

use tpch_parquet;
with subquery as (select /* +straight_join */ count(*) from lineitem a join /* +shuffle */
orders b on a.l_orderkey = b.o_orderkey where a.l_orderkey = b.o_orderkey and l_shipdate
between "1992-01-01" and "1994-01-01" and o_orderdate between "1992-01-01" and
"1994-01-01" group by a.l_orderkey having count(*) > 9999999999) select * from (select *
from subquery) a1, (select * from subquery) a2, (select * from subquery) a3, (select *
from subquery) a4, (select * from subquery) a5, (select * from subquery) a6, (select *
from subquery) a7, (select * from subquery) a8, (select * from subquery) a9, (select *
from subquery) a10, (select * from subquery) a11, (select * from subquery) a12, (select *
from subquery) a13, (select * from subquery) a14, (select * from subquery) a15, (select *

henryr

use tpch_300_parquet;
with subquery as (select /* +straight_join */ count(*) from lineitem a join /* +shuffle */ orders b on a.l_orderkey = b.o_orderkey where a.l_orderkey = b.o_orderkey and l_shipdate between "1992-01-01" and "1994-01-01" and o_orderdate between "1992-01-01" and "1994-01-01" group by a.l_orderkey having count(*) > 9999999999) select * from (select * from subquery) a1, (select * from subquery) a2, (select * from subquery) a3, (select * from subquery) a4, (select * from subquery) a5, (select * from subquery) a6, (select * from subquery) a7, (select * from subquery) a8, (select * from subquery) a9, (select * from subquery) a10, (select * from subquery) a11, (select * from subquery) a12, (select * from subquery) a13, (select * from subquery) a14, (select * from subquery) a15, (select * from subquery) a16, (select * from subquery) a17, (select * from subquery) a18, (select * from subquery) a19, (select * from subquery) a20, (select * from subquery) a21, (select * from subquery) a22, (select * fro

henryr

=============================================================================
== Baseline KRPC
== many_independent_fragments takes 378s

-  13.12%  impalad  impalad              [.] tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)
   - tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**)
      - 93.95% tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)
         - tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long)
            - 98.16% operator new[](unsigned long)
                 29.20% impala::RowDescriptor::RowDescriptor(impala::RowDescriptor const&)

henryr

#!/bin/bash

# If this is an amend, not a new commit, need to take diff against HEAD~1, not HEAD, to do
# formatting.
RELATIVE_COMMIT="HEAD"
GIT_CMD="$(ps -ocommand= -p $PPID)"
if [[ ${GIT_CMD} == *"--amend"* ]]
then
    RELATIVE_COMMIT="HEAD~1"
fi

henryr

#!/bin/bash

# If this is an amend, not a new commit, need to take diff against HEAD~1, not HEAD, to do
# formatting.
RELATIVE_COMMIT="HEAD"
GIT_CMD="$(ps -ocommand= -p $PPID)"
if [[ ${GIT_CMD} == *"--amend"* ]]
then
    RELATIVE_COMMIT="HEAD~1"
fi

henryr

Error Message

test failure
Stacktrace

metadata/test_ddl.py:157: in test_truncate_table
>         multiple_impalad=self._use_multiple_impalad(vector))
common/impala_test_suite.py:254: in run_test_case
>                          pytest.config.option.update_results)
common/test_result_verifier.py:346: in verify_raw_results

henryr

com.cloudera.impala.planner.PlannerTest.testJoinOrder

Failing for the past 1 build (Since Failed#594 )
Took 0.21 sec.
add description
Error Message

section PLAN of query:
select
  n_name,

henryr

The claim for 'external consistency' is as following (all quotes are from the journal paper):

"external-consistency invariant: if the start of a transaction T2 occurs after the commit of a transaction T1, then the commit time-stamp of T2 must be greater than the commit timestamp of T1." 
But when assigning a commit timestamp, section 4.2.1 has:

"The commit timestamp s must be greater than or equal to all pre- pare timestamps (to satisfy the constraints discussed in Section 4.1.3), greater than TT.now().latest at the time the coordinator received its commit message, and greater than any timestamps the leader has assigned to previous transactions (again, to pre- serve monotonicity)."

This lead me down the following path:

1. Is TT.now().latest monotonic increasing? Presumably not, otherwise the third requirement (about being larger than any previous transaction) would be implicit, and also epsilon is not monotonic increasing.

henryr

import java.io.ByteArrayOutputStream;

public class TestByteArray {

  static byte[] chunk = new byte[1024 * 1024];
  public static void main(String[] args) {
    ByteArrayOutputStream baos = new ByteArrayOutputStream();
    int numChunks = 2 * 1024 * 1024;
    for (int  i = 0; i < numChunks; ++i) {
      long start = System.currentTimeMillis();