Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRILL-6820: Msgpack format reader #1500

Open
wants to merge 47 commits into
base: master
Choose a base branch
from
Open

DRILL-6820: Msgpack format reader #1500

wants to merge 47 commits into from

Commits on Sep 26, 2018

  1. added a msgpack format plugin and a zstd codec

    @jcmcote
    jcmcote committed Sep 26, 2018
    Configuration menu
    Copy the full SHA
    482dbc9 View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2018

  1. add support to skip over malformed MAP 

    add support for msgpack extended types
fix issue with columns of INT which then encounter a BIGINT
    @jcmcote
    jcmcote committed Sep 29, 2018
    Configuration menu
    Copy the full SHA
    c63fefa View commit details
    Browse the repository at this point in the history

  2. moved project to format-msgpack

    @jcmcote
    jcmcote committed Sep 29, 2018
    Configuration menu
    Copy the full SHA
    815f9db View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2018

  1. returning error codes when parsing error detected

    @jcmcote
    jcmcote committed Sep 30, 2018
    Configuration menu
    Copy the full SHA
    861730f View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2018

  1. added test case for batch with no columns

    @jcmcote
    jcmcote committed Oct 2, 2018
    Configuration menu
    Copy the full SHA
    16b03fb View commit details
    Browse the repository at this point in the history

  2. avoiding creating the MapOrList instances

    @jcmcote
    jcmcote committed Oct 2, 2018
    Configuration menu
    Copy the full SHA
    a536786 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2018

  1. simple clean ups

    Your Name committed Oct 4, 2018
    Configuration menu
    Copy the full SHA
    7cc9e47 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2018

  1. added support for learning schema and applying it

    Your Name committed Oct 7, 2018
    Configuration menu
    Copy the full SHA
    0044508 View commit details
    Browse the repository at this point in the history

  2. less verbose message when record is not valid

    Your Name committed Oct 7, 2018
    Configuration menu
    Copy the full SHA
    eb3efd5 View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2018

  1. testing complete model 

    fixed bug in reader (array of array)
new test case that require schema
added useSchema property to turn off schema utilization
    Your Name committed Oct 8, 2018
    Configuration menu
    Copy the full SHA
    cc26f2a View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2018

  1. implemented apply schema so the reader knows what column types to output

    Your Name committed Oct 10, 2018
    Configuration menu
    Copy the full SHA
    3f00445 View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2018

  1. reader can skip records that do not match schema

    Your Name committed Oct 11, 2018
    Configuration menu
    Copy the full SHA
    d751fa4 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2018

  1. when not using a schema the reader should ensureAtLeastOneField

    Your Name committed Oct 12, 2018
    Configuration menu
    Copy the full SHA
    c87a19d View commit details
    Browse the repository at this point in the history

  2. increased batch size to 16k

    Your Name committed Oct 12, 2018
    Configuration menu
    Copy the full SHA
    fed1fb2 View commit details
    Browse the repository at this point in the history

  3. fixed import

    Your Name committed Oct 12, 2018
    Configuration menu
    Copy the full SHA
    d575bfd View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2018

  1. clean up

    Your Name committed Oct 13, 2018
    Configuration menu
    Copy the full SHA
    9fbcfba View commit details
    Browse the repository at this point in the history

  2. drill timestamp is in milliseconds

    Your Name committed Oct 13, 2018
    Configuration menu
    Copy the full SHA
    bade33e View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2018

  1. ignoring junit test case which show internal drill issue

    Your Name committed Oct 15, 2018
    Configuration menu
    Copy the full SHA
    2594324 View commit details
    Browse the repository at this point in the history

  2. support to skip invalid elements in a list

    Your Name committed Oct 15, 2018
    Configuration menu
    Copy the full SHA
    cf20d98 View commit details
    Browse the repository at this point in the history

  3. consolidated printing warnings into context class

    Your Name committed Oct 15, 2018
    Configuration menu
    Copy the full SHA
    417e41e View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2018

  1. support for coercing values according to schema

    Your Name committed Oct 26, 2018
    Configuration menu
    Copy the full SHA
    497e5e0 View commit details
    Browse the repository at this point in the history

  2. added discovery and loading of extended type readers

    Your Name committed Oct 26, 2018
    Configuration menu
    Copy the full SHA
    25169fa View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2018

  1. error handling is not using exceptions 

    after doing some performance tests I concluded that throwing exceptions
1 out of 10000 records had not significant impact on performance and
makes the code much easier to understand.

also consolidated the reader count reader into a single reader class
that can do count or actual reading of records. Again much easier to
understand the code like this.
    Your Name committed Oct 27, 2018
    Configuration menu
    Copy the full SHA
    69f74cc View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2018

  1. refactored the writing of values 

    coercing values into target schema types
    Your Name committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    e558be9 View commit details
    Browse the repository at this point in the history

  2. refactored the writing of values 

    coercing values into target schema types
    Your Name committed Oct 30, 2018
    Configuration menu
    Copy the full SHA
    5629136 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2018

  1. work in progress

    Your Name committed Nov 3, 2018
    Configuration menu
    Copy the full SHA
    f2aff31 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2018

  1. better tracking of parsing

    Your Name committed Nov 5, 2018
    Configuration menu
    Copy the full SHA
    0626fd0 View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2018

  1. move schema to it's own package

    Your Name committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    b0fd31f View commit details
    Browse the repository at this point in the history

  2. code review fixes

    Your Name committed Nov 6, 2018
    Configuration menu
    Copy the full SHA
    4d748fa View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2018

  1. refactoring, documentation and better logging

    Your Name committed Nov 9, 2018
    Configuration menu
    Copy the full SHA
    ed7373c View commit details
    Browse the repository at this point in the history

  2. documentation

    Your Name committed Nov 9, 2018
    Configuration menu
    Copy the full SHA
    d927316 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2018

  1. added example of using msgpack

    Your Name committed Nov 10, 2018
    Configuration menu
    Copy the full SHA
    ba39754 View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2018

  1. unit test work

    Your Name committed Nov 16, 2018
    Configuration menu
    Copy the full SHA
    5e770be View commit details
    Browse the repository at this point in the history

  2. unit test work

    Your Name committed Nov 16, 2018
    Configuration menu
    Copy the full SHA
    1cd9470 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2018

  1. using ColumnMetadata

    Your Name committed Nov 18, 2018
    Configuration menu
    Copy the full SHA
    fc2c804 View commit details
    Browse the repository at this point in the history

  2. removed todo

    Your Name committed Nov 18, 2018
    Configuration menu
    Copy the full SHA
    8aedaec View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2018

  1. performance test

    Your Name committed Nov 20, 2018
    Configuration menu
    Copy the full SHA
    8dd32d4 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master'

    @jcmcote
    jcmcote committed Nov 20, 2018
    Configuration menu
    Copy the full SHA
    29affb8 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2018

  1. performance improvements

    @jcmcote
    jcmcote committed Nov 24, 2018
    Configuration menu
    Copy the full SHA
    cea0d12 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/apache/drill

    @jcmcote
    jcmcote committed Nov 24, 2018
    Configuration menu
    Copy the full SHA
    a07dfb2 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2018

  1. using a map to cache the field names 

    refactored the extension mechanism
    Your Name committed Nov 29, 2018
    Configuration menu
    Copy the full SHA
    df7b378 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2018

  1. performance optimization 

    avoiding creating objects and copying data into byte arrays
not decoding field names but instead map lookup on bytes
    Your Name committed Dec 1, 2018
    Configuration menu
    Copy the full SHA
    859d1fd View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2018

  1. small code improvements

    Your Name committed Dec 4, 2018
    Configuration menu
    Copy the full SHA
    e1e802d View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2018

  1. refactored FieldPathTracker 

    it now uses a hashmap to avoid re-creating String for the map keys
    Your Name committed Dec 6, 2018
    Configuration menu
    Copy the full SHA
    586a4d9 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2018

  1. fix usage of drillbuf and tupleschema

    Your Name committed Dec 10, 2018
    Configuration menu
    Copy the full SHA
    8f1cd78 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2018

  1. serialize schema as JSON with ordered childrens

    Your Name committed Dec 16, 2018
    Configuration menu
    Copy the full SHA
    3e552fa View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2019

  1. working on msgpack

    Your Name committed Jan 12, 2019
    Configuration menu
    Copy the full SHA
    cba47ad View commit details
    Browse the repository at this point in the history