Storage Plugin

Create the Storage Plugin

In this section we create the skeleton of our storage plugin:

• The Maven module
• The Plugin Config
• The Plugin Class

We build the plugin in two passes. In the first, we create the basic structure. Provide just enough behavior to plan and execute a query. Some data can appear in the plugin config (connection endpoints, credentials, etc.) You will find it useful to hard-code the more advanced bits (query-specific parameters) just to get things going. Later, you will learn how to us filter-push down and other techniques to replace the hard-coded bits with proper implementations.

Create the Module

Choose a name for the Maven module that will hold your storage plugin. Let's call ours "example". Storage plugins typically go in the contrib module, in a subdirectory named storage-example.

To create the project in Eclipse:

• Select the drill-contrib-parent node in the Eclipse Package Explorer.
• Choose Maven → New Maven Module Project from the context menu.
• Give your project the storage-example name.
• Accept the other defaults and create the project.
• Edit the resulting pom.xml file to add your needed dependencies. Use other storage plugins as a "cheat sheet" to see what is needed.
• Edit drill-contrib-parent/pom.xml to verify your module appears in the module list. Add it if missing:

  <modules>
    ...
    <module>storage-example</module>
    <module>data</module>
    ...
  </modules>

• (Review this part.) Get Eclipse to know about your project. Use File → Import → Existing Maven Projects. Select the Drill root. Your project should now appear in the Package Explorer as storage-example. (If this does not work, try selecting drill and Maven → Update Project.
• Eclipse named your new package org.storage.example. Rename this package to org.apache.drill.exec.store.example.
• Do the same for the test package.
• Run the Maven-provided 'App' class to ensure everything works. Select the class name, then Debug as → Java Application from the context menu.
• Delete the example App and AppTest classes.

Storage Plugin Config Class

The config class provides all configuration information needed to execute your plugin, except the query-specific information.

• Create the ExampleStoragePluginConfig class:

@JsonTypeName(ExampleStoragePluginConfig.NAME)
public class ExampleStoragePluginConfig extends StoragePluginConfigBase {

  public static final String NAME = "example";

  public int field1;
  public String field2;

  @JsonCreator
  public ExampleStoragePluginConfig(
    @JsonProperty("field1") int field1,
    @JsonProperty("field2") String field2) {
    this.field1 = field1;
    this.field2 = field2;
  }

  @JsonProperty("field1")
  public int getField1() { return field1; }
  @JsonProperty("field2")
  public String getField2() { return field2; }

  @Override
  public boolean equals(Object o) {
     if (o == this) {
      return true;
    }
    if (o == null || !(o instanceof ExampleStoragePluginConfig)) {
      return false;
    }
   return Objects.equal(field1 field2);
  }

  @Override
  public int hashCode() {
    return Objects.hashCode(field1, field2);
  }

  @Override
  public String toString() {
    return new PlanStringBuilder(SumoStoragePluginConfig.NAME)
        .field("field1", field1)
        .field("field2", field2)
        .toString();
  }
}

Add the config fields needed by your plugin. You will need:

• The field itself, should be a simple Java type such as an int or String. (More complex types are possible, but can be tricky. Look at other config classes for examples.)
• While we'd like to make our fields private and final, if you want to use your plugin with table functions, the fields must be public and not final.
• Add either a public setFoo method or add the field to the constructor.
• A good practice is to add @JsonProperty to identify each property. You can change the variable name, but the property must be the same across releases to allow users to upgrade.
• Create a public getFoo method for each field.
• Add the field to the hashCode, equals and toString methods.
• toString uses the PlanStringBuilder class to build the string since the string will appear as part of the EXPLAIN PLAN FOR output.

Storage Plugin Class

The storage plugin class is Drill's main entry point to your plugin.

• Create the storage plugin class in Eclipse. The initial file should look like this:

public class SumoStoragePlugin extends BaseStoragePlugin<BaseStoragePluginConfig> {
  private final ExampleStoragePluginConfig config;

  public SumoStoragePlugin(SumoStoragePluginConfig config,
      DrillbitContext context, String name) throws IOException {
    super(context, config, name, buildOptions());
    schemaFactory = new SumoSchemaFactory(this);
  }

  private static StoragePluginOptions buildOptions() {
    StoragePluginOptions options = new StoragePluginOptions();
    options.supportsRead = true;
    options.supportsProjectPushDown = true;
    return options;
  }

  @Override
  public void registerSchemas(SchemaConfig schemaConfig, SchemaPlus parent)
      throws IOException {
    // TODO Auto-generated method stub
  }
}

The constructor has special meaning to Drill. Drill looks up the storage plugin config to locate the storage plugin class. Specifically, Drill uses the type of the first argument (here ExampleStoragePluginConfig) to match a storage plugin class with a storage plugin config class.

StoragePluginOptions is a feature of the "base" storage plugin that gathers options that would otherwise be specified via a collection of method overrides. Your storage plugin will likely support just read. It can, however, support both read and write, or just write.

All plugins based on EVF support project push-down. The other fields will be explained as we proceed.

Drill Config File

Create the plugin's config file:

• Under your src folder, create a resources subfolder.
• Select the resources folder in the Package Explorer, then, from the context menu: Build Path → Use as Source Folder.
• Under resources, create drill-module.conf (easiest to just copy from another storage plugin such as OpenTSDB):

drill.classpath.scanning: {
  packages += "org.apache.drill.exec.store.example"
}

The above tells Drill about your storage plugin.

First Test

Create a test for your plugin:

public class TestExamplePlugin extends ClusterTest {

  @BeforeClass
  public static void setup() throws Exception {
    ClusterFixtureBuilder builder = new ClusterFixtureBuilder(dirTestWatcher);
    startCluster(builder);

    StoragePluginRegistry pluginRegistry = cluster.drillbit().getContext().getStorage();
    ExampleStoragePluginConfig config =
        new ExampleStoragePluginConfig(/* Your fields here */);
    config.setEnabled(true);
    pluginRegistry.createOrUpdate(ExampleStoragePluginConfig.NAME, config, true);
  }

  @Test
  public void test() {
    fail("Not yet implemented");
  }
}

The setup() method does some "magic" to start an in-process Drill "cluster" (really, one Drillbit), create an instance of your plugin config, and register that config in Drill.

Notice that the test case does nothing yet.

To test, set a breakpoint in the constructor of your storage plugin. Run your test. If Eclipse hits the breakpoint, then you've got everything wired up correctly. If not, go back and review the above steps:

• The config class is marked as Jackson serializable.
• The storage plugin takes the config as its first option as shown above.
• The drill-module.conf file exists and adds the correct package to the class path.
• The test code is correct.