Remove unused scripts, update documentation wrt contributing & testing (#182)

* Remove unused scripts, update documentation

* Increase heading levels

Technically KafkaEx is the top-level header

* Clarify in CONTRIBUTING.md

After updating the README

Author: dantswain

CONTRIBUTING.md

@@ -3,6 +3,10 @@
 - [Fork](https://github.com/kafkaex/kafka_ex/fork), then clone the repo: `git clone git@github.com:your-username/kafka_ex.git`
 - Create a feature branch: `git checkout -b feature_branch`
 - Make your changes
+- Make sure the tests all pass with the dockerized test cluster.  See the "Testing" section of the README.
+```
+mix test --include integration --include consumer_group --include server_0_p_9_p_0
+```
 - Make sure the unit tests pass: `mix test --no-start`
 - Make sure the integration tests pass: `mix test --only integration`
 - Make sure the consumer group tests pass: `mix test --only consumer_group`

README.md

@@ -10,8 +10,7 @@ KafkaEx
 
 [Apache Kafka](http://kafka.apache.org/) (>= 0.8.0) client for Elixir/Erlang.
 
-Usage
------
+## Usage
 
 Add KafkaEx to your mix.exs dependencies:
 
@@ -244,54 +243,95 @@ KafkaEx.produce(produce_request)
 
 Compression is handled automatically on the consuming/fetching end.
 
-### Test
+## Testing
+
+It is strongly recommended to test using the Dockerized test cluster described
+below.  This is required for contributions to KafkaEx.
+
+**NOTE** You may have to run the test suite twice to get tests to pass.  Due to
+asynchronous issues, the test suite sometimes fails on the first try.
+
+### Dockerized Test Cluster
+
+Testing KafkaEx requires a local SSL-enabled Kafka cluster with 3 nodes: one
+node listening on each port 9092, 9093, and 9093.  The easiest way to do this
+is using the scripts in
+this repository that utilize [Docker](https://www.docker.com) and
+[Docker Compose](https://www.docker.com/products/docker-compose) (both of which
+are freely available).  This is the method we use for our CI testing of
+KafkaEx.
+
+To launch the included test cluster, run
 
-#### Unit tests
 ```
-mix test --no-start
+./scripts/docker_up.sh
 ```
 
-#### Integration tests
-Add the broker config to `config/config.exs` and run:
-##### Kafka >= 0.8.2
+The `docker_up.sh` script will attempt to determine an IP address for your
+computer on an active network interface.  If it has trouble with this, you can
+try manually specifying a network interface in the `IP_IFACE` environment
+variable:
+
 ```
-mix test --only consumer_group --only integration
+IP_IFACE=eth0 ./scripts/docker_up.sh
 ```
-##### Kafka < 0.8.2
+
+The test cluster runs Kafka 0.9.2.
+
+### Running the KafkaEx Tests
+
+The KafkaEx tests are split up using tags to handle testing multiple scenarios
+and Kafka versions.
+
+#### Unit tests
+
+These tests do not require a Kafka cluster to be running.
+
 ```
-mix test --only integration
+mix test --no-start
 ```
 
-#### All tests
-##### Kafka >= 0.8.2
+#### Integration tests
+
+If you are not using the Docker test cluster, you may need to modify
+`config/config.exs` for your set up.
+
+The full test suite requires Kafka 0.9+.
+
+##### Kafka >= 0.9.0
+
+The 0.9 client includes functionality that cannot be tested with older
+clusters.
+
 ```
-mix test --include consumer_group --include integration
+mix test --include integration --include consumer_group --include server_0_p_9_p_0
 ```
-##### Kafka < 0.8.2
+
+##### Kafka >= 0.8.2 and < 0.9.0
+
+Kafka 0.8.2 introduced the consumer group API.
+
 ```
-mix test --include integration
+mix test --include consumer_group --include integration
 ```
 
-### Testing with Docker
+##### Kafka < 0.8.2
 
-Assuming you have Docker 1.12 or later installed, you can use the included
- scripts to launch a Kafka 0.9 cluster for testing.
+If your test cluster is older, the consumer group tests must be omitted.
 
 ```
-./scripts/docker_up.sh
-mix test --include integration --include consumer_group --include server_0_p_9_p_0
+mix test --include integration
 ```
 
-If `docker_up.sh` has trouble finding the correct network interface, you can
-manually specify one by running, e.g., `IP_IFACE=eth0 ./scripts/docker_up.sh`.
-
 ### Static analysis
 
+This requires Elixir 1.3.2+.
+
 ```
 mix dialyzer
 ```
 
-### Contributing
+## Contributing
 
 All contributions are managed through the
 [kafkaex github repo](https://github.com/kafkaex/kafka_ex).

scripts/README.md

@@ -1,8 +1,3 @@
-This directory contains three scripts to:
-* Install the Kafka version preferred for developing kafka_ex;
-* Start a three-node Kafka (and Zookeeper cluster);
-* Clean everything back up again.
-
-The script `kafka_base_dir.sh` is used by the others to decide where to place Kafka; this way
-we can drop Kafka in a directory cached betweeen builds by Travis, hopefully speeding up builds
-and making them less dependent on the availability of Apache mirrors.
+The scripts in this directory are used for testing KafkaEx and are not part of
+the release package.  Each script should have a comment near the top explaining
+its purpose.

scripts/active_ifaces.sh

@@ -1,7 +1,12 @@
 #!/usr/bin/awk -f
-# print only active ifaces from ifconfig
+
+# Print only active ifaces from ifconfig
 #    from http://unix.stackexchange.com/questions/103241/how-to-use-ifconfig-to-show-active-interface-only
 #    (using a version without pcregrep because that may not be available)
+#
+# We need the ip of an active network interface in order to properly launch the
+# dockerized Kafka cluster.
+
 BEGIN            { print_it = 0  }
 /status: active/ { print_it = 1  }
 /^($|[^\t])/     { if(print_it) print buffer; buffer = $0; print_it = 0  }

scripts/ci_tests.sh

@@ -1,6 +1,10 @@
 #!/bin/bash
 
 # Runs the test suite for the travis build
+#
+# If COVERALLS is true, then we report test coverage to coveralls.
+#
+# This script could be used for local testing as long as COVERALLS is not set.
 
 set -ev
 

scripts/cleanup_kafka.sh

@@ -1,9 +1,14 @@
 #!/bin/bash
 
-# this script is used when the docker kafka container starts
-# this version is specialized from
+# This script is used when the docker kafka container starts
+#
+# This version is specialized from
 # https://github.com/wurstmeister/kafka-docker/blob/master/start-kafka.sh
 # to work with our setup (esp. with ssl)
+#
+# Note that we mainly manage the server.properties file during `docker_up.sh`;
+# the wurstmeister/kafka behavior of translating KAFKA_* env vars to settings
+# is removed from this script.
 
 set -ev
 

scripts/docker-start-kafka.sh

@@ -1,6 +1,14 @@
 #!/bin/bash
 
-# launches the dockerized kafka cluster
+# Launches a dockerized kafka cluster configured for testing with KafkaEx
+#
+# This script attempts to auto-detect the ip address of an active network
+# interface using `./scripts/active_ifaces.sh`.  You can override this by
+# supplying the name of an interface through the IP_IFACE env var - e.g.,
+#
+# IP_IFACE=eth0 ./scripts/docker_up.sh
+#
+# This script should be run from the project root
 
 set -e