Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split MAP and REDUCE tasks into individual mesos tasks #60

Open
wants to merge 20 commits into
base: master
Choose a base branch
from
Open

Split MAP and REDUCE tasks into individual mesos tasks #60

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
c3445a1
Update README.md
hansbogert Apr 13, 2015
b11cd64
Merge pull request #48 from hansbogert/patch-1
tarnfeld Apr 13, 2015
b69bf8f
Use MESOS_NATIVE_JAVA_LIBRARY in README.md
hansbogert Apr 24, 2015
5c85bf7
Merge pull request #50 from hansbogert/patch-1
tarnfeld Apr 25, 2015
73569c5
* Restructure o.a.h.mapred.MesosScheduler#assignTasks to always pass …
briantopping May 8, 2015
6e93e75
Fixes per PR changes. Addiiton of release plugin to follow.
briantopping May 9, 2015
ee1c876
Missed a comment, rolled back version change until another PR.
briantopping May 9, 2015
6352915
Add release plugin and changes to facilitate publishing to Sonatype OSS
briantopping May 12, 2015
7ca77dd
Merge remote-tracking branch 'briantopping/master'
tarnfeld May 13, 2015
bd0fc09
Merge remote-tracking branch 'briantopping/ReleaseChanges'
tarnfeld May 13, 2015
b413b25
Added Framework Authentication.
DarinJ May 1, 2015
c99b6e9
Merge pull request #54 from DarinJ/master
tarnfeld May 15, 2015
d6b0d0c
Sync the tracker.stop() call to the scheduler
tarnfeld May 19, 2015
f547cb0
Refactor code to determine if a TT is idle
tarnfeld May 19, 2015
b7a8278
Merge pull request #59 from duedil-ltd/fix/tt-respawning
tarnfeld May 20, 2015
dc3e288
Split MAP and REDUCE tasks into individual mesos tasks
tarnfeld May 20, 2015
05b7acd
Remove killTasks() synchronized block to avoid a deadlock
tarnfeld May 21, 2015
b9ff376
Don't synchronized() inside the idle check callback
tarnfeld May 21, 2015
0d54030
Revert back to using a simpler TaskLauncher interrupt model
tarnfeld May 21, 2015
3f8c161
Ensure the suicide timer repeats itself
tarnfeld May 21, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ operation](http://hadoop.apache.org/docs/stable/single_node_setup.html#PseudoDis
</property>
```

[More details on configuration propertios can be found here.](configuration.md)
[More details on configuration properties can be found here.](configuration.md)

#### Start ####

Expand All @@ -145,13 +145,13 @@ to the Mesos native library.
On Linux:

```
$ MESOS_NATIVE_LIBRARY=/path/to/libmesos.so hadoop jobtracker
$ MESOS_NATIVE_JAVA_LIBRARY=/path/to/libmesos.so hadoop jobtracker
```

And on OS X:

```
$ MESOS_NATIVE_LIBRARY=/path/to/libmesos.dylib hadoop jobtracker
$ MESOS_NATIVE_JAVA_LIBRARY=/path/to/libmesos.dylib hadoop jobtracker
```

> **NOTE: You do not need to worry about distributing your Hadoop
Expand Down
30 changes: 30 additions & 0 deletions configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,36 @@ default values.
role configured in "mapred.mesos.role".
</description>
</property>
<property>
<name>mapred.mesos.framework.name</name>
<value>hadoop</value>
<description>
This is the Mesos framework name. Defaults to Hadoop plus port information.
</description>
</property>
<property>
<name>mapred.mesos.framework.principal</name>
<value>hadoop</value>
<description>
This is the Mesos framework principal. It is used for framework authentication.
Consult the Mesos documentation for details.
</description>
</property>
<property>
<name>mapred.mesos.framework.secretfile</name>
<value>/location/secretfile</value>
<description>
Location of the file holding the Mesos framework secret. It is used for framework authentication.
Consult the Mesos documentation for details. Caution: avoid newline characters, some editor place these before end of file.
</description>
</property>
<property>
<name>mapred.mesos.framework.user</name>
<value>hadoop</value>
<description>
This is the user the Mesos framework runs as. If left unset, it defaults to the user running the scheduler.
</description>
</property>

<!-- If you're using a custom Mesos Containerizer -->
<property>
Expand Down
67 changes: 42 additions & 25 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<parent>
<artifactId>oss-parent</artifactId>
<groupId>org.sonatype.oss</groupId>
<version>9</version>
</parent>

<modelVersion>4.0.0</modelVersion>

<groupId>org.apache.mesos</groupId>
<artifactId>hadoop-mesos</artifactId>
<version>0.1.0</version>
<groupId>com.github.mesos</groupId>
<artifactId>mesos-hadoop-mr1</artifactId>
<version>0.1.1-SNAPSHOT</version>

<url>https://github.com/mesos/hadoop</url>

<properties>
<encoding>UTF-8</encoding>
Expand All @@ -15,8 +23,8 @@
<!-- runtime deps versions -->
<commons-logging.version>1.1.3</commons-logging.version>
<commons-httpclient.version>3.1</commons-httpclient.version>
<hadoop-client.version>2.5.0-mr1-cdh5.2.0</hadoop-client.version>
<mesos.version>0.21.0</mesos.version>
<hadoop-client.version>2.6.0-mr1-cdh5.4.0</hadoop-client.version>
<mesos.version>0.21.1</mesos.version>
<protobuf.version>2.5.0</protobuf.version>
<metrics.version>3.1.0</metrics.version>
<snappy-java.version>1.0.5</snappy-java.version>
Expand All @@ -26,18 +34,6 @@
<mockito.version>1.9.5</mockito.version>
</properties>

<!-- TODO(benh): Remove repository once we've published a JDK6 JAR. -->
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
</repositories>

<dependencies>
<dependency>
<groupId>commons-logging</groupId>
Expand Down Expand Up @@ -108,23 +104,23 @@
</exclusions>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>${snappy-java.version}</version>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>${snappy-java.version}</version>
</dependency>

<!-- Test scope -->
<dependency>
<groupId>junit</groupId>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
Expand All @@ -134,9 +130,7 @@
<showWarnings>true</showWarnings>
</configuration>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.2</version>
<configuration>
Expand All @@ -160,7 +154,30 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.1</version>
<configuration>
</configuration>
</plugin>
</plugins>
</build>

<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
</repositories>

<scm>
<connection>scm:git:[email protected]:mesos/hadoop.git</connection>
<developerConnection>scm:[email protected]:mesos/hadoop.git</developerConnection>
<url>scm:git:[email protected]:mesos/hadoop.git</url>
</scm>
</project>
Loading