Thursday, January 30, 2014

How to create Eclipse plugin for Hadoop-2.2.0

Hadoop 2.2.0 doesn't come with a compiled Plugin jar for Eclipse. This tutorial will guide you through how to build your own Hadoop-2.2.0 eclipse Plugin.

Requirements:

  • git
  • ant
  • Eclipse Kepler
  • Hadoop srouce code

I assume you already have Eclipse installed. If not:

Install Eclipse:
  1. Download Eclipse Kepler from "http://www.eclipse.org/downloads/", then choose "Eclipse IDE for Java EE Developers".
  2. Uncomnpress it to "/opt/eclipse"
  3. Export /opt/eclipse to your shell PATH.
Install git and ant:
If you don't have git and ant, just "yum install git" and "yum install ant" to install both packages.

Build up Hadoop2x Eclipse plugin:
Download the souce code for Hadoop plugin from https://github.com/winghc/hadoop2x-eclipse-plugin.git
$ git clone https://github.com/winghc/hadoop2x-eclipse-plugin.git

Compile and build the jar file:
$ cd hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin
$ ant jar  -Dversion=2.2.0 -Declipse.home=/opt/eclipse-kepler-sr1/ -Dhadoop.home=/home/lxu/opt/hadoop-2.2.0/
Buildfile: build.xml
check-contrib:
init:
     [echo] contrib: eclipse-plugin
init-contrib:

ivy-download:
      [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      [get] To: /home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/ivy/ivy-2.1.0.jar
      [get] Not modified - so not downloaded
ivy-probe-antlib:
ivy-init-antlib:
ivy-init:
[ivy:configure] impossible to define new type: class not found: org.apache.ivy.plugins.signer.bouncycastle.OpenPGPSignatureGenerator in [] nor Ivy classloader
[ivy:configure] :: Apache Ivy 2.3.0-local-20131108183505 - 20131108183505 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/ivy/ivysettings.xml

ivy-resolve-common:

ivy-retrieve-common:
[ivy:cachepath] impossible to define new type: class not found: org.apache.ivy.plugins.signer.bouncycastle.OpenPGPSignatureGenerator in [] nor Ivy classloader
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
...
...
     [copy] Copying /home/lxu/opt/hadoop-2.2.0/share/hadoop/common/lib/hadoop-auth-2.2.0.jar to /home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/lib/hadoop-auth-2.2.0.jar
     [copy] Copying 1 file to /home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/lib
     [copy] Copying /home/lxu/opt/hadoop-2.2.0/share/hadoop/common/lib/netty-3.6.2.Final.jar to /home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/lib/netty-3.6.2.Final.jar
      [jar] Building jar: /home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.2.0.jar

BUILD SUCCESSFUL

Total time: 12 seconds

Copy the jar file to eclise's Plugin directory:
$ cp home/lxu/workspace/eclipse-kp/hadoop2x-eclipse-plugin/build/contrib/eclipse-plugin/hadoop-eclipse-plugin-2.2.0.jar /opt/eclipse-kepler-sr1/plugins/hadoop-eclipse-plugin-2.2.0.jar

Restart Eclipse.

2 comments:

Unknown said...

I downloaded hadoop 2.3.0. I tried tweaking this a bit, to change create some of the symlinks to be poining to what 2.2.0 is expecting like below.

./hadoop/common/lib/commons-lang-2.5.jar
./hadoop/common/lib/hadoop-auth-2.2.0.jar

After the jar was created, I tried copying to eclipse/plugins/ folder and restarted eclipse. I wasn't able to get it done. Any thoughts

Unknown said...

I have hadoop-2.3.0 on my system. I tried usign that for creating the plugin jar. It failed and i had to symlink the following jar files.

./hadoop/common/lib/commons-lang-2.5.jar
./hadoop/common/lib/hadoop-auth-2.2.0.jar

the build was sucessful, when i placed the plugin jar and copied it to eclipse/plugins, it wasn't sucessfull. any thoughts?