Hadoop Example in Programmer's Guide

Moderator: NorbertKrupa

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Hadoop Example in Programmer's Guide

Post by becky » Tue Jul 02, 2013 9:11 pm

Hi,

I am trying to run through the Hadoop connector example in the Programmer's Guide.

When I try to compile the example java program I get all kinds of errors.

Has anyone else tried the example?

Code: Select all

[root@hadoop01]# javac -classpath /opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar:/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop/lib/hadoop-vertica.jar -d classes VerticaExample.java && jar -cvf hadoop-vertica-example2.jar -C classes .
VerticaExample.java:14: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configuration;
                             ^
VerticaExample.java:15: package org.apache.hadoop.conf does not exist
import org.apache.hadoop.conf.Configured;
                             ^
VerticaExample.java:16: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.DoubleWritable;
                           ^
VerticaExample.java:17: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.LongWritable;
                           ^
VerticaExample.java:18: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
VerticaExample.java:22: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.Tool;
                             ^
VerticaExample.java:23: package org.apache.hadoop.util does not exist
import org.apache.hadoop.util.ToolRunner;
                             ^
VerticaExample.java:29: cannot find symbol
symbol: class Configured
public class VerticaExample extends Configured implements Tool {
                                    ^
VerticaExample.java:29: cannot find symbol
symbol: class Tool
public class VerticaExample extends Configured implements Tool {
                                                          ^
VerticaExample.java:31: cannot find symbol
symbol  : class LongWritable
location: class com.vertica.hadoop.VerticaExample
    Mapper < LongWritable, VerticaRecord, Text, DoubleWritable > {
             ^
VerticaExample.java:31: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample
    Mapper < LongWritable, VerticaRecord, Text, DoubleWritable > {
                                          ^
VerticaExample.java:31: cannot find symbol
symbol  : class DoubleWritable
location: class com.vertica.hadoop.VerticaExample
    Mapper < LongWritable, VerticaRecord, Text, DoubleWritable > {
                                                ^
VerticaExample.java:33: cannot find symbol
symbol  : class LongWritable
location: class com.vertica.hadoop.VerticaExample.Map
        public void map(LongWritable key, VerticaRecord value, Context context)
                        ^
VerticaExample.java:45: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample
    Reducer < Text, DoubleWritable, Text, VerticaRecord > {
              ^
VerticaExample.java:45: cannot find symbol
symbol  : class DoubleWritable
location: class com.vertica.hadoop.VerticaExample
    Reducer < Text, DoubleWritable, Text, VerticaRecord > {
                    ^
VerticaExample.java:45: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample
    Reducer < Text, DoubleWritable, Text, VerticaRecord > {
                                    ^
VerticaExample.java:62: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample.Reduce
        public void reduce(Text key, Iterable < DoubleWritable > values,
                           ^
VerticaExample.java:62: cannot find symbol
symbol  : class DoubleWritable
location: class com.vertica.hadoop.VerticaExample.Reduce
        public void reduce(Text key, Iterable < DoubleWritable > values,
                                                ^
VerticaExample.java:39: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample.Map
                context.write(new Text((String) value.get(3)),
                                  ^
VerticaExample.java:40: cannot find symbol
symbol  : class DoubleWritable
location: class com.vertica.hadoop.VerticaExample.Map
                    new DoubleWritable((Long) value.get(0)));
                        ^
VerticaExample.java:39: cannot access org.apache.hadoop.util.Progressable
class file for org.apache.hadoop.util.Progressable not found
                context.write(new Text((String) value.get(3)),
                       ^
VerticaExample.java:56: cannot access org.apache.hadoop.conf.Configuration
class file for org.apache.hadoop.conf.Configuration not found
                record = new VerticaRecord(context.getConfiguration());
                                                                   ^
VerticaExample.java:125: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample.Reduce
            context.write(new Text("mrtarget"), record);
                              ^
VerticaExample.java:131: cannot find symbol
symbol  : class Configuration
location: class com.vertica.hadoop.VerticaExample
        Configuration conf = getConf();
        ^
VerticaExample.java:131: cannot find symbol
symbol  : method getConf()
location: class com.vertica.hadoop.VerticaExample
        Configuration conf = getConf();
                             ^
VerticaExample.java:143: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample
        job.setMapOutputKeyClass(Text.class);
                                 ^
VerticaExample.java:144: cannot find symbol
symbol  : class DoubleWritable
location: class com.vertica.hadoop.VerticaExample
        job.setMapOutputValueClass(DoubleWritable.class);
                                   ^
VerticaExample.java:148: cannot find symbol
symbol  : class Text
location: class com.vertica.hadoop.VerticaExample
        job.setOutputKeyClass(Text.class);
                              ^
VerticaExample.java:127: method does not override or implement a method from a supertype
    }@
     ^
VerticaExample.java:167: cannot find symbol
symbol  : class Configuration
location: class com.vertica.hadoop.VerticaExample
        int res = ToolRunner.run(new Configuration(), new VerticaExample(),
                                     ^
VerticaExample.java:167: cannot find symbol
symbol  : variable ToolRunner
location: class com.vertica.hadoop.VerticaExample
        int res = ToolRunner.run(new Configuration(), new VerticaExample(),
                  ^
Note: VerticaExample.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
31 errors
THANKS - BECKSTER

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1811
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: Hadoop Example in Programmer's Guide

Post by JimKnicely » Tue Jul 02, 2013 9:29 pm

Hey Becky,

Instead of passing in the jar file names yourself, try using the hadoop classpath command. Like this:

Code: Select all

javac -classpath `hadoop classpath` \
-d classes VerticaExample.java \
&& jar -cvf hadoop-vertica-example.jar -C classes .
PS. Make sure that you enclose the hadoop classpath command in back ticks and not apostrophes :)
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Hadoop Example in Programmer's Guide

Post by becky » Tue Jul 02, 2013 10:12 pm

Thanks Jim! I was able to compile the program! But now when I try and run it, I seem to have another issue :cry:

Code: Select all

[root@hadoop01]# hadoop jar hadoop-vertica-example.jar \
> com.vertica.hadoop.VerticaExample \
> -Dmapred.vertica.hostnames=vertica01,vertica02,vertica03,vertica04 \
> -Dmapred.vertica.port=5433 \
> -Dmapred.vertica.username=dbadmin \
> -Dmapred.vertica.database=hadooptest
13/07/02 16:59:05 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/07/02 16:59:05 INFO mapred.JobClient: Cleaning up the staging area hdfs://hadoop01:8020/user/root/.staging/job_201307021107_0002
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
        at com.vertica.hadoop.VerticaOutputFormat.checkOutputSpecs(VerticaOutputFormat.java:74)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
        at com.vertica.hadoop.VerticaExample.run(VerticaExample.java:163)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at com.vertica.hadoop.VerticaExample.main(VerticaExample.java:167)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
THANKS - BECKSTER

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1811
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: Hadoop Example in Programmer's Guide

Post by JimKnicely » Wed Jul 03, 2013 12:10 pm

Becky,

Check out this post:

http://www.vertica-forums.com/viewtopic.php?p=1814

There is a version incompatibility. It seems Hadoop changed a class to an interface in version 2.

I would suggest going to the download section of Vertica's web site, scroll down to the "Add HP Vertica Connectors, Plugins and Packages" section and down load and install the "HP Vertica Hadoop Connector for Hadoop 2.0, supports CDH4".

Thanks!
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Hadoop Example in Programmer's Guide

Post by becky » Wed Jul 03, 2013 12:56 pm

Thanks, again, Jim!

I installed the new Coudera hadoop jar files, re-compiled the java and re-ran my job.

It's getting better :)

Now I get these errors:
  • 13/07/03 07:06:49 INFO mapred.JobClient: Task Id : attempt_201307021107_0004_r_000002_0, Status : FAILED
    java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.vertica.hadoop.VerticaOutputFormat not found
Last edited by becky on Wed Jul 03, 2013 1:23 pm, edited 1 time in total.
THANKS - BECKSTER

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Hadoop Example in Programmer's Guide

Post by becky » Wed Jul 03, 2013 1:22 pm

Hi Jim,

Note that the job is partially working as the mrtarget table is created in Vertica:

Code: Select all

dbadmin=> select * from mrtarget;
 a | b | c | d | f | t | v | z
---+---+---+---+---+---+---+---
(0 rows)
Thanks so much for your help on this!!!
THANKS - BECKSTER

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1811
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: Hadoop Example in Programmer's Guide

Post by JimKnicely » Wed Jul 03, 2013 4:05 pm

Hey Becky,

Looks like your job can't find the hadoop-vertica.jar file.

You can use the -libjars option to add your libraries.

Do it like this:

1. Create an environment variable that stores the location of the jar(s):

Code: Select all

export LIBJARS=/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/hadoop/lib/vertica-jdk5-6.1.2-0.jar,/opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/l/hadoop/lib/hadoop-vertica.jar
2. Modify your hadoop command as follows:

Code: Select all

hadoop jar hadoop-vertica-example.jar \
 com.vertica.hadoop.VerticaExample -libjars ${LIBJARS} \
 -Dmapred.vertica.hostnames=vertica01,vertica02,vertica03,vertica04 \
 -Dmapred.vertica.port=5433 \
 -Dmapred.vertica.username=dbadmin \
 -Dmapred.vertica.database=hadooptest
Let us know if this works for you!
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

Post Reply

Return to “Hadoop Connector”