How to connect Vertica to hadoop.

Moderator: NorbertKrupa

Post Reply
vicky_tsc
Newbie
Newbie
Posts: 12
Joined: Wed Apr 24, 2013 7:16 am

How to connect Vertica to hadoop.

Post by vicky_tsc » Thu Apr 25, 2013 3:37 am

I have downloaded the hadoop connnectors and examples. Also the hadoop is running. how can i connect to hadoop/how to test the examples.
I gave like as below as job,

hadoop jar hadoop-vertica-example.jar com.vertica.hadoop.VerticaExample
-libjars /usr/lib/hadoop/lib/hadoop-vertica.jar,/usr/lib/hadoop/lib/vertica-jdk5.jar
-Dmapred.vertica.hostnames=192.168.x.x -Dmapred.vertica.port=5433
-Dmapred.vertica.username=hdfs -Dmapred.vertica.database=HADOOP
-Dmapred.vertica.debug=1

it show the error as,
Exception in thread "main" java.lang.RuntimeException: java.sql.SQLException: [Vertica][VJDBC](100176) Failed to connect to host 192.168.x.x on port 5433. Reason: Connection timed out:connect

How to resolve this.

vicky_tsc
Newbie
Newbie
Posts: 12
Joined: Wed Apr 24, 2013 7:16 am

Re: How to connect Vertica to hadoop.

Post by vicky_tsc » Mon Apr 29, 2013 7:36 am

I have successively connected the had0op 0.20.2, pig with Vertica using pig script.

vicky
Newbie
Newbie
Posts: 15
Joined: Fri Mar 22, 2013 10:54 am

Re: How to connect Vertica to hadoop.

Post by vicky » Tue Apr 30, 2013 2:26 pm

Hey buddy can you please explain whole procedure how u connected vertica with hadoop

ranjan14ravi
Newbie
Newbie
Posts: 3
Joined: Wed Nov 20, 2013 7:18 pm

Re: How to connect Vertica to hadoop.

Post by ranjan14ravi » Wed Nov 20, 2013 7:41 pm

Hello,

I am trying to connect Hadoop with Vertica, but not able to do it.

As per documentation, I created the VerticaExample.java file, but not able to compile it.

Can anybody help me on this.
Attachments
error2.txt
(8.27 KiB) Downloaded 870 times
error.txt
(1.75 KiB) Downloaded 849 times

id10t
GURU
GURU
Posts: 732
Joined: Mon Apr 16, 2012 2:44 pm

Re: How to connect Vertica to hadoop.

Post by id10t » Thu Nov 21, 2013 10:36 am

Hi!

[DELETED]
Last edited by id10t on Sat May 09, 2015 3:16 pm, edited 1 time in total.

ranjan14ravi
Newbie
Newbie
Posts: 3
Joined: Wed Nov 20, 2013 7:18 pm

Re: How to connect Vertica to hadoop.

Post by ranjan14ravi » Fri Nov 22, 2013 8:46 am

thanks...

I got the resolution... it was compilation issue.

ranjan14ravi
Newbie
Newbie
Posts: 3
Joined: Wed Nov 20, 2013 7:18 pm

Re: How to connect Vertica to hadoop.

Post by ranjan14ravi » Fri Nov 22, 2013 11:32 am

Hello,
I am unable to load a Text File from HDFS into HP Vertica using Hadoop streaming method. I have followed exact steps as mentioned in "HP Vertica Analytics Platform Version 6.1.x Documentation" but no success yet. Can any one help !!

Below is the log details:
Error: Could not find or load main class job_201311211158_0011
[hduser@centos ~]$ hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-*.jar -libjars $HADOOP_HOME/lib/hadoop-vertica.jar -Dmapred.reduce.tasks=0 -Dmapred.vertica.output.table.name=streaming -Dmapred.vertica.output.table.def="intcol integer, floatcol float, varcharcol varchar" -Dmapred.vertica.hostnames=localhost -Dmapred.vertica.port=5433 -Dmapred.vertica.username=hduser -Dmapred.vertica.password=ranjan14 -Dmapred.vertica.database=Vmart -Dmapred.vertica.output.delimiter="|" -Dmapred.vertica.output.terminator="~" -input /tmp/textdata.txt -output /tmp/output -mapper "python /home/hduser/work/mapper.py" -outputformat com.vertica.hadoop.deprecated.VerticaStreamingOutput
Warning: $HADOOP_HOME is deprecated.

packageJobJar: [/tmp/hadoop-hduser/hadoop-unjar4618000824638461660/] [] /tmp/streamjob5803503291157496379.jar tmpDir=null
13/11/22 02:22:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/11/22 02:22:17 WARN snappy.LoadSnappy: Snappy native library not loaded
13/11/22 02:22:17 INFO mapred.FileInputFormat: Total input paths to process : 1
13/11/22 02:22:17 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-hduser/mapred/local]
13/11/22 02:22:17 INFO streaming.StreamJob: Running job: job_201311211158_0012
13/11/22 02:22:17 INFO streaming.StreamJob: To kill this job, run:
13/11/22 02:22:17 INFO streaming.StreamJob: /usr/local/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201311211158_0012
13/11/22 02:22:17 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?j ... 11158_0012
13/11/22 02:22:18 INFO streaming.StreamJob: map 0% reduce 0%
13/11/22 02:22:59 INFO streaming.StreamJob: map 100% reduce 100%
13/11/22 02:22:59 INFO streaming.StreamJob: To kill this job, run:
13/11/22 02:22:59 INFO streaming.StreamJob: /usr/local/hadoop/hadoop-1.2.1/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201311211158_0012
13/11/22 02:22:59 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?j ... 11158_0012
13/11/22 02:22:59 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201311211158_0012_m_000001
13/11/22 02:22:59 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

Thanks in advance!!!

Ravi Ranjan

Post Reply

Return to “Hadoop Connector”