Hadoop Example in Programmer's Guide

Moderator: NorbertKrupa

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Hadoop Example in Programmer's Guide

Post by becky » Wed Jul 03, 2013 6:36 pm

Wow! Thank you so much, Jim! The -libjars solution worked perfectly!
THANKS - BECKSTER

muaythai_duan
Newbie
Newbie
Posts: 18
Joined: Sun Aug 25, 2013 11:43 am

Re: Hadoop Example in Programmer's Guide

Post by muaythai_duan » Tue Sep 03, 2013 8:48 am

hi,everyone!
when I test Hadoop-vertica-connect,I had some wrong.

Code: Select all

[root@h05 hadoop]# hadoop jar hadoop-vertica-example.jar \
>  com.vertica.hadoop.VerticaExample \
>  -Dmapred.vertica.hostnames=h02,h03,h04,h05 \
>  -Dmapred.vertica.port=5433 \
>  -Dmapred.vertica.username=dbadmin \
>  -Dmapred.vertica.password=test \
>  -Dmapred.vertica.database=test
13/09/03 23:43:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/09/03 23:43:57 INFO mapred.JobClient: Running job: job_201309032332_0001
13/09/03 23:43:58 INFO mapred.JobClient:  map 0% reduce 0%
13/09/03 23:44:07 INFO mapred.JobClient:  map 20% reduce 0%
13/09/03 23:44:08 INFO mapred.JobClient:  map 90% reduce 0%
13/09/03 23:44:10 INFO mapred.JobClient:  map 100% reduce 0%
13/09/03 23:44:11 INFO mapred.JobClient:  map 100% reduce 20%
13/09/03 23:44:12 INFO mapred.JobClient:  map 100% reduce 60%
13/09/03 23:44:17 INFO mapred.JobClient:  map 100% reduce 97%
13/09/03 23:44:20 INFO mapred.JobClient: Task Id : attempt_201309032332_0001_r_000000_0, Status : FAILED
java.io.IOException: java.sql.SQLException: [Vertica][VJDBC](100024) IOException while communicating with server: null. Refer to the inner exception for more detail.
        at com.vertica.hadoop.VerticaExample$Reduce.setup(VerticaExample.java:55)
        at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:162)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: [Vertica][VJDBC](100024) IOException while communicating with server: null. Refer to the inner exception for more detail.
        at com.vertica.io.ProtocolStream.readMessage(Unknown Source)
        at com.vertica.core.VConnection.handleErro
13/09/03 23:44:20 INFO mapred.JobClient: Task Id : attempt_201309032332_0001_r_000002_0, Status : FAILED
java.io.IOException: java.sql.SQLException: [Vertica][VJDBC](ERROR: Node failure during execution) {1}
        at com.vertica.hadoop.VerticaRecordWriter.close(VerticaRecordWriter.java:90)
        at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:611)
        at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.sql.SQLException: [Vertica][VJDBC](ERROR: Node failure during execution) {1}
        at com.vertica.dataengine.VQueryExecutor.checkForCancelError(Unknown Source)
        at com.vertica.dataengine.VQueryExecutor.handleExecuteResponse(Unknown Source)
        at com.vertica.dataengine.VQueryExecutor.execute(Unknown Source)
        at com.vertica.jdbc.SPreparedStatement.executeBatch(Unknown Sourc
13/09/03 23:44:21 INFO mapred.JobClient:  map 100% reduce 60%
What's wrong!help me!

muaythai_duan
Newbie
Newbie
Posts: 18
Joined: Sun Aug 25, 2013 11:43 am

Re: Hadoop Example in Programmer's Guide

Post by muaythai_duan » Tue Sep 03, 2013 10:57 am

I had new question!

Code: Select all

2013-09-04 01:17:59.030 DistCall Dispatch:0x7fab240d7820-a0000000000bfa <PANIC> @v_test_node0004: VX001/4064: NewPool::addChunk talloc mmap error: 'Cannot allocate memory'; size 2097152
        LOCATION:  addChunk, /scratch_a/release/vbuild/vertica/Basics/talloc.cpp:149
2013-09-04 01:17:59.037 DistCall Dispatch:0x7fab240d76e0-a0000000000bfb <PANIC> @v_test_node0004: VX001/4064: NewPool::addChunk talloc mmap error: 'Cannot allocate memory'; size 2097152
        LOCATION:  addChunk, /scratch_a/release/vbuild/vertica/Basics/talloc.cpp:149
2013-09-04 01:17:59.050 DistCall Dispatch:0x7fab240d6810-c0000000000fb2 <PANIC> @v_test_node0004: VX001/4064: NewPool::addChunk talloc mmap error: 'Cannot allocate memory'; size 2097152
        LOCATION:  addChunk, /scratch_a/release/vbuild/vertica/Basics/talloc.cpp:149
2013-09-04 01:17:59.115 DistCall Dispatch:0x7fab24057190-c0000000000fb5 <PANIC> @v_test_node0004: VX001/4064: NewPool::addChunk talloc mmap error: 'Cannot allocate memory'; size 2097152
        LOCATION:  addChunk, /scratch_a/release/vbuild/vertica/Basics/talloc.cpp:149
2013-09-04 01:17:59.230 DistCall Dispatch:0x7fab24009be0-b0000000001685 [Txn] <INFO> Starting Commit: Txn: b0000000001685 'select * from (select type_name as data_type_name, case odbc_type when 9 then odbc_subtype when 10 then 1111 else odbc_type end as data_type, column_size, case when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_prefix, case when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_suffix, creation_parameters as create_param, 1 as nullable, 1 as case_sensitive, 3 as searchable, 0 as unsigned_attribute, case odbc_type when 2 then 1 else 0 end as fixed_prec_scale, 0 as auto_unique, type_name as local_type_name, min_scale as minimum_scale, max_scale as maximum_scale, odbc_type as sql_data_type, odbc_subtype as sql_datetime_sub, 10 as num_prec_radix from v_catalog.types order by odbc_type, odbc_subtype) as vmd '
2013-09-04 01:17:59.273 DistCall Dispatch:0x7fab24055e80-c0000000000fb8 [Txn] <INFO> Starting Commit: Txn: c0000000000fb8 'select * from (select type_name as data_type_name, case odbc_type when 9 then odbc_subtype when 10 then 1111 else odbc_type end as data_type, column_size, case when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_prefix, case when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_suffix, creation_parameters as create_param, 1 as nullable, 1 as case_sensitive, 3 as searchable, 0 as unsigned_attribute, case odbc_type when 2 then 1 else 0 end as fixed_prec_scale, 0 as auto_unique, type_name as local_type_name, min_scale as minimum_scale, max_scale as maximum_scale, odbc_type as sql_data_type, odbc_subtype as sql_datetime_sub, 10 as num_prec_radix from v_catalog.types order by odbc_type, odbc_subtype) as vmd '
2013-09-04 01:17:59.368 DistCall Dispatch:0x7fab24009be0 [Txn] <INFO> Commit Complete: Txn: b0000000001685 at epoch 0x2b
2013-09-04 01:17:59.439 DistCall Dispatch:0x7fab24009be0-b0000000001687 [Txn] <INFO> Starting Commit: Txn: b0000000001687 'select * from allTypes where key = ?'
2013-09-04 01:17:59.486 DistCall Dispatch:0x7fab24055e80 [Txn] <INFO> Commit Complete: Txn: c0000000000fb8 at epoch 0x2b
2013-09-04 01:17:59.498 DistCall Dispatch:0x7fab24009be0 [Txn] <INFO> Commit Complete: Txn: b0000000001687 at epoch 0x2b
2013-09-04 01:17:59.587 DistCall Dispatch:0x7fab24055e80-c0000000000fba [Txn] <INFO> Starting Commit: Txn: c0000000000fba 'select * from allTypes where key = ?'
2013-09-04 01:17:59.588 DistCall Dispatch:0x7fab24055e80 [Txn] <INFO> Commit Complete: Txn: c0000000000fba at epoch 0x2b
2013-09-04 01:18:00.532 DistCall Dispatch:0x7fab240108e0-c0000000000fbd [Txn] <INFO> Starting Commit: Txn: c0000000000fbd 'select * from (select type_name as data_type_name, case odbc_type when 9 then odbc_subtype when 10 then 1111 else odbc_type end as data_type, column_size, case when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_prefix, case when odbc_type in (-5, 2, 8) then null else E'\'' end as literal_suffix, creation_parameters as create_param, 1 as nullable, 1 as case_sensitive, 3 as searchable, 0 as unsigned_attribute, case odbc_type when 2 then 1 else 0 end as fixed_prec_scale, 0 as auto_unique, type_name as local_type_name, min_scale as minimum_scale, max_scale as maximum_scale, odbc_type as sql_data_type, odbc_subtype as sql_datetime_sub, 10 as num_prec_radix from v_catalog.types order by odbc_type, odbc_subtype) as vmd '
2013-09-04 01:18:00.533 DistCall Dispatch:0x7fab240108e0 [Txn] <INFO> Commit Complete: Txn: c0000000000fbd at epoch 0x2b
2013-09-04 01:18:00.600 DistCall Dispatch:0x7fab240108e0-c0000000000fbf [Txn] <INFO> Starting Commit: Txn: c0000000000fbf 'select * from allTypes where key = ?'
2013-09-04 01:18:00.600 DistCall Dispatch:0x7fab240108e0 [Txn] <INFO> Commit Complete: Txn: c0000000000fbf at epoch 0x2b
2013-09-04 01:18:01.726 EEThread:0x7faae808aa10-b0000000001682 [Main] <PANIC> Wrote backtrace to ErrorReport.txt
how can I Assign a newpool memory size,Thanks!

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1825
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: Hadoop Example in Programmer's Guide

Post by JimKnicely » Tue Sep 03, 2013 4:31 pm

Hi,

Do you want to create a new resource pool or just alter a user's resource pool?
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

muaythai_duan
Newbie
Newbie
Posts: 18
Joined: Sun Aug 25, 2013 11:43 am

Re: Hadoop Example in Programmer's Guide

Post by muaythai_duan » Wed Sep 04, 2013 2:32 am

Re: Hadoop Example in Programmer's Guide
by knicely87 » Wed Sep 04, 2013 12:31 am

Hi,

Do you want to create a new resource pool or just alter a user's resource pool?

the newpool is created by vertica-hadoop-example.jar,I try to create new resource!

Post Reply

Return to “Hadoop Connector”