Error about pload!!!

Moderator: NorbertKrupa

Post Reply
muaythai_duan
Newbie
Newbie
Posts: 18
Joined: Sun Aug 25, 2013 11:43 am

Error about pload!!!

Post by muaythai_duan » Mon Oct 28, 2013 2:11 pm

The file is 2GB,I use pload to copy record into vertica,without any error!

Code: Select all

smith=> copy test.tmp_gn_pdp_src_t with source PloadDelimitedSource(file='/opt/disk1/drdata/gnpdp001.txt') delimiter '|' null '' direct;
 Rows Loaded 
-------------
     6114952
(1 row)
Time: First fetch (1 row): 47326.315 ms. All rows formatted: 47326.338 ms

smith=> select * from storage_containers;
    node_name     | schema_name |   projection_id   |   projection_name   | storage_type |    storage_oid    | total_row_count | deleted_row_count | used_bytes | start_epoch | end_epoch |  grouping  | segment_lower_bound | segment_upper_bound | is_sorted | location_label | delete_vector_count 
------------------+-------------+-------------------+---------------------+--------------+-------------------+-----------------+-------------------+------------+-------------+-----------+------------+---------------------+---------------------+-----------+----------------+---------------------
 v_smith_node0001 | test        | 45035996273720636 | tmp_gn_pdp_src_t_b1 | ROS          | 45035996273727717 |         1528733 |                 0 |  182757077 |          13 |        13 | PROJECTION |          3221225472 |          4294967295 | t         |                |                   0
 v_smith_node0001 | test        | 45035996273720502 | tmp_gn_pdp_src_t_b0 | ROS          | 45035996273727977 |         1527932 |                 0 |  182646367 |          13 |        13 | PROJECTION |                   0 |          1073741823 | t         |                |                   0
 v_smith_node0004 | test        | 45035996273720636 | tmp_gn_pdp_src_t_b1 | ROS          | 58546795155836565 |         1529905 |                 0 |  182892209 |          13 |        13 | PROJECTION |          2147483648 |          3221225471 | t         |                |                   0
 v_smith_node0004 | test        | 45035996273720502 | tmp_gn_pdp_src_t_b0 | ROS          | 58546795155836825 |         1528733 |                 0 |  182757077 |          13 |        13 | PROJECTION |          3221225472 |          4294967295 | t         |                |                   0
 v_smith_node0002 | test        | 45035996273720636 | tmp_gn_pdp_src_t_b1 | ROS          | 49539595901095573 |         1527932 |                 0 |  182646367 |          13 |        13 | PROJECTION |                   0 |          1073741823 | t         |                |                   0
 v_smith_node0002 | test        | 45035996273720502 | tmp_gn_pdp_src_t_b0 | ROS          | 49539595901095833 |         1528382 |                 0 |  182710634 |          13 |        13 | PROJECTION |          1073741824 |          2147483647 | t         |                |                   0
 v_smith_node0003 | test        | 45035996273720636 | tmp_gn_pdp_src_t_b1 | ROS          | 54043195528466069 |         1528382 |                 0 |  182710634 |          13 |        13 | PROJECTION |          1073741824 |          2147483647 | t         |                |                   0
 v_smith_node0003 | test        | 45035996273720502 | tmp_gn_pdp_src_t_b0 | ROS          | 54043195528466329 |         1529905 |                 0 |  182892209 |          13 |        13 | PROJECTION |          2147483648 |          3221225471 | t         |                |                   0
(8 rows)
I change the file to 4GB,the vertica report error:

Code: Select all

vertica.log:
2013-07-01 07:40:13.835 EEThread:0x7ff50c6152a0-b00000000001ee <PANIC> @v_load_test_node0001: VX001/4064: NewPool::addChunk talloc mmap error: 'Cannot allocate memory'; size 8388608
2013-07-01 07:40:14.000 Timer Service:0x5dc68e0 <ERROR> @v_load_test_node0001: {run} 53000/5001: ThreadManager failed to create thread Cluster Inviter: boost::thread_resource_error
        HINT:  Please raise the number of processes per user
        LOCATION:  createThread, /scratch_a/release/vbuild/vertica/Session/ThreadManager.cpp:76
2013-07-01 07:40:15.000 Timer Service:0x5dc68e0 <ERROR> @v_load_test_node0001: {run} 53000/5001: ThreadManager failed to create thread WOSReaper: boost::thread_resource_error
        HINT:  Please raise the number of processes per user
        LOCATION:  createThread, /scratch_a/release/vbuild/vertica/Session/ThreadManager.cpp:76
2013-07-01 07:40:15.000 Timer Service:0x5dc68e0 <ERROR> @v_load_test_node0001: {run} 53000/5001: ThreadManager failed to create thread AgeOutEvents: boost::thread_resource_error
        HINT:  Please raise the number of processes per user
        LOCATION:  createThread, /scratch_a/release/vbuild/vertica/Session/ThreadManager.cpp:76
2013-07-01 07:40:15.000 Timer Service:0x5dc68e0 <ERROR> @v_load_test_node0001: {run} 53000/5001: ThreadManager failed to create thread UDxSideProcessMonitor: boost::thread_resource_error
        HINT:  Please raise the number of processes per user
        LOCATION:  createThread, /scratch_a/release/vbuild/vertica/Session/ThreadManager.cpp:76
2013-07-01 07:40:16.000 Timer Service:0x5dc68e0 <ERROR> @v_load_test_node0001: {run} 53000/5001: ThreadManager failed to create thread LogFlusher: boost::thread_resource_error
        HINT:  Please raise the number of processes per user
        LOCATION:  createThread, /scratch_a/release/vbuild/vertica/Session/ThreadManager.cpp:76
2013-07-01 07:40:29.295 EEThread:0x7ff50c6152a0-b00000000001ee [Main] <PANIC> Wrote backtrace to ErrorReport.txt

Code: Select all

ErrorReport.txt
Backtrace Requested
/opt/vertica/bin/vertica(_ZN6Basics9Backtrace11DoBacktraceEiiPvS1_+0x8cc)[0x2ab5aac]
/opt/vertica/bin/vertica(_ZN6Basics20GlobalSignalHandlers20logBacktraceFromHereEb+0xa6)[0x2b3055a]
/opt/vertica/bin/vertica(errthrow+0x5c)[0x2e533b6]
/opt/vertica/bin/vertica(_ZN6Basics7NewPool5allocEm+0x1d0)[0x2b2b51c]
/opt/vertica/bin/vertica(_Z7eeAllocm+0xc2)[0x1fd63b2]
/opt/vertica/bin/vertica(_ZN3SAL3SAL18allocCompressorForEN3CAT15CompressionTypeEjj+0x387)[0x29c7a09]
/opt/vertica/bin/vertica(_ZN2EE9SortQueue18sampleAndSetUpSortEPhPNS_8DataAreaEi+0x633)[0x2094b59]
/opt/vertica/bin/vertica(_ZN2EE11SortManager13queueSortTaskEv+0x1d8)[0x2015d1c]
/opt/vertica/bin/vertica(_ZN2EE11SortManager12ingestTuplesEPNS_15BlockDescriptorEyb+0x469)[0x201739b]
/opt/vertica/bin/vertica(_ZN2EE10DataTarget7runSortEv+0xe1)[0x2064627]
/opt/vertica/bin/vertica(_ZN2EE10DataTarget3runEv+0x105)[0x2064e0f]
/opt/vertica/bin/vertica(_ZN2EE9NOperator7executeEv+0x10)[0x20136d0]
/opt/vertica/bin/vertica(_ZN2EE2EE11runOperatorEv+0x152)[0x1fc4ffa]
/opt/vertica/bin/vertica(_ZN2EE2EE13startEEThreadEv+0x1f6)[0x1fc7ce4]
/opt/vertica/bin/vertica(_ZN2EE2EE15runOperatorShimEPS0_+0x2a3)[0x1fcad0d]
/opt/vertica/bin/vertica(_ZNK5boost9function0IvEclEv+0x1bb)[0x159607b]
/opt/vertica/bin/vertica(_ZN7Session13ThreadManager12launchThreadERKN5boost9function0IvEE+0x57)[0x15926ab]
/opt/vertica/bin/vertica(thread_proxy+0x80)[0x3457b90]
/lib64/libpthread.so.0[0x36daa07851]
/lib64/libc.so.6(clone+0x6d)[0x36da6e890d]
END BACKTRACE
THREAD CONTEXT
Thread type: EE Worker Thread
Request: copy test.tmp_gn_pdp_src_t with source PloadDelimitedSource(file='/opt/disk1/drdata/gnpdp.txt') delimiter '|' null '' direct;
          87: Root
           5: GroupByNothing
           4: Union
           6: Union
FAULT =>  20: DataTarget
           1: Router
           0: ValExpr
          77: Copy
          88: NewEENode
              ExprEval
              LoadUnion
              ExprEval
              Load
              ExprEval
              Load
              ExprEval
              Load
              ExprEval
              Load
Transaction: [0x00b00000000001ee]
END THREAD CONTEXT
what can I do?
linux parameter:
[root@h02 vm]# more /proc/sys/vm/max_map_count
755127
[root@h02 vm]# more /proc/sys/kernel/threads-max
[root@h02 kernel]# more /proc/sys/kernel/pid_max
655127
4 core cpu 3.3GB,12GB memery.

User avatar
Julie
Master
Master
Posts: 221
Joined: Thu Apr 19, 2012 9:29 pm

Re: Error about pload!!!

Post by Julie » Tue Oct 29, 2013 12:35 pm

Thanks,
Juliette

muaythai_duan
Newbie
Newbie
Posts: 18
Joined: Sun Aug 25, 2013 11:43 am

Re: Error about pload!!!

Post by muaythai_duan » Wed Oct 30, 2013 7:25 am

Julie wrote:Maybe this link will help?

http://dustycodes.wordpress.com/2012/02 ... r-process/
Hi,Julie
Thank for you reply.I had changed the linux parameters,for example
linux parameter:
[root@h02 vm]# more /proc/sys/vm/max_map_count
755127
[root@h02 kernel]# more /proc/sys/kernel/pid_max
655127
end,I create a new 2GB resource pool.I copy the data is ok!

Post Reply

Return to “Vertica Data Load”