Vertica Nodes Randomly Fail

Moderator: NorbertKrupa

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Vertica Nodes Randomly Fail

Post by becky » Fri Jul 19, 2013 6:51 pm

Scutter,

Thanks a bunch for you help in this thread. I decided to drop all the nodes and use Red Hat as opposed to CentOS.

This time when I did the install I used the -T -S default options.

So far, the cluster has stayed up longer than ever on CentOS.

I'll keep you posted!
THANKS - BECKSTER

scutter
Master
Master
Posts: 302
Joined: Tue Aug 07, 2012 2:15 am

Re: Vertica Nodes Randomly Fail

Post by scutter » Fri Jul 19, 2013 7:10 pm

Glad to hear that the cluster is more stable. For the record, CentOS should be fine for Vertica.

--Sharon
Sharon Cutter
Vertica Consultant, Zazz Technologies LLC

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Vertica Nodes Randomly Fail

Post by becky » Fri Jul 19, 2013 7:13 pm

Hi Scutter,

I totally agree that CentOS work's with Vertica. I've set it up many times with out incident. Maybe it was the VM template I was using. It was not provided to me from the IT department.

Anyway, I used Jim's query (http://www.vertica-forums.com/viewtopic ... 1194#p3818) to find out how long the db nodes have been up:

dbadmin=> SELECT n.node_name, n.node_state, n.node_address, sysdate - MAX(ns.event_timestamp) AS uptime
dbadmin-> FROM node_states ns JOIN nodes n ON n.node_name = ns.node_name
dbadmin-> WHERE ns.node_state = 'UP'
dbadmin-> GROUP BY n.node_name, n.node_state, n.node_address
dbadmin-> ORDER BY n.node_name;
node_name | node_state | node_address | uptime
---------------------+------------+-----------------+-----------------
v_testdb_node0001 | UP | XXX.XXX.XXX.181 | 02:26:15.074865
v_testdb_node0002 | UP | XXX.XXX.XXX.182 | 02:26:09.016887
v_testdb_node0003 | UP | XXX.XXX.XXX.183 | 02:26:05.000407
(3 rows)

Cool! Nodes would go down in < 15 minutes before I switched to Red Hat.
THANKS - BECKSTER

scutter
Master
Master
Posts: 302
Joined: Tue Aug 07, 2012 2:15 am

Re: Vertica Nodes Randomly Fail

Post by scutter » Fri Jul 19, 2013 7:39 pm

> Nodes would go down in < 15 minutes before I switched to Red Hat.

Switched to Red Hat and changed to use -T for spread configuration, to be clear for anybody revisiting this topic later.

Regardless - glad that the nodes are stable now!
Sharon Cutter
Vertica Consultant, Zazz Technologies LLC

vcarusi
Beginner
Beginner
Posts: 29
Joined: Mon Apr 20, 2015 11:03 am

Re: Vertica Nodes Randomly Fail

Post by vcarusi » Mon May 16, 2016 1:15 pm

Hi,
I have a similar issue with a cluster, 3 nodes,installed 1 year ago. For 3 times a node shutdown.

2016-05-16 01:23:26.005 Init Session:0x7ff3b8023460 [Session] <INFO> [Query] TX:0(verticanode1-22017:0x2fc749) select 1 /*jdbc Connection.isValid() check*/
2016-05-16 01:23:26.057 Init Session:0x7ff3b8021a50 [Session] <INFO> [Query] TX:0(verticanode1-22017:0x2fc866) select 1 /*jdbc Connection.isValid() check*/

2016-05-16 01:23:26.296 TM Mergeout(00):0x7ff3c8014d00-a0000000e27579 [TM] <INFO> Tuple Mover: nothing to merge out
2016-05-16 01:23:26.316 TM Mergeout(00):0x7ff3c8014d00-a0000000e27579 [TM] <INFO> Tuple Mover: no DV to merge out
2016-05-16 01:23:26.316 TM Mergeout(00):0x7ff3c8014d00-a0000000e27579 [Txn] <INFO> Rollback Txn: a0000000e27579 'Mergeout: Tuple Mover'
2016-05-16 01:23:26.318 TM Mergeout(01):0x7ff3c8014a80-a0000000e2757a [TM] <INFO> Tuple Mover: nothing to merge out
2016-05-16 01:23:26.329 TM Mergeout(01):0x7ff3c8014a80-a0000000e2757a [TM] <INFO> Tuple Mover: no DV to merge out
2016-05-16 01:23:26.329 TM Mergeout(01):0x7ff3c8014a80-a0000000e2757a [Txn] <INFO> Rollback Txn: a0000000e2757a 'Mergeout: Tuple Mover'
2016-05-16 01:23:26.343 TM Mergeout(00):0x7ff3c8014d00-a0000000e27582 [Txn] <INFO> Begin Txn: a0000000e27582 'collectMoveStorageJobs'
2016-05-16 01:23:26.358 TM Mergeout(01):0x7ff3c8014a80 [Util] <INFO> Task 'TM Mergeout(01)' enabled
2016-05-16 01:23:26.358 TM Mergeout(00):0x7ff3c8014d00-a0000000e27582 [Txn] <INFO> Rollback Txn: a0000000e27582 'collectMoveStorageJobs'
2016-05-16 01:23:26.359 TM Mergeout(00):0x7ff3c8014d00 [Util] <INFO> Task 'TM Mergeout(00)' enabled

2016-05-16 01:23:28.312 Init Session:0x7ff3b80213e0 <LOG> @v_mydb_node0001: 00000/2705: Connection received: host=99.999.99.99 port=36429 (connCnt 20)
2016-05-16 01:23:28.312 Init Session:0x7ff3b80213e0 <LOG> @v_mydb_node0001: 00000/4540: Received SSL negotiation startup packet
2016-05-16 01:23:28.312 Init Session:0x7ff3b80213e0 <LOG> @v_mydb_node0001: 00000/4691: Sending SSL negotiation response 'N'
2016-05-16 01:23:28.323 Init Session:0x7ff3b80213e0 <FATAL> @v_mydb_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 99.999.99.99:36429

2016-05-16 01:23:29.368 Init Session:0x7ff3b801da00 <LOG> @v_mydb_node0001: 00000/2705: Connection received: host=99.999.99.99 port=36445 (connCnt 20)
2016-05-16 01:23:29.368 Init Session:0x7ff3b801da00 <LOG> @v_mydb_node0001: 00000/4540: Received SSL negotiation startup packet
2016-05-16 01:23:29.368 Init Session:0x7ff3b801da00 <LOG> @v_mydb_node0001: 00000/4691: Sending SSL negotiation response 'N'
2016-05-16 01:23:29.368 Init Session:0x7ff3b801da00 <FATAL> @v_mydb_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 99.999.99.99:36445

2016-05-16 01:23:29.388 Init Session:0x7ff3b801dbb0 <LOG> @v_mydb_node0001: 00000/2705: Connection received: host=99.999.99.99 port=36447 (connCnt 20)
2016-05-16 01:23:29.388 Init Session:0x7ff3b801dbb0 <LOG> @v_mydb_node0001: 00000/4540: Received SSL negotiation startup packet
2016-05-16 01:23:29.388 Init Session:0x7ff3b801dbb0 <LOG> @v_mydb_node0001: 00000/4691: Sending SSL negotiation response 'N'
2016-05-16 01:23:29.388 Init Session:0x7ff3b801dbb0 <FATAL> @v_mydb_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 99.999.99.99:36447

2016-05-16 01:23:29.921 Init Session:0x7ff3b801b4d0 <LOG> @v_mydb_node0001: 00000/2705: Connection received: host=99.999.99.99 port=36458 (connCnt 20)
2016-05-16 01:23:29.921 Init Session:0x7ff3b801b4d0 <LOG> @v_mydb_node0001: 00000/4540: Received SSL negotiation startup packet
2016-05-16 01:23:29.921 Init Session:0x7ff3b801b4d0 <LOG> @v_mydb_node0001: 00000/4691: Sending SSL negotiation response 'N'
2016-05-16 01:23:29.921 Init Session:0x7ff3b801b4d0 <FATAL> @v_mydb_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 99.999.99.99:36458

2016-05-16 01:23:30.454 Init Session:0x7ff3b801d0e0 <LOG> @v_mydb_node0001: 00000/2705: Connection received: host=99.999.99.99 port=36468 (connCnt 20)
2016-05-16 01:23:30.454 Init Session:0x7ff3b801d0e0 <LOG> @v_mydb_node0001: 00000/4540: Received SSL negotiation startup packet
2016-05-16 01:23:30.454 Init Session:0x7ff3b801d0e0 <LOG> @v_mydb_node0001: 00000/4691: Sending SSL negotiation response 'N'
2016-05-16 01:23:30.454 Init Session:0x7ff3b801d0e0 <FATAL> @v_mydb_node0001: {SessionRun} 57V03/5785: Cluster Status Request by 99.999.99.99:36468

2016-05-16 01:23:34.548 Spread Client:0x8669a50 [Comms] <INFO> Saw membership message 8192 on V:mydb
2016-05-16 01:23:34.548 Spread Client:0x8669a50 [Comms] <INFO> Saw transitional message; watch for lost daemons
2016-05-16 01:23:34.548 Spread Client:0x8669a50 [Comms] <INFO> Saw membership message 8192 on Vertica:all
2016-05-16 01:23:34.548 Spread Client:0x8669a50 [Comms] <INFO> Saw transitional message; watch for lost daemons
2016-05-16 01:23:34.548 Spread Client:0x8669a50 [Comms] <INFO> Saw membership message 8192 on Vertica:join
2016-05-16 01:23:34.548 Spread Client:0x8669a50 [Comms] <INFO> Saw transitional message; watch for lost daemons
2016-05-16 01:23:34.562 Spread Client:0x8669a50 [Comms] <INFO> Saw membership message 6144 on V:mydb
2016-05-16 01:23:34.562 Spread Client:0x8669a50 [Comms] <INFO> NETWORK change with 2 VS sets
2016-05-16 01:23:34.562 Spread Client:0x8669a50 [Comms] <INFO> DB Group changed

2016-05-16 01:23:34.563 Spread Client:0x8669a50 [VMPI] <INFO> DistCall: Set current group members called with 1 members
2016-05-16 01:23:34.563 Spread Client:0x8669a50 [VMPI] <INFO> Removing 45035996273720622 from list of initialized nodes for session verticanode1-22017:0x18a358
2016-05-16 01:23:34.563 Spread Client:0x8669a50 [VMPI] <INFO> Removing 45035996273720626 from list of initialized nodes for session verticanode1-22017:0x308fb0
2016-05-16 01:23:34.563 Spread Client:0x8669a50 [VMPI] <INFO> Ending session verticanode2-10186:0x3473ce due to loss of 45035996273720622
2016-05-16 01:23:34.564 Spread Client:0x8669a50 [VMPI] <INFO> Removing 45035996273720622 from list of initialized nodes for session verticanode1-22017:0x34bb5d
2016-05-16 01:23:34.564 Spread Client:0x8669a50 [VMPI] <INFO> Removing 45035996273720626 from list of initialized nodes for session verticanode1-22017:0x34bb5d
2016-05-16 01:23:34.564 Spread Client:0x8669a50 [Comms] <INFO> nodeSetNotifier: node v_mydb_node0002 left the cluster
2016-05-16 01:23:34.565 AnalyzeRowCount:0x7ff3c8013be0 <WARNING> @v_mydb_node0001: 01000/4539: Received no response from v_mydb_node0002 in GetRowCountMessage
2016-05-16 01:23:34.567 Spread Client:0x8669a50 [Recover] <INFO> Node left cluster, reassessing k-safety...
2016-05-16 01:23:34.568 AnalyzeRowCount:0x7ff3c8013be0 <WARNING> @v_mydb_node0001: 01000/4539: Received no response from v_mydb_node0003 in GetRowCountMessage
2016-05-16 01:23:34.569 Spread Client:0x8669a50 [Recover] <INFO> Checking Deps:Down bits: 010 Deps:
2016-05-16 01:23:34.571 Spread Client:0x8669a50 <LOG> @v_mydb_node0001: 00000/3298: Event Posted: Event Code:3 Event Id:0 Event Severity: Critical [2] PostedTimestamp: 2016-05-16 01:23:34.569099 ExpirationTimestamp: 2084-06-03 04:37:41.569099 EventCodeDescription: Current Fault Tolerance at Critical Level ProblemDescription: Loss of node v_mydb_node0001 will cause shutdown to occur. K=1 total number of nodes=3 DatabaseName: mydb Hostname: verticanode1
2016-05-16 01:23:34.571 Spread Client:0x8669a50 [Comms] <INFO> nodeSetNotifier: node v_mydb_node0003 left the cluster
2016-05-16 01:23:34.571 Spread Client:0x8669a50 [Recover] <INFO> Node left cluster, reassessing k-safety...
2016-05-16 01:23:34.571 Spread Client:0x8669a50 [Recover] <INFO> Cluster partitioned: 3 total nodes, 1 up nodes, 2 down nodes
2016-05-16 01:23:34.571 Spread Client:0x8669a50 [Recover] <INFO> Setting node v_mydb_node0001 to UNSAFE
2016-05-16 01:23:34.577 Spread Client:0x8669a50 <LOG> @v_mydb_node0001: 00000/3298: Event Posted: Event Code:6 Event Id:5 Event Severity: Informational [6] PostedTimestamp: 2016-05-16 01:23:34.57123 ExpirationTimestamp: 2084-06-03 04:37:41.57123 EventCodeDescription: Node State Change ProblemDescription: Changing node v_mydb_node0001 startup state to UNSAFE DatabaseName: mydb Hostname: verticanode1
2016-05-16 01:23:34.577 Spread Client:0x8669a50 <LOG> @v_mydb_node0001: 00000/3293: Event Cleared: Event Code:6 Event Id:6 Event Severity: Informational [6] PostedTimestamp: 2016-05-16 01:23:34.577322 ExpirationTimestamp: 2016-05-16 01:23:34.577322 EventCodeDescription: Node State Change ProblemDescription: Changing node v_mydb_node0001 leaving startup state UP DatabaseName: mydb Hostname: verticanode1
2016-05-16 01:23:34.577 Spread Client:0x8669a50 [Recover] <INFO> Changing node v_mydb_node0001 startup state from UP to UNSAFE
2016-05-16 01:23:34.577 Spread Client:0x8669a50 <LOG> @v_mydb_node0001: 00000/3298: Event Posted: Event Code:2 Event Id:0 Event Severity: Emergency [0] PostedTimestamp: 2016-05-16 01:23:34.577492 ExpirationTimestamp: 2016-05-16 01:33:34.577492 EventCodeDescription: Loss Of K Safety ProblemDescription: System is not K-safe: K=1 total number of nodes=3 DatabaseName: mydb Hostname: verticanode1
2016-05-16 01:23:34.577 Spread Client:0x8669a50 [Comms] <INFO> stop: disconnecting #node_a#N010164030020 from spread daemon, Mbox=9
2016-05-16 01:23:34.578 Spread Client:0x8669a50 [Comms] <INFO> Vertica pid=22017; found spread pid=22015 from pidfile /D/vertica/databases/catalog/mydb/v_mydb_node0001_catalog/spread.pid
2016-05-16 01:23:34.586 SafetyShutdown:0x7ff3c4006cc0 [Shutdown] <INFO> Shutting down this node
2016-05-16 01:23:34.586 SafetyShutdown:0x7ff3c4006cc0-fff00000000005c0 [Txn] <INFO> Begin Txn: fff00000000005c0 'prepareShutdown'
2016-05-16 01:23:34.588 RebalanceCluster:0x7ff3c8012990-a0000000e27577 [TM] <WARNING> Rebalance cluster (background) failed: Received no response from v_mydb_node0002, v_mydb_node0003 in transaction bind
2016-05-16 01:23:34.588 SafetyShutdown:0x7ff3c4006cc0-fff00000000005c0 <LOG> @v_mydb_node0001: 00000/3298: Event Posted: Event Code:6 Event Id:8 Event Severity: Informational [6] PostedTimestamp: 2016-05-16 01:23:34.586988 ExpirationTimestamp: 2084-06-03 04:37:41.586988 EventCodeDescription: Node State Change ProblemDescription: Changing node v_mydb_node0001 startup state to SHUTDOWN DatabaseName: mydb Hostname: verticanode1
2016-05-16 01:23:34.589 SafetyShutdown:0x7ff3c4006cc0-fff00000000005c0 <LOG> @v_mydb_node0001: 00000/3293: Event Cleared: Event Code:6 Event Id:5 Event Severity: Informational [6] PostedTimestamp: 2016-05-16 01:23:34.588941 ExpirationTimestamp: 2016-05-16 01:23:34.588941 EventCodeDescription: Node State Change ProblemDescription: Changing node v_mydb_node0001 leaving startup state UNSAFE DatabaseName: mydb Hostname: verticanode1
2016-05-16 01:23:34.589 SafetyShutdown:0x7ff3c4006cc0-fff00000000005c0 [Recover] <INFO> Changing node v_mydb_node0001 startup state from UNSAFE to SHUTDOWN
2016-05-16 01:23:34.597 Init Session:0x7ff3b8023460-a0000000e27580 <ERROR> @v_mydb_node0001: 08006/4539: Received no response from v_mydb_node0002, v_mydb_node0003 in transaction bind
2016-05-16 01:23:34.597 Init Session:0x7ff3b80192e0-a0000000e2757e <ERROR> @v_mydb_node0001: 08006/4539: Received no response from v_mydb_node0002, v_mydb_node0003 in transaction bind
2016-05-16 01:23:34.597 Init Session:0x7ff3b8021a50-a0000000e27581 <ERROR> @v_mydb_node0001: 08006/4539: Received no response from v_mydb_node0002, v_mydb_node0003 in transaction bind
2016-05-16 01:23:34.597 Init Session:0x7ff3b80126e0-a0000000e2757d <ERROR> @v_mydb_node0001: 08006/4539: Received no response from v_mydb_node0002, v_mydb_node0003 in transaction bind
2016-05-16 01:23:35.586 Spread Client:0x8669a50 [Comms] <INFO> Spread daemon pid=22015 has terminated
2016-05-16 01:23:35.586 Spread Client:0x8669a50 [Comms] <INFO> connected: false
2016-05-16 01:23:35.586 Spread Client:0x8669a50 [Comms] <INFO> DB Group changed




node_name |start_timestamp_utc |request |success
v_mydb_node0001 |2016-05-16 01:23:35 |select 1 /*jdbc Connection.isValid() check*/ |false
v_mydb_node0001 |2016-05-16 01:23:35 |select 1 /*jdbc Connection.isValid() check*/ |false
v_mydb_node0001 |2016-05-16 01:23:35 |select 1 /*jdbc Connection.isValid() check*/ |false
v_mydb_node0003 |2016-05-16 01:23:28 |select * from v_internal.vs_nodes; |true

Any ideas ? What can be the cause ?
Thank you,
Veronica
Last edited by vcarusi on Mon May 16, 2016 2:58 pm, edited 2 times in total.

vcarusi
Beginner
Beginner
Posts: 29
Joined: Mon Apr 20, 2015 11:03 am

Re: Vertica Nodes Randomly Fail

Post by vcarusi » Mon May 16, 2016 2:22 pm

The spread.conf seems to be OK.
Also, I don't think it is an installation problem. The cluster works 95% of the time very well.
OS: Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-48-generic x86_64)

I restarted the node.
After that, when I wanted to retrieve data from my tables, connecting to the node just restarted, I obtained the following error, :
ROLLBACK 4236: One or more nodes did not open a data connection to this node. This may indicate a network configuration problem. Check that the private interfaces used for communication among the cluster hosts reside in the same subnet and are returned first by host address lookup

But, the node responded well if I queried v_catalog tables.
The others nodes worked well.
Finally, I had to restart entire database/cluster.

Each node, has the own system tables ?
Thank you,
Veronica

vcarusi
Beginner
Beginner
Posts: 29
Joined: Mon Apr 20, 2015 11:03 am

Re: Vertica Nodes Randomly Fail

Post by vcarusi » Tue May 17, 2016 9:15 am

I found in syslog:

May 16 01:00:11 verticanode1 kernel: [7745522.508054] INFO: task jbd2/xvda1-8:259 blocked for more than 120 seconds.
May 16 01:00:11 verticanode1 kernel: [7745522.511461] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:00:11 verticanode1 kernel: [7745522.514210] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:18:00 verticanode1 dhclient: DHCPREQUEST of 10.164.30.20 on eth0 to 10.164.30.1 port 67 (xid=0x556f4ef3)
May 16 01:18:00 verticanode1 dhclient: DHCPACK of 10.164.30.20 from 10.164.30.1
May 16 01:18:00 verticanode1 dhclient: bound to 10.164.30.20 – renewal in 1752 seconds.
May 16 01:23:17 verticanode1 kernel: [7745522.518139] jbd2/xvda1-8 D ffff88078fcf34c0 0 259 2 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745522.518143] ffff88076358fbd8 0000000000000046 ffff880763840000 ffff88076358ffd8
May 16 01:23:17 verticanode1 kernel: [7745522.518146] 00000000000134c0 00000000000134c0 ffff880763840000 ffff88078fcf3d58
May 16 01:23:17 verticanode1 kernel: [7745522.518148] ffff88078ff98478 0000000000000002 ffffffff811ef790 ffff88076358fc50
May 16 01:23:17 verticanode1 kernel: [7745522.518150] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745522.518158] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745522.518162] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745522.518165] [<ffffffff811ef79e>] sleep_on_buffer+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745522.518167] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.518169] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745522.518172] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.518176] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.518179] [<ffffffff811f0aca>] __wait_on_buffer+0x2a/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.518182] [<ffffffff8128adb0>] jbd2_journal_commit_transaction+0xf30/0x1ab0
May 16 01:23:17 verticanode1 kernel: [7745522.518186] [<ffffffff810755bf>] ? try_to_del_timer_sync+0x4f/0x70
May 16 01:23:17 verticanode1 kernel: [7745522.518189] [<ffffffff8128fa0d>] kjournald2+0xbd/0x250
May 16 01:23:17 verticanode1 kernel: [7745522.518192] [<ffffffff810ab120>] ? prepare_to_wait_event+0x100/0x100
May 16 01:23:17 verticanode1 kernel: [7745522.518194] [<ffffffff8128f950>] ? commit_timeout+0x10/0x10
May 16 01:23:17 verticanode1 kernel: [7745522.518197] [<ffffffff8108b5b2>] kthread+0xd2/0xf0
May 16 01:23:17 verticanode1 kernel: [7745522.518199] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745522.518202] [<ffffffff81731f0c>] ret_from_fork+0x7c/0xb0
May 16 01:23:17 verticanode1 kernel: [7745522.518204] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745522.518206] INFO: task jbd2/xvdb-8:416 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745522.521383] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745522.524022] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745522.527895] jbd2/xvdb-8 D ffff88078fc134c0 0 416 2 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745522.527898] ffff8807638f1bd8 0000000000000046 ffff880763891800 ffff8807638f1fd8
May 16 01:23:17 verticanode1 kernel: [7745522.527900] 00000000000134c0 00000000000134c0 ffff880763891800 ffff88078fc13d58
May 16 01:23:17 verticanode1 kernel: [7745522.527902] ffff88078ffabc30 0000000000000002 ffffffff811ef790 ffff8807638f1c50
May 16 01:23:17 verticanode1 kernel: [7745522.527904] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745522.527909] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745522.527912] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745522.527914] [<ffffffff811ef79e>] sleep_on_buffer+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745522.527916] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.527918] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745522.527921] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.527924] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.527926] [<ffffffff811f0aca>] __wait_on_buffer+0x2a/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.527929] [<ffffffff8128adb0>] jbd2_journal_commit_transaction+0xf30/0x1ab0
May 16 01:23:17 verticanode1 kernel: [7745522.527932] [<ffffffff810755bf>] ? try_to_del_timer_sync+0x4f/0x70
May 16 01:23:17 verticanode1 kernel: [7745522.527935] [<ffffffff8128fa0d>] kjournald2+0xbd/0x250
May 16 01:23:17 verticanode1 kernel: [7745522.527937] [<ffffffff810ab120>] ? prepare_to_wait_event+0x100/0x100
May 16 01:23:17 verticanode1 kernel: [7745522.527940] [<ffffffff8128f950>] ? commit_timeout+0x10/0x10
May 16 01:23:17 verticanode1 kernel: [7745522.527942] [<ffffffff8108b5b2>] kthread+0xd2/0xf0
May 16 01:23:17 verticanode1 kernel: [7745522.527944] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745522.527946] [<ffffffff81731f0c>] ret_from_fork+0x7c/0xb0
May 16 01:23:17 verticanode1 kernel: [7745522.527948] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745522.527953] INFO: task java:1387 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745522.530926] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745522.533579] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745522.537577] java D ffff88078fc534c0 0 1387 1 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745522.537579] ffff880764cbd9c8 0000000000000082 ffff8807651fe000 ffff880764cbdfd8
May 16 01:23:17 verticanode1 kernel: [7745522.537582] 00000000000134c0 00000000000134c0 ffff8807651fe000 ffff88078fc53d58
May 16 01:23:17 verticanode1 kernel: [7745522.537584] ffff88078ffa01e8 0000000000000002 ffffffff81287630 ffff880764cbda40
May 16 01:23:17 verticanode1 kernel: [7745522.537586] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745522.537590] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745522.537592] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745522.537595] [<ffffffff8128763e>] sleep_on_shadow_bh+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745522.537597] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.537599] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745522.537601] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.537604] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.537607] [<ffffffff810aad35>] ? wake_up_bit+0x25/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.537609] [<ffffffff81288a5d>] do_get_write_access+0x2ad/0x4f0
May 16 01:23:17 verticanode1 kernel: [7745522.537612] [<ffffffff811f109d>] ? __getblk+0x2d/0x2e0
May 16 01:23:17 verticanode1 kernel: [7745522.537614] [<ffffffff81288cc7>] jbd2_journal_get_write_access+0x27/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.537618] [<ffffffff8126e9cb>] __ext4_journal_get_write_access+0x3b/0x80
May 16 01:23:17 verticanode1 kernel: [7745522.537622] [<ffffffff81243d20>] ext4_reserve_inode_write+0x70/0xa0
May 16 01:23:17 verticanode1 kernel: [7745522.537624] [<ffffffff81247010>] ? ext4_dirty_inode+0x40/0x60
May 16 01:23:17 verticanode1 kernel: [7745522.537627] [<ffffffff81243d94>] ext4_mark_inode_dirty+0x44/0x1f0
May 16 01:23:17 verticanode1 kernel: [7745522.537629] [<ffffffff81247010>] ext4_dirty_inode+0x40/0x60
May 16 01:23:17 verticanode1 kernel: [7745522.537631] [<ffffffff811e7cba>] __mark_inode_dirty+0x10a/0x2d0
May 16 01:23:17 verticanode1 kernel: [7745522.537635] [<ffffffff811d8cc1>] update_time+0x81/0xd0
May 16 01:23:17 verticanode1 kernel: [7745522.537638] [<ffffffff811c0559>] ? __sb_start_write+0x49/0xe0
May 16 01:23:17 verticanode1 kernel: [7745522.537641] [<ffffffff811d8ed0>] file_update_time+0x80/0xd0
May 16 01:23:17 verticanode1 kernel: [7745522.537643] [<ffffffff812471f5>] ext4_page_mkwrite+0x55/0x410
May 16 01:23:17 verticanode1 kernel: [7745522.537646] [<ffffffff81177f17>] do_wp_page+0x637/0x7c0
May 16 01:23:17 verticanode1 kernel: [7745522.537649] [<ffffffff81364781>] ? cpumask_any_but+0x31/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.537651] [<ffffffff8117a10c>] handle_mm_fault+0x65c/0xf10
May 16 01:23:17 verticanode1 kernel: [7745522.537654] [<ffffffff8172d534>] __do_page_fault+0x184/0x560
May 16 01:23:17 verticanode1 kernel: [7745522.537657] [<ffffffff811826f5>] ? change_protection+0x65/0xb0
May 16 01:23:17 verticanode1 kernel: [7745522.537659] [<ffffffff81182891>] ? mprotect_fixup+0x151/0x290
May 16 01:23:17 verticanode1 kernel: [7745522.537663] [<ffffffff81009ee0>] ? xen_clocksource_get_cycles+0x20/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.537666] [<ffffffff810cd8da>] ? __getnstimeofday+0x3a/0xc0
May 16 01:23:17 verticanode1 kernel: [7745522.537668] [<ffffffff8172d92a>] do_page_fault+0x1a/0x70
May 16 01:23:17 verticanode1 kernel: [7745522.537670] [<ffffffff81729d68>] page_fault+0x28/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.537682] INFO: task vertica:22019 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745522.540882] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745522.543541] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745522.547430] vertica D ffff88078fc534c0 0 22019 1 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745522.547432] ffff88076357da28 0000000000000082 ffff880108a74800 ffff88076357dfd8
May 16 01:23:17 verticanode1 kernel: [7745522.547435] 00000000000134c0 00000000000134c0 ffff880108a74800 ffff88078fc53d58
May 16 01:23:17 verticanode1 kernel: [7745522.547437] ffff88078ff9a9f8 0000000000000002 ffffffff81287630 ffff88076357daa0
May 16 01:23:17 verticanode1 kernel: [7745522.547439] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745522.547442] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745522.547445] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745522.547447] [<ffffffff8128763e>] sleep_on_shadow_bh+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745522.547449] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.547451] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745522.547454] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.547456] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.547459] [<ffffffff810aad35>] ? wake_up_bit+0x25/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.547461] [<ffffffff81288a5d>] do_get_write_access+0x2ad/0x4f0
May 16 01:23:17 verticanode1 kernel: [7745522.547463] [<ffffffff811f109d>] ? __getblk+0x2d/0x2e0
May 16 01:23:17 verticanode1 kernel: [7745522.547466] [<ffffffff81288cc7>] jbd2_journal_get_write_access+0x27/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.547468] [<ffffffff8126e9cb>] __ext4_journal_get_write_access+0x3b/0x80
May 16 01:23:17 verticanode1 kernel: [7745522.547471] [<ffffffff81243d20>] ext4_reserve_inode_write+0x70/0xa0
May 16 01:23:17 verticanode1 kernel: [7745522.547473] [<ffffffff81247010>] ? ext4_dirty_inode+0x40/0x60
May 16 01:23:17 verticanode1 kernel: [7745522.547476] [<ffffffff81243d94>] ext4_mark_inode_dirty+0x44/0x1f0
May 16 01:23:17 verticanode1 kernel: [7745522.547478] [<ffffffff81247010>] ext4_dirty_inode+0x40/0x60
May 16 01:23:17 verticanode1 kernel: [7745522.547480] [<ffffffff811e7cba>] __mark_inode_dirty+0x10a/0x2d0
May 16 01:23:17 verticanode1 kernel: [7745522.547482] [<ffffffff811d8cc1>] update_time+0x81/0xd0
May 16 01:23:17 verticanode1 kernel: [7745522.547485] [<ffffffff811d8ed0>] file_update_time+0x80/0xd0
May 16 01:23:17 verticanode1 kernel: [7745522.547487] [<ffffffff811509c0>] __generic_file_aio_write+0x180/0x3d0
May 16 01:23:17 verticanode1 kernel: [7745522.547489] [<ffffffff81150c68>] generic_file_aio_write+0x58/0xa0
May 16 01:23:17 verticanode1 kernel: [7745522.547491] [<ffffffff8123b982>] ext4_file_write+0xa2/0x3f0
May 16 01:23:17 verticanode1 kernel: [7745522.547495] [<ffffffff811cec1a>] ? do_filp_open+0x3a/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.547497] [<ffffffff811bd7ea>] do_sync_write+0x5a/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.547499] [<ffffffff811bdf74>] vfs_write+0xb4/0x1f0
May 16 01:23:17 verticanode1 kernel: [7745522.547502] [<ffffffff811bd328>] ? do_sys_open+0x1b8/0x280
May 16 01:23:17 verticanode1 kernel: [7745522.547504] [<ffffffff811be9a9>] SyS_write+0x49/0xa0
May 16 01:23:17 verticanode1 kernel: [7745522.547506] [<ffffffff81731fbd>] system_call_fastpath+0x1a/0x1f
May 16 01:23:17 verticanode1 kernel: [7745522.547513] INFO: task kworker/u30:1:17634 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745522.551108] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745522.553800] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745522.557796] kworker/u30:1 D ffff88078fc734c0 0 17634 2 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745522.557801] Workqueue: writeback bdi_writeback_workfn (flush-202:16)
May 16 01:23:17 verticanode1 kernel: [7745522.557803] ffff880103a0d6f8 0000000000000046 ffff8801e3d4e000 ffff880103a0dfd8
May 16 01:23:17 verticanode1 kernel: [7745522.557805] 00000000000134c0 00000000000134c0 ffff8801e3d4e000 ffff88078fc73d58
May 16 01:23:17 verticanode1 kernel: [7745522.557807] ffff88078ffa91e8 0000000000000002 ffffffff81287630 ffff880103a0d770
May 16 01:23:17 verticanode1 kernel: [7745522.557809] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745522.557812] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745522.557815] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745522.557817] [<ffffffff8128763e>] sleep_on_shadow_bh+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745522.557820] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.557822] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745522.557824] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745522.557827] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.557829] [<ffffffff810aad35>] ? wake_up_bit+0x25/0x30
May 16 01:23:17 verticanode1 kernel: [7745522.557832] [<ffffffff81288a5d>] do_get_write_access+0x2ad/0x4f0
May 16 01:23:17 verticanode1 kernel: [7745522.557834] [<ffffffff81288cc7>] jbd2_journal_get_write_access+0x27/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.557837] [<ffffffff8126e9cb>] __ext4_journal_get_write_access+0x3b/0x80
May 16 01:23:17 verticanode1 kernel: [7745522.557840] [<ffffffff812755a0>] ext4_mb_mark_diskspace_used+0xd0/0x4c0
May 16 01:23:17 verticanode1 kernel: [7745522.557842] [<ffffffff81276d3d>] ext4_mb_new_blocks+0x2ad/0x550
May 16 01:23:17 verticanode1 kernel: [7745522.557845] [<ffffffff8126941c>] ? ext4_ext_find_extent+0x12c/0x2d0
May 16 01:23:17 verticanode1 kernel: [7745522.557847] [<ffffffff8126cf14>] ext4_ext_map_blocks+0x664/0xf40
May 16 01:23:17 verticanode1 kernel: [7745522.557852] [<ffffffff81240001>] ? ext4_da_get_block_prep+0x561/0x5d0
May 16 01:23:17 verticanode1 kernel: [7745522.557854] [<ffffffff812413f5>] ext4_map_blocks+0x2d5/0x540
May 16 01:23:17 verticanode1 kernel: [7745522.557857] [<ffffffff81244306>] ? ext4_writepages+0x3c6/0xd20
May 16 01:23:17 verticanode1 kernel: [7745522.557859] [<ffffffff8124455f>] ext4_writepages+0x61f/0xd20
May 16 01:23:17 verticanode1 kernel: [7745522.557863] [<ffffffff8115b45e>] do_writepages+0x1e/0x40
May 16 01:23:17 verticanode1 kernel: [7745522.557865] [<ffffffff811e7640>] __writeback_single_inode+0x40/0x220
May 16 01:23:17 verticanode1 kernel: [7745522.557867] [<ffffffff811e8407>] writeback_sb_inodes+0x247/0x3e0
May 16 01:23:17 verticanode1 kernel: [7745522.557869] [<ffffffff811e863f>] __writeback_inodes_wb+0x9f/0xd0
May 16 01:23:17 verticanode1 kernel: [7745522.557871] [<ffffffff811e88b3>] wb_writeback+0x243/0x2c0
May 16 01:23:17 verticanode1 kernel: [7745522.557873] [<ffffffff811ea189>] bdi_writeback_workfn+0x1b9/0x430
May 16 01:23:17 verticanode1 kernel: [7745522.557877] [<ffffffff81083a82>] process_one_work+0x182/0x450
May 16 01:23:17 verticanode1 kernel: [7745522.557879] [<ffffffff81084871>] worker_thread+0x121/0x410
May 16 01:23:17 verticanode1 kernel: [7745522.557882] [<ffffffff81084750>] ? rescuer_thread+0x430/0x430
May 16 01:23:17 verticanode1 kernel: [7745522.557884] [<ffffffff8108b5b2>] kthread+0xd2/0xf0
May 16 01:23:17 verticanode1 kernel: [7745522.557886] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745522.557888] [<ffffffff81731f0c>] ret_from_fork+0x7c/0xb0
May 16 01:23:17 verticanode1 kernel: [7745522.557890] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745642.556078] INFO: task jbd2/xvda1-8:259 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745642.560264] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745642.563492] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745642.568241] jbd2/xvda1-8 D ffff88078fcf34c0 0 259 2 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745642.568244] ffff88076358fbd8 0000000000000046 ffff880763840000 ffff88076358ffd8
May 16 01:23:17 verticanode1 kernel: [7745642.568247] 00000000000134c0 00000000000134c0 ffff880763840000 ffff88078fcf3d58
May 16 01:23:17 verticanode1 kernel: [7745642.568249] ffff88078ff98478 0000000000000002 ffffffff811ef790 ffff88076358fc50
May 16 01:23:17 verticanode1 kernel: [7745642.568252] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745642.568259] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745642.568263] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745642.568266] [<ffffffff811ef79e>] sleep_on_buffer+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745642.568269] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745642.568271] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745642.568273] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745642.568277] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745642.568280] [<ffffffff811f0aca>] __wait_on_buffer+0x2a/0x30
May 16 01:23:17 verticanode1 kernel: [7745642.568284] [<ffffffff8128adb0>] jbd2_journal_commit_transaction+0xf30/0x1ab0
May 16 01:23:17 verticanode1 kernel: [7745642.568288] [<ffffffff810755bf>] ? try_to_del_timer_sync+0x4f/0x70
May 16 01:23:17 verticanode1 kernel: [7745642.568291] [<ffffffff8128fa0d>] kjournald2+0xbd/0x250
May 16 01:23:17 verticanode1 kernel: [7745642.568293] [<ffffffff810ab120>] ? prepare_to_wait_event+0x100/0x100
May 16 01:23:17 verticanode1 kernel: [7745642.568296] [<ffffffff8128f950>] ? commit_timeout+0x10/0x10
May 16 01:23:17 verticanode1 kernel: [7745642.568299] [<ffffffff8108b5b2>] kthread+0xd2/0xf0
May 16 01:23:17 verticanode1 kernel: [7745642.568301] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745642.568303] [<ffffffff81731f0c>] ret_from_fork+0x7c/0xb0
May 16 01:23:17 verticanode1 kernel: [7745642.568305] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745642.568308] INFO: task jbd2/xvdb-8:416 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745642.572516] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745642.575964] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745642.580991] jbd2/xvdb-8 D ffff88078fc134c0 0 416 2 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745642.580993] ffff8807638f1bd8 0000000000000046 ffff880763891800 ffff8807638f1fd8
May 16 01:23:17 verticanode1 kernel: [7745642.580996] 00000000000134c0 00000000000134c0 ffff880763891800 ffff88078fc13d58
May 16 01:23:17 verticanode1 kernel: [7745642.580998] ffff88078ffabc30 0000000000000002 ffffffff811ef790 ffff8807638f1c50
May 16 01:23:17 verticanode1 kernel: [7745642.581000] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745642.581004] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745642.581008] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745642.581010] [<ffffffff811ef79e>] sleep_on_buffer+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745642.581012] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745642.581015] [<ffffffff811ef790>] ? generic_block_bmap+0x50/0x50
May 16 01:23:17 verticanode1 kernel: [7745642.581017] [<ffffffff81726167>] out_of_line_wait_on_bit+0x77/0x90
May 16 01:23:17 verticanode1 kernel: [7745642.581020] [<ffffffff810ab160>] ? autoremove_wake_function+0x40/0x40
May 16 01:23:17 verticanode1 kernel: [7745642.581023] [<ffffffff811f0aca>] __wait_on_buffer+0x2a/0x30
May 16 01:23:17 verticanode1 kernel: [7745642.581026] [<ffffffff8128adb0>] jbd2_journal_commit_transaction+0xf30/0x1ab0
May 16 01:23:17 verticanode1 kernel: [7745642.581029] [<ffffffff810755bf>] ? try_to_del_timer_sync+0x4f/0x70
May 16 01:23:17 verticanode1 kernel: [7745642.581032] [<ffffffff8128fa0d>] kjournald2+0xbd/0x250
May 16 01:23:17 verticanode1 kernel: [7745642.581034] [<ffffffff810ab120>] ? prepare_to_wait_event+0x100/0x100
May 16 01:23:17 verticanode1 kernel: [7745642.581037] [<ffffffff8128f950>] ? commit_timeout+0x10/0x10
May 16 01:23:17 verticanode1 kernel: [7745642.581039] [<ffffffff8108b5b2>] kthread+0xd2/0xf0
May 16 01:23:17 verticanode1 kernel: [7745642.581041] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745642.581043] [<ffffffff81731f0c>] ret_from_fork+0x7c/0xb0
May 16 01:23:17 verticanode1 kernel: [7745642.581045] [<ffffffff8108b4e0>] ? kthread_create_on_node+0x1c0/0x1c0
May 16 01:23:17 verticanode1 kernel: [7745642.581050] INFO: task java:1387 blocked for more than 120 seconds.
May 16 01:23:17 verticanode1 kernel: [7745642.585563] Not tainted 3.13.0-48-generic #80-Ubuntu
May 16 01:23:17 verticanode1 kernel: [7745642.589528] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 16 01:23:17 verticanode1 kernel: [7745642.595205] java D ffff88078fc534c0 0 1387 1 0x00000000
May 16 01:23:17 verticanode1 kernel: [7745642.595208] ffff880764cbd9c8 0000000000000082 ffff8807651fe000 ffff880764cbdfd8
May 16 01:23:17 verticanode1 kernel: [7745642.595210] 00000000000134c0 00000000000134c0 ffff8807651fe000 ffff88078fc53d58
May 16 01:23:17 verticanode1 kernel: [7745642.595212] ffff88078ffa01e8 0000000000000002 ffffffff81287630 ffff880764cbda40
May 16 01:23:17 verticanode1 kernel: [7745642.595214] Call Trace:
May 16 01:23:17 verticanode1 kernel: [7745642.595218] [<ffffffff81287630>] ? start_this_handle+0x590/0x590
May 16 01:23:17 verticanode1 kernel: [7745642.595220] [<ffffffff81725c3d>] io_schedule+0x9d/0x140
May 16 01:23:17 verticanode1 kernel: [7745642.595222] [<ffffffff8128763e>] sleep_on_shadow_bh+0xe/0x20
May 16 01:23:17 verticanode1 kernel: [7745642.595225] [<ffffffff817260c2>] __wait_on_bit+0x62/0x90
May 16 01:23:17 verticanode1 kernel: [7745642.595227] [<ffffffff81287630>] ?

Post Reply

Return to “Vertica Database Administration”