Unable to bring up the Vertica Cluster

[Roopalini]

Hi Forum,

Our Vertica DB is on a 3 node cluster. One of the nodes went down and I am unable to bring the database up. I think , I didn't shut it down properly before starting the database and something got messed up. Here are the errors in the log file. Can someone tell me what I need to do. It does not even show me the status of the nodes.

[dbadmin@c000vert03na01 v_c000_temp_wh_node0003_catalog]$ cat vertica.log| grep 'ERROR'
2014-03-24 21:39:07.097 Main:0x5040900 <LOG> @v_c000_temp_wh_node0003: 00000/3298: Event Posted: Event Code:6 Event Id:9 Event Severity: Informational [6] PostedTimestamp: 2014-03-24 21:39:07.097573 ExpirationTimestamp: 2082-04-12 00:53:14.097573 EventCodeDescription: Node State Change ProblemDescription: Changing node v_c000_temp_wh_node0003 startup state to SHUTDOWN_ERROR DatabaseName: c000_temp_wh Hostname: c000vert03na01
2014-03-24 21:39:07.097 Main:0x5040900 [Recover] <INFO> Changing node v_c000_temp_wh_node0003 startup state from UNSAFE to SHUTDOWN_ERROR
2014-03-24 21:46:00.715 Main:0x6666900 <LOG> @v_c000_temp_wh_node0003: 00000/3298: Event Posted: Event Code:6 Event Id:9 Event Severity: Informational [6] PostedTimestamp: 2014-03-24 21:46:00.715335 ExpirationTimestamp: 2082-04-12 01:00:07.715335 EventCodeDescription: Node State Change ProblemDescription: Changing node v_c000_temp_wh_node0003 startup state to SHUTDOWN_ERROR DatabaseName: c000_temp_wh Hostname: c000vert03na01
2014-03-24 21:46:00.715 Main:0x6666900 [Recover] <INFO> Changing node v_c000_temp_wh_node0003 startup state from UNSAFE to SHUTDOWN_ERROR
2014-03-25 00:02:41.757 Main:0x593d900 <LOG> @v_c000_temp_wh_node0003: 00000/3298: Event Posted: Event Code:6 Event Id:9 Event Severity: Informational [6] PostedTimestamp: 2014-03-25 00:02:41.757057 ExpirationTimestamp: 2082-04-12 03:16:48.757057 EventCodeDescription: Node State Change ProblemDescription: Changing node v_c000_temp_wh_node0003 startup state to SHUTDOWN_ERROR DatabaseName: c000_temp_wh Hostname: c000vert03na01
2014-03-25 00:02:41.757 Main:0x593d900 [Recover] <INFO> Changing node v_c000_temp_wh_node0003 startup state from UNSAFE to SHUTDOWN_ERROR
2014-03-25 00:08:46.335 Main:0x6b8a900 <LOG> @v_c000_temp_wh_node0003: 00000/3298: Event Posted: Event Code:6 Event Id:9 Event Severity: Informational [6] PostedTimestamp: 2014-03-25 00:08:46.335504 ExpirationTimestamp: 2082-04-12 03:22:53.335504 EventCodeDescription: Node State Change ProblemDescription: Changing node v_c000_temp_wh_node0003 startup state to SHUTDOWN_ERROR DatabaseName: c000_temp_wh Hostname: c000vert03na01
2014-03-25 00:08:46.335 Main:0x6b8a900 [Recover] <INFO> Changing node v_c000_temp_wh_node0003 startup state from UNSAFE to SHUTDOWN_ERROR
[dbadmin@c000vert03na01 v_c000_temp_wh_node0003_catalog]$

[NorbertKrupa]

Take a look at the Administrator's guide. You'll most likely need to perform a recovery after an unclean shutdown which involves rolling back to the last good epoch.

If some hosts are UP and some DOWN, restart the specific host that is down using Restart HP Vertica on Host from the Administration Tools, or you can start the database as described in Starting and Stopping the Database (page 233) (unless you have a known node failure and want to continue in that state.)

[Roopalini]

Forcefully starting the database resolved the issue

admintools -t start_db -d c000_temp_wh -F

Now the node is in the "RECOVERING STATE". Can you please tell me , how can I find the % of recovery. Is there anything that I can do to speed up the RECOVERY process?

Thanks

[JimKnicely]

Check these system tables:

Code: Select all

dbadmin=> select table_schema, table_name from system_tables where table_name ilike '%recov%';
 table_schema |      table_name       
--------------+-----------------------
 v_monitor    | projection_recoveries
 v_monitor    | recovery_status
(2 rows)