Database Won't Start!

[JeffSatler]

Hello,

Our database went down and I can not get it to restart using the admintools. When I try and start it, I get the following messages:

Code: Select all

*** Starting database: finance_wh01 ***
        Node Status: v_finance_wh01_node0001: (DOWN) v_finance_wh01_node0002: (DOWN) v_finance_wh01_node0003: (DOWN)
        Node Status: v_finance_wh01_node0001: (DOWN) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (DOWN) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
        Node Status: v_finance_wh01_node0001: (INITIALIZING) v_finance_wh01_node0002: (INITIALIZING) v_finance_wh01_node0003: (INITIALIZING)
Nodes DOWN: v_finance_wh01_node0001, v_finance_wh01_node0002, v_finance_wh01_node0003 (may be still initializing).
It is suggested that you continue waiting.
Do you want to continue waiting? (yes/no) [yes] no
        Database startup successful, but it may be incomplete. Some nodes
        remain in a transitional state. Waiting can sometimes resolve this
        problem.  See Database Cluster State in Main Menu for updated
        cluster state. If this state persists, try using the Advanced menu
        to Stop Vertica on Host, then Restart Database.
Press RETURN to continue

When I hit enter I get a message that says that all databases will be shut down.

Please help!

Thanks,
Jeff

[JimKnicely]

Jeff - It looks like it is your node 3 that won't start. Can you check if the spread daemon is running on that server?

You can use the ps -ef | grep -i spread command at the Linux prompt to find out.

Example:

Code: Select all

[root]# ps -ef | grep -i spread
spread   19896     1  0  2011 ?        00:08:58 /opt/vertica/spread/sbin/spread -n N010255017039 -c /opt/vertica/config/vspread.conf
root     26289 26258  0 13:51 pts/0    00:00:00 grep -i spread

If its not running you can start it like this:

Code: Select all

[root]# /etc/init.d/spreadd start
Starting spread daemon: spread (pid  30516) is running...
                                                           [  OK  ]

Sometimes for me it won't start up correctly so I have to issue a stop and then a start twice... I'm not sure why it acts like my car on cold mornings.

Example:

Code: Select all

[root@verticatst03 init.d]# /etc/init.d/spreadd stop
Stopping spread daemon:                                    [FAILED]
[root@verticatst03 init.d]# /etc/init.d/spreadd start
Starting spread daemon:                                    [FAILED]
[root@verticatst03 init.d]# /etc/init.d/spreadd start
Starting spread daemon: spread (pid  30516) is running...
                                                           [  OK  ]

I hope this helps.