Unable to start Vertica database

Moderator: NorbertKrupa

Phani
Newbie
Newbie
Posts: 4
Joined: Wed Aug 27, 2014 4:39 pm

Unable to start Vertica database

Post by Phani » Wed Aug 27, 2014 4:45 pm

Hi,

Am new to vertica database and I created a two node vertica cluster for testing it but when I try to create a database it does not start and shows INITIALIZING state forever.

Node Status: v_raj_node0001: (INITIALIZING)
Node Status: v_raj_node0001: (INITIALIZING)
Node Status: v_raj_node0001: (UP)
Creating database nodes
Creating node v_raj_node0002 (host x.x.x.x)
Generating new configuration information
Stopping bootstrap node
Issuing shutdown command to database
Starting all nodes
Starting nodes:
v_raj_node0001 (x.x.x.x)
v_raj_node0002 (x.x.x.x)

Starting Vertica on all nodes. Please wait, databases with large catalogs may take a while to initialize.

Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Nodes DOWN: v_raj_node0001, v_raj_node0002 (may be still initializing).
It is suggested that you continue waiting.
Do you want to continue waiting? (yes/no) [yes] yes
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Node Status: v_raj_node0001: (INITIALIZING) v_raj_node0002: (DOWN)
Nodes DOWN: v_raj_node0001, v_raj_node0002 (may be still initializing).
It is suggested that you continue waiting.
Do you want to continue waiting? (yes/no) [yes] yes


Noting I can find in the vertica.lof or dbLog

Here is my dbLog:


Connected to spread on local domain socket 4803
08/27/14 15:30:00 SP_connect: DEBUG: Auth list is: NULL
/===========================================================================\
| The Spread Toolkit. |
| Copyright (c) 1993-2012 Spread Concepts LLC |
| All rights reserved. |
| |
| The Spread toolkit is licensed under the Spread Open-Source License. |
| You may only use this software in compliance with the License. |
| A copy of the license can be found at http://www.spread.org/license |
| |
| This product uses software developed by Spread Concepts LLC for use |
| in the Spread toolkit. For more information about Spread, |
| see http://www.spread.org |
| |
| This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF |
| ANY KIND, either express or implied. |
| |
| Creators: |
| Yair Amir yairamir@cs.jhu.edu |
| Michal Miskin-Amir michal@spreadconcepts.com |
| Jonathan Stanton jstanton@gwu.edu |
| John Schultz jschultz@spreadconcepts.com |
| |
| Major Contributors: |
| Ryan Caudy rcaudy@gmail.com - contribution to process groups.|
| Claudiu Danilov claudiu@acm.org - scalable, wide-area support. |
| Cristina Nita-Rotaru crisn@cs.purdue.edu - GC security. |
| Theo Schlossnagle jesus@omniti.com - Perl, autoconf, old skiplist. |
| Dan Schoenblum dansch@cnds.jhu.edu - Java interface. |
| |
| Special thanks to the following for discussions and ideas: |
| Ken Birman, Danny Dolev, Jacob Green, Mike Goodrich, Ben Laurie, |
| David Shaw, Gene Tsudik, Robbert VanRenesse. |
| |
| Partial funding provided by the Defense Advanced Research Project Agency |
| (DARPA) and the National Security Agency (NSA) 2000-2004. The Spread |
| toolkit is not necessarily endorsed by DARPA or the NSA. |
| |
| For a full list of contributors, see Readme.txt in the distribution. |
| |
| WWW: http://www.spread.org http://www.spreadconcepts.com |
| Contact: info@spreadconcepts.com |
| |
| Version 4.02.00 Built 18/June/2012 (Vertica) |
\===========================================================================/
Conf_load_conf_file: using file: /home/dbadmin/raj/v_raj_node0001_catalog/spread.conf
Successfully configured Segment 0 [x.x.x.x:4803] with 2 procs:
Nxxx3: x.x.x.x
Nxxx: x.x.x.x
Connected to spread on local domain socket 4803


Here is vertica.log:

HINT: Cluster State: raj
INITIALIZING: 1 of 2 (v_raj_node0001)
----
LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429
2014-08-27 15:36:27.000 Timer Service:0x8955520 [Util] <INFO> Task 'ManageEpochs' enabled
2014-08-27 15:36:27.000 Timer Service:0x8955520 [Util] <INFO> Task 'AnalyzeRowCount' enabled
2014-08-27 15:36:27.000 Timer Service:0x8955520 [Util] <INFO> Task 'FeatureUseLogger' enabled
2014-08-27 15:36:27.000 Timer Service:0x8955520 [Util] <INFO> Task 'LicenseSizeAuditor' enabled
2014-08-27 15:36:27.000 Timer Service:0x8955520 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-08-27 15:36:33.213 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/2705: Connection received: host=x.x.x.x3 port=55375 (connCnt 1)
2014-08-27 15:36:33.213 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/4540: Received SSL negotiation startup packet
2014-08-27 15:36:33.214 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/4691: Sending SSL negotiation response 'N'
2014-08-27 15:36:33.214 Init Session:0x7f33fc00f8c0 <FATAL> @v_raj_node0001: {SessionRun} 57V03/5785: Cluster Status Request by x.x.x.x:55375
HINT: Cluster State: raj
INITIALIZING: 1 of 2 (v_raj_node0001)
----
LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429
2014-08-27 15:36:37.000 Timer Service:0x8955520 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-08-27 15:36:45.258 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/2705: Connection received: host=x.x.x.x port=55377 (connCnt 1)
2014-08-27 15:36:45.258 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/4540: Received SSL negotiation startup packet
2014-08-27 15:36:45.258 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/4691: Sending SSL negotiation response 'N'
2014-08-27 15:36:45.259 Init Session:0x7f33fc00f8c0 <FATAL> @v_raj_node0001: {SessionRun} 57V03/5785: Cluster Status Request by x.x.x.x:55377
HINT: Cluster State: raj
INITIALIZING: 1 of 2 (v_raj_node0001)
----
LOCATION: initSession, /scratch_a/release/vbuild/vertica/Session/ClientSession.cpp:429
2014-08-27 15:36:47.000 Timer Service:0x8955520 [Util] <INFO> Task 'FeatureUseLogger' enabled
2014-08-27 15:36:47.000 Timer Service:0x8955520 [Util] <INFO> Task 'LicenseSizeAuditor' enabled
2014-08-27 15:36:47.000 Timer Service:0x8955520 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-08-27 15:36:57.000 Timer Service:0x8955520 [Util] <INFO> Task 'LowDiskSpaceCheck' enabled
2014-08-27 15:36:59.305 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/2705: Connection received: host=x.x.x.x port=55379 (connCnt 1)
2014-08-27 15:36:59.305 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/4540: Received SSL negotiation startup packet
2014-08-27 15:36:59.305 Init Session:0x7f33fc00f8c0 <LOG> @v_raj_node0001: 00000/4691: Sending SSL negotiation response 'N'
2014-08-27 15:36:59.306 Init Session:0x7f33fc00f8c0 <FATAL> @v_raj_node0001: {SessionRun} 57V03/5785: Cluster Status Request by x.x.x.x:55379
HINT: Cluster State: raj
INITIALIZING: 1 of 2 (v_raj_node0001)
Last edited by Phani on Wed Aug 27, 2014 5:00 pm, edited 3 times in total.

NorbertKrupa
GURU
GURU
Posts: 527
Joined: Tue Oct 22, 2013 9:36 pm
Location: Chicago, IL
Contact:

Re: Unable to start Vertica database

Post by NorbertKrupa » Wed Aug 27, 2014 4:57 pm

Most likely your network configuration. What version are you running and are firewalls enabled? If any firewalls are enabled, are ports open?
Checkout vertica.tips for more Vertica resources.

Phani
Newbie
Newbie
Posts: 4
Joined: Wed Aug 27, 2014 4:39 pm

Re: Unable to start Vertica database

Post by Phani » Wed Aug 27, 2014 5:09 pm

Iptables are not running on the servers.

/etc/init.d/iptables status
iptables: Firewall is not running.

Thanks,
Phani

NorbertKrupa
GURU
GURU
Posts: 527
Joined: Tue Oct 22, 2013 9:36 pm
Location: Chicago, IL
Contact:

Re: Unable to start Vertica database

Post by NorbertKrupa » Wed Aug 27, 2014 5:34 pm

What version of Vertica are you running?

In Admin Tools, try to Stop Vertica on Host on all nodes, then restart the database.

Have you tried attempting to reach the opposite host from both hosts?
Checkout vertica.tips for more Vertica resources.

Phani
Newbie
Newbie
Posts: 4
Joined: Wed Aug 27, 2014 4:39 pm

Re: Unable to start Vertica database

Post by Phani » Wed Aug 27, 2014 6:29 pm

Am using vertica 7.0.1-0.

I tried to restart it but still the same issue. I can do a password-less ssh between the nodesas dbadmin user.

Thanks,
Phani

User avatar
JimKnicely
Site Admin
Site Admin
Posts: 1825
Joined: Sat Jan 21, 2012 4:58 am
Contact:

Re: Unable to start Vertica database

Post by JimKnicely » Thu Aug 28, 2014 11:58 am

How's it going with the DB start up?

Have you tried answering the question "Do you want to continue waiting? (yes/no) [yes]" yes at least 3 times? If it fails, try it again three times. I remember on several occasions that worked for me.

Also, have you stopping and restarting spread on each server?

Thanks!
Jim Knicely

Image

Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.

Phani
Newbie
Newbie
Posts: 4
Joined: Wed Aug 27, 2014 4:39 pm

Re: Unable to start Vertica database

Post by Phani » Thu Aug 28, 2014 4:58 pm

Database started now. It is strange.. I did not changed anything just reboot the servers.

Thanks for your help.

Thanks,
Phani

Post Reply

Return to “New to Vertica Database Administration”