Backup Errors!

Moderator: NorbertKrupa

Post Reply
User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Backup Errors!

Post by becky » Mon Sep 30, 2013 4:32 pm

Hi,

I am getting errors when trying to backup a three node cluster:

Code: Select all

rmdir: failed to remove `/home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done': Directory not empty
Traceback (most recent call last):
  File "/opt/vertica/bin/vbr.py", line 1971, in backup
    remoteCopyAll()
  File "/opt/vertica/bin/vbr.py", line 1821, in remoteCopyAll
    recoverAll()
  File "/opt/vertica/bin/vbr.py", line 1743, in recoverAll
    subprocess.check_call(g["sshBackup"] + [mapping['backupHost'], 'rmdir %s.done'%dirName])
  File "/opt/vertica/oss/python/lib/python2.7/subprocess.py", line 504, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['ssh', '-x', '10.10.10.1', 'rmdir /home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done']' returned non-zero exit status 1
backup failed unexpectedly!
Here is the command I am running:

Code: Select all

/opt/vertica/bin/vbr.py --task backup --config-file /home/dbadmin/vertica_backups/sales_db_full.ini
Here is my /home/dbadmin/vertica_backups/sales_db_full.ini file:

Code: Select all

[Misc]
snapshotName = sales_db_full
verticaConfig = True
restorePointLimit = 7
tempDir = /tmp/vbr
retryCount = 2
retryDelay = 1

[Database]
dbName = sales_db
dbUser = dbadmin
dbPassword = XXXXXXXXXXXXXXXXX

[Transmission]
encrypt = False
checksum = False
port_rsync = 50000
bwlimit = 0

[Mapping0]
dbNode = v_sales_db_node0001
backupHost = vertica01
backupDir = /home/dbadmin/vertica_backups

[Mapping1]
dbNode = v_sales_db_node0002
backupHost = vertica01
backupDir = /home/dbadmin/vertica_backups

[Mapping2]
dbNode = v_sales_db_node0003
backupHost = vertica01
backupDir = /home/dbadmin/vertica_backups
Anyone else run into this issue? The backups were running fine for weeks :(
THANKS - BECKSTER

id10t
GURU
GURU
Posts: 732
Joined: Mon Apr 16, 2012 2:44 pm

Re: Backup Errors!

Post by id10t » Mon Sep 30, 2013 5:02 pm

Hi!

Can you post a list of files that in directory: /home/dbadmin/vertica_backups/v_sales_db_node0001/ ?
`rmdir` fails to remove directory in case that directory contains files.

Code: Select all

$ mkdir foo
$ touch foo/file.txt
$ rmdir foo
rmdir: failed to remove `foo': Directory not empty

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Backup Errors!

Post by becky » Mon Sep 30, 2013 6:26 pm

Hi sKwa,

I tried delete all the files in that directory manually and re-ran the backup command...

Code: Select all

[dbadmin@vertica01 ~]$ ls -lrt /home/dbadmin/vertica_backups/v_snowfall_node0001
total 0

Code: Select all

[dbadmin@vertica01]$ /opt/vertica/bin/vbr.py --task backup --config-file /home/dbadmin/vertica_backups/sales_db_full.ini
Preparing...
Found Database port:  5433
rmdir: failed to remove `/home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done': Directory not empty
Traceback (most recent call last):
  File "/opt/vertica/bin/vbr.py", line 1971, in backup
    remoteCopyAll()
  File "/opt/vertica/bin/vbr.py", line 1821, in remoteCopyAll
    recoverAll()
  File "/opt/vertica/bin/vbr.py", line 1743, in recoverAll
    subprocess.check_call(g["sshBackup"] + [mapping['backupHost'], 'rmdir %s.done'%dirName])
  File "/opt/vertica/oss/python/lib/python2.7/subprocess.py", line 504, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['ssh', '-x', '10.10.10.1', 'rmdir /home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done']' returned non-zero exit status 1
backup failed unexpectedly!
cleaning up...
Retrying... #1
rmdir: failed to remove `/home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done': Directory not empty
Traceback (most recent call last):
  File "/opt/vertica/bin/vbr.py", line 1971, in backup
    remoteCopyAll()
  File "/opt/vertica/bin/vbr.py", line 1821, in remoteCopyAll
    recoverAll()
  File "/opt/vertica/bin/vbr.py", line 1743, in recoverAll
    subprocess.check_call(g["sshBackup"] + [mapping['backupHost'], 'rmdir %s.done'%dirName])
  File "/opt/vertica/oss/python/lib/python2.7/subprocess.py", line 504, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['ssh', '-x', '10.10.10.1', 'rmdir /home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done']' returned non-zero exit status 1
backup failed unexpectedly!
cleaning up...
Retrying... #2
rmdir: failed to remove `/home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done': Directory not empty
Traceback (most recent call last):
  File "/opt/vertica/bin/vbr.py", line 1971, in backup
    remoteCopyAll()
  File "/opt/vertica/bin/vbr.py", line 1821, in remoteCopyAll
    recoverAll()
  File "/opt/vertica/bin/vbr.py", line 1743, in recoverAll
    subprocess.check_call(g["sshBackup"] + [mapping['backupHost'], 'rmdir %s.done'%dirName])
  File "/opt/vertica/oss/python/lib/python2.7/subprocess.py", line 504, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['ssh', '-x', '10.10.10.1', 'rmdir /home/dbadmin/vertica_backups/v_sales_db_node0001/.sales_db_full.done']' returned non-zero exit status 1
backup failed unexpectedly!
cleaning up...
Same error. Weird!
THANKS - BECKSTER

User avatar
becky
Intermediate
Intermediate
Posts: 118
Joined: Sat Apr 28, 2012 11:37 am

Re: Backup Errors!

Post by becky » Mon Sep 30, 2013 6:31 pm

Ok, so I deleted the following three directories and re-ran the backup command and it worked.

Code: Select all

[dbadmin@vertica01 vertica_backups]$ rm -fR v_sales_db_node0001
[dbadmin@vertica01 vertica_backups]$ rm -fR v_sales_db_node0002
[dbadmin@vertica01 vertica_backups]$ rm -fR v_sales_db_node0003
[dbadmin@vertica01 vertica_backups]$ /opt/vertica/bin/vbr.py --task backup --config-file /home/dbadmin/vertica_backups/sales_db_full.ini
Preparing...
Found Database port:  5433
Copying...
[==================================================] 100%
All child processes terminated successfully.
Committing changes on all backup sites...
backup done!
Unfortunately that means I lost all my prior backups :(
THANKS - BECKSTER

Post Reply

Return to “Vertica Backup & Recovery”