Hi all,
From the admin guide I read this:
Partitions and ROS Containers
1. Data is automatically split into partitions during load / refresh / recovery operations.
2. The Tuple Mover maintains physical separation of partitions.
3. Each ROS container contains data for a single partition, though there can be multiple ROS containers for a single partition.
So for #3, under what circumstances would there be "multiple ROS containers for a single partition"?
Partitions and ROS Containers
Moderator: NorbertKrupa
- JimKnicely
- Site Admin
- Posts: 1825
- Joined: Sat Jan 21, 2012 4:58 am
- Contact:
Re: Partitions and ROS Containers
Prior to a MERGEOUT operation, you could see extra ROS containers for partitioned data.
Simple example:
After the inserts, we have two ROS containers (see ros_count column):
After these inserts, we now have four ROS containers. Vertica did not at this point combine the partition data into a single container for each partition:
But after a MERGEOUT, Vertica did combine (i.e. merge) the partitioned data into a single container for each partition:
Make sense?
Simple example:
Code: Select all
dbadmin=> drop table test;
DROP TABLE
dbadmin=> create table test (c1 int not null, c2 int) partition by c1;
CREATE TABLE
dbadmin=> insert /*+ direct */ into test values (1, 1);
OUTPUT
--------
1
(1 row)
dbadmin=> insert /*+ direct */ into test values (2, 1);
OUTPUT
--------
1
(1 row)
Code: Select all
dbadmin=> select projection_name, ros_row_count, ros_count from projection_storage where anchor_table_name = 'test';
projection_name | ros_row_count | ros_count
-----------------+---------------+-----------
test_super | 2 | 2
(1 row)
Code: Select all
dbadmin=> insert /*+ direct */ into test values (1, 1);
OUTPUT
--------
1
(1 row)
dbadmin=> insert /*+ direct */ into test values (2, 1);
OUTPUT
--------
1
(1 row)
Code: Select all
dbadmin=> select projection_name, ros_row_count, ros_count from projection_storage where anchor_table_name = 'test';
projection_name | ros_row_count | ros_count
-----------------+---------------+-----------
test_super | 4 | 4
(1 row)
dbadmin=> select do_tm_task('mergeout', 'test');
do_tm_task
----------------------------------------------------------------------
Task: mergeout
(Table: public.test) (Projection: public.test_super)
(1 row)
Code: Select all
dbadmin=> select projection_name, ros_row_count, ros_count from projection_storage where anchor_table_name = 'test';
projection_name | ros_row_count | ros_count
-----------------+---------------+-----------
test_super | 4 | 2
(1 row)
Jim Knicely
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Re: Partitions and ROS Containers
A few additional comments beyond what Jim wrote.
- The “active” partitions (1 or more) will have more than one ROS container, until they become “past partitions”
- Past partitions may have more than one ROS container. Originally the TM would aggressively merge past partitions into a single ROS container, but in current versions it won’t always merge multiple ROS containers in past partitions to a single ROS. If you have a ROS container in a past partition it’s not necessary worthwhile to merge it with a large ROS, so the TM won’t do that.
- DVROSes (for delete vectors) may also co-exist and count towards the number of ROS containers.
- I’ve observed that recent versions have a higher limit for ROS containers than the technical limit. For example if the limit in the configuration parameter is 1024, it lets you go to 2048. I haven’t spotted anything in the documentation for this, so there’s no commitment that this would remain true, but it does give you more head room.
—Sharon
- The “active” partitions (1 or more) will have more than one ROS container, until they become “past partitions”
- Past partitions may have more than one ROS container. Originally the TM would aggressively merge past partitions into a single ROS container, but in current versions it won’t always merge multiple ROS containers in past partitions to a single ROS. If you have a ROS container in a past partition it’s not necessary worthwhile to merge it with a large ROS, so the TM won’t do that.
- DVROSes (for delete vectors) may also co-exist and count towards the number of ROS containers.
- I’ve observed that recent versions have a higher limit for ROS containers than the technical limit. For example if the limit in the configuration parameter is 1024, it lets you go to 2048. I haven’t spotted anything in the documentation for this, so there’s no commitment that this would remain true, but it does give you more head room.
—Sharon
Sharon Cutter
Vertica Consultant, Zazz Technologies LLC
Vertica Consultant, Zazz Technologies LLC
Re: Partitions and ROS Containers
Thank you Jim and Sharon! Your posts are very, very helpful!!!