Containers count mismatch

Moderator: NorbertKrupa

User avatar
nnani
Master
Master
Posts: 302
Joined: Fri Apr 13, 2012 6:28 am
Contact:

Containers count mismatch

Post by nnani » Tue Mar 25, 2014 7:55 am

Hello All,

I am checking if the number of ROS container for a particular table are equal to the partitions it has.

The table is partition on period_key/7 (period_key is date)
When I check the count for distinct period_key it is 742 so by the calculation the containers should be 106 (please correct me if this wrong :roll: )

When I check the storage_container for the projection related to that table it shows 1296 containers.

There is vast difference between these numbers. Any wrong assumption I am making ?
nnani........
Long way to go

You can check out my blogs at vertica-howto

id10t
GURU
GURU
Posts: 732
Joined: Mon Apr 16, 2012 2:44 pm

Re: Containers count mismatch

Post by id10t » Tue Mar 25, 2014 7:27 pm

Hi!

[DELETED]
Last edited by id10t on Wed May 06, 2015 5:55 pm, edited 1 time in total.

id10t
GURU
GURU
Posts: 732
Joined: Mon Apr 16, 2012 2:44 pm

Re: Containers count mismatch

Post by id10t » Tue Mar 25, 2014 7:39 pm

Hi!

[DELETED]
Last edited by id10t on Wed May 06, 2015 5:53 pm, edited 1 time in total.

User avatar
nnani
Master
Master
Posts: 302
Joined: Fri Apr 13, 2012 6:28 am
Contact:

Re: Containers count mismatch

Post by nnani » Wed Mar 26, 2014 7:45 am

about Calculations

Partition - 106
Scale factor = ? ( Is this the k-saftey value)
Nodes = 5

Where can I see How many containers does a projection have.

Code: Select all

select node_name,count(distinct(storage_oid)) from v_monitor.storage_containers where storage_type='ROS' and
schema_name=xx and projection_name like 'xx' group by node_name
I tried the above sql

It gives me a count of 1296 across the cluster

Can you please help me understand this.
I was under the assumption, if there are 106 partitions on a table the containers will also be 106 per node for that table

Thanks in Advance
nnani........
Long way to go

You can check out my blogs at vertica-howto

User avatar
nnani
Master
Master
Posts: 302
Joined: Fri Apr 13, 2012 6:28 am
Contact:

Re: Containers count mismatch

Post by nnani » Wed Mar 26, 2014 9:48 am

Hi,

I went through documentation for scaling factor.

How is Local segmentation different from Projection Segmentation?
What is the relation between ROS containers and Local Segmentation
Local data segmentation increases the number of storage containers stored on each node. This is not an issue unless a table contains many partitions. For example, if the table is partitioned by day and contains one or more years. If local data segmentation is enabled, then each of these table partitions is broken into multiple local storage segments, which potentially results in a huge number of files which can lead to ROS "pushback."
According to the document:
A partition is further segmented if local segmentation is enabled....what does this mean ?
How many container will a partition result into if the scaling factor is 4?


Confused with this New Topic :roll: But happy to leanr something new :)
nnani........
Long way to go

You can check out my blogs at vertica-howto

NorbertKrupa
GURU
GURU
Posts: 527
Joined: Tue Oct 22, 2013 9:36 pm
Location: Chicago, IL
Contact:

Re: Containers count mismatch

Post by NorbertKrupa » Wed Mar 26, 2014 1:00 pm

Locally segmented means the segmentation takes place on each node. Normally segmented projections are distributed across multiple nodes.
Checkout vertica.tips for more Vertica resources.

scutter
Master
Master
Posts: 302
Joined: Tue Aug 07, 2012 2:15 am

Re: Containers count mismatch

Post by scutter » Wed Mar 26, 2014 2:44 pm

Local segmentation is used to speed up cluster rebalancing. For each partition, instead of getting a single ROS container per partition, the data will be pre-split into multiple ROS containers according to the scaling factor. Each of those is a “local segment”. If set to 4, you’ll have 4 ROS containers / local segments per partition. When a cluster rebalance occurs, those “local segments” are redistributed around the cluster.

When local segmentation is on, the data is pre-split into these local segments and the rebalance will be as fast as possible. If local segmentation is off, then rebalance will first split up ROS containers into local segments according to the scaling factor, and then redistribute the segments.

Whether to have local segmentation on or off depends on how often you rebalance, and your needs for partition granularlity and data retention. If you have daily partitioning, local segmentation on, and scaling factor set to 4, then a year’s worth of data would already exceed the maximum number of ROS containers per projection.

Note that “local resegmentation” is also used in explain plans for GROUP BY operations. That is a different concept.

—Sharon
Sharon Cutter
Vertica Consultant, Zazz Technologies LLC

Post Reply

Return to “Vertica Database Development”