Page 1 of 2

Containers count mismatch

Posted: Tue Mar 25, 2014 7:55 am
by nnani
Hello All,

I am checking if the number of ROS container for a particular table are equal to the partitions it has.

The table is partition on period_key/7 (period_key is date)
When I check the count for distinct period_key it is 742 so by the calculation the containers should be 106 (please correct me if this wrong :roll: )

When I check the storage_container for the projection related to that table it shows 1296 containers.

There is vast difference between these numbers. Any wrong assumption I am making ?

Re: Containers count mismatch

Posted: Tue Mar 25, 2014 7:27 pm
by id10t
Hi!

[DELETED]

Re: Containers count mismatch

Posted: Tue Mar 25, 2014 7:39 pm
by id10t
Hi!

[DELETED]

Re: Containers count mismatch

Posted: Wed Mar 26, 2014 7:45 am
by nnani
about Calculations

Partition - 106
Scale factor = ? ( Is this the k-saftey value)
Nodes = 5

Where can I see How many containers does a projection have.

Code: Select all

select node_name,count(distinct(storage_oid)) from v_monitor.storage_containers where storage_type='ROS' and
schema_name=xx and projection_name like 'xx' group by node_name
I tried the above sql

It gives me a count of 1296 across the cluster

Can you please help me understand this.
I was under the assumption, if there are 106 partitions on a table the containers will also be 106 per node for that table

Thanks in Advance

Re: Containers count mismatch

Posted: Wed Mar 26, 2014 9:48 am
by nnani
Hi,

I went through documentation for scaling factor.

How is Local segmentation different from Projection Segmentation?
What is the relation between ROS containers and Local Segmentation
Local data segmentation increases the number of storage containers stored on each node. This is not an issue unless a table contains many partitions. For example, if the table is partitioned by day and contains one or more years. If local data segmentation is enabled, then each of these table partitions is broken into multiple local storage segments, which potentially results in a huge number of files which can lead to ROS "pushback."
According to the document:
A partition is further segmented if local segmentation is enabled....what does this mean ?
How many container will a partition result into if the scaling factor is 4?


Confused with this New Topic :roll: But happy to leanr something new :)

Re: Containers count mismatch

Posted: Wed Mar 26, 2014 1:00 pm
by NorbertKrupa
Locally segmented means the segmentation takes place on each node. Normally segmented projections are distributed across multiple nodes.

Re: Containers count mismatch

Posted: Wed Mar 26, 2014 2:44 pm
by scutter
Local segmentation is used to speed up cluster rebalancing. For each partition, instead of getting a single ROS container per partition, the data will be pre-split into multiple ROS containers according to the scaling factor. Each of those is a “local segment”. If set to 4, you’ll have 4 ROS containers / local segments per partition. When a cluster rebalance occurs, those “local segments” are redistributed around the cluster.

When local segmentation is on, the data is pre-split into these local segments and the rebalance will be as fast as possible. If local segmentation is off, then rebalance will first split up ROS containers into local segments according to the scaling factor, and then redistribute the segments.

Whether to have local segmentation on or off depends on how often you rebalance, and your needs for partition granularlity and data retention. If you have daily partitioning, local segmentation on, and scaling factor set to 4, then a year’s worth of data would already exceed the maximum number of ROS containers per projection.

Note that “local resegmentation” is also used in explain plans for GROUP BY operations. That is a different concept.

—Sharon