I'm a new user on HP Vertica and I'm trying to load some data on it.
My server is: 6 cores and 16GB memory (around 11GB free).
When I run my load process I get the following error :
[Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 2142854, Free = 2 (Limit = 12865318, Used = 12865316)]
I thought that the problem was that VERTICA it was trying to use more memory then available, so I reduced MAXMEMORYSIZE of general pool to 10G and also to 5G. But the problem kept:
[Vertica][VJDBC](3587) ERROR: Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 1154611, Free = 0 (Limit = 4626636, Used = 4626636)]
Changing the general pool parameters, my load process worked only when I raised planned_concurrency from 6 (AUTO) to 10.
Why do I get a memory error if i have more concurrency threads than planned_concurrency even if memory_inuse_kb is less than max_memory_size_kb?
Kleyson Rios.
[Vertica][VJDBC](3587) ERROR: Insufficient resources to ...
Moderator: NorbertKrupa
- JimKnicely
- Site Admin
- Posts: 1825
- Joined: Sat Jan 21, 2012 4:58 am
- Contact:
Re: [Vertica][VJDBC](3587) ERROR: Insufficient resources to
Hi,
Do you get any results from this query?
error_level: WARNING
message: Vertica suggests 2GB of memory or more per core
Do you get any results from this query?
- select distinct error_level, message from error_messages where message ilike 'Vertica suggests%';
error_level: WARNING
message: Vertica suggests 2GB of memory or more per core
Jim Knicely
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Re: [Vertica][VJDBC](3587) ERROR: Insufficient resources to
From this query
I get the following messages:
During my loding process I have peaks of 10 running_query_count and when running_query_count reach the same size of planned_concurrency the process stops until get the error.
Do I have to be sure that planned_concurrency allways will be greater than running_query_count ?
Kleyson Rios.
Code: Select all
select error_level, message from error_messages order by event_timestamp desc;
When my process is waiting my general pool is:ERROR | Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 9152, Free = 1 (Limit = 9607373, Used = 9607372)]
ERROR | Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 4544, Free = 1 (Limit = 9607373, Used = 9607372)]
ERROR | Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 1919836, Free = 1 (Limit = 9607373, Used = 9607372)]
ERROR | Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 1919836, Free = 1 (Limit = 9607373, Used = 9607372)]
ERROR | Insufficient resources to execute plan on pool general [Timedout waiting for resource request: Request exceeds limits: Memory(KB) Exceeded: Requested = 1919836, Free = 1 (Limit = 9607373, Used = 9607372)]
My server has 12 cores and 16GB.node_name | v_biprod_node0001
pool_oid | 45035996273718906
pool_name | general
is_internal | t
memory_size_kb | 10113024
memory_size_actual_kb | 10113024
memory_inuse_kb | 9607372
general_memory_borrowed_kb | 0
queueing_threshold_kb | 9607373
max_memory_size_kb | 10113024
running_query_count | 5
planned_concurrency | 5
max_concurrency |
is_standalone | t
queue_timeout_in_seconds | 300
execution_parallelism | AUTO
priority | 0
runtime_priority | MEDIUM
runtime_priority_threshold | 2
single_initiator | false
query_budget_kb | 1919836
During my loding process I have peaks of 10 running_query_count and when running_query_count reach the same size of planned_concurrency the process stops until get the error.
Do I have to be sure that planned_concurrency allways will be greater than running_query_count ?
Kleyson Rios.
- JimKnicely
- Site Admin
- Posts: 1825
- Joined: Sat Jan 21, 2012 4:58 am
- Contact:
Re: [Vertica][VJDBC](3587) ERROR: Insufficient resources to
If I were you I'd leave PLANNEDCONCURRENCY set to auto (Memory/2GB) and change the MAXCONCURRENCY for the general pool. It seems like you are simply running out of resources running the 10 COPY commands concurrently. Remember that the general pool is limited to 95% of available RAM.
Jim Knicely
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Re: [Vertica][VJDBC](3587) ERROR: Insufficient resources to
I am not sure about running out of resources.
As I said before, my loading process has peak of 10 running_query_count. If I set up planned_concurrency for some number below I get the error, but any number above, such as 12 or 20, the process finish successfuly. So, my resources is enough to run my process. The problem is when running_query_count reach planned_concurrency number.
I would like to understand that behavior.
Thanks for the help.
Kleyson Rios.
As I said before, my loading process has peak of 10 running_query_count. If I set up planned_concurrency for some number below I get the error, but any number above, such as 12 or 20, the process finish successfuly. So, my resources is enough to run my process. The problem is when running_query_count reach planned_concurrency number.
I would like to understand that behavior.
Thanks for the help.
Kleyson Rios.
- JimKnicely
- Site Admin
- Posts: 1825
- Joined: Sat Jan 21, 2012 4:58 am
- Contact:
Re: [Vertica][VJDBC](3587) ERROR: Insufficient resources to
Hi,
I'm taking a stab here ...
In the "Target Memory Determination for Queries in Concurrent Environments" section of the Admin Guide, there is this statement:
Again, I'm guessing
I'm taking a stab here ...
In the "Target Memory Determination for Queries in Concurrent Environments" section of the Admin Guide, there is this statement:
Your queing thresh hold is 9607373 KB (~9 GB) and your memory size is 10113024 KB (~10 GB). So as you increase the PLANNEDCONCURRENCY, you are increasing the amount of memory available to each query. I guess you eventually freed up enough so that each command could complete. With a low PLANNEDCONCURRENCY, the queries that run early, get the bulk of available memory, leaving the later queries to starve for memory.If the resource pool for the query has the MEMORYSIZE parameter set, and the pool is standalone (i.e. cannot borrow from General pool) then the target memory is to use the amount of memory in the Queuing Threshold of the pool / PLANNEDCONCURRENCY.
Again, I'm guessing
Jim Knicely
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Re: [Vertica][VJDBC](3587) ERROR: Insufficient resources to
Hi,
I think the explanation by Jim makes sense
Thanks for that Jim.
I think the explanation by Jim makes sense
If you increase the PLANNEDCONCURRENCY then each of your 10 queries will try to get equal resources. and if the resources are adequate to make them run, then they will run without any errors.You are increasing the amount of memory available to each query. I guess you eventually freed up enough so that each command could complete. With a low PLANNEDCONCURRENCY, the queries that run early, get the bulk of available memory, leaving the later queries to starve for memory.
Thanks for that Jim.