- Guarantee completion in a reasonable time frame for critical workloads.
- Support reasonable cluster scheduling between groups of users based on fair allocation of resources per group.
- Prevent ad-hoc users from damaging the cluster for other users.
Cloudera Manager 4.5 introduced the ability to partition resources across HBase, HDFS, Impala, MapReduce, and YARN services by setting configuration properties that were enforced by Linux control groups (Linux cgroups). With Cloudera Manager 5, the ability to statically allocate resources using cgroups is now configurable through a single static service pool user interface. Within those static allocations, for YARN applications and Impala queries, you can also allocate resources using YARN dynamic resource pools.
Static partitioning isolates the services in your cluster from one another, enabling predictable behavior across services so that load on one service has a bounded impact on other services. Static partitioning is implemented using cgroups, but the configuration interface allows you to allocate resources across all services at once allowing Cloudera Manager to compute the amount of resources allocated to each service.
Dynamic partitioning (resources shared by consumers across a service) controls access to a service's resources based on a user's access to pools of cluster resources. Pools can be organized by groups of users, types of applications, types of queries, or any other framework that makes sense in the user environment. Users may submit requests to specific pools, and resources are allocated to their jobs based on a number of factors including scheduling algorithms, availability of resources and so on.
For information on configuring schedulers for use with MapReduce as opposed to YARN, see Configuring the MapReduce Scheduler.