Hard Partitioning With Oracle VM Server for SPARC
Introduction
This document describes hard partitioning with Oracle VM Server for SPARC, and how to use it to conform to the Oracle licensing policies for partitioned environments.
CPU Cores and CPU Threads
Oracle VM Server for SPARC software runs on SPARC servers. Each SPARC processor has multiple CPU cores, and each CPU core has multiple CPU threads (virtual CPUs). By default, domains created with Oracle VM Server for SPARC are configured in units of CPU threads. Oracle VM Server for SPARC uses CPU affinity to try to assign each domain’s virtual CPUs on its own cores for optimal performance. This can be made mandatory by using the “whole core constraint” to require that CPUs are assigned in units of entire CPU cores. When binding such a domain, the system provisions the specified number of CPU cores and all their CPU threads to the domain.
For a list of supported SPARC systems and more information about CPU whole-core configurations, see the Release Notes and Administration Guide of Oracle VM Server for SPARC Documentation.
Hard Partitioning and CPU Whole Cores
Beginning with the Oracle VM Server for SPARC 2.0 release, hard partitioning is enforced by using CPU whole-core configurations and specifying the maximum number of cores that can be assigned to the domain. In this case, domains are configured with CPU whole cores, instead of the default of individual CPU threads. Using a whole-core configuration and setting a maximum CPU cap limits the number of CPU cores that can be dynamically assigned to a bound or active domain.
Hard Partitioning With Oracle VM Server for SPARC
Oracle Hard Partition Licensing
To conform to the Oracle hard partition licensing requirement, you must use at least Oracle VM Server for SPARC 2.0 release and must use CPU whole cores as follows:
If a domain runs applications using Oracle hard partition licensing, that domain must be configured with CPU whole cores, and the CPU cap (maximum number of CPU cores) must be set for the domain. The CPU cap must be the number of CPU cores that is specified by the Oracle software license.
If a domain does not run applications that use Oracle hard partition licensing, that domain does not need to be configured with CPU whole cores or a CPU cap. For example, if you do not run any Oracle applications in a domain, that domain does not need to be configured with CPU whole cores or a CPU cap.
Hard Partitioning With Oracle VM Server for SPARC
Checking the Configuration of a Domain
To Check Whether a Domain is Configured With CPU Whole Cores and a CPU Cap
1. Obtain the resources management information for the domain.
# ldm list -o resmgmt domain
Verify that the whole-core constraint appears in the output and that the max-cores keyword specifies the maximum number of CPU cores configured for the domain.
For more information, see the ldm(1M) man page of the Reference Manual in the Oracle VM Server for SPARC Documentation.
EXAMPLE – Checking Whether a Domain Is Configured With CPU Whole Cores and a CPU Cap
The following command shows that ldg1 domain is configured with CPU whole cores and a maximum of five cores:
# ldm list -o resmgmt ldg1
NAME ldg1
CONSTRAINT
whole-core
max-cores=5
To List the CPU Cores That Are Assigned to a Domain
CPU cores are assigned to a domain when a domain is bound.
1. Obtain the list of CPU cores that are assigned to a domain.
# ldm list -o core domain
EXAMPLE – Listing the CPU Cores That Are Assigned to a Domain
The following command shows the cores that are assigned to domain ldg1:
# ldm list -o core ldg1
NAME ldg1
CORE
CID PCPUSET
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
Configuring a Domain With CPU Whole Cores and a CPU Cap
Use the following commands to configure a domain to use CPU whole cores and set the maximum number of CPU cores (CPU cap) for the domain:
ldm set-core number-of-cpu-cores domain
ldm set-domain max-cores=max-number-of-cpu-cores domain
If you are using version 2.0 or 2.1 of the Logical Domains Manager to assign whole cores to domains, the ldm add-core, ldm set-core, ldm remove-core, and ldm set-domain max-core commands are not available. Instead, you must use the ldm add-vcpu –c, ldm set-vcpu –c, and ldm remove-vcpu –c commands. These commands also set the CPU cap for the domain. See the ldm(1M) man page of the Reference Manual in the Oracle VM Server for SPARC Documentation.
To Create a New Domain With CPU Whole Cores and a CPU Cap
When creating a new domain, you can configure it to use CPU whole cores.
1. Create the domain.
# ldm create domain
2. Set the number of CPU whole cores and the CPU cap for the domain.
# ldm set-core number-of-cpu-cores domain
# ldm set-domain max-cores=max-number-of-cpu-cores domain
3. Configure the domain.
Caution –During this configuration, do not use the ldm add-vcpu, ldm set-vcpu or ldm rm-vcpu command. If you do so, the domain is reconfigured with individual CPU threads instead of CPU whole cores. Ensure that you use the ldm add-core, ldm set-core, or ldm remove-core command, or the –c option of the ldm add-vcpu, ldm set-vcpu or ldm rm-vcpu command if you use version 2.0 or 2.1 of Logical Domains Manager.
4. Bind and start the domain.
# ldm bind domain
# ldm start domain
EXAMPLE – Creating a New Domain With Two CPU Whole Cores and a CPU Cap of Two Cores
This example shows how to create a domain, ldg1, with two CPU whole cores. The following command creates the ldg1 domain:
# ldm create ldg1
The following commands configure the ldg1 domain with two CPU whole cores and set the CPU cap to two cores:
# ldm set-core 2 ldg1
# ldm set-domain max-cores=2 ldg1
At this point, you can perform further configuration of the domain subject to the restrictions described in Step 3 of the previous procedure. Before you use the ldg1 domain, you must bind and start it, which is done by the following commands:
# ldm bind ldg1
# ldm start ldg1
To Configure an Existing Domain With CPU Whole Cores and a CPU Cap
If a domain already exists and is configured to use CPU threads, you can change its configuration to use CPU whole cores and a CPU cap.
1. Stop and unbind the domain. This step is not needed to configure the domain to use CPU whole cores; it is only needed to set the CPU cap.
# ldm stop domain
# ldm unbind domain
2. Set the number of CPU whole cores for the domain and set the CPU cap.
# ldm set-core number-of-cpu-cores domain
# ldm set-domain max-cores=max-number-of-cpu-cores domain
3. Rebind and restart the domain if it was stopped in step 1.
# ldm bind domain
# ldm start domain
EXAMPLE – Configuring an Existing Domain With Four Whole Cores and a CPU Cap of Eight Cores
This example shows how to update the configuration of an existing domain, ldg1. Four CPU whole cores are configured for the domain and the CPU cap is set to eight cores. This enables you to add up to four more whole cores to the domain, if needed.
The following commands stop and unbind the ldg1 domain:
# ldm stop ldg1
# ldm unbind ldg1
The following commands configure the ldg1 domain with four CPU whole cores and set the CPU cap to eight. Note that the ldm set-core command can be done while the domain is active.
# ldm set-core 4 ldg1
# ldm set-domain max-cores=8 ldg1
The following commands bind and restart the ldg1 command:
# ldm bind ldg1
# ldm start ldg1
To Configure the Primary Domain With CPU Whole Cores and a CPU Cap
If the primary domain is configured to use CPU threads, you can change its configuration to use CPU whole cores and a CPU cap.
1. Set the number of CPU whole cores for the primary domain.
# ldm set-core number-of-cpu-cores primary
2. Put the primary domain in delayed reconfiguration mode. This step is not needed to change the domain to use whole cores, but it is needed in order to set a CPU cap.
# ldm start-reconf primary
3. Set the CPU cap for the primary domain.
# ldm set-domain max-cores=max-number-of-cpu-cores primary
4. Reboot the primary domain.
Use the appropriate procedure to reboot the primary domain, which depends on the system configuration. See “Rebooting the primary Domain” in the Administration Guide of the Oracle VM Server for SPARC Documentation.
EXAMPLE – Configuring the Primary Domain With Two Whole Cores and a CPU Cap of Two Cores
This example shows how to configure CPU whole cores on the primary domain. First, the following command specifies the number of CPU whole cores to assign to the primary domain:
# ldm set-core 2 primary
The following command initiates delayed reconfiguration mode on the primary domain:
# ldm start-reconf primary
The following command configures the CPU cap for the primary domain to two:
# ldm set-domain max-cores=2 primary
The following command reboots the primary domain:
# shutdown -i 6
Interaction With Other Features
CPU Dynamic Reconfiguration
You can use CPU dynamic reconfiguration with domains that are configured with CPU whole cores. However, you can only add or remove entire CPU cores, not individual CPU threads. Using CPU dynamic reconfiguration in this way maintains the hard partitioning of the system. In addition, when you dynamically add CPU cores to a domain, the command fails if it exceeds the maximum number of CPU cores for the domain.
Use the following commands to dynamically add, set, or remove CPU whole cores to a bound or active domain:
ldm add-core number-of-cpu-cores domain
ldm set-core number-of-cpu-cores domain
ldm remove-core number-of-cpu-cores domain
To conform to the Oracle licensing policy requirements, you must set the CPU cap by using the following command:
ldm set-domain max-cores=max-number-of-cpu-cores domain
Or, if you use version 2.0 or 2.1 of Logical Domains Manager, use the ldm add-vcpu –c, ldm set-vcpu –c, and ldm remove-vcpu –c commands for dynamic reconfiguration. These commands also set the CPU cap.
EXAMPLE – Dynamically Adding Two CPU Whole Cores to a Domain
This example shows how to dynamically add two CPU whole cores to domain ldg1, which is an active domain configured with CPU whole cores. The first command shows that the ldg1 domain is active. The second command shows that the ldg1 domain is configured with CPU whole cores, and a maximum of four CPU cores.
# ldm list ldg1
NAME STATE FLAGS CONS VCPU MEMORY UTIL UPTIME
ldg1 active -n---- 5000 16 2G 0.4% 5d 17h 49m
# ldm list -o resmgmt ldg1
NAME
ldg1
CONSTRAINT
whole-core
max-cores=4
The following command shows the CPU cores assigned to the domain before adding two CPU whole cores:
# ldm list -o core ldg1
NAME
ldg1
CORE
CID PCPUSET
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
The following command dynamically adds two CPU whole cores to the ldg1 domain. This is permitted because the total number of cores does not exceed the maximum of four:
# ldm add-core 2 ldg1
The following command shows the CPU cores assigned to the domain after adding two CPU whole cores:
# ldm list -o core ldg1
NAME ldg1
CORE
CID PCPUSET
1 (8, 9, 10, 11, 12, 13, 14, 15)
2 (16, 17, 18, 19, 20, 21, 22, 23)
3 (24, 25, 26, 27, 28, 29, 30, 31)
4 (32, 33, 34, 35, 36, 37, 38, 39)
CPU Dynamic Resource Management
If Dynamic Resource Management (DRM) is used to automatically manage CPU resources on some domains, DRM policies do not apply to those domains that are configured with CPU whole cores.
A DRM policy can include a domain that is configured with CPU whole cores. However, when such a policy is activated, the DRM policy is automatically disabled for that domain, which remains configured with CPU whole cores. If the domain is later reconfigured with CPU threads instead of CPU whole cores, the DRM policy is automatically re-enabled for that domain.
CPU Power Management
You can use CPU power management on domains that have either the performance or elastic policy set and configure the domains with CPU whole cores. Using CPU power management maintains the hard partitioning of the system.
Domain Reboot or Rebind
A domain that is configured with CPU whole cores remains configured with CPU whole cores when it is restarted, or if the entire platform is restarted. A domain uses the same physical CPU cores for the entire time it remains bound. For example, if a domain is rebooted, it uses the exact same physical CPU cores both before and after the reboot. Or, if the entire platform is powered off while a domain is bound, that domain will be configured with the exact same physical CPU cores when the platform is powered on again. If you unbind a domain and then rebind it, or if the entire platform is restarted with a new configuration, the domain might use different physical CPU cores.
Domain Migration
Starting with Oracle VM Server for SPARC 3.0, CPU whole-core configuration is compatible with domain migration. You can migrate a domain that is configured with CPU whole cores, and whole core allocation is enforced on the target system. Maximum CPU cores, if set, are also enforced on the target system. In prior releases, the whole-core configuration and the maximum number of CPU cores are not preserved by the migration to the target system.
Live migration of guest domains with whole cores and a CPU cap does not conform to the terms of the hard partitioning license.
Conclusion
With the Oracle VM Server for SPARC software, you must use the whole-core constraint and the CPU cap constraint to conform to the Oracle hard partition licensing requirement. In such a case, domains are configured with CPU whole cores, instead of the default of individual CPU threads. Using a CPU whole-core configuration with the CPU cap set also adds some restrictions to features, such as CPU dynamic reconfiguration beyond the CPU cap, CPU dynamic resource management, and domain migration.
For more information about Oracle's virtualization solutions, visit oracle.com/virtualization.
Source:
http://www.oracle.com/technetwork/server-storage/vm/ovm-sparc-hard-partitioning-1403135.pdf
