RHEL7 Control Groups on Oracle Database Servers

After some linux patching on some database servers I was surprised to see an entry in an alert log I hadn’t seen before…complete with a WARNING message.  After digging around I found it was due to Linux control groups.  In an effort to manage a server with multiple workloads, RHEL7 introduced control groups and they’ve increased features/abilities with each dot release.  On the affected  Oracle nodes in RHEL7 we see entries on startup in the alert log:

************************ Large Pages Information *******************

Parameter use_large_pages = ONLY

Per process system memlock (soft) limit = UNLIMITED

Large page usage restricted to processor group "user.slice"

Total Shared Global Region in Large Pages = 60 GB (100%)

WARNING:

  The parameter _linux_prepage_large_pages is explicitly disabled.

  Oracle strongly recommends setting the _linux_prepage_large_pages

  parameter since the instance  is running in a Processor Group. If there is

  insufficient large page memory, instance may encounter SIGBUS error

  and may terminate abnormally.

Large Pages used by this instance: 30721 (60 GB)

Large Pages unused in Processor Group user.slice = 1722 (3444 MB)

Large Pages configured in Processor Group user.slice = 120000 (234 GB)

Large Page size = 2048 KB

You may also see:

Large page usage restricted to processor group "system.slice/oracle-ohasd.service"

…this concerned me because if our large page usage is restricted and we can’t do a prepage, eventually the db_cache would warm up and allocate memory beyond the limitation, causing an instance crash.  Also, “WARNING” on startup is something to take a closer look at.  That undocumented parameter was most definitely NOT explicitly disabled.  This message appears in every database that’s being controlled by processor groups, so it appears prepage of large pages is disabled automatically.  There’s a note (2414778.1) that says this message can be ignored and its removed in 12.2+.

So…what amount of memory is our group being limited to by default?

>systemctl show user.slice|grep Mem

MemoryCurrent=18446744073709551615

MemoryAccounting=no

MemoryLimit=18446744073709551615

…which works out to be 17,179,869,184GB, (roughly 17 Exabytes) which is probably a sufficient amount of RAM (per node) for any database out there…so we’re basically being limited to slightly less than infinity….nothing to worry about.

Are control groups useful in a multi-database instance server environment?  To set up CPU limits per instance, you could use this or instance caging…I would prefer to set the cpu_count parameter and use instance caging.  Memory Limits?  Memory utilization for the SGA is set via sga_target/max_size…so I don’t think there’s a play for it here either.  Maybe the accounting features could be useful…if you wanted to see how much CPU one instance in a multiple instance db server was using historically?  There are awr tables that track that…but getting the information from CG might be easier….

One potentially useful way to use control groups for a database server with multiple instances is to throttle bandwidth on IO to prevent one instance from using up all the IO and affecting the performance of a different instance (assuming each instance is on its own asm disks or luns.)  There’s a note about how to get that to work on Metalink: 2495924.1.  This has been a useful feature in Exadata for years with IORM…now you can have similar functionality without Exadata.

One other interesting feature of control groups is the ability to look at the resources each group is using in real time (if you have accounting on) with the systemd-cgtop command.  If you don’t have accounting, you’ll see something like this, which is a summary of process counts, cpu, memory and IO for each group:

It took some time to find the details about what the new alert log entries were telling me with the new warning and processor group information.  Hopefully this information will save you some time.