Configuring HugePages for Oracle on Linux (x86-64)

Introduction

For large SGA sizes, HugePages can give substantial benefits in virtual memory management. Without HugePages, the memory of the SGA is divided into 4K pages, which have to be managed by the Linux kernel. Using HugePages, the page size is increased to 2MB (configurable to 1G if supported by the hardware), thereby reducing the total number of pages to be managed by the kernel and therefore reducing the amount of memory required to hold the page table in memory. In addition to these changes, the memory associated with HugePages can not be swapped out, which forces the SGA to stay memory resident. The savings in memory and the effort of page management make HugePages pretty much mandatory for Oracle 11g systems running on x86-64 architectures.

Just because you have a large SGA, it doesn‘t automatically mean you will have a problem if you don‘t use HugePages. It is typically the combination of a large SGA and lots database connections that leads to problems. To determine how much memory you are currently using to support the page table, run the following command at a time when the server is under normal/heavy load.

# grep PageTables /proc/meminfo
PageTables:      1244880 kB
#

Automatic Memory Management (AMM) is not compatible with Linux HugePages, so apart from ASM instances and small unimportant databases, you will probably have no need for AMM on a real database running on Linux. Instead, Automatic Shared Memory Management and Automatic PGA Management should be used as they are compatible with HugePages.

Configuring HugePages

Run the following command to determine the current HugePage usage. The default HugePage size is 2MB on Oracle Linux 5.x and as you can see from the output below, by default no HugePages are defined.

$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$

Depending on the size of your SGA, you may wish to increase the value of Hugepagesize to 1G.

Create a file called "hugepages_setting.sh" with the following contents.

#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
# Check for the kernel version
KERN=`uname -r | awk -F. ‘{ printf("%d.%d\n",$1,$2); }‘`
# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk {‘print $2‘}`
# Start from 1 pages to be on the safe side and guarantee 1 free HugePage
NUM_PG=1
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | awk {‘print $5‘} | grep "[0-9][0-9]*"`
do
   MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
   if [ $MIN_PG -gt 0 ]; then
      NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
   fi
done
# Finish with results
case $KERN in
   ‘2.4‘) HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
          echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
   ‘2.6‘ | ‘3.8‘) echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    *) echo "Unrecognized kernel version $KERN. Exiting." ;;
esac
# End

Thanks to Bjoern Rost for pointing out the issue when using the script against UEK3 and the suggested fix.

Make the file executable.

$ chmod u+x hugepages_setting.sh

Make sure all the Oracle services are running as normal on the server, then run the script and make a note of the recommended "vm.nr_hugepages" value.

$ ./hugepages_setting.sh
Recommended setting: vm.nr_hugepages = 305
$

Edit the "/etc/sysctl.conf" file as the "root" user, adding the following entry, adjusted based on your output from the script. You should set the value greater than or equal to the value displayed by the script. You only need 1 or 2 spare pages.

vm.nr_hugepages=306

One person reported also needing the hugetlb_shm_group setting on Oracle Linux 6.5. I did not and it is listed as a requirement for SUSE only. If you want to set it, get the ID of the dba group.

# fgrep dba /etc/group
dba:x:54322:oracle
#

Use the resulting group ID in the "/etc/sysctl.conf" file.

vm.hugetlb_shm_group=54322

Run the following command as the "root" user.

# sysctl -p

Alternatively, edit the "/etc/grub.conf" file, adding "hugepages=306" to the end of the kernel line for the default kernel and reboot.

You can now see the HugePages have been created, but are currently not being used.

$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     306
HugePages_Free:      306
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
$

Add the following entries into the "/etc/security/limits.conf" script, where the setting is at least the size of the HugePages allocation in KB (HugePages * Hugepagesize). In this case the value is 306*2048=626688.

* soft memlock 626688
* hard memlock 626688

If you prefer, you can set these parameters to a value just below the size of physical memory of the server. This way you can forget about it, unless you add more physical memory.

Check the MEMORY_TARGET parameters are not set for the database and SGA_TARGET and PGA_AGGREGATE_TARGET parameters are being used instead.

SQL> show parameter target

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
archive_lag_target                   integer     0
db_flashback_retention_target        integer     1440
fast_start_io_target                 integer     0
fast_start_mttr_target               integer     0
memory_max_target                    big integer 0
memory_target                        big integer 0
parallel_servers_target              integer     16
pga_aggregate_target                 big integer 200M
sga_target                           big integer 600M
SQL>

Restart the server and restart the database services as required.

Check the HugePages information again.

$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:     306
HugePages_Free:       98
HugePages_Rsvd:       93
HugePages_Surp:        0
Hugepagesize:       2048 kB
$

You can see the HugePages are now being used.

Remember, if you increase your memory allocation or add new instances, you need to retest the required number of HugePages, or risk Oracle running without them.

Force Oracle to use HugePages (USE_LARGE_PAGES)

Sizing the number of HugePages correctly is important because prior to 11.2.0.3, if the whole SGA doesn‘t fit into the available HugePages, the instance will start up without using any. From 11.2.0.3 onward, the SGA can run partly in HugePages and partly not, so the impact of this issue is not so great. Incorrect sizing may not be obvious to spot. Later releases of the database display a "Large Pages Information" section in the alert log during startup.

****************** Large Pages Information *****************

Total Shared Global Region in Large Pages = 602 MB (100%)

Large Pages used by this instance: 301 (602 MB)
Large Pages unused system wide = 5 (10 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 306 (612 MB)
Large Page size = 2048 KB
***********************************************************

If you are running Oracle 11.2.0.2 or later, you can set the USE_LARGE_PAGES initialization parameter to "only" so the database fails to start if it is not backed by hugepages. You can read more about this here.

ALTER SYSTEM SET use_large_pages=only SCOPE=SPFILE;
SHUTDOWN IMMEDIATE;
STARTUP;

On startup the "Large Page Information" in the alert log reflects the use of this parameter.

****************** Large Pages Information *****************
Parameter use_large_pages = ONLY

Total Shared Global Region in Large Pages = 602 MB (100%)

Large Pages used by this instance: 301 (602 MB)
Large Pages unused system wide = 5 (10 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 306 (612 MB)
Large Page size = 2048 KB
***********************************************************

Attempting to start the database when there aren‘t enough HugePages to hold the SGA will now return the following error.

SQL> STARTUP
ORA-27137: unable to allocate large pages to create a shared memory segment
Linux-x86_64 Error: 12: Cannot allocate memory
SQL>

The "Large Pages Information" section of the alert log output describes the startup failure and the appropriate action to take.

****************** Large Pages Information *****************
Parameter use_large_pages = ONLY

Large Pages unused system wide = 0 (0 KB) (alloc incr 4096 KB)
Large Pages configured system wide = 0 (0 KB)
Large Page size = 2048 KB

ERROR:
  Failed to allocate shared global region with large pages, unix errno = 12.
  Aborting Instance startup.
  ORA-27137: unable to allocate Large Pages to create a shared memory segment

ACTION:
  Total Shared Global Region size is 608 MB. Increase the number of
  unused large pages to atleast 304 (608 MB) to allocate 100% Shared Global
  Region with Large Pages.
***********************************************************

Disabling Transparent HugePages (RHEL6/OL6)

Starting from RHEL6/OL6, Transparent HugePages are implemented and enabled by default. They are meant to improve memory management by allowing HugePages to be allocated dynamically by the "khugepaged" kernel thread, rather than at boot time like conventional HugePages. That sounds like a good idea, but unfortunately Transparent HugePages don‘t play well with Oracle databases and are associated with node reboots in RAC installations and performance problems on both single instance and RAC installations. As a result Oracle recommends disabling Transparent HugePages on all servers running Oracle databases, as described in this MOS note.

You can check the current setting using the following command, which is displaying the default value of "enabled=[always]".

# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
#

The preferred method to disable Transparent HugePages is to add "transparent_hugepage=never" to the kernel boot line in the "/etc/grub.conf" file.

title Oracle Linux Server (2.6.39-400.24.1.el6uek.x86_64)
        root (hd0,0)
        kernel /vmlinuz-2.6.39-400.24.1.el6uek.x86_64 ro root=/dev/mapper/vg_ol6112-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=uk
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16  rd_NO_DM rd_LVM_LV=vg_ol6112/lv_swap rd_LVM_LV=vg_ol6112/lv_root rhgb quiet numa=off
transparent_hugepage=never
        initrd /initramfs-2.6.39-400.24.1.el6uek.x86_64.img

The server must be rebooted for this to take effect.

Alternatively, add the following lines into the "/etc/rc.local" file and reboot the server.

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
   echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

Whichever method you choose, remember to check the change has work after reboot.

# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
#

With Transparent HugePages disabled, you should proceed to configure conventional HugePages, as described above.

Configuring 1G Hugepagesize

As mentioned by Eugene in the comments, Oracle currently don‘t recommend using 1G Hugepagesize. You can read more about this in MOS Doc ID 1607545.1. With that in mind, the rest of this section should probably be considered more of an academic exercise.

Check if your current hardware can support a Hugepagesize of 1G. If the following command produces any output, it can.

# cat /proc/cpuinfo | grep pdpe1gb

Thanks to Kevin Closson for pointing out the hardware support requirement.

Edit the "/etc/grub.conf" file, adding the following entries on to the kernel line of the default grub entry. Adjust the "hugepages" entry to the desired number of 1G pages. Notice this includes the disabling of Transparent HugePages, which is not mandatory, but a good idea.

transparent_hugepage=never hugepagesz=1G hugepages=1 default_hugepagesz=1G

Check the current HugePages setup.

# grep Huge /proc/meminfo
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
#

Reboot and check the HugePages setup again.

#  grep Huge /proc/meminfo
HugePages_Total:       1
HugePages_Free:        1
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
#

For more information see:

Hope this helps. Regards Tim...

Back to the Top.

时间： 2024-10-12 16:02:22

Configuring HugePages for Oracle on Linux (x86-64)的相关文章

Linux x86 64内核终止D状态的进程

在上一篇文章<Linux x86内核终止D状态的进程>中,我展示了32位x86系统中如何编码杀死D进程.本文我将展示一种64位x86系统上的方法.        说实话,64位系统上做这样的事是比较难的,因为你无法通过修改p->thread.ip来到达将进程拽出死循环的目的.要想知道64位系统上到底该怎么把进程执行绪引出,我们得先看看"标准"的做法是什么. 标准的做法就是fork时的行为,一个新进程刚刚被创建,它第一次进入运行状态之前,并不是通过switch_to切出

栈溢出攻击系列：shellcode在linux x86 64位攻击获得root权限（二）shellcode

shellcode 是一组指令opcode, 是可以被程序运行,因为shellcode是要直接操作寄存器和函数,所以opcode 必须是十六进制的形式. 既然是攻击,那shellcode 主要的目的是调用系统函数,而在x86下 在linux下有两种方式. 第一种是通过直接调用中断 int 0x80进入内核态,从而达到调用目的. 第二种是通过调用libc里syscall(64位)和sysenter(32位) 而目前在64位linux中推荐使用syscall,因为opcode是16进制的指令集合,可

栈溢出攻击系列：shellcode在linux x86 64位攻击获得root权限（一）函数如何执行

栈溢出网上已经有很多的例子了,但是很少会涉及到在64位的和操作系统linux相关的,而最近刚好在一直研究这个,所以写着一系列博文,一来是为了帮助自己记忆,二来也是为了更多的大家互相探讨. 寄存器 X86-64有16个64位寄存器,分别是:%rax,%rbx,%rcx,%rdx,%esi,%edi,%rbp,%rsp,%r8,%r9,%r10,%r11,%r12,%r13,%r14,%r15.其中: %rax 作为函数返回值使用 %rsp 栈指针寄存器,指向栈顶 %rdi,%rsi,%rdx,%r

栈溢出攻击系列：shellcode在linux x86 64位攻击获得root权限（七）利用寄存器攻击

在(六)中我们提到了使用固定栈地址的攻击方式,但在实际中,系统默认的参数不会为0 cat /proc/sys/kernel/randomize_va_space 那么在系列中的六失去攻击的意义,但是任何事情都会有漏洞,我们来讲另一个基于寄存器的攻击 漏洞代码 vulnerableret2reg.c #include <stdio.h> #include <string.h> void evilfunction(char* input) { char buffer[1000]; st

【翻译mos文章】Linux x86 and x86-64 系统中的SHMMAX最大值

Linux x86 and x86-64 系统中的SHMMAX最大值 参考原文: Maximum SHMMAX values for Linux x86 and x86-64 (文档 ID 567506.1) 目标: 问题1:32-bit (x86) Linux system 系统中的最大值? 问题2:64-bit (x86-64) Linux system系统中的最大值? 解决方案: 答案1: Oracle Global Customer Support 官方推荐 SHMMAX 的最大值是小于

【翻译mos文章】Linux x86 and x86-64 系统SHMMAX最大

Linux x86 and x86-64 系统SHMMAX最大值 参考原始: Maximum SHMMAX values for Linux x86 and x86-64 (文件 ID 567506.1) 目标: 问题1:32-bit (x86) Linux system 系统中的最大值? 问题2:64-bit (x86-64) Linux system系统中的最大值? 解决方式: 答案1: Oracle Global Customer Support 官方推荐 SHMMAX 的最大值是小于4G

[书接上一回]在Oracle Enterprise Linux (v5.7) 中安装DB - (1/4)

在上一回中,我们安装了OEL了,现在就要安装Oracle数据. 首先登录root用户,输入账号密码或,输入命令行:startx,启动图形界面. 先将虚拟机中插入光碟(Enterprise-R5-U7-Server-x86_64-dvd.iso),如图下, 击Computer,再点击如下CD-RW/DVD-RAM Drive. 此时,会自动弹出如下的窗口. 进入插入光碟的路径,如下: 查看Server的文件和检查安装以下 RPM packages. 需要的包: libaio-devel-03.10

Oracle基于Linux 7下的安装

一.环境: 1.WMware Workstation 2.linux 7.0 64位 iso 3.安装包: p13390677_112040_Linux-x86-64_1of7.zip p13390677_112040_Linux-x86-64_2of7.zip 二.创建新虚拟机: 文件--新建虚拟机--典型(推荐)--下一步--稍后安装操作系统--linux--版本:Red Hat Enterprise Linux 7 64 位--下一步--虚拟机名称:自定义--位置(L):自行选择虚拟机存放

Oracle Enterprise Linux 64-bit 下Oracle11g的监听配置改动及測试步骤

測试环境:Oracle Enterprise Linux 64-bit (5.8版本号) + Oracle 11g 64位 相关说明: Oracle11g64位软件的安装位置为/u01/app/oracle/product/11.2.0/dbhome_1 ,数据库名为默认的orcl,Linux虚拟机的IP设置为192.168.1.121 一.改动listener.ora文件内容 命令:[[email protected] ~]$ vi /u01/app/oracle/product/11.2.0