最近程序报错:Failed to connect to redis: Connection timed out
发现THP的有关信息,决定做测试及修改
1.page 与 Huge Pages
page:
一般而言,内存管理的最小块级单位叫做page,一个page是4096bytes,1M的内存会有256个page,1GB的话就会有256,000个page。
CPU通过内置的内存管理单元维护着page表记录。
Huge pages:
Huge Pages就是大小为2M到1GB的内存page,主要用于管理数千兆的内存,比如1GB的page对于1TB的内存来说是相对比较合适的。
HugePage广泛启用开始于Kernal 2.6,一些版本下2.4内核也可以是用。在操作系统Linux环境中,内存是以页Page的方式进行分配,默认大小为4K。如果需要比较大的内存空间,则需要进行频繁的页分配和管理寻址动作。
HugePage是传统4K Page的替代方案。顾名思义,是用HugePage可以让我们有更大的内存分页大小。无论是HugePage还是传统的正常Page,这个过程都涉及到OS内存寻址过程
当一个进程访问内存的时候,并不是直接进行内存位置访问,是需要通过Page Table进行转移变换。在使用HugePage的情况下,PageTable具有了额外的属性,就是判断该页记录是HugePage还是Regular Page。
2.THP
THP(Transparent Huge Pages)是一个使管理Huge Pages自动化的抽象层。
Linux本身的页大小是固定的4KB,在2.6.38内核新增了THP,透明地支持huge page(2MB)的使用,并且默认开启。
THP优点:
(1) 减少page fault。一次page fault可以加载更大的内存块。
(2) 更小的页表。相同的内存大小,需要更少的页。
(3) 由于页表更小,虚拟地址到物理地址的翻译也更快。
THP缺点:
(1)降低分配内存效率。需要大块、连续内存块,内核线程会比较激进的进行compaction,解决内存碎片,加剧锁争用。
(2)降低IO吞吐。由于swapable huge page,在swap时需要切分成原有的4K的页。Oracle的测试数据显示会降低30%的IO吞吐。
3.测试性能
查看系统版本:
uname -r Linux ckl-master1 2.6.32-504.el6.x86_64
查看当前的Huge pages设置:
grep Huge /proc/meminfo AnonHugePages: 5971968 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB
查看是否开启了Huge pages:
cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise never
测试THP开启redis性能:
# /usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000 ====== SET ====== 10000000 requests completed in 41.43 seconds 50 parallel clients 3 bytes payload keep alive: 1 98.62% <= 1 milliseconds 98.62% <= 2 milliseconds 98.63% <= 4 milliseconds 98.63% <= 5 milliseconds 98.63% <= 7 milliseconds 98.63% <= 12 milliseconds 98.64% <= 13 milliseconds 98.64% <= 14 milliseconds 98.65% <= 15 milliseconds 98.66% <= 16 milliseconds 98.68% <= 17 milliseconds 98.69% <= 18 milliseconds 98.74% <= 19 milliseconds 98.83% <= 20 milliseconds 98.96% <= 21 milliseconds 99.13% <= 22 milliseconds 99.32% <= 23 milliseconds 99.49% <= 24 milliseconds 99.60% <= 25 milliseconds 99.67% <= 26 milliseconds 99.74% <= 27 milliseconds 99.79% <= 28 milliseconds 99.81% <= 29 milliseconds 99.84% <= 30 milliseconds 99.85% <= 31 milliseconds 99.86% <= 32 milliseconds 99.86% <= 33 milliseconds 99.87% <= 34 milliseconds 99.87% <= 35 milliseconds 99.88% <= 36 milliseconds 99.89% <= 37 milliseconds 99.89% <= 38 milliseconds 99.89% <= 39 milliseconds 99.89% <= 40 milliseconds 99.90% <= 42 milliseconds 99.90% <= 43 milliseconds 99.91% <= 44 milliseconds 99.91% <= 49 milliseconds 99.91% <= 53 milliseconds 99.91% <= 54 milliseconds 99.92% <= 55 milliseconds 99.92% <= 56 milliseconds 99.93% <= 57 milliseconds 99.93% <= 62 milliseconds 99.93% <= 63 milliseconds 99.93% <= 65 milliseconds 99.93% <= 66 milliseconds 99.93% <= 67 milliseconds 99.94% <= 68 milliseconds 99.94% <= 74 milliseconds 99.94% <= 76 milliseconds 99.94% <= 77 milliseconds 99.94% <= 78 milliseconds 99.95% <= 79 milliseconds 99.95% <= 81 milliseconds 99.95% <= 82 milliseconds 99.95% <= 83 milliseconds 99.95% <= 85 milliseconds 99.96% <= 86 milliseconds 99.96% <= 91 milliseconds 99.96% <= 92 milliseconds 99.97% <= 93 milliseconds 99.97% <= 95 milliseconds 99.97% <= 96 milliseconds 99.98% <= 97 milliseconds 99.99% <= 98 milliseconds 99.99% <= 99 milliseconds 99.99% <= 100 milliseconds 99.99% <= 101 milliseconds 99.99% <= 126 milliseconds 100.00% <= 157 milliseconds 100.00% <= 158 milliseconds 100.00% <= 168 milliseconds 100.00% <= 169 milliseconds 100.00% <= 169 milliseconds 241382.62 requests per second
关闭THP:
echo never > /sys/kernel/mm/transparent_hugepage/enabled cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never]
测试关闭THPredis性能:
/usr/local/redis/bin/redis-benchmark -P 4 -t set -r 5000000 -n 10000000 ====== SET ====== 10000000 requests completed in 26.72 seconds 50 parallel clients 3 bytes payload keep alive: 1 99.70% <= 1 milliseconds 99.71% <= 2 milliseconds 100.00% <= 3 milliseconds 100.00% <= 3 milliseconds 374321.53 requests per second
关闭THP后,redis的响应时间明显缩短,建议关闭THP,需要重启redis进程