Enhancing the Scalability of Memcached

原文地址： https://software.intel.com/en-us/articles/enhancing-the-scalability-of-memcached-0

1 Introduction - Memcached and Web Services

Memcached is a Key-Value cache used by cloud and web service delivery
companies, such as Facebook [1], Twitter [2], Reddit [3], and YouTube [4], to
reduce latency in serving web data to consumers and to ease the demand on
database and computational servers [5]. In addition to lowering latency,
memcached’s scale-out architecture supports increased throughput by simply
adding memcached servers. Scaling up, however, is less rewarding as more than 4
cores causes performance degradation [1] [6].

In this paper, we introduce memcached optimizations to utilize concurrent
data structures, an updated cache replacement strategy, and network
optimizations to increase throughput over 6X versus the current open-source
version (v1.6) on Intel? Xeon? E5 server processors while maintaining latency
Service-Level Agreements (SLAs) of 1ms or less for Round-Trip Time (RTT). Figure
1 illustrates the 6X speedup of the optimized version over the v1.6 memcached
baseline, with the optimized version on 32 logical cores providing 3.15 Million
requests per second (RPS), with a median round-trip time(RTT) less than one
millisecond.

Figure
1 – Best throughput measured using any configuration from the base open-source
code version 1.6.0_beta1 and the optimized version

Memcached operates on simple data types of key-value pairs, similar to NoSQL
databases, but is not persistent like NoSQL. Memcached stores all the key-value
pairs in non-persistent memory, so in the event of failure, all the stored data
is lost. Throughout this paper we use the term cache item to specify the
combination of a key and its associated value. Keys are, by definition, unique
values. In a web-serving architecture, the memcached application sits between
the Front-End Web Servers, or Web Tier, and the Back-End Database as shown in
figure 2.

Figure
2 – memcached server sits between the Front-end Web Tier and Back-end
Database

Memcached’s purpose is to intercept data requests and satisfy them out of its
logical cache (i.e. system memory) when possible, avoiding trips to disk storage
attached to the back-end database. There is also the potential to avoid
compute-intensive tasks by retrieving a pre-computed value from the logical
cache. In both cases, the time spent retrieving or computing data results is
reduced (i.e. transaction latency). A cluster of servers participates in a
memcached logical cache, with each server offering its system memory as a
portion of the complete logical cache [7] [8] [9]. For example, a memcached
cluster of 10 nodes, with each node having a 128 GB memory footprint, provides a
1.28 TB logical cache for servicing data requests. The logical cache is
populated from data in the back-end database, the persistent data store for
web-serving, or from computational servers. Cache items are maintained using
Least Recently Used (LRU) policy as well as Time To Live (TTL). When an item is
evicted its slot is filled by a more recent item.

A web service request can require multiple trips to memcached, the database,
and other services. If the caching strategy is efficient, the number of trips to
memcached will be an order of magnitude greater than the required trips to the
database (i.e. 10:1). Data retrieval from system memory, as provided by
memcached, is an order of magnitude faster than retrieving the same data from
the database (microseconds versus milliseconds) and, in most cases, orders of
magnitude faster than computing the data on-the-fly. Therefore, avoiding
database accesses and computation is necessary to provide acceptable user
response times for web service requests.

1.1 Terminology

Throughout the paper, we will reference several items with abbreviations and
short-hand.

LRU (Least Recently Used) -- This is the eviction scheme
used in memcached to determine which items are removed from the cache to free
space for additional cache items.

Base (Baseline) – This is the unmodified 1.6.beta_0 version
of Memcached, which can be downloaded directly from memcached.org.

Bags – This is short-hand for the modified version of
memcached optimized to increase performance. Bags is the modified LRU cache
update strategy, called a “bag LRU”. This was the first section of code
developed.

RTT (Round Trip Time) – This metric is the elapsed time of
a Memcached request, including the elapsed time from request from client to
server until response back to client.

SLA (Service Level Agreement) – This metric is the maximum
RTT allowed to maintain acceptable web service request response time to the
User. Response time above the SLA will result in an unacceptable response time
for generating the web service request. There can be multiple memcached requests
(to memcached logical cache) per web service request (to Front-end Web Tier).
For the purpose of this paper, the memcached SLA is 1 millisecond response time
to the requesting Front-end Web Tier client.

NIC (Network Interface Card) -- This is the network card
used in the system.

2 Current Memcached Architecture

When memcached was introduced in 2003 [5], there were few X86 processors with
multiple cores, and multi-socket servers were extremely expensive. Intel? Xeon?
dual-core and quad-core processors weren’t released until 2005 and 2009
respectively. Today, dual-socket, 8 and 16-core server systems are the building
block of web service delivery, with roadmaps for processors showing increasing
core counts [10]. These advanced systems provide ample opportunity for workloads
to execute in parallel using multiple threads or processes, yet memcached is
unable to do so using the current data structures and software architecture [1]
[6].

2.1 Data Structures

Memcached has 4 main data structures, including

• A hash table to locate a cache item.


• Least Recently Used (LRU) list to determine cache item eviction order when
  the cache is full.


• A cache item data structure for holding the key, data, flags, and
  pointers.


• A slab allocator, which is the memory manager for cache item data
  structures.

The hash table data structure is an array of buckets. The array size (k) is
always a power of 2 to make finding the correct bucket quicker by taking the
value of 2^k-1 and using it as a hash mask. Executing a bit-wise AND (e.g.
hash_value & hash_mask) quickly determines the bucket that contains the hash
value. The buckets are constructed as single linked-lists of cache items that
are NULL-terminated. Figure 3 illustrates this data structure.

Figure
3 – Data Structure of hash table used to lookup cache items

The LRU, used to determine eviction order, is a double linked-list which
holds all cache items in a single-sized slab (i.e. unit of memory allocation,
described below). The double linked-list pointers are maintained in each cache
item structure, similar to the hash table, and are modified each time the cache
item is manipulated. In the event of an eviction, the tail of this list is
checked for the oldest cache item, which is removed for reuse. Figure 4
Illustrates the LRU data structure.

Figure
4 – Data Structure of the current open-source LRU used to determine cache item
eviction order

The cache item data structure holds the key-value pair data. It is an
intersection of the following reference data:

• pointers used in the hash table for the single linked-list


• pointers used in the LRU for the double linked-list


• reference counter to determine how many threads are currently accessing a
  cache item


• flags for cache item status


• the key


• length of the value in bytes


• value

The slab allocator provides memory management for cache items. Since the
cache items are relatively
small, allocating and freeing of these
small chunks of memory using system calls would be slow and likely cause
thrashing. Instead, memcached uses slabs as the unit of memory allocation. A
slab is a larger piece of memory that can hold many cache items within it. For
example, a slab could be a 1024 byte chunk of memory which will hold 16 cache
items that are 64 bytes or less. The slab allocator uses these larger memory
allocations, and then maintains a list of free cache items in the slabs. Each
time a cache item is requested, the slab allocator checks the size of the value
being stored and returns a cache item from a slab large enough to accommodate.
In some cases this solution wastes space, but the time saved and memory
thrashing avoided offsets this disadvantage.

2.2 Commands

1. GET to retrieve a cache item from the cache


2. STORE to add a cache item to the cache


3. DELETE to remove a cache item from the cache

Additionally, there are 15 other specialty requests including stats,
replacements, arithmetic, flush, and updates. These code paths generally follow
some order of the above primary commands. For instance, a replacement first
DELETEs the cache item and then STOREs a new cache item into the hash table and
LRU. In analysis, we focus effort on GETs, as this operation constitutes the
bulk of the command requests in memcached operation. [11]

For the three primary commands (GET, STORE, and DELETE), the flow of
memcached is as follows:

1. Request arrives at NIC and is processed by libevent


2. A worker thread
   
   • Takes connection and data packet
   
   
   • Determines the command and data


3. A hash value is created from the key data.


4. The cache lock is acquired to begin hash table and LRU
   processing. (Critical Section)
   
   • (STORE only)
     
     • Cache item memory is allocated.
     
     
     • Data is loaded into the cache item data structure (flags, key, and
       value).
   
   
   • Hash table is traversed to GET, STORE, or DELETE the cache item from the
     hash table.
   
   
   • LRU is modified to place move cache item in the front of the LRU (GET,
     STORE) or remove it (DELETE).
   
   
   • Cache item flags are updated


5. Cache lock is released. (End of Critical Section)


6. Response is constructed.


7. (GET only) Cache lock is asserted.
   
   • Cache item reference counter is decremented.
   
   
   • Cache lock is released.


8. Response is transmitted back to requesting client (i.e. Front-end Web
   server).

2.3 Process Flow

Figure 5 documents the process flow of memcached when processing data
requests from the perspective of a memcached server in the cluster. Multiple
servers work in conjunction to act as single larger logical data cache. In
essence, these servers constitute a large, distributed hash table. Reviewing
figure 5 from top to bottom, the clients (these are usually in the form of web
tier servers building a response for a user or computational servers needing
data for calculations) are responsible for submitting operation requests to
memcached. When memcached is initiated, a static number of worker threads are
created to process these client requests, normally aligned with the number of
physical processors in the server system.

Figure
5 – Process flow for cache item operations (STORE/GET/DELETE) in open-source
v1.6

Client requests are distributed to worker threads for processing. For a GET
request, each thread must complete a hash table lookup to identify where the
value data resides associated with the key. Additionally, to indicate that the
key-value is recently accessed and update the eviction order, it is moved to the
front of the Least Recently Used (LRU) list. Unfortunately, both of these
activities require protection (i.e. locks) for these shared data structures.
After the data is located, it is retrieved from system memory by the worker
thread and transmitted back to the requesting client.

Hash table thread safety is provided by a single global cache lock that is
asserted for hash table access. Additionally, the LRU linked-list, maintained to
manage cache evictions, uses the same global cache lock when cache items’ LRU
linked-list is manipulated to ensure thread safety.

Threads are initiated by libevent, which is a portable network protocol
wrapper. Threads wait in libevent for a request, and upon
receiving one, load the data from the packet and decode the command. Assuming a
request for a cache item (GET/STORE/DELETE/ARITHMETIC), the thread computes a
hash value from the key and acquires the global cache lock. The thread will hold
the lock until the command is completed, then release the lock and create a
response. Doing this, memcached is effectively serialized to ensure data
consistency and thread safety.

2.3.1 Hash Table Lookup

After acquiring the lock, the hash table traversal and/or manipulation is
executed. Serialized hash table entry and exit eliminates concern for thrashing
pointers and corrupting the linked-list bucket during a STORE or DELETE. If we
leave the hash table implementation as-is and remove the locks, there is a
scenario where two cache items “close” to each other (i.e. adjacent in the
linked-list) are being removed and, upon completion of the removals, the
linked-list no longer points to cache items in the correct chain. This is
illustrated in Figure 6, with each row representing a time progression.

Figure
6 - Linked-list corruption from two threads removing cache items
concurrently.

In the second row, one thread is removing the yellow cache item (#3) and one
is removing the orange cache item (#4). They both modify the pointer of the
previous cache item. In row 3, they both NULL out the next pointer. Finally, in
the fourth row, the cache items are removed and the remaining hash chain is left
with a gap in the middle of the chain, causing item #5 to be lost from the hash
chain.

The lock is also required for GET requests so they are not traversing a
bucket while the linked-list is changing. Traversing a linked-list during update
could cause the thread to follow a pointer into non-existent memory, causing a
segmentation fault or a miss of cache item that does exist. Refer to Figure 3
for the layout of the hash table data structure.

2.3.2 LRU Manipulation

Upon completion of the hash table operation, the cache item is either added
(STORE) or removed (DELETE) from the double linked-list structure (Figure 4)
used for the LRU eviction scheme. When processing a GET request, the cache item
is removed from its current spot in the LRU and added to the head of the list.
The cache lock is held for all LRU manipulation as two cache items being removed
from the same location, or inserted at the head simultaneously, can cause list
corruption. This is the same corruption described above with the hash table
linked-list, except with an additional pointer to corrupt.

2.3.3 Other Serial operations

The Hash table lookup and LRU Manipulation operations are the primary code
segments protected by the global cache lock, but there are others. Cache
evictions, allocations, and flushes also require holding the global cache lock.
These operations access the either the LRU or hash table, so simultaneous access
by multiple threads can cause corruption. Additionally, cache item flags that
are modified by these operations are protected by the global cache lock to
ensure thread safety. An example of flag use is the ITEM_LINKED flag, which
needs to be cleared when removing a cache item from the hash table. The global
cache lock ensures only one worker thread can manipulate this flag at a
time.

2.3.4 Parallel operations

The bulk of the process flow for memcached operation requires protection by
the global cache lock. Outside of this functionality, there are some operations
that are completed in parallel, including: network transmission (both receive
and transmit), packet decode, key hashing, and building of the data
response.

2.4 Performance Bottleneck

The global cache lock is a performance bottleneck for more than four threads,
as discussed in [1] and [6]. This behavior was also confirmed in testing (Figure
7 and Table 1). Figure 7 illustrates that application throughput degrades for
open-source memcached past 4 cores. The lock is coarse-grained; that is, time
spent in serial execution is substantial, and contention for the lock among
worker threads is high.

To verify this, Intel? Vtune? Amplifier XE was used to produce stack traces
on this code and indicated that a majority of the execution time spent in a lock
(i.e. waiting for the global cache lock). Looking deeper into the stack trace,
we determined the global cache lock was causing contention. To quickly confirm
the problem, we removed the global cache lock (which we knew would be unsafe,
but effective) and got a substantial throughput increase on a read-only
workload.

¹ Vtune is a trademark of Intel Corporation in the U.S. and
other
countries.
² http://www.opencompute.org
³ Other
brands and names are the property of their respective owners.

Abundant research illustrates shared lock contention can cripple a program’s
parallel performance. Reference [12] provides an in-depth analysis of contention
and how it affects a program and degrades performance. We concluded that
avoiding the contention for the shared global cache lock would improve parallel
performance and application scalability.

Table
1 - Captured "perf top" from a server running memcached with 8 threads

Figure
7 - Scaling performance of Base (version 1.6) memcached showing performance peak
around 3-4 cores

3 Memcached Optimizations

This section outlines the optimizations to memcached to improve application
scalability for multi-threaded execution on many-core servers. After examining
the current memcached architecture, we investigated techniques to minimize or
eliminate the global cache lock, the primary performance bottleneck.

Performance goals include:

• Equivalent performance regardless of number of keys (i.e. 1 key versus 1
  million key lookups)


• Updated cache replacement strategy for LRU Eviction should provide
  equivalent (or better) hit rates. The expectation is a 90%+ hit rate for
  cached value lookups in the synthetic workload we are using.


• LRU correctness for cache management; that is, the correct value is
  evicted based on its LRU status.

To measure progress versus these goals, we have baseline measurements using a
synthetic workload to replay a set of transactions and measure the hit rate.

3.1 Data Structures

For our modifications two data structures are modified: the hash table and
the LRU. The following optimizations were implemented:

• Hash table locking mechanism changed to allow for parallel access


• Bag LRU – The data structure is changed to an array of different sized
  LRU’s with Single linked-list bags (i.e. collections) of cache items.

While we are changing the algorithms in the hash table, we make no physical
changes to the hash table data structures. We implement striped locks which are
required to modify the hash table buckets. A striped lock is a fine-grain set of
locks (versus the global hash table lock) implemented on sections of the hash
table. For this implementation, we first create and initialize a set of Z locks,
where Z is a power of 2. Keeping this a power of 2, a bitmask is used to
calculate the lock number quickly. To traverse the linked-list, the lock
guarding that bucket is acquired the performing a bit-wise AND of the value Z-1
with the bucket to-be-locked. This protocol provides a shared lock for every Zth
bucket (Figure 8), reducing contention for hash table locks. In testing, Z=32
was a suitable value for the number of locks.

The LRU is where the bulk of the data structure changes are required. To
build a more parallel LRU to take advantage of the parallel hash table, we
researched ideas for parallel LRU data structures. There are two good ideas for
lock-mitigating LRU algorithms and data structures described in [13] and
[14].

Figure
8 - Addition of the striped locks that guard each bucket

Taking inspiration from both of these, and the concept of a Bag LRU, we
implement single linked-list “bags” of cache items. Each “bag” contains cache
items with similar insertion or last-touched times, and can be inserted
atomically. There is a cache item “clean-up” functionality required with this
model; a cleaner thread exists for this purpose. It evicts expired/invalid cache
items and performs maintenance on the bags. A more detailed explanation of the
new data structure including diagrams and logic is in section 3.5.

3.2 Commands

There are no changes to any of the memcached commands or protocols described
in section 2.2. The difference is solely in the methodology for command
execution.

3.3 Process Flow

Figure 9 relates the updated process flow with modifications. From top to
bottom, the client, libevent, and worker thread functionality remains the same.
The serialized operations for hash table lookup and LRU processing are modified
to operate in a parallel fashion.

DELETE and STORE operations now use a parallel hash table approach with
striped locks (explained earlier) (Figure 8), and GET operations now execute
non-blocking and parallel.

Figure
9 - Process flow for cache item operations (STORE/GET/DELETE) in optimized
version of 1.6.0_beta1

The algorithmic changes, described below, prevent hash chain corruption and
maintain pointers of cache items being removed. In the event a thread is
pre-empted on a removed cache item, it will likely continue in the correct
bucket. There is a slight nuance to the unblocked GET. If a thread is preempted
for a long duration on a cache item, and that cache item is moved, the GET can
return a “false negative” (i.e. memcached will return “no cache item found” when
the cache item is actually in the hash table).

The pros and cons of this were weighed, and, since memcached is a cache for
data that should be persistent elsewhere (e.g. Database Tier); the extra
throughput realized from removing the locks outweighs the possibility of a false
negative. If there is a case where the false negative presents a problem, you
could use the striped locks, which minimally decrease performance. However, the
expectation is, as core counts increase, that striped locks could again become a
point of contention.

LRU processing is now non-blocking and parallel, as described above in
section 3.2, and below in section 3.5. In bags, a single linked-list of cache
items is stored so they can be inserted atomically, removing the need for a lock
while placing a new cache item into a bag. There is still a Compare-and- Swap
(CAS) command required, which could be a point of contention if only STOREs are
performed. However, optimizing for the majority of requests (i.e. GETs), the Bag
LRU is implemented with no locks or CAS required for GETs.

3.4 Parallel Hash Table Lookup

The first performance impediment encountered when executing a memcached
transaction is the serialized hash table lookup. Previous work on increasing
parallel performance of memcached has focused on a partitioned hash table in
which threads access only specific portions (i.e. partitions) of the cache. This
works well for removing lock contention and increasing performance as seen in
[16]. The disadvantage of this approach is that multiple frequently accessed
cache items in a single partition can overwhelm the threads available to service
that partition. Similarly, traffic patterns that favor a partition can be
starved for processing power. Overall performance of a partition is limited to
the threads that can access that cache without causing lock contention.

This implementation optimizes throughput for the majority of traffic, GET
operations. Reference [11] shows that traffic on the majority of Facebook’s
cache servers are GET requests. This balance is as high as 99.8% GETs in some
cases, with only a few caching tiers being write-dominated. This is consistent
with the performance testing done by [15] and [16] as both testing methodologies
only measure throughput for GET requests.

With this in mind, a parallel hash table is designed that doesn’t require any
locks when processing GET requests. Lock contention, no matter how small,
impacts performance (transaction latency, overall throughput under Latency SLA),
as shown in [12]. In the future, as CPUs execute faster and core-counts
increase, lock contention will increase. The new designs accounts for these
trends by completely eliminating lock contention for GETs.

3.4.1 GET Lock Removal

In order to remove locks on GETs, there are two main cases to address:
expansions and hash chain modifications (i.e. insertions and removals).

An expansion is required when the hash table has a much
larger amount of cache items then buckets. In this case, adding more cache items
will increase the length of the hash chain, increasing the time required to find
a cache item and RTT. Instead of suffering this time penalty, a new hash table
with twice the amount of buckets is allocated, as follows:

• all cache items are taken from the old table


• the key is hashed /li>


• the cache item is inserted into the new table/li>

Assoc_get logic:

1. mask hash to get the bucket

2. If (!expanding)

3. Check hash chain bucket – return cache item if found

4. If (cache item not found)

5. If(expanding && expansion bucket is greater than the
bucket)

6. Find bucket in new larger table

7. Wait for old bucket to be empty

8. Check new hash bucket – return cache item if found

9. else if expanding

10. Ensure bucket is in old table

11. Check hash chain bucket – return cache item if found

12. If (cache item not found)

13. If(expansion bucket is greater than the bucket

14. Find bucket in new larger table

15. Wait for old bucket to be empty

16. Check new hash bucket – return cache item if found

To accomplish cache item retrieval, a check for an expansion is required. If
there is no expansion in progress, the GET request executes normally and returns
the cache item, if found. However, there is the possibility the expansion
started after the initial check and moved the cache item. In this case, the
process waits for the old bucket (the bucket where the item would be located in
the previous hash table) to be empty and then checks the new hash table and
bucket for the cache item.

If the hash table is currently being expanded, there is a check to see if the
old bucket has been moved already. If the old bucket has not been moved, it is
checked for the cache item. Like before, if the cache item isn’t found, a check
is required to see if a “bucket move” occurred on the old bucket. If so, the new
hash table will then be checked for the item.

The second case to address, hash chain modifications, is handled in the STORE
and DELETE code. As long as pointers are changed in correct order, GETs will
have no problem with hash chain traversal.

3.4.2 STORE/DELETE Lock Removal

The STORE and DELETE commands are very similar to the current open-source
implementation. To ensure safety in hash chains and multiple inserts/deletes on
the same chain or cache item, STOREs and DELETEs use the striped locks that we
described earlier (Figure 8).

Assoc_delete logic:

1. Determine bucket of cache item

2. Get the bucket lock

3. If(expanding)

4. If(expansion bucket is greater than the bucket){

5. Release the bucket lock

6. Determine new bucket

7. Get bucket lock

8. Delete cache item from bucket if found

9. Release bucket lock

10. Else if not expanding

11. Delete cache item from bucket if found

12. Release bucket lock

The striped lock is required for all hash chain manipulation, including
moving buckets during an expansion. This makes STOREs and DELETEs simpler since
they get the bucket lock and then check for an expansion. If the expansion is
moving the bucket the STORE or DELETE is going to manipulate, they are locked
out until the old bucket is empty and know to check the new one. If the move
started after the lock was acquired, then the expansion waits until the lock is
released and can be acquired by the expansion to move the bucket. The logic is
identical for contending STOREs, DELETEs, and MOVEs as they are all locked out
of the buckets until the manipulations are complete.

Assoc_insert logic:

3.5 Parallel LRU Eviction

A parallel replacement for the double linked-list LRU implementation is
required to determine cache item eviction order. The concept behind the “bag”
implementation is to group cache items into “bags” based on their timestamp. In
these bags, a single linked-list of cache items is stored so they can be
inserted atomically to the end of the list using a CAS. This removes the need
for a lock while placing a new cache item into a bag although the CAS could be a
point of contention if only STOREs are performed. Again, optimizing for the
majority (GET requests), the Bag LRU is implemented with no locks or CAS for
GETs.

The new design also needs a way to evict old cache items and maintain new bag
LRU structures. To accomplish this, we implement a cleaner thread to run in the
background. This thread removes expired cache items, invalid cache items, and
performs maintenance on the LRU. Work that GET requests were doing in the old
version (i.e. Removal on expirations) is moved to this background thread,
increasing the throughput of GET requests and decreasing overhead per GET
request transaction. The cleaner thread also provides a benefit of increasing
hit rates of the cache. Cache items inserted with quick expirations are now
cleaned up within a few minutes of expiring, instead of taking space until aging
out of the cache.

3.5.1 Linked-list structure

The linked-list structure reuses
the previous and next pointers from the
current LRU, so no extra memory allocation is required. The next pointer still
points to the next cache item in the new single linked-list and the previous
pointer is used to mark the cache item’s bag membership which is the newest bag
at the time the cache item was last accessed. The cleaner thread uses the
residence pointer while cleaning the bags.

3.5.2 Additional Data structures

Figure
10 - Bag data structure

Figure
11 – Bag List data structure

There are two data structures required for the bag LRU, an array of LRUs for
bag identification, and the actual bag data structure.

The first data structure is the bag LRU list, an array of bag heads, with one
per slab id (Figure 10). These structures are referenced in a global variable
and initialized when the LRU is initialized. Each bag head in this array
maintains:

• A pointer to the newest bag


• An alternate to the newest bag


• The oldest bag in the bag list


• A count of the number of bags for statistics

The newest bag is the bag that is currently filled with newly-allocated cache
items. The oldest bag is where the cleaner thread starts cleaning bags and where
eviction threads evict cache items from. The newest alternate bag is used by the
cleaner thread to avoid locks and contention with inserts into the newest bag.
Any cache item that already exists in a bag, but has been accessed recently and
needs to be moved to the newest bag by the cleaner thread to maintain eviction
order (the one currently being filled), will be placed into the newest alternate
bag.

The second data structure created is a bag, which maintains a single
linked-list of cache items (Figure 11). Each bag has:

• A pointer to the newest and oldest cache items in the bag


• A counter for the number of cache items in the bag


• A lock


• Bag open and close timestamps that are currently set, but not
  used.

When cleaning a bag, the cleaner thread begins its operation from the oldest
cache item in the bag. The evictions done by worker threads begin in the same
location. The newest cache item pointer is to allow for cache items to be added
to the end of the bag quickly when being inserted or as bags are merged. The
count is for determining when to open a new bag and when to merge an old bag
into a newer one. This count can also be used for statistics and debugging as
needed. The reason for a lock on each bag is to ensure thread safety between the
cleaner thread and evictions done by worker threads. If an eviction thread and a
cleaner thread are moving cache items in the same bag, it is possible for one of
the threads to corrupt a bag (we have shown linked list corruption in Figure 6).
Since the cleaner thread spends very little time in each bag, it is unlikely
that there is contention on this lock between cleaner and worker threads. Also,
since worker threads are serialized higher up in the call stack, so there is no
contention on this lock between these threads.

3.5.3 Operations on the Data Structures

This section reviews operations on the new data structures and algorithms for
initialization, inserting cache items, and cleaning the LRU List.

First, all added locks need to be initialized and an initial bag is created
for each slab size to be used. After all the structures are initialized, the
cleaner thread is created and launched in a cleaning loop.

Bag LRU initialization

1. Initialize the global eviction lock

2. Initialize the cleaner locks – (currently only one lock could be one
cleaner thread per slab id)

3. Create the array of bag head structures

4. For each slab id

5. Initialize the first bag

6. Initialize the bag lock

7. Point the bag array head to this as the newest bag

8. Increment the bag counter

9. Create and launch the cleaner thread

In the cleaning loop, the thread continually checks to see if bags are full
and adds new bags to those slabs as needed. Periodically, the cleaner thread
traverses every cache item in the bags, removing invalid or expired cache items,
and merging bags below a minimum cache item count threshold.

Cleaner thread loop

Cleaner thread loop

1. Loop forever

2. Acquire cleaner lock

3. For each slab id

4. if the newest bag is full

5. create and initialize a new bag

6. point the newest bags next bag pointer to the newly initialized
bag

7. set a bag holder = newest bag

8. Newest bag = newly initialized bag

9. Newest alternate = bag holder

10. atomic increment of bag count

11. Every Nth iteration, clean the bags

12. Release cleaner lock

13. Sleep for a given interval

When a cache item is first inserted into a LRU, it goes into the newest bag
available for that cache item’s slab size. Once the bag is located, the cache
item is placed at the end of the newest cache item in the newest bag using a
CAS. If this fails, the thread traverses the chain for that bag until it finds a
NULL pointer (indicating the end of the bag), and tries the CAS again. This
continues until the CAS passes indicating a successful insertion.

Inserting New Cache item not already in LRU

1. New cache item next pointer = NULL

2. If(current open bag is empty)

3. If( CAS (current bags newest cache item, NULL, new cache item)){

4. New cache item previous pointer = current bag

5. Current bag newest cache item = new cache item

6. Atomic increment bag cache item counter

7. Return

8. Swap cache item = current bags newest cache item

9. While( !CAS(swap cache item next pointer, NULL, new cache item)){

10. While(swap cache item next pointer != NULL)

11. Follow the swap cache item next pointer until you find a null and
try the swap again

12. New cache item previous pointer = current bag

13. Current bag newest cache item = new cache item

14. Atomic increment bag cache item counter

For a GET, since cache item already exists in the LRU, we are simply touching
the cache item to update its timestamp and location in the LRU. The touch copies
the new timestamp into the cache item last accessed field and points the cache
item’s previous pointer (the bag the cache item should now reside in) to the
newest bag in that slab size.

Updating Cache item already in the LRU

1. Cache item previous pointer = current bag

2. Update cache item timestamp

3.5.3.1 Cleaning the LRU list

Within the context of cleaning the LRU list, there are several operations to
support; including cache item reordering, merging buckets, cache item eviction,
and dumping the cache.

As the cleaner thread traverses the bag lists, it checks the cache item
previous pointer. If the pointer points to the current bag it is cleaning, it
leaves the item in its current location. If the item points to different bag,
that means that the item has been touched since it was first inserted. In this
case, the cleaner thread will remove the item from its current bag, and insert
it into the bag it should be residing in. As described above, if this is the
newest bag in the LRU size, it will be placed into the newest alternate to avoid
contention.

Merging two buckets

1. Older buckets newest cache item next pointer = newer buckets oldest cache
item

2. Newest buckets oldest cache item = older buckets oldest cache item

3. For each cache item we added

4. If the cache item was pointing to the old bag

5. Point it to the new bag

6. Increment the newer bags counter by the amount of cache items in the
older bag

The logic to merge two buckets is to concatenate the two linked-lists
together, put the combined linked-list into the newer of the two bags. Then it
must ensure every cache item that had its previous pointer pointing to the old
bag now points to the newer bag and reclaim the older bag.

Evicting a cache item

1. Get the eviction lock for the slab id

2. Get the oldest bag from the slab id

3. Find the first bag in the list that is before the newest alternate and has
cache items in it

4. if we can’t find the previous bag, add two new bags to this slab id, then
check those two bags

5. if we still can’t find a bag

6. stop trying to evict cache items just return a failure

7. release eviction lock

8. lock the bag we found

9. starting at the oldest cache item in the bag

10. try to find an cache item with a refcount of 1 or 0 or a delete locked
cache item in the first N cache items of the bag

11. if cache item is found evict it and free it

12. try and get the newly free cache item from the slab allocator

13. if we evicted an cache item but didn’t get it from the allocator –

14. evict another cache item and try again

15. else

16. stop trying to evict cache items just return a failure

17. unlock the bag

18. release eviction lock

To evict a cache item from the cache, the worker threads are serialized with
a global eviction lock to ensure correctness. The bag is then locked to avoid
conflicts with the cleaner thread. After this is done, the thread is the only
thread touching the bag and can remove any cache item that meets the eviction
criteria (which is the same as the open-source version).

Dumping the cache

1. get cleaner lock

2. get eviction lock

3. add new bags so we can evict without effecting other threads

4. for each slab id

5. for each cache item

6. If (cache item older than the cache dump time)

7. Free cache Item

8. release eviction lock

9. release cleaner lock

Dumping the cache is the same as the eviction of a cache item as before, but
this time all cache items older then the time specified in the command are
evicted.

3.6 Removal of global cache locks

With functionality in place to manage parallel hash lookups, LRU ordering,
and parallel cache evictions, the global cache locks are nearly ready to be
removed from the GET/STORE/DELETE code paths. Management of reference counts
must by changed; instead of a lock to change the reference count, atomic
operations are used to increment and decrement the count. Upon completion, the
global cache lock/unlock pairs are removed from the operations, as the global
lock is no longer necessary.

1. Lock (global_cache_lock) <- Remove these locks

2. . . . . . (GET/STORE/DELETE)

1. 3. Unlock (global_cache_lock) <- Remove these unlocks

4 Test Environment Configuration and Metrics

This section reviews the hardware configuration of all the systems being
used.

4.1.1 System under Test (i.e. SUT)

Based on early evaluation, a 1 GbE (Gigabit Ethernet) NIC is easily
saturated, so a 10GbE NIC is required to remove the network as a performance
bottleneck.

Memcached is primarily run in large server farms where Total Cost of
Ownership (TCO) is important. The objective is to maximize performance/watt
results to decrease the TCO. The test system is a power-optimized 2-Socket Open
Compute 2.0 System Board manufactured by Quanta Computer.

System Hardware:

• Dual-Socket Open Compute 2.0 System board (half width) in 1.5U
  chassis²


• Quantity 2 -- Intel? Xeon? E5-2660 2.2Ghz Processors (95W TDP) [10]


• 128 GB DRAM – Quantity 16, 8 GB DDR3 PC3 12800 ECC


• Intel? “Niantic” 82599 10Gb Direct Attach SFP+ Ethernet Controller
  [10]


• Arista 7124 Switch – 24 Port SFP+⁴

4.1.2 Load generators

With the open-source version of memcached on the previous generations of
servers, a single load generator is sufficient. However, with the modifications,
an increase in the number of clients is required to stress a single SUT. Using
one client, it is possible to get approximately 800K requests per second (RPS);
therefore, we configure four clients for driving the load. All load generators
are configured identically and confirmed individually to be capable of
generating 800K+ RPS with the following Hardware configuration:

• Dual-Socket Open Compute 2.0 System board (half width) in 1.5U
  chassis⁵


• Quantity 2 -- Intel? Xeon? E5-2660 2.2Ghz Processors (95W TDP) [10]


• 16 GB DRAM – Quantity 2, 8 GB DDR3 PC3 12800 ECC


• Intel? “Niantic” 82599 10Gb Direct Attach SFP+ Ethernet Controller
  [10]

4.2 Software (SW) Configuration

This section reviews the SW configurations for the SUT and client systems,
network optimizations, and power monitoring configuration.

4.2.1 Test System Configuration

The Following software is installed to run the Memcached tests:

²http://www.opencompute.org 
³Other brands
and names are the property of their respective owners.
⁴http://www.aristanetworks.com/en/products/7100sbr
/> ⁵http://www.opencompute.org

• CentOS (Community ENTerprise Operating System), version 6.2 – Linux Kernel
  2.6.32⁶


• Libevent 1.4.14b-stable


• Intel? ixgbe 3.8.14 NIC driver


• Memcached-1.6.0_beta1 and Memcached-1.6.0_beta1-bags

• Memcached –p 11211 -u nobody –t -m 128000 –d

With this command, thread is the number of worker threads to run in
Memcached. For all tests, the only variable modified for testing memcached is
the number of threads; this workload driver and input parameters are utilized to
determine peak throughput and scalability.

4.2.2 Client System Configuration

The client systems are configured with the following software:

• CentOS (Community ENTerprise Operating System), version 6.2 – Linux Kernel
  2.6.32 ⁷


• Intel? ixgbe (10 GbE) version 3.8.14 NIC driver⁸


• Mcblaster⁹

With the operating system and network driver software installed, mcblaster is
utilized to drive a synthetic cache lookup workload and measure performance
(i.e. transaction throughput, latency/response time) of the memcached instance.
There are two steps involved in this testing. First, the keys and values must be
loaded into the memcached instance, using:

• ./mcblaster –p 11211 -t 16 -z 64 –k 1000000 –r 10 –W 75000 –d 20

This command loads the cache with 1 million keys, all with 64 Byte values. In
the next phase of testing, all clients send requests for those keys, in random
order, simultaneously at a given rate to measure throughput and latency at that
rate.

• ./mcblaster –p 11211 -t 16 -z 64 –k 1000000 –r –W 0 –d 120

With these tests, the rate is increased as memcached’s average RTT is
monitored. Maximum throughput is the highest rate achieved without exceeding the
1ms average RTT SLA.

Using previous work on measuring performance of memcached [15] and [16] as a
guide, all measurements are gathered using GET requests. Many key-value caching
servers are GET-dominated [11] and the number of STOREs in the workload mix has
little impact on GET performance [16].

⁶http://www.centos.org/
⁷http://www.centos.org/
⁸http://sourceforge.net/projects/e1000/files/ixgbe%20stable/ -
3.8.14 was replaced by 3.8.21 (2-24-2012)
⁹https://github.com/fbmarc/facebook-memcached-old/tree/master/test/mcblaster

4.2.3 Network Setup

For maximum throughput, there are two different networking configurations
depending on the Intel? Hyper-Threading (HT) Technology setting
(enable/disable).

The following setup is used when Intel? HT Technology was disabled:

• Reduce the Ring buffer size to 96 for both rx (receive) and tx(transact,
  send)
  
  • Goal: Reduce latency ( RTT)
  
  
  • Command
    
    • ethtool –G ethX rx 96
    
    
    • ethtool –G ethX tx 96


• Disable generic receive offload
  
  • Goal: Reduce latency ( RTT)
  
  
  • Command: ethtool –K ethX gro off


• Affinitize NIC interrupts to CPUs by executing the
  set_irq_affinity.sh¹⁰shell script
  
  • Goal: Reduce network interrupt contention
  
  
  • Command: ./set_irq_affinity.sh ethX
  
  
  • Note: Ensure that the irqbalance service is disabled in the kernel or it
    will overwrite these changes. (killall irqbalance)

These settings provide best throughput when combined with the latest ixgbe
driver (3.8.14 when the test was run). Flow director (included with the ixgbe
driver) is used to steer incoming packets evenly across the physical CPU cores
distributing the load of network interrupts. This allows CPUs to maintain
balanced utilization across cores.

With Intel? HT Technology enabled, the optimal network configuration uses 1
memcached thread per physical CPU and 2 NIC queues which are now affinitized to
the additional 16 logical cores (Table 2). Using process affinity, this
configuration allows a majority of the NIC handling to be off-loaded to the
logical cores adjacent to the Memcached threads, providing a boost in
performance vs. Intel? HT Technology disabled.

Table
2 – Logical Cores share Memcached and Network traffic in an Intel? HT Technology
memcached setup

¹⁰This script is provided in the ixgbe driver download.

4.2.4 Power measurements

For testing, power is monitored with a Yokogawa WT210 digital power
meter¹¹ . The wattage is collected every 5 seconds while the
system is under load. The power is then averaged to give the power readings
listed in the charts in the next section.

As a note, all of these servers were run with a video card and a
less-than-optimal cooling solution. We expect less power consumption per server
in a data center environment versus the numbers reported in this paper.

5 Findings

To summarize the performance gains, Figure 12 provides the maximum server
capacity (i.e. RPS) for the SUT while maintaining the Latency SLA of <1ms RTT
(Round-trip time). Consistent with our research, these measurements are taken
with GET requests.

Testing confirmed that the optimized memcached performs best with one thread
per physical processor, so optimized test results use 16 threads to align with
16 physical cores in the SUT. Intel? HT Technology and Turbo mode both provide
performance upside when enabled.

The two open-source baseline results illustrate no performance gain from
increasing 8 to 16 threads (i.e. lack of scalability due to lock contention).
The optimized “Bags” results show increasing performance gains as we add Intel?
HT Technology (+8%) and Turbo (+23%). The performance improvement with Intel? HT
Technology is attributed to additional logical processors handling the network
interrupts concurrently with worker threads on the same physical core, removing
the need for context swaps. Turbo mode provides an increase in processor
frequency, contributing a significant performance improvement as well. Our best
throughput result, at 3.15M RPS, is achieved with Intel? HT Technology and Turbo
enabled (+31%), with each feature providing an additive increase in performance.
When we compare the best optimized result (3.15M) with an identically-configured
baseline (.525M), we realize a 6X performance gain.

Figure 13 relates the average round-trip time (RTT) for the four optimized
memcached cases from Figure 12. It is interesting to note that there is a point
(260 us, 285 us, 215 us, and 205 us, respectively) where RTT degrades
immediately past the 1 ms SLA.

The reality is that, regardless of RPS load, the response time SLA must be
managed to less than 300 us to avoid the point on the curve where RTT degrades;
however, our “capacity” or RPS per server provides consistent RTT under the
300us threshold, allowing each server to be fed millions of RPS to maximize
per-server throughput.

Figure 14 shows the maximum RPS throughput with a median RTT < 1ms SLA as
core counts increase. Turbo was enabled and Intel? HT Technology disabled for
these tests. As core counts are increased, memcached scales linearly due to
parallel data structures and algorithms operating on many-core processors. We
expect this linear scaling to continue on next-generation many-core processors,
as there is no blocking required between threads for execution of the GET
requests.

¹¹http://tmi.yokogawa.com/products/digital-power-analyzers/digital-power-analyzers/wt210wt230-digital-power-meters/#tm-wt210_01.htm

Figure
12 - Maximum GET requests while maintaining <1ms RTT SLA

Figure
13 - Median RTT as GET RPS rate is increased

Figure
14 - Maximum throughput with a median RTT < 1ms SLA as core counts
increase

Table 3 adds power metrics to the best baseline and optimized memcached
results at maximum load under the SLA. In addition to the 6X performance delta,
performance per Watt improves 3.4X over the baseline. In the data center, where
power and cooling is a large portion of the total cost of ownership, maximizing
the work completed per watt invested is critical. These per-server power numbers
would be lower in a data center environment with no graphics card and a better
cooling solution than our SUT.

Table
3 - Power measurements at the wall while running at max throughput < 1ms
SLA

6 Conclusions and Future Work

Optimized memcached increases throughput by 6X increase and performance per
watt by 3.4X over the baseline. Extrapolating these numbers from the half-width
server board to a full chassis (i.e. two boards), the SUT configuration supplies
6.3 Million RPS per 1.5U server with 256GB of memory, with capability to scale
to 1 TB (32 GB * 16 DIMM slots) of memory per 1.5U chassis. The speedup is
attributed to the redesign of the hash table and LRU to utilize parallel data
structures. These data structures allow for the removal of all locks for GET
requests, and mitigate the contention of locks for SET and DELETE requests,
enabling linear speedup when measured from 1 to 16 cores on a 2-socket Intel?
system.

Future research should examine memcached on 4-socket servers and micro-server
platforms to determine if the bag version will continue to scale with additional
cores, and if an increase in performance per watt occurs with the amortization
of the power drawn from shared components.

A second area for research is the cache replacement policy. With updated
cache item ordering and eviction, as well as the addition of a cleaner thread,
new replacement policies could be assessed (e.g. LFU, Adaptive Replacement
Cache, and MQ). A key feature improvement of the cleaner thread is its
autonomous nature, allowing cache management without worker threads’
involvement. Assessing new replacement policies can now be completed without
modifying the flow of the program, potentially increasing the cache hit
rate.

7 Acknowledgment

The authors recognize the contributions of project team members; including
Rajiv Kapoor (Intel Corporation), George Chaltas (Intel Corporation), Alex Duyck
(Intel Corporation), Dean Chandler (Intel Corporation), and Mike Bailey (Oregon
State University)

8 References

[1] P. Saab, "Scaling memcached at Facebook," 12 December 2008. [Online].
Available: https://www.facebook.com/note.php?note_id=39391378919&ref=mf.
[Accessed 1 April 2012].

[2] Twitter Engineering, "Memcached SPOF Mystery," Twitter Engineering, 20
April 2010. [Online]. Available:
http://engineering.twitter.com/2010/04/memcached-spof-mystery.html. [Accessed 1
April 2012].

[3] Reddit Admins, "reddit‘s May 2010 "State of the Servers" report,"
Reddit.com, 11 May 2011. [Online]. Available:
http://blog.reddit.com/2010/05/reddits-may-2010-state-of-servers.html. [Accessed
1 April 2012].

[4] C. Do, "YouTube Scalability," Youtube / Google, Inc., 23 June 27.
[Online]. Available:
http://video.google.com/videoplay?docid=-6304964351441328559. [Accessed 1 April
2012].

[5] memcached.org, "memcached - a distributed memory object caching system,"
Memcached.org, 2009. [Online]. Available: http://memcached.org/about. [Accessed
27 February 2012].

[6] N. Gunther, S. Subramanyam and S. Parvu, "Hidden Scalability Gotchas in
Memcached and Friends," in Velocity 2010 Web Performance and Operation
Conference, Santa Clara, 2010.

[7] B. Fitzpatrick, "Distributed caching with memcached," Linux J., vol.
2004, no. 1075-3583, p. 124, 2004.

[8] B. Fitzpatrick, "LiveJournal‘s Backend A history of scaling," August
2005. [Online]. Available:
http://www.slideshare.net/vishnu/livejournals-backend-a-history-of-scaling.
[Accessed 1 April 2012].

[9] N. Shalom, "Marrying memcached and NoSQL," 24 October 2010. [Online].
Available:
http://natishalom.typepad.com/nati_shaloms_blog/2010/10/marrying-memcache-and-nosql.html.
[Accessed 1 April 2012].

[10] Intel Corporation, "ARK - Your Source for Intel Product Information,"
Intel.com, 2012. [Online]. Available: http://ark.intel.com/. [Accessed 2012 1
April].

[11] B. Atikoglu, Y. Xu, E. Frachtenberg, S. Jiang and M. Paleczny, "Workload
Analysis of a Large-Scale Key-Value Store," in ACM SIGMETRICS/Performance 2012
Conference, London, 2012.

[12] J. Preshing, "Locks Aren‘t Slow; Lock Contention Is," Preshing on
Programming, 18 November 2011. [Online]. Available:
http://preshing.com/20111118/locks-arent-slow-lock-contention-is. [Accessed 1
April 2012].

[13] B. Agnes, "A High Performance Multi-Threaded LRU Cache,"
codeproject.com, 3 February 2008. [Online]. Available:
http://www.codeproject.com/Articles/23396/A-High-Performance-Multi-Threaded-LRU-Cache.
[Accessed 1 September 2011].

[14] M. Spycher, "High-Throughput, Thread-Safe, LRU Caching," Ebay Tech Blog,
30 August 2011. [Online]. Available:
http://www.ebaytechblog.com/2011/08/30/high-throughput-thread-safe-lru-caching/.
[Accessed 1 September 2011].

[15] D. R. Hariharan, "Scaling Memcached – Harnessing the Power of
Virtualization," in VMWorld 2011, Las Vegas, 2011.

[16] M. Berezecki, E. Frachtenberg, M. Paleczny and K. Steele, "Many-Core
Key-Value Store," in Green Computing Conference and Workshops (IGCC), 2011
International, Orlando, 2011.

[17] S. Hart, E. Frachtenberg and M. Berezecki, "Predicting Memcached
Throughput using Simulation and Modeling," Orlando, 2012.