Nessie: A decoupled, client-driven, key-value store using RDMA

Abstract

The increasing use of key-value storage systems in performance-critical data centre applications has motivated new storage system designs that use Remote Direct Memory Access (RDMA) to reduce communication overhead. However, existing approaches that achieve low latency and high throughput do so by dedicating entire cores to PDMA message handling, including polling local memory for incoming RDMA messages. In this paper we describe and demonstrate why polling-based RDMA is not suitable for many data centre applications with significant data processing and data storage requirements. We then propose, design, implement and evaluate an alternative communication approach that strictly uses one-sided RDMA operations, which eliminates polling on the server-side and is therefore a serverless (client-driven) design. This approach is used to build Nessie, a distributed client-driven RDMA-enabled key-value store that uses native RDMA compare-and-swap operations to co-ordinate conflicting operations between clients. Nessie further reduces communication requirements by decoupling the key-value location from the key lookup mechanism, which enables local write operations, thus avoiding comparatively expensive RDMA operations. Nessie's one-sided design ensures that only the clients in the system consume any CPU, which in turn allows it to maintain high performance despite the need for computation by the application utilizing the key-value store or contention from other applications.