OpenStack and NVMe-over-Fabrics – Getting High Performance for Network Connected SSDs

What is NVMe over Fabrics?

Evolvement of NVMe interface protocol is the boon to SSD based storage arrays. It further powered SSDs to obtain high performance and reduced latency for accessing data. Benefits further extended by NVMe over Fabrics network protocol which brings NVMe feature retained over network fabric while accessing the storage array remotely. Let’s understand how.

While leveraging NVMe protocol with storage arrays consists of high-speed NAND and SSDs, a latency was experienced when NVMe based storage arrays access through shared storage or storage area networks (SAN). In SAN, data should be transferred between host (initiator) and the NVMe-enabled storage array (target) over Ethernet, RDMA technologies (iWARP/RoCE), or Fibre Channel. Latency caused due to a translation of SCSI commands into NVMe commands in process of transportation of data. To address this bottleneck, NVM express introduced NVMe over Fabrics protocol to get replaced with iSCSI as storage networking protocol. With this, NVMe benefits taken onto network fabrics in SAN kind of architecture to have a complete end to end NVMe based storage model which is highly efficient for new age workloads. NVMe-oF supports all available network fabrics technologies like RDMA (RoCE, iWARP), Fibre Channel (FC-NVMe), Infiniband, Future Fabrics, and Intel Omni-Path architecture.

Click to Download Calsoft’s NVMe ebook

NVMe over Fabrics and OpenStack

As we know that OpenStack consists of a library of open source projects for the centralized management of data center operations. OpenStack provides an ideal environment to implement efficient NVMe based storage model for high throughput. OpenStack Nova and Cinder are components used in proposed NVMe-oF with OpenStack solution. This consists of creation and integration of Cinder NVME-oF target driver along with OpenStack Nova.

OpenStack Cinder is a block storage service project for OpenStack deployments mainly used to create services which provide persistent storage to cloud-based applications. It provides APIs to users to access storage resources without disclosing storage location information.

OpenStack Nova is component within OpenStack which helps is providing on-demand access to compute resources like virtual machines, containers, and bare metal servers. In NVMe-oF with OpenStack solutions Nova in attaching NVMe volumes to VMs.

A support of NVMe-oF in OpenStack is available from “Rocky” release. A proposed solution requires RDMA NICs and supports kernel initiator and kernel target.

NVMe-oF Targets Supported

Based on the proposed solution above we get 2 choices to implement NVMe-oF with OpenStack. First with a Kernel NVMe-oF target driver which is supported as of OpenStack “R” release. The second implementation is Intel’s SPDK based NVMe-oF implementation containing SPDK NVMe-oF target driver and SPDK LOLVOL (Logical Volume Manager) backend which is anticipated in OpenStack S release.

Kernel NVMe-oF Target (Supported as of OpenStack “R” release)

Here is the implementation consist of support for kernel target and kernel initiator. But kernel based NVMe-oF target implementation has limitations in terms of number of IOPs per CPU core. Also, kernel-based NVMe-oF suffers a latency issue due to CPU interrupts, many systems calls to read data, and time take to transfer data between threads.

Kernel Based NVMe-oF + OpenStack Implementation
Fig – Kernel Based NVMe-oF + OpenStack Implementation

SPDK NVMe-oF Target (Anticipated in Openstack “S” release)


SPDK architecture achieved high performance for NVMe-oF with OpenStack by moving all necessary application drivers to userspaces (apart from the kernel) and enables operation in polled mode rather interrupt mode and lockless (avoiding the use of CPU cycles synchronizing data between threads) processing.

Let’s understand what it means.

In SPDK implementation, storage drivers which are utilized for storage operations like storing, updating, deleting data are isolated from kernel space where general purpose computing processes run. This isolation of storage drivers from kernel saves an amount of time required for processing in the kernel and enables CPU cycles to spend more time for execution of storage drivers at user space. This avoids interruption and locking of storage driver with other general purpose computing drivers in kernel space.

In typical I/O model, application request read/write data access and waits till I/O cycle to complete. In polled mode, once application places a request for data access it goes at other execution and comes back after a defined interval to check completion of an earlier request. This reduces latency and process overheads and further improves the efficiency of I/O operations.

By summarizing,

SPDK specially designed to extract performance from non-volatile media, containing tools and libraries for scalable and efficient storage applications utilized userspace, and polled mode components to enable millions IO/s per core. SPDK architecture is open source BSD licensed blocks optimized for bringing out high throughput from the latest generation of CPUs and SSDs.

SPDK Architecture
Fig – SPDK Architecture

Why SPDK NVMe-oF Target?

As per performance benchmarking report of NVMe-oF using SPDK, it has been seen that

    • Throughput scales up and latency decreases almost linearly with the scaling of SPDK NVMe-oF target and initiator I/O cores.
    • SPDK NVMe-oF target performed up to 7.3x better w.r.t IOPS/core than Linux Kernel NVMe-oF target while running 4K 100% random write workload with increasing number of connections (16) per NVMe-oF subsystem.
    • SPDK NVMe-oF initiator is 3x 50GbE faster than Kernel NVMe-oF initiator with null bdev based backend.
    • SPDK reduces NVMe-oF software overhead up to 10x.
      SPDK vs. Kernel NVMe-oF I/O Efficiency
      Fig – SPDK vs. Kernel NVMe-oF I/O Efficiency
      NVMe over Fabrics Target Features Realized Benefit
      Utilizes NVM Express* (NVMe) Polled Mode Driver Reduced overhead per NVMe I/O
      RDMA Queue Pair Polling No interrupt overhead
      Connections pinned to CPU cores No synchronization overhead
  • SPDK saturates 8 NVMe SSDs with a single CPU core

SPDK NVMe-oF implementation

This is the first implementation of NVMe-oF integrating with OpenStack (Cinder and Nova) which leverages NVMe-oF target driver and SPDK LVOL (Logical Volume Manager) based SDS storage backend. This provides a high-performance alternative to kernel LVM and kernel NVMe-oF target.

SPDK Based NVMe-oF + OpenStack Implementation
Fig – SPDK Based NVMe-oF + OpenStack Implementation

The implementation is demonstrated at OpenStack Summit 2018 Vancouver. You can watch the demonstration video here.

If compared with Kernel-based implementation, SPDK reduces NVMe-oF software overheads and yields high throughput and performance. Let’s see how this will be added to upcoming OpenStack “S” release.

This article is based on a session at OpenStack Summit 2018 Vancouver – OpenStack and NVMe-over-Fabrics – Network connected SSDs with local performance. The session was presented by Tushar Gohad (Intel), Moshe Levi (Mellanox) and Ivan Kolodyazhny (Mirantis).


[Tweet “OpenStack and NVMe-over-Fabrics – Getting High Performance for Network Connected SSDs ~ via @CalsoftInc”]


Related Posts

Generative AI and the changing face of Software Development Lifecycle

Generative AI and the changing face of Software Development Lifecycle

Generative AI is revolutionizing many IT segments, and one of such segments is software product development lifecycle (SDLC). This blog summarizes how Generative AI is transforming SDLC with its applications, benefits, and examples.

Generative AI: Transforming Industries for Success

Generative AI : Transforming Industries for Success

Generative AI is the hot topic of discussion everywhere and is being embraced by everyone. Read this blog to explore how different sectors are leveraging Generative AI to drive innovation, enhance efficiency, and deliver superior experiences.

Private 5G Promising Industry 4.0 Transformation blog

Private 5G: Promising Industry 4.0 Transformation

The potential of Private 5G in ensuring super connectivity and higher data rates in Industry 4.0 is achieving traction worldwide. Private 5G together with other key emerging technologies such as Artificial Intelligence (AI), automation, and Internet of Things (IoT) support operators to generate innovative revenue streams. These advancements make Private 5G an apt choice for all types of enterprise ecosystem (big/small/mid-sized) to realize digital transformation. Read the latest blog to know what, why, and how Private 5G is fueling Industry 4.0.

Potential of 5G in Manufacturing and Industrial Automation

Potential of 5G in Manufacturing and Industrial Automation

Manufacturing industries are probing for novelties and modernization to gain better profitability and productivity. 5G technology with its key capabilities such as higher speed, greater availability, support for ultra-reliable and low-latency communication has potential to revolutionize the manufacturing industry. The technology promises to facilitate digital infrastructure to realize automated and advanced operations which will lead to an enhancement in business output. Read the latest blog to learn how 5G can impact and benefit manufacturing and industrial automation.

How to Choose the Right Partner for Outsourcing Product Development Services

How to Choose the Right Partner for Outsourcing Product Development Services

Outsourcing can be a great way for technology companies to access specialized expertise, accelerate their development process, and ultimately bring innovative products to market faster. Read this blog to know how to choose the right partner for outsourcing your product development.


Edge-to-Cloud in Manufacturing

In the manufacturing industry, an Edge-to-Cloud strategy can bring a number of benefits and challenges. Read this blog to know all about it.


Leave a comment / Query / Feedback

Your email address will not be published. Required fields are marked *