Using NVMe on KVM
You can use NVMe (Non-Volatile Memory Express) both as a storage and cache device on CentOS 6/7 KVM compute resources. However, since NVMe benefits most from a direct connection between internal physical server peripherals, we recommend using NVMe as a cache device and SATA SSD as a storage device.
NVMe SSD has an advantage over older storage protocols. NVMe speeds up operations and reduces CPU overhead that allows achieving lower latency and higher IOPS. NVMe also provides a more efficient mechanism for queuing messages. For instance, SAS and SATA support only one I/O queue at a time. The SAS queue can include up to 256 outstanding commands and the SATA queue up to 32. NVMe can support up to 65535 queues and up to 64000 commands per queue.
You can use NVMe as a storage device but it may cause some performance issues because of the Integrated Storage overhead; network and driver latency. Depending on the testing method, hardware, and network configuration, the expected performance drop in linear speeds may be up to 17%. However, it can be barely noticeable on production workloads.
OnApp doesn't provide support for NVMe Over Fabrics at the moment. When the NVMe Over Fabrics support is announced, it will provide a huge performance boost to Integrated Storage.