On Wed, 2017-03-29 at 19:06 +0000, George Kondiles wrote: I'm glad you're using it! Note that this is not even part of an official release yet. Further, the API we're going to release as part of SPDK 17.03 is not even the API that I envision the blobstore will have when all is said and done. I just want to correctly set expectations - I'm going to change the API quite a bit and not everything in the API currently makes sense. I also reserve the right to change the on-disk format still, for at least a few more months. Feedback of any kind is very much welcome. You'll always need to call spdk_allocate_thread when each new thread starts up that is using the blobstore (unless you are using our event framework from lib/event, which does that for you). If you want the blobstore to talk to the bdev layer, then you'll want to call spdk_get_io_channel and pass it the bdev as the io_device parameter. There is a full example of how to do this in lib/blob/bdev/blob_bdev.c. I highly recommend for this first version that you just follow that example. If you want the blobstore to talk directly to the NVMe driver, however, I haven't written an example to show you how just yet. I think the easiest way to implement spdk_bs_dev::create_channel directly on the NVMe driver is to make it call spdk_nvme_ctrlr_alloc_io_qpair and then return (and cast) the queue pair to an spdk_io_channel object. That's cheating a bit but I think it will work out. I'll try to write up an example that demonstrates the best way to do this in the next week or two. There are some other challenges here, such as who polls each queue pair for completions, that using the bdev layer just solves for you. The spdk_bs_dev::create_channel function will only be called on the thread that will be using that channel. That thread should have already been set up with the spdk_allocate_thread when it started, but you can just call spdk_get_io_channel from within the create_channel callback. See lib/blob/bdev/blob_bdev.c for an example that you can probably just use outright. I think you're on the right track here. Our spdk_io_channel structure is just a software construct for tracking per-thread contexts up and down the I/O stack. The bottom of that stack is an NVMe queue pair typically. This idea is a powerful one, but one that we haven't done a great job explaining just yet. It is a dramatic departure from concepts present in POSIX too, so it will be unfamiliar to most people. Yes. SATA calls it TRIM. NVMe calls it dealloc. SCSI calls it UNMAP. Maybe I should call it dealloc because that's actually the most descriptive term and we're very NVMe-centric. Of the three terms, I'm sure that's the one least used though. Blob metadata is cached, but only while a blob is open. If you close the blob all of the memory is released. I don't have exact counts (and it is very much subject to change), but you can expect maybe ~128B per open blob. There are a few operations (i.e. opening a blob) that are currently O(N) where N is the number of OPEN blobs. This is just because I haven't had a chance to implement a better algorithm yet. There aren't any operations that are O(N) where N is the total number of blobs. In general, blobs are entirely independent of one another because they each have their own blocks for metadata and data and the location of that metadata can be determined entirely from the blobid with no shared data structure. That's the real key to this design - with the exception of a bit mask that requires central coordination for a brief, synchronous period only when doing a few rare metadata operations (create, sync, delete), every other operation on the blobstore can happen entirely in parallel with no locks.