September 2020 - SPDK

Does BlobFS Asynchronous API support multi thread writing?

by chen.zhenghua＠zte.com.cn

Hi everyone, I simply tested the BlobFS Asynchronous API by using SPDK events framework to execute multi tasks, each task writes one file. But it doesn't work, the spdk_file_write_async() reported an error when resizing the file size. The call stack looks like this: spdk_file_write_async() -> __readwrite() -> spdk_file_truncate_async() -> spdk_blob_resize() The resize operation must be done in the metadata thread which invoked the spdk_fs_load(), so only the task dispatched to the metadata CPU core works. That's to say only one thread can be used to write files. It's hard to use, and performance issues may arise. Does anyone knows further more about this? thanks very much

2 months, 1 week

2
1
0 / 0

Reporting intermittent test failures

by Harris, James R

Hi all, I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue. Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked. Thanks, -Jim

1 year, 7 months

3
2
0 / 0

Rocksdb with SPDK

by dimitra.giantsidi＠gmail.com

Hi all, I am running Rocksdb on top of a blobfs filesystem and I have a couple of questions. 1. What exactly does the ```sanitize_path()``` function do and why do we need to sanitize the path before opening a file? If the fname is /mnt/rocksdb/log, the sanitize path returns /log .. What does that means for my application? 2. the file->Sync() operations fails with a message "Not Implemented". In particular, I am having the following code : ``` if (!FLAGS__spdk.empty()) { FLAGS_env = rocksdb::NewSpdkEnv(FLAGS_env, FLAGS_dir, FLAGS__spdk, FLAGS__spdk_bdev, FLAGS__spdk_cache_size); fprintf(stdout, "Using rocksdb::NewSpdkEnv() environment.\n"); if (FLAGS_env == nullptr) { fprintf(stderr, "Could not load SPDK blobfs - check that SPDK mkfs was run " "against block device %s.\n", FLAGS__spdk_bdev.c_str()); exit(1); } } unique_ptr<WritableFile> file; EnvOptions soptions; rocksdb::Slice data = return_data(FLAGS__value_size); Status s = FLAGS_env->NewWritableFile(FLAGS_file, &file, soptions); if (!s.ok()) { fprintf(stderr, "oops ..\n"); abort(); } size_t size = 0; size_t total_size = FLAGS_total_size*1024; struct timeval start_time, end_time; gettimeofday(&start_time, NULL); while (size < total_size) { s = file->Append(data); if (s.ok()) { s = file->Sync(); } size += FLAGS__value_size; if (!s.ok()) { fprintf(stdout, "written %d bytes to %s (%s)..\n", size, FLAGS_file.c_str(), s.ToString().c_str()); abort(); } } ``` However, I am seeing that the rocksdb::SpdkWritableFile::Sync() is implemented .. Any ideas? Thanks for your help :) Dimitra

1 year, 8 months

2
2
0 / 0

Re: Potential defect in pci_unplug()

by Harris, James R

Hi Niu, I agree, this doesn't look right. Could you file an SPDK issue for this to make sure we track it? And then try sending an e-mail to the DPDK mailing list? I'm open to submitting a patch to our DPDK submodule short-term, but only if we get agreement with DPDK community that this fix is correct. Thanks, Jim On 9/28/20, 12:17 AM, "Niu, Yawei" <yawei.niu(a)intel.com> wrote: Hi, In the pci_unplug() (dpdk/drivers/bus/pci/pci_common.c), why do we call rte_devargs_remove() to remove the saved device args? That looks a defect to me, since that’ll result in the hot removed device failed to be detected when it’s plugged back (when white list option was provided), the situation is described following: 1. App starts with white list option provided, let’s suppose the device in white list is: 0000:81:00.0; 2. rte_devargs_add() is called to add this device arg on rte_eal_init(); 3. Issue “echo 1 > /sys/bus/pci/devices/0000\:81\:00.0/remove” to hot remove the device; 4. pci_unplug() is called to remove the device, and rte_devargs_remove() is called, so this device arg for white list is remove too; 5. Issue “echo 1 > /sys/bus/pci/rescan” then “PCI_WHITELIST=”0000:81:00.0” spdk/script/setup.sh” to do hot plug; 6. In pci_scan_one(), new dev is generated, however, the dev->device.devargs is set NULL since the device args has been removed on device unplug; 7. rte_pci_probe() does white list scan, however, it unexpectedly skips the device because the devargs of the device is NULL now; I don’t fully understand the DPDK code, but this rte_devargs_remove() in pci_unplug() doesn’t make sense to me (when I commented it out, seems everything will work as expected). Is this a glitch or I missed anything? Thanks -Niu _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

1 year, 8 months

2
1
0 / 0

Potential defect in pci_unplug()

by Niu, Yawei

Hi, In the pci_unplug() (dpdk/drivers/bus/pci/pci_common.c), why do we call rte_devargs_remove() to remove the saved device args? That looks a defect to me, since that’ll result in the hot removed device failed to be detected when it’s plugged back (when white list option was provided), the situation is described following: 1. App starts with white list option provided, let’s suppose the device in white list is: 0000:81:00.0; 2. rte_devargs_add() is called to add this device arg on rte_eal_init(); 3. Issue “echo 1 > /sys/bus/pci/devices/0000\:81\:00.0/remove” to hot remove the device; 4. pci_unplug() is called to remove the device, and rte_devargs_remove() is called, so this device arg for white list is remove too; 5. Issue “echo 1 > /sys/bus/pci/rescan” then “PCI_WHITELIST=”0000:81:00.0” spdk/script/setup.sh” to do hot plug; 6. In pci_scan_one(), new dev is generated, however, the dev->device.devargs is set NULL since the device args has been removed on device unplug; 7. rte_pci_probe() does white list scan, however, it unexpectedly skips the device because the devargs of the device is NULL now; I don’t fully understand the DPDK code, but this rte_devargs_remove() in pci_unplug() doesn’t make sense to me (when I commented it out, seems everything will work as expected). Is this a glitch or I missed anything? Thanks -Niu

1 year, 9 months

1
0
0 / 0

Cancelled: 1st October Asia-friendly SPDK community meeting and 8th October Bug Scrub meeting

by Liu, Changpeng

Hi everyone, The next Asia-friendly SPDK community meeting and Bug Scrub meeting will be cancelled due to PRC holidays.

1 year, 9 months

1
0
0 / 0

a distributed storage system base on SPDK

by peng yu

hello, sorry if this is not the correct place to send such an email. I'm writing a distributed storage system, which uses SPDK as the data plane. My purpose is using it as a persistent volume for k8s or a backend driver for openstack cinder. It is a personal open source project. I hope to find someone who is interested in it too. My idea is creating a virtual disk array, using SPDK as the disk array's controller and disk. The SPDK on the disk exports the physical disk to the controller. The SPDK on the controller aggregates multiple disks to a single volume and exports to a host. We could set raid level or encryption on the controller. Now I have a workable version, and my next step is writing a CSI plugin for it, let it work on k8s. Hope to hear some feedback about it. Here is the home page of the project: https://virtual-disk-array.readthedocs.io/en/latest/Introduction.html Best regards.

1 year, 9 months

6
9
0 / 0

thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

by sunshihao＠huawei.com

hello，maintainer： when i use spdk api spdk_thread_send_msg to send io read/write msg，i found a bug： multiple threads send msg to one spdk_thread，they may get msg form local_thread->msg_cache，there is not a lock or cas to ensure the safety of the linked list。 msg = NULL; if (local_thread != NULL) { if (local_thread->msg_cache_count > 0) { msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */ assert(msg != NULL); SLIST_REMOVE_HEAD(&local_thread->msg_cache, link); local_thread->msg_cache_count--; } } i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe，so please give me some advice。 Thank you！

1 year, 9 months

4
5
0 / 0

How to do mmio correctly when vfio-pci is used?

by oscar.huang＠microchip.com

Hi, Gugus, I have a simple SPDK application, which maps a device BAR, and write some data onto the device with spdk_mmio_write_8 function. The code flow is like this: ... spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size); addrToWrite = virBarAddr + offset; //offset is where I'd like to write data in the device memory spdk_mmio_write_8(addrToWrite, data); ... The code has been working with uio_pci_generic. But recently when I switched to vfio-pci (iommu enabled), I got a segmentation fault the program was executing that spdk_mmio_write_8 function. If I do the following, the program can run: ... spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size); physAddr = phys +offset; int fd = open("/dev/mem", O_RDWR | O_SYNC); void *map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, physAddr & ~MAP_MASK); addrToWrite = (uint64_t)map_base + (physAddr & MAP_MASK); spdk_mmio_write_8(addrToWrite, data); ... I don't understand why it is so. Basically I need a routine to map device physical address to process virtual address, and then call spdk_mmio_write_x functions. It looks like the virtual address from spdk_pci_device_map_bar cannot be used. I noticed the virtual address from spdk_pci_device_map_bar was 0x2010008000, but the one by mmap was 0x7ffff7fef000, very different. Is there a unified mmio approach that works for both uio_pci_generic and vfio_pci?

1 year, 9 months

2
1
0 / 0

Cancelled: SPDK Asia-Friendly Thursday, September 17, 2020, 12:00 PM GMT+8 Bug Scrub meeting Community Meeting

by Liu, Changpeng

1 year, 9 months

1
0
0 / 0

2022

2021

2020

2019

2018

2017

2016

SPDK September 2020