September 2020 - SPDK

by Harris, James R

Hi all, I’ve seen a lot of cases recently where -1 votes from the test pool have been removed from a patch due to a failure unrelated to the patch, but then nothing was filed in GitHub for that failure. The filing in GitHub could be a new issue, or a comment on an existing issue. Please make those GitHub updates a priority. It’s the only way the project can understand the frequency of those intermittent failures and gather to get them fixed. If you’re not sure if a failure has been seen before, search GitHub issues with the “Intermittent Failure” label, or ask on Slack if anyone else has seen the issue. There is no harm in filing a new issue that may be a duplicate – we can always clean these up later during the next bug scrub meeting. The important thing is that we get the failure tracked. Thanks, -Jim

7 months

3
2
0 / 0

Rocksdb with SPDK

by dimitra.giantsidi＠gmail.com

Hi all, I am running Rocksdb on top of a blobfs filesystem and I have a couple of questions. 1. What exactly does the ```sanitize_path()``` function do and why do we need to sanitize the path before opening a file? If the fname is /mnt/rocksdb/log, the sanitize path returns /log .. What does that means for my application? 2. the file->Sync() operations fails with a message "Not Implemented". In particular, I am having the following code : ``` if (!FLAGS__spdk.empty()) { FLAGS_env = rocksdb::NewSpdkEnv(FLAGS_env, FLAGS_dir, FLAGS__spdk, FLAGS__spdk_bdev, FLAGS__spdk_cache_size); fprintf(stdout, "Using rocksdb::NewSpdkEnv() environment.\n"); if (FLAGS_env == nullptr) { fprintf(stderr, "Could not load SPDK blobfs - check that SPDK mkfs was run " "against block device %s.\n", FLAGS__spdk_bdev.c_str()); exit(1); } } unique_ptr<WritableFile> file; EnvOptions soptions; rocksdb::Slice data = return_data(FLAGS__value_size); Status s = FLAGS_env->NewWritableFile(FLAGS_file, &file, soptions); if (!s.ok()) { fprintf(stderr, "oops ..\n"); abort(); } size_t size = 0; size_t total_size = FLAGS_total_size*1024; struct timeval start_time, end_time; gettimeofday(&start_time, NULL); while (size < total_size) { s = file->Append(data); if (s.ok()) { s = file->Sync(); } size += FLAGS__value_size; if (!s.ok()) { fprintf(stdout, "written %d bytes to %s (%s)..\n", size, FLAGS_file.c_str(), s.ToString().c_str()); abort(); } } ``` However, I am seeing that the rocksdb::SpdkWritableFile::Sync() is implemented .. Any ideas? Thanks for your help :) Dimitra

8 months, 2 weeks

2
2
0 / 0

Re: Potential defect in pci_unplug()

by Harris, James R

Hi Niu, I agree, this doesn't look right. Could you file an SPDK issue for this to make sure we track it? And then try sending an e-mail to the DPDK mailing list? I'm open to submitting a patch to our DPDK submodule short-term, but only if we get agreement with DPDK community that this fix is correct. Thanks, Jim On 9/28/20, 12:17 AM, "Niu, Yawei" <yawei.niu(a)intel.com> wrote: Hi, In the pci_unplug() (dpdk/drivers/bus/pci/pci_common.c), why do we call rte_devargs_remove() to remove the saved device args? That looks a defect to me, since that’ll result in the hot removed device failed to be detected when it’s plugged back (when white list option was provided), the situation is described following: 1. App starts with white list option provided, let’s suppose the device in white list is: 0000:81:00.0; 2. rte_devargs_add() is called to add this device arg on rte_eal_init(); 3. Issue “echo 1 > /sys/bus/pci/devices/0000\:81\:00.0/remove” to hot remove the device; 4. pci_unplug() is called to remove the device, and rte_devargs_remove() is called, so this device arg for white list is remove too; 5. Issue “echo 1 > /sys/bus/pci/rescan” then “PCI_WHITELIST=”0000:81:00.0” spdk/script/setup.sh” to do hot plug; 6. In pci_scan_one(), new dev is generated, however, the dev->device.devargs is set NULL since the device args has been removed on device unplug; 7. rte_pci_probe() does white list scan, however, it unexpectedly skips the device because the devargs of the device is NULL now; I don’t fully understand the DPDK code, but this rte_devargs_remove() in pci_unplug() doesn’t make sense to me (when I commented it out, seems everything will work as expected). Is this a glitch or I missed anything? Thanks -Niu _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

8 months, 2 weeks

2
1
0 / 0

Potential defect in pci_unplug()

by Niu, Yawei

Hi, In the pci_unplug() (dpdk/drivers/bus/pci/pci_common.c), why do we call rte_devargs_remove() to remove the saved device args? That looks a defect to me, since that’ll result in the hot removed device failed to be detected when it’s plugged back (when white list option was provided), the situation is described following: 1. App starts with white list option provided, let’s suppose the device in white list is: 0000:81:00.0; 2. rte_devargs_add() is called to add this device arg on rte_eal_init(); 3. Issue “echo 1 > /sys/bus/pci/devices/0000\:81\:00.0/remove” to hot remove the device; 4. pci_unplug() is called to remove the device, and rte_devargs_remove() is called, so this device arg for white list is remove too; 5. Issue “echo 1 > /sys/bus/pci/rescan” then “PCI_WHITELIST=”0000:81:00.0” spdk/script/setup.sh” to do hot plug; 6. In pci_scan_one(), new dev is generated, however, the dev->device.devargs is set NULL since the device args has been removed on device unplug; 7. rte_pci_probe() does white list scan, however, it unexpectedly skips the device because the devargs of the device is NULL now; I don’t fully understand the DPDK code, but this rte_devargs_remove() in pci_unplug() doesn’t make sense to me (when I commented it out, seems everything will work as expected). Is this a glitch or I missed anything? Thanks -Niu

8 months, 2 weeks

1
0
0 / 0

Cancelled: 1st October Asia-friendly SPDK community meeting and 8th October Bug Scrub meeting

by Liu, Changpeng

Hi everyone, The next Asia-friendly SPDK community meeting and Bug Scrub meeting will be cancelled due to PRC holidays.

8 months, 2 weeks

1
0
0 / 0

a distributed storage system base on SPDK

by peng yu

hello, sorry if this is not the correct place to send such an email. I'm writing a distributed storage system, which uses SPDK as the data plane. My purpose is using it as a persistent volume for k8s or a backend driver for openstack cinder. It is a personal open source project. I hope to find someone who is interested in it too. My idea is creating a virtual disk array, using SPDK as the disk array's controller and disk. The SPDK on the disk exports the physical disk to the controller. The SPDK on the controller aggregates multiple disks to a single volume and exports to a host. We could set raid level or encryption on the controller. Now I have a workable version, and my next step is writing a CSI plugin for it, let it work on k8s. Hope to hear some feedback about it. Here is the home page of the project: https://virtual-disk-array.readthedocs.io/en/latest/Introduction.html Best regards.

8 months, 3 weeks

6
9
0 / 0

thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

by sunshihao＠huawei.com

hello，maintainer： when i use spdk api spdk_thread_send_msg to send io read/write msg，i found a bug： multiple threads send msg to one spdk_thread，they may get msg form local_thread->msg_cache，there is not a lock or cas to ensure the safety of the linked list。 msg = NULL; if (local_thread != NULL) { if (local_thread->msg_cache_count > 0) { msg = SLIST_FIRST(&local_thread->msg_cache); /* here maybe not thread safe */ assert(msg != NULL); SLIST_REMOVE_HEAD(&local_thread->msg_cache, link); local_thread->msg_cache_count--; } } i don't konw whether is my wrong way to use spdk_thread_send_msg or the list is not safe，so please give me some advice。 Thank you！

8 months, 4 weeks

4
5
0 / 0

How to do mmio correctly when vfio-pci is used?

by oscar.huang＠microchip.com

Hi, Gugus, I have a simple SPDK application, which maps a device BAR, and write some data onto the device with spdk_mmio_write_8 function. The code flow is like this: ... spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size); addrToWrite = virBarAddr + offset; //offset is where I'd like to write data in the device memory spdk_mmio_write_8(addrToWrite, data); ... The code has been working with uio_pci_generic. But recently when I switched to vfio-pci (iommu enabled), I got a segmentation fault the program was executing that spdk_mmio_write_8 function. If I do the following, the program can run: ... spdk_pci_device_map_bar(dev, bir, &virBarAddr, &phys, &size); physAddr = phys +offset; int fd = open("/dev/mem", O_RDWR | O_SYNC); void *map_base = mmap(0, MAP_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, physAddr & ~MAP_MASK); addrToWrite = (uint64_t)map_base + (physAddr & MAP_MASK); spdk_mmio_write_8(addrToWrite, data); ... I don't understand why it is so. Basically I need a routine to map device physical address to process virtual address, and then call spdk_mmio_write_x functions. It looks like the virtual address from spdk_pci_device_map_bar cannot be used. I noticed the virtual address from spdk_pci_device_map_bar was 0x2010008000, but the one by mmap was 0x7ffff7fef000, very different. Is there a unified mmio approach that works for both uio_pci_generic and vfio_pci?

8 months, 4 weeks

2
1
0 / 0

Cancelled: SPDK Asia-Friendly Thursday, September 17, 2020, 12:00 PM GMT+8 Bug Scrub meeting Community Meeting

by Liu, Changpeng

8 months, 4 weeks

1
0
0 / 0

Re: Print backtrace in SPDK

by Wenhua Liu

Hi Ziye, I'm using SPDK NVMe-oF target. I used some other way and figured out the following call path: posix_sock_group_impl_poll -> _sock_flush <------------------ failed -> spdk_sock_abort_requests -> _pdu_write_done -> nvmf_tcp_qpair_disconnect -> spdk_nvmf_qpair_disconnect -> _nvmf_qpair_destroy -> spdk_nvmf_poll_group_remove -> nvmf_transport_poll_group_remove -> nvmf_tcp_poll_group_remove -> spdk_sock_group_remove_sock -> posix_sock_group_impl_remove_sock -> spdk_sock_abort_requests -> _nvmf_ctrlr_free_from_qpair -> _nvmf_transport_qpair_fini -> nvmf_transport_qpair_fini -> nvmf_tcp_close_qpair -> spdk_sock_close The _sock_flush calls sendmsg to write the data to the socket. It's sendmsg failing with return value -1. I captured wire data. In Wireshark, I can see the READ command has been received by the target as a TCP packet. As the response to this TCP packet, a TCP packet with FIN flag set is sent to the initiator. The FIN is to close the socket connection. I'm running SPDK target inside a VM. My NVMe/TCP initiator runs inside another VM. I'm going to try with another SPDK target which runs on a physical machine. By the way, I noticed there is a uring based sock implementation, how do I switch to this sock implementation. It seems the default is posix sock implementation. Thanks, -Wenhua On 8/23/20, 9:55 PM, "Yang, Ziye" <ziye.yang(a)intel.com> wrote: Hi Wenhua, Which applications are you using from SPDK? 1 SPDK NVMe-oF target in target side? 2 SPDK NVMe perf or others? For nvmf_tcp_close_qpair will be called in the following possible cases (not all listed) for TCP transport. But it will be called by spdk_nvmf_qpair_disconnect as the entry. 1 qpair is not in polling group spdk_nvmf_qpair_disconnect nvmf_transport_qpair_fini 2 spdk_nvmf_qpair_disconnect .... _nvmf_qpair_destroy nvmf_transport_qpair_fini .. nvmf_tcp_close_qpair 3 spdk_nvmf_qpair_disconnect .... _nvmf_qpair_destroy _nvmf_ctrlr_free_from_qpair _nvmf_transport_qpair_fini .. nvmf_tcp_close_qpair spdk_nvmf_qpair_disconnect is called by nvmf_tcp_qpair_disconnect in tcp.c. nvmf_tcp_qpair_disconnect is called in the following cases: (1) _pdu_write_done (if there is error for write); (2) nvmf_tcp_qpair_handle_timeout.( No response from initiator in 30s if targets sends c2h_term_req) (3) nvmf_tcp_capsule_cmd_hdr_handle. (Cannot get tcp req) (4) nvmf_tcp_sock_cb. TCP PDU related handling issue. Also in lib/nvmf/ctrlr.c Target side has a timer poller: nvmf_ctrlr_keep_alive_poll. If there is no keep alive command sent from host, it will call spdk_nvmf_qpair_disconnect in related polling group assoicated with the controller. Best Regards Ziye Yang -----Original Message----- From: Wenhua Liu <liuw(a)vmware.com> Sent: Saturday, August 22, 2020 3:15 PM To: Storage Performance Development Kit <spdk(a)lists.01.org> Subject: [SPDK] Print backtrace in SPDK Hi, Does anyone know if there is a function in SPDK that prints the backtrace? I run into a “Connection Reset by Peer” issue on host side when testing NVMe/TCP. I identified it’s because some queue pairs are closed unexpectedly by calling nvmf_tcp_close_qpair, but I could not figure out how/why this function is called. I thought if the backtrace can be printed when calling this function, it might be helpful to me to find the root cause. Thanks, -Wenhua _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org

9 months

2
11
0 / 0

2021

2020

2019

2018

2017

2016

SPDK September 2020