April 2020 - SPDK - Ml01.01.Org

EAL: set_mempolicy failed: Invalid argument

by Andrey Kuzmin

Hi guys, I'm running 20.01 (default configuration) on a CentOS 7 Epyc-based system, and getting a lot of subj warnings like in the below example. Does that point to a system configuration issue that needs to be fixed? Thanks, Andrey Starting SPDK v20.01 git sha1 5ffffe9 / DPDK 19.11.0 initialization... [ DPDK EAL parameters: hotplug --no-shconf -c 0x1 --log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6 --iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations --file-prefix=spdk_pid109760 ] EAL: No available hugepages reported in hugepages-1048576kB EAL: VFIO support initialized EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Initializing NVMe Controllers EAL: set_mempolicy failed: Invalid argument EAL: using IOMMU type 1 (Type 1) EAL: set_mempolicy failed: Invalid argument Attaching to 0000:45:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:46:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4b:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4c:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4d:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attaching to 0000:4e:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:45:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:46:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4b:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4c:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4d:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Attached to 0000:4e:00.0 EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument EAL: set_mempolicy failed: Invalid argument Initialization complete. Starting I/O...

10 months, 3 weeks

3
6
0 / 0

how to deploy and run spdk in centos7

by mengxianghong2008＠126.com

I build spdk at unbuntu, after build, how can I deploy it and run at centos7? Anyone can give me a doc or link to instructure me, best wishes!!!

10 months, 4 weeks

4
3
0 / 0

spdk_zmalloc error

by cfrancisy＠gmail.com

Hi, I meet a weird problem. The error below only happens in one machine of my cluster, but in other nodes, the program run successfully without bug. I try to build a memory pool, which consists of 256 buffers, each is 32M. Here is my code snippet. For a memory pool, it will execute many times. ---------------------------------------------- #define SPDK_PAGE_SIZE (4096) #define SPDK_BUFFER_SIZE (32ull<<20) buffer_ = (char*)spdk_zmalloc(SPDK_BUFFER_SIZE, SPDK_PAGE_SIZE, NULL, SPDK_ENV_SOCKET_ID_ANY, SPDK_MALLOC_DMA); if(buffer_ == nullptr){ fprintf(stderr, "%s:%d: SPDK allocate memory failed\n", __FILE__, __LINE__); } --------------------------------------- I found that in some buffer allocations, the variable buffer_ after the spdk_zmalloc is still a nullptr. It makes the program throw error output. I start up spdk enviroment with hugemem=16384, the total memory pool size 8GB. SPDK vesion is v20.01 Could someone give me some comments? thanks.

10 months, 4 weeks

3
2
0 / 0

SPDK ABI Improvements

by Howell, Seth

Hello All, We are taking several steps towards solidifying the ABI strategy for SPDK. These include the following: 1. Creating an ABI major/minor versioning scheme for all shared libraries. When a new symbol is added to a library, the shared object minor version will be incremented to show that the library is still backwards compatible. When a symbol is removed or changed, the ABI major version will be updated to indicate that the library is no longer backwards compatible. This versioning scheme will be enforced by the following new test in the SPDK CI: https://review.spdk.io/gerrit/c/spdk/spdk/+/1068. 2. Incrementing the shared object version of each library independently. 3. Creating individual map files for each shared library. As we create individual map files for the libraries, ABIs are likely to change, so please keep an eye out for changes in the major/minor version of SPDK libraries as you continue to link your applications to them. Thank you, Seth Howell

10 months, 4 weeks

3
4
0 / 0

Re: A few thoughts about spdk_top

by Harris, James R

Hi Darek, On 4/7/20, 3:30 AM, "Stojaczyk, Dariusz" <dariusz.stojaczyk(a)intel.com> wrote: I like the idea of spdk_top application, although I'm slightly concerned about our implementation. Is it a right choice to develop it in C? It's already a huge amount of code and it already has a huge amount of limitations, not even mentioning possible bugs. I actually spent a lot of time developing in curses, GTK, GTKmm and even WinAPI and I don't do it anymore because it's difficult and not worth it. I switched to javascript as well as server-side javascript for my other projects and I find it incredibly easier to write user apps there. I'm not trying to push on javascript specifically, but could we consider writing spdk_top in a higher level language, where we don't care about memory allocation, json parsing, and printing data to the screen? [Jim] I think there's room for multiple user interfaces. Personally, I like a terminal-based application like this - I typically run tmux with 4 panes, and having something like this in one pane while I'm running a target in the foreground in another works well for me. Maybe something in Python could end up as a bit less code? It's possible, but I look at something like spdkcli and it has quite a bit of code too. [Jim] Regarding json parsing, I think it is nice to have another use case for our client-side JSON RPC APIs. But you're right, there's about 300 lines of code in there specific to issuing RPCs to the app. Potentially some of that long-term could be moved into a common library for other applications that have a need for issuing RPCs from a C application. Currently spdk_top seems to require the same, detailed review as any spdk patch and honestly I'm a bit reluctant to review it. Moreover, I find one major feature missing there - a view with all pollers on a specific, single thread and busy time % for each poller, so that I can see what my thread is most busy with. I think it would be the first view I check when doing any spdk optimization. Yet I see it's quite a bit of effort to add it to the current spdk_top code, which brings me back to my first question. [Jim] I think now's the time to provide that feedback on what might be missing. But the basic infrastructure is all in place, and knowing Maciek I'm sure he's open to suggestions! I'd encourage everyone to pull down the code and kick the proverbial tires. The end of the series can be found at https://review.spdk.io/gerrit/c/spdk/spdk/+/1717/4. I've also started reviewing individual patches and am suggesting areas where the code could be simplified a bit. Regards, -Jim

11 months

2
1
0 / 0

A few thoughts about spdk_top

by Stojaczyk, Dariusz

I like the idea of spdk_top application, although I'm slightly concerned about our implementation. Is it a right choice to develop it in C? It's already a huge amount of code and it already has a huge amount of limitations, not even mentioning possible bugs. I actually spent a lot of time developing in curses, GTK, GTKmm and even WinAPI and I don't do it anymore because it's difficult and not worth it. I switched to javascript as well as server-side javascript for my other projects and I find it incredibly easier to write user apps there. I'm not trying to push on javascript specifically, but could we consider writing spdk_top in a higher level language, where we don't care about memory allocation, json parsing, and printing data to the screen? Currently spdk_top seems to require the same, detailed review as any spdk patch and honestly I'm a bit reluctant to review it. Moreover, I find one major feature missing there - a view with all pollers on a specific, single thread and busy time % for each poller, so that I can see what my thread is most busy with. I think it would be the first view I check when doing any spdk optimization. Yet I see it's quite a bit of effort to add it to the current spdk_top code, which brings me back to my first question. D.

11 months

1
0
0 / 0

Re: Fail to start cinder service

by Szwed, Maciej

Hi Ranjan, I happy that this helped to proceed further. The log you pasted below is cut (lines are cut after some number of words) – could you please try to get the rest of the lines as it seems that most important information was there? Thanks, Maciek From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com> Sent: Tuesday, March 31, 2020 7:42 AM To: Szwed, Maciej <maciej.szwed(a)intel.com> Cc: Storage Performance Development Kit <spdk(a)lists.01.org> Subject: RE: Fail to start cinder service Thanks a ton for such instant response Szwed. My bad, I just missed this email. I removed the comments, and at least the service is running. I am yet to check if I can access the volume from horizon. Will update you soon on this. root@ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack@c-vol.service> ● devstack(a)c-vol.service<mailto:devstack@c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack@c-vol.service> Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack@c-vol.service>; enabled; vendor preset: enabled) Active: active (running) since Tue 2020-03-31 11:09:03 IST; 23s ago Main PID: 18375 (cinder-volume) Tasks: 2 (limit: 6143) CGroup: /system.slice/system-devstack.slice/devstack(a)c-vol.service<mailto:/system.slice/system-devstack.slice/devstack@c-vol.service> ├─18375 /usr/bin/python3.6 /usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf └─18421 /usr/bin/python3.6 /usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=err_msg) Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Could no Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG cinder.service [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Creating RPC server for service cinder-volume {{(pid=18421) start Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG oslo_db.sqlalchemy.engines [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_A Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG cinder.service [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Pinning object versions for RPC server serializer to 1.38 {{(pid= Mar 31 11:09:13 ubuntu cinder-volume[18375]: INFO cinder.volume.manager [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Initializing RPC dependent components of volume driver SPDK Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.utils [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Volume driver SPDKDriver not initialized Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Cannot complete RPC initialization because driver isn't in Mar 31 11:09:23 ubuntu cinder-volume[18375]: ERROR cinder.service [-] Manager for service cinder-volume ubuntu@spdk is reporting problems, not sending heartbeat. Service will appear "down". Regards, Ranjan From: Szwed, Maciej <maciej.szwed(a)intel.com<mailto:maciej.szwed@intel.com>> Sent: Tuesday, March 24, 2020 7:38 PM To: Kumar Ranjan <Kumar.Ranjan(a)wdc.com<mailto:Kumar.Ranjan@wdc.com>> Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>> Subject: RE: Fail to start cinder service CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe. Hi Ranjan, Nice to meet you. I’m adding SPDK mailing list for wider audience (someone may find it helpful someday 😊 ). Maybe it has to do something with additional comments on each line in configuration file – try to remove them. Also in “spdk_rpc_port =4420” you are missing space after “=” – I doubt that this may has something to do with that missing space, but it is always worth trying removing it. Everything else looks good. Let me know if that won’t work – I’ll try to reproduce environment on my side. Regards, Maciej From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com<mailto:Kumar.Ranjan@wdc.com>> Sent: Monday, March 23, 2020 6:57 PM To: Szwed, Maciej <maciej.szwed(a)intel.com<mailto:maciej.szwed@intel.com>> Subject: Fail to start cinder service Hi Maciej, I am Ranjan from WDC. Request your help in understanding as why I am not able to start the cinder service. I saw some of your patches and looks like you are more into this field. Hence pinging you. For experiment : devstack as well my client is running in the same server. My host IP for Horizon (private network) : 192.168.1.14 My target IP where rpc is listening : 192.168.1.111 My target has 2 data ports , one is : 192.168.1.111 and another : 172.168.1.111 Spdk entried in my Cinder.conf [spdk] spdk_rpc_ip = 192.168.1.111 #address to machine with SPDK spdk_rpc_port =4420 #port to use to send commands to SPDK once running with -r #spdk_rpc_username = admin#username set in remote_rpc.py #spdk_rpc_password = admin#password set in remote_rpc.py target_ip_address = 172.168.1.111 #NVMe-oF interface target_port = 4420 target_protocol = nvmet_rdma target_helper = spdk-nvmeof target_prefix = nqn.1992-05.com.wdc volume_driver = cinder.volume.drivers.spdk.SPDKDriver volume_backend_name = SPDK --------------- root@ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack@c-vol.service> ● devstack(a)c-vol.service<mailto:devstack@c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack@c-vol.service> Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack@c-vol.service>; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2020-03-23 23:16:59 IST; 5s ago Process: 15636 ExecStart=/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf (code=exited, status=1/FAILURE) Main PID: 15636 (code=exited, status=1/FAILURE) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume return self._conf._get(name, self._group) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2639, in _get Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume value, loc = self._do_get(name, group, namespace) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2713, in _do_get Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume raise ConfigFileValueError(message) Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume oslo_config.cfg.ConfigFileValueError: Value for option spdk_rpc_port from LocationInfo(location=<Locations.user: (4, Tru Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume [None req-5dddc9ec-a2a6-4c94-913c-caea68014ac5 None None] No volume service(s) started successfully, terminating. Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack@c-vol.service>: Main process exited, code=exited, status=1/FAILURE Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack@c-vol.service>: Failed with result 'exit-code'. Target : root@KAMSHED:/usr/share/spdk/scripts# ps aux | grep nvmf root 14676 691 5.0 4271148 409508 pts/0 SLl 01:28 37:21 /usr/sbin/nvmf_tgt -o -j -c /usr/share/spdk/nvmf.conf -r 192.168.1.111:4420 root 15844 0.0 0.0 2316 464 pts/0 S+ 01:33 0:00 grep nvmf Regards, Ranjan

11 months

1
0
0 / 0

is it possible to add a blacklist for the lvol module?

by peng yu

I have a below scenario: A disk is controlled by SPDK APP A. SPDK APP A exports the disk as a NVMEoF target SPDK APP B is a NVMEoF initiator, it connects to that disk. SPDk APP B creates a lvs and a lvol on the disk. When I restart the APP A and APP B. The APP A will examine the disk and find there is a lvs, and then the lvol module on APP A will claim this disk, then this disk can't be exported over NVMEoF. Could we add a api to the lvol module? The api could add a dev name pattern to a black list. When the lvol module examine a disk, it will check against the black list, if a dev name matches the black list, it will ignore this device. Best regards.

11 months, 1 week

3
5
0 / 0

2021

2020

2019

2018

2017

2016

SPDK April 2020