EAL: set_mempolicy failed: Invalid argument
by Andrey Kuzmin
Hi guys,
I'm running 20.01 (default configuration) on a CentOS 7 Epyc-based
system, and getting a lot of subj warnings like in the below example.
Does that point to a system configuration issue that needs to be
fixed?
Thanks,
Andrey
Starting SPDK v20.01 git sha1 5ffffe9 / DPDK 19.11.0 initialization...
[ DPDK EAL parameters: hotplug --no-shconf -c 0x1
--log-level=lib.eal:6 --log-level=lib.cryptodev:5 --log-level=user1:6
--iova-mode=pa --base-virtaddr=0x200000000000 --match-allocations
--file-prefix=spdk_pid109760 ]
EAL: No available hugepages reported in hugepages-1048576kB
EAL: VFIO support initialized
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Initializing NVMe Controllers
EAL: set_mempolicy failed: Invalid argument
EAL: using IOMMU type 1 (Type 1)
EAL: set_mempolicy failed: Invalid argument
Attaching to 0000:45:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attaching to 0000:46:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attaching to 0000:4b:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attaching to 0000:4c:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attaching to 0000:4d:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attaching to 0000:4e:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attached to 0000:45:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attached to 0000:46:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attached to 0000:4b:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attached to 0000:4c:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attached to 0000:4d:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Attached to 0000:4e:00.0
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
EAL: set_mempolicy failed: Invalid argument
Initialization complete. Starting I/O...
10 months, 3 weeks
how to deploy and run spdk in centos7
by mengxianghong2008@126.com
I build spdk at unbuntu, after build, how can I deploy it and run at centos7? Anyone can give me a doc or link to instructure me, best wishes!!!
10 months, 4 weeks
spdk_zmalloc error
by cfrancisy@gmail.com
Hi,
I meet a weird problem. The error below only happens in one machine of my cluster, but in other nodes, the program run successfully without bug.
I try to build a memory pool, which consists of 256 buffers, each is 32M.
Here is my code snippet. For a memory pool, it will execute many times.
----------------------------------------------
#define SPDK_PAGE_SIZE (4096)
#define SPDK_BUFFER_SIZE (32ull<<20)
buffer_ = (char*)spdk_zmalloc(SPDK_BUFFER_SIZE,
SPDK_PAGE_SIZE, NULL,
SPDK_ENV_SOCKET_ID_ANY,
SPDK_MALLOC_DMA);
if(buffer_ == nullptr){
fprintf(stderr, "%s:%d: SPDK allocate memory failed\n", __FILE__, __LINE__);
}
---------------------------------------
I found that in some buffer allocations, the variable buffer_ after the spdk_zmalloc is still a nullptr.
It makes the program throw error output.
I start up spdk enviroment with hugemem=16384, the total memory pool size 8GB.
SPDK vesion is v20.01
Could someone give me some comments? thanks.
10 months, 4 weeks
SPDK ABI Improvements
by Howell, Seth
Hello All,
We are taking several steps towards solidifying the ABI strategy for SPDK. These include the following:
1. Creating an ABI major/minor versioning scheme for all shared libraries. When a new symbol is added to a library, the shared object minor version will be incremented to show that the library is still backwards compatible. When a symbol is removed or changed, the ABI major version will be updated to indicate that the library is no longer backwards compatible. This versioning scheme will be enforced by the following new test in the SPDK CI: https://review.spdk.io/gerrit/c/spdk/spdk/+/1068.
2. Incrementing the shared object version of each library independently.
3. Creating individual map files for each shared library.
As we create individual map files for the libraries, ABIs are likely to change, so please keep an eye out for changes in the major/minor version of SPDK libraries as you continue to link your applications to them.
Thank you,
Seth Howell
10 months, 4 weeks
Re: A few thoughts about spdk_top
by Harris, James R
Hi Darek,
On 4/7/20, 3:30 AM, "Stojaczyk, Dariusz" <dariusz.stojaczyk(a)intel.com> wrote:
I like the idea of spdk_top application, although I'm slightly concerned about our implementation. Is it a right choice to develop it in C? It's already a huge amount of code and it already has a huge amount of limitations, not even mentioning possible bugs. I actually spent a lot of time developing in curses, GTK, GTKmm and even WinAPI and I don't do it anymore because it's difficult and not worth it. I switched to javascript as well as server-side javascript for my other projects and I find it incredibly easier to write user apps there. I'm not trying to push on javascript specifically, but could we consider writing spdk_top in a higher level language, where we don't care about memory allocation, json parsing, and printing data to the screen?
[Jim] I think there's room for multiple user interfaces. Personally, I like a terminal-based application like this - I typically run tmux with 4 panes, and having something like this in one pane while I'm running a target in the foreground in another works well for me. Maybe something in Python could end up as a bit less code? It's possible, but I look at something like spdkcli and it has quite a bit of code too.
[Jim] Regarding json parsing, I think it is nice to have another use case for our client-side JSON RPC APIs. But you're right, there's about 300 lines of code in there specific to issuing RPCs to the app. Potentially some of that long-term could be moved into a common library for other applications that have a need for issuing RPCs from a C application.
Currently spdk_top seems to require the same, detailed review as any spdk patch and honestly I'm a bit reluctant to review it. Moreover, I find one major feature missing there - a view with all pollers on a specific, single thread and busy time % for each poller, so that I can see what my thread is most busy with. I think it would be the first view I check when doing any spdk optimization. Yet I see it's quite a bit of effort to add it to the current spdk_top code, which brings me back to my first question.
[Jim] I think now's the time to provide that feedback on what might be missing. But the basic infrastructure is all in place, and knowing Maciek I'm sure he's open to suggestions! I'd encourage everyone to pull down the code and kick the proverbial tires. The end of the series can be found at https://review.spdk.io/gerrit/c/spdk/spdk/+/1717/4. I've also started reviewing individual patches and am suggesting areas where the code could be simplified a bit.
Regards,
-Jim
11 months
A few thoughts about spdk_top
by Stojaczyk, Dariusz
I like the idea of spdk_top application, although I'm slightly concerned about our implementation. Is it a right choice to develop it in C? It's already a huge amount of code and it already has a huge amount of limitations, not even mentioning possible bugs. I actually spent a lot of time developing in curses, GTK, GTKmm and even WinAPI and I don't do it anymore because it's difficult and not worth it. I switched to javascript as well as server-side javascript for my other projects and I find it incredibly easier to write user apps there. I'm not trying to push on javascript specifically, but could we consider writing spdk_top in a higher level language, where we don't care about memory allocation, json parsing, and printing data to the screen?
Currently spdk_top seems to require the same, detailed review as any spdk patch and honestly I'm a bit reluctant to review it. Moreover, I find one major feature missing there - a view with all pollers on a specific, single thread and busy time % for each poller, so that I can see what my thread is most busy with. I think it would be the first view I check when doing any spdk optimization. Yet I see it's quite a bit of effort to add it to the current spdk_top code, which brings me back to my first question.
D.
11 months
Re: Fail to start cinder service
by Szwed, Maciej
Hi Ranjan,
I happy that this helped to proceed further. The log you pasted below is cut (lines are cut after some number of words) – could you please try to get the rest of the lines as it seems that most important information was there?
Thanks,
Maciek
From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com>
Sent: Tuesday, March 31, 2020 7:42 AM
To: Szwed, Maciej <maciej.szwed(a)intel.com>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: RE: Fail to start cinder service
Thanks a ton for such instant response Szwed. My bad, I just missed this email.
I removed the comments, and at least the service is running. I am yet to check if I can access the volume from horizon. Will update you soon on this.
root@ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack@c-vol.service>
● devstack(a)c-vol.service<mailto:devstack@c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack@c-vol.service>
Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack@c-vol.service>; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-03-31 11:09:03 IST; 23s ago
Main PID: 18375 (cinder-volume)
Tasks: 2 (limit: 6143)
CGroup: /system.slice/system-devstack.slice/devstack(a)c-vol.service<mailto:/system.slice/system-devstack.slice/devstack@c-vol.service>
├─18375 /usr/bin/python3.6 /usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf
└─18421 /usr/bin/python3.6 /usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager raise exception.VolumeBackendAPIException(data=err_msg)
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Could no
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager
Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG cinder.service [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Creating RPC server for service cinder-volume {{(pid=18421) start
Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG oslo_db.sqlalchemy.engines [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_A
Mar 31 11:09:13 ubuntu cinder-volume[18375]: DEBUG cinder.service [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Pinning object versions for RPC server serializer to 1.38 {{(pid=
Mar 31 11:09:13 ubuntu cinder-volume[18375]: INFO cinder.volume.manager [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Initializing RPC dependent components of volume driver SPDK
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.utils [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Volume driver SPDKDriver not initialized
Mar 31 11:09:13 ubuntu cinder-volume[18375]: ERROR cinder.volume.manager [None req-9d2d3102-a466-443b-a606-a43ff7b0b8db None None] Cannot complete RPC initialization because driver isn't in
Mar 31 11:09:23 ubuntu cinder-volume[18375]: ERROR cinder.service [-] Manager for service cinder-volume ubuntu@spdk is reporting problems, not sending heartbeat. Service will appear "down".
Regards,
Ranjan
From: Szwed, Maciej <maciej.szwed(a)intel.com<mailto:maciej.szwed@intel.com>>
Sent: Tuesday, March 24, 2020 7:38 PM
To: Kumar Ranjan <Kumar.Ranjan(a)wdc.com<mailto:Kumar.Ranjan@wdc.com>>
Cc: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk@lists.01.org>>
Subject: RE: Fail to start cinder service
CAUTION: This email originated from outside of Western Digital. Do not click on links or open attachments unless you recognize the sender and know that the content is safe.
Hi Ranjan,
Nice to meet you. I’m adding SPDK mailing list for wider audience (someone may find it helpful someday 😊 ). Maybe it has to do something with additional comments on each line in configuration file – try to remove them. Also in “spdk_rpc_port =4420” you are missing space after “=” – I doubt that this may has something to do with that missing space, but it is always worth trying removing it. Everything else looks good. Let me know if that won’t work – I’ll try to reproduce environment on my side.
Regards,
Maciej
From: Kumar Ranjan <Kumar.Ranjan(a)wdc.com<mailto:Kumar.Ranjan@wdc.com>>
Sent: Monday, March 23, 2020 6:57 PM
To: Szwed, Maciej <maciej.szwed(a)intel.com<mailto:maciej.szwed@intel.com>>
Subject: Fail to start cinder service
Hi Maciej,
I am Ranjan from WDC. Request your help in understanding as why I am not able to start the cinder service.
I saw some of your patches and looks like you are more into this field. Hence pinging you.
For experiment : devstack as well my client is running in the same server.
My host IP for Horizon (private network) : 192.168.1.14
My target IP where rpc is listening : 192.168.1.111
My target has 2 data ports , one is : 192.168.1.111 and another : 172.168.1.111
Spdk entried in my Cinder.conf
[spdk]
spdk_rpc_ip = 192.168.1.111 #address to machine with SPDK
spdk_rpc_port =4420 #port to use to send commands to SPDK once running with -r
#spdk_rpc_username = admin#username set in remote_rpc.py
#spdk_rpc_password = admin#password set in remote_rpc.py
target_ip_address = 172.168.1.111 #NVMe-oF interface
target_port = 4420
target_protocol = nvmet_rdma
target_helper = spdk-nvmeof
target_prefix = nqn.1992-05.com.wdc
volume_driver = cinder.volume.drivers.spdk.SPDKDriver
volume_backend_name = SPDK
---------------
root@ubuntu:~# systemctl status devstack(a)c-vol.service<mailto:devstack@c-vol.service>
● devstack(a)c-vol.service<mailto:devstack@c-vol.service> - Devstack devstack(a)c-vol.service<mailto:devstack@c-vol.service>
Loaded: loaded (/etc/systemd/system/devstack(a)c-vol.service<mailto:/etc/systemd/system/devstack@c-vol.service>; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2020-03-23 23:16:59 IST; 5s ago
Process: 15636 ExecStart=/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf (code=exited, status=1/FAILURE)
Main PID: 15636 (code=exited, status=1/FAILURE)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume return self._conf._get(name, self._group)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2639, in _get
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume value, loc = self._do_get(name, group, namespace)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume File "/usr/local/lib/python3.6/dist-packages/oslo_config/cfg.py", line 2713, in _do_get
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume raise ConfigFileValueError(message)
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume oslo_config.cfg.ConfigFileValueError: Value for option spdk_rpc_port from LocationInfo(location=<Locations.user: (4, Tru
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume
Mar 23 23:16:59 ubuntu cinder-volume[15636]: ERROR cinder.cmd.volume [None req-5dddc9ec-a2a6-4c94-913c-caea68014ac5 None None] No volume service(s) started successfully, terminating.
Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack@c-vol.service>: Main process exited, code=exited, status=1/FAILURE
Mar 23 23:16:59 ubuntu systemd[1]: devstack(a)c-vol.service<mailto:devstack@c-vol.service>: Failed with result 'exit-code'.
Target :
root@KAMSHED:/usr/share/spdk/scripts# ps aux | grep nvmf
root 14676 691 5.0 4271148 409508 pts/0 SLl 01:28 37:21 /usr/sbin/nvmf_tgt -o -j -c /usr/share/spdk/nvmf.conf -r 192.168.1.111:4420
root 15844 0.0 0.0 2316 464 pts/0 S+ 01:33 0:00 grep nvmf
Regards,
Ranjan
11 months
is it possible to add a blacklist for the lvol module?
by peng yu
I have a below scenario:
A disk is controlled by SPDK APP A.
SPDK APP A exports the disk as a NVMEoF target
SPDK APP B is a NVMEoF initiator, it connects to that disk.
SPDk APP B creates a lvs and a lvol on the disk.
When I restart the APP A and APP B. The APP A will examine the disk
and find there is a lvs, and then the lvol module on APP A will claim
this disk, then this disk can't be exported over NVMEoF.
Could we add a api to the lvol module? The api could add a dev name
pattern to a black list. When the lvol module examine a disk, it will
check against the black list, if a dev name matches the black list, it
will ignore this device.
Best regards.
11 months, 1 week