Hi Shuhei,
Thank you for your work on this! I will start reviewing these patches today.
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 松本周平 / MATSUMOTO,SHUUHEI
<shuhei.matsumoto.xt(a)hitachi.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Monday, September 25, 2017 at 9:48 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Proposal to Dynamic Reconfiguration of SPDK iSCSI Target
Hi,
I spent much time but I just submitted a first patch series made of 8 patches related with
portal, portal group, and initiator group of iSCSI target.
I hope this can start review process about our proposal and I will appreciate your any
feedback.
Now I’m preparing another patch series about dynamic LUN reconfiguration.
If prepared I will submit these too.
Thank you,
Shuhei Matsumoto
From: 松本周平 / MATSUMOTO,SHUUHEI
Sent: Tuesday, September 19, 2017 8:53 AM
To: Storage Performance Development Kit
Subject: RE: [SPDK] Proposal to Dynamic Reconfiguration of SPDK iSCSI Target
Hi James, John, and All
I added Backlog/In-Progress/Done lists and cards to those lists in Trello.
Next I will split up our patch and push each to GerritHub.
Please wait for a little time and I will inform you if ready.
Thank you,
Shuhei Matsumoto
From: 松本周平 / MATSUMOTO,SHUUHEI
Sent: Saturday, September 16, 2017 6:30 AM
To: Storage Performance Development Kit
Subject: RE: [SPDK] Proposal to Dynamic Reconfiguration of SPDK iSCSI Target
Hi James,
Thank you so much for your great classification.
Almost all are covered and I would like to another item to 2).
e. Mapping of initiator/portal group
an array of PG-IG mappings is not easy to understand
PG-IGs mapping list (PG is head, IGs are elements) is easy to understand for accessibility
control.
I hope this make sense for you.
Thank you,
Shuhei Matsumoto
From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Harris, James R
Sent: Saturday, September 16, 2017 2:45 AM
To: Storage Performance Development Kit
Subject: [!]Re: [SPDK] Proposal to Dynamic Reconfiguration of SPDK iSCSI Target
Hi Matsumoto-san,
We are glad to have you join the community!
You have brought up a lot of very good topics. I’d like to summarize the major topics
here and then maybe we can break this up into separate e-mail threads and/or Trello cards.
I think that will make it easier for all of us to track each topic.
1) Adherence to iSCSI specification for target topology (specifically portals and
portal groups)
2) Runtime portal/portal group configuration (depends on agreement on item 1)
a. RPCs for add/remove portal/portal group
b. Design changes for iscsi_conn<->iscsi_portal relationship
c. spdk_scsi_dev changes needed to support dynamic set of target ports
d. Portal acceptor poller changes
3) iSCSI target node – add/remove LUN
a. RPCs for add/remove scsi_lun from scsi_dev
b. Unit Attention queues
c. LUN0 handling
Please let me know if there’s anything major I’ve missed. I’ll respond to #1 and #3 later
today.
Do you have patches for any/all of these items already? If so, it would be great to get
those patches pushed to GerritHub to help facilitate the discussion.
Thanks,
-Jim
From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 松本周平 / MATSUMOTO,SHUUHEI
<shuhei.matsumoto.xt(a)hitachi.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Friday, September 15, 2017 at 12:48 AM
To: "'spdk(a)lists.01.org'" <spdk(a)lists.01.org>
Subject: [SPDK] Proposal to Dynamic Reconfiguration of SPDK iSCSI Target
Hello,
This is my first post and I’m very glad to join the SPDK community.
We are working on the improvement of SPDK iSCSI target for Hitachi’s new storage
product.
I would like to post a few proposals about flexibility of SPDK iSCSI target.
What we want to do is to support dynamic adding/removing ports and LUNs for iSCSI target
and
by this improvement, we make SPDK iSCSI target have at least as flexible as SPDK
VHOST-SCSI.
This is my first post and I appreciate your any feedback.
Thank you,
Shuhei Matsumoto
What we want to do
==================
Adding/removing portal to/from iscsi target dynamically,
- We make portal-portal_group pair for each portal (each portal group has only
one portal and each portal belongs to only one portal group).
- New JSON-RPC commands, will add/remove portals to/from an existing iSCSI
target.
Adding an LUN (BDEV) to an existing iSCSI target
- VHOST SCSI support this function.
- Comparison between VHOST SCSI and iSCSI target
- VHOST SCSI
- each SCSI controller can have multiple SCSI devices
- each SCSI device can have only one LUN
- we can add a SCSI device with an LUN to an existing SCSI controller
- iSCSI target
- a iSCSI target can have only a SCSI device.
- a SCSI device can have multiple LUNs.
- we cannot add an additional SCSI device to an existing iSCSI target
-> Hence we would like to add an LUN to an existing SCSI device as same as
VHOST-SCSI. We should support the following:
- JSON-RPC command.
- Unit Attention function
- add an LUN to an existing SCSI device.
- callback function to notify Unit Attention at the hot-add event.
Removing an LUN (BDEV) from an existing iSCSI target
- VHOST SCSI support this function.
- Comparison between VHOST SCSI and iSCSI target
- VHOST SCSI
- Removing a BDEV will remove an LUN binded to the BDEV from an existing
SCSI device. Empty SCSI device will be remained.
- Removing a SCSI device will remove an LUN and its binded BDEV from
the removed SCSI device.
- Notify the hot-remove event to the host by the VIRTIO event through
the registration of the callback function at the SCSI device creation.
- iSCSI target
- Removing a BDEV will leave an LUN binded to the BDEV from an existing
SCSI device as same as VHOST SCSI.
- This hot-remove event is not notified to the host due to lack of
Unit Attention function.
-> Hence we would like to support the following:
- Unit Attention function
- callback function to notify Unit Attention at the hot-remove event.
Design
======
1. iSCSI Portal
1.1 iscsi_portal_acceptor poller for each portal (currently one global
iscsl_portal_acceptor poller for all portals)
One iscsi_portal_acceptor poller walks all portal_group_list and portal_list is
not safe when adding/removing portal and portal_groups are done by dynamically.
Currently each portal can be registered into only one portal_group. Hence one
iscsi_portal_acceptor for each portal is correct.
1.2 Synchronous removing the iscsi_portal_acceptor poller
Currently removing the iscsi_portal_acceptor does not wait completion.
When the requester runs on the different core from the to-be-stopped
iscsi_portal_acceptor, the requester will wait completion by using
event_call+semaphore+timer.
To achieve this, the lcore variable is added to the spdk_iscsi_conn structure.
Without spdk_iscsi_conn->lcore, the requester cannot know where the
iscsl_portal_acceptor runs. We referred the vhost_scsi code about
event_call+semaphore+timer.
1.3 Decouple spdk_iscsi_conn and spdk_iscsi_portal to remove a portal from the
active connections.
Currently spdk_iscsi_conn refers obsolete spdk_iscsi_portal after
spdk_iscsi_portal removal. We copied member data of the iscsi_portal to the
iscsi_conn during login processing, and after login completion, iscsi_conn
refer not iscsi_portal but copied data of iscsi_portal.
About iSCSI target, we do not change as it was. iSCSI target can be removed
dynamically if no active connection.
1.4 Support dynamic adding/removing target ports to/from an existing iSCSI/SCSI
target.
Current spdk_scsi_dev has an array of target ports. However, this array
supports only adding target ports and has some bugs.
2. Internal Data Structure
2.1 PG-IG Mapping (PG: Portal Group, IG: Initiator Group)
Each target node has PG-IG mappings.
Current PG-IG mappings is an array and not flexible.
We do not change parameter of JSON-RPC but change the internal data
structure of PG-IG mappings.
Current:
target -- (1..n)----> PG_map (1..1) --> PG
|
--> IG_map (1..1) --> IG
Our proposal
target -- (1..n) --> PG_map --- (1..1) --> PG
|
-- (1..m) --> IG_map -- (1..1) --> IG
Our proposal will make the accessibility control of PG and IG comprehensible.
We keep the upper limit of number of PG-IG mappings.
PG_map must have at least one IG_map, and if all IG_maps of a PG_map are
removed, the PG_map will be also removed from the target.
2.2 Global iSCSI Portal List
Currently each portal can be registered into only one portal group. However
duplicated registration of a portal can be detected at the failure of socket
open system call of the portal.
By adding global portal list, we can detect duplicated registration without
system call failure.
3. Dynamic Adding/Removing LUNs to/from an existing iSCSI/SCSI target
3.1 UA queue for each I_T (or I_T_L) nexus (UA: Unit Attention)
Each SCSI device must have a queue of UA for each I_T (or I_T_L) nexus.
Currently the appropriate data structure for I_T nexus is the
spdk_iscsi_session. Hence we add a UA queue for each I_T_L nexus into
the spdk_iscsi_session.
3.2 Lockless concurrency control by event based callback function
Currently all connections of an iSCSI target will run on the same CPU.
by using SPDK event_call and callback function, we will implement Unit
Attention function without using locks.
3.3 How to add an LUN when LUN0 is not active
To make the initiator detect newly added LUNs, the iSCSI target must have
the active LUN0.
When LUN1, LUN2, and LUN3 are acitive and LUN0 is not active,
if you add an LUN as LUN4 the iSCSI target will not run correctly.
When LUN1, LUN2, and LUN3 are active and LUN0 is not active,
adding an LUN as LUN0 is required for the iSCSI target to run correctly.
Hence we will handle the LUN ID parameter as optional of the ADD LUN command.
If the LUN ID parameter is omitted, the new LUN will have the smallest LUN ID
of unused IDs.
4. JSON-RPC for iSCSI target
4.1 construct_target_node
By our change, this command can skip registration of PG-IG mappings and LUNs
at target creation, and later we can add/remove PG-IG mappings and LUNs.
4.2 add_pg_ig_mappings_to_target_node
This new command supports adding the specified PG-IG mappings to the existing
target.
4.3 remove_pg_ig_mappings_from_target_node
This new command supports removing the specified PG-IG mappings from the
existing target.
4.4 clear_pg_ig_mappings_from_target_node
This new command supports removing all PG-IG mappings from the existing target.
4.5 add_lun_to_target_node
This new command supports adding an LUN to an existing target.
When a LUN is added, the target does not immediately notify the host. When an
I/O is sent to an existing LUN, the I/O will get a “check condition” status.
When the request sense is issued the sense data will indicate that the
"REPORT LUNS DATA HAS CHANGED".
To add/remove LUNs to/from an existing target correctly, LUN0 is special and
if we remove LUN0, LUN0 has to be assigned to the next added LUN.
To satisfy this requirement, LUN ID parameter of the ADD LUN command is not
mandatory but optional, and if it is omitted, the smallest free LUN ID will be
assigned to the newly added LUN.
4.6 remove_luns_from_target_node
This new command supports removing LUNs from the existing target.
5. Bug fix of python code (rpc.py) for JSON-RPC
Due to erroneous sorting, we may not be able to construct the iSCSI target node
with multiple LUNs when cofigured by rpc.py. Sorting should be done by LUN ID
but currently is done by LUN name. Hence we observed this unintentional failure.