6:34 p.m.
On Mon, Jan 08, 2018 at 11:09:17AM -0700, Jason Gunthorpe wrote:
> As usual we implement what actually has a consumer. On top of
that the
> R/W API is the only core RDMA API that actually does DMA mapping for the
> ULP at the moment.
Well again the same can be said for dma_map_page vs dma_map_sg...
I don't understand this comment.
> For SENDs and everything else dma maps are done by the ULP (I'd like
> to eventually change that, though - e.g. sends through that are
> inline to the workqueue don't need a dma map to start with).
> That's because the initial design was to let the ULPs do the DMA
> mappings, which fundamentally is wrong. I've fixed it for the R/W
> API when adding it, but no one has started work on SENDs and atomics.
Well, you know why it is like this, and it is very complicated to
unwind - the HW driver does not have enough information during CQ
processing to properly do any unmaps, let alone serious error tear
down unmaps, so we'd need a bunch of new APIs developed first, like RW
did. :\
Yes, if it was trivial we would have done it already.
> > And on that topic, does this scheme work with HFI?
>
> No, and I guess we need an opt-out. HFI generally seems to be
> extremely weird.
This series needs some kind of fix so HFI, QIB, rxe, etc don't get
broken, and it shouldn't be 'fixed' at the RDMA level.
I don't think rxe is a problem as it won't show up a pci device.
HFI and QIB do show as PCI devices, and could be used for P2P transfers
from the PCI point of view. It's just that they have a layer of
software indirection between their hardware and what is exposed at
the RDMA layer.
So I very much disagree about where to place that workaround - the
RDMA code is exactly the right place.
> > This is why P2P must fit in to the common DMA framework
somehow, we
> > rely on these abstractions to work properly and fully in RDMA.
>
> Moving P2P up to common RDMA code isn't going to fix this. For that
> we need to stop preting that something that isn't DMA can abuse the
> dma mapping framework, and until then opt them out of behavior that
> assumes actual DMA like P2P.
It could, if we had a DMA op for p2p then the drivers that provide
their own ops can implement it appropriately or not at all.
Eg the correct implementation for rxe to support p2p memory is
probably somewhat straightfoward.
But P2P is _not_ a factor of the dma_ops implementation at all,
it is something that happens behind the dma_map implementation.
Think about what the dma mapping routines do:
(a) translate from host address to bus addresses
and
(b) flush caches (in non-coherent architectures)
Both are obviously not needed for P2P transfers, as they never reach
the host.
Very long term the IOMMUs under the ops will need to care about
this,
so the wrapper is not an optimal place to put it - but I wouldn't
object if it gets it out of RDMA :)
Unless you have an IOMMU on your PCIe switch and not before/inside
the root complex that is not correct.
6:44 p.m.
New subject: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
On 08/01/18 11:34 AM, Christoph Hellwig wrote:
But P2P is _not_ a factor of the dma_ops implementation at all,
it is something that happens behind the dma_map implementation.
Think about what the dma mapping routines do:
(a) translate from host address to bus addresses
and
(b) flush caches (in non-coherent architectures)
Both are obviously not needed for P2P transfers, as they never reach
the host.
Isn't pci_p2pdma_map_sg doing (a)? It's just translating from a host
address to a PCI bus address.
> Very long term the IOMMUs under the ops will need to care about
this,
> so the wrapper is not an optimal place to put it - but I wouldn't
> object if it gets it out of RDMA :)
Unless you have an IOMMU on your PCIe switch and not before/inside
the root complex that is not correct.
Per the ACS discussion, in the future we might want to implement "ACS
Direct Translated P2P" as Alex described. I expect it would be the IOMMU
that needs to set that up. So, I think, we also have the dma_map
implementations also doing something like:
(c) setup/manage any security permissions on mappings
Which P2P may at some point be concerned with.
Logan
6:57 p.m.
New subject: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
On Mon, Jan 08, 2018 at 11:44:41AM -0700, Logan Gunthorpe wrote:
> Think about what the dma mapping routines do:
>
> (a) translate from host address to bus addresses
>
> and
>
> (b) flush caches (in non-coherent architectures)
>
> Both are obviously not needed for P2P transfers, as they never reach
> the host.
Isn't pci_p2pdma_map_sg doing (a)? It's just translating from a host
address to a PCI bus address.
It does, sort of - but in a different way then the normal DMA map
ops. And only to work around the fact that we need to map our
P2P space into struct pages. Without that we could just pass the
bus address around, but the Linux stack and VM isn't anywhere near
ready for something that advanced.
>> Very long term the IOMMUs under the ops will need to care
about this,
>> so the wrapper is not an optimal place to put it - but I wouldn't
>> object if it gets it out of RDMA :)
>
> Unless you have an IOMMU on your PCIe switch and not before/inside
> the root complex that is not correct.
Per the ACS discussion, in the future we might want to implement "ACS
Direct Translated P2P" as Alex described. I expect it would be the IOMMU
that needs to set that up. So, I think, we also have the dma_map
implementations also doing something like:
(c) setup/manage any security permissions on mappings
Which P2P may at some point be concerned with.
Maybe once root complexes with iommus actually support P2P. But until
then we have a lot more more important problems to solve.
7:05 p.m.
New subject: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
On 08/01/18 11:57 AM, Christoph Hellwig wrote:
It does, sort of - but in a different way then the normal DMA map
ops. And only to work around the fact that we need to map our
P2P space into struct pages. Without that we could just pass the
bus address around, but the Linux stack and VM isn't anywhere near
ready for something that advanced.
Yeah, wouldn't that be nice.
Ok, so if we shouldn't touch the dma_map infrastructure how should the
workaround to opt-out HFI and QIB look? I'm not that familiar with the
RDMA code.
Thanks,
Logan
4:47 p.m.
New subject: [PATCH 06/12] IB/core: Add optional PCI P2P flag to rdma_rw_ctx_[init|destroy]()
On Mon, Jan 08, 2018 at 12:05:57PM -0700, Logan Gunthorpe wrote:
Ok, so if we shouldn't touch the dma_map infrastructure how
should the
workaround to opt-out HFI and QIB look? I'm not that familiar with the RDMA
code.
We can add a no_p2p quirk, I'm just not sure what the right place
for it is. As said they device don't really mind P2P at the PCIe
level, it's just that their RDMA implementation is really strange.
So I'd be tempted to do it in RDMA.
1683
days inactive
1684
days old
4 comments
2 participants
participants (2)
-
Christoph Hellwig -
Logan Gunthorpe