Why would a switch not identify that as a peer address? We use the
together with ATS to identify the address space which a transaction
I think you are conflating two types of TLPs here. If the device supports ATS then it will
issue a TR TLP to obtain a translated address from the IOMMU. This TR TLP will be
addressed to the RP and so regardless of ACS it is going up to the Root Port. When it gets
the response it gets the physical address and can use that with the TA bit set for the
p2pdma. In the case of ATS support we also have more control over ACS as we can disable it
just for TA addresses (as per 22.214.171.124.2 of the spec).
If I'm not completely mistaken when you disable ACS it is
possible that a bridge identifies a transaction as belonging to a peer
address, which isn't what we want here.
You are right here and I think this illustrates a problem for using the IOMMU at all when
P2PDMA devices do not support ATS. Let me explain:
If we want to do a P2PDMA and the DMA device does not support ATS then I think we have to
disable the IOMMU (something Mike suggested earlier). The reason is that since ATS is not
an option the EP must initiate the DMA using the addresses passed down to it. If the IOMMU
is on then this is an IOVA that could (with some non-zero probability) point to an IO
Memory address in the same PCI domain. So if we disable ACS we are in trouble as we might
MemWr to the wrong place but if we enable ACS we lose much of the benefit of P2PDMA.
Disabling the IOMMU removes the IOVA risk and ironically also resolves the IOMMU grouping
So I think if we want to support performant P2PDMA for devices that don't have ATS
(and no NVMe SSDs today support ATS) then we have to disable the IOMMU. I know this is
problematic for AMDs use case so perhaps we also need to consider a mode for P2PDMA for
devices that DO support ATS where we can enable the IOMMU (but in this case EPs without
ATS cannot participate as P2PDMA DMA iniators).