相关的资料显示的都是与bios或者和PCI有关，最有可能是主板的异常：
I tried the command line parameter: i915.i915_enable_rc6=0
but without success. Is it still useful for ivy bridge CPU
Another (rather unlikely, but I‘ll still mention it just in case) reason could be that you‘re using igb_uio
module that is really old (I don‘t remember when igb_uio gained IOMMU support, probably 1.4.x), which is
why I asked about DPDK version, but since you‘re using 1.6.0 you should be fine.
Please test with intel_iommu=igfx_off added to the kernel cmdline. Probably broken dmar tables in your bios, so please also check
for a bios upgrade
Does it work on 3.17-rc1? Are all of the 8169 NICs on bus 05 up and running?
Please provide lspci -vv info for 04:00.0
lspci -vvs 04:00.0
04:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge
(rev 03) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=04, secondary=05, subordinate=05, sec-latency=32
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: f7800000-f78fffff
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v1) PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns,
L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr-
TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
L0 <2us, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk-
DLActive- BWMgmt- ABWMgmt-
Capabilities: [c0] Subsystem: Micro-Star International Co., Ltd. Device
7758
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01
Status: NegoPending- InProgress-
79305f iommu/vt-d: Update to use PCI DMA aliases
e17f9ff iommu/vt-d: Use iommu_group_get_for_dev()
104a1c1 iommu/core: Create central IOMMU group lookup/creation interface
Ok, then it‘s probably not a result of the PCIe-to-PCI bridge since 05:00.0 is
the correct requester ID for all the devices behind the bridge. Unfortunately
that means that the problem may not be fixable. We‘re only seeing reads to a
single address, which may mean the NIC is using that read to synchronize
transaction ordering, ex. using a DMA read to flush a DMA write from the
device. If the NIC driver has visibility of this address, then it could
attempt to do a coherent mapping for the device(s) to avoid the fault. If it
doesn‘t, then these NICs may simply be incompatible with the IOMMU.
Are these 3 separate NICs plugged into PCI slots on the motherboard or is this
a single triple-port card with embedded PCIe-to-PCI bridge?
You might be able to run the IOMMU in passthrough mode with iommu=pt
r8169.use_dac=1, but note the warning in modinfo "use_dac:Enable PCI DAC.
Unsafe on 32 bit PCI slot." Unfortunately if you don‘t enable use_dac, then
intel_iommu will ignore the passthrough option for these devices.
Also note that this problem has nothing to do with Virtualization/KVM.
Drivers/Network or perhaps Drivers/PCI would be a more appropriate
classification
Since you‘re apparently trying to use VT-d on this system for KVM and therefore
presumably device assignment, I‘ll note that you will never be able to
successfully assign the conventional PCI devices separately between guests or
between host and guests. The IOMMU does not have the granularity to create
separate IOMMU domains per PCI slot in this topology. Also, some (all?)
Realtek NICs have some strange backdoors to PCI configuration space that make
them poor targets for PCI device assignment:
I think ASUS iommu errors are all the same issue. Bad BIOSes. I‘ve had a P45M (ASUS laptop) and a new core i5 desktop (ASUS
motherboard) have DMAR errors spewing out from the kernel. Turning off VT-d solves the problem. USB becomes broken on the
desktop with VT-d on. It‘s unfortunate that this issue was not seen on 2.6.30 and lower kernels. Will a work-around be applied to
2.6.31+ kernels or are kernel maintainers going to give us a cold shoulder and say talk to ASUS