Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux

powerpc/powernv/npu: Do not try invalidating 32bit table when 64bit table is enabled

GPUs and the corresponding NVLink bridges get different PEs as they
have separate translation validation entries (TVEs). We put these PEs
to the same IOMMU group so they cannot be passed through separately.
So the iommu_table_group_ops::set_window/unset_window for GPUs do set
tables to the NPU PEs as well which means that iommu_table's list of
attached PEs (iommu_table_group_link) has both GPU and NPU PEs linked.
This list is used for TCE cache invalidation.

The problem is that NPU PE has just a single TVE and can be programmed
to point to 32bit or 64bit windows while GPU PE has two (as any other
PCI device). So we end up having an 32bit iommu_table struct linked to
both PEs even though only the 64bit TCE table cache can be invalidated
on NPU. And a relatively recent skiboot detects this and prints
errors.

This changes GPU's iommu_table_group_ops::set_window/unset_window to
make sure that NPU PE is only linked to the table actually used by the
hardware. If there are two tables used by an IOMMU group, the NPU PE
will use the last programmed one which with the current use scenarios
is expected to be a 64bit one.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Michael Ellerman <mpe@ellerman.id.au>

authored by

Alexey Kardashevskiy and committed by
Michael Ellerman
d41ce7b1 b574df94

+24 -3
+24 -3
arch/powerpc/platforms/powernv/pci-ioda.c
··· 2681 2681 static long pnv_pci_ioda2_npu_set_window(struct iommu_table_group *table_group, 2682 2682 int num, struct iommu_table *tbl) 2683 2683 { 2684 + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group); 2685 + int num2 = (num == 0) ? 1 : 0; 2684 2686 long ret = pnv_pci_ioda2_set_window(table_group, num, tbl); 2685 2687 2686 2688 if (ret) 2687 2689 return ret; 2688 2690 2689 - ret = pnv_npu_set_window(gpe_table_group_to_npe(table_group), num, tbl); 2690 - if (ret) 2691 + if (table_group->tables[num2]) 2692 + pnv_npu_unset_window(npe, num2); 2693 + 2694 + ret = pnv_npu_set_window(npe, num, tbl); 2695 + if (ret) { 2691 2696 pnv_pci_ioda2_unset_window(table_group, num); 2697 + if (table_group->tables[num2]) 2698 + pnv_npu_set_window(npe, num2, 2699 + table_group->tables[num2]); 2700 + } 2692 2701 2693 2702 return ret; 2694 2703 } ··· 2706 2697 struct iommu_table_group *table_group, 2707 2698 int num) 2708 2699 { 2700 + struct pnv_ioda_pe *npe = gpe_table_group_to_npe(table_group); 2701 + int num2 = (num == 0) ? 1 : 0; 2709 2702 long ret = pnv_pci_ioda2_unset_window(table_group, num); 2710 2703 2711 2704 if (ret) 2712 2705 return ret; 2713 2706 2714 - return pnv_npu_unset_window(gpe_table_group_to_npe(table_group), num); 2707 + if (!npe->table_group.tables[num]) 2708 + return 0; 2709 + 2710 + ret = pnv_npu_unset_window(npe, num); 2711 + if (ret) 2712 + return ret; 2713 + 2714 + if (table_group->tables[num2]) 2715 + ret = pnv_npu_set_window(npe, num2, table_group->tables[num2]); 2716 + 2717 + return ret; 2715 2718 } 2716 2719 2717 2720 static void pnv_ioda2_npu_take_ownership(struct iommu_table_group *table_group)