DW PCIE Linux驱动整理
1.几个基本概念
TLP(Transaction Layer Packet): TLP包是由PCIe的Endpoint或者Root Complex发送的数据包。在PCIe体系中的事务层生成;TLP包由头（Hander）、数据（Data）、ECRC（校验）几个部分组成。TLP是用户程序和PCIe设备交互的唯一渠道.。
CDM:Configuration Dependent Module.This is an internal block in the native controller that has the PCIe configuration registers and some user-accessible registers.包括 PCIE配置寄存器和 厂家定义的寄存器（比如ATU和DMA等）。
ELBI:External Local Bus Interface.Delivers an inbound register RD/WR received by the controller to external application registers when the controller is expected to generate the PCIe completion of this register
RD/WR. For switch applications: the ELBI is intended for incoming requests that are targeted to local switch application registers, while TRGT1 is intended for TLPs that are passing through the switch. The controller automatically generates completions for requests that are routed to the ELBI。ELBI可以理解为通过PCIE访问一个CPU的内部空间寄存器。ELBI 主要是在EP端，使用RC没有该部分功能。
LBC:Local Bus Controller .This is an internal block that allows the DBI interface (from your application side), or the wire side interface (through the TRGT0 interface), to access the CDM or your external application registers on the ELBI.本地的寄存器控制，比如RC中断清除等相关都需要操作。
XALI:Transmit Client Interface.理解为给外部传输的接口
DBI:Data Bus Interface.You can use this interface to locally access the controller’s internal registers in the CDM, or your external application registers on the ELBI. You can optionally connect a local CPU or controller to this port.
ATU:Address Translation Unit ATU是一个地址转换单元，负责将一段存储器域的地址转换到PCIe总线域地址。
TLP中的地址哪里来？ATU转换过来的。ATU是一个地址转换单元，负责将一段存储器域的地址转换到PCIe总线域地址，除了地址转换外，还能提供访问类型等信息，这些信息都是ATU根据总线上的信号自己做的，数据都打包到TLP中，不用软件参与。
软件需要做的是配置ATU，所以如果ATU配置完成，并且能正常工作，那么CPU访问PCIe空间就和访问本地存储器空间方法是一样的，只要读写即可。
iATU mapping directions:
iATU can do inbound and outbound mapping.
Inbound mapping is PCI address to internal address.
Outbound mapping is internal address to PCI address.
iATU mapping modes:
On device, iATU supports two mapping modes, address match mode and BAR match mode.
For address match mode:
PCI address ------mapping------- internal address
For BAR match mode:
BAR number ------mapping------ internal address
Generally speaking, if your card has SoC, the FW on the SoC will configure the iATU mapping with BAR match mode. And don't let host side driver to configure it.
inbound/outbound
Inbound:PCI域訪问存储器域
Outbound:存储器域訪问PCI域
RC訪问EP: RC存储器域->outbound->RC PCI域->EP PCI域->inbound->EP存储器域
EP訪问RC：EP存储器域->outbound->EP PCI域->RC PCI域->inbound->RC存储器域
Out即出去，发起訪问的一側，须要进行outbound，去訪问对端
In即进来，被訪问的一側，须要进行inbound，使得对端能够訪问
EP訪问RC演示样例（蓝色箭头）：
（1）首先，EP须要配置outbound，RC须要inbound(一般RC端不用配)，这样就建立了EP端0x20000000到RC端0x50000000的映射
（2）在RC端改动0x50000000的内容，EP端能够看到对应的变化。从EP端读/写0x20000000和从RC端读/写0x50000000，结果是一样的
RC訪问EP演示样例（黑色箭头）：
（1）首先，RC端须要配置outbound(一般内核中配好)，EP端须要inbound(0x5b000000 inbound到BAR2)，这样就建立了RC端0x20100000（BAR2）到EP端0x5b000000的映射
（2）在EP端改动0x5b000000内存的内容，在RC端0x20100000能够看到对应的变化，从RC端读/写0x20100000和从EP端读/写0x5b000000，结果是一样的
2.DWC PCIE Register
根据文档<DWC_pcie_ctl_rc_databook.pdf>、<DWC_pcie_ctl_rc_registers.pdf>,寄存器的分布图如下所示：
DBI内部划分了2个部分，DBI和DBI2通过片选（CS2）来进行选择。
When you are using an AXI DBI slave, shadow registers can only be accessed through the DBI where you select between the two registers using the CS2 address bit.This is called DBI2, CS2, dbi_cs2, DBI_CS2, or Dbi2 access; all of these terms mean the same thing.
上图左边是DBI访问空间内容， 右边是DBI2访问的空间内容。DBI2可以访问特殊的Shadow寄存器（IP厂家定制寄存器）。
使用AXI方式访问Slave DBI有两种方式，分别为专用方式（AXI Dedicated DBI Slave Access）和共享方式(AXI Shared DBI Slave Access)。
其中，专用方式可以直接通过DBI Slave通道访问DBI总线，进而访问CDM空间：
共享方式没有专用的DBI通道，，是和XALI访问共享一个通道（CPU作RC主模式时对内部的LBC控制寄存器的访问和对外部EP设备的寄存器访问使用的是同一套总线，并没有使用专用的DBI总线），两者通过一个Switch开关进行切换；
当配置开关slv_armisc_info[21]和slv_awmisc_info[21] =1时，开关切换到DBI通道，此时可以访问到CDM空间；否则将通过outbound方式访问XALI空间。
具体实现时，可以通过设置PCIE的SIDEBAND的2个寄存器slv_armisc_info/slv_awmisc_info来切换，保证从AXI接口访问访问到LBC。
3.CDM空间-配置寄存器
<DWC_pcie_ctl_rc_registers.pdf>文档描述了CDM空间的寄存器分布，如下图：
第一章DWC_PCIE_DSP/PF0_TYPE1_HDR Registers对应了PCIE协议中的配置空间寄存器：
第二章开始文档开始描述CAP 寄存器，即对应功能的Capability（能力）说明，对应的寻址方式都是B+0x**：
那么每个CAP结构的基地址B从哪里获得？
PCI-X 和PCIe 总线规范要求其设备必须支持Capabilities 结构。在PCI 总线的基本配置空间中，包含一个Capabilities Pointer 寄存器，该寄存器存放Capabilities 结构链表的头指针。在一个PCIe 设备中，可能含有多个Capability 结构，这些寄存器组成一个链表，如下图所示：
其中每一个Capability 结构都有唯一的ID 号，每一个Capability 寄存器都有一个指针，这个指针指向下一个Capability 结构，从而组成一个单向链表结构，这个链表的最后一个Capability 结构的指针为0。链表开始的指针地址为0x34处的1byte数值，以以imx6q为例，寻址过程如下：
1FF_C034 :00000040 所属内容 PF0_TYPE1_HDR :NEXT=0X40
1FF_C040: 5bc35001 所属内容 PF0_PM_CAP ID=0X01 NEXT=0X50
1FF_C050 :038a7005 所属内容 PCI_MSI_CAP_ID_NEXT_CTRL ID=0X05 NEXT=0X70
1FF_C070 :0042d010 所属内容 PF0_PCIE_CAP ID=0X10 NEXT=0Xd0
1FF_C0D0 :00000003 所属内容 PF0_MSI_CAP ID=0X3 NEXT=0X0 (最后一个CAP)
注意指针寄存器bit[15:8]对应偏移地址，[7:0]对应CAP ID。偏移地址是绝对地址，不是从本身CAP计算的偏移地址。
4.CDM空间-Port Logic 寄存器
从上文图表中可以得出，Port Logic寄存器主要包括了iATU寄存器和DMA寄存器，其偏移地址固定为0x700。
根据文档描述，对ATU寄存器的访问，由于IP版本的兼容性问题，可以有两种方式。
第一种方式为通过iATU_VIEWPORT_OFF寄存器进行间接访问：
实现方式是首先配置 0x700+0x200地址的 (iATU_VIEWPORT_OFF)寄存器，表示设置第N个iATU以及ATU的方式（inbound/outbound）；然后设置iATU Lower Base/Upper Base/Ctrl1/Ctrl2 Register等实际配置：
#define PCIE_ATU_VIEWPORT 0x900 #define PCIE_ATU_REGION_INBOUND (0x1 << 31) #define PCIE_ATU_REGION_OUTBOUND (0x0 << 31) #define PCIE_ATU_REGION_INDEX2 (0x2 << 0) #define PCIE_ATU_REGION_INDEX1 (0x1 << 0) #define PCIE_ATU_REGION_INDEX0 (0x0 << 0) #define PCIE_ATU_CR1 0x904 #define PCIE_ATU_TYPE_MEM (0x0 << 0) #define PCIE_ATU_TYPE_IO (0x2 << 0) #define PCIE_ATU_TYPE_CFG0 (0x4 << 0) #define PCIE_ATU_TYPE_CFG1 (0x5 << 0) #define PCIE_ATU_CR2 0x908 #define PCIE_ATU_ENABLE (0x1 << 31) #define PCIE_ATU_BAR_MODE_ENABLE (0x1 << 30) #define PCIE_ATU_LOWER_BASE 0x90C #define PCIE_ATU_UPPER_BASE 0x910 #define PCIE_ATU_LIMIT 0x914 #define PCIE_ATU_LOWER_TARGET 0x918 #define PCIE_ATU_BUS(x) (((x) & 0xff) << 24) #define PCIE_ATU_DEV(x) (((x) & 0x1f) << 19) #define PCIE_ATU_FUNC(x) (((x) & 0x7) << 16) #define PCIE_ATU_UPPER_TARGET 0x91C void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type, u64 cpu_addr, u64 pci_addr, u32 size) { u32 retries, val; if (pci->ops->cpu_addr_fixup) cpu_addr = pci->ops->cpu_addr_fixup(cpu_addr); if (pci->iatu_unroll_enabled) { dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr, pci_addr, size); return; } dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT, PCIE_ATU_REGION_OUTBOUND | index); dw_pcie_writel_dbi(pci, PCIE_ATU_LOWER_BASE, lower_32_bits(cpu_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_BASE, upper_32_bits(cpu_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_LIMIT, lower_32_bits(cpu_addr + size - 1)); dw_pcie_writel_dbi(pci, PCIE_ATU_LOWER_TARGET, lower_32_bits(pci_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET, upper_32_bits(pci_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type); dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE); /* * Make sure ATU enable takes effect before any subsequent config * and I/O accesses. */ for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) { val = dw_pcie_readl_dbi(pci, PCIE_ATU_CR2); if (val & PCIE_ATU_ENABLE) return; usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX); } dev_err(pci->dev, "outbound iATU is not being enabled\n"); }
第二种方式为直接访问，也被称为Unroll方式：
直接访问方式对于iATU寄存器，其地址为DBI基地址+0x300000+Offset, 来源下图：
例如，通过直接方式访问iATU 第N个Outbound iATU Region Control 1 Register.为：DBI基地址+0x300000+0x200*N。具体实现如下:
/* * iATU Unroll-specific register definitions * From 4.80 core version the address translation will be made by unroll */ #define PCIE_ATU_UNR_REGION_CTRL1 0x00 #define PCIE_ATU_UNR_REGION_CTRL2 0x04 #define PCIE_ATU_UNR_LOWER_BASE 0x08 #define PCIE_ATU_UNR_UPPER_BASE 0x0C #define PCIE_ATU_UNR_LIMIT 0x10 #define PCIE_ATU_UNR_LOWER_TARGET 0x14 #define PCIE_ATU_UNR_UPPER_TARGET 0x18 /* Register address builder */ #define PCIE_GET_ATU_OUTB_UNR_REG_OFFSET(region) \ ((0x3 << 20) | ((region) << 9)) #define PCIE_GET_ATU_INB_UNR_REG_OFFSET(region) \ ((0x3 << 20) | ((region) << 9) | (0x1 << 8)) static void dw_pcie_prog_outbound_atu_unroll(struct dw_pcie *pci, int index, int type, u64 cpu_addr, u64 pci_addr, u32 size) { u32 retries, val; dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_LOWER_BASE, lower_32_bits(cpu_addr)); dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_BASE, upper_32_bits(cpu_addr)); dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_LIMIT, lower_32_bits(cpu_addr + size - 1)); dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_LOWER_TARGET, lower_32_bits(pci_addr)); dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_UPPER_TARGET, upper_32_bits(pci_addr)); dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL1, type); dw_pcie_writel_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2, PCIE_ATU_ENABLE); /* * Make sure ATU enable takes effect before any subsequent config * and I/O accesses. */ for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) { val = dw_pcie_readl_ob_unroll(pci, index, PCIE_ATU_UNR_REGION_CTRL2); if (val & PCIE_ATU_ENABLE) return; usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX); } dev_err(pci->dev, "outbound iATU is not being enabled\n"); } static void dw_pcie_writel_ob_unroll(struct dw_pcie *pci, u32 index, u32 reg, u32 val) { u32 offset = PCIE_GET_ATU_OUTB_UNR_REG_OFFSET(index); dw_pcie_writel_dbi(pci, offset + reg, val); }
5. DTS的描述
以imx6q为例，该SOC的DTS中对PCIE控制器的描述（对应dts文件：linux-4.14.75/arch/arm/boot/dts/imx6qd.dtsi）
pcie: pcie@1ffc000 { compatible = "fsl,imx6q-pcie", "snps,dw-pcie"; reg = <0x01ffc000 0x04000>, //PCIE控制器基地址 <0x01f00000 0x80000>; //PCIE配置空间基地址 reg-names = "dbi", "config"; #address-cells = <3>; #size-cells = <2>; device_type = "pci"; bus-range = <0x00 0xff>; ranges = <0x81000000 0 0 0x01f80000 0 0x00010000 /* downstream I/O */ //BAR空间地址 0x82000000 0 0x01000000 0x01000000 0 0x00f00000>; /* non-prefetchable memory */ //BAR空间地址 num-lanes = <1>; interrupts = <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>; interrupt-names = "msi"; #interrupt-cells = <1>; interrupt-map-mask = <0 0 0 0x7>; interrupt-map = <0 0 0 1 &gpc GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>, <0 0 0 2 &gpc GIC_SPI 122 IRQ_TYPE_LEVEL_HIGH>, <0 0 0 3 &gpc GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>, <0 0 0 4 &gpc GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>; clocks = <&clks IMX6QDL_CLK_PCIE_AXI>, <&clks IMX6QDL_CLK_LVDS1_GATE>, <&clks IMX6QDL_CLK_PCIE_REF_125M>; clock-names = "pcie", "pcie_bus", "pcie_phy"; status = "disabled"; };
对应的寄存器手册：
可以看到，0x1FFC0000是处理器中PCIE控制器的基地址，对应了DTS中<reg-names = "dbi">表项。
6.Linux驱动解析
(1)平台驱动的匹配
static const struct of_device_id imx6_pcie_of_match[] = { { .compatible = "fsl,imx6q-pcie", .data = (void *)IMX6Q, }, { .compatible = "fsl,imx6sx-pcie", .data = (void *)IMX6SX, }, { .compatible = "fsl,imx6qp-pcie", .data = (void *)IMX6QP, }, { .compatible = "fsl,imx7d-pcie", .data = (void *)IMX7D, }, {}, }; static struct platform_driver imx6_pcie_driver = { .driver = { .name = "imx6q-pcie", .of_match_table = imx6_pcie_of_match, .suppress_bind_attrs = true, }, .probe = imx6_pcie_probe, .shutdown = imx6_pcie_shutdown, }; static int __init imx6_pcie_init(void) { /* * Since probe() can be deferred we need to make sure that * hook_fault_code is not called after __init memory is freed * by kernel and since imx6q_pcie_abort_handler() is a no-op, * we can install the handler here without risking it * accessing some uninitialized driver state. */ hook_fault_code(8, imx6q_pcie_abort_handler, SIGBUS, 0, "external abort on non-linefetch"); return platform_driver_register(&imx6_pcie_driver); } device_initcall(imx6_pcie_init);
imx6_pcie_of_match匹配表中的"fsl,imx6q-pcie"与dts吻合，即可进入probe函数。
(2)probe函数
linux-4.14.75/drivers/pci/dwc/pci-imx6.c
static int imx6_pcie_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct dw_pcie *pci; struct imx6_pcie *imx6_pcie; struct resource *dbi_base; struct device_node *node = dev->of_node; int ret; imx6_pcie = devm_kzalloc(dev, sizeof(*imx6_pcie), GFP_KERNEL); if (!imx6_pcie) return -ENOMEM; pci = devm_kzalloc(dev, sizeof(*pci), GFP_KERNEL); //分配PCI数据结构 if (!pci) return -ENOMEM; pci->dev = dev; pci->ops = &dw_pcie_ops; //指向DW通用OPS操作接口 imx6_pcie->pci = pci; imx6_pcie->variant = (enum imx6_pcie_variants)of_device_get_match_data(dev); dbi_base = platform_get_resource(pdev, IORESOURCE_MEM, 0); //获取设备树节点中的第0个reg属性，即0x01ffc000，对应DBI物理基地址 pci->dbi_base = devm_ioremap_resource(dev, dbi_base); //将DBI物理基地址通过ioremap映射为虚拟地址，记入pci->dbi_base变量 if (IS_ERR(pci->dbi_base)) return PTR_ERR(pci->dbi_base); /* Fetch GPIOs */ imx6_pcie->reset_gpio = of_get_named_gpio(node, "reset-gpio", 0); imx6_pcie->gpio_active_high = of_property_read_bool(node, "reset-gpio-active-high"); if (gpio_is_valid(imx6_pcie->reset_gpio)) { ret = devm_gpio_request_one(dev, imx6_pcie->reset_gpio, imx6_pcie->gpio_active_high ? GPIOF_OUT_INIT_HIGH : GPIOF_OUT_INIT_LOW, "PCIe reset"); if (ret) { dev_err(dev, "unable to get reset gpio\n"); return ret; } } else if (imx6_pcie->reset_gpio == -EPROBE_DEFER) { return imx6_pcie->reset_gpio; } /* Fetch clocks */ imx6_pcie->pcie_phy = devm_clk_get(dev, "pcie_phy"); if (IS_ERR(imx6_pcie->pcie_phy)) { dev_err(dev, "pcie_phy clock source missing or invalid\n"); return PTR_ERR(imx6_pcie->pcie_phy); } imx6_pcie->pcie_bus = devm_clk_get(dev, "pcie_bus"); if (IS_ERR(imx6_pcie->pcie_bus)) { dev_err(dev, "pcie_bus clock source missing or invalid\n"); return PTR_ERR(imx6_pcie->pcie_bus); } imx6_pcie->pcie = devm_clk_get(dev, "pcie"); if (IS_ERR(imx6_pcie->pcie)) { dev_err(dev, "pcie clock source missing or invalid\n"); return PTR_ERR(imx6_pcie->pcie); } switch (imx6_pcie->variant) { case IMX6SX: imx6_pcie->pcie_inbound_axi = devm_clk_get(dev, "pcie_inbound_axi"); if (IS_ERR(imx6_pcie->pcie_inbound_axi)) { dev_err(dev, "pcie_inbound_axi clock missing or invalid\n"); return PTR_ERR(imx6_pcie->pcie_inbound_axi); } break; case IMX7D: imx6_pcie->pciephy_reset = devm_reset_control_get_exclusive(dev, "pciephy"); if (IS_ERR(imx6_pcie->pciephy_reset)) { dev_err(dev, "Failed to get PCIEPHY reset control\n"); return PTR_ERR(imx6_pcie->pciephy_reset); } imx6_pcie->apps_reset = devm_reset_control_get_exclusive(dev, "apps"); if (IS_ERR(imx6_pcie->apps_reset)) { dev_err(dev, "Failed to get PCIE APPS reset control\n"); return PTR_ERR(imx6_pcie->apps_reset); } break; default: break; } /* Grab GPR config register range */ imx6_pcie->iomuxc_gpr = syscon_regmap_lookup_by_compatible("fsl,imx6q-iomuxc-gpr"); if (IS_ERR(imx6_pcie->iomuxc_gpr)) { dev_err(dev, "unable to find iomuxc registers\n"); return PTR_ERR(imx6_pcie->iomuxc_gpr); } /* Grab PCIe PHY Tx Settings */ if (of_property_read_u32(node, "fsl,tx-deemph-gen1", &imx6_pcie->tx_deemph_gen1)) imx6_pcie->tx_deemph_gen1 = 0; if (of_property_read_u32(node, "fsl,tx-deemph-gen2-3p5db", &imx6_pcie->tx_deemph_gen2_3p5db)) imx6_pcie->tx_deemph_gen2_3p5db = 0; if (of_property_read_u32(node, "fsl,tx-deemph-gen2-6db", &imx6_pcie->tx_deemph_gen2_6db)) imx6_pcie->tx_deemph_gen2_6db = 20; if (of_property_read_u32(node, "fsl,tx-swing-full", &imx6_pcie->tx_swing_full)) imx6_pcie->tx_swing_full = 127; if (of_property_read_u32(node, "fsl,tx-swing-low", &imx6_pcie->tx_swing_low)) imx6_pcie->tx_swing_low = 127; /* Limit link speed */ ret = of_property_read_u32(node, "fsl,max-link-speed", &imx6_pcie->link_gen); if (ret) imx6_pcie->link_gen = 1; imx6_pcie->vpcie = devm_regulator_get_optional(&pdev->dev, "vpcie"); if (IS_ERR(imx6_pcie->vpcie)) { if (PTR_ERR(imx6_pcie->vpcie) == -EPROBE_DEFER) return -EPROBE_DEFER; imx6_pcie->vpcie = NULL; } platform_set_drvdata(pdev, imx6_pcie); ret = imx6_add_pcie_port(imx6_pcie, pdev); //下面展开 if (ret < 0) return ret; return 0; }
static int imx6_add_pcie_port(struct imx6_pcie *imx6_pcie, struct platform_device *pdev) { struct dw_pcie *pci = imx6_pcie->pci; struct pcie_port *pp = &pci->pp; struct device *dev = &pdev->dev; int ret; if (IS_ENABLED(CONFIG_PCI_MSI)) { pp->msi_irq = platform_get_irq_byname(pdev, "msi"); //获取中断信息 if (pp->msi_irq <= 0) { dev_err(dev, "failed to get MSI irq\n"); return -ENODEV; } ret = devm_request_irq(dev, pp->msi_irq, imx6_pcie_msi_handler, IRQF_SHARED | IRQF_NO_THREAD, "mx6-pcie-msi", imx6_pcie); if (ret) { dev_err(dev, "failed to request MSI irq\n"); return ret; } } pp->root_bus_nr = -1; pp->ops = &imx6_pcie_host_ops; ret = dw_pcie_host_init(pp); //调用DW通用host初始化接口，下面展开 if (ret) { dev_err(dev, "failed to initialize host\n"); return ret; } return 0; }
linux-4.14.75/drivers/pci/dwc/pcie-designware-host.c
int dw_pcie_host_init(struct pcie_port *pp) { struct dw_pcie *pci = to_dw_pcie_from_pp(pp); struct device *dev = pci->dev; struct device_node *np = dev->of_node; struct platform_device *pdev = to_platform_device(dev); struct pci_bus *bus, *child; struct pci_host_bridge *bridge; struct resource *cfg_res; int i, ret; struct resource_entry *win, *tmp; cfg_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config"); //获取设备树节点中的config reg属性，即0x01f00000，对应cfg基地址，即配置空间基地址 if (cfg_res) { pp->cfg0_size = resource_size(cfg_res) / 2; //将cfg空间一分为二 pp->cfg1_size = resource_size(cfg_res) / 2; pp->cfg0_base = cfg_res->start; //cfg0_base 对应 0x01f00000 pp->cfg1_base = cfg_res->start + pp->cfg0_size; //cfg1_base 对应 0x01f00000 + 0x80000/2 即 0x01f40000 } else if (!pp->va_cfg0_base) { dev_err(dev, "missing *config* reg space\n"); } bridge = pci_alloc_host_bridge(0); if (!bridge) return -ENOMEM; ret = of_pci_get_host_bridge_resources(np, 0, 0xff, &bridge->windows, &pp->io_base); if (ret) return ret; ret = devm_request_pci_bus_resources(dev, &bridge->windows); if (ret) goto error; /* Get the I/O and memory ranges from DT */ resource_list_for_each_entry_safe(win, tmp, &bridge->windows) { switch (resource_type(win->res)) { case IORESOURCE_IO: ret = pci_remap_iospace(win->res, pp->io_base); if (ret) { dev_warn(dev, "error %d: failed to map resource %pR\n", ret, win->res); resource_list_destroy_entry(win); } else { pp->io = win->res; pp->io->name = "I/O"; pp->io_size = resource_size(pp->io); pp->io_bus_addr = pp->io->start - win->offset; } break; case IORESOURCE_MEM: pp->mem = win->res; pp->mem->name = "MEM"; pp->mem_size = resource_size(pp->mem); pp->mem_bus_addr = pp->mem->start - win->offset; break; case 0: pp->cfg = win->res; pp->cfg0_size = resource_size(pp->cfg) / 2; pp->cfg1_size = resource_size(pp->cfg) / 2; pp->cfg0_base = pp->cfg->start; pp->cfg1_base = pp->cfg->start + pp->cfg0_size; break; case IORESOURCE_BUS: pp->busn = win->res; break; } } if (!pci->dbi_base) { pci->dbi_base = devm_pci_remap_cfgspace(dev, //如果dbi_base地址没有被映射，重新进行映射 pp->cfg->start, resource_size(pp->cfg)); if (!pci->dbi_base) { dev_err(dev, "error with ioremap\n"); ret = -ENOMEM; goto error; } } pp->mem_base = pp->mem->start; if (!pp->va_cfg0_base) { pp->va_cfg0_base = devm_pci_remap_cfgspace(dev, //将cfg0物理基地址通过ioremap映射为虚拟地址，记入pp->va_cfg0_base变量 pp->cfg0_base, pp->cfg0_size); if (!pp->va_cfg0_base) { dev_err(dev, "error with ioremap in function\n"); ret = -ENOMEM; goto error; } } if (!pp->va_cfg1_base) { pp->va_cfg1_base = devm_pci_remap_cfgspace(dev, //将cfg0物理基地址通过ioremap映射为虚拟地址，记入pp->va_cfg1_base变量 pp->cfg1_base, pp->cfg1_size); if (!pp->va_cfg1_base) { dev_err(dev, "error with ioremap\n"); ret = -ENOMEM; goto error; } } ret = of_property_read_u32(np, "num-viewport", &pci->num_viewport); if (ret) pci->num_viewport = 2; if (IS_ENABLED(CONFIG_PCI_MSI)) { if (!pp->ops->msi_host_init) { pp->irq_domain = irq_domain_add_linear(dev->of_node, MAX_MSI_IRQS, &msi_domain_ops, &dw_pcie_msi_chip); if (!pp->irq_domain) { dev_err(dev, "irq domain init failed\n"); ret = -ENXIO; goto error; } for (i = 0; i < MAX_MSI_IRQS; i++) irq_create_mapping(pp->irq_domain, i); } else { ret = pp->ops->msi_host_init(pp, &dw_pcie_msi_chip); if (ret < 0) goto error; } } if (pp->ops->host_init) { ret = pp->ops->host_init(pp); if (ret) goto error; } pp->root_bus_nr = pp->busn->start; bridge->dev.parent = dev; bridge->sysdata = pp; bridge->busnr = pp->root_bus_nr; bridge->ops = &dw_pcie_ops; bridge->map_irq = of_irq_parse_and_map_pci; bridge->swizzle_irq = pci_common_swizzle; if (IS_ENABLED(CONFIG_PCI_MSI)) { bridge->msi = &dw_pcie_msi_chip; dw_pcie_msi_chip.dev = dev; } ret = pci_scan_root_bus_bridge(bridge); //开始扫描PCI总线 if (ret) goto error; bus = bridge->bus; if (pp->ops->scan_bus) pp->ops->scan_bus(pp); pci_bus_size_bridges(bus); pci_bus_assign_resources(bus); list_for_each_entry(child, &bus->children, node) pcie_bus_configure_settings(child); pci_bus_add_devices(bus); //添加PCI设备 return 0; error: pci_free_host_bridge(bridge); return ret; }
（3）配置空间OPS读写操作函数
static struct pci_ops dw_pcie_ops = { .read = dw_pcie_rd_conf, .write = dw_pcie_wr_conf, }; static int dw_pcie_rd_conf(struct pci_bus *bus, u32 devfn, int where, int size, u32 *val) { struct pcie_port *pp = bus->sysdata; if (!dw_pcie_valid_device(pp, bus, PCI_SLOT(devfn))) { *val = 0xffffffff; return PCIBIOS_DEVICE_NOT_FOUND; }
//read接口分成两个入口，分别为读取自身配置（RC）和其他设备（EP)配置 if (bus->number == pp->root_bus_nr) return dw_pcie_rd_own_conf(pp, where, size, val); //读取自身配置 return dw_pcie_rd_other_conf(pp, bus, devfn, where, size, val); //读取其他设备配置 } static int dw_pcie_rd_own_conf(struct pcie_port *pp, int where, int size, u32 *val) { struct dw_pcie *pci; if (pp->ops->rd_own_conf) return pp->ops->rd_own_conf(pp, where, size, val); pci = to_dw_pcie_from_pp(pp); return dw_pcie_read(pci->dbi_base + where, size, val); //读取自身配置，直接用内存读取的方式读取，地址为DBI基地址+偏移 } static int dw_pcie_rd_other_conf(struct pcie_port *pp, struct pci_bus *bus, u32 devfn, int where, int size, u32 *val) { int ret, type; u32 busdev, cfg_size; u64 cpu_addr; void __iomem *va_cfg_base; struct dw_pcie *pci = to_dw_pcie_from_pp(pp); if (pp->ops->rd_other_conf) return pp->ops->rd_other_conf(pp, bus, devfn, where, size, val); busdev = PCIE_ATU_BUS(bus->number) | PCIE_ATU_DEV(PCI_SLOT(devfn)) | PCIE_ATU_FUNC(PCI_FUNC(devfn)); if (bus->parent->number == pp->root_bus_nr) { type = PCIE_ATU_TYPE_CFG0; cpu_addr = pp->cfg0_base; //配置空间物理基地址 cfg_size = pp->cfg0_size; va_cfg_base = pp->va_cfg0_base; //配置空间虚拟基地址 } else { type = PCIE_ATU_TYPE_CFG1; cpu_addr = pp->cfg1_base; cfg_size = pp->cfg1_size; va_cfg_base = pp->va_cfg1_base; } dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX1, //首先写入ATU配置 type, cpu_addr, busdev, cfg_size); ret = dw_pcie_read(va_cfg_base + where, size, val); //写入ATU后可以直接读取，读取地址为配置空间虚拟基地址+偏移 if (pci->num_viewport <= 2) dw_pcie_prog_outbound_atu(pci, PCIE_ATU_REGION_INDEX1, PCIE_ATU_TYPE_IO, pp->io_base, pp->io_bus_addr, pp->io_size); return ret; }
（4） read/write底层实现
int dw_pcie_read(void __iomem *addr, int size, u32 *val) { if ((uintptr_t)addr & (size - 1)) { *val = 0; return PCIBIOS_BAD_REGISTER_NUMBER; } if (size == 4) { *val = readl(addr); } else if (size == 2) { *val = readw(addr); } else if (size == 1) { *val = readb(addr); } else { *val = 0; return PCIBIOS_BAD_REGISTER_NUMBER; } return PCIBIOS_SUCCESSFUL; } void dw_pcie_prog_outbound_atu(struct dw_pcie *pci, int index, int type, u64 cpu_addr, u64 pci_addr, u32 size) { u32 retries, val; if (pci->ops->cpu_addr_fixup) cpu_addr = pci->ops->cpu_addr_fixup(cpu_addr); if (pci->iatu_unroll_enabled) { dw_pcie_prog_outbound_atu_unroll(pci, index, type, cpu_addr, pci_addr, size); return; } dw_pcie_writel_dbi(pci, PCIE_ATU_VIEWPORT, PCIE_ATU_REGION_OUTBOUND | index); dw_pcie_writel_dbi(pci, PCIE_ATU_LOWER_BASE, lower_32_bits(cpu_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_BASE, upper_32_bits(cpu_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_LIMIT, lower_32_bits(cpu_addr + size - 1)); dw_pcie_writel_dbi(pci, PCIE_ATU_LOWER_TARGET, lower_32_bits(pci_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_UPPER_TARGET, upper_32_bits(pci_addr)); dw_pcie_writel_dbi(pci, PCIE_ATU_CR1, type); dw_pcie_writel_dbi(pci, PCIE_ATU_CR2, PCIE_ATU_ENABLE); /* * Make sure ATU enable takes effect before any subsequent config * and I/O accesses. */ for (retries = 0; retries < LINK_WAIT_MAX_IATU_RETRIES; retries++) { val = dw_pcie_readl_dbi(pci, PCIE_ATU_CR2); if (val & PCIE_ATU_ENABLE) return; usleep_range(LINK_WAIT_IATU_MIN, LINK_WAIT_IATU_MAX); } dev_err(pci->dev, "outbound iATU is not being enabled\n"); } static inline void dw_pcie_writel_dbi(struct dw_pcie *pci, u32 reg, u32 val) { __dw_pcie_write_dbi(pci, pci->dbi_base, reg, 0x4, val); } void __dw_pcie_write_dbi(struct dw_pcie *pci, void __iomem *base, u32 reg, size_t size, u32 val) { int ret; if (pci->ops->write_dbi) { pci->ops->write_dbi(pci, base, reg, size, val); return; } ret = dw_pcie_write(base + reg, size, val); if (ret) dev_err(pci->dev, "write DBI address failed\n"); } int dw_pcie_write(void __iomem *addr, int size, u32 val) { if ((uintptr_t)addr & (size - 1)) return PCIBIOS_BAD_REGISTER_NUMBER; if (size == 4) writel(val, addr); else if (size == 2) writew(val, addr); else if (size == 1) writeb(val, addr); else return PCIBIOS_BAD_REGISTER_NUMBER; return PCIBIOS_SUCCESSFUL; }
参考：
https://blog.csdn.net/qq_42254853/article/details/123050004
https://blog.csdn.net/qq_29832469/article/details/124911677
https://blog.csdn.net/qq_42254853/article/details/120903907
https://elinux.org/Device_Tree_Usage#PCI_Address_Translation
https://www.cnblogs.com/fortunely/p/16405592.html#5reg
https://blog.csdn.net/zz2633105/article/details/125058094
https://blog.csdn.net/yumiano0/article/details/127267094
https://blog.csdn.net/linjiasen/article/details/87944672
https://blog.csdn.net/u013253075/article/details/119361574
https://blog.csdn.net/llxxyy507/article/details/114786875
https://zhuanlan.zhihu.com/p/26244141
