5-Octeon as a PCI host

1. Introduction

PCI host mode refers to when Octeon is configured as the PCI bus arbiter controlling access to other devices. In this mode, Octeon most closely resembles a desktop PC or a rack mount server. This document provides detailed information on the following aspects of Octeon as a PCI host:

2. Hardware Coverage of this Document

This document does not cover all of the modes and registers available for the Octeon PCI block. Only the most common setup used for host mode is described. For a complete PCI block reference, refer to the Octeon Hardware Manual.

3. 32bit PCI Device Limitations

Most PCI devices implement 32bit memory addressing and don't support full 64bit accesses. This greatly constrains the address space available to Octeon. Normally Octeon can map devices to any 64bit address, but 32bit addresses must be mapped below 4GB. Adding this limitation to the fact that all BAR registers must not overlap (including Octeon's), limits memory accesses to Octeon through BAR1 only. BAR1 covers a 128MB range, divided into 32 4MB entries. Each entry may be mapped to an arbitrary Octeon memory address. By convention, memory mapped through BAR1 is visible to PCI devices as the lower 128MB address region.

Octeon's BAR2 can't be used by 32bit devices. Since BAR2 is 39bits wide, it overlaps with all possible 32bit addresses when mapped at 0. The next possible mapping (1<<39) is outside of the 32bit address region.
BAR1 entries are dynamically allocated as needed by calls to dma_map_single(). Overlapping regions are merged. If more than 32 regions are needed, the kernel will panic.
64bit devices can use Octeon's BAR2 to access all of memory. BAR2 must be mapped outside of the 32bit region.

4. PCI Host Bus Reset and Initialization

The recommended host mode PCI reset and initialization sequence requires quite a few register setups and strict timing constraints. Refer tolinux/kernel_2.6/linux/arch/mips/cavium-octeon/pci.c for the exact code for PCI initialization. Below are the general steps for reference, but please use pci.c as the definitive reference.

Reset the bus using CIU_SOFT_PRST
Delay for 2ms
Set the max write combining to 1 using NPI_CTL_STATUS
Deassert PCI reset and advertize PCX Host Mode Device Capability
Wait 2 ms after deasserting PCI reset
Setup NPI_PCI_CTL_STATUS_2
Wait 2 ms before doing PCI reads
Read NPI_PCI_CTL_STATUS_2 to determine the PCI mode (64/32, PCI-X/PCI)
Setup NPI_PCI_CFG01
1. Memory Space Access Enable
2. Master Enable
3. PERR# Enable
4. System Error Enable
5. Fast Back to Back Transaction Enable
Enable the internal arbiter using NPI_PCI_INT_ARB_CFG
Disable the Master Latency Timer using NPI_PCI_CFG16
Setup NPI_PCI_CFG22
1. Master Retry Value [1..255] and 0=infinite
2. AM_DO_FLUSH_I control
Setup NPI_PCI_CFG56
1. Data Parity Error Recovery Enable
2. Relaxed Ordering Enable
3. Maximum Memory Byte Count 1024B
4. Maximum outstanding Split transactions to 3
Setup Octeon BARs for best performance
1. NPI_PCI_READ_CMD_6 = 0x22
2. NPI_PCI_READ_CMD_C = 0x33
3. NPI_PCI_READ_CMD_E = 0x33
Setup PCI memory accesses from Octeon using NPI_MEM_ACCESS_SUBID3
1. Endian-Swap on read.
2. Endian-Swap on write.
3. No-Snoop on read.
4. No-Snoop on write.
5. Relax Read on read.
6. Relax Order on write.
Remap the Octeon BAR 0 to disable it.
Remap the Octeon BAR 1 to map 0-128MB
1. Put in L2 cache.
2. Byte swapping
Remap the Octeon BAR 2 to disable it.

5. PCI config space addressing

PCI configuration space is accessed through Octeon DID 3, SUBDID 1. The recommended method of referencing PCI configuration space is by constructing an Octeon IO space address from the needed bus, device, function, and register. The following structure defines the format for the IO address.

    typedef union    {        uint64_t    u64;        uint64_t *  u64_ptr;        uint32_t *  u32_ptr;        uint16_t *  u16_ptr;        uint8_t *   u8_ptr;        struct        {            uint64_t    upper       : 2;            uint64_t    reserved    : 13;            uint64_t    io          : 1;            uint64_t    did         : 5;            uint64_t    subdid      : 3;            uint64_t    reserved2   : 4;            uint64_t    endian_swap : 2;            uint64_t    reserved3   : 10;            uint64_t    bus         : 8;            uint64_t    dev         : 5;            uint64_t    func        : 3;            uint64_t    reg         : 8;        } s;    } octeon_pci_address_t;

To access a PCI device, set the fields of the address:

    octeon_pci_address_t pci_addr;    pci_addr.u64 = 0;    pci_addr.s.upper = 2;    pci_addr.s.io = 1;    pci_addr.s.did = 3;    pci_addr.s.subdid = 1;    pci_addr.s.endian_swap = 1;    pci_addr.s.bus = bus_number;    pci_addr.s.dev = device;    pci_addr.s.func = function;    pci_addr.s.reg = register;

A read or write can then be performed using the *_ptr union members.

Note:: PCI configuration space is in little endian format. All accesses need to be byte swapped before a store and after a load.

6. PCI port IO addressing

All PCI IO port addressing is based at 0x00011a0400000000. To reference a PCI IO at 0x1234, you must read and write to 0x00011a0400001234.

Endian swapping

Octeon supports multiple swapping modes across the PCI bus. For host mode development, only mode 1 is useful.

Legacy ISA Devices

Legacy PC devices, such as floppy controllers and CMOS RTCs, use IO ports below 0x1000. These ports are treated special by PCI bridges. In PCI bridge poweron defaults, these ports are normally blocked completely. If your board has a legacy PC device, such as a south bridge or video card behind a PCI bridge, the bridge must be specially configured. The bridge must be configured to pass all IO addresses 0x0 to 0x1000 without any alias filtering. This is done in the Linux kernel for Intel and PLX bridge chips. This must be changed if your board uses multiple bridge chips or a different brand. The code for this can be found in pci_chips.c.

Here is configuration changes made by Linux:

2 bytes at PCI config space 0x3e
1. Disable bit 2, ISA port filtering
2. Enable bit 3, VGA
3. Enable bit 4, VGA Alias Filter
1 byte at PCI config space 0x1c and 0x30 to set the IO base to 0

VIA South Bridge

The Via 686 series of south bridges, common on Octeon motherboards, have an IO address limitation. These devices require that the IO ports be below 0x10000 for the PCI devices embedded in the chip. The recommended way of doing this is to start allocation IO ports at 0x4000. Keep in mind there are ISA emulated devices using the port number below 0x1000 inside these chips.

7. PCI memory IO addressing

Octeon's PCI controller uses did=3, subdid=3,4,5 for PCI memory. In normal usage a single subdid is enough for all devices (0x00011b0000000000). It is important the PCI devices don't overlap with the Octeon BAR registers. It is recommended that PCI devices start at 0x8000000, right after the 0-128MB BAR1 mapping.

Endian swapping

Octeon supports multiple swapping modes across the PCI bus. For host mode development, only mode 1 is useful.

Octeon BAR setup

The following code sets the Octeon BAR registers in the recommended setup for PCI host mode.

    typedef union    {        uint32_t u32;        struct        {            uint64_t reserved                : 14;      // Reserved            uint64_t addr_idx                : 14;      // Address bits [35:22] sent to L2C            uint64_t ca                      : 1;       // Set '1' when access is to be cached in L2.            uint64_t end_swp                 : 2;       // Endian Swap Mode            uint64_t addr_v                  : 1;       // Set '1' when the selected address range is valid.        } s;    } octeon_pci_bar1_indexx_t;    // Write a 32bit value to the Octeon NPI register space    static void npi_write32(uint64_t address, uint32_t val)    {        volatile uint32_t *ptr = (volatile uint32_t *)(address ^ 4);        *ptr = val;    }    // Read a 32bit value from the Octeon NPI register space    static uint32_t npi_read32(uint64_t address)    {        volatile uint32_t *ptr = (volatile uint32_t *)(address ^ 4);        return *ptr;    }    // Remap the Octeon BAR 0 to disable it. We'll map to to the highest possible address    npi_write32(OCTEON_NPI_PCI_CFG04, 0xffffffff);    npi_write32(OCTEON_NPI_PCI_CFG05, 0xffffffff);    // Remap the Octeon BAR 1 to map 0-128MB    octeon_pci_bar1_indexx_t bar1_index;    int index;    bar1_index.u32 = 0;    bar1_index.s.ca = 1;            // 1 = Put in L2 cache    bar1_index.s.end_swp = 1;       // 1 = Byte swapping    bar1_index.s.addr_v = 1;        // This entry is valid    for (index=0; index<32; index++)    {        bar1_index.s.addr_idx = index;        npi_write32(OCTEON_NPI_PCI_BAR1_INDEXX(index), bar1_index.u32);    }    npi_write32(OCTEON_NPI_PCI_CFG06, 0);    npi_write32(OCTEON_NPI_PCI_CFG07, 0);    // Remap the Octeon BAR 2 to disable it. We'll map to to the highest possible address    npi_write32(OCTEON_NPI_PCI_CFG08, 0xffffffff);    npi_write32(OCTEON_NPI_PCI_CFG09, 0xffffffff);

8. DMA Memory Coherency

PCI DMA accesses go through the Octeon's L2 cache just like any other memory access. The L2 performs all core invalidates necessary to make sure memory is always coherent. This means that drivers don't need to worry about cache coherency.

9. PCI Interrupts

The PCI INT-A,B,C,D interrupt lines route to CIU_INTX_EN0 bits 36-39. The status of these lines can be monitored using CIU_INTX_SUM0. Depending on the setup of CIU_INTX_EN0, these will be routed to Mips interrupt 2 or 3. Under Linux, these are always mapped to Mips interrupt 2, which is then decoded into Linux IRQ 44-47 based on the status of CIU_INTX_SUM0.

Note:: PCI interrupts are level triggered active low. Don't connect active high or edge triggered devices to the PCI interrupts. Use GPIO pins for these devices.

Interrupt routing

To determine the interrupt line used for each device, you must consult with the board schematic. Unfortunately most designers do not follow the PCI spec guidelines for choosing interrupt lines. This means the only way to determine where each interrupt line goes is through the board schematic. This board specific setup is performed for Linux inpcibios_map_irq and octeon_get_pci_interrupts (linux/kernel_2.6/linux/arch/mips/cavium-octeon/{pci.c, hal.c}).

GPIO Interrupts

The GPIO pins should be used for interrupts from any devices that do not follow the PCI specification for interrupt generation. Each GPIO pin can be independently configured for edge / level triggered as well as active high/low. In addition, these pins have configurable debouce logic per pin. It is recommended that devices be hooked to GPIO 15, then 14, and so on. This allows boards to work properly with Octeon Pass 1 as well as newer chips.

VIA South Bridge Interrupts

The Via 686 series of south bridges are designed to be directly connected to x86 processors. These devices do not create standard PCI interrupts. The 686 interrupts are active high. This interrupt line must be connected to an Octeon GPIO register, preferably GPIO15. Inside the 686 chip, the original PC 8259 dual interrupt controllers is emulated. In order for interrupts from the 686 chip to work, both the master and slave 8259s must be programmed. Refer to linux/arch/mips/cavium-octeon/i8259.c for details.

Interrupt Affinity

Interrupts are normally handled by the cpu that loads the driver. This is normally cpu 0. This can be changed through the /proc/irq interface. Writing a cpumask to/proc/irq/#/smp_affinity will route an interrupt to the cores in the mask.

For example, lets say this is what "cat /proc/interrupts" shows:

                  CPU00      CPU01          3:        215        277          Octeon  IPC          7:     404782     404029          Octeon  timer, Perfcounter         23:      15242      15203          Octeon  Ethernet         42:          0        143          Octeon  serial         45:     987869          0          Octeon  eth0         54:          0          0          Octeon  ECC         64:          0          0          Octeon  dwc_otg, dwc_otg_pcd, dwc_otg_hcd:usb1

To switch the interrupts for eth0 to the second core:

        echo 2 > /proc/irq/45/smp_affinity

To have both cores process the interrupt:

        echo 3 > /proc/irq/45/smp_affinity

10. Configuring Linux for PCI Host

Linux uses the bootloader flag CVMX_BOOTINFO_CFG_FLAG_PCI_HOST to determine if the board supports PCI in host mode. If this bit is set in the config_flags field, PCI is setup and enumerated. If it isn't set, Linux doesn't configure PCI.

11. Tested Linux Device Drivers

The following Linux device drivers are known to work with Octeon in host mode. Many other drivers should work without modification, but only these have been tested so far.

Via Rhine II 100Mbps Ethernet
Realtech 8139D 100Mbps Ethernet
NEC D720100 USB 2.0
Intel E1000 1Gbps Ethernet (Wired and Fiber)
CMD640 Parallel IDE Controller
Silicon Motion SATA controller
AirLink AWILL3026/NA USB 802.11b/g
USB Mass storage devices

本站仅提供存储服务，所有内容均由用户发布，如发现有害或侵权内容，请点击举报。