Corundum is very interesting project because somehow it managed to include implementations of Ethernet, PCIe and AXI-S. This is my attempt to deconstruct the source code and understand the the engineering miracle that Corundum is.
Installation Link to heading
Let’s start by installing some dependencies
sudo apt install -y \
git make gcc g++ clang \
python3 python3-pip python3-venv \
verilator \
iverilog \
gtkwave \
libpcap-dev \
libelf-dev
Then in a .venv, We need to install cocotb and some other libraries
pip install cocotb cocotb-test cocotbext-axi cocotbext-eth scapy
Running cocotb tests Link to heading
Running mqnic_core proved to bit tricky, because for some reason running mqnic_core_axi fails out-of-box with the iverilog compilation errors. So, I have to move some parameters in 2 files.
diff --git a/fpga/common/rtl/mqnic_interface_rx.v b/fpga/common/rtl/mqnic_interface_rx.v
index d993b2bd..37f71f9a 100644
--- a/fpga/common/rtl/mqnic_interface_rx.v
+++ b/fpga/common/rtl/mqnic_interface_rx.v
@@ -79,7 +79,9 @@ module mqnic_interface_rx #
parameter AXIS_KEEP_WIDTH = AXIS_DATA_WIDTH/8,
parameter AXIS_RX_ID_WIDTH = PORTS > 1 ? $clog2(PORTS) : 1,
parameter AXIS_RX_DEST_WIDTH = RX_QUEUE_INDEX_WIDTH+1,
- parameter AXIS_RX_USER_WIDTH = (PTP_TS_ENABLE ? PTP_TS_WIDTH : 0) + 1
+ parameter AXIS_RX_USER_WIDTH = (PTP_TS_ENABLE ? PTP_TS_WIDTH : 0) + 1,
+ parameter DMA_CLIENT_TAG_WIDTH = $clog2(RX_DESC_TABLE_SIZE),
+ parameter DMA_CLIENT_LEN_WIDTH = DMA_LEN_WIDTH
)
(
input wire clk,
@@ -215,8 +217,7 @@ module mqnic_interface_rx #
input wire [DMA_CLIENT_LEN_WIDTH-1:0] mtu
);
-parameter DMA_CLIENT_TAG_WIDTH = $clog2(RX_DESC_TABLE_SIZE);
-parameter DMA_CLIENT_LEN_WIDTH = DMA_LEN_WIDTH;
+
parameter REQ_TAG_WIDTH = $clog2(RX_DESC_TABLE_SIZE);
diff --git a/fpga/common/rtl/mqnic_interface_tx.v b/fpga/common/rtl/mqnic_interface_tx.v
index 5251bd7b..78edf7a6 100644
--- a/fpga/common/rtl/mqnic_interface_tx.v
+++ b/fpga/common/rtl/mqnic_interface_tx.v
@@ -66,7 +66,9 @@ module mqnic_interface_tx #
parameter AXIS_KEEP_WIDTH = AXIS_DATA_WIDTH/8,
parameter AXIS_TX_ID_WIDTH = TX_QUEUE_INDEX_WIDTH,
parameter AXIS_TX_DEST_WIDTH = $clog2(PORTS)+4,
- parameter AXIS_TX_USER_WIDTH = TX_TAG_WIDTH + 1
+ parameter AXIS_TX_USER_WIDTH = TX_TAG_WIDTH + 1,
+ parameter DMA_CLIENT_TAG_WIDTH = $clog2(TX_DESC_TABLE_SIZE),
+ parameter DMA_CLIENT_LEN_WIDTH = DMA_LEN_WIDTH
)
(
input wire clk,
@@ -188,8 +190,7 @@ module mqnic_interface_tx #
input wire [DMA_CLIENT_LEN_WIDTH-1:0] mtu
);
-parameter DMA_CLIENT_TAG_WIDTH = $clog2(TX_DESC_TABLE_SIZE);
-parameter DMA_CLIENT_LEN_WIDTH = DMA_LEN_WIDTH;
Running top mqnic core Link to heading
To run the simulation, from core AXI cocotb tests
cd fpga/common/tb/mqnic_core_axi/
make
885364.00ns INFO cocotb.mqnic_core_axi.s_axil_ctrl Read complete addr: 0x000101f0 prot: 2 resp: 0 data: 00 00 00 00
885364.00ns INFO cocotb.mqnic_core_axi.s_axil_ctrl Read start addr: 0x000101f8 prot: 2 length: 4
885432.00ns INFO cocotb.mqnic_core_axi.s_axil_ctrl Read complete addr: 0x000101f8 prot: 2 resp: 0 data: 00 00 00 00
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5336, 954442, 167423, 0, 8056, 232373, 0, 0, 0, 0, 44967, 0, 0, 0, 0, 0, 4641, 890538, 134033, 0, 7233, 189328, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
885440.00ns INFO cocotb.regression run_test_nic passed
885440.00ns INFO cocotb.regression ******************************************************************************************
** TEST STATUS SIM TIME (ns) REAL TIME (s) RATIO (ns/s) **
******************************************************************************************
** test_mqnic_core_axi.run_test_nic PASS 885440.00 273.98 3231.76 **
******************************************************************************************
** TESTS=1 PASS=1 FAIL=0 SKIP=0 885440.00 274.35 3227.43 **
******************************************************************************************
Block level test cmac_pad Link to heading
Going deeper in hierarchy, running one of block tests,cmac_pad in this example
cd fpga/common/tb/cmac_pad
make
5152.00ns INFO cocotb.regression ****************************************************************************************
** TEST STATUS SIM TIME (ns) REAL TIME (s) RATIO (ns/s) **
****************************************************************************************
** test_cmac_pad.run_test_001 PASS 800.00 0.16 5094.65 **
** test_cmac_pad.run_test_002 PASS 3104.00 0.21 14532.15 **
** test_cmac_pad.run_test_pad_001 PASS 272.00 0.05 4982.60 **
** test_cmac_pad.run_test_pad_002 PASS 976.00 0.07 13656.54 **
****************************************************************************************
** TESTS=4 PASS=4 FAIL=0 SKIP=0 5152.00 0.57 9023.16 **
****************************************************************************************
Generating block diagram of mqnic_core Link to heading
I thought it would be useful to have block diagram of mqnic_core. Using run.ys, we can run yosys to generate json and pass it to netlistsvg.
prep -top mqnic_core_pcie;
write_json elaborated.json;
write_verilog elaborated.v;
cd fpga/common/rtl
sed -i 's?$finish?//$finish?' *.v
sed -i 's?$error?//$error?' *.v
yosys -s run.ys
netlistsvg elaborated.json -o elaborated.svg
convert elaborated.svg elaborated.png
