White Rabbit on general purpose FPGA boards (without the dedicated external oscillators)

Based on the original contribution by Missing Link Electronics at the 13th White Rabbit Workshop at CERN in 2024 and with the help of Nikhef's Peter Jansweijer CLBv3 (found on the OHWR gitlab repository), this project demonstrates a functional White Rabbit implementation on the Acorn CLE215+ Artix7 FPGA.

Synthesizing for the Acorn CLE215+

The patch applied to the White Rabbit PTP Core repository allowing to synthetsize the project for the Acorn CLE215+ as well as the instructions are found in the patch repository (tested with Xilinx Vivado 2022.2).

PLL gain settings

From the shell connected to the embdded RISC-V processor, the main PLL gain coefficients (proportional and integral) are set using the command

pll gain 0 0 -100 -2 12

and similartly for the helper PLL (any negative number will do instead of -1 as 2nd argument)

pll gain -1 0 -100 -2 12

Once both PLLs are locked, the Kp and Ki can be increased for better tracking performance.

Results

MDEV measurement as a function of PLL loop gain coefficients, changing Kp while keeping Ki fixed (-2) except for two measurements, of the time interval between the White Rabbit Switche (WRS) and the CLE215+ 1-PPS outputs. For reference, WRS is the Modified Allan Deviation of the two PPS generated by two WRS (Creotech), matching the expected value of about 60 ps jitter at 1 s integration time.

Beyond the long-term time-domain characteristics, the phase noise of a 10-MHz output locked on the White Rabbit control signal is measurement as a function of control loop parameters. Measurements performed using a Rohde & Schwarz FSWP Phase Noise Analyzer with 10-correlations in the 1-Hz bin.

All Modified Allan deviation of the 1-PPS outputs plot exhibit a $1/\tau$ slope, i.e. $1/\tau^2$ for the variance, characteristic of flicker phase noise. The MVAR of flicker PM [1] level is $0.0855 h_1/\tau^2$: at 1 Hz, the phase noise of $-70$ dBc/Hz or $-67$ dBrad $^2$ /Hz would lead to $\sqrt{10^{-6.7}/(10^7)^2\cdot 0.0855}=1.4\cdot 10^{-11}$. The observed $5.5\cdot 10^{-11}$ hints at a noise model too simple to account for all contributions.

From this analysis, we conclude that "optimal" loop coefficients are Kp=-4000 and Ki=-2.

The command pll stat must display something like

softpll: mode:3 seq:ready n_ref 1 n_out 1
irqs:400961 alignment_state:0 HL1 ML1 HY=25714 MY=42355 DelCnt=0 setpoint:17364 refcnt:209789 tagcnt:5

where HL1 means that the helper PLL is locked and ML1 means that the main PLL is locked.

[1] E. Rubiola, Enrico's Chart of Phase Noise and Two-Sample Variances, https://rubiola.org/pdf-static/Enrico%27s-chart-EFTS.pdf (2025)

SFP brands

Some SFP seem to operate properly with the Acorn, some not (but are functional with the M2SDR).

Tested as functional: Cisco GLC-BX-U BlueOptics BO15C3149620D and FS SFP-1G34-BX20

Tested as non functional: AXCEN AXGE-1254-0531

Application to the M2SDR

git clone https://github.com/enjoy-digital/litex_wr_nic
cd litex_wr_nic
git checkout m2sdr
./m2sdr_wr_nic.py --build
openFPGALoader --fpga-part xc7a200tsbg484 --cable ft4232 --freq 20000000 --write-flash --bitstream litex_m2sdr_platform.bin

where we manually openFPGALoader rather than --flash to synthesize and flash on different computers.

Then for communicating between PC and M2SDR: either

bar=`lspci | grep Xil | cut -d\  -f1`
litex_server --pcie --pcie-bar $bar

and then

litex_term crossover --csr-csv csr.csv

where csr.csv was found in the test/ of litex_wr_nic after synthesis of the gateware, but the resulting terminal can hardly display the gui output of the White Rabbit, so the preferred method is to compile the litex_wr_nic/software/kernel modules and

sudo rmmod m2sdr        # in case it was loaded
sudo insmod liteuart.ko
sudo insmod litepcie.ko
minicom -D /dev/ttyLXU0

The WR clock output is measured using a frequency counter by adding in litex_wr_nic/m2sdr_wr_nic.py the signal self.comb += platform.request("sync_clk_in").eq(ClockSignal("wr")) as the last instruction of the __init()__ function just before the main():, after commenting platform.request("pps_out").eq(pps), to avoid a conflict on the SYNCDBG_CLK pin.

When loading a new gateware, the computer must usually be rebooted. Sometimes disabling the PCI board, flashing the FPGA, and then enumerated again using

bar=`lspci | grep Xil | cut -d\  -f1`
echo 1 > /sys/bus/pci/devices/0000:$bar/remove
openFPGALoader -c ft4232 -b litex-acorn-baseboard-mini -f gateware.bit
echo 1 > /sys/bus/pci/rescan

and then rmmod and insmod the m2sdr.ko module might succeed (or fail).

Note: in case from litex_m2sdr import Platform fails when synthesizing the m2sdr branch of https://github.com/enjoy-digital/litex_wr_nic, make sure to

pip install --user -e .

in the litex_m2sdr repository.

Note: m2sdr_wr_nic.py --build synthesis fails with Vivado 2019.2 or 2020.1 and succeeds with Vivado 2022.2.

Results

Phase noise (M2SDR scaled 62.5-MHz v.s White Rabbit switch 10-MHz output):

Allan deviation:

M2SDR SDR with WR support

In litex_m2sdr, assuming litex_wr_nic is at the same filesystem tree level (possibly update litex_m2sdr.py with the two FIXME: Avoid harcoded path/platform directory information accordingly for different tree structure) and was installed using pip install --user -e .:

./litex_m2sdr.py --variant=baseboard --with-pcie --with-white-rabbit --build

on the build computer running Vivado (in our case 2022.2), and then on the target computer the PCIe with the M2SDR board is connected to:

openFPGALoader --fpga-part xc7a200tsbg484 --cable ft4232 --freq 20000000 --write-flash --bitstream ./build/litex_m2sdr_baseboard_pcie_x1_white_rabbit/gateware/litex_m2sdr_baseboard_pcie_x1_white_rabbit.bin
sudo echo 1 > /sys/bus/pci/rescan

Then from a kernel module perspective, scp -r litex_m2sdr/software/ from the build computer to the target computer and in software/kernel run make to compile the kernel modules, leading to m2sdr.ko and liteuart.ko. Once these two kernel modules are loaded

sudo insmod m2sdr.ko
sudo insmod liteuart.ko

the communication interface /dev/ttyLXU0 is created and minicom -D /dev/ttyLXU0 displays a prompt

wrc# ver                                                                                        
WR Core build: wrpc-v5.0-ohwr-9-g5ac04dd5-dirt (unsupported developer build)
Built: Jul 18 2025 08:28:12 by JM Friedt
Built for RISCV, 128 kB RAM, stack is 2048 bytes

DO NOT attempt to use the kernel modules found in litex_wr_nic: despite similar names, they will kernel panic. Also make sure to recompile the kernel according to the new gateware configuration since header files csr.h, mem.h and soc.h are automagically generated during gateware synthesis, so that kernel modules must be recompiled accordingly on the target computer.

Result: phase-lock of the M2SDR with SDR capability on the WR switch master signal

SAWR WRPC Monitor wrpc-v5.0-ohwr-9-g5ac04dd5-dirt | Esc/q = exit; r = redraw

TAI Time: 2023-05-12-12:28:06  UTC offset: 37   PLL mode: BC  state: Locked 
---+-------------------+-------------------------+---------+---------+-----
 # |        MAC        |       IP (source)       |    RX   |    TX   | VLAN
---+-------------------+-------------------------+---------+---------+-----
 0 | 22:33:44:55:66:77 |                         |    3105 |    1291 |    0

--- HAL ---|------------- PPSI ------------------------------------------------
 Itf | Frq |  Config   | MAC of peer port  |    PTP/EXT/PDETECT States   | Pro 
-----+-----+-----------+-------------------+-----------------------------+-----
 wr0 | Lck | auto      | 70:b3:d5:91:ea:f0 | SLAVE    /IDLE      /EXT_ON | R-W 
Pro(tocol): R-RawEth, V-VLAN, U-UDP

--------------------------- Synchronization status ----------------------------
Servo state:          White-Rabbit: TRACK_PHASE                         

--- Timing parameters ---------------------------------------------------------
meanDelay        :              285.439 ns
delayMS          :              285.439 ns 
delayMM          :             1095.745 ns 
delayAsymmetry   :                0.000 ns
delayCoefficient :   +0.000000000000000000  fpa   0
ingressLatency   :                0.000 ns
egressLatency    :                0.000 ns
semistaticLatency:                6.400 ns
offsetFromMaster :               -0.025 ns
Phase setpoint   :                1.548 ns
Skew             :                0.067 ns
Update counter   :                  626 times
Master PHY delays TX:               238.809 ns   RX:           279.657 ns 
Slave  PHY delays TX:                 0.000 ns   RX:             6.400 ns 

If the finite state machine remains in Servo state: White-Rabbit: SYNC_PHASE (wait for hw) then execute ptrack ps-freeze.

WR clock output for phase noise measurement

To output the 62.5 MHz WR disciplined clock on the WFL output, comment https://github.com/enjoy-digital/litex_m2sdr/blob/main/litex_m2sdr.py#L522C1-L528C60

self.gpio.connect_to_pads(pads=platform.request("gpios"))

# Drive led from GPIO0 when in loopback mode to ease test/verification.
self.sync += If(self.gpio._control.fields.loopback, led_pad.eq(self.gpio.i_async[0]))

# Use GPIO0 as ClkIn.
self.comb += si5351_clk_in.eq(self.gpio.i_async[0])

and replace with

self.comb += platform.request('debug').eq(ClockSignal('wr'))

Changing the beatnote frequency

To change the beatnote from $62.5*(1-2^{14}/(2^{14}+1))=3814$ Hz, update the content of litex_wr_nic/firmware/wrpc-sw/include/spll_defs.h where HPLL_N is defined with the default value of 14.

Firmware update without gateware generation

litex_wr_nic provides, in the test directory, the test_cpu.py script which allows uploading the firmware without generating the full gateware.

1. Install litex_wr_nic by running pip install --user -e . in the git cloned litex_wr_nic directory
2. In the git cloned litex_m2sdr directory, synthesize the gateware (requires Vivado and litex standard install):

./litex_m2sdr.py --variant=baseboard --with-pcie --with-white-rabbit --build
cp csr.csv ../litex_wr_nic/test/
openFPGALoader --fpga-part xc7a200tsbg484 --cable ft4232 --freq 20000000 --write-flash --bitstream ./build/litex_m2sdr_baseboard_pcie_x1_white_rabbit/gateware/litex_m2sdr_baseboard_pcie_x1_white_rabbit.bin
# probably need to shutdown and restart the computer as the PCIe bus might be corrupt when rescanning
cd litex_m2sdr/software/kernel
make
sudo insmod m2sdr.ko

resulting in dmesg with

m2sdr 0000:06:00.0: Version LiteX-M2SDR SoC / baseboard variant / built on 2025-09-04 12:41:57

and at this point executing minicom -D /dev/ttyLXU0 allows for

wrc# ver                                                                                              
WR Core build: wrpc-v5.0-ohwr-9-g5ac04dd5-dirt (unsupported developer build)

When using the M2SDR on a different computer than the one used for the synthesis, we need to scp the litex_m2_sdr/software/kernel and csr.csv to the remote computer, as well as build/litex_m2sdr_baseboard_pcie_x1_white_rabbit/gateware/litex_m2sdr_baseboard_pcie_x1_white_rabbit.bi*.

3. execute litex_server --jtag --jtag-config=openocd_xc7_ft4232.cfg in one terminal or one screen session At the moment litex_server --jtag does not seem functional for transfering the gateware through JTAG so we

sudo insmod m2sdr.ko  # if not already loaded, e.g. on a remote computer
bar=`lspci | grep Xil | cut -d\  -f1`
sudo chown -R 777 /sys/bus/pci/devices/0000\:$bar
litex_server --pcie --pcie-bar $bar

5. Now that we have the csr.csv memory configuration file in the litex_wr_nic/test directory, go there and

cd ../litex_wr_nic/test
./test_cpu.py --build-firmware
./test_cpu.py --load-firmware ../litex_wr_nic/firmware/wrpc-sw/wrc.bin
minicom -D /dev/ttyLXU0

Make sure to remove lines 72-73 which execute a git checkout in firmware/build.py that would delete all updates made on the spll_main.c firmware file