Different event counters for each ASIC in local triggering

[ckierans]

When running local triggering in the Layer 0 testing (i.e. with a detector connected and biased), and only a single ASIC with triggering enabled, I see a different number of events on each ASIC. For example, see the output below when I had ASIC 3 as the only ASIC with local triggering enabled. Logically, all other ASICs should report the same event count as ASIC 3.

Clearing the event data and then starting the data stream:
client.reset_ASIC_event_data()
client.print_ASIC_event_data()
print("------------------------")
sleep(0.1)
client.start_data_stream()
sleep(1000)
client.stop_data_stream()
client.print_ASIC_event_data() 

The output:
ASIC 00 --                    N_fifo:          0                    event count:          0
ASIC 01 --                    N_fifo:          0                    event count:          0
ASIC 02 --                    N_fifo:          0                    event count:          0
ASIC 03 --                    N_fifo:          0                    event count:          0
ASIC 04 --                    N_fifo:          0                    event count:          0
ASIC 05 --                    N_fifo:          0                    event count:          0
ASIC 06 --                    N_fifo:          0                    event count:          0
ASIC 07 --                    N_fifo:          0                    event count:          0
ASIC 08 --                    N_fifo:          0                    event count:          0
ASIC 09 --                    N_fifo:          0                    event count:          0
ASIC 10 --                    N_fifo:          0                    event count:          0
ASIC 11 --                    N_fifo:          0                    event count:          0
------------------------
ASIC 00 --                    N_fifo:          0                    event count:      12555
ASIC 01 --                    N_fifo:          0                    event count:      12555
ASIC 02 --                    N_fifo:          0                    event count:      12471
ASIC 03 --                    N_fifo:          0                    event count:      12437
ASIC 04 --                    N_fifo:          0                    event count:      12415
ASIC 05 --                    N_fifo:          0                    event count:      12555
ASIC 06 --                    N_fifo:          0                    event count:      12535
ASIC 07 --                    N_fifo:          0                    event count:      12555
ASIC 08 --                    N_fifo:          0                    event count:      12503
ASIC 09 --                    N_fifo:          0                    event count:      12474
ASIC 10 --                    N_fifo:          0                    event count:      12484
ASIC 11 --                    N_fifo:          0                    event count:      12433

[SeanCGriffin]

@ckierans Please post your exact trigger configuration (i.e. the output of the client call) so we know exactly what state the FPGA was in.

[ckierans]

The trigger configuration for each ASIC for this test was as follows:

for vata in range(0, 12):
    print(f"ASIC{vata:02d}: ", client.get_trigger_enable_mask(vata)) 

ASIC00:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC01:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC02:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC03:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC04:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC05:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC06:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC07:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC08:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC09:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC10:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}
ASIC11:  {'tm_hit': False, 'tm_ack': False, 'force_trigger': False, 'asics': [False, False, False, True, False, False, False, False, False, False, False, False]}

[adamschoenwald]

Does the differential between asics increase as you wait longer before checking counts, or is it always around the same. If the amount they differ by doesnt scale w/ time counting perhaps its an issue with processor time spent between enabling asic to asic or time spent between one asic and the next?

[SeanCGriffin]

My initial reaction was that VATA cores might be failing to get their data into the FIFOs, but the number of events doesn't quite agree with that. We (@lucasparker mostly?) need to think about how triggers get accepted and where the event counters get incremented.

Does the differential between asics increase as you wait longer before checking counts, or is it always around the same. If the amount they differ by doesnt scale w/ time counting perhaps its an issue with processor time spent between enabling asic to asic or time spent between one asic and the next?

I said something similar in one of my responses; I need to remember if the event counters still go when the data emitter isn't running or if the FIFOs just fill up and everything just chills out.