Skip to content

Add NeighborDiscovery caching checks - #1368

Merged
kstribrnAmzn merged 15 commits into
FreeRTOS:mainfrom
kstribrnAmzn:ndCacheFixes
Oct 15, 2026
Merged

kstribrnAmzn merged 15 commits into
FreeRTOS:mainfrom
kstribrnAmzn:ndCacheFixes

Conversation

@kstribrnAmzn

Copy link
Copy Markdown
Member

Description

  1. Protect against unsolicited neighbor advertisements writing to the cache
  2. Protect against MAC caching for spoofed packets

We thank Causal Security (https://causalsecurity.com/) for their help identifying this issue.
We thank @htibosch for the extensive help developing source code changes for this fix

Test Steps

Tested with unit tests, CBMC proofs

Checklist:

  • I have tested my changes. No regression in existing tests.
  • I have modified and/or added unit-tests to cover the code changes in this Pull Request.

Related Issue

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@kstribrnAmzn
kstribrnAmzn force-pushed the ndCacheFixes branch 2 times, most recently from 1d7af29 to dda05dc Compare September 25, 2026 17:07
@kstribrnAmzn
kstribrnAmzn marked this pull request as ready for review September 28, 2026 20:52
@kstribrnAmzn

Copy link
Copy Markdown
Member Author

The CBMC proofs timed out yesterday. Trying a rerun now.

Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_ND.c Resolved
Comment thread source/FreeRTOS_UDP_IPv6.c Resolved
Comment thread source/FreeRTOS_ND.c Outdated
break;
}

if( pxICMPPacket->xIPHeader.ucHopLimit != 255 )

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking, test): This guard is what keeps off-link NAs out, but its only test uses ucHopLimit = 64 on an empty cache. The mutant ucHopLimit < 254 survives the suite. Please add a boundary test where 254 is the only defect:

prvSeedSolicitedEntry( 0, &xDefaultIPAddress );
prvBuildNaPacket( &xPkt, ndTEST_FLAG_SOLICITED | ndTEST_FLAG_OVERRIDE, &xDefaultIPAddress, &xNewMAC );
xPkt.xIPHeader.ucHopLimit = 254U;
( void ) prvProcessICMPMessage_IPv6( &xBuf );
TEST_ASSERT_EQUAL( eND_INCOMPLETE, xNDCache[ 0 ].ucState );

Also:

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added the ndHOP_LIMIT macro (this can be merged with the raDEFAULT_HOP_LIMIT macro later) as well as adjusted the print statemnet, casting, and test.

Comment thread source/FreeRTOS_ND.c Outdated
{
/* This answers a solicitation of ours, so the new MAC-address is the
* legitimate way for a neighbour to announce that it changed. */
eNaAction = eNA_UPDATE_REACHABLE;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking, security): Case F takes the S bit as proof that we asked, but REACHABLE/STALE/DELAY rows have no solicitation outstanding. One NA claiming S=1/O=1 therefore replaces the MAC of any existing row, including a router's. A unit test at head confirms this: the row keeps REACHABLE and takes the new MAC.

RFC 4861 §7.2.5 does allow it. But the PR already deviates for S=0/O=1 (Case G keeps the MAC and probes). Could Case F do the same unless the row is INCOMPLETE or PROBE? Storing the tick of our last NS in the row, in place of the write-only ulLastMatchingNA, would also work. The comments at :842-844, :1157-1161 and :1601-1603 state this invariant as already enforced.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated this to now outright reject the solicitation update.

Comment thread source/FreeRTOS_ND.c
* remote device re-inject the parked packet at will: the packet would
* miss the cache again, trigger another solicitation and be parked
* once more. */
if( ( xAction == eNA_UPDATE_REACHABLE ) ||

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking, test): This gating fixes the re-injection issue, but replacing the condition with ( 1 ) survives the suite. Suggested test:

  • Park a packet from xDefaultIPAddress.
  • Seed a REACHABLE row with xDefaultMACAddress.
  • Send an unsolicited NA carrying that same MAC (gives eNA_MAINTAIN).
  • Register no uxIPHeaderSizePacket/xSendEventStructToIPTask expectations.
  • Assert pxNDWaitingNetworkBuffer is unchanged.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should now be covered by test_prvProcessNA_MaintainAction_LeavesParkedPacket

Comment thread source/FreeRTOS_ND.c Outdated
size_t uxNeededSize;
NetworkBufferDescriptor_t * pxNetworkBuffer;
/* Move to next option */
if( uxRemaining < ( ( size_t ) ucLen * 8U ) )

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: This branch is unreachable: :987 already breaks on ( size_t ) ucLen * 8U > uxRemaining, and neither value changes in between. Deleting either check alone survives the suite; deleting both is caught by ASan. Please drop :1002-1006; it is an uncovered branch against the gate. Since this loop is in view: 1-7 trailing bytes after the last option are not treated as malformed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've deleted the uncovered branch.

Comment thread source/FreeRTOS_ND.c Outdated
break;
}

/* Simplified: Just walk the remaining buffer space */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (non-blocking): We should probably avoid comments to the author
Please replace them with the actual precondition: xCheckIPv6SizeFields() (FreeRTOS_IPv6.c:128), or the checksum path, guarantees ETH + IPv6 + usPayloadLength <= xDataLength. State it as an @pre with a configASSERT(); that also gives the CBMC harness the assumption it is missing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@@ -0,0 +1,198 @@
import socket

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chore (non-blocking): This new file has no MIT header. It hard-codes one lab setup (DUT_IP_V4 = "192.168.1.11", IFACE_NAME = "Gigabit", a fixed link-local), needs a DUT CLI that isn't in this repo, and isn't run by CI. Suggestions:

  • Drop it from this PR, or add a README and take the parameters from argv/env.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This info can be added later. This PR is already 4400 lines. I don't want to add more unless its critical.

Comment thread source/FreeRTOS_ND.c
case ( uint8_t ) eND_REACHABLE:

/* The neighbor is known and reachability was recently confirmed. */
if( xNDCache[ x ].ucAge <= ( uint8_t ) ndMAX_CACHE_AGE_BEFORE_NEW_ND_SOLICITATION )

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): In base, a valid row sent an NS when ucAge <= 3, and the answer reset it. Here REACHABLE→STALE keeps the leftover age (≤3). It does not reset it the way vNDPCacheSetState( ..., eND_STALE ) does at :722. STALE is then freed at 0 (:488), with no NS ever sent. A probe test confirms that a REACHABLE router row goes to FREE after ipconfigMAX_ND_AGE ticks with no buffer allocation, i.e. no NS. Recovery then depends on the next send:

  • UDP/ping re-solicit through prvStartLookup().
  • The TCP reply path does not (FreeRTOS_TCP_Transmission_IPv6.c:236-244, pre-existing). That was masked before, when the receive path re-created the row.

Suggest resetting ucAge on REACHABLE→STALE (as :722 does), or probing at ucAge <= 3 as before. The docs at FreeRTOSIPConfigDefaults.h:3072-3073 and FreeRTOS_ND.h:149-152 still describe the old pre-expiry solicitation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to resetting ucAge on the state

Comment thread source/FreeRTOS_ND.c Outdated
* way its MAC-address is learned, and without it the first reply to such
* a peer would be dropped. */
if( ( eType == eIPv6_LinkLocal ) ||
( ( eType == eIPv6_Global ) && ( FreeRTOS_FindEndPointOnNetMask_IPv6( pxIPAddress ) != NULL ) ) )

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

issue (blocking): With ipconfigCOMPATIBLE_WITH_SINGLE == 1, FreeRTOS_FindEndPointOnNetMask_IPv6() returns the single end-point for any address (FreeRTOS_Routing.c:1357-1360), so this condition is true for every global source. Inbound TCP (all ports) and UDP to a bound socket from an off-link IPv6 host then send an NS to that host's solicited-node address. No one on the link can answer it, so the packet is parked and dropped after ipND_RESOLUTION_MAX_DELAY. Base only resolved link-local sources here. The same over-match occurs in multi-end-point builds for an IPv6 end-point with uxPrefixLength == 0: xCompareIPv6_Address() returns a match (FreeRTOS_IPv6.c:425-427). Could this compare against ipv6_settings.xPrefix/uxPrefixLength directly and require a non-zero prefix length? A unit test in the single-end-point routing variant would catch it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've replaced this with prvIsAddressOnLink(), which compares xPrefix/uxPrefixLength directly and skips zero-length prefixes.

Comment thread source/FreeRTOS_DHCP.c Outdated

#if ( ipconfigDHCP_FALL_BACK_AUTO_IP != 0 )
/* Define the Link Layer IP address: 169.254.x.x */
/* Define the Link Layer IP address: 169.254.x.x */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit (suggestion): We should probably not add formatting changes to the PR

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That won't work. We enforce formatting on each PR.

Comment thread source/FreeRTOS_ND.c Resolved
1. Protect against unsolicited neighbor advertisements writing to the cache
2. Protect against MAC caching for spoofed packets

We thank Causal Security (https://causalsecurity.com) for their help identifying this issue.
We thank @htibosch for the extensive help developing source code changes for this fix
Fix build, unit-test, doxygen and spell-check failures on the neighbor
discovery cache protection change:

- build/unittest (-Werror): remove redundant pxNDWaitingNetworkBuffer
  extern (already in FreeRTOS_IP.h), cast xTaskGetTickCount() to uint32_t
  and ~ndpFLAG_IS_ROUTER to uint8_t, hoist ucZeroMac and pcHostType to
  block top (declaration-after-statement), cast xMatch to int for %d,
  drop unused pcBuffer, enlarge pcSpace[16]->[24] to avoid snprintf
  truncation, and move the ipconfigMAX_ND_AGE fallback define to the
  top macro block.
- doxygen: document the ND config macros and pcNDStateName /
  pcNDActionName / vNDCheckWaitingPacket, and remove the duplicate
  @PARAM on the pxNDPCacheLookup prototype.
- spell-check: fix "faciliate" typo, rename xhasReleased -> xHasReleased,
  and add "neighbours" and "TLLA" to the cSpell wordlist.
I did finally break down and verify this on my own
linux machine. It passes
Raise FreeRTOS_ND.c coverage from 86.8%/80.2% (line/branch) to
98.7%/97.1%, with functions at 100%.

Tests added:
- vNDAgeCache state machine: INCOMPLETE timeout (matching/non-matching/
  no parked buffer), DELAY->PROBE, PROBE-exhausted->FREE, entries not
  expiring this tick (age-guard false sides), and PROBE buffer-alloc
  failure
- prvProcessNA / prvIsValidNa / prvDetermineAction: TLLA multicast-MAC
  and zero-MAC rejection, malformed zero-length option drop, MAINTAIN /
  UPDATE_STALE / CONFIRM_REACHABLE actions, solicited/unsolicited and
  Override=0/1 arms, router-flag recording on insert and update,
  cache-full lowest-age eviction, and cache-full all-INCOMPLETE (no
  eviction) path
- prvProcessICMPMessage_IPv6: unimplemented ICMP types (Dest Unreachable,
  Packet Too Big, Time Exceeded, Parameter Problem)
- pcNDStateName / pcNDActionName: every enum case plus the default arm
- pcMessageType: multicast listener report cases

Stabilize the pre-existing NeighborAdvertisement3 test, which relied on
uninitialized stack: zero its packet structs and allow the incoming-buffer
release, making it order-independent. New tests that leave non-FREE cache
entries wipe xNDCache on exit, since the suite has no setUp/tearDown.

Mark genuinely unreachable branches with LCOV_EXCL_BR_LINE: defensive NULL
checks after cache lookups, config-gated debug prints (ipconfigHAS_PRINTF /
ipconfigHAS_DEBUG_PRINTF), the configASSERT-guarded prefix-length failure
path in FreeRTOS_CreateIPv6Address, and an unreachable partial-byte bounds
combination.
At this point everything should actually pass
Comment thread source/FreeRTOS_DNS.c Resolved
Comment thread source/FreeRTOS_IP_Timers.c Resolved
Comment thread source/FreeRTOS_ND.c Outdated

for( x = 0; x < ( BaseType_t ) ipconfigND_CACHE_ENTRIES; x++ )
{
if( ( xNDCache[ x ].ucFlags & ndpFLAG_IS_ROUTER ) == 0U )

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression: an on-link attacker can permanently fill the ND cache.

ndpFLAG_IS_ROUTER comes straight from the R bit of a received NA (vNDPCacheUpdate), and this loop never evicts router-flagged entries. On main, vNDRefreshCacheEntry always overwrote the oldest entry, so a new binding could always get in.

How it happens:

  1. Send ipconfigND_CACHE_ENTRIES (default 24) TCP/UDP frames from spoofed link-local sources. xCheckRequiresNDResolution sends an NS for each one and prvNDCacheRecordSolicitation records it as INCOMPLETE.
  2. Answer each NS with an NA that has S=1, R=1 and a TLLA. Every entry becomes REACHABLE with the router flag set.
  3. prvGetFreeNDPCacheEntry now returns NULL for every caller. No new INCOMPLETE entry can be recorded, so every later NA, including the gateway's after its entry ages out, hits pxEntry == NULL and becomes eNA_DROP.
  4. The attacker keeps the entries alive by sending occasional traffic from the matching MAC (vNDRefreshCacheEntryAge).

The result is that no new neighbours can be resolved until reboot. test_vNDRefreshCacheEntry_NoMatchingEntryCacheFull currently asserts this behaviour ("Routers are never evicted").

Suggested minimal fix: let router entries be evicted as a last resort (fall through to the oldest router entry when nothing else qualifies), or only grant the eviction exemption to an address that matches the configured or RA-learned gateway instead of trusting the NA's R bit.

@kstribrnAmzn kstribrnAmzn Oct 14, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll simplify this to evicting the oldest cached entry regardless of the source. Introducing further logic to have a two-stage eviction process will grow this PR further and I already hate the size.

Adding a final stage eviction of routers was fairly trivial. Added this behavior.

Comment thread source/FreeRTOS_ND.c Outdated
/* The binding that is being kept must be verified, so let
* vNDAgeCache() send the first unicast NS straight away. */
pxEntry->ucAge = ( uint8_t ) 1U;
pxEntry->ucNumProbes = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression: a wrong or changed MAC can never be corrected, and the PROBE entry never expires.

prvDetermineAction maps every NA with O=1 and a different TLLA to eNA_REJECT_MAC_PROBE, including a solicited answer to our own probe. Each time, this branch resets ucNumProbes = 0, so the PROBE → FREE exit in vNDAgeCache (ucNumProbes >= ipconfigMAX_ND_RE_LOOKUP_ATTEMPTS) is never reached while the neighbour keeps replying. The probes are also sent through vNDSendNeighbourSolicitation, which always uses the solicited-node multicast address, even though the comment above says "unicast NS". So any node can answer them and confirm the cached MAC.

Consequences:

  • Legitimate MAC change (VRRP/HSRP failover, NIC swap, VM migration): the new owner answers every probe with S=1, O=1 and its new MAC. Every answer resets the counter, so traffic keeps going to the old MAC indefinitely. On main, the last NA won and the entry recovered.
  • Poisoning: an attacker who wins the INCOMPLETE race keeps the binding for good. The real owner's O=1 replies only keep the entry in PROBE, and the attacker can confirm each multicast probe with a matching-MAC NA.

test_prvProcessNA_ExistingEntry_OverrideUnsolicited_DiffMac_KeepsMacAndProbes asserts the reset.

Suggested minimal fix: do not reset ucNumProbes (or ucAge) when the entry is already in DELAY or PROBE. Then unconfirmed probes run out, the entry is freed, and the next resolution learns the current MAC. Separately (can be a follow-up): send NUD probes unicast to the cached MAC per RFC 4861 §7.3.3, and accept a solicited O=1 NA received during PROBE.

@kstribrnAmzn kstribrnAmzn Oct 14, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a feature enhancement which can be added later. I do not want to grow this PR with further features.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THis was small enough to include here. I've included the probe exhaustion fix. vNDPCacheSetState will now store the state before overwriting it and skip while verification is already running.

Comment thread source/FreeRTOS_ND.c Outdated
pxICMPHeader_IPv6->ucTypeOfMessage = ipICMP_NEIGHBOR_ADVERTISEMENT_IPv6;
pxICMPHeader_IPv6->ucTypeOfService = 0;
pxICMPHeader_IPv6->ulReserved = ndICMPv6_FLAG_SOLICITED | ndICMPv6_FLAG_UPDATE;
pxICMPHeader_IPv6->ulReserved = ndICMPv6_FLAG_SOLICITED | ndICMPv6_FLAG_OVERRIDE;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression (interop): FreeRTOS nodes will now drop each other's unsolicited advertisements.

This sends an NA to ff02::1 (all-nodes multicast) with ndICMPv6_FLAG_SOLICITED set. The new check in prvIsValidNa drops any NA that is sent to a multicast destination and has S=1:

if( ( pxNa->xDestinationIsMulticast != pdFALSE ) && ( pxNa->xSolicited != pdFALSE ) )
{
    break;
}

So advertisements from this function are now rejected by peers running this PR. RFC 4861 §7.2.6 requires S=0 for unsolicited advertisements. The sender is unchanged from main, but the rejection is new, so the breakage is new.

Suggested fix: send only the override flag here:

pxICMPHeader_IPv6->ulReserved = ndICMPv6_FLAG_OVERRIDE;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied your suggested fix. Thanks for the feedback :)

* Update cache eviction to include router values
* Corrected NumProbes counter reset
* Do not drop override, unsolicited messages
@kstribrnAmzn
kstribrnAmzn merged commit ece06d3 into FreeRTOS:main Oct 15, 2026
10 checks passed
@kstribrnAmzn
kstribrnAmzn deleted the ndCacheFixes branch October 15, 2026 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants