This matches the recent updates to the EndpointSliceTracker for the EndpointSlice controller in #99345 that accomplished the same thing.
This mirrors an earlier fix to the EndpointSlice controller. I'll make a follow up PR to move this component to a shared package, but that seems beyond the scope of a bug fix PR.
This will mirror custom Endpoints to EndpointSlices to ensure that applications will not need to maintain both separately.