The KEP specifies that the controller will "mirror all labels from the Endpoints resource and all endpoints and ports from the corresponding subset". I'd missed that in my initial implementation, this should fix that.
This will mirror custom Endpoints to EndpointSlices to ensure that applications will not need to maintain both separately.