I think that it is worth to consider migrating to https://kubernetes.io/docs/reference/using-api/server-side-apply/ instead of current client-side patching logic (please correct me if I'm wrong).