Simulator fails to update pod annotations when testing against a large number of nodes and pods

[ronaknnathani]

Problem statement

I am testing a scheduler configuration with 3k+ nodes and 3k+ pods. While pods are getting scheduled, they don't have the annotations for node scores. I see the following error in the simulator container:

  - Pod "pod-26c-2msbq" is invalid: metadata.annotations: Too long: must have at most 262144 bytes
E0805 03:27:35.147947       1 storereflector.go:137] failed to update the pod with retry to record store: update pod:
    sigs.k8s.io/kube-scheduler-simulator/simulator/scheduler/storereflector.(*reflector).ResisterResultSavingToInformer.(*reflector).storeAllResultToPodFunc.func1.1
        /go/src/simulator/scheduler/storereflector/storereflector.go:132

Seems like for large cluster sizes, it is hitting the annotation limit on client-side apply.

Version used

I am running v0.4.0 version of the scheduler simulator.

[sanposhiho]

This is already fixed in #418. Probably I should cut a new release soon, but there's in-flight patches to several bugs, which I'd love to include.
@ronaknnathani Can you try using the latest master meanwhile?