fix(operator): Blocking gRPC calls

[domsolutions]

Motivation

There was an issue where ServerNotify gRPC call blocks and doesn't allow the Server CR reconcile loop to complete. This blocks other Server CRs from reconciling, thus scheduler will not be notified about them and Servers will remiain in an unready state

Summary of changes

• changed from gRPC in-built expoential backoff retry (retrying 100 times which is effetively forever) to a constant backoff, reconcile will manage exponential backoff if it eventually fails
• ctx generally should not be part of state of struct for request - changed to be passed as func arg
• added few missing places where ctx was not being used in IO calls

Note:

A further impovement, to mitigate blocking is to allow for multiple reconcile workers, separate story has been created for this.

Checklist

• Added/updated unit tests
• Added/updated documentation
• Checked for typos in variable names, comments, etc.
• Added licences for new files

Testing

[lc525]

lgtm, with some minor comments/questions. A much better end-state than where we started.