Test discrepancies identified with kubetest2 - cloud-provider-gcp vs k/k

[michaelmdresser]

What happened:
There are new CI jobs that use kubetest2 that are intended to serve as benchmarks to ensure parity of functionality between kubetest and kubetest2 before migrating more jobs from kubetest to kubetest2.

1. ci-cloud-provider-gcp-conformance-latest is a periodic that runs the conformance E2E tests against cloud-provider-gcp using kubetest2.

2. pull-kubernetes-e2e-gce-kubetest2 is a copy of pull-kubernetes-e2e-gce using kubetest2. This is the standard merge-blocking presubmit that runs on all PRs against k/k.

3. ci-kubernetes-gce-conformance-latest-kubetest2 is a copy of ci-kubernetes-gce-conformance-latest using kubetest2. This is a periodic that runs conformance E2E tests against k/k.

Interesting behavior is being observed in TestGrid as 1 and 3 run. ci-cloud-provider-gcp-conformance-latest and ci-kubernetes-gce-conformance-latest-kubetest2 are essentially the same job - they both run conformance tests against a version of kubernetes (k/k vs cloud-provider-gcp) with kubetest2. However, comparing the dashboard for ci-kubernetes-gce-conformance-latest-kubetest2 with that of ci-cloud-provider-gcp-conformance-latest shows a striking difference. The easiest thing to point out is the set of AdmissionWebhook tests that succeed on ci-cloud-provider-gcp-conformance-latest yet fail on ci-kubernetes-gce-conformance-latest-kubetest2. Both tests use the latest version of kubetest2 and pull the latest available Ginkgo E2E test package. Kubetest2 calls (effectively) the same script in k/k and in cloud-provider-gcp to bring the cluster up, and it does so with the same set of environment variables.

The easy conclusion is that k/k has some regression or other difference when compared to cloud-provider-gcp. Yet, the original k/k conformance job ci-kubernetes-gce-conformance-latest has no failing tests.

What you expected to happen:
The same set of tests fail on ci-kubernetes-gce-conformance-latest as do on ci-cloud-provider-gcp-conformance-latest.

How to reproduce it (as minimally and precisely as possible):
N/A, see TestGrid.

Please provide links to example occurrences, if any:
See problem description.

[michaelmdresser]

/cc @amwat

[amwat]

/cc @cheftako @cici37

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[cheftako]

/cc @jpbetz

[cheftako]

/remove-lifecycle stale

[cheftako]

/lifecycle frozen