[FIX] handle race conditions that could lead to job running twice #859
+34
−17
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Do not wait for locks and start jobs that are not in the expected state.
In a nutshell this PR replaces two
SELECT FOR UPDATEbySELECT FOR UPDATE SKIP LOCKED. This is because if the job to run is already locked or not in the expected state there is no need to wait: it means the job is being executed by another worker already.Also since there is commit between the check that the job is in enqueued state and set started, and the actual start of execution, there was window there for two workers to start the same job in some rare situations. This PR should avoid this case.
Maybe fixes #858