Policy bot status update issue

[BoseKarthikeyan]

We are using policy bot for checking the PR raised is passing the required conditions so that once the conditions are met then bulldozer bot should merge this PR. For one of our repository we are observing an issue that even when all checks are passed the policy bot is showing 0/1 rules are approved (seen in policy bot deployment pod logs)

Also when we click on the policy bot status which shows has pending only then the status gets updated to 1/1 rules approved in pod logs and then the PR gets merged.

Policy bot pod logs-->

{"level":"info","rid":"d04u24grha0c73ejg7hg","github_event_type":"pull_request_review","github_delivery_id":"84c2217e-20d9-11f0-954f-3745e41a5a0e","github_installation_id":61001852,"github_repository_owner":"ford-cloud","github_repository_name":"cloud-platform","github_pr_num":253,"github_sha":"be0eee9e5859882fd10bbdf30cde690eae5511d8","time":"2025-04-24T06:58:28.439081604Z","message":"Setting "policy-bot: main" status on be0eee9e5859882fd10bbdf30cde690eae5511d8 to pending: 0/1 rules approved"}
997

Screenshot from Github:

ScreenShot from Policy-bot UI:

Screen shot PR after clicking the Policy-bot status check:

.policy.yaml file:

policy:
  approval:
    - and:
        - Cycode Secrets check passed
        - All commits are signed
        - or:
            - team team has approved
            - and:
              - only app-registration files have changed
              - CI / registration status check is successful
              - target is main branch
              - Validation is successful

approval_rules:
  - name: All commits are signed
    requires:
      conditions:
        has_valid_signatures: true

  - name: only app-registration files have changed
    requires:
      conditions:
        only_changed_files:
          paths:
            - '^app-registrations/.+/.+\.(tf|tfvars|json|md)$'
    options:
      invalidate_on_push: true

  - name: team has approved
    requires:
      count: 1
      teams:
        - "infra-auto-developers"
        - "cloudtooling-admins"
    options:
      invalidate_on_push: true
      request_review:
        enabled: false
        mode: teams

  - name: "CI / registration status check is successful"
    requires:
      conditions:
        has_successful_status:
          - Pipelines as Code CI / app-registration
    options:
      invalidate_on_push: true

  - name: "CLDCTL-Validation is successful"
    requires:
      conditions:
        has_successful_status:
          - CLDCTL-Validation
    options:
      invalidate_on_push: true

  - name: target is main branch
    requires:
      conditions:
        targets_branch:
          pattern: "^main$"

  - name: Cycode Secrets check passed
    requires:
        conditions:
          has_successful_status:
            - "Cycode: Secrets"
    options:
      invalidate_on_push: true

[asvoboda]

This sounds like the github app might not have all the relevant webhooks enabled to send to the bot. Policy-bot will re-evaluate once you load the ui. Make sure you follow the readme for setup https://github.com/palantir/policy-bot?tab=readme-ov-file#deployment, and validate from the github ui that events are being sent as you expect.

[BoseKarthikeyan]

@asvoboda, We are using the same GitHub app (bot) for another repository where it is working as expected. The only difference is that in the other repository, someone from the team has to approve, whereas in this repository, approval is done by a bot. Additionally, please note that when someone clicks on the policy check, which appears as pending in the policy-bot UI, the PR gets merged.

[asvoboda]

What does the bot do to post a review? Is it via a github review, a comment, or something else?

[wmaben]

Hey @asvoboda , In this case we have a github app setup for approval similar to this - https://github.com/palantir/go-githubapp, where we add a webhook to the repo and using the payload(PR events) the go app will check the status of PR if all checks ran success then it will approve (shown in below tagged image).

Also, we have another repo where we see this issue of policy bot not updating the status untill we manually click the PR check and then the PR is merged, in this case we have a task in pipelinerun which checks if all tasks are success and the other dependent checks are complete then using the gh command is used to approve the PR
gh pr review $GIT_URL/pull/$GIT_PR_NUMBER --approve

[bluekeyes]

As Andrew mentioned, viewing the details page forces Policy Bot to re-evaluate the current state of the PR. If nothing else changes about the PR, but viewing the details changes the approval status, it's a good indication that something went wrong when handling the webhook for the latest event.

Some reasons this could happen:

• The GitHub App configuration is missing one or more required webhook events, so GitHub never sent the webhook
• Something happened between GitHub and Policy Bot and it never received the webhook
• Policy Bot received the webhook, but something failed during processing before it could post a status

From the information shared so far, it's kind of hard to tell which situation is happening here. Would you be able to share:

• The version of Policy Bot you are running
• Whether this is using GitHub.com or GitHub Enterprise, and if using GitHub Enterprise, the version
• The full logs for one of the problematic PRs; you may need to redact internal information like organization and repository names

It may also be worth enabling debug logging temporarily to collect more information while you reproduce the problem.

[wmaben]

Hi @bluekeyes, We have verified the Github App configuration and looks all good. So sharing the details requested below along with the debug logs for one of the PR number 260.

POLICYBOT_VERSION=1.36.4

Attached the debug logs--->

policy-bot-debug-logs.json

[wmaben]

@bluekeyes, Did you find a chance to check the debug logs for any particular issue that the policy check is on hold even after meeting all the requirements ? Do you think the bot we are using to approve the PR's is not having sufficient permission or something..

[bluekeyes]

Sorry, I haven't had a chance to review these logs in detail yet. The permissions of your approval bot itself shouldn't matter here - Policy Bot can't tell what those permissions are and the fact that viewing the details page evaluates correctly means that the comments are counted as approval.

[bluekeyes]

Hey @wmaben and @BoseKarthikeyan, I reviewed the logs, screenshots, and information provided here and I'm still a bit confused about what's happening. From what I can gather, you are experiencing this problem on two different repositories:

1. Repository A - approval from your custom GitHub App should make Policy Bot pass
2. Repository B - approval from a pipeline run using the gh CLI should make Policy Bot pass

Then there is some third repository, Repository C, that requires human approval and works correctly.

Is that right? Do all of these repositories use the same policy file or do they have different policies?

Looking at the logs for PR 260, which I think is from Repository A, two things stand out to me. First, the only rule that requires approval ignores the approval from cldctl-validation[bot]:

{..., "rule":"cloud-enablement-team team has approved", "user":"cldctl-validation[bot]", "message":"ignoring approval by non-required user"}

Which rule of the policy is the approval from this bot supposed to satisfy?

Second, one of the other rules is missing a required status:

{..., "rule":"Pipelines as Code CI / app-registration status check is successful", "message":"found 0/1 required conditions"}

Your screenshot suggests that the Pipelines as Code CI / app-registration status appears eventually, but I don't see that happening in the logs here.

Are there any additional logs? If viewing the details page updates the policy-bot status correctly, can you share the logs from that as well? If you want to include all of the server logs that happen from the time you open a PR to when the PR merges, I can filter it down given the PR number to make sure nothing is missed.

Finally, are you trying to set this up policy for the first time or did it used to work and then broke at some point?

[wmaben]

@bluekeyes, All these repository(A,B &C) uses different policy files and also we observe this issue intermittently on A & B repos where sometime when all checks are passed the PR is merged automatically by bulldozer bot. But most of the times during this issue even after all checks getting passed and policy status will not turn green untill someone manually clicks on the status through browser.

I have attached the complete logs of a PR# - 309 (including the logs after clicking the details page, which merges the PR)

Image 1 ---> for #309 PR the policy check shows - Waiting for status to be reported — 0/1 rules approved

Image 2 --> all checks are green on policy UI when clicked on details page (so ideally it show as waiting status at this point)

Image 3 --> After clicking on the details page manually the PR# - 309 checks get updated and PR is merged.

Complete logs for PR >>

policy-PR-309.json

[bluekeyes]

Thank you for the updated logs. Assuming these logs are complete, it seems like the problem is that Policy Bot never performed an evaluation for the successful Pipelines as Code CI / app-registration status. Is this a traditional commit status or is it a check run?

Three evaluations happened for this PR: on the initial pull_request event, on a pull_request_review event, and then loading the details page. On the first two events, the Pipelines as Code CI status was not present. Seven minutes later when viewing the details page, the status was present on the commit.

While we see 11 status and check_run events in the logs between the last two evaluations, these don't lead to evaluations. Policy Bot doesn't log much about these unprocessed events, but the handler code is relatively simple so I think the only reasons these might be skipped are:

• A traditional commit status is not success
• A check run is not completed with a conclusion of success
• A check run is completed with a conclusion of success, but for some reason the event included no associated PRs
• Policy Bot did not receive the event

The next thing I'd try to find is a record of the webhook delivery. If you have access to the GitHub App settings for your Policy Bot app, you can find these on the "Advanced" tab. You'll want to look for the webhook delivery corresponding to the successful Pipelines as Code CI status. Usually you can find these by looking at the timestamps, but you can also get the delivery IDs from the Policy Bot logs - look for lines where the github_event_type field is status or check_run and then copy the github_delivery_id value.

From the webhook delivery, we should be able to see:

• The payload body, which can help confirm things like the pull requests included in a check_run event
• The response from the server - this should be successful (a 200) and include a request ID that matches up with one or more log lines

For the case where Policy Bot did not receive the event, how do you deploy Policy Bot? Are there multiple instances? Does it run behind some kind of proxy/router/load balancer?

[bluekeyes]

I also released Policy Bot 1.37.0, which adds some additional debug logs for check_run events. It may be worth upgrading to this version if we don't discover anything from looking at the webhook deliveries.