Failure: Page already exists

[GruenSein]

Hi all,

as my usage of this nice tool intensifies, I have stumbled across another issue that I cannot seem to fix. When I push my documentation to confluence, I receive the following error:

atlassian DATA:
unexpected rest response detected; retrying in 5 seconds...

sphinxcontrib.confluencebuilder error:

---
Unsupported Confluence API call

An unsupported Confluence API call has been made. See the following
details for more information:

REQ: POST
RSP: 400
URL: https://voith.atlassian.net/wiki/
API: rest/api/content
DATA: {
  "statusCode": 400,
  "data": {
    "authorized": true,
    "valid": true,
    "errors": [],
    "successful": true
  },
  "message": "com.atlassian.confluence.api.service.exceptions.BadRequestException: A page with this title already exists: A page already exists with the same TITLE in this space"
}
---

While the response is relatively clear, it unfortunately does not tell me which page resulted in this error. Additionally, I am a bit confused how this can be because:

• I wiped the space before trying to publish when I first encountered this issue
• I was under the impression that existing pages were supposed to be overwritten
• The issue seems to occur with the same page every time but the error message does not tell me which page it is
• As I used the API myself before switching to confluence builder, I tried to take care that all pages have individual names

Is there some way to further investigate? Also, being able to skip pages with errors and continue would already help a lot. I have almost 10k pages about configuration files and the one failing is around number 3k leading to a situation in which 7k configs remain undocumented. Is this possible?

Update:

I have actually spent the time watching the publishing process to figure out which page is to blame. The page in question is also 800kb in size and therefore the biggest one I have. Is it possible that this is a problem? It is always this page that seems to cause the issue. It is interesting that there is a retry before and then it states that the page already exists. Is it possible that the first attempt actually succeeds and then the retry fails?

Update 2:

After increasing the time out to a minute, my documentation is correctly being published again. This leads me to believe that there is some issue with the retry being initiated even though the primary request ends up succeeding. The other issue is that this completely cancels the publishing process as a whole

[jdknight]

Is there some way to further investigate?

Enable Sphinx to run with a higher verbosity may provide more insights.

Is it possible that the first attempt actually succeeds and then the retry fails?

This very well may be the case. Confluence sometimes reports 500 errors which request from this extension may perform a generic retry or some special retry events. There is a use case where a title conflict should fallback to a page update. My random guess (from a fresh state) maybe that some page (maybe the large one you mentioned) is being published, accepted by Confluence but then the instance reports a 500-like error. An automatic retry is made:

unexpected rest response detected; retrying in 5 seconds...

Which will repeat the same request to create a "new" page instead of updating and Confluence reports the title already existing. The extension then retries for find the title that should exist to perform a page update, but maybe it fails to find it? This will in turn re-raise the original exception that a duplicate title exists.

Being able to skip pages with errors and continue would already help a lot... Is this possible?

There does not exist such a capability at this time. We can try to introduce something like this, but it might get a bit complex (e.g. if a page fails to publish, how to gracefully handle not publishing nested pages and attachment that could later fail as well).

The extension is designed in a way where Sphinx can be re-run and only newer pages are updated. With respect to Confluence, there should be a hash check made on the page's metadata to verify if a new publish is warranted. However, this does not apply if you plan to do a fresh state each time.

If the issue only occurs on one or a subset of "large" documents in the entire set, a workaround could be to run Sphinx twice -- once to publish most of the documents as normal and then re-run with only publishing the large one with a larger delay. Granted, not ideal.

The page in question is also 800kb in size and therefore the biggest one I have. Is it possible that this is a problem?

In theory, page sizes are not a concern for the extension itself. It is just a matter of how Confluence handles it. This extension tries to handle various undocumented and unexpected states reported by Confluence instances, but the process of handling all the corner cases is less then ideal.

---

If you can reproduce the error, running with confluence_publish_debug = 'headers' might give more information of where the retry occurs and what requests were made before. This could help us to test and improve a workaround for such a scenario.

[GruenSein]

Hi @jdknight

thanks for your explanation. The wiping of the space was the hackish solution to avoid following errors which seem to pop up randomly (I cannot make out a pattern, at least). Only updating changed pages is preferable, orf course. The error that occurs randomly when not whiping the space is the following:

REQ: PUT
RSP: 409
URL: https://voith.atlassian.net/wiki/
API: api/v2/pages/1254493933/properties
DATA: {
 "errors": [
 {
 "status": 409,
 "code": "CONFLICT",
 "title": "Encountered conflict updating property with id [1252870446] on content id [1254493933] of type [page]",
 "detail": null
 }
 ]
}

or

Unsupported Confluence API call

An unsupported Confluence API call has been made. See the following
details for more information:

REQ: POST
RSP: 400
URL: https://voith.atlassian.net/wiki/
API: api/v2/pages
DATA: {
  "errors": [
    {
      "status": 400,
      "code": "BAD_REQUEST",
      "title": "A page with this title already exists: A page already exists with the same TITLE in this space",
      "detail": null
    }
  ]
}

I am not sure what that means. My initial assumption was that changes take precedence over the current state which is why I am having a hard time why there can even be a conflict. Shouldn't the property simply be overwritten?

After whiping the space, the apparent timing issue can occur which I tried to fix with the longer timeout. The clear downside is that the documentation is gone while the very slow publishing process is active. Would it be possible to query pages before retries to make sure that they are really not present? IMHO, it is sad that the confluence API is so unreliable to necessitate this but maybe this approach could remedy the situation.

I am rerunning the publication on a freshly wiped space right now and will update this posts if I receive a more detailed error message.

EDIT:

Here's a failure that even occured with a freshly wiped space:

DEBUG:urllib3.connectionpool:https://voith.atlassian.net:443 "GET /wiki/api/v2/pages?body-format=storage&space-id=21266439&status=current&title=Herdecke%2FU5 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://voith.atlassian.net:443 "GET /wiki/api/v2/pages?body-format=storage&space-id=21266439&status=archived&title=Herdecke%2FU5 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:https://voith.atlassian.net:443 "POST /wiki/api/v2/pages HTTP/1.1" 504 None
DEBUG:urllib3.connectionpool:https://voith.atlassian.net:443 "POST /wiki/api/v2/pages HTTP/1.1" 504 None
DEBUG:urllib3.connectionpool:https://voith.atlassian.net:443 "POST /wiki/api/v2/pages HTTP/1.1" 504 None

sphinxcontrib.confluencebuilder error:

---
Unsupported Confluence API call

An unsupported Confluence API call has been made. See the following
details for more information:

REQ: POST
RSP: 504
URL: https://voith.atlassian.net/wiki/
API: api/v2/pages
DATA: <not-or-invalid-json>
---

[jdknight]

There appears to be a couple of error situations being mentioned here.

Encountered conflict updating property with id

Properly-related issues have been a bit of a pain since their introduction in extension. The original Confluence API allowed content like page labels to be applied as part of a page update. Now that they need to be served by another (property) request, there have been some unexpected scenarios observed. For example, creating a new page and then waiting* to publish a proper update for a page, Confluence may or may not have already created that property (either it being forced by design or maybe through some other marketplace addon). So, we attempt to put our property update and it may fail; which then we try to be nice by asking Confluence what it thinks it should be, make our changes with version bump and re-submit again. It appears to have been working with some success, but maybe your situation it is not.

One thing I did notice is we only apply property retry events for all properties except an extension-specific one. I have updated the implementation to ensure all property updates attempt the crude retry attempt to see if it might prevent some publishing issues. If interested, you can try to install the development version of this extension to see if it helps.

A page with this title already exists

My only other guess where this might be happening is a situation where a child page has the same title has a parent page and Confluence denies the child page publish since it would result in an invalid hierarchy. But I cannot recall what come from that investigation.

Shouldn't the property simply be overwritten?

Yes... but properties (like pages/etc.) need to have a version increment to be accepted. So, if a proper entry was versioned 41, the next update needs to include a property with a version hint of 42. There might be another scenario where conflict events are triggered that I am not yet aware of.

Would it be possible to query pages before retries to make sure that they are really not present?

Funny enough, we already do this in some fashion. While it was for another use case, if a new page update fails with a "title already exists" error, it should perform another page search and then do another update (e.g. 500 error on failure, retry event reports page exist, which gets caught in a re-check if the page exists with a name search, and if found, retry the page update). Although, I guess for this error case, this might result in the large/troubled page being attempted to be published twice... but I do not immediately see any reported output that shows that is happening.

REQ: POST
RSP: 504

There is not much this extension can do for this use case. It appears that either the chance of this publish event occurred during limited availability event for Confluence Cloud or that you may have been publishing in a proxy environment which the proxy was not playing nice.

There is a maximum of three attempts made here for the 500-series error. With that in mind, I have introduced two new configuration options confluence_publish_retry_attempts and confluence_publish_retry_duration which can give a more tailored experience for retry scenarios (at the time of writing, available in the development tree). You can possible try with the options changed to the following, if availability when publish is only limited in a minute window:

confluence_publish_retry_attempts = 10
confluence_publish_retry_duration = 10

... I receive a more detailed error message ...

I appreciate the additional logs and content. It does help give a more detailed picture of where some areas in the code are having trouble. Instead of, which what I assume you logged with urllib3, it may be better to re-run your attempt with the headers option. It should give a much better understanding of what explicit API calls and header content used/reported, which might reveal something else:

confluence_publish_debug = 'headers'

However, no worries if not desired/possible.

[jdknight]

May I recommend trying with the latest release. Corrections have been made in properly raising on certain API failures which may have mislead to certain reported error states. In addition, some additional workarounds have been added to help support retry attempts to rebuild conflicting property values on Confluence pages.

[GruenSein]

Hi there, sorry for the delayed response. I have updated to the newest version and things went well for a while. However, some failures still occur. As the confluence builder runs in a CI pipeline, it is a bit bothersome to get the logs. Due to the large number of pages, the log cuts off relatively early. I will try the proposed options and report back, though. Thanks for your continued efforts!

Last Messages
=============

    hydro/Machines/Foyers_U2/VS_U2_Tu.DrTu_Vib.Abs.Y.RMS.Mean_OpMode.SCP_[Foyers_U2]


    publishing documents... [ 19%]
    hydro/Machines/Foyers_U2/HSI_Implementation_Review_for_Foyers_U2


    publishing documents... [ 19%]
    hydro/Machines/Herdecke_U5

Loaded Extensions
=================

* sphinx.ext.mathjax (8.2.3)
* alabaster (1.0.0)
* sphinxcontrib.applehelp (2.0.0)
* sphinxcontrib.devhelp (2.0.0)
* sphinxcontrib.htmlhelp (2.1.0)
* sphinxcontrib.serializinghtml (2.0.0)
* sphinxcontrib.qthelp (2.0.0)
* sphinx.ext.autosectionlabel (8.2.3)
* sphinx.ext.autodoc.preserve_defaults (8.2.3)
* sphinx.ext.autodoc.type_comment (8.2.3)
* sphinx.ext.autodoc.typehints (8.2.3)
* sphinx.ext.autodoc (8.2.3)
* sphinx.ext.napoleon (8.2.3)
* sphinxcontrib.confluencebuilder (2.12.0)
* sphinx.ext.autosummary (8.2.3)
* sphinx_toolbox.collapse (3.9.0)

Traceback
=========

      File "C:\Users\RUBBERA\micromamba\envs\adison_313\Lib\site-packages\sphinxcontrib\confluencebuilder\rest.py", line 377, in put
        raise ConfluenceBadApiError(rsp.status_code, errdata)
    sphinxcontrib.confluencebuilder.exceptions.ConfluenceBadApiError:
    ---
    Unsupported Confluence API call

    An unsupported Confluence API call has been made. See the following
    details for more information:

    REQ: PUT
    RSP: 409
    URL: https://voith.atlassian.net/wiki/
    API: api/v2/pages
    DATA: {
      "errors": [
        {
          "status": 409,
          "code": "CONFLICT",
          "title": "Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update CONTENT set HIBERNATEVERSION=?, TITLE=?, LOWERTITLE=?, VERSION=?, CREATOR=?, CREATIONDATE=?, LASTMODIFIER=?, LASTMODDATE=?, VERSIONCOMMENT=?, PREVVER=?, CONTENT_STATUS=?, NAVIGATIONTYPE=?, PAGEID=?, DRAFTTYPE=?, SPACEID=?, CHILD_POSITION=?, PARENTID=?, MESSAGEID=? where CONTENTID=? and HIBERNATEVERSION=?; nested exception is org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: update CONTENT set HIBERNATEVERSION=?, TITLE=?, LOWERTITLE=?, VERSION=?, CREATOR=?, CREATIONDATE=?, LASTMODIFIER=?, LASTMODDATE=?, VERSIONCOMMENT=?, PREVVER=?, CONTENT_STATUS=?, NAVIGATIONTYPE=?, PAGEID=?, DRAFTTYPE=?, SPACEID=?, CHILD_POSITION=?, PARENTID=?, MESSAGEID=? where CONTENTID=? and HIBERNATEVERSION=?",
          "detail": null
        }
      ]
    }
    ---


The full traceback has been saved in:
C:\Users\RUBBERA\AppData\Local\Temp\sphinx-err-a95k_rkp.log

To report this error to the developers, please open an issue at <https://github.com/sphinx-doc/sphinx/issues/>. Thanks!
Please also report this if it was a user error, so that a better error message can be provided next time.

[jdknight]

A less than ideal API state. Looks like an attempt to update a page that does not exist. I'm not sure what to say right now on the best way forward, aside from trying to re-run with additional logging. For example, enabling debug logs to trace what API calls where performed, to see why the extension believe a page update is warranted over a new page attempt:

  (in conf.py)
confluence_publish_debug = 'headers-and-data'  # (or `headers`)
  (or, your environment)
export CONFLUENCE_PUBLISH_DEBUG="headers-and-data"

Maybe appending something such as the following in CI to get a full traceback could help:

for %f in (%LocalAppData%\Temp\sphinx-err-*.log) do type "%f"

[jdknight]

Any updates?

[GruenSein]

Hi James, sorry for the long time with no update. Unfortunately, I am occupied otherwise. Currently, I can only offer some unspecific observations. Logging all header and data for my current implementation might be overkill since we are talking about almost 10k pages.

Observations:

• Atlassian overhauled their UI so I think some work might have gone into the backend
• I updated to the latest confluence builder a few weeks ago which seems to work better but still fails randomly
• wiping a space does not seem to work anymore
• In the CI/CD cli output, I cannot see any events as even the preprocessing of the large number of pages exhausts the viewable log length (think label warnings and so on)
• your logging suggestion appears to assume a windows host. My CI/CD is a linux container. I am also afraid of having a log file that ends up being 100s of MBs in size. Is there a way to pipe only failures to a log file? If so, how can I implement it? Once it exists, it is no problem to store it as an artifact. I will, however, need to redact all content as this is not ment for the public.

[jdknight]

Three things that might be helpful.

First, a configuration that has not been discussed is confluence_publish_delay. This is a float value which can be used to tailor the rate of API requests made to a Confluence instance (in addition to any rate limiting Confluence's API asks the extension to adhere to). If the Confluence instance is not happen with the large amount of API requests, slightly tweaking this value higher might help.

For large logs, temporarily limiting to only the last N lines might help capture the last messages. For example:

<sphinx-build-call> | tail -n 1000

For any traceback logs Sphinx may generate, the artifact path would be the following:

/tmp/sphinx-err-*.log