CloudantV1.postPartitionSearch does not return order field in Node.JS

[PatrikHysky]

Describe the bug

CloudantV1.postPartitionSearch method should return an order field with each result, containing the relevance score. This works when using the Python library, however, it does not work in Node.JS using @ibm-cloud/cloudant package.

To Reproduce

In Node.JS, create an API that uses CloudantV1.postPartitionSearch. The result is missing the order field. Example response in Node.JS:
{"totalRows":4,"bookmark":"g2wAAAA...","rows":[{"fields":{"FormNumber":"SG24-8573-00"},"id":"rb:SG24-8573-00"}...
Using the same database, same partition, same method and same query, an equivalent Python script correctly returns the order field for each result.
Example response in Python:
{'total_rows': 4, 'bookmark': 'g2wAAAA...', 'rows': [{'id': 'rb:SG24-8573-00', 'order': [0.910089373588562, 2766], 'fields': {'FormNumber': 'SG24-8573-00'}...

Expected behavior

As per Cloudant documentation:
The response to a search query includes an order field for each of the results. The order field is an array where the first element is the field or fields that are specified in the sort parameter.

Screenshots

Must gather (please complete the following information):

• SDK version 0.12.7
• Node.js Version 20
• Name of service that you're trying to use: @ibm-cloud/cloudant in Node.JS
• method CloudantV1.postPartitionSearch

Additional context

No workaround found using the Node.JS library. The order field is simply missing from the response.

[ricellis]

Thanks for reporting this.
The omission of order in the SDKs is intentional as it is marked deprecated in the Cloudant API docs.

Regarding the Python case, it is not present in the SearchResultRow class model. The ability to access the order field in Python is a side-effect of using the raw response dict and the fact that the server does still include order in the response.

Please could you describe your use case for the order field? Are you using it for the geographical search distance? It would help us to understand whether we need to reevaluate its omission.

[ricellis]

I should say as well that if you want to workaround in the meantime you can use postPartitionSearchAsStream to access the raw server response.

[PatrikHysky]

The relevance score (which is what the order field contains) is useful in storing the original "by relevance"sort order and optimizing search results in search APIs. it is indispensible In scenarios where a search API queries multiple databases and combines hits based on the respective relevance scores. Without the relevance scores, it is impossible to syndicate the search results into one list "by relevance."

I looked at the Cloudant API docs and the example under postSearch still contains the order field:

Example SearchResult response.

{
  "bookmark": "g2wAAAEPeJzLYWBgYMlgTmGQT0lKzi9KdUhJMtFLykxPyilN1UvOyS9NScwr0ctLLckBKmRKZEiy____f1YGk5v9iztMQCGGRGZU3Wa4dSc5AMmkepgBlxdXgQ1gINr6PBaQhgYgBTRjP9iQDwxgkMhEtCsghhyAGAJxybeak2BDGLMAi8FWFQ",
  "rows": [
    {
      "fields": {},
      "id": "exampleid",
      "order": [
        "0.8410536050796509",
        "1"
      ]
    }
  ],
  "total_rows": 1
}

I don't understand why this useful feature was removed but I hope it can be restored.

[rnewson]

The relevance score is calculated from a number of factors that make cross-database comparisons incorrect. For example, the number of times a query term appears in the matching documents and the number of times that term appears in the entire index. The information you would need to apply a cross-database normalisation to account for those factors is not available through the API.

A side-note, but this is the reason each index defined in the same design document is built as separate Lucene indexes (c.f map/reduce views in a design document which all build into the same index), in order that the relevance calculation is not contaminated by an index of potentially radically different content.