Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Oct 1, 2025

Problem

When migrating data from Cosmos DB NoSQL to MongoDB, the tool was generating new ObjectIds for MongoDB's _id field instead of preserving the source document's id field. This made it difficult to maintain consistent identifiers across systems.

For example, with source data:

{
    "id": "BSKT_1",
    "CustomerId": 111
}

The tool would create:

{
    "_id": ObjectId("659f9bbcb128ee4f195bdf8d"),
    "id": "BSKT_1",
    "CustomerId": 111
}

Solution

This PR adds an optional IdFieldName parameter to the MongoDB sink settings that allows users to specify which field from the source data should be mapped to MongoDB's _id field. The original field is also preserved in the document.

Usage

{
    "Source": "Cosmos-NoSql",
    "Sink": "MongoDB",
    "SourceSettings": {
        "ConnectionString": "AccountEndpoint=https://...",
        "Database": "cosmicworks",
        "Container": "baskets"
    },
    "SinkSettings": {
        "ConnectionString": "mongodb://localhost:27017",
        "DatabaseName": "mydb",
        "Collection": "baskets",
        "IdFieldName": "id"
    }
}

With this configuration, the result in MongoDB will be:

{
    "_id": "BSKT_1",
    "id": "BSKT_1",
    "CustomerId": 111
}

Changes

  • Added IdFieldName property to MongoSinkSettings and MongoLegacySinkSettings
  • Updated MongoDataSinkExtension and MongoLegacyDataSinkExtension to map the specified field to _id
  • Field name matching is case-insensitive
  • Added comprehensive unit tests (21 tests total, all passing)
  • Updated documentation in README.md and ExampleConfigs.md

Features

  • ✅ Optional parameter - when not specified, MongoDB generates ObjectId as usual
  • ✅ Case-insensitive field name matching
  • ✅ Preserves original field in addition to setting _id
  • ✅ Works with both MongoDB and MongoDB-Legacy (Wire v2) extensions
  • ✅ Handles various data types (string, int, long, etc.)
  • ✅ Gracefully handles null values

Fixes #[issue-number]

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://crl3.digicert.com:443/sha2-assured-cs-g1.crl
    • Triggering command: dotnet build Extensions/Mongo/Cosmos.DataTransfer.MongoExtension/Cosmos.DataTransfer.MongoExtension.csproj (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>mapping cosmos-nosql id to mongodb _id</issue_title>
<issue_description>I'm going to migrate my data from cosmos-nosql to mongodb .
the tool doesn't map the cosmos id field to mongodb _id field.
for example:
cosmos data:

 {  "id": "BSKT_1",
    "CustomerId": 111,
}

the mapped data in mongo

{
"_id": ObjectId("659f9bbcb128ee4f195bdf8d"),
"id" : "BSKT_1",
"CustomerId": 111
}

how can I resolve this issue?</issue_description>

<agent_instructions>This issue is asking for being able to set a custom _id value for the mongo _id field as supposed to a uniquely generated one. Users should be able to indicate this behavior via the sink settings. I'm thinking the ideal behavior would be to provide the key in the data that should be mapped to the _id column. We still want to add the key in the data as well since the _id is unique to mongodb. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

Fixes #104

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 3 commits October 1, 2025 12:44
Co-authored-by: philnach <19275540+philnach@users.noreply.github.com>
Co-authored-by: philnach <19275540+philnach@users.noreply.github.com>
Co-authored-by: philnach <19275540+philnach@users.noreply.github.com>
Copilot AI changed the title [WIP] mapping cosmos-nosql id to mongodb _id Add IdFieldName parameter to MongoDB sink extensions for custom _id mapping Oct 1, 2025
Copilot AI requested a review from philnach October 1, 2025 12:52
Copilot finished work on behalf of philnach October 1, 2025 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mapping cosmos-nosql id to mongodb _id

2 participants