Skip to content

Conversation

mixmix
Copy link
Member

@mixmix mixmix commented Apr 11, 2024

Noticed that when adding lots of messages to the database quickly, they go in right,
but when you stream the out of the db, they come out with the wrong sequence order

This is a problem for legacy replication (createHistoryStream)

Copy link

socket-security bot commented Apr 11, 2024

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package New capabilities Transitives Size Publisher
npm/tap-arc@1.2.2 None +7 950 kB architectci

🚮 Removed packages: npm/tap-arc@0.3.5

View full report↗︎


await Promise.all(
many.map((content) => {
return p(db.create)({ feedFormat: 'classic', content })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promise.all() is not the right way of doing this, you need to do them sequentially. Otherwise it's very likely indeed that they'll end up in the wrong order, because this has to do with jitdb indexing the log.

ssb-db2 is optimized for fast reads not for fast writes, so be patient if it's not the best at bulk creating msgs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote a test where I did it sequentially with a pull-stream, and it still got them wrong I think...

the fast writes thing is gonna be a challenge with Ahau importing some records. Working on that though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants