Training data ordering bias

[lukebarton]

Depending on which order the examples are provided, the model can be extremely biased - is this expected?

https://colab.research.google.com/drive/1nkzYd9m6Q2aIx5v3coepbmvC-mipGeSm

swim, expect zing
[('zoob', 0.7240370727788529), ('zing', 0.275962927221147)]
[('zing', 0.5323253870010376), ('zoob', 0.4676746129989624)] zoobs (cats) first
meow, expect zoob
[('zing', 0.5290462532355265), ('zoob', 0.47095374676447355)]
[('zoob', 0.8305825838620241), ('zing', 0.16941741613797592)] zoobs (cats) first

If adding the zoob label to the classifier first, we can see a ~26% and ~36% swing in confidence, giving completely different answers depending on the training input ordering.

[codelion]

Thank you for reporting this issue. This issue is a bit subtle, to support online and active learning it is not possible to know the order of labels and training data so there will be some order dependency in the learning by the neural network. However within the same batch we can do some consistent sorting to avoid this issue. I will try and address it in #47 as well.

[codelion]

After doing the fix for ordering I noticed there is still this issue for the example you had since the underlying model doesn't seem to be mapping the similarities to concepts.

When testing single words against training examples, here's what different models produce:

google-bert/bert-large-cased (problematic)

Word similarities:
  swim → fish: 0.33, swim → cat: 0.16
  meow → fish: 0.20, meow → cat: 0.34

Sentence similarities for "swim":
  "fish like to swim": 0.06
  "cats like to meow": 0.07
  "cats are felines": 0.15 (highest!)

The model gives very low similarities overall and sometimes higher similarity to wrong categories.

bert-base-uncased (better but still issues)

Word similarities:
  swim → fish: 0.92, swim → cat: 0.91
  meow → fish: 0.92, meow → cat: 0.94

Too similar across all categories - poor discrimination.

sentence-transformers/all-MiniLM-L6-v2 (works correctly)

Word similarities:
  swim → fish: 0.75, swim → cat: 0.67
  meow → fish: 0.65, meow → cat: 0.77

Clear semantic distinctions with correct associations.

Test Results

With your exact example:

BERT models:

• ❌ "swim" → "zoob" (cats) - WRONG
• ❌ "meow" → "zing" (fish) - WRONG

sentence-transformers/all-MiniLM-L6-v2:

• ✅ "swim" → "zing" (fish) - CORRECT
• ✅ "meow" → "zoob" (cats) - CORRECT