refactor(web): use tokenization analysis to maintain delayed reversion feature 🚂 #14880

jahorton · 2025-10-02T18:02:08Z

In #14877, we kept the old computeAlignment pathway alive because it was necessary for determining the portion of the most recent keystroke's effects that should not be affected by applied suggestions. With #14880 in place, we can now replace the other scenarios previously reliant upon computeAlignment, allowing us to discontinue its use and rely fully on the new tokenization strategy.

User Testing

Run all tests via Keyman for Android. Use the keyboard in-app.

TEST_PRED_CHAIN_APPLY: Apply numerous suggestions in sequence and verify that they apply properly.

TEST_PRED_TEXT_SUGGEST_ANDROID: Verify that predictive text produces appropriate suggestions and that they apply properly.

Test with gff_amharic.
Verify that reversions work properly. (Tap backspace after applying a suggestion, then the left-most "suggestion".)

TEST_CLEAR_CONTEXT: Verify that predictive text throws no errors when clearing the context / deleting all text.

TEST_PRED_LONG_TEXT_REVERSIONS: Verify that predictive text produces appropriate suggestions and that they apply properly.

Test with sil_euro_latin.

Type the following text, in full:

applesauce and orange juice don't seem like they'd make for the best breakf

Tap the breakfast suggestion.
Tap backspace.
Tap the reversion ("breakf")
Verify that the reversion is properly applied and that Android does not signal any errors (via toasts, notifications, or otherwise).

TEST_PERIOD_APPLY_OVERWRITE: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

Type testi
Tap the testing suggestion
Then tap the . key.
Verify that testing. is the result, with no appended whitespace.

TEST_SPACE_APPLY_OVERWRITE: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

Type testi
Tap the testing suggestion
Then tap the key.
Verify that testing is the result.

TEST_PERIOD_APPLICATION: Using the sil_euro_latin keyboard, check if punctuation typed with a highlighted suggestion applies it and appears afterward.

Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
Then tap the . key.
Verify that testing. is the result, with no appended whitespace.
Tap . again and verify that it outputs a new ..

TEST_SPACE_APPLICATION: Using the sil_euro_latin keyboard, check if punctuation typed with a highlighted suggestion applies it and appears afterward.

Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
Then tap the key.
Verify that testing is the result, with no appended whitespace.
Tap and verify that it outputs a new .

TEST_PERIOD_OVERWRITE_REVERSION: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

Type testi
Tap the testing suggestion
Then tap the . key.
Verify that testing. is the result, with no appended whitespace.
Backspace over the . and verify that a proper reversion is displayed: "testi"
Apply the reversion and verify that no errors result.
Clear the context and type 3 words, then a space, and then repeat this test until reaching this line.

TEST_PERIOD_APPLICATION_REVERSION: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
Then tap the . key.
Verify that testing. is the result, with no appended whitespace.
Backspace over the ., landing at testing, and verify that a proper reversion is displayed: "testi"
Apply the reversion and verify that no errors result.
Clear the context and type 3 words, then a space, and then repeat this test until reaching this line.

TEST_PERIOD_APPLICATION_DELAYED_REVERSION: Using the sil_euro_latin keyboard, check if punctuation typed immediately after applying a suggestion erases the whitespace.

Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
Then tap the . key.
Verify that testing. is the result, with no appended whitespace.
Tap . again and verify that it outputs a new ..
Backspace over the . and verify that a proper reversion is displayed: "testi"
Apply the reversion and verify that no errors result.
Clear the context and type 3 words, then a space, and then repeat this test until reaching this line.

TEST_MULTITAPS: Verify that the . key on english_punct_rota rotates through the desired punctuation marks.

Install the keyboard package found at https://jahorton.github.io/english_punct_rota.kmp.

It may be installed via the following QR link as well:

Erase all text.
Tap the . key four times, rotating through ., ,, and ?, landing on !.
Type the following text: Testing is important . (Include the space mark.)
Repeat step 2 above and verify that only the punctuation mark is altered.

TEST_MULTITAP_OVERWRITE: Using a keyboard with a multitap punctuation rota key, check if punctuation typed immediately after applying a suggestion erases the whitespace, even on repeated taps.

Install the keyboard package found at https://jahorton.github.io/english_punct_rota.kmp.

It may be installed via the following QR link as well:
Type testi
Manually apply the testing suggestion.
Then tap the . key four times, rotating through ., ,, and ?, landing on !.
Verify that the results are testing., testing,, testing?, and the final result is testing!, with no appended whitespace.

TEST_MULTITAP_APPLICATION: Using a keyboard with a multitap punctuation rota key, check if punctuation typed with a highlighted suggestion applies it and appears afterward, even on repeated taps.

Install the keyboard package found at https://jahorton.github.io/english_punct_rota.kmp.

It may be installed via the following QR link as well:
Type testi and verify that it is selected / highlighted. (If it isn't, delete its text and retype - tap accuracy may help.)
Then tap the . key four times, rotating through ., ,, and ?, landing on !.
Verify that the results are testing., testing,, testing?, and the final result is testing!, with no appended whitespace.

TEST_SUGGESTIONS_ON_BKSP: Using Keyman for Android, verify that reasonable suggestions are displayed for text after using BKSP to reach the end of a previous word.

Type term
Type
Tap backspace once.
Multiple suggestions should be displayed, all based on 'term'.

TEST_DELAYED_ON_WORD: Using the sil_euro_latin keyboard, check if deleting characters up to the end of an applied suggestion's main body displays a reversion.

Type trials .
Tap the and suggestion.
Type testi, then tap space to automatically apply it. (Restart this test if it is not automatically highlighted somehow.)
Type ab.
Delete ab so that you end up immediately after testing.
Verify that a "testi" reversion is displayed.
Apply it and verify that the full text reverts to trials and testi.
Verify that the original suggestions are displayed, but no suggestion is automatically selected / highlighted.
Type 'a' and verify that no errors, notifications, or warnings are displayed.

TEST_DELAYED_ON_SPACE: Using the sil_euro_latin keyboard, check if deleting characters up to the end of an applied suggestion's main body displays a reversion.

Type trials .
Tap the and suggestion.
Type testi, then tap space to automatically apply it. (Restart this test if it is not automatically highlighted somehow.)
Type ab.
Delete ab so that you end up immediately after the whitespace after testing.
Verify that a "testi" reversion is displayed.
Apply it and verify that the full text reverts to trials and testi.
Verify that the original suggestions are displayed, but no suggestion is automatically selected / highlighted.
Type 'a' and verify that no errors, notifications, or warnings are displayed.

TEST_DOUBLE_DELAYED: Using the sil_euro_latin keyboard, check if two suggestions can be reverted in rapid succession.

With the context empty, type that.
Type a p, then apply the point suggestion.
Apply the of suggestion, giving you that point of .
Tap backspace and verify that a "" suggestion is displayed.
Tap the "" suggestion and verify that the text becomes that point .
Tap backspace again and verify that a "p" suggestion is displayed.
Tap the "p" suggestion and verify that the text becomes that p.

TEST_ROBUSTNESS: Spend at least 5 minutes trying to "break" predictive text and/or cause it to work improperly. Report back on any issues discovered.

keymanapp-test-bot · 2025-10-02T18:02:32Z

User Test Results

Test specification and instructions

✅ TEST_PRED_CHAIN_APPLY (PASSED) (notes)
✅ TEST_PRED_TEXT_SUGGEST_ANDROID (PASSED) (notes)
✅ TEST_CLEAR_CONTEXT (PASSED) (notes)
✅ TEST_PRED_LONG_TEXT_REVERSIONS (PASSED) (notes)
✅ TEST_PERIOD_APPLY_OVERWRITE (PASSED) (notes)
✅ TEST_SPACE_APPLY_OVERWRITE (PASSED) (notes)
✅ TEST_PERIOD_APPLICATION (PASSED) (notes)
✅ TEST_SPACE_APPLICATION (PASSED) (notes)
✅ TEST_PERIOD_OVERWRITE_REVERSION (PASSED) (notes)
✅ TEST_PERIOD_APPLICATION_REVERSION (PASSED) (notes)
✅ TEST_PERIOD_APPLICATION_DELAYED_REVERSION (PASSED) (notes)
✅ TEST_MULTITAPS (PASSED) (notes)
✅ TEST_MULTITAP_OVERWRITE (PASSED) (notes)
✅ TEST_MULTITAP_APPLICATION (PASSED) (notes)
✅ TEST_SUGGESTIONS_ON_BKSP (PASSED) (notes)
✅ TEST_DELAYED_ON_WORD (PASSED) (notes)
✅ TEST_DELAYED_ON_SPACE (PASSED) (notes)
✅ TEST_DOUBLE_DELAYED (PASSED) (notes)
✅ TEST_ROBUSTNESS (PASSED) (notes)

After changes added in #14880, one of our unit tests failed. After lots of investigation, it turns out that the `isSubstitutionAlignable` does not catch certain cases that `determineContextSlideTransform` does. The latter does need some improvement to work as a fix, but a small change lets it pick up the slack and replace the former. As it's also used during context-state transition, re-using the same method in both places gives us far more confidence that we won't have an error the second time around. Build-bot: skip build:web Test-bot: skip

…n feature

…/delayed-reversion-via-tokenization-analysis

…nstructor parameterization Build-bot: build Test-bot: test

Meng-Heng · 2025-10-17T09:11:09Z

Test Specification

MacBook 15.7.1
Android Studio
Android Emulator API 36.0
Keyman 19.0.137-alpha-test-14880

Here are the following results:

Test Results

TEST_PRED_CHAIN_APPLY (PASSED):

With sil_euro_latin keyboard
Applied numerous suggestions in sequence
Verified: they applied properly.

TEST_PRED_TEXT_SUGGEST_ANDROID (PASSED):

Installed gff_amharic
Typed ህሕ, applied ህሕቡን suggestion by tapping on it
Tapped backspace
Chose the left-most suggestion
The word returned to ህሕ.

TEST_CLEAR_CONTEXT (PASSED):

Confirmed: No error when erasing all text using backspacing & using Clear text

TEST_PRED_LONG_TEXT_REVERSIONS (PASSED):

Switched back to sil_euro_latin keyboard
Typed the following text, in full:

applesauce and orange juice don't seem like they'd make for the best breakf

Tapped the breakfast suggestion
Tapped backspace
Tapped the reversion ("breakf")
Verified:
- The reversion was properly applied.
- Android did not signal any errors (via toasts, notifications, or otherwise).

TEST_PERIOD_APPLY_OVERWRITE (PASSED):

Typed testi
Tapped the testing suggestion
Then tapped the . key
Verified: testing. is the result, no appended whitespace.

TEST_SPACE_APPLY_OVERWRITE (PASSED):

Typed testi
Tapped the testing suggestion
Then tapped the key
Verified: testing is the result.

TEST_PERIOD_APPLICATION (PASSED):

Enabled "Prediction with auto-suggestion"
Typed testi
The testing suggestion was highlighted
Then tapped the . key
Verified: testing. was the result, no appended whitespace
Tapped . again
Result: testing...

TEST_SPACE_APPLICATION (PASSED):

Typed testi
The testing suggestion was highlighted
Then tapped the key
Verified: testing was the result, with only one whitespace
Tapped
Verified: it outputted a new
Result: testing .

TEST_PERIOD_OVERWRITE_REVERSION (PASSED):

Typed testi
Tapped the testing suggestion
Then tapped the . key
Verified: testing. was the result, no appended whitespace
Backspaced over the .
Verified: only "testi" reversion was displayed
Applied the reversion
Verified: no errors result
Result: testi
Pressed "Clear text"
Repeated the above after typing 3 words:
1. Typed MengHeng is doing testi
2. Tapped the testing suggestion
3. Then tapped the . key
4. Verified: testing. was the result, no appended whitespace
5. Backspaced over the .
6. Verified: only "testi" reversion was displayed
7. Applied the reversion
8. Verified: no errors result
9. Result: MengHeng is doing testi.

TEST_PERIOD_APPLICATION_REVERSION (PASSED):

Typed testi
The testing suggestion was highlighted
Then tapped the . key
Verified:testing. was the result, with no appended whitespace
Backspaced over the ., landed at testing, verified that a proper reversion is displayed: "testi"
Applied the reversion
Verified: no errors result
Pressed "Clear text"
Repeated the above after typing 3 words:
1. Typed MengHeng did the testi
2. The testing suggestion was highlighted
3. Then tapped the . key
4. Verified: testing. was the result, no appended whitespace
5. Backspaced over the .
6. Verified: only "testi" reversion was displayed
7. Applied the reversion
8. Verified: no errors result
9. Result: MengHeng did the testi.

TEST_PERIOD_APPLICATION_DELAYED_REVERSION (PASSED):

Typed testi
The testing suggestion was highlighted
Then tapped the . key
Verified: testing. was the result, with no appended whitespace.
Tapped . again
Verified: it outputted a new .
Backspace over the ., text result was testing.
Verified: only the "testi" reversion was displayed
Applied the reversion
Verified: no errors result
Pressed "Clear text"
Repeated the above after typing 3 words:
1. Typed MengHeng finished doing testi
2. The testing suggestion highlighted
3. Then tapped the . key
4. Verified: testing. is the result, no appended whitespace
5. Tapped . again
6. Verified: it outputted a new .
7. Backspace over the ., text result was testing.
8. Verified: only the "testi" reversion was displayed
9. Applied the reversion
10. Verified: no errors result
11. Ended up with MengHeng finished doing testi.

TEST_MULTITAPS (PASSED):

Installed english_punct_rota.kmp
Erased all text
Repeatedly tapped the . key four times, rotated through ., ,, and ?, landed on !
Typed the following text: Testing is important (Include the space mark)
Repeated step 2 above
Verified: only the punctuation mark was altered.

TEST_MULTITAP_OVERWRITE (PASSED):

Typed testi
Manually applied the testing suggestion
Then repeatedly tapped the . key four times, rotated through ., ,, and ?, landed on !
Verified: the results were testing., testing,, testing?, and the final result was testing!, with no appended whitespace.

TEST_MULTITAP_APPLICATION (PASSED):

Enabled "Prediction with auto-suggestion"
Typed testi
testing was highlighted
Then tapped the . key four times, rotated through ., ,, and ?, landed on !
Verified: the results were testing., testing,, testing?, and the final result was testing!, with no appended whitespace.

TEST_SUGGESTIONS_ON_BKSP (PASSED):

With "Prediction with auto-suggestion" on
Used the english_punct_rota keyboard
Typed term; term highlighted
Typed
Tapped backspace once
Multiple suggestions were displayed, all based on 'term'; with term stayed highlighted (before and after a backspace).

TEST_DELAYED_ON_WORD (PASSED):

Switched to sil_euro_latin keyboard
Typed trials
Tapped the and suggestion
Typed testi, then tapped space to automatically applied the highlighted suggestion
Typed ab
Deleted ab and ended up immediately after testing
Verified: a "testi" reversion is displayed among testing & resting.
Applied it
Verified: the full text reverts to trials and testi
Verified:
- The original suggestions were displayed
- No suggestion was automatically selected / highlighted.
Typed 'a' --> Result: trials and testia
Verified: no errors, notifications, or warnings were displayed.

TEST_DOUBLE_DELAYED (PASSED):

With the context empty, typed that
Typed a p, then applied the point suggestion
Applied the of suggestion, result in that point of
Tapped backspace
Verified: a "" suggestion was displayed
Tapped the "" suggestion
Verified: the text became that point
Tapped backspace again
Verified: a "p" suggestion was displayed
Tapped the "p" suggestion
Verified: the text became that p.

TEST_ROBUSTNESS (PASSED):

I was not able to break the predictive text. It seemed that it is working well.

Meng-Heng · 2025-10-17T09:18:37Z

Isolating this:

TEST_DELAYED_ON_SPACE (FAILED):

Typed trials
Tapped the and suggestion
Typed testi, then tapped space to automatically applied it
Typed ab
Deleted ab and ended up immediately after the whitespace after testing
Result: trials and testing
The default suggestions showed up, not the reversion.

On step 3, if the word suggestion manually applied via a tap, the reversion worked. Here are the test:

Typed trials
Tapped the and suggestion
Typed testi, then tapped the suggested word
Result: trials and testing
Typed ab
Deleted ab and ended up immediately after the whitespace after testing
Result: trials and testing
The "testi" suggestion displayed.

jahorton · 2025-10-17T15:41:01Z

Isolating this:

* **TEST_DELAYED_ON_SPACE (FAILED):**

This works fine in standalone Web and in the Android app (under emulation) for me; can you confirm that this reproduces, @Meng-Heng?

Double-checked this and confirmed that my locally-built version has this PR's code in place.

Meng-Heng · 2025-10-20T01:41:24Z

Prerequisites

Deleted and Reinstalled Keyman
Launched Keyman and Dismissed "Start up" dialog
Enable "Prediction with auto-suggestion"

TEST_DELAYED_ON_SPACE (PASSED):

Typed trials
Tapped the and suggestion.
Typed testi, then tapped space to automatically apply it
Typed ab.
Deleted ab, now ended up immediately after the whitespace after testing
Verified: a "testi" reversion displayed.
Applied it and verified that the full text reverted to trials and testi
Verified: the original suggestions displayed, but no suggestion was automatically selected / highlighted.
Typed 'a' next to testi, and verified that no errors, notifications, or warnings displayed.

github-project-automation bot added this to Keyman Oct 2, 2025

github-project-automation bot moved this to Todo in Keyman Oct 2, 2025

github-actions bot added web/ web/predictive-text/ refactor labels Oct 2, 2025

keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Oct 2, 2025

keymanapp-test-bot bot added the epic-autocorrect label Oct 2, 2025

keymanapp-test-bot bot changed the title ~~refactor(web): use tokenization analysis to maintain delayed reversion feature~~ refactor(web): use tokenization analysis to maintain delayed reversion feature 🚂 Oct 2, 2025

keymanapp-test-bot bot added this to the A19S13 milestone Oct 2, 2025

jahorton mentioned this pull request Oct 2, 2025

fix(web): fix base context-state validation 🚂 #14883

Merged

jahorton force-pushed the feat/web/link-new-tokenization-transitioner branch from c96a604 to 0cf7930 Compare October 3, 2025 13:32

jahorton force-pushed the refactor/web/delayed-reversion-via-tokenization-analysis branch from 3541216 to 79abb65 Compare October 3, 2025 13:35

jahorton added 2 commits October 7, 2025 11:29

refactor(web): use tokenization analysis to maintain delayed reversio…

604b252

…n feature

fix(web): restore unit tests

aa477a9

jahorton force-pushed the refactor/web/delayed-reversion-via-tokenization-analysis branch from 79abb65 to aa477a9 Compare October 7, 2025 17:00

jahorton changed the base branch from feat/web/link-new-tokenization-transitioner to refactor/web/preservation-transforms October 8, 2025 16:40

darcywong00 modified the milestones: A19S13, A19S14 Oct 11, 2025

jahorton added 2 commits October 16, 2025 10:18

Merge branch 'refactor/web/preservation-transforms' into refactor/web…

1ce8b75

…/delayed-reversion-via-tokenization-analysis

fix(web): adjust unit tests to match change in ContextTokenization co…

646fdc1

…nstructor parameterization Build-bot: build Test-bot: test

jahorton force-pushed the refactor/web/delayed-reversion-via-tokenization-analysis branch from 18a0647 to 646fdc1 Compare October 16, 2025 16:10

keymanapp-test-bot bot added has-user-test user-test-required User tests have not been completed and removed user-test-missing User tests have not yet been defined for the PR labels Oct 16, 2025

jahorton mentioned this pull request Oct 16, 2025

refactor(web): utilize new tokenization-transition methodology 🚂 #14877

Merged

jahorton requested review from ermshiperete and mcdurdin October 16, 2025 16:20

jahorton marked this pull request as ready for review October 16, 2025 16:20

Meng-Heng self-assigned this Oct 17, 2025

keymanapp-test-bot bot added user-test-failed and removed user-test-required User tests have not been completed labels Oct 17, 2025

ermshiperete approved these changes Oct 17, 2025

View reviewed changes

keymanapp-test-bot bot removed the user-test-failed label Oct 20, 2025

Base automatically changed from refactor/web/preservation-transforms to epic/autocorrect October 20, 2025 13:36

jahorton merged commit 84d22fe into epic/autocorrect Oct 20, 2025
14 of 15 checks passed

github-project-automation bot moved this from Todo to Done in Keyman Oct 20, 2025

jahorton deleted the refactor/web/delayed-reversion-via-tokenization-analysis branch October 20, 2025 13:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

refactor(web): use tokenization analysis to maintain delayed reversion feature 🚂 #14880

refactor(web): use tokenization analysis to maintain delayed reversion feature 🚂 #14880

Uh oh!

jahorton commented Oct 2, 2025 •

edited

Loading

Uh oh!

keymanapp-test-bot bot commented Oct 2, 2025 •

edited

Loading

Uh oh!

Meng-Heng commented Oct 17, 2025 •

edited

Loading

Uh oh!

Meng-Heng commented Oct 17, 2025 •

edited

Loading

Uh oh!

jahorton commented Oct 17, 2025 •

edited

Loading

Uh oh!

Meng-Heng commented Oct 20, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

refactor(web): use tokenization analysis to maintain delayed reversion feature 🚂 #14880

refactor(web): use tokenization analysis to maintain delayed reversion feature 🚂 #14880

Uh oh!

Conversation

jahorton commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User Testing

Uh oh!

keymanapp-test-bot bot commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User Test Results

Uh oh!

Meng-Heng commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Specification

Test Results

Uh oh!

Meng-Heng commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jahorton commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Meng-Heng commented Oct 20, 2025

Prerequisites

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jahorton commented Oct 2, 2025 •

edited

Loading

keymanapp-test-bot bot commented Oct 2, 2025 •

edited

Loading

Meng-Heng commented Oct 17, 2025 •

edited

Loading

Meng-Heng commented Oct 17, 2025 •

edited

Loading

jahorton commented Oct 17, 2025 •

edited

Loading