Skip to content

Commit 6fb0b8d

Browse files
fix: Catch AssertionError in countTokens() (#445)
1 parent 7d89650 commit 6fb0b8d

File tree

1 file changed

+3
-2
lines changed

1 file changed

+3
-2
lines changed

src/main/java/ee/carlrobert/codegpt/EncodingManager.java

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -59,9 +59,10 @@ public int countMessageTokens(String role, String content) {
5959

6060
public int countTokens(String text) {
6161
try {
62+
// #444: Cl100kParser.split() throws AssertionError "Input is not UTF-8: "
6263
return encoding.countTokens(text);
63-
} catch (Exception ex) {
64-
LOG.warn(ex);
64+
} catch (Exception | Error ex) {
65+
LOG.warn("Could not count tokens for: " + text, ex);
6566
return 0;
6667
}
6768
}

0 commit comments

Comments
 (0)