Skip to content

Commit f05019b

Browse files
author
Hendrik van Antwerpen
committed
Add script to copy and process benchmark figures
1 parent 0c66cab commit f05019b

File tree

10 files changed

+192
-780
lines changed

10 files changed

+192
-780
lines changed

crates/bpe/.gitignore

Lines changed: 0 additions & 10 deletions
This file was deleted.

crates/bpe/README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -203,16 +203,16 @@ If the requirement of correct BPE output can be relaxed, then the Greedy approac
203203

204204
Results for counting o200k tokens for random 10000 byte slices. The setup time of the interval encoder is comparable to backtracking. After setup counting of slices of the original data are approximately constant time.
205205

206-
<img src="./benches/result/reports/counting-o200k/lines.svg" style="background-color: white" />
206+
![counting runtime comparison](./benches/result/counting-o200k.svg)
207207

208208
### Encoding results
209209

210210
Results for encoding o200k tokens for random 1000 bytes. The backtracking encoder consistently outperforms tiktoken by a constant factor.
211211

212-
<img src="./benches/result/reports/encoding-o200k/lines.svg" style="background-color: white" />
212+
![encoding runtime comparison](./benches/result/encoding-o200k.svg)
213213

214214
### Incremental encoding results
215215

216-
Results for incrementally encoding o200k tokens by appending 10000 random bytes. The appending encoder is slower by a constant factor but overall has similar performance curve as the backtracking encoder encoding all data at once.
216+
Results for incrementally encoding o200k tokens by appending 10000 random bytes. The appending encoder is slower by a constant factor but overall has similar performance curve as the backtracking encoder encoding all data at once.
217217

218-
<img src="./benches/result/reports/appending-o200k/lines.svg" style="background-color: white" />
218+
![appending runtime comparison](./benches/result/appending-o200k.svg)

crates/bpe/benches/result/appending-o200k.svg

Lines changed: 52 additions & 0 deletions
Loading

crates/bpe/benches/result/counting-o200k.svg

Lines changed: 48 additions & 0 deletions
Loading

crates/bpe/benches/result/encoding-o200k.svg

Lines changed: 76 additions & 0 deletions
Loading

crates/bpe/benches/result/reports/appending-o200k/lines.svg

Lines changed: 0 additions & 232 deletions
This file was deleted.

crates/bpe/benches/result/reports/counting-o200k/lines.svg

Lines changed: 0 additions & 217 deletions
This file was deleted.

0 commit comments

Comments
 (0)