readme: add model card

README.md

@@ -1,3 +1,107 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+---
+
+Anthropic's client side tokenizer.
+
+Accuracy compared to actual Claude 3 Haiku tokenizer (Claude 3 family has the same tokenizer):
+
+```python
+Tokenization results saved to __temp.txt.tokens
+Text: Hello, world! This is a simple...
+Actual tokens: 17
+Predicted tokens: 10
+Accuracy: 58.82%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: The quick brown fox jumps over...
+Actual tokens: 19
+Predicted tokens: 10
+Accuracy: 52.63%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: In computer programming, a hel...
+Actual tokens: 29
+Predicted tokens: 21
+Accuracy: 72.41%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: Artificial intelligence (AI) i...
+Actual tokens: 30
+Predicted tokens: 24
+Accuracy: 80.00%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: The Eiffel Tower is a wrought-...
+Actual tokens: 56
+Predicted tokens: 48
+Accuracy: 85.71%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: To be, or not to be, that is t...
+Actual tokens: 60
+Predicted tokens: 50
+Accuracy: 83.33%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: In the beginning God created t...
+Actual tokens: 38
+Predicted tokens: 31
+Accuracy: 81.58%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: Four score and seven years ago...
+Actual tokens: 41
+Predicted tokens: 34
+Accuracy: 82.93%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: I have a dream that one day th...
+Actual tokens: 51
+Predicted tokens: 43
+Accuracy: 84.31%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: That's one small step for man,...
+Actual tokens: 22
+Predicted tokens: 14
+Accuracy: 63.64%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: Here are the key points about ...
+Actual tokens: 203
+Predicted tokens: 195
+Accuracy: 96.06%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: This appears to be an excerpt ...
+Actual tokens: 179
+Predicted tokens: 180
+Accuracy: 99.44%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: This is the beginning of the b...
+Actual tokens: 194
+Predicted tokens: 191
+Accuracy: 98.45%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: That is the opening lines of t...
+Actual tokens: 177
+Predicted tokens: 163
+Accuracy: 92.09%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: That's a powerful and inspirin...
+Actual tokens: 193
+Predicted tokens: 190
+Accuracy: 98.45%
+--------------------------------------------------
+Tokenization results saved to __temp.txt.tokens
+Text: That famous quote is from Neil...
+Actual tokens: 131
+Predicted tokens: 122
+Accuracy: 93.13%
+--------------------------------------------------
+Average accuracy: 82.69%
+```