huh? tokenizer.decode([389]) + tokenizer.decode([6376]) == tokenizer.decode([389, 6376]) # True โœ… tokenizer.decode([364]) + tokenizer.decode([6389]) == tokenizer.decode([364, 6389]) # False โŒ โ€œ1 + 1 + 1 = 3?โ€: Depends on the tokens, bro. ๐Ÿ˜‚

Jul 11, 2025 ยท 6:27 PM UTC

1
1
5
Replying to @dskhudia
Welcome to the war on tokenizers Daya
5