Ankith Reddy Subhanpuram·Nov 2From Raw Text to the Unit Hypersphere: Building and Visualizing a Modern Text Embedding Model from…A complete guide to engineering a dense text embedding pipeline in PyTorch — covering BPE tokenization, Unsupervised SimCSE, InfoNCE…
Pralhad Teggi·Oct 16Tokenization in LLMs: How “I Love Machine Learning” Becomes NumbersBy Dr. Pralhad P. Teggi
Sujal Charati·Oct 16Building a BPE Tokenizer from Scratch in JavaScriptHow GPT-style models turn text into numbers in about 120 lines of code.
Arun Jagota·Oct 15Learnable Tokenization for LLMs for Code GenerationWe treat text as a sequence of words. If we could tokenize code this way as well, that is as a sequence of tokens — lets call them words —…
Akshat A. Mistry·Sep 24Byte-Pair Encoding (BPE) TokenizersAn end-to-end practical guide to how modern LLM tokenizers work
