Elif Nur Yılmaz·Mar 26Tokenization: The Foundational Layer of LLMsAn examination of how tokenizers transform raw text into model-ready inputs, and why this seemingly simple step determines model…A response icon1A response icon1
Zaina Haider·Nov 4, 2025Why Byte-Pair Encoding Still Rules (and What Comes Next)Understanding Tokenization
Kevin Haritmonds·Aug 6, 2025LLaMA 2 in Python 3: No Transformers, just PyTorch and SentencePieceI’ve always been fascinated by how large language models (LLMs) can interpret natural language inputs and respond with surprisingly…
InAI-EnthusiastbyDeepankar Singh·Dec 9, 2024Unlocking the Power of SentencePiece Encoding: A Comprehensive GuideLearn SentencePiece Encoding for NLP: explore theory, advantages, real-world use cases, and Python implementation for effective tokenization
Lars Wiik·Apr 21, 2024How Modern Tokenization Works 🌟Multilingual Tokenization — Byte-pair Encoding, SentencePiece, and WordPiece. Unigram Language Model. LLM, Embedding, NLP, Machine…
Divyanshu Dimri·Jan 7, 2024SentencePiece — The NLP Architect’s Tool for Building Bridges Between LanguagesOverview
Dr. Nimrita Koul·Dec 8, 2023Tokenizers for Natural Language Transformer Models — Simply ExplainedIn this article, we will talk about tokenizers used for tokenization of text to be used for various NLP tasks using Transformer models.A response icon1A response icon1
Sieun Park·Mar 5, 2023Subword RegularizationPrerequisite for understanding the commonly used `SentencePiece` tokenizer. Improve your tokenization using subword regularization!
