Removing stop words with NLTK in Python - GeeksforGeeks

Removing stop words with NLTK in Python

Last Updated : 14 Oct, 2026

Stop words are frequently occurring words such as “is”, “the”, “and”, and “of” that may contribute little to certain text analysis tasks. Removing them is a common NLP preprocessing technique used to reduce unnecessary tokens and focus on relevant words.

NLTK (Natural Language Toolkit) provides predefined stop-word lists for multiple languages. We can use its stopwords corpus along with word tokenization to identify and remove these words from text. However, stop-word removal is task-dependent and may not improve every NLP application.

  • Stop words are high-frequency, low-meaning words in text data
  • They are often removed during preprocessing in NLP pipelines
  • Removal helps improve efficiency and model performance in many tasks
  • Different libraries provide built-in stopword lists (NLTK, SpaCy, etc.)
consider_the_sentence_the_quick_brown_fox_jumps_over_the_lazy_dog_

Implementation with NLTK

NLTK provides a simple and effective way to remove stop words using built-in English stopword lists.

Installation

pip install nltk

1. Importing Required Modules and Download Resources

Import the sotpwords corpus and word_tokenize function. Download the required NLTK resources before processing the text.

Python 
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download("stopwords")
nltk.download("punkt")
nltk.download("punkt_tab")

Explanation:

  • stopwords provides predefined lists of common words for different languages.
  • word_tokenize() splits text into individual tokens, including punctuation.
  • nltk.download("stopwords") downloads the stop-word corpus.
  • punkt and punkt_tab provide tokenizer resources used by relevant NLTK versions.

2. Define and Tokenize Text

Define a sample sentence and convert it into lowercase tokens before filtering.

Python 
text = "This is a simple example showing how stop word removal works."

tokens = word_tokenize(text.lower())

print("Original tokens:", tokens)

Output

Original tokens: ['this', 'is', 'a', 'simple', 'example', 'showing', 'how', 'stop', 'word', 'removal', 'works', '.']

Explanation:

  • Text variable stores the sample sentence.
  • lower() converts the sentence to lowercase so that words can be compared consistently with the stop-word list.
  • word_tokeinze() separates the sentence into words and punctuation marks.

3. Loading Stop words and Filter Tokens

Load the English stop-word list and remove matching tokens from the sentence.

Python 
stop_words = set(stopwords.words("english"))

filtered_tokens = [
    word for word in tokens
    if word not in stop_words
]

print("Filtered tokens:", filtered_tokens)

Output:

Filtered tokens: ['simple', 'example', 'showing', 'stop', 'word', 'removal', 'works', '.']

Explanation:

  • stopwords.words("english") returns the predefined English stop-word lists.
  • set() converts the list into a set for efficient membership checks.
  • List comprehension retains only tokens that are not present in stop-words.
  • Period remains because punctuation is not automatically removed by this filtering operations.

Complete Program

Following program combines the steps into a single implementation.

Python 
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download("stopwords")
nltk.download("punkt")
nltk.download("punkt_tab")

text = "This is a simple example showing how stop word removal works."

tokens = word_tokenize(text.lower())

stop_words = set(stopwords.words("english"))

filtered_tokens = [
    word for word in tokens
    if word not in stop_words
]

print("Original tokens:", tokens)
print("Filtered tokens:", filtered_tokens)

Output

Original tokens: ['this', 'is', 'a', 'simple', 'example', 'showing', 'how', 'stop', 'word', 'removal', 'works', '.']
Filtered tokens: ['simple', 'example', 'showing', 'stop', 'word', 'removal', 'works', '.']

Comment

Explore