Why Proper Prompting Makes a Big Difference in Legal AI
Welcome back! This post continues our series on leveraging AI to elevate your 1L studies. In Lessons 1 and 2, we examined how to construct a database of case files and query them in natural language using large language models (LLMs).
In this lesson, we will explore why proper prompting is essential — that is, how the phrasing of your question impacts the LLM’s response.
Broad, open-ended prompts like “What was this case about?” or “Brief this case using IRAC framework” may seem straightforward, but they leave too much open to interpretation by the AI model. Without clear guidance on what information is most relevant or important, the AI’s response may miss the mark.
Let’s run an experiment and compare two prompts that are aimed to extract the same information from a case. We will use the database that we built in Lesson 2.
from langchain import OpenAI
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.document_loaders import PyPDFDirectoryLoader
from langchain.indexes import VectorstoreIndexCreatoros.environ["OPENAI_API_KEY"] = "your_api_key"loader = PyPDFDirectoryLoader('supreme_court/')
index = VectorstoreIndexCreator().from_loaders([loader])One of the cases that we uploaded into a ‘supreme_court’ folder was Marbury v. Madison (1803). Let’s run two queries:
Prompt 1:
query = "Brief Marbury v. Madison case using the IRAC model. Present in a bullet point format"
result = index.query_with_sources(query)
print(result['answer'])“IRAC” is a commonly used framework for analyzing and briefing legal cases. It stands for:
- Issue: Identify the key question or legal issue addressed in the case. This guides the entire analysis.
- Rule: Explain the legal rule, standard, or test used by the court to resolve the issue. Discuss the source of law the rule is derived from (e.g. statute, precedent).
- Application: Apply the rule to the facts of the case. Explain how the rule leads to the court’s decision. You may weigh evidence for multiple sides here before concluding.
- Conclusion: State the court’s finding or holding. Explain the outcome and any subsequent impact or implications.
Using the IRAC framework helps to systematically unpack the reasoning in a case by isolating each element of the analysis. For law students, IRAC is a useful structure for both studying and discussing cases to gain a full understanding of how the law was applied to reach the ultimate conclusion.
Get legaltextai’s stories in your inbox
Join Medium for free to get updates from this writer.
In Prompt 2, we will articulate for the LLM, with greater specificity, what we aim to convey:
query = '''
Summarize Marbury v. Madison according to the following requirements:
Facts: Name of the case and its parties, what happened factually and
procedurally, and the judgment.
Issues: What were the main issues at stake.
Procedural history: What happened procedurally.
Holding: The applied rule of law.
Rationale: Reasons for the holding.
Decision: What did the court decide, e.g. affirmed, overruled.
Policy implications: What are the wider policy and social consequences of this decision.
Present in a bullet point format.
'''
result = index.query_with_sources(query)
print(result['answer'])Let’s compare the results from these two prompts:
The answer from Prompt 1:
“Marbury v. Madison was a case in which William Marbury sued James Madison, the Secretary of State of the United States, for failing to deliver Marbury’s commission as a justice of the peace.
The Supreme Court held that Marbury had a right to the commission, but that the Court could not issue a writ of mandamus to enforce it because the Judiciary Act of 1789, which authorized the Court to issue such writs, was unconstitutional.”
The answer from Prompt 2:
“Marbury v. Madison:
- Facts: Case between William Marbury and James Madison, Marbury was appointed as a justice of the peace but his commission was not delivered to him.
- Issues: Whether Marbury had a right to the commission and if so, whether the Supreme Court had the power to issue a writ of mandamus to Madison.
- Procedural History: Marbury filed a petition for a writ of mandamus in the Supreme Court.
- Holding: The Supreme Court held that Marbury had a right to the commission, but the Court did not have the power to issue a writ of mandamus to Madison.
- Rationale: The Court reasoned that the Judiciary Act of 1789, which gave the Supreme Court the power to issue writs of mandamus, was unconstitutional because it exceeded the scope of the Supreme Court’s authority as defined in Article III of the Constitution.
- Decision: The Court overruled Marbury’s petition.
- Policy Implications: The decision established the principle of judicial review, which is the power of the Supreme Court to declare laws unconstitutional.”
In contrast, with a broad prompt to simply “brief the case using IRAC,” the AI is left to determine what belongs in each section of the IRAC model. It may fail to identify legally meaningful facts, confuse issues and holdings, or lack a complete rationale. The AI is still learning, so we must provide the clearest criteria possible to yield the output we need.
In summary, carefully crafted prompting is key to enabling AI to develop sophisticated legal analytical skills. Providing clear guidelines on the relevant elements to cover in a case summary or brief helps the AI model illuminate the legally significant details and arguments — rather than generating a superficial response lacking reasoned analysis. With practice at structured prompting, AI comprehension and reasoning in law will continue to improve over time. But for now, thoughtful prompting remains foundational.
