EDD (Evaluation-Driven Development)

As a participant in the OpenAI FDE Bootcamp, I observed a strong emphasis on evaluations and context optimization as foundational elements of building reliable AI systems. Rather than viewing these as theoretical concepts, I plan to apply them in practice by building an end-to-end project and experimenting with these techniques firsthand to better understand their real-world impact.

The end goal is to create small, hands-on project for learning modern agentic application development. The application is an AI Architecture Tutor built with Python, FastAPI, the OpenAI Agents SDK, and plain HTML/JavaScript.

The project has evolved through these stages:

Web page
  -> FastAPI endpoint
  -> OpenAI agent
  -> Repository and runtime instructions
  -> Reusable skill
  -> Function tools
  -> Local traces
  -> Evaluation dataset
  -> Deterministic and model graders
  -> Complete eval pipeline
  -> Context-growth experiment

The hands-on project is planned to be create in below below parts:

1. Web-App:Web application, agent, instructions, skill, tools, and tracing
2. Eval Pipeline:Evaluation dataset, deterministic graders, model judge, and full eval pipeline
3. Context Analysis: Context-growth strategies, measurements, and visual report
4. Integration: Integrate with one of the open source framework like Promptfoo

This is a multi-part series, with this article serving as the foundation. In this first installment, we will establish the core concepts and principles that will be built upon in the subsequent articles.rage to build there on

The project intentionally avoids databases, authentication, frontend frameworks, containers, and other production infrastructure. Its purpose is to make the agent workflow visible and understandable.

agent-learning-lab/
|-- AGENTS.md                     Instructions for coding agents
|-- README.md                     Project guide
|-- requirements.txt              Minimal Python dependencies
|-- .env.example                  API-key example
|-- app/
|   |-- main.py                   FastAPI routes
|   |-- agent.py                  Tutor definition and runtime instructions
|   |-- tools.py                  Model-callable tools and glossary
|   |-- skills.py                 SKILL.md loader
|   `-- local_tracing.py          Local JSON trace processor
|-- web/
|   `-- index.html                Plain HTML/JavaScript interface
|-- skills/
|   `-- explain-concept/
|       `-- SKILL.md              Reusable concept-explanation workflow
|-- evals/
|   |-- datasets/
|   |   |-- dataset.jsonl         Standard behavior cases
|   |   `-- context_dataset.jsonl Context-growth experiment
|   |-- graders/
|   |   |-- deterministic.py      Exact answer and tool checks
|   |   `-- model.py              Semantic model judge
|   |-- context/
|   |   |-- strategies.py         Context-selection strategies
|   |   |-- report.py             HTML report generator
|   |   `-- report_template.html  Dependency-free report template
|   |-- evaluate_case.py          Run one standard case
|   |-- run_eval.py               Run the complete standard dataset
|   |-- run_context_eval.py       Run the context experiment
|   `-- results/                  Generated JSON and HTML reports
|-- tests/                        Unit tests
`-- traces/                       Generated local traces

Application:

The first part of the project is deliberately a very small web application.

web/index.html contains:

• A text area for the technical question
• An Ask button
• A response area
• A small JavaScript fetch call

When the form is submitted, JavaScript sends this request:

POST /api/ask
Content-Type: application/json

{"question": "What is context engineering?"}

app/main.py defines two FastAPI routes:

• GET / returns the HTML page.
• POST /api/ask validates the question, runs the tutor, and returns {"answer": "..."}.

The complete web request is:

Browser
  -> POST /api/ask
  -> FastAPI ask_question()
  -> answer_question()
  -> AI tutor
  -> JSON response
  -> Browser displays the answer

Agent Ingeration:

To have the right foundation in place, I created an Agent as below:

Get Neeraj Pandey’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

app/agent.py creates one Agent named AI Architecture Tutor using the gpt-5.6-sol model.

The agent has four important parts:

• instructions: its role and runtime behavior
• model: the model used for decisions and answers
• tools: Python functions the model may request
• input: the question for the current run

run_tutor() calls Runner.run(...) and returns the complete Agents SDK run result. That object contains more than the final text:

result.final_output   # Final tutor answer
result.new_items      # Messages, tool calls, and tool results
result.raw_responses  # Raw model responses and token usage

The web page only needs result.final_output. Evaluation code keeps the full result so it can inspect tool calls and token usage.

Each web question currently starts an independent run. The application does not yet retain a long-running conversation between browser requests.

Blending Together

To tie all these together, add Agents.md, Skills.md, Tools, a local trace app/local_tracing.py records completed agent runs as JSON under traces/. These files show model and function spans so the agent loop can be inspected without dashboard access.

Run

python -m venv .venv .\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

Copy .env.example to .env and set the API KEY

Start the application: python -m uvicorn app.main:app — reload

If all works out well, you shall see the app under: Open http://127.0.0.1:8000 and ask What is RAG?. After the answer appears, inspect the newest JSON file under traces/.

Tests:

python -m unittest discover -s tests -v

Finally, the source code is here: bhobho/evalpipeline at v1-agent-app

With above in place, you have all the foundation for a working application that we will build form here on.

..stay tuned, next in series to follow.