Machine Learning In Production , MLOps & Machine Learning Lifecycle Part2: Scoping

Don’t build your system on sand, why ML scoping is the most important step

This story is the second part of the series :machine-learning-in-production-mlops-machine-learning-lifecycle-part1

image made by me (inspired by prof Andrew Ng’s course)

“You will spend 80% of your time on data preprocessing”. Yes, this is the first and most common sentence both of you and me have heard when starting to learn AI and data science. We also hear things like, “The more time you spend on data preprocessing, the better your AI system will perform.”

Are those sentences true? Yes, they are.

Should they be our top priority? No.”

Because spending 80% of your time at data preprocessing without studying system feasibility is a waste of resources, what is the purpose of spending 80% or more at data processing for a project that doesn’t make sense in the real world. This is exactly why project scoping is essential.

What is scoping:

According to the Machine Learning in Production course from DeepLearning.AI, scoping means picking the right project to work on and planning out the scope of the project. It involves going through several key phases.

To help you understand this well, I will provide an analogy: let’s say a ML system is a house you want to build. So what are the most important things to focus on before you start building?

1. Check Land and Soil Quality: Evaluating soil quality before building a house is a crucial step to ensure the long-term safety and stability of your home, as not all soils are equally fit to build upon. We take this exact same approach in the first step of building an AI project: we check if AI is a suitable solution for our specific problem. After all, not every problem can be solved with AI.
2. Defining ideas about the design of a house: This is the same as translating the AI solution you find into an ML task. Let’s say we want to build an AI system that analyzes a customer’s past purchases and browsing behavior to predict and display hyper-personalized product recommendations in real-time, thereby increasing both user engagement and average order value.
3. Assess feasibility and value of potential design: Evaluating whether the chosen house design can be executed within defined constraints — such as time, budget, resources, and legal permits — before construction begins.
4. Determine Success Metrics: Define the success criteria. The house must be built up to modern building codes, pass all electrical and plumbing inspections, and be completed by a specific date. This is the exact same approach we take while scoping ML projects: we define both business metrics (e.g., response, latency) and ML metrics (e.g., 95% accuracy or a specific F1-score).
5. Budget for resources: Secure the construction loan, hire general contractors and architects, and schedule material deliveries (lumber, concrete, appliances); similarly, for an ML project, we must calculate the required budget, including computing power, cloud infrastructure, domain experts, data labeling costs, and project timelines.

Keep in mind: A house is only as strong as its foundation. You can’t build a strong ML system on a weak foundation.

image made by me (inspired by prof Andrew Ng’s course)

How to assess the feasibility and value of potential solutions?

Feasibility: is about determining whether a project can be completed. Sometimes, you must use external benchmarks (e.g., industry literature, organizational publications, or company reports) to check if anyone has successfully managed the task before.

Value: Is the project worth building?

Assessing project feasibility depends on the type of data, and whether the system is being developed for the first time or if an existing one is being modified. The figure below outlines how to do this:

image made by me (inspired by prof Andrew Ng’s course)

But what is HLP?

HLP (Human-Level Performance) is a way to establish a baseline, which we use during scoping. This is especially true for unstructured data (audio, images, and video) that humans are good at understanding and interpreting. It helps us with:

1. Setting a performance threshold: by determining the highest achievable accuracy, we can establish a baseline. For example, if the HLP can achieve 90% accuracy on a task while our model only achieves 60%, this serves as a flag indicating we must improve the model’s performance.
2. Guiding error analysis: If our model fails to reach the HLP, we will shift our focus from model performance to other issues, such as poor data quality or the need for data augmentation.
3. Prove that the ML system is superior to humans doing the job.
4. Resolving labeling ambiguity: By comparing the performance of multiple human annotators, HLP helps identify inconsistent labels and clarify ambiguities in the training dataset.

Some considerations to keep in mind when assessing the value and feasibility of projects:

Get Ula Yousef’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Ethical consideration: even if a project is technically and financially feasible, we should determine whether it is legal and does not violate laws and ethical standards.

Technical consideration: Do we have the required technology, infrastructure, and technical skills to carry out the task?

How to determine milestones (and define success metrics?)

Legend has it that there is a constant conflict between the Machine Learning team and the business team. Because the ML team focuses entirely on technical metrics — such as precision, accuracy, and the F1 score — the business team focuses strictly on business metrics, like revenue, profit, and customer lifetime value. To bridge this gap, the two teams should collaborate to set technical and business metrics that they both agree on.

Another important metrics is visibility (stakeholder alignment): is about ensure that teams, sponsors and users agree on the final outcome of the project, it also means build a system that people trust it and know how to use it.

Another important metric is visibility (stakeholder alignment): it is about ensuring that teams, sponsors, and users agree on the final outcome of the project. It also means building a system that people trust and know how to use.

Your role as a Machine Learning engineer is to find the technical metrics that align well with business goals and objectives. So, instead of saying, “We want to build an E-Commerce recommendation system that achieves specific accuracy”, think, “We need to increase the total sales of certain items on an E-commerce platform”. Therefore, we will optimize the model for precision on high-margin items rather than just overall accuracy.

Keep in mind: “AI usage is a tool, not a goal. As ML and AI engineers, our primary job is solving business problems that require AI while striking a balance between model success metrics and business success metrics. Therefore, our objective is not just achieving high accuracy or precision, but ensuring that those metrics drive the final business goals.”

key specifications of resources:

Personnel required include:

• Labelers: for raw data annotation.
• Machine learning engineers and data scientists: to build and fine-tune models.”

Data Acquisition: Costs for gathering, labeling, and storing data.

Timeline: setting a deadlines to schedule every project phase from data gathering to final product release.

key timelines tracked:

• Data gathering periods.
• Model training starts.
• Deployment dates.
• First version release.

Computer resources include the costs of the CPU, GPU, and memory.

Summary:

In this story we emphasized the importance of scoping as a way to study the foundation of your project before you start building it.

We discussed the steps of the scoping process, which include defining the business problem and brainstorming an AI solution that fits ethical and technical considerations without violating laws. Finally, we covered the key ideas for determining success metrics and milestones.

I will keep writing about MLOps, so stay tuned and enjoy reading!