Andrew Ng: Six Tough Challenges in Building and Deploying AI Applications

Professor Andrew Ng has just published an article specifically dissecting the core competencies of AI engineers.

After analyzing numerous job postings, conducting systematic expert interviews, and carrying out surveys, he ultimately broke down the skill set of "building AI applications" into six components.

These six competencies are: LLM fundamentals, laying the data foundation for models, building agent systems, evaluation-driven development, production environment operations, and machine learning basics.

It may sound like a table of contents, but each component holds substantial depth when expanded upon.

How Do AI Applications Differ from Ordinary Software?

Let’s start with a fundamental difference.

Traditional software features deterministic inputs and outputs. If you write a sorting function, it will produce the same result every time you run it, even ten thousand times.

AI applications are different. You cannot know in advance what an LLM will output, nor can you predict what errors the model might make. This uncertainty necessitates repeated iteration throughout the development process—write a version, check the results, and then decide how to adjust next.

Therefore, the core competency of an AI engineer is essentially the ability to make decisions under uncertainty.

Component 1: Understanding the Underlying Mechanics of LLMs

Many people use LLMs like a black box: they throw in a question, wait for an answer, and neither understand why it got it right nor why it got it wrong.

Andrew Ng believes this is insufficient.

You need to understand how an LLM splits text into tokens and how it generates output step by step. Only then can you judge: Can it handle this task well? Under what circumstances is it prone to errors?

Beyond this, there is a host of engineering details to master: when to use multimodal models, how to make trade-offs within limited context windows, how caching works, where the model's knowledge cutoff lies, how to control computational power for inference, and when to use tool calling.

Digging deeper, there is another question: When is fine-tuning necessary, and when should you consider self-hosting deployment?

Making the right judgments on these matters requires a solid understanding of the underlying mechanics.

Component 2: Feeding the Right Data to the Model

The quality of an LLM's output largely depends on its input.

The early popular approach was RAG (Retrieval-Augmented Generation), which involves stuffing relevant content into the context via vector search. However, current practices go far beyond this.

You need to determine: Which content should be written directly into the prompt, and which should the model retrieve using tools as needed?

How should data organization be selected? What scenarios are best suited for vector indices, knowledge graphs, or semantic layers on structured data?

How should documents be processed to ensure the model can understand them? Different formats such as PDF, HTML, and images require different processing methods.

Another easily overlooked aspect is that data quality requires continuous maintenance; it cannot be handled once and then ignored.

Component 3: Building Agent Systems with Care

Agent systems vary significantly in form.

Simple systems follow fixed workflows: a series of LLM call steps are designed in advance and executed sequentially. Complex systems allow the LLM to determine the next step itself, making cyclic decisions until the task is completed.

How to choose? Andrew Ng offers several criteria for judgment:

Which steps can run in parallel, and which must be serial? Where is code more reliable, and where is an LLM more flexible? How to handle rollbacks when errors occur?

Regarding tool calling, it is necessary to design which tools the model can use and what commands it can execute. How is memory managed—especially when conversations are long and the context window is full? When is multi-agent collaboration needed rather than relying on a single agent to handle everything?

Another crucial matter, often skipped, is: Once the prototype works, how do you transform it into a reliable system ready for production? This requires considering security boundaries, adversarial inputs, and risks such as data leakage.

Component 4: Evaluation-Driven Development, The True Watershed

On this point, Andrew Ng states:

Among the AI engineers he has observed, the ability to implement a rigorous evaluation closed-loop is the most critical trait distinguishing excellent engineers from average ones.

What does this mean?

Many people modify systems based on intuition, changing whatever seems off and subjectively judging that it "seems a bit better" afterward.

Skilled engineers take a different approach: First, clarify what needs to be measured, then establish an evaluation mechanism. After each change, run evaluations and decide on the next adjustments based on data.

Evaluation itself is a technical skill. You need to analyze the system's output and intermediate processes, combining business objectives to determine metrics. There are many forms of evaluation: hard-coded rules, scoring by LLMs, and human judgment. Determining which method suits which scenario also requires discernment.

Moreover, evaluation is not a one-time task; it must evolve alongside system iterations.

Component 5: Production Environment, A Different World

Building a system is one thing; running it in a production environment is another.

Operations for AI applications differ from traditional software operations in several key aspects.

First is observability. You need to know how the system performs with real users, not just on test sets. Performance must be monitored, shifts in data distribution must be detected, and sudden drops in model performance or injection attacks must be responded to quickly.

Second is the testing framework. Regression testing for AI systems is more complex than for traditional software—many results do not have a single correct answer, requiring statistical methods to determine whether performance has improved or degraded. The intensity of testing should match the cost of errors.

Third is cost and latency control. Decisions on which model to select, whether distillation is needed, and whether workflows can be simplified will directly impact the system's viability as user volume grows.

Component 6: Machine Learning Basics, Unavoidable

For the final component, Andrew Ng says:

All the engineers he knows who excel in LLM applications have a certain depth of understanding of machine learning and deep learning.

Why?

Because LLMs themselves are products of machine learning, and many judgments can only be made correctly from an ML perspective. Furthermore, many practical application scenarios still require traditional machine learning—whether using pre-trained models from others or training your own.

More importantly, several core concepts in machine learning are extremely useful for building AI applications: the bias-variance tradeoff, error analysis, and data engineering. These concepts provide a mental framework that helps you make better judgments when dealing with systems featuring uncertain outputs.

Risk Warning and Disclaimer

The market carries risks; investment requires caution. This article does not constitute personal investment advice, nor does it take into account the specific investment objectives, financial status, or needs of individual users. Users should consider whether any opinions, views, or conclusions in this article align with their specific circumstances. Investors bear full responsibility for their own decisions.