When should a business invest in AI automation?

The question most businesses ask is "how do we use AI?" That phrasing almost guarantees a weak outcome, because it starts from the technology and goes looking for a problem. The more productive question is narrower: which specific, repeated, expensive piece of work in this organization has characteristics that language models handle well?

Here is the framework we use to answer that, including the cases where the honest recommendation is to build something conventional instead.

The four conditions worth checking first

In our experience, AI automation tends to pay for itself when all four of these hold. Miss one and the economics get marginal; miss two and the project usually should not proceed.

1. The work is genuinely repetitive and high-volume

Automation economics are driven by frequency. A task performed two hundred times a day is worth significant engineering; the same task performed twice a week almost never is, regardless of how tedious it feels. Count the actual volume before scoping anything - intuition about frequency is unreliable, and the number is usually lower than people estimate.

2. The input is unstructured, but the output is well-defined

This is the shape language models are genuinely good at: messy input, structured output. An email thread that needs to become a support ticket with a category and priority. A scanned invoice that needs to become line items. A long document that needs to become a summary with citations.

Reverse the shape and the fit disappears. If the input is already structured - rows in a database, fields in a form - you want a query or a rule, not a model. Using an LLM for deterministic logic is slower, more expensive, and less reliable than the five lines of code it replaced.

3. You can define what a correct answer looks like

This is the condition that eliminates most proposals, and it is worth being strict about. If you cannot write down fifty examples with their expected outputs, you cannot build an evaluation set. Without an evaluation set you cannot tell whether a change improved the system, you cannot detect regression when a model is updated, and you have no basis for trusting it in production.

"We will know it is good when we see it" is not a specification. It is the reason a great many AI pilots never reach production: nobody could ever establish that they were working.

4. A wrong answer is recoverable

Language models are probabilistic. Over enough volume they will produce confident, wrong output. The question is not whether that happens but what it costs when it does.

Drafting an internal summary that a person reviews: cheap error, strong candidate. Classifying support tickets with an easy reassignment path: cheap error, good candidate. Approving a refund, submitting a regulatory filing, or sending a customer communication unreviewed: expensive error, needs a human approval gate - which is fine, but it changes the economics, because you are now saving review time rather than eliminating the task.

What it actually costs to run

Build cost gets scrutinized; run cost gets discovered. The second one is what tends to kill AI features in their first year.

Cost per request scales with the amount of context you send and the model tier you send it to. A retrieval system that stuffs twenty documents into every request to be safe can cost an order of magnitude more than one that retrieves three good ones - for worse accuracy, since irrelevant context degrades output.

• Model your expected request volume and context size before building, not after launch.
• Instrument actual cost per request from the first day in production.
• Route by difficulty: small models for classification and reformulation, large models only for synthesis.
• Cache aggressively - identical and near-identical requests are more common than teams expect.
• Set a budget alert. An unbounded retry loop can produce a remarkable bill overnight.

Where conventional software wins

A meaningful share of what arrives described as an AI project is better solved another way, and saying so early saves a great deal of money.

• "Our team spends hours finding information" - often a search and information-architecture problem first. Fix retrieval before adding generation on top of it.
• "We need to predict demand" - frequently a reporting problem. Many organizations have not yet looked carefully at the data they already hold.
• "Customers ask the same questions repeatedly" - sometimes an assistant; often a clearer product, better documentation, or a redesigned form.
• "We re-key data between systems" - that is an integration project. An LLM in the middle adds cost and a new failure mode to a solved problem.

The best AI projects we have worked on started as a specific, measurable complaint from the people doing the work - never as a mandate to adopt AI.

Sequencing: what to do first

If the four conditions hold, the order of work matters more than the choice of model.

1. Pick one workflow with a measurable baseline. Not a department, not a strategy - one workflow where you know the current time cost or error rate.
2. Build the evaluation set before the product. Fifty to two hundred graded examples. This takes days and determines everything afterward.
3. Establish the retrieval or data layer. Most quality problems attributed to models are actually retrieval problems.
4. Build the thinnest usable version and put it in front of a small group of real users behind a flag.
5. Measure against the baseline, instrument cost, and only then decide whether to widen scope.

Teams that follow roughly this order tend to end up with something in production. Teams that start by selecting a platform and running a broad pilot tend to produce an impressive demonstration and no operational change.

A closing note on honesty

AI is genuinely useful for a specific and growing class of problems, and it is being sold for a far wider class than it serves well. The organizations getting real value from it are not the ones with the largest ambitions - they are the ones that picked a narrow, well-measured problem, built the evaluation discipline to know whether it was working, and expanded from evidence.

If you are not sure which category your idea falls into, that uncertainty is worth resolving with a short feasibility exercise rather than a long build.