Building an AI assistant before the AI exists

The problem: a feature built on a service that didn’t exist

One of our clients was adding an AI assistant to their platform. Organisations using the platform describe a situation in a few sentences and get a short answer that links to the relevant entries on the site.

The work was split between two teams. A separate team is building the AI backend in Python: retrieval, prompts and the language model. We are building everything on the platform side:

• The question screen, where an organisation asks and rates the answer.
• A setup area, where each organisation tailors the assistant with its own instructions and material.
• A log of every question and answer, for the organisation and for the client’s back office.
• A nightly export of all active entries, so the AI only recommends what is actually on the platform.

Both teams started at the same time. On day one we had a draft of the interface between the two systems and nothing to call. Waiting for the backend would have left the platform work idle for weeks and then squeezed every screen into the last few days. Building against guesses would have meant rewriting the screens once real responses showed up.

The fix: an API spec and a mock that answers

We turned the draft into an OpenAPI file with all endpoints, their schemas, every error and an example for every response. Both teams change that file when the interface changes. It stays the source of truth until the real backend is running and generates its own spec. Next to it we wrote a small Express server that implements the whole API from the same examples.

A mock that always returns the happy path only proves that the happy path works, so we made it steerable from the UI. Words in the question choose the response:

The setup screens work the same way. New material moves from processing to indexed on a timer, and trigger words in a name produce each failure the real backend can report, so every status and error message could be checked in the browser.

Within days, every state of every screen could be clicked through in a browser. hang turned out to be the most useful row. The mock accepts the request and then says nothing, which is a case you would never get to test against a real service, and it’s the one that proves our own 30-second timeout fires.

Building against the mock also brought up problems that would otherwise have appeared in production. These are the ones worth writing down, and how we solved each.

Issue 1: two systems wanting the same data

The draft gave the backend endpoints for the organisation’s instructions, the question log and the feedback on answers. That meant two copies of the organisation’s settings, and a log owned by the service most likely to be down.

We moved all three to the platform and dropped those endpoints from the spec:

• Instructions are stored with the organisation’s other settings and sent along with every question. The backend keeps no copy, so there’s nothing to keep in sync.
• The log is written by the platform from each response. When the backend is down, it can’t log anything, but the platform still can, and a question that got no answer is exactly the one worth counting.
• Feedback is stored on that log entry, matched by the request ID.

The backend kept what it has to index: the material organisations add and the entry data. The rule we ended up with: the AI service owns what it needs to search, and the platform owns everything people edit or read back later.

Issue 2: errors that never reach the backend

The spec described the errors the backend can return. The UI also had to cope with failures where the backend never answers at all: the network drops, a request times out, or something in between returns an error page of its own.

We gave every error from the backend the same simple shape, with a stable code and a message for the logs. The platform reacts to the code only, so the backend team can reword messages freely.

On our side, the client wraps every failure, from the backend or from the network, in the same error type. The rest of the code only ever checks one code, and on the question screen those codes map to a handful of messages a user can act on: try a different question, wait a moment, or try again later.

Issue 3: retries that cost money

Every question costs money, because the model reads several thousand tokens of entries, documents and instructions to answer it. An HTTP client that retries on failure would quietly pay twice for every slow answer.

The platform never retries a question automatically. If it fails, the user sees why and clicks “Ask” again.

The timeouts are layered on purpose. In the spec, the backend gives up after 25 seconds and returns a proper 504 timeout. Our client waits 30. Because the backend’s limit is the shorter one, a slow answer normally ends with a typed error from the backend, and our own timeout only fires when the backend is completely stuck. With the values the other way round, every slow answer would end in a vague client-side timeout while the backend kept working, and billing, on an answer nobody would read.

Every question is logged, including the ones that failed:

} catch (error) {
  // No automatic retry: every call to the backend costs money
  logEntry.status = "error";
  logEntry.errorCode = error instanceof AiBackendError ? error.code : "unknown";
  result = { success: false, error: toAskErrorCode(error) };
}

// Every question is logged, including failed ones, to see how many went unanswered
await createItem("logs", logEntry);

Cost also shaped the security rules. The spec has the backend accept requests only with a valid key from known servers, and each organisation has a limit on questions per minute. On top of that sits a global limit, which puts a ceiling on the bill if the key ever leaks.

The nightly export is the opposite case. It sends the full list every time, so a repeat is harmless, and it retries once after ten minutes if the backend was busy or unreachable. Whether to retry depends on what a repeat costs.

Issue 4: templated responses that looked like errors

Some questions should never reach the model. For certain kinds of questions, the backend will reply with a fixed, pre-written template and skips the model entirely.

The first draft treated these as a special kind of error. On the platform, that would have put them in the wrong place everywhere: errors are shown as “please try again”, counted as failures and hidden behind generic messages. A templated response is a real answer. It needs its own layout, it needs to be marked in the log, and the organisation needs to be able to find it there later.

We made it one of the normal response types, next to a regular answer and “nothing suitable found”. It comes back as a 200 with a flag, and its metadata shows no model and zero tokens: a template is faster, costs nothing and always says exactly what it was written to say. The mock returns one for a trigger word, so the screen was designed and checked before the backend could recognise anything.

Issue 5: links in model output

Answers come back as Markdown with links to entries on the platform. The model also reads the material that organisations add, and any of it can contain a link, or an instruction to include one. A model can also simply invent a URL that looks right. Rendering the answer with a Markdown library and dangerouslySetInnerHTML would have passed all of that straight to the user.

The spec allows only paragraphs, lists, bold text and links, and the renderer builds React elements from that small subset, so the answer never becomes HTML. A link becomes clickable only if its URL is one of the entries returned with the answer:

nodes.push(
  allowed.has(linkUrl) ? (
    <a href={linkUrl} target="_blank" rel="noopener noreferrer">{linkText}</a>
  ) : (
    linkText
  )
);

Any other link shows up as plain text, so the assistant can only send people to pages that exist on the platform.

Issue 6: a nightly sync that could empty the index

The nightly export sends every active entry, and the backend is meant to work out what changed and removes whatever is missing. That design has a sharp edge: a bug that sends a tenth of the entries would quietly delete most of the index overnight.

The fix is a guard written into the spec. Any export that drops by more than a set threshold compared to the previous one is rejected with suspicious_drop, and nothing is deleted. If the drop is real, someone sends it again with ?force=true. Every run is recorded and shown in the back office, with a button to run it by hand and, after a rejected drop, a second one to force it through after a confirmation.

The data itself needed attention too. The draft described it as a clean schema, so we exported 30 real entries, picked for variety, with fill rates for every field across the whole data set. That cleared up several things the schema couldn’t have shown.

The result

Less than three weeks after the first draft of the interface, every platform screen was built and tested against the mock in every state the spec describes, from a normal answer to a backend that hangs. The assistant can’t be used until the real backend is ready. When it is, connecting the two takes two environment variables, plus testing against real answers.

The mock also changed how we designed the feature. With no real answers to look at, we had to decide up front what the platform does when the AI is slow, down, expensive, wrong or facing a question it should leave to a template. Those decisions are easy to put off once a real model is in place, because the happy path works most of the time. Most of the effort went into the spec and the mock, and the useful questions were plain ones: who owns this data, what does a repeat cost, which errors does a user need to tell apart, and what happens if this response is hostile.