MA Technologies LLC logo
← Blog

AI observability

LLM Audit Logging: Why AI Observability Is Different

August 2026 · 6 min read

Written by Murtaza Dabbawala

AI observability: prompt, model, context, sources, tools, output and human oversight

Traditional application logging was built around a fairly simple question: what happened?

An API received a request, a service processed it, a database was updated, and the application returned a response. If something broke, engineers could usually follow the trail backwards through logs and traces and work out where things went wrong. That works because conventional software is, for the most part, predictable. Give the system the same inputs and conditions, and you generally get the same behaviour.

LLM applications make that much harder.

Imagine an AI system tells someone, “This participant appears eligible for the study.” Three months later, somebody challenges that recommendation. Your application log says the request succeeded, the API returned 200, and the whole thing took 2.4 seconds. Technically, everything looks healthy. But can you actually explain the answer? What was the AI asked? Which prompt was running? Which model version? What protocol or documents did it retrieve? Was a recent amendment included? What patient information was available to it? Did it call another system before producing the answer?

The application can be working perfectly while the reasoning context behind the output has disappeared.

A prompt is rarely just a prompt

We often picture an AI request as Prompt → Model → Response. In production, it is usually closer to a small supply chain. A user’s question might be combined with a system prompt, previous conversation, retrieved documents, database records, business rules and tool outputs before the model ever sees it.

The model may then call another tool, receive new information, generate a response, pass that response through a guardrail and only then return something to the user.

So if the answer is questioned later, keeping the original user prompt doesn’t tell you very much. It’s like trying to understand why a person made a decision by recording only the question they were asked, but none of the information sitting on the desk in front of them.

What a real AI trace preserves

Prompt

The prompt as assembled, and the version it came from.

Model

Model, version and configuration in effect at that moment.

Context

Conversation, records and business rules that shaped the input.

Sources

Retrieved documents and the exact versions used.

Tools

Tool and API calls made, with what they returned.

Guardrails

Safety and policy decisions applied to the output.

Human

Who intervened, approved, edited or released the result.

The trace is the story

Flat logs become less useful as AI workflows become more sophisticated. A single request might trigger retrieval, several API calls, one or more LLM calls and a final safety check. You can log every one of those events and still end up with a pile of disconnected facts.

Tracing gives you the missing relationship. Instead of asking engineers to piece together what happened from dozens of log entries, you can follow one request from beginning to end and see how the pieces interacted.

If an engineer had to investigate one response six months from now, could they follow one trace and reconstruct the journey — or would they be searching through logs hoping to find the right pieces?

Then comes the uncomfortable part

Privacy

There is a natural reaction to all of this: fine, just log everything. But that creates another problem. The information you're trying to preserve may itself be sensitive. Prompts, retrieved documents and responses can contain PII, PHI, financial information, proprietary data or confidential research information. An AI audit trail can therefore become a second place where sensitive information accumulates.

The answer isn't to capture everything. Good observability needs the same discipline as good data architecture: collect what you need, protect it properly, and don't duplicate sensitive information simply because you can. Sometimes recording a document ID and version is enough to establish provenance without copying the entire document into an audit record. Redaction, encryption, access controls and retention policies belong in the logging architecture — not bolted on afterwards.

Trace enough to explain. Protect enough to trust.

What the solution looks like

This isn't something you can go back and fix later. Six months from now, when someone asks why the system produced a particular result, either the context was captured at the time or it's gone. There's no reconstructing it after the fact.

So the move is to treat every AI interaction like a transaction worth tracing, not just another API call. Give it a trace ID. Capture the prompt version, the model and its configuration, which documents it pulled and which version of them, what tools it reached for, what guardrails kicked in, and what actually made it to the user, all tied to that same trace.

That's really the whole idea. One chain you can walk back through, instead of five different logs nobody thought to connect. And you can build this with LLM observability platforms such as Langfuse or LangSmith, OpenTelemetry-based tracing, or a custom audit layer integrated with your existing application monitoring stack.

So, would your team actually pass the six-month test today? Or would you be piecing it together after someone's already standing in your office asking?

Traditional logs tell you that something happened. AI observability should help you understand the conditions under which it happened.

See how we build governed AIRead the case studyTalk to us