
1. Fact Lifecycle

For most of the last five years, generative AI has been an assistant. You prompt it for a draft, a summary, a snippet of code, and it returns one. The next phase of the technology is different. Agentic AI is built to pursue goals on its own, decide what to do at each step, and run sequences of work across live systems with little human intervention. The promise is enterprise operations that anticipate problems, adjust their own processes, and respond to changing conditions in something close to real time.
That promise is harder to deliver on than the demo videos suggest. Gartner predicts that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls. The failures rarely come from the model being insufficiently smart. They come from the surrounding infrastructure: agents that lose track of their own state during a run, tool integrations that break the moment an API drifts, reasoning loops that spiral without anyone able to tell why. The work of running an agent reliably in production lives in the harness around it: the validation, state management, concurrency control, and guardrails that turn flexible reasoning into accurate output. The next frontier of agentic AI is less about expanding model capability and more about engineering that surrounding layer.
Data ingestion is one of the clearest places to see this gap. Conventional pipelines are rigid by design. They rely on static rules that struggle with natural language and break the moment a source changes its format. Agentic systems sidestep that brittleness, adapting to varied inputs, interpreting context on their own, and pulling out only the information that matters.
The same flexibility is what makes them difficult in production. Because agents reason probabilistically, their output often fails to conform to the strict schemas that deterministic downstream systems expect. Large or complex inputs can exhaust an agent's context window, causing it to silently drop instructions or degrade its reasoning without raising an error. And these failures tend to be invisible rather than loud. An agent can confidently assert information that was never there, or be steered by untrusted content into altering what it captures. The challenge, then, is less in the ingestion itself than in the harness around it.
In one of our customer engagements, we were tasked with building a data ingestion pipeline that could turn a business's scattered, unstructured material into a reliable, queryable knowledge base. The brief sounded straightforward: gather information from a range of sources, strip out the filler, and extract the specific facts that mattered. In practice, each of those steps hid a layer of difficulty. The inputs were heterogeneous and unpredictable. No two sources were structured the same way, and the pipeline had to handle them without any configuration tailored to each source.
From that raw material, the agent had to pull out atomic facts (an individual price, a single policy, a specific operational detail) rather than loose summaries. The extraction had to be dynamic, capturing only what a given requirement called for instead of following a fixed schema.
The harder requirement came after extraction. We weren't building a pipeline that would run once and stop. We needed a living database of facts: one that would grow as new sources arrived, recognize when a newly extracted fact duplicated or contradicted something already stored, and resolve those conflicts on its own so the knowledge base stayed internally consistent as it evolved. The one thing we took as given was that each client had enough usable material to draw from; the pipeline could organize and verify, not invent.
Zero technical input. Whoever owned the data could not be expected to configure schemas, map fields, or write rules tailored to a particular source. The pipeline had to handle that interpretation itself.
Source heterogeneity. Inconsistent source structures and varied formats were the norm, with no guarantee of clean or uniform input. The pipeline had to be resilient to messy material that came in whatever shape the source produced.
Grounding. Every fact in the database had to be traceable back to a real source. No standalone claims, no facts without provenance.
Coverage. The pipeline had to populate enough of the expected categories to reliably answer the queries the downstream application would actually receive.
Latency. A full ingestion run for a new source had to finish within roughly 30 minutes.
Scalability. The pipeline had to run many ingestion jobs at once across clients, not work through them one at a time.
The first version was a straightforward sequential pipeline. A user would provide a URL, upload a file, or type directly; the system would read the content and pass it to the LLM for fact extraction. There was just a fixed sequence of steps applied uniformly to every input. That simplicity was a deliberate production decision. When a system's failure modes are still unknown, the cheapest way to find them is to ship the smallest thing that works end to end and watch where it breaks. The requirements were clear and the inputs well defined, so we assumed the LLM could absorb the variability in source content without per-source configuration, and let the real failure modes surface on their own terms.
The same instinct shaped the knowledge base. Before building anything custom, we evaluated the managed options. Most were built for a different problem: Mem0, like most memory tools, is oriented around user conversations rather than ingesting facts from documents and websites. Zep was closer, but its knowledge-graph approach was overkill for our needs. A graph database answers questions about how entities relate, whereas we just needed a reliable, growing store of atomic facts. A custom approach driven by the LLM was simpler to build and cheaper to maintain.
Source content is noisy. Every source carries far more than the business information being targeted. Web pages bury it under navigation menus, cookie notices, and repeated footers; documents surround it with boilerplate headers, disclaimers, and formatting artifacts. Whatever the format, the ingestion step pulled in all of it. The LLM then received the full payload and was expected to work out what mattered. Results were inconsistent: some sources produced clean facts, while others returned a mix of useful content and filler. The effect scaled with how much irrelevant material a source contained, and on the noisiest sources that filler could account for the majority of the output.
A second problem followed from the first. The pipeline split content into fixed-size chunks and asked the LLM to pull a set number of facts from each. On information-dense sources this worked acceptably. On sparse ones, the model produced the same number of facts regardless, padding the count with low-confidence extractions or near-duplicate entries. Nothing let the extraction step respond to how much a given chunk actually contained, so the quota dictated the output rather than the source.
Together, these two limitations meant the knowledge base accumulated entries that were either low quality or redundant, which compounded the downstream challenge of maintaining a consistent, queryable knowledge base.
A sequential pipeline has no capacity for judgment. Every source receives the same treatment regardless of its structure, relevance, or information density. The LLM was simultaneously determining what was worth extracting and performing the extraction itself, from whatever the previous step happened to return, with no understanding of how the source was organized and no ability to seek additional context when a page turned out to be shallow or incomplete.
TL;DR: We replaced the fixed pipeline with an agent that adapts to each source: it surveys what's there, processes only the relevant parts, and extracts as many facts as the content actually supports. The real wins, though, came from the harness around the agent rather than the agent itself: guardrails that cap how much work it can do per run, validation that rejects malformed or unsafe output, and observability to debug failures at scale.
Addressing the problems required changes at two levels: how the agent reasoned about its work, and the harness around it that made those decisions safe to act on. The static pipeline was replaced with an agentic workflow that could decide what to process and how, rather than applying a fixed sequence of steps regardless of what it encountered.
The agent now surveys a source before processing it. Whether the source is a website, a document, or a structured export, it first builds a picture of what is there, identifies the parts relevant to the categories it needs to populate, and processes only those, pulling in what it has determined is worth pulling in rather than gathering all of the content available. Fact extraction became dynamic in the same way. Instead of producing a fixed count per chunk, the agent extracts based on what the content actually contains. Dense sources produce more facts, sparse sources fewer, and the extraction step responds to the source rather than to a predetermined quota.
An agent is only as good as the harness around it. The model decides what to do. The harness decides what it is allowed to do, what it sees, what it remembers, and what shape its output has to take before the rest of the system will accept it. Moving from a static pipeline to an agentic workflow meant the harness, not the prompt, became the primary engineering surface, and each new failure mode that the agentic workflow introduced was addressed as a harness responsibility.
The first responsibility is the envelope the agent runs inside. Three ingestion profiles, named quick, standard, and deep, define hard ceilings on the units of work processed per run, the number of agent requests, and the number of tool calls. Evidence budgets cap the characters of content and the number of discovered references that can enter the agent's context, so a single dense source cannot crowd out the rest of the run, and timeouts on every external call prevent a slow source from stalling ingestion indefinitely. Within that envelope, independent extraction tasks run in parallel and in batches, with semaphores capping the number of LLM calls in flight at any moment so concurrency cannot push the system past its rate limits. The agent chooses what to do inside the envelope. It cannot expand the envelope itself.
Memory and scope come next. The agent tracks every reference it has already processed within a run, so the same item reached through multiple paths is still handled once, and runs over the same source produce the same set of processed units every time. Extracted facts are grouped by entity type before deduplication, so comparisons happen inside a relevant cluster rather than across the whole fact set. This eliminates the false positives that occurred when unrelated categories matched on surface similarity, and it makes each LLM call cheaper than a global one would have been, since the call operates on one cluster at a time.
The last layer is what the harness refuses to let through. We use Pydantic AI because it validates every LLM response against a declared schema before it reaches the application.
from pydantic import BaseModel
from pydantic_ai import Agent
class Fact(BaseModel):
content: str
source: str
class Response(BaseModel):
facts: list[Fact]
agent = Agent(model, output_type=Response, system_prompt=prompt)
result = await agent.run(input_text)
An invalid response is retried automatically with the validation error fed back in as context, so the application never sees malformed data. This is paired with deliberate retry asymmetry. Read tools get one retry to avoid loop spirals, while write tools get three to tolerate transient failures, and exceptions are explicitly split between those that should be retried and those that should not. Safety lives in the same layer. Ingested content is treated as adversarial regardless of where it came from and screened for prompt injection before reaching the agent. Extracted facts are passed through Bedrock Guardrails on the way out, batched for efficiency, with a fallback that checks each fact individually when a batch fails, and a safe default that drops the batch on API error so unchecked content never escapes. Every guardrail check, LLM call, and pipeline stage emits a structured Logfire span, which is what makes the system debuggable once parallel LLM calls are running at volume.
The model did not get smarter between the static pipeline and this one. The harness got better at constraining, observing, and recovering from it.
TL;DR: The improved pipeline worked, but running multiple jobs concurrently maxed out CPU and froze the app, since the heavy agent work was sharing an instance with the main application. We moved that work to a managed SQS queue with dedicated workers and added stage-level checkpointing so failed jobs resume where they left off. The result: concurrent runs that stayed within the latency budget and failed gracefully.
The improved pipeline gave us the performance and latency we were after, but it did so at the expense of compute. The semaphores in the harness cap concurrent LLM calls within a single ingestion run; what they cannot do is stop two or three runs from each maxing out the same instance. Once we began running ingestion jobs concurrently, even though every job was asynchronous by design, CPU utilization on the application instance would climb past 80%, at which point the entire application would freeze and stall the very jobs we were trying to run. The cause was the pipeline itself. Each agent was now doing heavy parallel work across stages, at times driving multiple browser sessions and issuing several LLM calls at the same time. Async scheduling alone was not enough; the contention was real, and it was happening on the same instance as the main application.
The same engineering thinking that shaped the harness inside a single run had to extend across runs once load became real. The obvious candidate for moving work off the main application was a Celery and Redis setup. The drawback was operational rather than architectural: choosing Celery meant standing up and scaling the Redis broker ourselves and owning the queue's availability, persistence, and scaling. For our use case, that was overhead we did not want to take on. We went instead with a worker architecture backed by an SQS queue. Because our stack was already on AWS, the integration was straightforward, and because SQS is fully managed, queue scalability and service maintenance were not problems we had to solve. Multiple agent workloads could now run concurrently without affecting the main application's performance, and ingestion finished reliably within the latency budget required by the product.
To push reliability further, we layered in checkpointing between stages along with retries. The pipeline itself is split into a fixed sequence of stages, and each completed stage is recorded in two places: a row in a relational index that tracks which stages have completed for a given run, and the stage's actual output stored compressed in an object store so the index stays cheap to query even when payloads are large. Resumption is implicit. A worker that picks a job up again walks through the same sequence of stages, finds the early ones already marked complete, loads their outputs from the object store, and starts real work at the first stage that is incomplete. The same loop that runs a fresh job runs a recovered one. Because an ingestion run can take half an hour, a failure 10 minutes in would otherwise mean repeating that work; with checkpoints in place, the agent picks up where it left off rather than starting over. Combined with the managed queue's visibility timeouts and dead letter handling, this gave us a pipeline that ran reliably and failed gracefully.
The combined effect of the agentic approach and the infrastructure separation produced measurable improvements across the metrics that defined the project's success criteria. The clearest gain was in throughput. Deep ingestion of a URL source, where the agent follows internal links, crawls multiple levels, and populates every relevant category, dropped from roughly 4 hours to 30 minutes. A shallow ingestion of 5 pages dropped from 30 minutes to 10 minutes once ingestion was moved off the main application.
Quality moved in step with speed. Fact coverage, the share of expected categories for which the pipeline produced a correct value when judged against ground truth, rose from 50% to 88%. The dynamic extraction step reached categories that the fixed pipeline had consistently left empty, and clustering facts by entity before deduplication removed the false matches that had been eroding accuracy as the knowledge base grew. The jump tracks closely with the change in how the agent was allowed to decide what to extract.
Operational consistency improved alongside both. Ingestion runs became deterministic, in the sense that the same source processed on different occasions produces the same output. The agent tracks every reference it has already processed within a run, so a source reached through multiple paths is still handled once. Checkpointing reinforces the same property across failures: a run that resumes from the last completed stage lands on the same result as one that ran straight through. That property is a prerequisite for debugging, regression testing, and confident deployment, and it was not something the earlier pipeline had. Before either was in place, variability across runs made it difficult to tell a genuine data quality problem apart from a crawl path that happened to take a different route on a different day.
Improvement▲▼ | Result▲▼ | Business outcome▲▼ |
|---|---|---|
Smarter extraction | Coverage 50% → 88% | Answers more questions correctly |
Work offloaded to SQS workers | Deep runs 4 hrs → 30 min | Onboard sources in minutes, not hours |
Concurrent, managed queue | Many jobs run in parallel | Scales across clients without ops strain |
Checkpointing + dedup | Deterministic, resumable runs | Reliable releases, no wasted compute |
Grounding + guardrails | Traceable, validated facts | Trustworthy, auditable answers |
Building this pipeline reframed what we thought we were building. We started out thinking the work was in the agent: the prompts, the reasoning, the model choice. By the end, most of the engineering attention had moved to the layer around it, the harness that constrains what the agent can do, validates what it produces, and recovers when it fails. None of those are things the model can be relied on to do itself. The happy path is the easy part. What separates a demo from a production system is how it behaves when an input is malformed, a tool times out, work fails partway through, or many copies of the agent run at once.
What's striking is that almost none of this is new. Failure handling, validation, observability, scalability: these are the same disciplines that have always separated a working system from a fragile one. The novelty of agentic systems is in what they fail at, not in how you fix it. The failure modes are unfamiliar, but the fixes are ordinary engineering, and that asymmetry is what makes the harness work feel deceptively boring even though it is what determines whether the project ships. Tool choices follow from the same logic. We chose SQS over Celery to avoid owning a broker, and Pydantic AI so we didn't hand-write retry logic for every malformed response. A tool whose maintenance cost competes with the work you are trying to do is the wrong tool, however capable it is.
So if you are building an agent for production, plan to spend more of your time on the engineering around it than on the agent itself. The model will improve on its own. The harness will not.