Building an Agent Harness

Deep dive into building an agent harness that gives an LLM access to arbitrary files and code execution to extract structured data from PDFs, Excel workbooks, and beyond.

Source code

Table of Contents

The agent harness

An agent harness is the runtime scaffolding that turns a language model into an agent: a loop that calls the model, picks tools, passes results back, manages context, and handles failures.

A good example of this is Claude Code — Anthropic’s models are capable on their own, but what makes Claude Code amazing is the harness built around them: the tools, context management, and execution loop. The harness matters as much as the model, if not more.

In this post we will explore data extraction from arbitrary files as an excuse to build our own harness. Simply sending raw bytes to the model doesn’t always work — serializing a PDF or Excel workbook as text will often cause data loss, whether from formatting or content hidden inside those files like charts and images. And if you work with enough files, it isn’t practical to do so at all. The harness is what gives your LLM access to those files and the ability to execute code, making the process more flexible and reliable.

Most model and cloud providers have their own version of an agent harness now, but for this post I’ll demonstrate how to build one using @strands-agents/sdk and Amazon Bedrock AgentCore Code Interpreter. Strands provides the abstraction to build an agent with skills and tools, while AgentCore Code Interpreter provides a sandbox environment with a file system and code execution capabilities.

Remember this is all an abstraction — in the end we are just making API calls to an LLM with tool calling capabilities.

The execute code tool

The way AgentCore Code Interpreter works is simple: we can create sessions, which are isolated environments in which we can issue commands to execute code, shell scripts or work with the file system. We then expose the desired capabilities to our agent as tools:

export function createCodeExecutionTool(
  sandbox: AgentCoreSandbox,
  sessionId: string,
) {
  return tool({
    name: "execute_code",
    description:
      "Execute Python code in the AgentCore code interpreter sandbox. Use this to run computations, data analysis, or any Python code.",
    inputSchema: z.object({
      code: z.string().describe("The Python code to execute"),
    }),
    callback: async ({ code }) => {
      try {
        const result = await sandbox.executeCode(sessionId, code);
        if (result.isError) {
          return `Error:\n${result.output}`;
        }
        return result.output || "(no output)";
      } catch (err) {
        console.error("execute_code failed:", err);
        throw err;
      }
    },
  });
}

The snippet above uses a simple abstraction to interact with the code interpreter, implementation details are available in the source repository as usual.

AgentCore lets you have up to 1,000 concurrent sessions, with a duration between 1 minute and 8 hours. After stopping a session all data will be lost. There is a default interpreter that comes with the AWS account, but you can create your own with different configuration options.

Our first agent

The following agent uses a Bedrock model and the previously defined code execution tool to perform calculations. If we ask for the Fibonacci sequence, it will generate the Python code to calculate it. This is generally more accurate than letting the LLM compute things itself.

import { Agent, BedrockModel } from "@strands-agents/sdk";
import { AgentCoreSandbox } from "./platform/agentcore-sandbox.js";
import { createCodeExecutionTool } from "./tools/execute-code-tool.js";

export async function fibonacci(sandbox: AgentCoreSandbox) {
  const sessionId = await sandbox.startSession();
  const agent = new Agent({
    model: new BedrockModel({
      region: "us-east-1",
      modelId: "us.anthropic.claude-haiku-4-5-20251001-v1:0",
      cacheConfig: { strategy: "auto" },
    }),
    tools: [createCodeExecutionTool(sandbox, sessionId)],
    systemPrompt:
      "You are an agent capable of executing Python code through the execute_code tool. Use it when performing calculations.",
  });
  await agent.invoke([
    { text: "Calculate the first 10 numbers of the fibonacci sequence" },
  ]);
}

There are many agent SDKs, but most of them are pretty similar — they allow defining a system prompt, tools, and skills, and most come with ways to manage context, conversations, and more complex workflows.

Making our files available

AgentCore Code Interpreter allows uploading files in multiple ways. We can directly post the files over HTTP, give the Interpreter an execution role to connect to S3 storage, or bring your own file system through access points — though I have no hands-on experience with that last option.

The first method is straightforward, with a 100MB per-file limit, but requires downloading files to your server before uploading them. The second supports up to 5GB and avoids that round-trip, but requires more care around setup and security.

If you go with S3, you can run aws s3 cp as many times as needed or write a shell script directly to the sandbox and execute it — making it straightforward to implement parallel downloads and retries. Keep in mind though that giving the sandbox S3 access comes with security considerations.

Giving the sandbox access to S3 means the LLM could potentially access files it isn’t intended to — imagine multiple users uploading confidential documents to the same bucket. You can greatly limit the attack surface by restricting the Code Interpreter role to s3:GetObject and using non-guessable names, for example hashing the user email to create a folder per user. The LLM could still execute commands to download any file, but without the ability to list bucket contents it is much harder to exploit. Also consider whether users interact with your agent directly — if so, be mindful of prompt injection.

Working with different file types

AgentCore Code Interpreter comes with a set of pre-installed libraries the agent can use to do a wide variety of tasks. We can create skills that leverage those to work with PDFs, Excel and other file formats.

For PDFs, the best approach is to analyze each page independently, and treat text, tables, and images differently. The same strategy applies to workbooks: first peek at the structure, then use different libraries depending on the content.

If you want a good reference implementation, ask Claude how it analyzes PDFs or spreadsheets, and have a peek at its own skills. You can ask your coding agent to create skills based on those but using the tools available in the sandbox.

Giving our agent vision capabilities

In order to analyze images or content that is not properly readable as text, we can instruct our agent to save images to the sandbox disk, and call a tool that will put it in the conversation context encoded in base 64. This works because multimodal models are capable of interpreting different content types at the same time.

You could also implement an OCR tool in the same fashion and explain your agent the trade-offs. Using the vision capabilities of the model is more accurate but consumes more tokens and should be used especially for images found inside the documents and charts, while OCR is cheaper and can be used as a fallback for scanned documents and the like.

This is a sample implementation of the view_image tool:

export function createViewImageTool(
  sandbox: AgentCoreSandbox,
  sessionId: string,
) {
  return tool({
    name: "view_image",
    description:
      "Download a PNG image from the sandbox and put it in context so you can see it.",
    inputSchema: z.object({
      path: z
        .string()
        .describe("Absolute or relative path to the PNG file in the sandbox"),
    }),
    callback: async ({ path }) => {
      try {
        const bytes = await sandbox.readFile(sessionId, path);
        return [{ image: { format: "png", source: { bytes } } }];
      } catch (err) {
        console.error("view_image failed:", err);
        return `Error reading image: ${err instanceof Error ? err.message : String(err)}`;
      }
    },
  });
}

With this set of tools and the appropriate skills, our agent can understand virtually any kind of document.

This is the final implementation:

export function createAgent(
  config: Config,
  sandbox: AgentCoreSandbox,
  sessionId: string,
): Agent {
  return new Agent({
    model: new BedrockModel({
      region: config.awsRegion,
      modelId: config.bedrockModelId,
      cacheConfig: { strategy: "auto" },
    }),
    tools: [
      createCodeExecutionTool(sandbox, sessionId),
      createViewImageTool(sandbox, sessionId),
    ],
    plugins: [new AgentSkills({ skills: ["./src/skills/"] })],
    systemPrompt:
      "You are a helpful data extraction assistant with access to a Python sandbox.\n\n" +
      "## Tools\n" +
      "- **execute_code**: Run Python in the sandbox for computation, file processing, and analysis.\n" +
      "- **view_image**: Download an image from the sandbox and view it. Use after saving any chart or plot.",
  });
}

If we upload some crafted company sample files and invoke it with the following prompt:

Look at the files in the documents directory. Extract the regional revenue figures from the PDF and the total cost per department from the Headcount sheet in the Excel file.

We will observe how the LLM uses the capabilities that we have made available to it to fulfill our request:

I'll help you extract that data. Let me first explore the documents directory to see what files are available.
  ⏳ execute_code

🔧 Tool #1: execute_code
✓ Tool completed
Great! I found the files. Now let me extract the regional revenue figures from the PDF and the total cost per department from the Excel file. First, let me load the relevant skills.
  ⏳ skills

  ⏳ skills

🔧 Tool #2: skills

🔧 Tool #3: skills
✓ Tool completed
✓ Tool completed
Now let me extract the data from both files:
  ⏳ execute_code

🔧 Tool #4: execute_code
✓ Tool completed
The PDF has a chart but the table data isn't being extracted as a table. Let me render it as an image to see the exact revenue figures:
  ⏳ execute_code

🔧 Tool #5: execute_code
✓ Tool completed

  ⏳ view_image

🔧 Tool #6: view_image
✓ Tool completed
Perfect! I can see the chart. Now let me extract the Excel data and then compile the results:
  ⏳ execute_code
  ...

See the full transcription here.

If you need even more flexibility you can give the sandbox access to the internet or use AgentCore harness directly, which allows you to create your own images with pre-installed libraries for the sandbox container. Anthropic also offers the same sandbox Claude uses with the same libraries and skills so you don’t need to build everything from scratch.

Deploying your agent

Remember, an agent is just an abstraction. You can have interactive agents with users in the loop, run them as part of automated workflows, or simply invoke them from a REST endpoint in your C# API. Where you deploy them depends entirely on your use case: a Node.js server in a Kubernetes pod, a Python server in AgentCore, or a Lambda function reacting to an SQS event are all valid options. For conversational agents, platforms like AgentCore Runtime are purpose-built and come with benefits worth considering, but weigh them against more general-purpose options that may give you more flexibility in terms of scaling, cost, and control over your infrastructure.

Limits, cost, and observability

Before optimizing for speed or cost, you need to understand the two constraints that can break your agent entirely: the model’s maximum context window and its maximum output tokens per message. If the context on the last turn is close to the limit, you may need context summarization or sub-agents to offload the conversation. If the agent’s output is consistently hitting the output token ceiling, consider whether it can be made more concise — or whether you need chunking at the application level. Err on the side of small, focused agents that own a specific part of your workflow rather than a few agents that try to do everything.

Once the hard limits are under control, the main cost driver to watch is total output tokens. Input tokens matter too, but the real lever is caching: when you cache the system prompt and tool definitions, repeated runs share that cost across calls rather than paying it fresh every time. This is inference-level caching, not a simple input/output ratio — keep an eye on cache read and write counts to confirm it’s actually working.

Finally, store the full transcription of every run. That’s the only way to understand what the agent is actually doing — and that understanding is what lets you improve its instructions, catch unnecessary steps or common errors, and tune it over time.