Observing and evaluating AI agentic workflows with Strands Brokers SDK and Arize AX

This submit is co-written with Wealthy Younger from Arize AI.

Agentic AI functions constructed on agentic workflows differ from conventional workloads in a single necessary approach: they’re nondeterministic. That’s, they will produce completely different outcomes with the identical enter. It is because the giant language fashions (LLMs) they’re based mostly on use possibilities when producing every token. This inherent unpredictability can lead AI utility designers to ask questions associated to the correction plan of motion, the optimum path of an agent and the proper set of instruments with the correct parameters. Organizations that wish to deploy such agentic workloads want an observability system that may be sure that they’re producing outcomes which might be right and could be trusted.

On this submit, we current how the Arize AX service can hint and consider AI agent duties initiated by way of Strands Brokers, serving to validate the correctness and trustworthiness of agentic workflows.

Challenges with generative AI functions

The trail from a promising AI demo to a dependable manufacturing system is fraught with challenges that many organizations underestimate. Based mostly on business analysis and real-world deployments, groups face a number of vital hurdles:

• Unpredictable conduct at scale – Brokers that carry out effectively in testing would possibly fail with surprising inputs in manufacturing, akin to new language variations or domain-specific jargon that trigger irrelevant or misunderstood responses.
• Hidden failure modes – Brokers can produce believable however fallacious outputs or skip steps unnoticed, akin to miscalculating monetary metrics in a approach that appears right however misleads decision-making.
• Nondeterministic paths – Brokers would possibly select inefficient or incorrect choice paths, akin to taking 10 steps to route a question that ought to take solely 5, resulting in poor consumer experiences.
• Instrument integration complexity – Brokers can break when calling APIs incorrectly, for instance, passing the fallacious order ID format so {that a} refund silently fails regardless of a profitable stock replace.
• Price and efficiency variability – Loops or verbose outputs may cause runaway token prices and latency spikes, akin to an agent making greater than 20 LLM calls and delaying a response from 3 to 45 seconds.

These challenges imply that conventional testing and monitoring approaches are inadequate for AI techniques. Success requires a extra considerate method that includes a extra complete technique.

Arize AX delivers a complete observability, analysis, and experimentation framework

Arize AX is the enterprise-grade AI engineering service that helps groups monitor, consider, and debug AI functions from improvement to manufacturing lifecycle. Incorporating Arize’s Phoenix basis, AX provides enterprise necessities such because the “Alyx” AI assistant, on-line evaluations, computerized immediate optimization, role-based entry management (RBAC), and enterprise scale and assist. AX presents a complete resolution to organizations that caters to each technical and nontechnical personas to allow them to handle and enhance AI brokers from improvement by way of manufacturing at scale. Arize AX capabilities embrace:

• Tracing – Full visibility into LLM operations utilizing OpenTelemetry to seize mannequin calls, retrieval steps, and metadata akin to tokens and latency for detailed evaluation.
• Analysis – Automated high quality monitoring with LLM-as-a-judge evaluations on manufacturing samples, supporting customized evaluators and clear success metrics.
• Datasets – Keep versioned, consultant datasets for edge circumstances, regression checks, and A/B testing, refreshed with actual manufacturing examples.
• Experiments – Run managed checks to measure the affect of adjustments to prompts or fashions, validating enhancements with statistical rigor.
• Playground – Interactive atmosphere to replay traces, check immediate variations, and examine mannequin responses for efficient debugging and optimization.
• Immediate administration – Model, check, and deploy prompts like code, with efficiency monitoring and gradual rollouts to catch regressions early.
• Monitoring and alerting – Actual-time dashboards and alerts for latency, errors, token utilization, and drift, with escalation for vital points.
• Agent visualization – Analyze and optimize agent choice paths to cut back loops and inefficiencies, refining planning methods.

These parts type a complete observability technique that treats LLM functions as mission-critical manufacturing techniques requiring steady monitoring, analysis, and enchancment.

Arize AX and Strands Brokers: A robust mixture

Strands Brokers is an open supply SDK, a strong low-code framework for constructing and operating AI brokers with minimal overhead. Designed to simplify the event of refined agent workflows, Strands unifies prompts, instruments, LLM interactions, and integration protocols right into a single streamlined expertise. It helps each Amazon Bedrock hosted and exterior fashions, with built-in capabilities for Retrieval Augmented Technology (RAG), Mannequin Context Protocol (MCP), and Agent2Agent (A2A) communication. On this part, we stroll by way of constructing an agent with Strands Agent SDK, instrumenting it with Arize AX for trace-based analysis, and optimizing its conduct.

The next workflow exhibits how a Strands agent handles a consumer job end-to-end—invoking instruments, retrieving context, and producing a response—whereas sending traces to Arize AX for analysis and optimization.

The answer follows these high-level steps:

1. Set up and configure the dependencies
2. Instrument the agent for observability
3. Construct the agent with Strands SDK
4. Take a look at the agent and generate traces
5. Analyze traces in Arize AI
6. Consider the agent’s conduct
7. Optimize the agent
8. Regularly monitor the agent

Conditions

You’ll want:

• An AWS account with entry to Amazon Bedrock
• An Arize account along with your House ID and API Key (join at no extra value at arize.com).

Set up dependencies:pip set up strands opentelemetry-sdk arize-otel

Resolution walkthrough: Utilizing Arize AX with Strands Brokers

The mixing between Strands Agent SDK and Arize AI’s observability system supplies deep, structured visibility into the conduct and choices of AI brokers. This setup allows end-to-end tracing of agent workflows—from consumer enter by way of planning, software invocation, and remaining output.

Full implementation particulars can be found within the accompanying pocket book and assets within the Openinference-Arize repository in GitHub.

Set up and configure the dependencies

To put in and configure the dependencies, use the next code:

from opentelemetry import hint
from opentelemetry.sdk.hint.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from strands_to_openinference_mapping import StrandsToOpenInferenceProcessor
from arize.otel import register
import grpc

Instrument the agent for observability

To instrument the agent for observability, use the next code.

•  The StrandsToOpenInferenceProcessor converts native spans to OpenInference format.
•  trace_attributes add session and consumer context for richer hint filtering.

Use Arize’s OpenTelemetry integration to allow tracing:

register(
    space_id="your-arize-space-id",
    api_key="your-arize-api-key",
    project_name="strands-project",
    processor=StrandsToOpenInferenceProcessor()
)
agent = Agent(
    mannequin=mannequin,
    system_prompt=system_prompt,
    instruments=[
        retrieve, current_time, get_booking_details,
        create_booking, delete_booking
    ],
    trace_attributes={
        "session.id": "abc-1234",
        "consumer.id": "user-email@instance.com",
        "arize.tags": [
            "Agent-SDK",
            "Arize-Project",
            "OpenInference-Integration"
        ]
    }
)

Construct the agent with Strands SDK

Create the Restaurant Assistant agent utilizing Strands. This agent will assist clients with restaurant info and reservations utilizing a number of instruments:

1. retrieve – Searches the information base for restaurant info
2. current_time – Will get the present time for reservation scheduling
3. create_booking – Creates a brand new restaurant reservation
4. get_booking_details – Retrieves particulars of an current reservation
5. delete_booking – Cancels an current reservation

The agent makes use of Anthropic’s Claude 3.7 Sonnet mannequin in Amazon Bedrock for pure language understanding and technology. Import the required instruments and outline the agent:

import get_booking_details, delete_booking, create_booking
from strands_tools import retrieve, current_time
from strands import Agent, software
from strands.fashions.bedrock import BedrockModel
import boto3
system_prompt = """You're "Restaurant Helper", a restaurant assistant serving to clients reserving tables in several eating places. You may speak concerning the menus, create new bookings, get the small print of an current reserving or delete an current reservation. You reply at all times politely and point out your title within the reply (Restaurant Helper)..........."""
mannequin = BedrockModel(
    model_id="us.anthropic.claude-3-7-sonnet-20250219-v1:0",
)
kb_name="restaurant-assistant"
smm_client = boto3.consumer('ssm')
kb_id = smm_client.get_parameter(
    Title=f'{kb_name}-kb-id',
    WithDecryption=False
)
os.environ["KNOWLEDGE_BASE_ID"] = kb_id["Parameter"]["Value"]
agent = Agent(
    mannequin=mannequin,
    system_prompt=system_prompt,
    instruments=[
        retrieve, current_time, get_booking_details,
        create_booking, delete_booking
    ],
    trace_attributes={
        "session.id": "abc-1234",
        "consumer.id": "user-email-example@area.com",
        "arize.tags": [
            "Agent-SDK",
            "Arize-Project",
            "OpenInference-Integration",
        ]
    }
)

Take a look at the agent and generate traces

Take a look at the agent with a few queries to generate traces for Arize. Every interplay will create spans in OpenTelemetry that will probably be processed by the customized processor and despatched to Arize AI.The primary check case is a restaurant info question. Ask about eating places in San Francisco. This may set off the information base retrieval software:

# Take a look at with a query about eating places
outcomes = agent("Hello, the place can I eat in New York?")
print(outcomes)

The second check case is for a restaurant reservation. Take a look at the reserving performance by making a reservation. This may set off the create_booking software:

# Take a look at with a reservation request
outcomes = agent("Make a reservation for tonight at Rice & Spice. At 8pm, for two folks within the title of Anna")
print(outcomes)

Analyze traces in Arize AI

After operating the agent, you’ll be able to view and analyze the traces within the Arize AI dashboard, proven within the following screenshot. Hint-level visualization exhibits the illustration of the hint to substantiate the trail that the agent took throughout execution. Within the Arize dashboard, you’ll be able to overview the traces generated by the agent. By choosing the strands-project you outlined within the pocket book, you’ll be able to view your traces on the LLM Tracing tab. Arize supplies highly effective filtering capabilities that will help you deal with particular traces. You may filter by OTel attributes and metadata, for instance, to research efficiency throughout completely different fashions.

You may also use Alyx AI assistant, to research your agent’s conduct by way of pure language queries and uncover insights. Within the instance under, we use Alyx to purpose about why a software was invoked incorrectly by the agent in one of many traces, serving to us determine the basis reason for the misstep

Selecting a selected hint provides detailed details about the agent’s runtime efficiency and decision-making course of, as proven within the following screenshot.

The graph view, proven within the following screenshot, exhibits the hierarchical construction of your agent’s execution and customers can examine particular execution paths to grasp how the agent made choices by choosing the graph.

You may also view session-level insights on the Classes tab subsequent to LLM Tracing. By tagging spans with session.id and consumer.id, you’ll be able to group associated interactions, determine the place conversations break down, monitor consumer frustration, and consider multiturn efficiency throughout periods.

Consider the agent’s conduct

Arize’s system traces the agent’s decision-making course of, capturing particulars akin to routing choices, software calls and parameters. You may consider efficiency by analyzing these traces to confirm that the agent selects optimum paths and supplies correct responses. For instance, if the agent misinterprets a buyer’s request and chooses the fallacious software or makes use of incorrect parameters, Arize evaluators will determine when these failures happen.Arize has pre-built analysis templates for each step of your Agent course of:

Create a brand new job below Evals and Duties and select LLM as a decide job kind. You should use a pre-built immediate template (software calling is used within the instance proven within the following screenshot) or you’ll be able to ask Alyx AI assistant to construct one for you. Evals will now mechanically run in your traces as they stream into Arize. This makes use of AI to mechanically label your knowledge and determine failures at scale with out human intervention.

Now each time the agent is invoked, hint knowledge is collected in Arize and the software calling analysis mechanically runs and labels the info with a right or incorrect label together with a proof by the LLM-as-a-judge for its labeling choice. Right here is an instance of an analysis label and clarification.

Optimize the agent

The LLM-as-a-judge evaluations mechanically determine and label failure circumstances the place the agent didn’t name the correct software. Within the under screenshot these failure circumstances are mechanically captured and added to a regression dataset, which can drive agent enchancment workflows. This manufacturing knowledge can now gasoline improvement cycles for bettering the agent.

Now, you’ll be able to join immediately with Arize’s immediate playground, an built-in improvement atmosphere (IDE) the place you’ll be able to experiment with varied immediate adjustments and mannequin selections, examine side-by-side outcomes and check throughout the regression dataset from the earlier step. When you’ve gotten an optimum immediate and mannequin mixture, it can save you this model to the immediate hub for future model monitoring and retrieval, as proven within the following screenshot.

Experiments from the immediate testing are mechanically saved, with on-line evaluations run and outcomes saved for fast evaluation and comparability to facilitate data-driven choices on what enhancements to deploy. Moreover, experiments could be integrated into steady integration and steady supply (CI/CD) workflows for automated regression testing and validation every time new immediate or utility adjustments are pushed to techniques akin to GitHub. The screenshot under exhibits hallucination metrics for immediate experiments.

Regularly monitor the agent

To take care of reliability and efficiency in manufacturing, it’s important to repeatedly monitor your AI brokers. Arize AI supplies out-of-the-box monitoring capabilities that assist groups detect points early, optimize value, and supply high-quality consumer experiences.Establishing displays in Arize AI presents:

• Early subject detection – Determine issues earlier than they affect customers
• Efficiency monitoring – Monitor traits and preserve constant agent conduct
• Price administration – Observe token utilization to keep away from pointless bills
• High quality assurance – Validate your agent is delivering correct, useful responses

You may entry and configure displays on the Displays tab in your Arize challenge. For particulars, consult with the Arize documentation on monitoring.

When monitoring your Strands Agent in manufacturing, pay shut consideration to those key metrics:

• Latency – Time taken for the agent to answer consumer inputs
• Token utilization – Variety of tokens consumed, which immediately impacts value
• Error fee – Frequency of failed responses or software invocations
• Instrument utilization – Effectiveness and frequency of software calls
• Consumer satisfaction indicators – Proxy metrics akin to software name correctness, dialog size, or decision charges

By frequently monitoring these metrics, groups can proactively enhance agent efficiency, catch regressions early, and ensure the system scales reliably in real-world use. In Arize, you’ll be able to create customized metrics immediately from OTel hint attributes or metadata, and even from analysis labels and metrics, such because the software calling correctness analysis you created beforehand. The screenshot under visualizes the software name correctness ratio throughout agent traces, serving to determine patterns in right versus incorrect software utilization

The screenshot under illustrate how Arize supplies customizable dashboards that allow deep observability into LLM agent efficiency, showcasing a customized monitoring dashboard monitoring core metrics akin to latency, token utilization, and the share of right software calls.

The screenshot under demonstrates prebuilt templates designed to speed up setup and provide fast visibility into key agent behaviors.

Clear up

Whenever you’re executed experimenting, you’ll be able to clear up the AWS assets created by this pocket book by operating the cleanup script: !sh cleanup.sh.

Conclusion

The important thing lesson is evident: observability, computerized evaluations, experimentation and suggestions loops, and proactive alerting aren’t non-compulsory for manufacturing AI—they’re the distinction between innovation and legal responsibility. Organizations that spend money on correct AI operations infrastructure can harness the transformative energy of AI brokers whereas avoiding the pitfalls which have plagued early adopters. The mix of Amazon Strands Brokers and Arize AI supplies a complete resolution that addresses these challenges:

• Strands Brokers presents a model-driven method for constructing and operating AI brokers
• Arize AI provides the vital observability layer with tracing, analysis, and monitoring capabilities

The partnership between AWS and Arize AI presents a strong resolution for constructing and deploying generative AI brokers. The totally managed framework of Strands Brokers simplifies agent improvement, and Arize’s observability instruments present vital insights into agent efficiency. By addressing challenges akin to nondeterminism, verifying correctness, and enabling continuous monitoring, this integration advantages organizations in that they will create dependable and efficient AI functions. As companies more and more undertake agentic workflows, the mixture of Amazon Bedrock and Arize AI units a brand new commonplace for reliable AI deployment.

Get began

Now that you just’ve realized learn how to combine Strands Brokers with the Arize Observability Service, you can begin exploring various kinds of brokers utilizing the instance offered on this pattern. As a subsequent step, strive increasing this integration to incorporate automated evaluations utilizing Arize’s analysis framework to attain agent efficiency and choice high quality.

Able to construct higher brokers? Get began with an account at arize.com for no extra value and start remodeling your AI brokers from unpredictable experiments into dependable, production-ready options. The instruments and information are right here; the one query is: what’s going to you construct?

In regards to the Authors

Wealthy Younger is the Director of Associate Options Structure at Arize AI, centered on AI agent observability and analysis tooling. Previous to becoming a member of Arize, Wealthy led technical pre-sales at WhyLabs AI. In his pre-AI life, Wealthy held management and IC roles at enterprise expertise firms akin to Splunk and Akamai.

Karan Singh is a Agentic AI chief at AWS, the place he works with top-tier third-party basis mannequin and agentic frameworks suppliers to develop and execute joint go-to-market methods, enabling clients to successfully deploy and scale options to unravel enterprise agentic AI challenges. Karan holds a BS in Electrical Engineering from Manipal College, a MS in Electrical Engineering from Northwestern College, and an MBA from the Haas College of Enterprise at College of California, Berkeley.

Nolan Chen is a Associate Options Architect at AWS, the place he helps startup firms construct revolutionary options utilizing the cloud. Previous to AWS, Nolan specialised in knowledge safety and serving to clients deploy high-performing large space networks. Nolan holds a bachelor’s diploma in mechanical engineering from Princeton College.

Venu Kanamatareddy is an AI/ML Options Architect at AWS, supporting AI-driven startups in constructing and scaling revolutionary options. He supplies strategic and technical steerage throughout the AI lifecycle from mannequin improvement to MLOps and generative AI. With expertise throughout startups and enormous enterprises, he brings deep experience in cloud structure and AI options. Venu holds a level in laptop science and a grasp’s in synthetic intelligence from Liverpool John Moores College.