Create an agentic RAG utility for superior data discovery with LlamaIndex, and Mistral in Amazon Bedrock

Agentic Retrieval Augmented Technology (RAG) functions symbolize a sophisticated strategy in AI that integrates basis fashions (FMs) with exterior data retrieval and autonomous agent capabilities. These programs dynamically entry and course of data, break down complicated duties, use exterior instruments, apply reasoning, and adapt to numerous contexts. They transcend easy query answering by performing multi-step processes, making selections, and producing complicated outputs.

On this submit, we reveal an instance of constructing an agentic RAG utility utilizing the LlamaIndex framework. LlamaIndex is a framework that connects FMs with exterior information sources. It helps ingest, construction, and retrieve data from databases, APIs, PDFs, and extra, enabling the agent and RAG for AI functions.

This utility serves as a analysis instrument, utilizing the Mistral Giant 2 FM on Amazon Bedrock generate responses for the agent circulation. The instance utility interacts with well-known web sites, corresponding to Arxiv, GitHub, TechCrunch, and DuckDuckGo, and might entry data bases containing documentation and inside data.

This utility may be additional expanded to accommodate broader use instances requiring dynamic interplay with inside and exterior APIs, in addition to the mixing of inside data bases to supply extra context-aware responses to consumer queries.

Resolution overview

This resolution makes use of the LlamaIndex framework to construct an agent circulation with two principal elements: AgentRunner and AgentWorker. The AgentRunner serves as an orchestrator that manages dialog historical past, creates and maintains duties, executes activity steps, and supplies a user-friendly interface for interactions. The AgentWorker handles the step-by-step reasoning and activity execution.

For reasoning and activity planning, we use Mistral Giant 2 on Amazon Bedrock. You need to use different textual content technology FMs accessible from Amazon Bedrock. For the complete record of supported fashions, see Supported basis fashions in Amazon Bedrock. The agent integrates with GitHub, arXiv, TechCrunch, and DuckDuckGo APIs, whereas additionally accessing inside data by a RAG framework to supply context-aware solutions.

On this resolution, we current two choices for constructing the RAG framework:

You possibly can choose the RAG implementation possibility that most accurately fits your choice and developer talent degree.

The next diagram illustrates the answer structure.

Within the following sections, we current the steps to implement the agentic RAG utility. You can too discover the pattern code within the GitHub repository.

Stipulations

The answer has been examined within the AWS Area us-west-2. Full the next steps earlier than continuing:

1. Arrange the next sources:
   1. Create an Amazon SageMaker
   2. Create a SageMaker area consumer profile.
   3. Launch Amazon SageMaker Studio, choose JupyterLab, and create an area.
   4. Choose the occasion t3.medium and the picture SageMaker Distribution 2.3.1, then run the area.
2. Request mannequin entry:
   1. On the Amazon Bedrock console, select Mannequin entry within the navigation pane.
   2. Select Modify mannequin entry.
   3. Choose the fashions Mistral Giant 2 (24.07), Amazon Titan Textual content Embeddings V2, and Rerank 1.0 from the record, and request entry to those fashions.
3. Configure AWS Identification and Entry Administration (IAM) permissions:
   1. Within the SageMaker console, go to the SageMaker consumer profile particulars and discover the execution function that the SageMaker pocket book makes use of. It ought to seem like AmazonSageMaker-ExecutionRole-20250213T123456.
4. Within the IAM console, create an inline coverage for this execution function. that your function can carry out the next actions:
   1. Entry to Amazon Bedrock companies together with:
      • Reranking capabilities
      • Retrieving data
      • Invoking fashions
      • Itemizing accessible basis fashions
   2. IAM permissions to:
      • Create insurance policies
      • Connect insurance policies to roles inside your account
   3. Full entry to Amazon OpenSearch Serverless service
5. Run the next command within the JupyterLab pocket book terminal to obtain the pattern code from GitHub:

git init
git distant add origin https://github.com/aws-samples/mistral-on-aws.git
git sparse-checkout init
git sparse-checkout set "notebooks/mistral-llamaindex-agentic-rag"
git pull origin principal

6. Lastly, set up the required Python packages by working the next command within the terminal:

cd mistral-llamaindex-agentic-rag
pip set up -r necessities.txt

Initialize the fashions

Initialize the FM used for orchestrating the agentic circulation with Amazon Bedrock Converse API. This API supplies a unified interface for interacting with varied FMs accessible on Amazon Bedrock. This standardization simplifies the event course of, permitting builders to write down code one time and seamlessly swap between completely different fashions with out adjusting for model-specific variations. On this instance, we use the Mistral Giant 2 mannequin on Amazon Bedrock.

Subsequent, initialize the embedding mannequin from Amazon Bedrock, which is used for changing doc chunks into embedding vectors. For this instance, we use Amazon Titan Textual content Embeddings V2. See the next code:

# Initialise and configure the BedrockConverse LLM with the Mistral Giant 2 mannequin and set it because the default in Settings

from llama_index.llms.bedrock_converse import BedrockConverse
from llama_index.core import Settings
llm = BedrockConverse(mannequin="mistral.mistral-large-2407-v1:0", max_tokens = 2048)
Settings.llm = BedrockConverse(mannequin="mistral.mistral-large-2407-v1:0", max_tokens = 2048)

# Initialise and configure the embedding mannequin with Amazon Titan Textual content Embeddings V2, and set it because the default in Settings

from llama_index.embeddings.bedrock import BedrockEmbedding
embed_model = BedrockEmbedding(model_name="amazon.titan-embed-text-v2:0")
Settings.embed_model = BedrockEmbedding(model_name="amazon.titan-embed-text-v2:0")

Combine API instruments

Implement two capabilities to work together with the GitHub and TechCrunch APIs. The APIs proven on this submit don’t require credentials. To offer clear communication between the agent and the inspiration mannequin, observe Python operate greatest practices, together with:

• Sort hints for parameter and return worth validation
• Detailed docstrings explaining operate goal, parameters, and anticipated returns
• Clear operate descriptions

The next code pattern exhibits the operate that integrates with the GitHub API. After the operate is created, use the FunctionTool.from_defaults() technique to wrap the operate as a instrument and combine it seamlessly into the LlamaIndex workflow.

See the code repository for the complete code samples of the operate that integrates with the TechCrunch API.

# Outline a operate to go looking GitHub repositories by subject, sorting by stars or replace date, and return high outcomes
import requests
def github_search(subject: str, num_results: int = 3, sort_by: str = "stars") -> record:
    """
    Retrieve a specified variety of GitHub repositories based mostly on a given subject, 
    ranked by the desired standards.

    This operate makes use of the GitHub API to seek for repositories associated to a 
    particular subject or key phrase. The outcomes may be sorted by the variety of stars 
    (reputation) or the latest replace, with essentially the most related repositories 
    showing first based on the chosen sorting technique.

    Parameters:
    -----------
    subject : str
        The subject or key phrase to seek for in GitHub repositories.
        The subject can not include clean areas.
    num_results : int, optionally available
        The variety of repository outcomes to retrieve. Defaults to three.
    sort_by : str, optionally available
        The criterion for sorting the outcomes. Choices embody:
        - 'stars': Kind by the variety of stars (reputation).
        - 'up to date': Kind by the date of the final replace (most up-to-date first).
        Defaults to 'stars'.

    Returns:
    --------
    record
        An inventory of dictionaries, the place every dictionary accommodates data 
        a few repository. Every dictionary contains:
        - 'html_url': The URL of the repository.
        - 'description': A quick description of the repository.
        - 'stargazers_count': The variety of stars (reputation) the repository has.
    """

    url = f"https://api.github.com/search/repositories?q=subject:{subject}&type={sort_by}&order=desc"
    response = requests.get(url).json()
    code_repos = [
        {
            'html_url': item['html_url'],
            'description': merchandise['description'],
            'stargazers_count': merchandise['stargazers_count'],
        }
        for merchandise in response['items'][:num_results]
    ]
    return code_repos

github_tool = FunctionTool.from_defaults(fn=github_search)

For arXiv and DuckDuckGo integration, we use LlamaIndex’s pre-built instruments as a substitute of making customized capabilities. You possibly can discover different accessible pre-built instruments within the LlamaIndex documentation to keep away from duplicating current options.

# Import and configure the ArxivToolSpec and DuckDuckGoSearchToolSpec from LlamaIndex prebuilt instruments

from llama_index.instruments.arxiv import ArxivToolSpec
from llama_index.instruments.duckduckgo import DuckDuckGoSearchToolSpec

arxiv_tool = ArxivToolSpec()
search_tool = DuckDuckGoSearchToolSpec()

api_tools = arxiv_tool.to_tool_list() + search_tool.to_tool_list()

# Consolidate all instruments into one record. 
api_tools.lengthen([news_tool, github_tool])

RAG possibility 1: Doc integration with Amazon OpenSearch Serverless

Subsequent, programmatically construct the RAG part utilizing LlamaIndex to load, course of, and chunk paperwork.  retailer the embedding vectors in Amazon OpenSearch Serverless. This strategy provides better flexibility for superior eventualities, corresponding to loading varied file sorts (together with .epub and .ppt) and deciding on superior chunking methods based mostly on file sorts (corresponding to HTML, JSON, and code).

Earlier than shifting ahead, you may obtain some PDF paperwork for testing from the AWS web site utilizing the next command, or you should use your personal paperwork. The next paperwork are AWS guides that assist in choosing the proper generative AI service (corresponding to Amazon Bedrock or Amazon Q) based mostly on use case, customization wants, and automation potential. Additionally they help in deciding on AWS machine studying (ML) companies (corresponding to SageMaker) for constructing fashions, utilizing pre-trained AI, and utilizing cloud infrastructure.

# obtain check paperwork from beneath hyperlinks
!wget -O docs/genai_on_aws.pdf https://docs.aws.amazon.com/pdfs/decision-guides/newest/generative-ai-on-aws-how-to-choose/generative-ai-on-aws-how-to-choose.pdf?did=wp_card&trk=wp_card#information
!wget -O docs/ml_on_aws.pdf https://docs.aws.amazon.com/pdfs/decision-guides/newest/machine-learning-on-aws-how-to-choose/machine-learning-on-aws-how-to-choose.pdf?did=wp_card&trk=wp_card#information

Load the PDF paperwork utilizing SimpleDirectoryReader() within the following code. For a full record of supported file sorts, see the LlamaIndex documentation.

# use Llamaindex to load paperwork 
from llama_index.core import SimpleDirectoryReader
loader = SimpleDirectoryReader('docs/')
paperwork = loader.load_data()

Subsequent, create an Amazon OpenSearch Serverless assortment because the vector database. Verify the utils.py file for particulars on the create_collection() operate.

# Create Amazon OpenSearch Serverless assortment 
from utils import *
import sagemaker 
import random

region_name = "us-west-2"
suffix = random.randrange(1, 500)
collection_name = "llamaindex-blog-"+str(suffix)
notebook_execution_role = sagemaker.get_execution_role()
endpoint = create_collection(collection_name, notebook_execution_role)

After you create the gathering, create an index to retailer embedding vectors:

## create an index within the assortment
index_name = "pdf-rag"
create_index(index_name, endpoint, emb_dim=1024)

Subsequent, use the next code to implement a doc search system utilizing LlamaIndex built-in with Amazon OpenSearch Serverless. It first units up AWS authentication to securely entry OpenSearch Service, then configures a vector shopper that may deal with 1024-dimensional embeddings (particularly designed for the Amazon Titan Embedding V2 mannequin). The code processes enter paperwork by breaking them into manageable chunks of 1,024 tokens with a 20-token overlap, converts these chunks into vector embeddings, and shops them within the OpenSearch Serverless vector index. You possibly can choose a unique or extra superior chunking technique by modifying the transformations parameter within the VectorStoreIndex.from_documents() technique. For extra data and examples, see the LlamaIndex documentation.

import boto3
from llama_index.vector_stores.opensearch import  OpensearchVectorStore,   OpensearchVectorClient
from opensearchpy import RequestsHttpConnection, AWSV4SignerAuth
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.core.node_parser import SentenceSplitter

## combine Amazon OpenSearch Serverless assortment and index to llamaindex 

dim = 1024 # Amazon Titan Embedding V2 mannequin dimension 
service="aoss"
credentials = boto3.Session().get_credentials()
awsauth = AWSV4SignerAuth(credentials, region_name, service)

shopper = OpensearchVectorClient(
    endpoint, 
    index_name, 
    dim, 
    embedding_field="vector", 
    text_field="chunk",
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
)

# initialise vector retailer and save doc chunks to the vector retailer 
vector_store = OpensearchVectorStore(shopper)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

index = VectorStoreIndex.from_documents(
    paperwork, 
    storage_context=storage_context,
    transformations=[SentenceSplitter(chunk_size=1024, chunk_overlap=20)]
)

You possibly can add a reranking step within the RAG pipeline, which improves the standard of data retrieved by ensuring that essentially the most related paperwork are offered to the language mannequin, leading to extra correct and on-topic responses:

from llama_index.postprocessor.bedrock_rerank import AWSBedrockRerank
reranker = AWSBedrockRerank(
    top_n=3,
    model_id="amazon.rerank-v1:0",#  one other rerank mannequin possibility is: cohere.rerank-v3-5:0
    region_name="us-west-2",
)
query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[reranker],
)

Use the next code to check the RAG framework. You possibly can evaluate outcomes by enabling or disabling the reranker mannequin.

response = query_engine.question(
    "By which state of affairs ought to I take advantage of Amazon Bedrock over Amazon SageMaker?",
)

Subsequent, convert the vector retailer right into a LlamaIndex QueryEngineTool, which requires a instrument identify and a complete description. This instrument is then mixed with different API instruments to create an agent employee that executes duties in a step-by-step method. The code initializes an AgentRunner to orchestrate the whole workflow, analyzing textual content inputs and producing responses. The system may be configured to help parallel instrument execution for improved effectivity.

# create QueryEngineTool based mostly on the OpenSearch vector retailer 

from llama_index.core.instruments import QueryEngineTool, ToolMetadata
oss_tool = QueryEngineTool(
        query_engine=query_engine,
        metadata=ToolMetadata(
            identify="oss_guide_tool",
            description="""
            These determination guides assist customers choose applicable AWS machine studying and generative AI companies based mostly on particular wants. 
            They cowl pre-built options, customizable platforms, and infrastructure choices for ML workflows, 
            whereas outlining how generative AI can automate processes, personalize content material, increase information, scale back prices, 
            and allow sooner experimentation in varied enterprise contexts.""",
        ),
    )

all_tools = api_tools +[oss_tool]

agent_worker = FunctionCallingAgentWorker.from_tools(
    all_tools, 
    llm=llm, 
    verbose=True, # Set verbose=True to show the complete hint of steps. 
    system_prompt = system_prompt,
    # allow_parallel_tool_calls = True  # Uncomment this line to permit a number of instrument invocations
)
agent = AgentRunner(agent_worker)
response = agent.chat(text_input)

You may have now accomplished constructing the agentic RAG utility utilizing LlamaIndex and Amazon OpenSearch Serverless. You possibly can check the chatbot utility with your personal questions. For instance, ask in regards to the newest information and options relating to Amazon Bedrock, or inquire in regards to the newest papers and hottest GitHub repositories associated to generative AI.

RAG possibility 2: Doc integration with Amazon Bedrock Information Bases

On this part, you employ Amazon Bedrock Information Bases to construct the RAG framework. You possibly can create an Amazon Bedrock data base on the Amazon Bedrock console or observe the supplied pocket book instance to create it programmatically. Create a brand new Amazon Easy Storage Service (Amazon S3) bucket for the data base, then add the beforehand downloaded information to this S3 bucket. You possibly can choose completely different embedding fashions and chunking methods that work higher in your information. After you create the data base, keep in mind to sync the info. Knowledge synchronization may take a couple of minutes.

To allow your newly created data base to invoke the rerank mannequin, it’s good to modify its permissions. First, open the Amazon Bedrock console and find the service function that matches the one proven within the following screenshot.

Select the function and add the next supplied IAM permission coverage as an inline coverage. This extra authorization grants your data base the required permissions to efficiently invoke the rerank mannequin on Amazon Bedrock.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Action": "bedrock:InvokeModel",
            "Resource": "arn:aws:bedrock:us-west-2::foundation-model/amazon.rerank-v1:0"
        },
        {
            "Effect": "Allow",
            "Action": "bedrock:Rerank",
            "Resource": "*"
        }
    ]
}

Use the next code to combine the data base into the LlamaIndex framework. Particular configurations may be supplied within the retrieval_config parameter, the place numberOfResults is the utmost variety of retrieved chunks from the vector retailer, and overrideSearchType has two legitimate values: HYBRID and SEMANTIC. Within the rerankConfiguration, you may optionally present a rerank modelConfiguration and numberOfRerankedResults to type the retrieved chunks by relevancy scores and choose solely the outlined variety of outcomes. For the complete record of obtainable configurations for retrieval_config, seek advice from the Retrieve API documentation.

# Configure a data base retriever utilizing AmazonKnowledgeBasesRetriever
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.retrievers.bedrock import AmazonKnowledgeBasesRetriever

# most variety of related textual content chunks that shall be retrieved
# If you happen to want fast, centered solutions: decrease numbers (1-3)
# If you happen to want detailed, complete solutions: greater numbers (5-10)
top_k = 10

# search mode choices: HYBRID, SEMANTIC
# HYBRID search combines the strengths of semantic search and key phrase search 
# Balances semantic understanding with actual matching
# https://docs.llamaindex.ai/en/steady/examples/retrievers/bedrock_retriever/
search_mode = "HYBRID"

kb_retriever = AmazonKnowledgeBasesRetriever(
    knowledge_base_id=knowledge_base_id,
    retrieval_config={
        "vectorSearchConfiguration": {
            "numberOfResults": top_k,
            "overrideSearchType": search_mode,
            'rerankingConfiguration': {
                'bedrockRerankingConfiguration': {
                    'modelConfiguration': {
                        'modelArn': 'arn:aws:bedrock:us-west-2::foundation-model/amazon.rerank-v1:0'
                    },
                    'numberOfRerankedResults': 3
                },
                'kind': 'BEDROCK_RERANKING_MODEL'
            }
        },
        
    }
)
kb_engine = RetrieverQueryEngine(retriever=kb_retriever)

Like the primary possibility, you may create the data base as a QueryEngineTool in LlamaIndex and mix it with different API instruments. Then, you may create a FunctionCallingAgentWorker utilizing these mixed instruments and initialize an AgentRunner to work together with them. Through the use of this strategy, you may chat with and reap the benefits of the capabilities of the built-in instruments.

# Create a question instrument for Bedrock Information Base
kb_tool = QueryEngineTool(
        query_engine=kb_engine,
        metadata=ToolMetadata(
            identify="kb_tool",
            description="""
            These determination guides assist customers choose applicable AWS machine studying and generative AI companies based mostly on particular wants. 
            They cowl pre-built options, customizable platforms, and infrastructure choices for ML workflows, 
            whereas outlining how generative AI can automate processes, personalize content material, increase information, scale back prices, 
            and allow sooner experimentation in varied enterprise contexts.""",
        ),
    )

# Replace the agent to incorporate all API instruments and the Information Base instrument.
all_tools = api_tools +[kb_tool]

agent_worker = FunctionCallingAgentWorker.from_tools(
    all_tools, 
    llm=llm, 
    verbose=True, # Set verbose=True to show the complete hint of steps. 
    system_prompt = system_prompt,
    # allow_parallel_tool_calls = True  # Uncomment this line to permit a number of instrument invocations
)
agent = AgentRunner(agent_worker)
response = agent.chat(text_input)

Now you have got constructed the agentic RAG resolution utilizing LlamaIndex and Amazon Bedrock Information Bases.

Clear up

If you end experimenting with this resolution, use the next steps to wash up the AWS sources to keep away from pointless prices:

1. Within the Amazon S3 console, delete the S3 bucket and information created for this resolution.
2. Within the OpenSearch Service console, delete the gathering that was created for storing the embedding vectors.
3. Within the Amazon Bedrock Information Bases console, delete the data base you created.
4. Within the SageMaker console, navigate to your area and consumer profile, and launch SageMaker Studio to cease or delete the JupyterLab occasion.

Conclusion

This submit demonstrated learn how to construct a robust agentic RAG utility utilizing LlamaIndex and Amazon Bedrock that goes past conventional query answering programs. By integrating Mistral Giant 2 because the orchestrating mannequin with exterior APIs (GitHub, arXiv, TechCrunch, and DuckDuckGo) and inside data bases, you’ve created a flexible know-how discovery and analysis instrument.

We confirmed you two complementary approaches to implement the RAG framework: a programmatic implementation utilizing LlamaIndex with Amazon OpenSearch Serverless, offering most flexibility for superior use instances, and a managed resolution utilizing Amazon Bedrock Information Bases that simplifies doc processing and storage with minimal configuration. You possibly can check out the answer utilizing the next code pattern.

For extra related data, see Amazon Bedrock, Amazon Bedrock Information Bases, Amazon OpenSearch Serverless, and Use a reranker mannequin in Amazon Bedrock. Check with Mistral AI in Amazon Bedrock to see the most recent Mistral fashions which are accessible on each Amazon Bedrock and AWS Market.

In regards to the Authors

Ying Hou, PhD, is a Sr. Specialist Resolution Architect for Gen AI at AWS, the place she collaborates with mannequin suppliers to onboard the most recent and most clever AI fashions onto AWS platforms. With deep experience in Gen AI, ASR, pc imaginative and prescient, NLP, and time-series forecasting fashions, she works intently with clients to design and construct cutting-edge ML and GenAI functions. Outdoors of architecting revolutionary AI options, she enjoys spending high quality time together with her household, getting misplaced in novels, and exploring the UK’s nationwide parks.

Preston Tuggle is a Sr. Specialist Options Architect with the Third-Get together Mannequin Supplier group at AWS. He focuses on working with mannequin suppliers throughout Amazon Bedrock and Amazon SageMaker, serving to them speed up their go-to-market methods by technical scaling initiatives and buyer engagement.