Utilizing Amazon OpenSearch ML connector APIs

When ingesting information into Amazon OpenSearch, clients usually want to reinforce information earlier than placing it into their indexes. As an example, you could be ingesting log recordsdata with an IP handle and need to get a geographic location for the IP handle, otherwise you could be ingesting buyer feedback and need to establish the language they’re in. Historically, this requires an exterior course of that complicates information ingest pipelines and might trigger a pipeline to fail. OpenSearch gives a variety of third-party machine studying (ML) connectors to help this augmentation.

This submit highlights two of those third-party ML connectors. The primary connector we show is the Amazon Comprehend connector. On this submit, we present you how you can use this connector to invoke the LangDetect API to detect the languages of ingested paperwork.

The second connector we show is the Amazon Bedrock connector to invoke the Amazon Titan Textual content Embeddings v2 mannequin to be able to create embeddings from ingested paperwork and carry out semantic search.

Answer overview

We use Amazon OpenSearch with Amazon Comprehend to show the language detection characteristic. That can assist you replicate this setup, we’ve supplied the required supply code, an Amazon SageMaker pocket book, and an AWS CloudFormation template. You will discover these assets within the sample-opensearch-ml-rest-api GitHub repo.

The reference structure proven within the previous determine reveals the parts used on this answer. A SageMaker pocket book is used as a handy approach to execute the code that’s supplied within the Github repository supplied above.

Stipulations

To run the total demo utilizing the sample-opensearch-ml-rest-api, ensure you have an AWS account with entry to:

Half 1: The Amazon Comprehend ML connector

Arrange OpenSearch to entry Amazon Comprehend

Earlier than you need to use Amazon Comprehend, you have to ensure that OpenSearch can name Amazon Comprehend. You do that by supplying OpenSearch with an IAM position that has entry to invoke the DetectDominantLanguage API. This requires the OpenSearch Cluster to have superb grained entry management enabled. The CloudFormation template creates a task for this referred to as --SageMaker-OpenSearch-demo-role. Use the next steps to connect this position to the OpenSearch cluster.

1. Open the OpenSearch Dashboard console—you’ll find the URL within the output of the CloudFormation template—and register utilizing the username and password you supplied.

2. Select Safety within the left-hand menu (in the event you don’t see the menu, select the three horizontal traces icon on the prime left of the dashboard).
3. From the safety menu, choose Roles to handle the OpenSearch roles.
4. Within the search field. enter ml_full_access position.
5. Choose the Mapped customers hyperlink to map the IAM position to this OpenSearch position.
6. On the Mapped customers display, select Handle mapping to edit the present mappings.
7. Add the IAM position talked about beforehand to map it to the ml_full_access position, it will enable OpenSearch to entry the wanted AWS assets from the ml-commons plugin. Enter your IAM position Amazon Useful resource Identify (ARN) (arn:aws:iam:::position/--SageMaker-OpenSearch-demo-role) within the backend roles subject and select Map.

Arrange the OpenSearch ML connector to Amazon Comprehend

On this step, you arrange the ML connector to attach Amazon Comprehend to OpenSearch.

1. Get an authorization token to make use of when making the decision to OpenSearch from the SageMaker pocket book. The token makes use of an IAM position connected to the pocket book by the CloudFormation template that has permissions to name OpenSearch. That very same position is mapped to the OpenSearch admin position in the identical approach you simply mapped the position to entry Amazon Comprehend. Use the next code to set this up:

awsauth = AWS4Auth(credentials.access_key,
credentials.secret_key,
area,
'es',
session_token=credentials.token)

2. Create the connector. It wants a number of items of data:
   1. It wants a protocol. For this instance, use aws_sigv4, which permits OpenSearch to make use of an IAM position to name Amazon Comprehend.
   2. Present the ARN for this position, which is similar position you used to arrange permissions for the ml_full_access position.
   3. Present comprehend because the service_name, and DetectDominateLanguage because the api_name.
   4. Present the URL to Amazon Comprehend and arrange how you can name the API and what information to go to it.

The ultimate name seems like:

comprehend = boto3.consumer('comprehend', region_name="us-east-1")
path="/_plugins/_ml/connectors/_create"
url = host + path

payload = {
  "identify": "Comprehend lang identification",
  "description": "comprehend mannequin",
  "model": 1,
  "protocol": "aws_sigv4",
  "credential": {
    "roleArn": sageMakerOpenSearchRoleArn
  },
  "parameters": {
    "area": "us-east-1",
    "service_name": "comprehend",
    "api_version": "20171127",
    "api_name": "DetectDominantLanguage",
    "api": "Comprehend_${parameters.api_version}.${parameters.api_name}",
    "response_filter": "$"
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://${parameters.service_name}.${parameters.region}.amazonaws.com",
      "headers": {
        "content-type": "application/x-amz-json-1.1",
        "X-Amz-Target": "${parameters.api}"
      },
      "request_body": "{"Text": "${parameters.Text}"}" 
    }
  ]
}

comprehend_connector_response = requests.submit(url, auth=awsauth, json=payload)
comprehend_connector = comprehend_connector_response.json()["connector_id"]

Register the Amazon Comprehend API connector

The subsequent step is to register the Amazon Comprehend API connector with OpenSearch utilizing the Register Mannequin API from OpenSearch.

• Use the comprehend_connector that you simply saved from the final step.

path="/_plugins/_ml/fashions/_register"
url = host + path

payload = {
    "identify": "comprehend lang id API",
    "function_name": "distant",
    "description": "API to detect the language of textual content",
    "connector_id": comprehend_connector
}
headers = {"Content material-Sort": "software/json"}

response = requests.submit(url, auth=awsauth, json=payload, headers=headers)
comprehend_model_id = response.json()['model_id']

As of OpenSearch 2.13, when the mannequin is first invoked, it’s mechanically deployed. Previous to 2.13 you would need to manually deploy the mannequin inside OpenSearch.

Take a look at the Amazon Comprehend API in OpenSearch

With the connector in place, you have to check the API to ensure it was arrange and configured appropriately.

1. Make the next name to OpenSearch.

path="/_plugins/_ml/fashions/"+ comprehend_model_id + '/_predict'
url = host + path

headers = {"Content material-Sort": "software/json"}
payload = {
    "parameters": {
        "Textual content": "你知道厕所在哪里吗"
    }
}

response = requests.submit(url, auth=awsauth, json=payload, headers=headers)
print(response.json())

2. You need to get the next outcome from the decision, displaying the language code as zh with a rating of 1.0:

{
   "inference_results":[
      {
         "output":[
            {
               "name":"response",
               "dataAsMap":{
                  "response":{
                     "Languages":[
                        {
                           "LanguageCode":"zh",
                           "Score":1.0
                        }
                     ]
                  }
               }
            }
         ],
         "status_code":200
      }
   ]
}

Create an ingest pipeline that makes use of the Amazon Comprehend API to annotate the language

The subsequent step is to create a pipeline in OpenSearch that calls the Amazon Comprehend API and provides the outcomes of the decision to the doc being listed. To do that, you present each an input_map and an output_map. You utilize these to inform OpenSearch what to ship to the API and how you can deal with what comes again from the decision.

path="/_ingest/pipeline/comprehend_language_identification_pipeline"
url = host + path

payload = {
  "description": "ingest establish lang with the comprehend API",
  "processors":[
    {
      "ml_inference": {
        "model_id": comprehend_model_id,
        "input_map": [
            {
               "Text": "Text"
            }
        ],
        "output_map": [
            {  
               "detected_language": "response.Languages[0].LanguageCode",
               "language_score": "response.Languages[0].Rating"
            }
        ]
      }
    }
  ]
}
headers = {"Content material-Sort": "software/json"}
response = requests.put(url, auth=awsauth, json=payload, headers=headers)

You’ll be able to see from the previous code that you’re pulling again each the highest language outcome and its rating from Amazon Comprehend and including these fields to the doc.

Half 2: The Amazon Bedrock ML connector

On this part, you utilize Amazon OpenSearch with Amazon Bedrock via the ml-commons plugin to carry out a multilingual semantic search. Just be sure you have the answer stipulations in place earlier than trying this part.

Within the SageMaker occasion that was deployed for you, you possibly can see the next recordsdata: english.json, french.json, german.json.

These paperwork have sentences of their respective languages that speak concerning the time period spring in numerous contexts. These contexts embody spring as a verb which means to maneuver all of the sudden, as a noun which means the season of spring, and eventually spring as a noun which means a mechanical half. On this part, you deploy Amazon Titan Textual content Embeddings mannequin v2 utilizing the ml connector for Amazon Bedrock. You then use this embeddings mannequin to create vectors of textual content in three languages by ingesting the completely different language JSON recordsdata. Lastly, these vectors are saved in Amazon OpenSearch to allow semantic searches for use throughout the language units.

Amazon Bedrock supplies streamlined entry to varied highly effective AI basis fashions via a single API interface. This managed service consists of fashions from Amazon and different main AI corporations. You’ll be able to check completely different fashions to search out the best match in your particular wants, whereas sustaining safety, privateness, and accountable AI practices. The service allows you to customise these fashions with your individual information via strategies equivalent to fine-tuning and Retrieval Augmented Technology (RAG). Moreover, you need to use Amazon Bedrock to create AI brokers that may work together with enterprise programs and information, making it a complete answer for growing generative AI functions.

The reference structure within the previous determine reveals the parts used on this answer.

(1) First we should create the OpenSearch ML connector by way of operating code inside the Amazon SageMaker pocket book. The connector basically creates a Relaxation API name to any mannequin, we particularly need to create a connector to name the Titan Embeddings mannequin inside Amazon Bedrock.

(2) Subsequent, we should create an index to later index our language paperwork into. When creating an index, you possibly can specify its mappings, settings, and aliases.

(3) After creating an index inside Amazon OpenSearch, we need to create an OpenSearch Ingestion pipeline that may enable us to streamline information processing and preparation for indexing, making it simpler to handle and make the most of the information. (4) Now that we’ve got created an index and arrange a pipeline, we will begin indexing our paperwork into the pipeline.

(5 – 6) We use the pipeline in OpenSearch that calls the Titan Embeddings mannequin API. We ship our language paperwork to the titan embeddings mannequin, and the mannequin returns vector embeddings of the sentences.

(7) We retailer the vector embeddings inside our index and carry out vector semantic search.

Whereas this submit highlights solely particular areas of the general answer, the SageMaker pocket book has the code and directions to run the total demo your self.

Earlier than you need to use Amazon Bedrock, you have to ensure that OpenSearch can name Amazon Bedrock. .

Load sentences from the JSON paperwork into dataframes

Begin by loading the JSON doc sentences into dataframes for extra structured group. Every row can comprise the textual content, embeddings, and extra contextual info:

import json
import pandas as pd

def load_sentences(file_name):
    sentences = []
    with open(file_name, 'r', encoding='utf-8') as file:
        for line in file:
            attempt:
                information = json.masses(line)
                if 'sentence' in information and 'sentence_english' in information:
                    sentences.append({
                        'sentence': information['sentence'],
                        'sentence_english': information['sentence_english']
                    })
            besides json.JSONDecodeError:
                # Skip traces that aren't legitimate JSON (just like the index traces)
                proceed
    
    return pd.DataFrame(sentences)

# Utilization
german_df = load_sentences('german.json')
english_df = load_sentences('english.json')
french_df = load_sentences('french.json')
# print(french_df.head())

Create the OpenSearch ML connector to Amazon Bedrock

After loading the JSON paperwork into dataframes, you’re able to arrange the OpenSearch ML connector to attach Amazon Bedrock to OpenSearch.

1. The connector wants the next info.
   1. It wants a protocol. For this answer, use aws_sigv4, which permits OpenSearch to make use of an IAM position to name Amazon Bedrock.
   2. Present the identical position used earlier to arrange permissions for the ml_full_access position.
   3. Present the service_name, mannequin, dimensions of the mannequin, and embedding kind.

The ultimate name seems like the next:

payload = {
  "identify": "Amazon Bedrock Connector: embedding",
  "description": "The connector to bedrock Titan embedding mannequin",
  "model": 1,
  "protocol": "aws_sigv4",
  "parameters": {
    "area": "us-east-1",
    "service_name": "bedrock",
    "mannequin": "amazon.titan-embed-text-v2:0",
    "dimensions": 1024,
    "normalize": True,
    "embeddingTypes": ["float"]
  },
  "credential": {
    "roleArn": sageMakerOpenSearchRoleArn
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://bedrock-runtime.${parameters.region}.amazonaws.com/model/${parameters.model}/invoke",
      "headers": {
        "content-type": "application/json",
        "x-amz-content-sha256": "required"
      },
      "request_body": "{ "inputText": "${parameters.inputText}", "dimensions": ${parameters.dimensions}, "normalize": ${parameters.normalize}, "embeddingTypes": ${parameters.embeddingTypes} }",
      "pre_process_function": "connector.pre_process.bedrock.embedding",
      "post_process_function": "connector.post_process.bedrock.embedding"
    }
  ]
}

bedrock_connector_response = requests.submit(url, auth=awsauth, json=payload, headers=headers)

bedrock_connector_3 = bedrock_connector_response.json()["connector_id"]
print('Connector id: ' + bedrock_connector_3)

Take a look at the Amazon Titan Embeddings mannequin in OpenSearch

After registering and deploying the Amazon Titan Embeddings mannequin utilizing the Amazon Bedrock connector, you possibly can check the API to confirm that it was arrange and configured appropriately. To do that, make the next name to OpenSearch:

headers = {"Content material-Sort": "software/json"}
payload = {
  "parameters": {
    "inputText": "It is good to see the flowers bloom and listen to the birds sing within the spring"
  }
}
response = requests.submit(url, auth=awsauth, json=payload, headers=headers)
print(response.json())

You need to get a formatted outcome, much like the next, from the decision that reveals the generated embedding from the Amazon Titan Embeddings mannequin:

{'inference_results': [{'output': [{'name': 'sentence_embedding', 'data_type': 'FLOAT32', 'shape': [1024], 'information': [-0.04092199727892876, 0.052057236433029175, -0.03354490175843239, 0.04398418962955475, -0.001235315459780395, -0.03284895047545433, -0.014197427779436111, 0.0098129278048…

The preceding result is significantly shortened compared to the actual embedding result you might receive. The purpose of this snippet is to show you the format.

Create the index pipeline that uses the Amazon Titan Embeddings model

Create a pipeline in OpenSearch. You use this pipeline to tell OpenSearch to send the fields you want embeddings for to the embeddings model.

pipeline_name = "titan_embedding_pipeline_v2"
url = f"{host}/_ingest/pipeline/{pipeline_name}"

pipeline_body = {
    "description": "Titan embedding pipeline",
    "processors": [
        {
            "text_embedding": {
                "model_id": bedrock_model_id,
                "field_map": {
                    "sentence": "sentence_vector"
                }
            }
        }
    ]
}

response = requests.put(url, auth=awsauth, json=pipeline_body, headers={"Content material-Sort": "software/json"})
print(response.textual content)

Create an index

With the pipeline in place, the subsequent step is to create an index that may use the pipeline. There are three fields within the index:

• sentence_vector – That is the place the vector embedding shall be saved when returned from Amazon Bedrock.
• sentence – That is the non-English language sentence.
• sentence_english – that is the English translation of the sentence. Embody this to see how nicely the mannequin is translating the unique sentence.

index_name="bedrock-knn-index-v2"
url = f'{host}/{index_name}'
mapping = {
    "mappings": {
        "properties": {
            "sentence_vector": {
                "kind": "knn_vector",
                "dimension": 1024,  
                "technique": {
                    "identify": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib"
                },
                "retailer":True
            },
            "sentence":{
                "kind": "textual content",
                "retailer": True
            },
            "sentence_english":{
                "kind": "textual content",
                "retailer": True
            }
        }
    },
    "settings": {
        "index": {
            "knn": True,
            "knn.space_type": "cosinesimil",
            "default_pipeline": pipeline_name
        }
    }
}

response = requests.put(url, auth=awsauth, json=mapping, headers={"Content material-Sort": "software/json"})
print(f"Index creation response: {response.textual content}")

Load dataframes into the index

Earlier on this part, you loaded the sentences from the JSON paperwork into dataframes. Now, you possibly can index the paperwork and generate embeddings for them utilizing the Amazon Titan Textual content Embeddings Mannequin v2. The embeddings shall be saved within the sentence_vector subject.

index_name = "bedrock-knn-index-v2"

def index_documents(df, batch_size=100):
    complete = len(df)
    for begin in vary(0, complete, batch_size):
        finish = min(begin + batch_size, complete)
        batch = df.iloc[start:end]

        bulk_data = []
        for _, row in batch.iterrows():
            # Put together the motion metadata
            motion = {
                "index": {
                    "_index": index_name
                }
            }
            # Put together the doc information
            doc = {
                "sentence": row['sentence'],
                "sentence_english": row['sentence_english']
            }
            
            # Add the motion and doc to the majority information
            bulk_data.append(json.dumps(motion))
            bulk_data.append(json.dumps(doc))

        # Be part of the majority information with newlines
        bulk_body = "n".be part of(bulk_data) + "n"

        # Ship the majority request
        bulk_url = f"{host}/_bulk"
        response = requests.submit(bulk_url, auth=awsauth, information=bulk_body, headers={"Content material-Sort": "software/x-ndjson"})

        if response.status_code == 200:
            print(f"Efficiently listed batch {begin}-{finish} of {complete}")
        else:
            print(f"Error indexing batch {begin}-{finish} of {complete}: {response.textual content}")

        # Optionally available: add a small delay to keep away from overwhelming the cluster
        time.sleep(1)

# Index your paperwork
print("Indexing German paperwork:")
index_documents(german_df)
print("nIndexing English paperwork:")
index_documents(english_df)
print("nIndexing French paperwork:")
index_documents(french_df)

Carry out semantic k-NN throughout the paperwork

The ultimate step is to carry out a k-nearest neighbor (k-NN) search throughout the paperwork.

# Outline your OpenSearch host and index identify
index_name = "bedrock-knn-index-v2"
def semantic_search(query_text, ok=5):
    search_url = f"{host}/{index_name}/_search"
    # First, index the question to generate its embedding
    index_doc = {
        "sentence": query_text,
        "sentence_english": query_text  # Assuming the question is in English
    }
    index_url = f"{host}/{index_name}/_doc"
    index_response = requests.submit(index_url, auth=awsauth, json=index_doc, headers={"Content material-Sort": "software/json"})
    
    if index_response.status_code != 201:
        print(f"Did not index question doc: {index_response.textual content}")
        return []
    
    # Retrieve the listed question doc to get its vector
    doc_id = index_response.json()['_id']
    get_url = f"{host}/{index_name}/_doc/{doc_id}"
    get_response = requests.get(get_url, auth=awsauth)
    query_vector = get_response.json()['_source']['sentence_vector']
    
    # Now carry out the KNN search
    search_query = {
        "dimension": 30,
        "question": {
            "knn": {
                "sentence_vector": {
                    "vector": query_vector,
                    "ok": 30
                }
            }
        },
        "_source": ["sentence", "sentence_english"]
    }

    search_response = requests.submit(search_url, auth=awsauth, json=search_query, headers={"Content material-Sort": "software/json"})
    
    if search_response.status_code != 200:
        print(f"Search failed with standing code {search_response.status_code}")
        print(search_response.textual content)
        return []

    # Clear up - delete the short-term question doc
    delete_url = f"{host}/{index_name}/_doc/{doc_id}"
    requests.delete(delete_url, auth=awsauth)

    return search_response.json()['hits']['hits']

# Instance utilization
question = "le soleil brille"
outcomes = semantic_search(question)

if outcomes:
    print(f"Search outcomes for: '{question}'")
    for lead to outcomes:
        print(f"Rating: {outcome['_score']}")
        print(f"Sentence: {outcome['_source']['sentence']}")
        print(f"English: {outcome['_source']['sentence_english']}")
        print()
else:
    print("No outcomes discovered or search failed.")

The instance question is in French and may be translated to the solar is shining. Protecting in thoughts that the JSON paperwork have sentences that use spring in numerous contexts, you’re in search of question outcomes and vector matches of sentences that use spring within the context of the season of spring.

Listed below are among the outcomes from this question:

Search outcomes for: ' le soleil brille'
Rating: 0.40515712
Sentence: Les premiers rayons de soleil au printemps réchauffent la terre.
English: The primary rays of spring sunshine heat the earth.

Rating: 0.40117615
Sentence: Die ersten warmen Sonnenstrahlen kitzeln auf der Haut im Frühling.
English: The primary heat solar rays tickle the pores and skin in spring.

Rating: 0.3999985
Sentence: Die ersten Sonnenstrahlen im Frühling wecken die Lebensgeister.
English: The primary rays of sunshine in spring awaken the spirits.

This reveals that the mannequin can present outcomes throughout all three languages. It is very important observe that the boldness scores for these outcomes could be low since you’ve solely ingested a pair paperwork with a handful of sentences in every for this demo. To extend confidence scores and accuracy, ingest a strong dataset with a number of languages and loads of sentences for reference.

Clear Up

To keep away from incurring future costs, go to the AWS Administration Console for CloudFormation console and delete the stack you deployed. This can terminate the assets used on this answer.

Advantages of utilizing the ML connector for machine studying mannequin integration with OpenSearch

There are lots of methods you possibly can carry out k-nn semantic vector searches; a preferred strategies is to deploy exterior Hugging Face sentence transformer fashions to a SageMaker endpoint. The next are the advantages of utilizing the ML connector method we confirmed on this submit, and why must you use it as an alternative of deploying fashions to a SageMaker endpoint:

• Simplified structure
  • Single system to handle
  • Native OpenSearch integration
  • Less complicated deployment
  • Unified monitoring

• Operational advantages
  • Much less infrastructure to take care of
  • Constructed-in scaling with OpenSearch
  • Simplified safety mannequin
  • Easy updates and upkeep

• Price effectivity
  • Single system prices
  • Pay-per-use Amazon Bedrock pricing
  • No endpoint administration prices
  • Simplified billing

Conclusion

Now that you simply’ve seen how you need to use the OpenSearch ML connector to reinforce your information with exterior REST calls, we suggest that you simply go to the GitHub repo in the event you haven’t already and stroll via the total demo yourselves. The total demo reveals how you need to use Amazon Comprehend for language detection and how you can use Amazon Bedrock for multilingual semantic vector search, utilizing the ml-connector plugin for each use instances. It additionally has pattern textual content and JSON paperwork to ingest so you possibly can see how the pipeline works.

In regards to the Authors

John Trollinger is a Principal Options Architect supporting the World Broad Public Sector with a deal with OpenSearch and Knowledge Analytics. John has been working with public sector clients over the previous 25 years serving to them ship mission capabilities. Outdoors of labor, John likes to gather AWS certifications and compete in triathlons.

Shwetha Radhakrishnan is a Options Architect for Amazon Internet Providers (AWS) with a spotlight in Knowledge Analytics & Machine Studying. She has been constructing options that drive cloud adoption and assist empower organizations to make data-driven choices inside the public sector. Outdoors of labor, she loves dancing, spending time with family and friends, and touring.