215111 Stack

Introduction to Amazon Bedrock Advanced Prompt Optimization

Amazon Bedrock has introduced a powerful new feature designed to refine how developers and organizations interact with large language models (LLMs). The Advanced Prompt Optimization tool empowers users to systematically improve their prompts for any model available on Bedrock, while simultaneously comparing original prompts against optimized versions across up to five different models. This capability is particularly valuable for teams looking to migrate to a new model, boost performance from their current setup, or simply ensure consistent output quality across a range of use cases.

Key Capabilities of the Tool

The prompt optimizer operates on a clear, data-driven workflow. It accepts your existing prompt template, example user inputs for variable values, ground truth answers, and an evaluation metric that guides the optimization process. One standout feature is its support for multimodal inputs—including PNG, JPG, and PDF files—making it especially useful for tasks like document analysis, image recognition, and other vision-based AI applications.

Custom Evaluation Metrics

Users have flexibility in defining how success is measured. You can provide:

• An AWS Lambda function to compute custom scores.
• An LLM-as-a-judge rubric that leverages another language model to evaluate responses.
• A simple natural language description that tells the optimizer what constitutes a good answer.

This metric-driven feedback loop iterates over the prompt, adjusting it based on the evaluation results, until it converges on an optimized version. The output includes the original and final prompt templates, along with evaluation scores, cost estimates, and latency metrics—providing a comprehensive before-and-after comparison.

How to Use Advanced Prompt Optimization

Getting started is straightforward from the Amazon Bedrock console. Follow these steps:

1. Navigate to the Advanced Prompt Optimization page and click Create prompt optimization.
2. Select up to five inference models for which you want to optimize your prompts. If you are migrating to a new model, choose your current model as a baseline alongside up to four candidate models. If you only want to improve an existing model, just select that single model to see the before-and-after results.
3. Prepare your prompt templates in JSONL format (one JSON object per line). Each object must include example user data, ground truth answers, and an evaluation metric or rewriting guidance. A sample schema is provided in the documentation.

Sample JSONL Structure

Below is a simplified representation of the required fields (refer to the official documentation for the full schema):

{
    "version": "bedrock-2026-05-14",
    "templateId": "string",
    "promptTemplate": "string",
    "steeringCriteria": ["string"],
    "customEvaluationMetricLabel": "string",
    "customLLMJConfig": {
        "customLLMJPrompt": "string",
        "customLLMJModelId": "string"
    },
    "evaluationMetricLambdaArn": "string",
    "evaluationSamples": [
        {
            "inputVariables": [
                {"variableName1": "string", "variableName2": "string"}
            ],
            "referenceResp": ...
        }
    ]
}

Benefits for Model Migration and Performance Improvement

The tool addresses two primary scenarios:

• Model migration: When switching from one LLM to another, you can test your optimized prompts against both the baseline and up to four new models. This helps ensure that your known use cases see no regressions, while also improving underperforming tasks.
• Performance tuning: Even without changing models, you can refine your prompts to achieve better accuracy, relevance, or cost efficiency. The optimizer automatically explores prompt variations guided by your chosen metric.

Because the optimization process is automated but transparent, you retain full control over the final prompt. The detailed evaluation report (including cost and latency) allows you to make informed decisions about trade-offs between accuracy and operational overhead.

Conclusion

Amazon Bedrock’s Advanced Prompt Optimization represents a significant step forward in prompt engineering for production LLM applications. By combining multimodal support, custom evaluation methods, and multi-model comparison, it enables developers to systematically improve AI outputs while reducing the manual trial-and-error typically involved. Whether you are migrating to a new model or fine-tuning an existing one, this tool provides the metrics and insights needed to optimize your prompts with confidence.

For more details, visit the Key Capabilities or How to Use sections above.