Evaluator-Optimizer Workflow

To use this agent, add the Maven dependency to your project, replacing ${agentic-patterns.version} with the latest version.

Latest version:

<dependency>
    <groupId>com.javaaidev.agenticpatterns</groupId>
    <artifactId>evaluator-optimizer</artifactId>
    <version>${agentic-patterns.version}</version>
</dependency>

See Javadoc here:

Usage

For this Evaluator-Optimizer workflow, there can be five steps. Only the step to generate initial result is required. All other steps are optional. If only the step to generate initial result is provided, then this workflow is downgraded to an normal task execution agent.

• Initialization step
• Initial result generation step
• Evaluation step
• Optimization step
• Finalization step

For these steps,

• Initial result generation step and Optimization step are typically implemented using an LLM.
• Evaluation step can be implemented using either an LLM or code logic.
• Initialization step and Finalization step are typically implemented using code logic, but they can also be implemented using an LLM.

EvaluatorOptimizerWorkflow

To create a Evaluator-Optimizer workflow, you can create an instance of EvaluatorOptimizerWorkflow.

EvaluatorOptimizerWorkflow is a parameterized type with following type parameters:

• Request: Type of workflow input.
• GenInput: Type of input of result generation.
• GenOutput: Type of output of result generation.
• ER extends EvaluationResult: Type of evaluation result.
• Response: Type of workflow output.

Each type of workflow step has an interface.

• InitializationStep<Request, GenInput>
• InitialResultGenerationStep<GenInput, GenOutput>
• EvaluationStep<GenInput, GenOutput, ER extends EvaluationResult>
• FinalizationStep<Request, GenInput, GenOutput, Response>

When constructing a EvaluatorOptimizerWorkflow, parameters in the table below can be provided.

Parameter	Required?	Type	Description
initializationStep	yes	InitializationStep	Initialization step
initialResultGenerationStep	yes	InitialResultGenerationStep	Initial result generation step
evaluationStep	no	EvaluationStep	Evaluation step
optimizationStep	no	OptimizationStep	Optimization step
finalizationStep	yes	FinalizationStep	Finalization step
evaluationPredicate	yes	Predicate	Check if the evaluation passed
maxNumberOfEvaluations	no	int	Maximum number of evaluation. The default value is 3.
name	no	String	Name of the workflow
observationRegistry	no	ObservationRegistry	Micrometer ObservationRegistry for tracing

EvaluationResult

Evaluation result is defined in the interface EvaluationResult.

EvaluationResult

package com.javaaidev.agenticpatterns.evaluatoroptimizer;

import org.jspecify.annotations.Nullable;

/**
 * Evaluation result
 */
public interface EvaluationResult {

  /**
   * Feedback of evaluation
   *
   * @return feedback
   */
  @Nullable
  String getFeedback();
}

There are two types of built-in EvaluationResults. BooleanEvaluationResult uses a boolean result.

BooleanEvaluationResult

package com.javaaidev.agenticpatterns.evaluatoroptimizer;

import com.fasterxml.jackson.annotation.JsonPropertyDescription;
import org.jspecify.annotations.Nullable;

/**
 * Boolean evaluation result
 *
 * @param passed   Passed or not passed
 * @param feedback Feedback if not passed
 */
public record BooleanEvaluationResult(
    @JsonPropertyDescription("If evaluation passed") boolean passed,
    @JsonPropertyDescription("Feedback of evaluation") @Nullable String feedback) implements
    EvaluationResult {

  @Override
  @Nullable
  public String getFeedback() {
    return feedback();
  }
}

NumericEvaluationResult uses a numeric score.

NumericEvaluationResult

package com.javaaidev.agenticpatterns.evaluatoroptimizer;

import com.fasterxml.jackson.annotation.JsonPropertyDescription;
import org.jspecify.annotations.Nullable;

/**
 * Numeric evaluation result
 *
 * @param score    Evaluation score
 * @param feedback Feedback if not passed
 */
public record NumericEvaluationResult(
    @JsonPropertyDescription("Evaluation score, value from 0 to 100") int score,
    @JsonPropertyDescription("Feedback of evaluation") @Nullable String feedback) implements
    EvaluationResult {

  @Override
  public @Nullable String getFeedback() {
    return feedback();
  }
}

Builder

It's easy to create EvaluatorOptimizerWorkflow using its builder. TaskExecutionAgent can be used as steps in the workflow.

Example

The example is an agent to generate code with evaluation feedbacks. This workflow is created using a builder.

CodeGenerationAgent

package com.javaaidev.agenticpatterns.examples.evaluatoroptimizer;

import com.javaaidev.agenticpatterns.core.AgentUtils;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.BooleanEvaluationResult;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.EvaluationStep.EvaluationInput;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.EvaluatorOptimizerWorkflow;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.NoopFinalizationStep;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.NoopInitializationStep;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.OptimizationStep.OptimizationInput;
import com.javaaidev.agenticpatterns.taskexecution.TaskExecutionAgent;
import io.micrometer.observation.ObservationRegistry;
import java.util.Map;
import java.util.Objects;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.SimpleLoggerAdvisor;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class CodeGenerationConfiguration {

  @Bean
  @Qualifier("codeGenerationWorkflow")
  public EvaluatorOptimizerWorkflow<CodeGenerationRequest, CodeGenerationRequest, CodeGenerationResponse, BooleanEvaluationResult, CodeGenerationResponse> codeGenerationWorkflow(
      ChatClient.Builder chatClientBuilder,
      SimpleLoggerAdvisor simpleLoggerAdvisor,
      ObservationRegistry observationRegistry
  ) {
    var chatClient = chatClientBuilder.defaultAdvisors(simpleLoggerAdvisor).build();
    return EvaluatorOptimizerWorkflow.<CodeGenerationRequest, CodeGenerationRequest, CodeGenerationResponse, BooleanEvaluationResult, CodeGenerationResponse>builder()
        .initializationStep(new NoopInitializationStep<>())
        .initialResultGenerationStep(
            TaskExecutionAgent.<CodeGenerationRequest, CodeGenerationResponse>defaultBuilder()
                .name("CodeGeneration_InitialResult")
                .chatClient(chatClient)
                .observationRegistry(observationRegistry)
                .responseType(CodeGenerationResponse.class)
                .promptTemplate(AgentUtils.loadPromptTemplateFromClasspath(
                    "prompt_template/code-generator/initial-result.st"))
                .build())
        .evaluationStep(
            TaskExecutionAgent.<EvaluationInput<CodeGenerationRequest, CodeGenerationResponse>, BooleanEvaluationResult>defaultBuilder()
                .name("CodeGeneration_Evaluation")
                .chatClient(chatClient)
                .observationRegistry(observationRegistry)
                .responseType(BooleanEvaluationResult.class)
                .promptTemplate(AgentUtils.loadPromptTemplateFromClasspath(
                    "prompt_template/code-generator/evaluation.st"))
                .promptTemplateContextProvider(evaluationInput -> Map.of(
                    "code", evaluationInput.genOutput().code()
                ))
                .build())
        .optimizationStep(
            TaskExecutionAgent.<OptimizationInput<CodeGenerationRequest, CodeGenerationResponse, BooleanEvaluationResult>, CodeGenerationResponse>defaultBuilder()
                .name("CodeGeneration_Optimization")
                .chatClient(chatClient)
                .observationRegistry(observationRegistry)
                .responseType(CodeGenerationResponse.class)
                .promptTemplate(AgentUtils.loadPromptTemplateFromClasspath(
                    "prompt_template/code-generator/optimization.st"))
                .promptTemplateContextProvider(optimizationInput -> Map.of(
                    "code", optimizationInput.genOutput().code(),
                    "feedback",
                    Objects.requireNonNullElse(optimizationInput.evaluationResult().feedback(), "")
                ))
                .build())
        .finalizationStep(new NoopFinalizationStep<>())
        .evaluationPredicate(BooleanEvaluationResult::passed)
        .name("CodeGeneration")
        .observationRegistry(observationRegistry)
        .build();
  }

  public record CodeGenerationRequest(String input) {

  }

  public record CodeGenerationResponse(String code) {

  }
}

We can also use a REST API to access the agent.

REST controller of agent

package com.javaaidev.agenticpatterns.examples.evaluatoroptimizer;

import com.javaaidev.agenticpatterns.evaluatoroptimizer.BooleanEvaluationResult;
import com.javaaidev.agenticpatterns.evaluatoroptimizer.EvaluatorOptimizerWorkflow;
import com.javaaidev.agenticpatterns.examples.evaluatoroptimizer.CodeGenerationConfiguration.CodeGenerationRequest;
import com.javaaidev.agenticpatterns.examples.evaluatoroptimizer.CodeGenerationConfiguration.CodeGenerationResponse;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequestMapping("/code_generation")
public class CodeGenerationController {

  private final EvaluatorOptimizerWorkflow<CodeGenerationRequest, CodeGenerationRequest, CodeGenerationResponse, BooleanEvaluationResult, CodeGenerationResponse> evaluatorOptimizerWorkflow;

  public CodeGenerationController(
      @Qualifier("codeGenerationWorkflow") EvaluatorOptimizerWorkflow<CodeGenerationRequest, CodeGenerationRequest, CodeGenerationResponse, BooleanEvaluationResult, CodeGenerationResponse> codeGenerationAgent) {
    this.evaluatorOptimizerWorkflow = codeGenerationAgent;
  }

  @PostMapping
  public CodeGenerationResponse generateCode(@RequestBody CodeGenerationRequest request) {
    return evaluatorOptimizerWorkflow.execute(request);
  }
}