Add verbose_filter feature and expose verbose parameter to PER #4358

pyek-bot · 2025-10-28T09:06:55Z

Description

Adds verbose_filter parameter to filter out the verbose response.
Exposes executor_verbose and executor_verbose_filter in PER
Use LLM in filter to get only LLM responses, otherwise use the tool name.
verbose_filter accepts a csv list of tools and LLM as the filter

Example:

POST /_plugins/_ml/agents/<agent_id>/_execute
{
    "parameters": {
        "question": "Search and return a document from any single index",
        "verbose": true,
        "verbose_filter": "ListIndexTool,IndexMappingTool"
    }
}

Related Issues

Resolves #4357

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2025-10-28T10:14:20Z

Codecov Report

❌ Patch coverage is 92.00000% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.16%. Comparing base (5964268) to head (ce553e4).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...thms/agent/MLPlanExecuteAndReflectAgentRunner.java	88.88%	0 Missing and 3 partials ⚠️
.../ml/engine/algorithms/agent/MLChatAgentRunner.java	95.23%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4358      +/-   ##
============================================
+ Coverage     80.09%   80.16%   +0.07%     
- Complexity    10199    10225      +26     
============================================
  Files           855      855              
  Lines         44374    44407      +33     
  Branches       5135     5145      +10     
============================================
+ Hits          35540    35598      +58     
+ Misses         6670     6647      -23     
+ Partials       2164     2162       -2

Flag	Coverage Δ
ml-commons	`80.16% <92.00%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…agent Signed-off-by: Pavan Yekbote <[email protected]>

Signed-off-by: Pavan Yekbote <[email protected]>

ylwu-amzn · 2025-10-29T01:49:41Z

Can we use include_output_in_agent_response ?

Zhangxunmt · 2025-10-29T03:50:14Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

+    public static final String EXECUTOR_VERBOSE = "executor_verbose";
+    public static final String EXECUTOR_VERBOSE_FILTER = "executor_verbose_filter";


In the example, this field name is "verbose" and "verbose_filter", which seems matching to the name in chatAgent. I think "verbose" and "verbose filter" is simpler.

Zhangxunmt · 2025-10-29T03:54:36Z

ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/agent/MLChatAgentRunner.java

        String parentInteractionId = tmpParameters.get(MLAgentExecutor.PARENT_INTERACTION_ID);
        boolean verbose = Boolean.parseBoolean(tmpParameters.getOrDefault(VERBOSE, "false"));
        boolean traceDisabled = tmpParameters.containsKey(DISABLE_TRACE) && Boolean.parseBoolean(tmpParameters.get(DISABLE_TRACE));
+        List<String> traceFilter = parseTraceFilter(tmpParameters.get(VERBOSE_FILTER));


Has any validation applied to this verbose filter value? like verboseFilter.matches("^[a-zA-Z]+(,[a-zA-Z]+)*$"). I think it's worth a check before using the filters.

Zhangxunmt · 2025-10-29T03:55:12Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

                    );

+                // Pass through verbose and verbose_filter if provided
+                if (allParams.containsKey(EXECUTOR_VERBOSE)) {


Consider validation too.

Zhangxunmt · 2025-10-29T03:59:50Z

.../main/java/org/opensearch/ml/engine/algorithms/agent/MLPlanExecuteAndReflectAgentRunner.java

+                    if (!allResponses.isEmpty()) {
+                        StringBuilder stepResult = new StringBuilder();
+                        stepResult.append(allResponses.getLast());
+                        if (allResponses.size() > 1) {
+                            stepResult.append("\n\n<step-traces>");
+                        }
+
+                        for (int i = 0; i < allResponses.size() - 1; i++) {
+                            stepResult.append("\n\n").append(allResponses.get(i));
+                            if (i == allResponses.size() - 2) {
+                                stepResult.append("\n</step-traces>");
+                            }
+                        }
+
+                        results.put(STEP_RESULT_FIELD, stepResult.toString());
+                    }
+


if (!allResponses.isEmpty()) { results.put(STEP_RESULT_FIELD, formatStepResultWithTraces(allResponses)); } private String formatStepResultWithTraces(List<String> responses) { StringBuilder result = new StringBuilder(responses.getLast()); if (responses.size() > 1) { List<String> traces = responses.subList(0, responses.size() - 1); result.append("\n\n<step-traces>\n\n") .append(String.join("\n\n", traces)) .append("\n</step-traces>"); } return result.toString(); }

This is a better version to avoid for loop.

pyek-bot · 2025-10-29T05:51:33Z

Can we use include_output_in_agent_response ?

Thanks, I can use this! I will raise a PR with PER processing this information if returned.

pyek-bot · 2025-10-29T06:19:44Z

@ylwu-amzn @Zhangxunmt #4369

Raised a new PR to process tool output if it is returned when using the include_output_in_agent_response

I still think this is a good feature to have if folks want to filter on the verbose output. I will remove the changes for PER and keep the chat agent changes for the feature itself.

mingshl · 2025-10-29T16:51:33Z

let's moved the discussion to #4369

pyek-bot requested review from HenryL27, Zhangxunmt, austintlee, b4sjoo, dhrubo-os, jngz-es, mingshl, model-collapse, rbhavna, sam-herman, xinyual, ylwu-amzn and zane-neo as code owners October 28, 2025 09:06

pyek-bot temporarily deployed to ml-commons-cicd-env October 28, 2025 09:10 — with GitHub Actions Inactive

pyek-bot had a problem deploying to ml-commons-cicd-env October 28, 2025 10:16 — with GitHub Actions Failure

pyek-bot had a problem deploying to ml-commons-cicd-env October 28, 2025 10:16 — with GitHub Actions Error

pyek-bot added 2 commits October 28, 2025 18:47

feat: add verbose_filter feature and expose verbose parameter to per …

208f4ee

…agent Signed-off-by: Pavan Yekbote <[email protected]>

remove test changes

ce553e4

Signed-off-by: Pavan Yekbote <[email protected]>

ylwu-amzn force-pushed the expose_verbose_parameter branch from eac5443 to ce553e4 Compare October 29, 2025 01:47

ylwu-amzn temporarily deployed to ml-commons-cicd-env October 29, 2025 01:48 — with GitHub Actions Inactive

ylwu-amzn temporarily deployed to ml-commons-cicd-env October 29, 2025 02:55 — with GitHub Actions Inactive

Zhangxunmt reviewed Oct 29, 2025

View reviewed changes

Merge branch 'main' into expose_verbose_parameter

8323321

Zhangxunmt had a problem deploying to ml-commons-cicd-env October 29, 2025 04:04 — with GitHub Actions Failure

Zhangxunmt had a problem deploying to ml-commons-cicd-env October 29, 2025 04:04 — with GitHub Actions Error

Zhangxunmt had a problem deploying to ml-commons-cicd-env October 29, 2025 04:04 — with GitHub Actions Failure

Zhangxunmt had a problem deploying to ml-commons-cicd-env October 29, 2025 04:04 — with GitHub Actions Error

mingshl closed this Oct 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add verbose_filter feature and expose verbose parameter to PER #4358

Add verbose_filter feature and expose verbose parameter to PER #4358

Uh oh!

pyek-bot commented Oct 28, 2025 •

edited

Loading

Uh oh!

codecov bot commented Oct 28, 2025 •

edited

Loading

Uh oh!

ylwu-amzn commented Oct 29, 2025

Uh oh!

Zhangxunmt Oct 29, 2025

Uh oh!

Zhangxunmt Oct 29, 2025

Uh oh!

Zhangxunmt Oct 29, 2025

Uh oh!

Zhangxunmt Oct 29, 2025

Uh oh!

pyek-bot commented Oct 29, 2025

Uh oh!

pyek-bot commented Oct 29, 2025

Uh oh!

mingshl commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		public static final String EXECUTOR_VERBOSE = "executor_verbose";
		public static final String EXECUTOR_VERBOSE_FILTER = "executor_verbose_filter";

Add verbose_filter feature and expose verbose parameter to PER #4358

Add verbose_filter feature and expose verbose parameter to PER #4358

Uh oh!

Conversation

pyek-bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

codecov bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ylwu-amzn commented Oct 29, 2025

Uh oh!

Zhangxunmt Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Zhangxunmt Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Zhangxunmt Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Zhangxunmt Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

pyek-bot commented Oct 29, 2025

Uh oh!

pyek-bot commented Oct 29, 2025

Uh oh!

mingshl commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pyek-bot commented Oct 28, 2025 •

edited

Loading

codecov bot commented Oct 28, 2025 •

edited

Loading