Refactor Storm and Add Custom Outline Generation

[movinam]

Summary:

The following PR introduces the following changes:

1. Refactors the storm agents configurations
2. Adds initial test coverage.
3. Allows users to specify custom outline sketches from which the initial draft outline for the article is generated.

Details

1. Since Aspect Search has been added as a separate agent, all code related to that has been removed from the storm agent.
2. Configuration files and default environment variables have been removed. Storm now relies on StormAgentConfig entirely.
3. Initial test coverage has been added - further tests will be added later.
4. Please note that for now, Storm uses an In memory VDB for writing the article.
5. When an outline sketch is part of the input schema, the refine_outline node will be skipped to conform to the sketch.
6. The state has been updated to return sections and the outline.
7. References now does not return all the search results, only those used to generate the article. To fetch all the search results, access search_results from the output schema.

Outline Sketch Specification

Users can now prompt Storm with the topic and a sketch of what their outline to look like. The outline sketches can be in markdown or just plain strings.

Example 1:

sketch = """# Geographic and Ecoregion Extent
Specify the precise geographic extent and range boundaries by detailing the minimum/maximum latitude, longitude, and elevation, and identifying all named regions, sub-regions, and ecoregions of occurrence with exact geographic limits, coordinates, and place names.

# Landscape and Vegetation
Characterize the dominant land cover and vegetation types, such as forest canopy type, grassland openness, and wetland presence, focusing on habitat structure and associations that are identifiable from Sentinel-2 imagery using bands B2–B4, B8, B11, and B12.

# Elevation and Topography
Describe the elevation limits and preferences, including specific elevation zones, and detail the influence of topographic contexts such as slopes, valleys, and plains. Also, identify any specific slope/aspect preferences and how elevational gradients and terrain features affect the species' occurrence."""

topic = 'Species distribution for Guanacos'

Generated outline:

Outline(page_title='Species Distribution for Guanacos', sections=[OutlineSection(section_title='Geographic and Ecoregion Extent', description='Detail the geographic extent and range boundaries of Guanacos, including latitude, longitude, and elevation limits. Identify all named regions, sub-regions, and ecoregions of occurrence with exact geographic limits, coordinates, and notable place names.', subsections=None), OutlineSection(section_title='Landscape and Vegetation', description='Characterize the dominant land cover and vegetation types in the habitats of Guanacos. Focus on habitat structure and associations that can be identified from Sentinel-2 imagery using specific bands, including forest canopy types, grassland openness, and wetland presence.', subsections=None), OutlineSection(section_title='Elevation and Topography', description='Describe the elevation limits and preferences of Guanacos, including specific elevation zones. Detail the influence of topographic contexts, such as slopes, valleys, and plains, and identify slope/aspect preferences as well as how elevational gradients and terrain features affect their occurrence.', subsections=None)])

Example 2:

user_def_outline = '''I want the outline to have three sections. The first an introduction to the topic. The second about datasets and experiments used to research the topic. And finally, the third section should discuss possible future directions.'''
topic = 'attention based attribution mechanisms for LLMs'

Generated outline:

Outline(page_title='Attention Based Attribution Mechanisms for LLMs', sections=[OutlineSection(section_title='Introduction', description='An overview of attention-based attribution mechanisms and their significance in understanding large language models (LLMs). This section will introduce the concept of attention in neural networks and explain the importance of attribution mechanisms in interpreting LLM behavior.', subsections=None), OutlineSection(section_title='Datasets and Experiments', description='A detailed discussion of the various datasets and experimental setups used to research attention-based attribution mechanisms. This section will cover key datasets that are commonly utilized, methods of evaluation, and notable experiments that have contributed to the field.', subsections=None), OutlineSection(section_title='Future Directions', description='An exploration of potential future research directions in attention-based attribution mechanisms for LLMs. This section will highlight gaps in current research, proposed advancements in methodologies, and the implications of these advancements for interpretability and model development.', subsections=None)])

Usage

Default behavior:

from akd.agents.storm.storm import StormInputSchema, StormAgent, StormAgentConfig
agent = StormAgent(config=StormAgentConfig(model_name='gpt-4o-mini',
                                           long_context_llm='gpt-4o-mini', debug=True))
topic = "test topic"
input = StormInputSchema(topic=topic)
out = await agent.arun(input)

With outline sketch:

from akd.agents.storm.storm import StormInputSchema, StormAgent, StormAgentConfig
agent = StormAgent(config=StormAgentConfig(model_name='gpt-4o-mini',
                                           long_context_llm='gpt-4o-mini', debug=True))
topic = "test topic"
outline_sketch = "sketch here"
input = StormInputSchema(topic=topic, outline_sketch=outline_sketch)
out = await agent.arun(input)

Checks

• Tested Changes
• Stakeholder Approval

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 248
• Failed: 7
• Warnings: 139
• Coverage: 72%

Branch: refactor/storm
PR: #181
Commit: 029e2e2

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 248
• Failed: 7
• Warnings: 139
• Coverage: 72%

Branch: refactor/storm
PR: #181
Commit: 029e2e2

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 248
• Failed: 7
• Warnings: 139
• Coverage: 72%

Branch: refactor/storm
PR: #181
Commit: 1025a32

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 250
• Failed: 5
• Warnings: 139
• Coverage: 73%

Branch: refactor/storm
PR: #181
Commit: fb3384b

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 287
• Failed: 3
• Warnings: 223
• Coverage: 74%

Branch: refactor/storm
PR: #181
Commit: 06a2c08

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

✅ Tests passed

📊 Test Results

• Passed: 355
• Failed: 0
• Warnings: 198
• Coverage: 79%

Branch: refactor/storm
PR: #181
Commit: d934209

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 350
• Failed: 5
• Warnings: 198
• Coverage: 79%

Branch: refactor/storm
PR: #181
Commit: 68ed1c0

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 354
• Failed: 1
• Warnings: 200
• Coverage: 79%

Branch: refactor/storm
PR: #181
Commit: 8da2806

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: )

📊 Test Results

• Passed: 0
• Failed: 0
• Skipped: 0
• Warnings: 0
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.

Branch: refactor/storm
PR: #181
Commit: 5eabcc1

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: )

📊 Test Results

• Passed: 0
• Failed: 0
• Skipped: 0
• Warnings: 0
• Coverage: 0%

⚠️ Note: Test counts are 0, which may indicate parsing issues or early test failure. Check the workflow logs for details.

Branch: refactor/storm
PR: #181
Commit: 5eabcc1

📋 Full coverage report and logs are available in the workflow run.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 551
• Failed: 5
• Skipped: 7
• Warnings: 245
• Coverage: 79%

Branch: refactor/storm
PR: #181
Commit: 579bd39

📋 Full coverage report and logs are available in the workflow run.

[NISH1001]

@movinam with the latest deveop and some refactor of SearchAgent, it's required that all the search agents confirm to some standard output schema. See #258

Can we reflect similar changes to AspectSearchAgent accordingly? Thank you.

[movinam]

Will create a separate PR for Aspect Search to not bloat this PR. The only changes to Aspect Search are the default headers.

[github-actions]

❌ Tests failed (exit code: 1)

📊 Test Results

• Passed: 551
• Failed: 5
• Skipped: 7
• Warnings: 243
• Coverage: 81%

Branch: refactor/storm
PR: #181
Commit: 7c268e9

📋 Full coverage report and logs are available in the workflow run.