Skip to content

Conversation

brianterry
Copy link

Issue #, if available:

Description of changes:

Problem
The WRAVAL application was missing critical configuration documentation, making it impossible for users to set up and run the tool successfully. Users encountered various errors including:
AttributeError: 'Settings' object has no attribute 'ENDPOINT_TYPE'
AWS credential errors
Bedrock model access issues
Missing SageMaker execution roles
Undefined S3 bucket references
The README only provided basic installation instructions (uv pip install . && wraval generate) without any guidance on the extensive AWS configuration required.

Root Cause
Through code analysis, I discovered that WRAVAL heavily depends on AWS services:
Amazon Bedrock for LLM inference
Amazon SageMaker for model deployment and custom endpoints
Amazon S3 for dataset storage
AWS STS for account ID retrieval
However, the configuration file (config/settings.toml) contained placeholder values like {aws_account} that needed to be replaced, and there was no documentation explaining:

  • How to configure AWS credentials
  • Which AWS permissions are required
  • How to enable Bedrock model access
  • How to create SageMaker execution roles
  • How to set up S3 buckets

Solution
I added comprehensive configuration documentation to address all setup requirements:

Enhanced README.md
Added a detailed "Configuration" section after "Quick start"
Documented three methods for AWS credential configuration
Listed specific AWS permissions required for each service
Provided step-by-step instructions for enabling Bedrock models
Added AWS CLI commands for SageMaker role creation
Included S3 bucket setup instructions
Added troubleshooting section for common configuration errors
Provided verification steps to test the setup

Benefits
Transforms WRAVAL from a broken out-of-the-box experience to a properly documented tool
Provides clear setup path for different use cases (Bedrock-only, SageMaker, local)
Reduces support burden by anticipating and addressing common configuration issues
Makes the tool accessible to users without deep AWS expertise
This fix enables users to successfully configure and run WRAVAL for LLM writing assistant evaluation tasks.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@EetuaLaine
Copy link
Contributor

EetuaLaine commented Jun 3, 2025

Thank you for the contribution. Overall LGTM. I would like to ask for two clarifications:

  1. How were these additional steps tested?
  2. Who reported this error and do you have context on what was the input? AttributeError: 'Settings' object has no attribute 'ENDPOINT_TYPE'. We would like to fix that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants