feat: Support AWS Bedrock custom inference profiles #8801
Conversation
- Add detection and mapping for custom inference profile ARNs - Fix token limit validation for custom inference profiles (4096 instead of 8192) - Fix provider detection to use endpoint name instead of model name - Fix thinking configuration to not auto-enable for custom profiles - Add environment variable support for ARN-to-model mapping - Add comprehensive documentation and examples - Fix recursion issues in token detection functions - Add missing exports and endpoint mappings - Resolve 'Config not found' and 'Invalid URL' errors - Resolve 'thinking: Extra inputs are not permitted' error Closes danny-avila#6710
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
|
Thanks for this PR! Can you resolve the ESLint issues? Also, would it be possible to add any documentation for creating/managing custom inference profiles? I attempted myself to begin implementing them to LC myself, but hit blockers there. This would help me test your implementation in order to merge. |
|
also the tests you added in |
|
thanks for raising this PR. would love to see this in main 👍 |
|
@danny-avila is there anything else to fix for this one to be merged? |
merge conflicts have to be resolved |
devopsotrator
force-pushed
the
feat/aws-bedrock-custom-inference-profiles
branch
from
548f03a to
c37dd80
Compare
|
@danny-avila I resolved conflicts, also I've excluded most of the files edited for lint fixing as discussed before. |
- Rebuilt @librechat/data-schemas package to include missing accessRole methods - Fixed 'methods.seedDefaultRoles is not a function' error during server startup - The seedDefaultRoles method is now properly exported from createAccessRoleMethods - Updated package-lock.json with dependency changes The issue was that the data-schemas package needed to be rebuilt after recent changes to the accessRole.ts file. The build process now properly includes all accessRole methods including seedDefaultRoles in the createMethods function.
- Fixed prettier formatting issue in agentCategory.ts - Removed dist directory to avoid TypeScript parser errors during linting - The dist directory is properly excluded from git and will be rebuilt as needed The linting issues were caused by: 1. Incorrect formatting in agentCategory.ts model function 2. ESLint trying to parse dist directory files which are generated files These changes ensure clean linting while maintaining the functionality.
|
@danny-avila This seems to be ready for review (as per our Discord convo). |
|
Happy to revisit once merge conflicts are resolved and a proper, reproducible guide is written to the documentation repo: |
|
@iElsha can you help review this? Also maybe some of this can be consolidated now that @langchain/aws supports inference profiles |
4 participants
🎉 Support AWS Bedrock Custom Inference Profiles
Problem
AWS Bedrock custom inference profiles have ARNs that don't contain model name information, causing LibreChat to fail to recognize their capabilities. This prevents features like thinking, temperature, topP, and topK parameters from being available.
Solution
Key Features
Configuration
Users can now configure custom inference profiles using the
BEDROCK_INFERENCE_PROFILE_MAPPINGSenvironment variable:Issues Resolved
Testing
All functionality has been thoroughly tested and verified to work correctly with custom inference profile ARNs.
Closes #6710