Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(llama.cpp): improve context shift handling #4820

Merged
merged 1 commit into from
Feb 14, 2025

Conversation

mudler
Copy link
Owner

@mudler mudler commented Feb 13, 2025

Description

This pull request includes changes to the backend/cpp/llama/grpc-server.cpp file to handle context capacity and prevent redundant checks. The most important changes include adding a check for context capacity and improving logging when context is exhausted.

Context capacity handling:

  • Added a check to set truncated and stopped_limit flags and log a message when the context capacity is reached. (backend/cpp/llama/grpc-server.cpp)

Redundant check removal:

  • Removed redundant checks and improved logging when context is exhausted by releasing the slot and logging an error message. (backend/cpp/llama/grpc-server.cpp)

Notes for Reviewers

Signed commits

  • Yes, I signed my commits.

Copy link

netlify bot commented Feb 13, 2025

Deploy Preview for localai ready!

Name Link
🔨 Latest commit 652d4b6
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/67af310860aaa80008ea81f9
😎 Deploy Preview https://deploy-preview-4820--localai.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@mudler mudler force-pushed the feat/improve_context_shift branch from bd0c800 to 652d4b6 Compare February 14, 2025 12:03
@mudler mudler added the bug Something isn't working label Feb 14, 2025
@mudler mudler merged commit 9e32fda into master Feb 14, 2025
25 checks passed
@mudler mudler deleted the feat/improve_context_shift branch February 14, 2025 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant