feat: Centralized Request Processing middleware #3847

dave-gray101 · 2024-10-16T07:35:56Z

This PR reworks how our endpoints parse and process user input, in order to centralize and commonize that code as much as possible. This will drastically cut down the endpoint-specific code to a minimum.

Additionally, the VAD subsystem was untested - making it hard to validate if this changeset was safe or not. Therefore, this PR also adds both manual testing tools (via bruno) and automated testing in the AIO image.

Note to @mudler - this PR looks worse than it is - here's some reviewing notes to help

core/http/middleware/request.go should be where you start - its the heart of the change and replaces core/http/ctx/fiber.go
configuration of defaults is now a responsibility of the route function - seems a bit weird at first, but allows endpoints to be easily reused with different defaults on different paths.
while all the endpoint files were updated, the changes all follow the same pattern:
- A new standard header to the function was added which pulls the relevant input and config from the earlier middleware stage
- A bunch of code is deleted as its now performed earlier by those middlewares
- The functional code specific to the endpoint is unchanged except for variable references
Specific to the AIO CI test: I was seeing very unpredictable behavior on CI versus locally reproducing this test. It turns out that setting LOCALAI_SINGLE_ACTIVE_BACKEND=true helps with a memory usage issue!

netlify · 2024-10-16T07:37:25Z

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	`f7dab99`
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/67a9c57513819e00084f4390
😎 Deploy Preview	https://deploy-preview-3847--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

dave-gray101 · 2024-11-21T07:49:47Z

~~@mudler merged in VAD, but I don't know how to test it manually yet since it's new. Please double check that :)~~

dave-gray101 · 2024-12-18T01:46:00Z

@mudler Ready for review, now with working VAD test.

Signed-off-by: Dave Lee <[email protected]>

dave-gray101 · 2025-02-05T19:17:23Z

merged and squashed this again.

@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!

mudler · 2025-02-06T18:32:28Z

merged and squashed this again.

@mudler - Can you decide if this PR is worth maintaining? It's been stable through my testing for a while now. I'd like to either find a merge window for this, or just scrap it if you dislike the central concept. Thanks!

Just had a quick look now and direction looks good, however since changes are big I need to take a closer look, I'll try to do this asap in these days as now would be a good merge window !

.bruno/LocalAI Test Requests/vad/vad test audio.bru

mudler · 2025-02-06T18:35:53Z

aio/cpu/vad.yaml

@@ -0,0 +1,8 @@
+backend: silero-vad


mudler · 2025-02-06T18:36:16Z

core/application/startup.go

-				config.LoadOptionF16(options.F16),
-				config.ModelPath(options.ModelPath),
-			)
+			cfg, err := application.BackendLoader().LoadBackendConfigFileByNameDefaultOptions(m, options)


mudler · 2025-02-06T18:37:14Z

core/backend/tokenize.go


-	inferenceModel, err = loader.Load(opts...)
+	// TODO: looks weird, seems to be a correct merge?


this looks going behind in history. this switch isn't needed anymore

I think I have corrected this :)

mudler · 2025-02-06T18:38:18Z

core/config/backend_config_loader.go

@@ -145,6 +147,15 @@ func (bcl *BackendConfigLoader) LoadBackendConfigFileByName(modelName, modelPath
 	return cfg, nil
 }

+func (bcl *BackendConfigLoader) LoadBackendConfigFileByNameDefaultOptions(modelName string, appConfig *ApplicationConfig) (*BackendConfig, error) {


cleanup really appreciated!

mudler · 2025-02-06T18:38:58Z

core/config/backend_config.go

+	}
+
+	if (u & FLAG_VAD) == FLAG_VAD {
+		if c.Backend != "silero-vad" {


note for later: at some point we should think about having these constants defined to be re-usable. There are some repetitions as well in the model initializer

Agreed - probably worth doing that as followup if you're OK with that

definitely, was just a reminder/note

Signed-off-by: Dave Lee <[email protected]>

tests/e2e-aio/e2e_test.go

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler · 2025-02-10T09:22:24Z

tests/e2e-aio/sample_data_test.go

this seems quite huge as a text golang file - can we maybe add the converted file to the repo in the fixture folder?

dave-gray101 force-pushed the feat-request-middleware branch from ed7f03e to 7dfddc7 Compare October 16, 2024 22:51

github-actions bot added the area/ai-model label Oct 16, 2024

dave-gray101 mentioned this pull request Oct 21, 2024

feat(templates): use a single template for multimodals messages #3892

Merged

1 task

dave-gray101 force-pushed the feat-request-middleware branch from 13a7411 to 85ed9ae Compare October 24, 2024 02:46

dave-gray101 requested a review from mudler October 24, 2024 03:50

dave-gray101 enabled auto-merge (squash) October 24, 2024 03:50

mudler mentioned this pull request Nov 15, 2024

chore(deps): bump llama-cpp to ae8de6d50a09d49545e0afab2e50cc4acfb280e2 #4157

Merged

1 task

mudler mentioned this pull request Dec 8, 2024

feat(template): read jinja templates from gguf files #4332

Merged

1 task

dave-gray101 force-pushed the feat-request-middleware branch 2 times, most recently from df37c12 to d08311f Compare December 17, 2024 20:06

github-actions bot added the dependencies label Jan 15, 2025

squash past, centralize request middleware PR

c1f30ba

Signed-off-by: Dave Lee <[email protected]>

dave-gray101 force-pushed the feat-request-middleware branch from 6050d80 to c1f30ba Compare February 5, 2025 19:14

Merge branch 'master' into feat-request-middleware

5e07c3c

mudler reviewed Feb 6, 2025

View reviewed changes

.bruno/LocalAI Test Requests/vad/vad test audio.bru Outdated Show resolved Hide resolved

mudler reviewed Feb 6, 2025

View reviewed changes

aio/cpu/vad.yaml

@@ -0,0 +1,8 @@

backend: silero-vad

Copy link

Owner

mudler Feb 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

dave-gray101 reacted with thumbs up emoji

mudler reviewed Feb 6, 2025

View reviewed changes

dave-gray101 added 4 commits February 8, 2025 17:10

Merge branch 'master' into feat-request-middleware

5b01500

migrate bruno request files to examples repo

aa9c995

Signed-off-by: Dave Lee <[email protected]>

fix

cac5f5a

Signed-off-by: Dave Lee <[email protected]>

Merge branch 'master' into feat-request-middleware

a33a4d5

mudler reviewed Feb 10, 2025

View reviewed changes

tests/e2e-aio/e2e_test.go Outdated Show resolved Hide resolved

Update tests/e2e-aio/e2e_test.go

b717f6f

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler reviewed Feb 10, 2025

View reviewed changes

Merge branch 'master' into feat-request-middleware

f7dab99

mudler approved these changes Feb 10, 2025

View reviewed changes

mudler disabled auto-merge February 10, 2025 11:06

mudler merged commit 3cddf24 into mudler:master Feb 10, 2025
23 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Centralized Request Processing middleware #3847

feat: Centralized Request Processing middleware #3847

dave-gray101 commented Oct 16, 2024 •

edited

Loading

netlify bot commented Oct 16, 2024 •

edited

Loading

dave-gray101 commented Nov 21, 2024 •

edited

Loading

dave-gray101 commented Dec 18, 2024

dave-gray101 commented Feb 5, 2025

mudler commented Feb 6, 2025

mudler Feb 6, 2025

mudler Feb 6, 2025

mudler Feb 6, 2025

dave-gray101 Feb 8, 2025

mudler Feb 6, 2025

mudler Feb 6, 2025 •

edited

Loading

dave-gray101 Feb 8, 2025

mudler Feb 10, 2025

mudler Feb 10, 2025


		inferenceModel, err = loader.Load(opts...)
		// TODO: looks weird, seems to be a correct merge?

feat: Centralized Request Processing middleware #3847

feat: Centralized Request Processing middleware #3847

Conversation

dave-gray101 commented Oct 16, 2024 • edited Loading

netlify bot commented Oct 16, 2024 • edited Loading

✅ Deploy Preview for localai ready!

dave-gray101 commented Nov 21, 2024 • edited Loading

dave-gray101 commented Dec 18, 2024

dave-gray101 commented Feb 5, 2025

mudler commented Feb 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mudler Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dave-gray101 commented Oct 16, 2024 •

edited

Loading

netlify bot commented Oct 16, 2024 •

edited

Loading

dave-gray101 commented Nov 21, 2024 •

edited

Loading

mudler Feb 6, 2025 •

edited

Loading