Skip to content

Commit ed012d2

Browse files
shaperkongmoumoulgrammel
authored
feat (provider): add metadata extraction mechanism to openai-compatible providers (#4397)
Co-authored-by: kongmoumou <[email protected]> Co-authored-by: Lars Grammel <[email protected]>
1 parent 3c5fafa commit ed012d2

14 files changed

+676
-18
lines changed

.changeset/big-bulldogs-behave.md

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@ai-sdk/deepseek': patch
3+
---
4+
5+
feat (provider/deepseek): extract cache usage as provide metadata

.changeset/curvy-ears-exercise.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
'@ai-sdk/openai-compatible': patch
3+
'@ai-sdk/provider-utils': patch
4+
---
5+
6+
feat (provider): add metadata extraction mechanism to openai-compatible providers

content/providers/02-openai-compatible-providers/index.mdx

+93
Original file line numberDiff line numberDiff line change
@@ -170,3 +170,96 @@ const provider = createOpenAICompatible({
170170

171171
For example, with the above configuration, API requests would include the query parameter in the URL like:
172172
`https://api.provider.com/v1/chat/completions?api-version=1.0.0`.
173+
174+
## Custom Metadata Extraction
175+
176+
The OpenAI Compatible provider supports extracting provider-specific metadata from API responses through metadata extractors.
177+
These extractors allow you to capture additional information returned by the provider beyond the standard response format.
178+
179+
Metadata extractors receive the raw, unprocessed response data from the provider, giving you complete flexibility
180+
to extract any custom fields or experimental features that the provider may include.
181+
This is particularly useful when:
182+
183+
- Working with providers that include non-standard response fields
184+
- Experimenting with beta or preview features
185+
- Capturing provider-specific metrics or debugging information
186+
- Supporting rapid provider API evolution without SDK changes
187+
188+
Metadata extractors work with both streaming and non-streaming chat completions and consist of two main components:
189+
190+
1. A function to extract metadata from complete responses
191+
2. A streaming extractor that can accumulate metadata across chunks in a streaming response
192+
193+
Here's an example metadata extractor that captures both standard and custom provider data:
194+
195+
```typescript
196+
const MyMetadataExtractor: MetadataExtractor = {
197+
// Process complete, non-streaming responses
198+
extractMetadata: ({ parsedBody }) => {
199+
// You have access to the complete raw response
200+
// Extract any fields the provider includes
201+
return {
202+
myProvider: {
203+
standardUsage: parsedBody.usage,
204+
experimentalFeatures: parsedBody.beta_features,
205+
customMetrics: {
206+
processingTime: parsedBody.server_timing?.total_ms,
207+
modelVersion: parsedBody.model_version,
208+
// ... any other provider-specific data
209+
},
210+
},
211+
};
212+
},
213+
214+
// Process streaming responses
215+
createStreamExtractor: () => {
216+
let accumulatedData = {
217+
timing: [],
218+
customFields: {},
219+
};
220+
221+
return {
222+
// Process each chunk's raw data
223+
processChunk: parsedChunk => {
224+
if (parsedChunk.server_timing) {
225+
accumulatedData.timing.push(parsedChunk.server_timing);
226+
}
227+
if (parsedChunk.custom_data) {
228+
Object.assign(accumulatedData.customFields, parsedChunk.custom_data);
229+
}
230+
},
231+
// Build final metadata from accumulated data
232+
buildMetadata: () => ({
233+
myProvider: {
234+
streamTiming: accumulatedData.timing,
235+
customData: accumulatedData.customFields,
236+
},
237+
}),
238+
};
239+
},
240+
};
241+
```
242+
243+
You can provide a metadata extractor when creating your provider instance:
244+
245+
```typescript
246+
const provider = createOpenAICompatible({
247+
name: 'my-provider',
248+
apiKey: process.env.PROVIDER_API_KEY,
249+
baseURL: 'https://api.provider.com/v1',
250+
metadataExtractor: MyMetadataExtractor,
251+
});
252+
```
253+
254+
The extracted metadata will be included in the response under the `providerMetadata` field:
255+
256+
```typescript
257+
const { text, providerMetadata } = await generateText({
258+
model: provider('model-id'),
259+
prompt: 'Hello',
260+
});
261+
262+
console.log(providerMetadata.myProvider.customMetric);
263+
```
264+
265+
This allows you to access provider-specific information while maintaining a consistent interface across different providers.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
import { deepseek } from '@ai-sdk/deepseek';
2+
import { generateText } from 'ai';
3+
import 'dotenv/config';
4+
import fs from 'node:fs';
5+
6+
const errorMessage = fs.readFileSync('data/error-message.txt', 'utf8');
7+
8+
async function main() {
9+
const result = await generateText({
10+
model: deepseek.chat('deepseek-chat'),
11+
messages: [
12+
{
13+
role: 'user',
14+
content: [
15+
{
16+
type: 'text',
17+
text: 'You are a JavaScript expert.',
18+
},
19+
{
20+
type: 'text',
21+
text: `Error message: ${errorMessage}`,
22+
},
23+
{
24+
type: 'text',
25+
text: 'Explain the error message.',
26+
},
27+
],
28+
},
29+
],
30+
});
31+
32+
console.log(result.text);
33+
console.log(result.usage);
34+
console.log(result.experimental_providerMetadata);
35+
// "prompt_cache_hit_tokens":1856,"prompt_cache_miss_tokens":5}
36+
}
37+
38+
main().catch(console.error);
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
import { deepseek } from '@ai-sdk/deepseek';
2+
import { streamText } from 'ai';
3+
import 'dotenv/config';
4+
import fs from 'node:fs';
5+
6+
const errorMessage = fs.readFileSync('data/error-message.txt', 'utf8');
7+
8+
async function main() {
9+
const result = streamText({
10+
model: deepseek('deepseek-chat'),
11+
messages: [
12+
{
13+
role: 'user',
14+
content: [
15+
{
16+
type: 'text',
17+
text: 'You are a JavaScript expert.',
18+
},
19+
{
20+
type: 'text',
21+
text: `Error message: ${errorMessage}`,
22+
},
23+
{
24+
type: 'text',
25+
text: 'Explain the error message.',
26+
},
27+
],
28+
},
29+
],
30+
});
31+
32+
for await (const textPart of result.textStream) {
33+
process.stdout.write(textPart);
34+
}
35+
36+
console.log();
37+
console.log('Token usage:', await result.usage);
38+
console.log('Finish reason:', await result.finishReason);
39+
console.log('Provider metadata:', await result.experimental_providerMetadata);
40+
// "prompt_cache_hit_tokens":1856,"prompt_cache_miss_tokens":5}
41+
}
42+
43+
main().catch(console.error);
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
import { deepSeekMetadataExtractor } from './deepseek-metadata-extractor';
2+
3+
describe('buildMetadataFromResponse', () => {
4+
it('should extract metadata from complete response with usage data', () => {
5+
const response = {
6+
usage: {
7+
prompt_cache_hit_tokens: 100,
8+
prompt_cache_miss_tokens: 50,
9+
},
10+
};
11+
12+
const metadata = deepSeekMetadataExtractor.extractMetadata({
13+
parsedBody: response,
14+
});
15+
16+
expect(metadata).toEqual({
17+
deepseek: {
18+
promptCacheHitTokens: 100,
19+
promptCacheMissTokens: 50,
20+
},
21+
});
22+
});
23+
24+
it('should handle missing usage data', () => {
25+
const response = {
26+
id: 'test-id',
27+
choices: [],
28+
};
29+
30+
const metadata = deepSeekMetadataExtractor.extractMetadata({
31+
parsedBody: response,
32+
});
33+
34+
expect(metadata).toBeUndefined();
35+
});
36+
37+
it('should handle invalid response data', () => {
38+
const response = 'invalid data';
39+
40+
const metadata = deepSeekMetadataExtractor.extractMetadata({
41+
parsedBody: response,
42+
});
43+
44+
expect(metadata).toBeUndefined();
45+
});
46+
});
47+
48+
describe('streaming metadata processor', () => {
49+
it('should process streaming chunks and build final metadata', () => {
50+
const processor = deepSeekMetadataExtractor.createStreamExtractor();
51+
52+
// Process initial chunks without usage data
53+
processor.processChunk({
54+
choices: [{ finish_reason: null }],
55+
});
56+
57+
// Process final chunk with usage data
58+
processor.processChunk({
59+
choices: [{ finish_reason: 'stop' }],
60+
usage: {
61+
prompt_cache_hit_tokens: 100,
62+
prompt_cache_miss_tokens: 50,
63+
},
64+
});
65+
66+
const finalMetadata = processor.buildMetadata();
67+
68+
expect(finalMetadata).toEqual({
69+
deepseek: {
70+
promptCacheHitTokens: 100,
71+
promptCacheMissTokens: 50,
72+
},
73+
});
74+
});
75+
76+
it('should handle streaming chunks without usage data', () => {
77+
const processor = deepSeekMetadataExtractor.createStreamExtractor();
78+
79+
processor.processChunk({
80+
choices: [{ finish_reason: 'stop' }],
81+
});
82+
83+
const finalMetadata = processor.buildMetadata();
84+
85+
expect(finalMetadata).toBeUndefined();
86+
});
87+
88+
it('should handle invalid streaming chunks', () => {
89+
const processor = deepSeekMetadataExtractor.createStreamExtractor();
90+
91+
processor.processChunk('invalid chunk');
92+
93+
const finalMetadata = processor.buildMetadata();
94+
95+
expect(finalMetadata).toBeUndefined();
96+
});
97+
98+
it('should only capture usage data from final chunk with stop reason', () => {
99+
const processor = deepSeekMetadataExtractor.createStreamExtractor();
100+
101+
// Process chunk with usage but no stop reason
102+
processor.processChunk({
103+
choices: [{ finish_reason: null }],
104+
usage: {
105+
prompt_cache_hit_tokens: 50,
106+
prompt_cache_miss_tokens: 25,
107+
},
108+
});
109+
110+
// Process final chunk with different usage data
111+
processor.processChunk({
112+
choices: [{ finish_reason: 'stop' }],
113+
usage: {
114+
prompt_cache_hit_tokens: 100,
115+
prompt_cache_miss_tokens: 50,
116+
},
117+
});
118+
119+
const finalMetadata = processor.buildMetadata();
120+
121+
expect(finalMetadata).toEqual({
122+
deepseek: {
123+
promptCacheHitTokens: 100,
124+
promptCacheMissTokens: 50,
125+
},
126+
});
127+
});
128+
129+
it('should handle null values in usage data', () => {
130+
const processor = deepSeekMetadataExtractor.createStreamExtractor();
131+
132+
processor.processChunk({
133+
choices: [{ finish_reason: 'stop' }],
134+
usage: {
135+
prompt_cache_hit_tokens: null,
136+
prompt_cache_miss_tokens: 50,
137+
},
138+
});
139+
140+
const finalMetadata = processor.buildMetadata();
141+
142+
expect(finalMetadata).toEqual({
143+
deepseek: {
144+
promptCacheHitTokens: NaN,
145+
promptCacheMissTokens: 50,
146+
},
147+
});
148+
});
149+
});

0 commit comments

Comments
 (0)