TimestampGranularities is not working properly

[danielrahn]

Currently it is only possible to get segment timestamps but no other.
It is possible to set other granularities in the client and in the api, but it looks like the form data request from the client is not correct.

package main

import (
	"context"
	"io"
	"log"
	"os"

	"github.com/openai/openai-go"
)

func main() {
	client := openai.NewClient()
	file, _ := os.Open("audio.m4a")
	transcription, _ := client.Audio.Transcriptions.New(context.Background(), openai.AudioTranscriptionNewParams{
		File:           openai.F[io.Reader](file),
		Model:          openai.F(openai.AudioModelWhisper1),
		ResponseFormat: openai.F(openai.AudioTranscriptionNewParamsResponseFormatVerboseJSON),
		TimestampGranularities: openai.F([]openai.AudioTranscriptionNewParamsTimestampGranularity{
			openai.AudioTranscriptionNewParamsTimestampGranularityWord,
			openai.AudioTranscriptionNewParamsTimestampGranularitySegment,
		}),
	})

	log.Printf("words.IsMissing(): %t\n", transcription.JSON.ExtraFields["words"].IsMissing())
	// words.IsMissing(): true
	log.Printf("segments.IsMissing(): %t\n", transcription.JSON.ExtraFields["segments"].IsMissing())
	// segments.IsMissing(): false
}

--3c013d85d22e8db1236e9831f28251c99e205b0326f97909758cea53ad82
Content-Disposition: form-data; name="timestamp_granularities.0"

word
--3c013d85d22e8db1236e9831f28251c99e205b0326f97909758cea53ad82
Content-Disposition: form-data; name="timestamp_granularities.1"

segment
--3c013d85d22e8db1236e9831f28251c99e205b0326f97909758cea53ad82--

Looks like the client sets the wrong name (timestamp_granularities.0 / timestamp_granularities.1 instead of timestamp_granularities[]). The request via curl works without problems.

curl https://api.openai.com/v1/audio/transcriptions \
  -H "Authorization: Bearer X" \
  -H "Content-Type: multipart/form-data" \
  -F file="@audio.mp4" \
  -F "timestamp_granularities[]=word" \
  -F "timestamp_granularities[]=segment" \
  -F model="whisper-1" \
  -F response_format="verbose_json"

--------------------------ivB13eW5FxAXMC0FGfs777
Content-Disposition: form-data; name="timestamp_granularities[]"

word
--------------------------ivB13eW5FxAXMC0FGfs777
Content-Disposition: form-data; name="timestamp_granularities[]"

segment
--------------------------ivB13eW5FxAXMC0FGfs777

{
  "task": "transcribe",
  "segments": [],
  "words": []
}

[RobertCraigie]

Thanks for the detailed report, we're investigating.

[Mau-MD]

+1. The following snippet only returns segment and not word granularity. Curl works fine

	transcriptionParams := openai.AudioTranscriptionNewParams{
		File:           openai.F(reader),
		Model:          openai.F(openai.AudioModelWhisper1),
		Language:       openai.F("en"),
		Prompt:         openai.F(promtp),
		ResponseFormat: openai.F(openai.AudioResponseFormatVerboseJSON),
		TimestampGranularities: openai.F([]openai.AudioTranscriptionNewParamsTimestampGranularity{
			openai.AudioTranscriptionNewParamsTimestampGranularityWord,
		}),
	}

[Blagoj5]

Current workaround that I did is by manually overwriting the RequestBody using option.WithRequestBody function. Code snippet example:

	body := &bytes.Buffer{}
	writer := multipart.NewWriter(body)
	fileWriter, err := writer.CreateFormFile("file", audioPath)
	if err != nil {
		return res, err
	}
	file, err := os.Open(audioPath)
	if err != nil {
		return res, err
	}
	_, err = io.Copy(fileWriter, file)
	if err != nil {
		return res, err
	}
	writer.WriteField("model", "whisper-1")
	writer.WriteField("timestamp_granularities[]", "word")
	writer.WriteField("timestamp_granularities[]", "segment")
	writer.WriteField("response_format", "verbose_json")
	writer.Close()

	transcription, err := client.Audio.Transcriptions.New(
		context.TODO(),
		openai.AudioTranscriptionNewParams{
			File:                   file,
			Model:                  openai.AudioModelWhisper1,
			TimestampGranularities: []string{"word", "segment"},
			ResponseFormat:         openai.AudioResponseFormatVerboseJSON,
		},
		option.WithRequestBody(writer.FormDataContentType(), body),
	)
      // ... continue

Not the prettiest workaround, but it works for now. I don't think you also need the full openai.AudioTranscriptionNewParams parameter (second parameter) since you're over-writing it later on.