audio works (archives collections)

[archivistsarah]

I'd like to be able to add audio works from our archival collections to the DC. These works may contain oral histories, but do not need to receive the same workflow and public facing display as the oral histories that SHI's department generates. They will also include works that are not oral histories, so any UI or display should be generic enough to accommodate different genres of audio.

I'm imagining a display similar to the way we handle video, with a player on the left and metadata on the right (and potentially the same transcript show/hide option as we use for video, although generating transcripts can be a secondary goal/separate ticket).

Some stuff that comes to mind immediately:

• Will these works need thumbnails and poster images?

• Can we use the player from the OH works? (I like some of its features and if we can reuse the work that was put into it...) Can it resize into the left hand side of the work page?

• Are there any technical limitations around file format? At the moment I'm planning for all of these to be MP3s, 44.1kHz, 192kbps stereo/96kbps mono -- but we may have outliers.

• Could we use the same WhisperAI workflow for audio transcriptions that we use for video?

• Copyright for audiovisual works is a nightmare, so this may trigger a larger conversation about copyright and our own risk tolerance.

[jrochkind]

Important question for architecture and UI is if these works can contain more than one audio file -- or more than one asset in general including non-audio files? And what the UI should look like in those cases if so.

Current video work UI does not support showing the user more than one asset, it supports only exactly one video asset and no other assets.

[jrochkind]

A question relative to digital formats and compression -- what's the nature of the audio files, are they all "people talking, mostly one at a time", or could they include music or field recordings or things like that?

[archivistsarah]

I can create access copies that are a single file, so we won't need to have multiple assets in a single work

the audio we currently have is 99% human voices, although we have a large collection of produced radio episodes coming in that will probably contain music and more layered audio, but I have zero details about it yet

[jrochkind]

Ah, so if you're creating additional access copies anyway, you can actually create them in whatever the format of our choice is? Yeah, let's talk about that before you start creating them!

[jrochkind]

We could have a poster image for search results containing these items? But can also do a generic "audio" icon -- if that's good enough, maybe you want to save time and not come up with artificial poster'? your choice, really!

On actual item page -- even if we have a poster image for search results, would we want it on item page, or just an audio player?

Your choice if you want us to develop this requiring poster image, optional poster image (and if it shows up in item page or just search results), or no poster image (only generic "audio" icon in search results). No poster image is probably quickest if urgency is a concern, can always add poster image feature later. But really up to you!

[archivistsarah]

Took a run at a very preliminary idea of what this could look like -- just a jumping off point so we can get started. This is based on reviewing some other audio collections + what we're already doing in the DC. The online audio interface I found and liked the most is (predictably), Aviary, which is specifically built for audiovisual cultural heritage collections.
Here's an example of an audio record with transcript in an Aviary repository: https://yalemssa.aviaryplatform.com/collections/56/collection_resources/17508

What I like most about it is the side-by-side display of media, description, and transcript (in a tab users can click to display instead of metadata). A lot of players I looked at took the full horizontal width of the screen and pushed a lot of information and functions below the fold. We already have this two column design in our image and video works, so I don't think it will be jarring to see it here, although we should discuss how much we want audio works (general) to mimic the oral histories program's interface, since these general audio works will include oral histories from the archives collections.

Some issues I can see:

• is this going to be absolutely awful on mobile? or will it vertically stack easily?
• where is the data for the individual asset titles and duration stored/coming from?
• can we offer a VTT download? is there a way we can offer it up with a .txt extension instead? (or PDF if that's not too much of an extra step?)
• how best to denote the order in which the assets should be displayed on the front end (the filenames contain sequence info but maybe not in a way the system can currently use)
• do we want duplicated title and date on the same page
• I didn't include a place for a thumbnail/poster, but we should talk about it
• is side by side really important? I like the idea of being able to stop/start the audio player, see where you are in the file, and generally interact with it while viewing the transcript and metadata. But this isn't a visual medium so it's not like users are missing out on content if they scroll away to read metadata or the transcript.

[jrochkind]

Thanks @archivistsarah !

Can you find an Aviary example with multiple media files?

I'm curious the thoughts/intentionality behind the choice of a "select menu popup" for multiple files rather than just listing them all on page in a list. Maybe because we expect very many files sometimes?

where is the data for the individual asset titles and duration stored/coming from?

This is already modelled in DC. A/V Assets have their lengths automatically extracted on ingest. And have a title field that can be filled out (defaults to filename).

how best to denote the order in which the assets should be displayed on the front end (the filenames contain sequence info but maybe not in a way the system can currently use)

Already modelled in DC, assets have a "position" and can be re-ordered. (Initial order is order they were selected on ingest I think). There might be a button already to re-order by filename order, I forget.

can we offer a VTT download? is there a way we can offer it up with a .txt extension instead? (or PDF if that's not too much of an extra step?)

Yes, no problem. Whisper-generated transcripts are in fact already in VTT format internally. PDF is an extra step but can be done.

This does make me want to revisit downloads for video though...

[jrochkind]

Is the transcript meant to be one transcript that goes across all files, or individual transcripts? Very curious to see what Aviary does with this, is one reason I'd love a multi-file aviary example (with transcript(s)!)

That's actually the most technically challenging thing to deal with either way that is occuring to me! Transcript when we have multiple files.

I guess whisper is going to give us multiple VTT's for multiple files; we may want to combine them into one VTT for download etc? That could be a challenge. Things like this.

[archivistsarah]

Work with multiple assets + transcript: https://yalemssa.aviaryplatform.com/collections/56/collection_resources/17507/transcript#transcript

The whole page refreshes when you select a different asset, and the page loads with the transcript for that file in the transcript tab.

[jrochkind]

Thanks. I don't love that UI but it gives us a baseline to discuss.

Aviary is doing separate VTT files downloadable for each asset too, looks like.

[archivistsarah]

I'm curious the thoughts/intentionality behind the choice of a "select menu popup" for multiple files rather than just listing them all on page in a list. Maybe because we expect very many files sometimes?

We could! I was mostly thinking of keeping the page uncluttered, but I totally defer to your UI expertise here. If we're keeping each side of each tape as its own file (which is how they come back from the vendor and how I save them when digitizing here), then some works for this first big project will have 12-16 audio files and the same number of transcript files. But those are outliers and most will have 2-6 files, from 1-3 tapes, 2 sides each.

(also totally on board with revisiting video downloads, but that's a separate conversation)

[jrochkind]

I am thinking of trying to re-use the video display template for audio too, also to keep thigns consistent.

But we can certainly have a flag saying re-use, but only provide downloads for audio for now!

[archivistsarah]

Oh, then definitely let's discuss as part of this project! Let's use what we've got.