Skip to content

Latest commit

 

History

History
130 lines (105 loc) · 8.44 KB

background_video_filters.md

File metadata and controls

130 lines (105 loc) · 8.44 KB

Integrating background filters into your Amazon Chime SDK for Android application

What is a Background Blur and Replacement?

The background blur and replacement APIs allow builders to apply background filter (blur or replacement) on frames received from a video source. The filter processors uses a TensorFlow Lite (TFLite) machine learning model to segment the foreground of a frame and then apply on top of the blurred background or a replacement image. Follow this guide for more information on how to use BackgroundBlurVideoFrameProcessor and BackgroundReplacementVideoFrameProcessor.

Background blur and replacement are integrated in the AmazonChimeSDKDemo app. To try it out, follow these steps:

  1. Run the AmazonChimeSDKDemo on your device.
  2. Join a meeting.
  3. Enable video.
  4. Click on the video tab.
  5. Click on the icon with three dots under your local video tile.
  6. Click Turn on background blur or Turn on background replacement filter from the menu.

Getting Started

Prerequisites

  • Have amazon-chime-sdk-machine-learning library imported. Follow README for more information on how to import these dependencies.
  • You have read the API overview and have a basic understanding of the components covered in that document.
  • You have completed Getting Started and have running application which uses the Amazon Chime SDK.
  • You have read the Custom Video Sources, Processors, and Sinks and have a basic understanding of APIs such as VideoSource.

Overview

BackgroundBlurVideoFrameProcessor and BackgroundReplacementVideoFrameProcessor uses VideoSource and VideoSink APIs to consume and modify frames which are then fanned out to downstream sinks. To use the processors, builders must wire up the processor to a video source external (e.g. DefaultCameraCaptureSource) using VideoSource.addVideoSink(sink). Then enable the local video with background blur or replacement processor as the source using AudioVideoFacade.startLocalVideo(source).

Implementing background blur in your application.

  1. Create a DefaultCameraCaptureSource. This requires a SurfaceTextureCaptureSourceFactory as a dependency. Both require a Logger and the camera capture source requires an application context to reach system camera APIs.
val surfaceTextureCaptureSourceFactory = DefaultSurfaceTextureCaptureSourceFactory(logger, eglCoreFactory)
val cameraCaptureSource = DefaultCameraCaptureSource(applicationContext, logger, surfaceTextureCaptureSourceFactory)
  1. Create a BackgroundBlurVideoFrameProcessor. Its constructor takes four parameters:
logger: Logger,
eglCoreFactory: EglCoreFactory,
context: Context,
configurations: BackgroundBlurConfiguration
  • Logger to log any warnings or errors.
  • A DefaultEglCoreFactory to share between the capturer and facade.
val eglCoreFactory = DefaultEglCoreFactory()
  • Application Context
  • BackgroundBlurConfiguration - blurStrength specifies blurValue that corresponds to blur radius used in gaussian blur. The higher the value, more blurrier the image will be. The processor will default to 7.0f if not provided by the builder.
val backgroundBlurVideoFrameProcessor = BackgroundBlurVideoFrameProcessor(
    ConsoleLogger(LogLevel.DEBUG),
    eglCoreFactory,
    applicationContext,
    BackgroundBlurConfiguration(12.5f)
)
  1. Add the background blur processor as sink to the video source (e.g. DefaultCameraCaptureSource)
cameraCaptureSource.addVideoSink(sink: backgroundBlurVideoFrameProcessor)
  1. Use background blur processor as source
audioVideo.startLocalVideo(backgroundBlurVideoFrameProcessor)

BackgroundBlurVideoFrameProcessor will receive the frames and apply the foreground on top of the blurred background image which is sent to the downstream sinks to render the modified frame.

Implementing background replacement in your application.

  1. Create a DefaultCameraCaptureSource. This requires a SurfaceTextureCaptureSourceFactory as a dependency. Both require a Logger and the camera capture source requires an application context to reach system camera APIs.
val surfaceTextureCaptureSourceFactory = DefaultSurfaceTextureCaptureSourceFactory(logger, eglCoreFactory)
val cameraCaptureSource = DefaultCameraCaptureSource(applicationContext, logger, surfaceTextureCaptureSourceFactory)
  1. Create a BackgroundReplacementVideoFrameProcessor. Its constructor takes four parameters:
logger: Logger,
eglCoreFactory: EglCoreFactory,
context: Context,
configurations: BackgroundReplacementConfiguration
  • Logger to log any warnings or errors.
  • A DefaultEglCoreFactory to share between the capturer and facade.
val eglCoreFactory = DefaultEglCoreFactory()
  • Application Context
  • BackgroundReplacementConfiguration - an image Bitmap to replace video background with. Defaults to shaded blue colored image. See BackgroundReplacementConfiguration for more information. The below code provides an example on how to load an image Bitmap from a URL string.
fun loadBackgroundReplacementVideoFrameProcessorWithImage(processor: BackgroundReplacementVideoFrameProcessor, imageUrl: String) {
    GlobalScope.launch(Dispatchers.IO) {
        val image: Bitmap = try {
            val url = URL(imageUrl)
            BitmapFactory.decodeStream(url.openConnection().getInputStream())
        } catch (exception: Exception) {
            throw exception
        }
        processor.configurations = BackgroundReplacementConfiguration(image)
    }
}

// Use the default until the image is loaded asynchronously.
val backgroundReplacementVideoFrameProcessor = BackgroundReplacementVideoFrameProcessor(
    ConsoleLogger(LogLevel.DEBUG),
    eglCoreFactory,
    applicationContext,
    BackgroundReplacementConfiguration()
)
loadBackgroundReplacementVideoFrameProcessorWithImage(backgroundReplacementVideoFrameProcessor, "https://...")
  1. Add the background replacement processor as sink to the video source (e.g. DefaultCameraCaptureSource)
cameraCaptureSource.addVideoSink(sink: backgroundReplacementVideoFrameProcessor)
  1. Use background replacement processor as source
audioVideo.startLocalVideo(backgroundReplacementVideoFrameProcessor)

BackgroundReplacementVideoFrameProcessor will receive the frames and apply the foreground on top of the replacement image which is sent to the downstream sinks to render the modified frame.