This feature defines an API surface for manipulating raw media carried by MediaStreamTracks such as the output of a camera, microphone, screen capture, or the decoder part of a codec and the input to the decoder part of a codec. It uses WebCodecs interfaces to represent raw media frames and exposes them using streams, similarly to the way the WebRTC Insertable Streams spec exposes encoded data from RTCPeerConnections.

Motivation

The motivation for this feature is to support some of the use cases described in WebRTC Next Version Use Cases. More specifically: * Funny Hats (https://www.w3.org/TR/webrtc-nv-use-cases/#funnyhats*): Refers to manipulation of media prior to encoding and after decoding to provide effects such as background removal, funny hats, voice effects. * Machine Learning (https://www.w3.org/TR/webrtc-nv-use-cases/#machinelearning*): Refers to applications such as real-time object identification/annotation.

Demo

• https://webrtc.github.io/samples/src/content/insertable-streams/video-processing/

Documentation

• https://w3c.github.io/mediacapture-insertable-streams/
• https://github.com/w3c/mediacapture-insertable-streams/blob/main/explainer.md

Specification

Editor's draft

Status in Chromium

Blink components: Blink>MediaStream

Origin trial (tracking bug)

Consensus & Standardization

• Firefox: No signal
• Edge: No signal
• Safari: No signal
• Web Developers: Positive

Owners

• guidou@chromium.org
• hta@chromium.org

Intent to Prototype url

Last updated on 2021-02-19