Drawing to a Frame (Skia)
What is Skia?​
Skia is a 2D graphics library that can be used to draw shapes, images, text, color-shaders and much more. Skia is GPU-accelerated by Metal on iOS and OpenGL on Android.
To provide a powerful cross-platform API for drawing directly to Camera Frames in realtime, VisionCamera provides a first-party react-native-skia integration via Skia Frame Processors:
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
const bananas = detectBananas()
frame.render()
for (const banana of bananas) {
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(banana.rect, paint)
}
}, [])
Installation​
Skia Frame Processors require @shopify/react-native-skia 1.2.1 or higher, and react-native-reanimated 3.0.0 or higher. Install the packages through npm and make sure you follow their installation instructions:
npm i @shopify/react-native-skia
npm i react-native-reanimated
Since the Skia integration is powered by C++ HardwareBuffers, set your Android's minSdkVersion to 26 or higher.
Skia Frame Processors​
A Skia Frame Processor, just like any other Frame Processor, runs synchronously for every Camera Frame.
Instead of a Frame, it is called with a DrawableFrame, which extends the Frame with a drawing canvas.
To create a Skia Frame Processor, use the useSkiaFrameProcessor hook.
In a Skia Frame Processor the caller is responsible for rendering the Camera Frame, so Frame.render() must always be called:
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render()
}, [])
Skia APIs​
To draw something to the Frame, use Skia's imperative APIs. For example, to draw a red rectangle in the center of the Frame, use drawRect(...):
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render()
const centerX = frame.width / 2
const centerY = frame.height / 2
const rect = Skia.XYWHRect(centerX, centerY, 150, 150)
const paint = Skia.Paint()
paint.setColor(Skia.Color('red'))
frame.drawRect(rect, paint)
}, [])
The Camera Frame is rendered like any other SkImage. You can pass a custom SkPaint object to the render(..) function to use a shader, for example to render the Frame with inverted colors use a RuntimeEffect:
const invertColorsFilter = Skia.RuntimeEffect.Make(`
uniform shader image;
half4 main(vec2 pos) {
vec4 color = image.eval(pos);
return vec4((1.0 - color).rgb, 1.0);
}
`)
const shaderBuilder = Skia.RuntimeShaderBuilder(invertColorsFilter)
const imageFilter = Skia.ImageFilter.MakeRuntimeShader(shaderBuilder, null, null)
const paint = Skia.Paint()
paint.setImageFilter(imageFilter)
const frameProcessor = useSkiaFrameProcessor((frame) => {
'worklet'
frame.render(paint)
}, [paint])
Coordinate system​
Each Frame, a Skia Frame Processor is rendering to an offscreen SkSurface.
The Camera Frame is a GPU-texture-backed SkImage, and it's coordinate system is in Frame dimensions and orientation.
- (
0,0) is top left - (
frame.width,frame.height) is bottom right
Performance​
Just like normal Frame Processors, Skia Frame Processors are really fast. Skia is GPU-accelerated by Metal and OpenGL, and VisionCamera Frames are streamed as efficiently as possible using GPU-buffers and textures.
A Skia Frame Processor can run and render at up to 500 FPS, depending on how complex the rendering code is.
Preview-only​
Skia Frame Processors are currently preview-only. Any content drawn to the Frame will not be visible in captured photos, snapshots or videos.
We at Margelo have worked a lot with 2D/3D graphics and Camera realtime processing (see the Snapchat-style mask filter on our website for example - that is running in VisionCamera/React Native!), if you need to capture drawn content to photos or videos, reach out to us and we'll build a customized/tailored solution for your company! :)