Free tool EbSynth turns video into hand-painted animation

Originally posted on 9 July 2019.

Start-up Secret Weapons has released EbSynth, an interesting free tool for converting video footage into stylised ‘motion paintings’.

The software transfers the look of one or more manually generated style frames to those from the video, creating the look of hand-painted animation.

Based on recent research in style transfer and texture synthesis

EbSynth is based on research from the Czech Technical University in Prague and Adobe Research, led by Secret Weapons founder Ondřej Jamriška, whose work has featured on CG Channel in the past.

The EbSynth website shows the demo video from the team’s most recent paper, Stylizing Video by Example.

Unlike other work done in the field, it isn’t based on machine learning, but uses a “state-of-the-art implementation” of non-parametric texture synthesis algorithms.

It now plans to develop EbSynth as a standalone tool.

Works from source footage and a single hand-painted reference frame

As input, EbSynth only requires the source video – converted to an image sequence in standard 2D file formats – and at least one still keyframe for reference.

The software then automatically transfers the style of the keyframe to the remaining frames of the footage, preserving “texture coherence, contrast and high frequency details”.

Since EbSynth isn’t working from any underlying 3D data, it can’t accurately stylise parts of objects that aren’t shown in the keyframe, or that are missing from it entirely.

That means that the keyframe should match the footage closely – ideally, it should be an overpaint of an actual video frame – and that a new keyframe is recommended after every significant change of perspective.

In addition, the user can supply masks to specify which parts of the video frame the stylisation should be applied to: for example, to stylise a character but not the background.

To judge from the demo video, the results are pretty good out of the box: there are noticeable transitions as the software moves from one keyframe to another, and a ‘halo’ region between character and background when not working with input masks, but the output is very watchable.

Updated 14 August 2020: EbSynth has now officially moved into beta.

New features include auto-filling of filenames, making it possible to drag directories of images into the software and have it process them automatically, rather than having to enter filepaths by hand.

The software also now exports directly to After Effects, with shots sequenced automatically in the After Effects timeline; while processing is now “10x faster” than the original release.

Interestingly, since EbSynth was originally released, some visual effects artists have begun to use the software to automate tasks like grading and blemish removal.

“They edit one frame and EbSynth distributes the changes into the whole scene, so it’s practical for retouching, colorising, and generally manipulating videos.”

Availability and system requirements

EbSynth is available in beta for Windows and macOS. The software is free for commercial use.

by Jim Thacker

https://ebsynth.com/