Jim Fan
@DrJimFan

Font design is about to get a huge upgrade. Traditionally, given the same font, the letter ‘A’ looks identical everywhere.

How about a smart & magical font that draws ‘A’ differently based on the semantic meaning of the word? Smells like Stable Diffusion?

Deep dive with me: 🧵

This really cool paper presents “word-as-image”, a semantic typography technique that illustrates a word as visualization of its actual meaning. It is a form of “constrained creativity” - the model still needs to respect the letter’s shape and preserve the readability.

1/

How does it work? The technique is pretty sophisticated. The authors use something called a “differentiable rasterizer” - it allows gradient to propagate from the rendered pixels back to the original shape parameters, which are the control points of the font.

2/

With the rasterizer, one can now construct an end-to-end pipeline. It applies Stable Diffusion to “nudge” the letter deformation towards an output image that reflects the semantic meaning. Additional loss functions are added to preserve the original font style.

3/

This method works for languages other than English as well! Here is a demo of the smart Chinese font. Since Chinese characters are already logograms, they allow for more degrees of freedom in the composition. IMHO these are even more awesome!

4/

The font doesn’t have to be black and white right? Stable Diffusion has you covered. Here are the colored ones: Frog, Swan, Spaceship, Owl!

5/

Fantastic work by the authors: Shir Iluz, @YVinker, @hertz_amir , Daniel Berio, @DanielCohenOr1, Ariel Shamir.

Website: https://wordasimage.github.io/Word-As-Image-Page/
Paper: https://arxiv.org/abs/2303.01818

For more deep dives 👇

Mar 6, 2023, 6:28:39 PM