During the hype of llama.ttf months ago, I was speculating the potential of WASM shaper for even crazier purpose, one that fitter to a font shaper's duty -- to synthesize font at runtime. This project as proof-of-concept implements a synthesizer that generates and rasterizes handwriting-style font, backed by a super-lightweight RNN model (~14MiB).
I bet this is what people felt when JavaScript was first demoed on the web.
In the year 2077, when each font will run it's own virtual machine on WASM containing a "minimal" Ubuntu image, some enlightened blogger will suggest server side rendered fonts as a performance improvement.
JS felt less impressive to me when I first saw it, simply because it couldn't do much. However, I remember the showcase Microsoft put together to show off what CSS could do in IE3 and that was very cool at the time.
Without trying to steal the thread, what I would care is actually the opposite direction.
Neither in Swift Playgrounds, nor in any other programming development environment apps for both mobile OSes, have I found a good development experience using pen instead of keyboard.
Given how many of us "program" in paper notebooks, it is quite incredible that besides a couple of research projects done by PhD students, no one cares to actually make it more widespread in a usable way.
I could see something like Apples new calculator drawing app taking this by storm. Writing out your code and getting syntax support and all the things in your ide but with your handwriting?
In the same way that the calculator writes in the answers in your handwriting, it could write in snippets of code in your handwriting.
If it's going to be made though, it has to be e-ink. Go big or go home!
Handwriting (in and out) support is very important IMO. Also being able to draw DAGs.
I'd like an e-ink device with high frame rate and HW powerful enough to run some models locally or with good enough connectivity and sensors that e.g. Computer Vision tasks can be offloaded to the users' smartphone.
Feel free to expose your ideas on there :) I welcome Open Source discussion!
I've read the README and watched the video, but I'm still not sure what this is doing? I know it can "synthesize [a] font at runtime", so does that mean it's creating a random handwritten font as you type? But it's not based on the user's handwriting?
This looks like it'd be quite useful for faking documents more convincingly. Existing handwriting fonts always have tell-tale regularities and there's so few that forensics analysis exposes them easily.
I wonder what makes SIMD an improvement here - in the end it all boils down to TTF bytecode and I don't think this comes with SIMD instructions, right?
OP is asking how SIMD is beneficial to this specific application. If there's a small neural network involved then evaluating that is probably a good place to use SIMD.
but... the program ends up entirely compiled and executed as TTF bytecode in the end right, since it's entirely contained in the TTF font ? And TTF bytecode is only the following instructions : https://developer.apple.com/fonts/TrueType-Reference-Manual/... and I don't see anything related to SIMD in there
You can run the demo locally following the Usage part. Currently the WASM shaper feature is not enabled in any browser, so the demo won't work in web pages.
The handwriting in fact consists of multiple "tiny black box" glyphs, similar to pixels in traditional rendering. The program takes the text (e.g., "Hello world") as input, and works out the (x, y) locations of these block boxes.
this is amazing! i'm guessing you can probably get it to antialias without much more work
as the demo video shows, it's probably not something you want to have in between you and the ability to scroll a web page or close a tab. but i guess using harfbuzz now means we're buying into a turing-complete virtual machine running an arbitrary program in order to display a glyph. how seriously crippled are the harfbuzzless rendering paths? i'm assuming opting out of harfbuzz means opting out of arabic, devanagari and other indic scripts, etc.? is there a less out-of-control alternative that doesn't leave two billion people out in the cold?
WASM shaper is still experimental and not yet shipped in and products, not before more limitation carried out as I think. So I'm not too worried about it.
So you trained a THAT small RNN to make this good handwriting? I'm actually even more impressed at that than at the crazy pipeline turning it into a font in realtime.
You can't test it in a browser, since no browser at present is linked against libharfbuzz with WASM shaper. Instead, one can test it with a modified local program such as gedit in the demo.
As convenience, I built a Docker image that packs both the ttf file and the modified gedit together. You can try it out via `make run` as stated in the instructions.
WASM is to libharfbuzz (when experimental compile time option is enabled) what javascript is to HTML.
So this is essentially native (albeit experimental) libharfbuzz. WASM is used because its how font files are scripted (when using this experimental version of lubharfbuzz)
Its important to keep in mind that wasm is a general technology and is not just used by web browsers.
The project must be tested in an application linked against libharfbuzz with WASM shaper enabled. Since it's not easy to build a library like this, I make a Docker image which contains both the ttf file and a modified gedit, so that anyone can test the project with a single command.
Link to llama.ttf, another fun font that abuses HarfBuzz for anyone interested (it runs a language model inside a font): https://fuglede.github.io/llama.ttf/
The llama.ttf video does a pretty good job explaining what the heck is going on
This is so cursed, there is so much that begs the question - "why?"...
I love it.
I can't wait for the next beautiful nightmare. Maybe someone should mix font rendering with PDF rendering. Of course, with a LLM doing something in the middle.
The answer is "no why", just for fun :) Though I learned a lot about how to stuff an NN model in a WASM binary and some tricks to optimize performance.
In the year 2077, when each font will run it's own virtual machine on WASM containing a "minimal" Ubuntu image, some enlightened blogger will suggest server side rendered fonts as a performance improvement.