This absolutely rocks: like an open and usable version of the once beloved and now deprecated Echo Nest Audio Analysis api [0]. There used to be a tool to use this data to manage your own (Spotify-only) music library and while it wasn’t magic, it sure felt like it.
11 years ago, NY Magazine published my favorite startup zeitgeist article of the Uber-For-X era: “Let’s, like, demolish Laundry”. This feels like a perfect revisit of that for our darker and more cynical era.
There have been several projects over the past few years to make text embeddings visually explorable (notably Nomic.ai's Nomic Atlas). However, Apple's just released a tool that makes this kind of analysis super accessible and insanely interactive.
Under the hood it's powered by Mosaic[0], a dataviz library built on top of DuckDB that's designed to handle coordinated interactive plots over huge datasets, the kind of thing where you interact with one plot and the rest all respond, which requires going back to the database to recalculate all the aggregations.
I've been fanboying Mosaic for the past year but finally have this to point to as an illustration of what's possible with it.
> Polygons, curves, or lines are an afterthought. For example, the polygonToCells function returns a set of cells that are entirely within the polygon. If you want to join, you'd need to also have the set of all cells that entirely contain the polygon. I've never found a reliable way to get that.
With v4 of h3 they (finally) have a clean syntax for this with polygonToCellsExperimental[0].
Now there’s options for
- Cell center is contained in the shape (default)
- Cell is fully contained in the shape
- Overlapping (covering): Cell overlaps the shape at any point
- BBOX: Cell bounding box overlaps shape
Makes life a fair bit easier if you’ve gotta deal with H3 polys. And if you’re working locally, DuckDB Spatial’s r-tree indexing[1] can make for a nice stand-in for PostGIS as a quick point-in-polygon solution without the need to spin up a service.
Nice, I did not know about polygonToCellsExperimental.
As for DuckDB & rtree, it's alas not a replacement of postgis yet and the indices cannot be used (yet) in joins. In fact, I even have workflows where I iterate over rows in python and run duckdb queries one after the other rather than joining in just one query because of this very issue.
Chronicling America is great, but as you’ve noticed it’s limited to issues that are in the public domain. It’s not just LC, the National Digital Newspaper Program[1] has been funding hundreds of libraries across all 50 USA states to make it possible since the mid-00s.
Australia’s national library offers Trove[2], which has a huge collection of Australian public domain newspapers.
Most of their repository is likely funded by your tax dollars and is there for the public to use.
Trove is fantastic. It came in handy very recently for the same project I used the Library of Congress for[1]. The Museums of History New South Wales has a great online archive as well[2].
It’s good to hear that PWAs are still on Firefox Android, even though they’re out of Desktop.
From my [fairly-out-of-the-loop-for-the-past-few-years] vantage point, Mozilla’s been a lot less invested in the PWA ecosystem since they abandoned their Firefox Phone / Boot2Gecko initiative, which was intended to create a middle tier between the expensive smartphones of the early 2010’s and ubiquitous and cheap feature phones (flip phones, classic Nokia candy bars), and expand access to the web across the world with it.
All the apps were PWAs, which made it simple to build out. Eventually Mozilla stopped the project, but KaiOS became a commercial implementation and it still runs on a fair number of feature phones to this day.
But without that pressure for PWA support in Firefox as a critical mobile feature, it was largely serving as an expensive bookmark launcher in the Firefox code base so that folks could alt-tab to the small number of sites that supported it on their desktops. Not a noble end for Mozilla’s support for what should have been / could have been an incredible leverage point for the web ecosystem and open development.
It’s not. If it holds up well in terms of operating it and extending it when things change, then it’s not “too simple”. You’ve just built on 4 extremely resilient and scalable technologies which have made the kind of “Big Data” hoop-jumping of the past 20+ years rather moot.
You know the drill: SQLite and DuckDB mean you’ve got transactional database and data warehouse that live in the single machine and unless it’s terabytes runs analytics performance comparable to bigquery and snowflake.
No need to write an etl pipeline that runs on a Hadoop cluster, provision that Hadoop cluster, spin up a Hive/Pig instance for analysis. Nah, it fits in *your* computer and is scripted using the same language in the pipelines as in the analysis without a performance cost.
If you need to scale it, it’s not technical scaling, it’s team knowledge scaling (still hard, but not a fundamental stumbling block). So bring in DBT/Dagster (or airflow) and now it’s got supported frameworks that others already know and use.
Thank you for your valuable feedback - much appreciated!
Team knowledge scaling is hard - I totally agree and learned a lot of lessons.
Top management usually works „email only“. No matter what cool dashboard you’re building: they don’t use it, because they’re working almost exclusively on their phones.
And that‘s one tough challenge in my opinion: making data easy to understand on small screens.
Then there is this group of CFOs … they love to connect their Excel to a live datastream. Once. Because at some point they return to static sheets just to prove that a 35 MB Excel file shows their latest forecast.
Likewise; this ticks so many boxes for me, but as a hippy-dippy non-gmail user I too have to second the IMAP/JMAP ask.
Gmail absolutely hits that sweet spot of API capabilities and where the users are, so I can’t fault the project creators (or most every email client business these days) for building first (or exclusively) for it.
That said, seeing Outlook as coming soon on their login page is reassuring that they’re building in a way that won’t tie them to Gmail forever. And while few email providers outside of Fastmail are offering JMAP support, as an API it’s much closer to the degree of functionality expected by anyone building on top of Gmail’s API today. A great new client that gives a big section of the public a better way to “do email” might be what it takes for more services to start offering JMAP.
So hats off to y’all and fingers crossed on incorporating open standards.
Hey Riordan, curious to know specifically what you're looking for?
But right on about Gmail, it's much easier for us to prototype and test than some of the other clients.
Also really appreciate the kind words! Means a lot to us at this early stage. Hopefully you do join the email list or discord so we can keep you posted on our progress!
