(Hi, Retool employee here who worked on this piece.)
This is the second installment[1] in what we hope will be a regular series telling the stories of novel programming environments that had a lot of influence on developers.
Creating these is a labor of love—we get to talk to the original teams who built products that were formational for us. We have an incredible in-house creative team that finds a way to sneak these fun mini-sites in amongst their other work.
For me, the most exciting part of working on a product like Retool is that it's an evolution of lot of ideas from the past. We're always looking at prior art to spark ideas and challenge our assumptions as we're building. There are many great ideas in old computing papers and products that were either before their time, or lost when there were major shifts in the industry. Pipes was a touchpoint for us as we built out one of our newer products[2].
We have lots of ideas for future installments in this series, but if you have any programming environments that were near and dear to you heart, let me know!
[1] Our first deep-dive was on Visual Basic. I'm very biased (I wrote it!), but if you like Glenn's Pipes piece, you'll probably enjoy this one too. :) https://retool.com/visual-basic
Hi Ryan, I appreciate the artistic approach taken, but thought you might want to know the page barely works for me in Firefox 120 on Linux 6.6. Everything is very laggy and the scrolljacking is terrible. Chromium is much smoother but the scrolljacking is still bad.
Generally speaking I don't think Linux is a target platform worth worrying too much about, but when it comes to programming-related content that might not be the case.
Generally speaking I don't think Linux is a target platform
Maybe, but you’d think an iOS iPad might be a target. I just closed the tab, as there’s no reader view, and the janky scrolling and waiting for text to load (text!) wasn’t worth the effort.
Same here on Firefox 120 Win10, but I don't mind it on the rare occasion such as this where it's half art project half write-up. If it were every article it would get tiresome.
Author here: Thank you so much for the amazing work! I am still finding more Easter eggs. Tips to other readers: click on EVERYTHING on the desktop in the opening splash screen.
My favorite “project awareness” technique for work was through pipes and I’ve never had something as quick as way back then.
I had a yahoo pipe that combined and deduped about 10-15 jira projects and about 30-50 GitHub commit streams into a single rss feed that I read through google reader.
It was glorious because it let me know quickly who changed what and when and where. And it was rss so I didn’t have to search through email or look at multiple notification pages.
I miss Yahoo Pipes so much. And YQL. In 2009 I implemented an effectively cross-domain XHR (ajax) plugin on top of jQuery so I could grab random webpages on the client-side and query them via CSS selectors (converted to XPath). YQL would give me JSONP back so it all felt quite magical. Ah the old days...
There are SaaS offerings for Node-RED or you can install it yourself but there isn't a one-click-Node-RED experience but https://deadred.flowhub.org provides that by being a serverless, static frontend-only version.
Yes pipes could be used as a CORS proxy, I built a full Spotify like app client side with jquery at the time, searching/streaming mp3s at radioblogclub and VK.com and fetching metadata/similar recommendations from last.fm etc...
Ah man, the open web we were promised. I miss these days of mashing up multiple services and just the general free for all. Can't do anything without an API key these days.
I too share dreams of that open web and want to re-create it. The first JS book I saw was the classic DHTML book with an attractive cover (can't find it now) in 2002 at Kinokuniya in Sydney. Good times.
i'm building a dev API on top of https://github.com/BrowserBox/BrowserBox that should provide the same functionality as a CORS-proxy / unrestricted iframe to be able to embed and interact with content across origins.
Ha, I bet we're similar age. I haven't heard the term DHTML in so long. Hows AJAX holding up these days? Then long polling aka Comet? Thankfully we've always embraced puns in tech.
Haha! :) Yes indeed. As they say, life begins at... :) hahaha. The old AJAX, DHTML and Comet. I never thought of them as puns, but that's clever! I'm a fan of the old MJPEG, SSE and marquee tag. Hahaha :)
Turns running free services is expensive with often little reward. And since essentially the only way to do micropayments currently is with ads, any platform that can't effectively deliver ads essential can't survive. Until users are willing to pay for web services, and provided an easy way to do so, there are a lot of cool things that will remain unsustainable.
Not everything has a profit motive behind it. We have this entire concept called charities and philanthropy that also show cases how people will spend their own money to accomplish goals or viewpoints.
You've never hosted a fan site or a squad or guild site for a game or movie? Or ran a discussion forum? Or hosted an irc server for a community.
Yes pipes could be used as a CORS proxy, I built a full Spotify like app client side with jquery at the time, searching/streaming mp3s at radioblogclub and VK.com and fetching metadata/similar recommendations from last.fm etc...
Already mentioned above before I saw this comment but I'll pile on. Firefox on Linux workstation is really bad. Chromium much better but the scrolljacking is still really bad
I've never seen the phone lag on any other website. Hell, my Pixel 1 or Nexus phones didn't have issues either. I've never seen lag like this on any website, period... I don't think it's a GPU thing. Probably there is some condition that makes it really laggy on some devices and not others. Hmm.
I'm on a 13900K with an RX 6600 and it is barely usable for me as well. Firefox tho... just tested in Chrome and it is better after the initial scroll but that first move is a killer.
Yeah, Firefox on my office desktop (10-core i9 with 16GB RAM. Crap GPU though) was unresponsive for a good 30 seconds before I was able to close the tab. Worried for a sec that I'd have to force close the browser.
Just signed up this seems like it will replace some docker containers I have running on a free tier of fly.io dumping into a supabase free tier. Very cool thank you for sharing!
The owner, tod sacerdoti, is using it to scrape top stories from lobste.rs and submit them to HN as a karma arbitrage pump. Search todsacerdoti on https://gerikson.com/hnlo/ to see what I’m talking about. Judging by the amount of HN karma he’s accumulated it seems to work very well.
Consider setting a delay of a few days before reposting, sometimes people post their own stuff on lobste.rs, want to also post on HN and you've already done so. It can be perceived negatively.
Readers may enjoy Flowpipe [1], an open source cloud scripting engine we launched this week. It allows creation of pipelines in HCL to connect HTTP queries, run containers, execute lambda-compatible functions, query databases, etc all from your own machine / CLI. It can also be combined with Steampie[2], our open source project to query cloud resources (139+ plugins) with SQL. We're more DevOps focused, but Yahoo Pipes was part of the inspiration for our naming of these "pipes" :-)
Thanks for your kind words - so great to hear that :-). We're really excited about Flowpipe and think it makes "cloud scripting" really fun & easy through pipelines - please let us know how you go and what we can improve!
I used pipes back in college to randomize our roommate ordering in picking dorm rooms. I had it take the top headline on The NY Times website, hash it, and pull out numbers from 1-6 in a deterministic way. It was the best way I could think of to do it from afar over the summer, and convince everyone it was fair without needing to run code ourselves. It was useful and fun to use!
Pipes was awesome for sure. It fit so nicely into a golden web 2.0 era sort of period where there was still loose open interoperability thanks to prevalent RSS and similar tech. Effective at cobbling together simple things to make simply useful things.
What's with this epic design and work put into this deep dive tho? What an effort from Retool. Suppose maybe just wondering why they didn't do it earlier! Ha
I’m interested in how an article like this comes to be. I can tell that it was made with a lot of care and I’m sure it took a while to polish to the final result (high quality content + polish = high cost). I’m still not sure what’s the purpose, it doesn’t seem to be a marketing piece, although the underlying message is clear between the lines: if you liked pipes, you’ll like retool. Or something along those lines.
Author of the story here! I've been a writer and journalist for nearly 30 years, and with the giant shift of advertising from traditional and then web publications to apps, social networks, and product sites (Amazon makes a fortune from ads), the number of places that publish long-form features has shrunk quite a bit. The ones that remain often have a tech billionaire or heir helping to fund them or they're the last survivors.
I can't speak for Retool, of course, but it was pitched to me by my fabulous editor there as the company wanting to show the great visual editing tools of the past and make sure they weren't forgotten. They had already run a wildly fascinating story about Visual Basic, something I never thought I would be interested in, and it was riveting. There was no marketing involvement in the story at all. There's an interesting line where producing compelling editorial work shines a nice light on what you're doing as a company? Stripe published Increment for years, both online and in a very lovely print edition, and I wrote (for the same editor) for years for them until that no longer served Stripe's particular purpose for it. But it was all essentially independent editorial with nothing to do with Stripe's core mission.
On the production side, I didn't see much of how it would look until it went live, and I think it tells an incredibly robust story—it's a great parallel to my reporting in a way that I've rarely seen in my whole career?!
Everytime I see Yahoo Pipes I remember the video [1] from Tom Scott about programming. ( The video title view count is updated in as close to real time as possible )
A lot of what I thought were good, accessible, productive, entry to middle level technology ( HyperCards. Yahoo Pipes. Flash, Delphi etc ) never gained enough market. They were all the sort of middle ground good enough options. Instead technology is often filtered from the top end extreme. So everyone tries to use what Google, Facebook, Amazon, Microsoft are using. Because those tools are deemed as what Professional should use. The only rare exception that is powering most of the word is Microsoft Excel.
I used Pipes to get a feed of specific categories of used video games and game systems from Craigslist in college and beyond. I got a lot of stuff that way, and visited many parts of the cities I lived in that I otherwise wouldn't have.
For resale or collection? I used to hit up our local vintage store (next to my high school) and pick up all the donated NESs and games to resell on eBay (probably back to the people who's parents donated that old useless junk...)
I loved Yahoo Pipes. I had so many pipes set up to filter RSS feeds and Podcasts down to just the types of content I cared about. Insanely useful, I miss it so much.
A big example, I listen to a podcast where they have standard episodes and episodes where they play D&D and I just don’t care about the D&D ones so it was really nice to be able to filter them out.
I use Handy Reading on f-droid/android and hide various things from feeds based on their title/category. Wonder if it could pass on a filtered feed to a podcast app somehow.
I feel like theres a potential for B2B solutions in this space. Especially for workers who want to bootstrap something quick, avoid something like Teams or other behemoths, create something for a one time survey, campaign, shorter process etc.
So much was lost transitioning from web 2.0 to whatever walled dystopian garbage we have now.
The concept of pipes are way back from the Unix days, and the idea of this modularity and later building stuff from a flow based interface like this somehow never translated into anything on the frontend side.
I've been met with simple specs that had to be built with custom web solutions a few times now where i've almost started building a generalised model like this that could scale because most apps could be defined like this, as entities related with filters between them in sequences related to each other and web views where people could interact.
The closest i found back then was something like Airtable/Zapier combinations that was extremely limited.
These days there's stuff like Make.com (formerly Integromat), IFTTT, Zapier, etc. to do some low code piping. Airtable also got automation and pipeline support.
I was the organizer of the first pipesCamp[1], in a sleepy town in india. What were we thinking! Ha. Fun times. We did get coverage. It truly was a powerful product, and I hat-tip all the wonderful folks at Yahoo involved.
Does the web need something like this today and in what form? All the replacements that are offered pale in comparison to excitement it was to use Pipes 15 years ago. Are we simply past that point?
I skipped to chapter 9 in the article ("Clogged"), and it looked like Pipes failed because it didn't have a large enough team or a well-defined mission. As a result they couldn't offer a super robust product that would lure in enterprise users. "You could not purchase some number of guaranteed-to-work Pipes calls per month" is the quote from the article.
The reason I think that interesting is because that's the model these days for everything from AI tokens to Monday.com seats. It makes me feel like Pipes was before its time.
That said I've been collecting different "business glue" products that are similar to Pipes. To me, like you say, they aren't as interesting, exciting and intuitive as Pipes was, but maybe it just takes a little more digging. I tried to focus on open source tools but some aren't.
Big Node-RED fan here. For me it has the perfect combination between visual flow based programming[0] and textual coding. It makes a great effort in making visual programming incredible efficient.
Node-RED does great job in implementing the ideas of FBP in visual manner.
There are so many quality of life improvements I could make for myself if only Pipes was still around. Nothing I've encountered since has ever matched or beat it.
The best part of Pipes was that no one blocked it or it was difficult to block. I used it to scrape data from web pages that I served up via Android widget apps.
If you need only that, it may be easiest to write a few lines of python/js and run it yourself or through some serverless offering. If you have more workflows to do, n8n.io may be useful.
I'd fire-up Linqpad, load-in a NuGet package for RSS feed parsing, then do something like `SomeRssLib.LoadFrom( "https://last.fm/yourfeed.rss" ).Select( async e => httpClient.GetAsync( "https://google.com/search?q=" + Uri.EscapeDataString( e["track_title"] ) ) ).StringJoin( resultHtml => "some quick-and-dirty-XML-string-concat-here" ).SomePostfixApplyExt( result => File.WriteAllText( "output.rss", result )`.
I used this service quite a bit in combination with drupal 5, shortly before it shut down. So 1.5 decades ago you could build quite large systems just by clicking stuff together.
When Pipes died I saw all the comments and wondered why no one built a successor project. When after a few years that still hadn't happened, and after an idea on how to get it to work had popped into my head, I built it myself: https://github.com/pipes-digital/pipes is the source code and https://www.pipes.digital/ a hosted version.
To call it a successor project is maybe arrogant and misleading - it can not do half of what the Yahoo project did, and is sorely missing time and work from my side to make it into what it should be. But if you are nostalgic for the concept and especially if you just want to filter or combine a few RSS feeds, or create one for a HTML page that is missing one, it's worth a try I still think. I personally use it regularly to get specific YouTube videos from a few channels that have many other videos I do not care about.
The article mentions one of the big problems for this kind of project: The APIs are missing or not open anymore. For example, a comment here mentions doing a google search for a song title and putting this in a feed - but you can't realistically parse Google. So you have to use the API, which I didn't know existed until looking it up now, and calculate the price tag attached to it very carefully. Maybe it would be a realistic integration to provide with a different search engine. Or a bigger project should run of the local search engines for the pipes block.
The article is also right in that there is a lot of influence of Pipes in different projects. I was always wondering why that influence was often so little - to react to output on a page in a wizard for example, instead of having blocks. I think I understand that a little bit better now - it is really a strange abstraction level. You are working with a flow of structured data, without necessarily having the correct data structure for that, and then that needs to be transformed somehow. The idea that you might have JSON on a page somewhere you want to extract and manipulate, but then that data is inside of your outer feed format - it feels wrong to confront the user with these details, it makes sure the target groups is developers, but it also does not seem like something that can be reasonably universally solved. But maybe I'm wrong here, and this is more a problem with my implementation (where RSS is piped from block to block) than with the concept itself.
The idea of a real Pipes 2 is very exciting to me though. Give it some initial funding, built all of the modern APIs into it that can be added (and in doubt just give users the option to use their own API keys) - especially things like ChatGPT, after all those LLMs are great in converting structured data - solve the data format question and crucially keep the interface block based (but maybe built with something like Flutter, so it can be better). Could work, not to win the world, but as something that can justify its existence next to ifttt and zapier. At least I regularly get the impression that the concept still fascinates people a lot, though their specific use case is very often very specialized and then often enough not something my small pipes project can cover. But if it were broader...
> it feels wrong to confront the user with these details, it makes sure the target groups is developers
That is, IMHO, not a real problem at all. Definitely not what people make it to be.
Developers aren't alien species. They aren't a special class of people, bred and trained to commune with the machine. In general, anyone could in theory become one. For this particular case - anyone can easily become one: that is, through search and thinking figure out enough to use Pipes-like software to solve some need, improve their lives a bit.
Software like this is within the range of tips&tricks people exchange on-line, or from their friends. Idiosyncratic, Ad-hoc, low complexity, high-value. The random non-tech person who's e.g. carefully curating a Google Sheet of board games they play with friends, is driven and able enough to add some Pipes-like automation to it. They've picked up how to do conditional formatting and link images from a tutorial, they can pick up how to Pipes-in ratings out of BGG or something just as well.
Point being: total domination isn't the only measure of success. Will a Pipes-like product be usable for absolutely everyone? Not at all. But people with a need and minimum ability to use a computing device, or even a minimum willingness to learn operating a simple tool, is still a huge addressable market - easily an order of magnitude larger than "developers".
If anything, IFTTT and Zapier are proofs there is a market for this. The real problem is that everything these days is extremely locked down, everyone tries to control everything within their reach. As you note, there's hardly any useful API left anymore. Companies like Zapier succeed, because they sign contracts to integrate other services. They become a service aggregator, leaving user with no agency - whereas Pipes were all about users controlling their own automation. That part is hard to solve.
I never used Yahoo Pipes. Was it primarily for automating for manipulating stuff grabbed from the web. Or could you also usefully use it to wrangle/ETL local data files and databases?
These days there's an abundance of public APIs that serve up various public information. What if you took a bunch of those, wrapped them up in a single typed SDK in TS or Python (using their existing SDKs if they exist), under a consistent devex, and just made it really easy to playfully string together different datasets
For anyone interested in visual programming, I would recommend having a look at Flow Based Programming[0] better known for being the inspiration for Unix pipes, which in turn were name giver for pipes in yahoo pipes.
This brings back soo many old memories of mine @ Yahoo! . We had soo many tooling - YTS, YCS benchmarks, Vespa, SherpaDb, Pipes, shimproxy the list goes on. Its unfortunate leadership didn't have the foresight to saasify them in a way AWS did. Tooling at Yahoo!! was soo ahead of its time.
I 'member it used to be a thing to blogspam AdSense with Y! Pipes via RSS subscriptions to source sites → translate to non-English language → translate back to English → auto-post to AdSense-encumbered Wordpress blog.
Congrats to the authors for taking a potentially great article and wrecking it with insane distractions and useless web BS. Pipes was an amazing piece of software and deserves better than that.
Web 2.0 times were great. It was just taken as a given that the future was open standards, interoperable web services, and creative tools like Pipes that let users and developers easily leverage them. Social media and the drive to monetize users at all costs destroyed that potential.
Yeah, I don't know why this hasn't become a thing. There seems to be so much "Latent value" locked up in the semistructured data (in HTML format but not in machine readable format) that could be unlocked via crowdsourcing with the right tool. That's what I want to build. I just think there's so much value in harnessing that. The interoperability you can enable of bringing data out of silos, and mashing up actions and values from different contexts...I mean, we need existing web content to be part of web authoring in a first class way.
This is the second installment[1] in what we hope will be a regular series telling the stories of novel programming environments that had a lot of influence on developers.
Creating these is a labor of love—we get to talk to the original teams who built products that were formational for us. We have an incredible in-house creative team that finds a way to sneak these fun mini-sites in amongst their other work.
For me, the most exciting part of working on a product like Retool is that it's an evolution of lot of ideas from the past. We're always looking at prior art to spark ideas and challenge our assumptions as we're building. There are many great ideas in old computing papers and products that were either before their time, or lost when there were major shifts in the industry. Pipes was a touchpoint for us as we built out one of our newer products[2].
We have lots of ideas for future installments in this series, but if you have any programming environments that were near and dear to you heart, let me know!
[1] Our first deep-dive was on Visual Basic. I'm very biased (I wrote it!), but if you like Glenn's Pipes piece, you'll probably enjoy this one too. :) https://retool.com/visual-basic
[2] Retool Workflows: https://retool.com/workflows