The thing that this demo shows in minutes could've been typed up in text in seconds. It is not a productive way to do things. There's a reason that these systems never catched on, they are unnecessary.
You may say, but "non-programmers" will use it! No, they won't. Designers will use real design tools to create (non-functional) visual designs. Programmers will bring those visual designs to functionality. That procedure works. It'll keep working. These systems are diversions, not improvements. Worthy of investigation, but not practical.
Yo - I did the Airbnb project linked ^^^ which I believe was the first of this wave of deep learning-powered sketch->UI projects (though standing on the shoulders of decades of R&D projects)
Our take was that we really do design on paper or whiteboard first & foremost, which is why our project emphasized the webcam + sharpie thing rather than drawing in-browser etc.
Here's a related thing I wrote about the need for design tools to design the real thing, rather than facsimiles of the thing: https://jon.gold/2017/08/dragging-rectangles/ - so so so much process waste is because developers have to re-implement static pictures of designs.
In our case, we didn't get buy-in to keep developing the project, but I'm kinda jazzed that so many people are running with the idea
Your solution covers only the "semantic" part. Just look at the data. It's basically a simple component tree. It would be more efficient to just type it up. It also would be more efficient to just drag rectangles from a toolshelf instead of defining the type of the rectangle by drawing extra hints.
As for the visual design, that's where you use a design tool like Illustrator or Photoshop. Typing that up (e.g. in CSS) is surely a pain, but sketching it all up is out of the question. I certainly do see room for improvement in the workflow here, but a sketchy interface isn't helping.
You have to question a lot of assumptions here, but also consider how designers are most efficient with the tools they already know and have used for years. Don't mistake something that you want to create for something that users will actually want to use.
Hey ! I'm a big fan. Obviously this project as it is constructed currently isn't much. But the idea that machines can learn the code behind what artists draw is especially intriguing. I also think we'll get there is due time, with the great work being done on GANS and better scene understanding algorithms coming out. I was inspired by your team's idea, even won a Hackathon with this idea ! Thanks for your contribution, I'm a big fan
Journey back in time with me to 1963, when that Sketchpad software to which I linked was unveiled. The same criticisms apply:
"The things this demo shows could have been typed up in text in seconds. Designers will use real design tools to create (rough) visual designs. Engineers will bring those visual designs to blueprints."
And yet half a century later, CAD is firmly in the domain of visual designers, where it seems so obvious that you would have to be crazy to think people would be designing in code. But hindsight is 20/20!
The way forward to visual programming might not be super clear, but we'll get there. If you don't think text-based REPL-style programming is limiting, I encourage you to check out Bret Victor's explorations of abstraction and direct manipulation. http://worrydream.com/
What a poor comparison. Have you actually used any CAD software? None of it works like Sketchpad. Design software in general doesn't use shape recognition, that's a pointless gimmick.
> The way forward to visual programming might not be super clear, but we'll get there.
This isn't even visual programming, nor is it a step in the right direction. My text editor has all kinds of visual tools. The data I edit however is textual, which has a lot of benefits.
> If you don't think text-based REPL-style programming is limiting...
I don't think REPLs are very useful for programming either.
> ...I encourage you to check out Bret Victor's explorations of abstraction and direct manipulation.
I'm aware of this stuff, it looks nice, but I don't think you need an entire visual programming language to get that benefit. If I need visualization, there are lots of tools to use.
Airplanes were once worthy of investigation but practical. As were automobiles. Generally, the fact that something works isn't a compelling argument that something else won't succeed.
I do aggree that it's pretty high bar in this case though - it's changing the flow, not just improving it. So it'd have to get very polished to be able to compete, which I just don't think it will.
> I do aggree that it's pretty high bar in this case though - it's changing the flow, not just improving it.
My whole point is that this is not an improvement, it's actually a worse way to enter a simple datastructure into the computer. It's even worse than using the already established UI paradigm of programs like Paint. Picking a tool and dragging out a box is faster, because you don't have to learn the visual language of how to draw these widgets.
It does look cool, because it makes the computer appear smart, but it's just not a good interface for actual use.
You may say, "but 'non-typesetters' will use it!" No, they won't. Authors will use real writing tools to create (non-legible) articles. Type-setters will bring those articles to print. That procedure works. It'll keep working. These systems are diversions, not improvements. Worthy of investigation, but not practical.
I worked on state of the art CASE software decades ago that would allegedly let users graphically model then generate their own systems without writing any code.
Yet looking around I see plenty of people still building systems by writing code.
Maybe one day the surf will come in for these ideas.
The thing that this demo shows in minutes could've been typed up in text in seconds. It is not a productive way to do things. There's a reason that these systems never catched on, they are unnecessary.
You may say, but "non-programmers" will use it! No, they won't. Designers will use real design tools to create (non-functional) visual designs. Programmers will bring those visual designs to functionality. That procedure works. It'll keep working. These systems are diversions, not improvements. Worthy of investigation, but not practical.