Hacker News new | past | comments | ask | show | jobs | submit login
A profitable, growing, useful, legal, well-loved... failure (apenwarr.ca)
436 points by rcfox on March 26, 2012 | hide | past | favorite | 90 comments



This is one of the most fantastic submissions to HN I've seen in a long while. It's relevant in many different ways; it's about a startup, it's about a minimum viable product, it's about untapped market potential, it's about economic success, it's about a jaw-dropping kludgety mother-of-all-hacks, and it offers insights about what matters, both to users in large corporations, and to the author.

And it's funny as hell.


Everything apenwarr (Avery Pennarun) writes is worth reading, be it code or essay. He's been writing much less since he's started working at Google, but there's a lot of history for you to read.

Of his open source stuff: bup is awesome, sshuttle is awesome, and wvdial is a lifesaver for those rare times you need it. I'm quite sure do/redo is awesome, although I can't provide 1st hand testimony.


Redo is great, I evaluated several make systems for our iOS C++ framework after I got too annoyed about the complexity of waf, and we ended up using redo.

Is redo awesome? Maybe. We have written a few "usability layers" on top of redo, so I can't evaluate it fully as it is.


redo as a concept is awesome. redo as an implementation just recently reached version 0.11. So I think your description of "great" above is more than generous enough at this phase :) Thanks for being patient enough to use it so early on.


All that technical horror is precisely why it's such a needed and profitable product. That's your barrier to entry.

The only happy ending I see here is if the founders learn to delegate, or work hard on finding the right people to sell this to. Not to a hacker -- to a businessman who would be genuinely happy to run a growing business, and not horrified by the technical kludges.

I always chuckle a little to myself when I see clearly talented programmers shying away from this kind of problem in favor of allegedly more interesting problems. I used to feel the same way. But it's mostly chasing an illusion. The real work of programming is always about dealing with the messiness of the real world.

If you find a problem that doesn't have any real-world nastiness, you can be sure it's either (1) irrelevant, or (2) already solved for free by countless other smart programmers, because it's fun.


This part really resonated with me.

So don't get me wrong. I like Access users. Access developers, in particular, are the anti-IT department, the rebels, the people who aren't willing to wait for the sysadmins to provision them a server, and they don't have to, because they can just share an Access file on the fileserver. IT departments hate them, which is how I know they're on to something. These are the kind of people I want to help. This is the sort of thing that's the reason I do the work that I do. No kidding.

While it feels a little weird hearing that talk in relation to a Microsoft product, I agree. Building and maintaining a successful (To whatever extent you would call such a niche market a "success") product that helps people solve their own problems instead of feeding the IT priesthood would be awesome.

And then when you talk about the implementation details. My smile turns to a horrified slack jaw expression.

I'd probably feel guilty about it, and I might even try to keep it running as long as I could, but I'd eventually have as much as I could take and give up.


True, it's a bit weird to think of a Microsoft product that way - but Access is a special case. Microsoft would dearly love to kill it and get everyone to migrate to MS SQL (and boy, do they push hard in that direction) but there are just too many satisfied users to let it die. So basically there's customer satisfaction in spite of their best efforts :)


> So basically there's customer satisfaction in spite of their best efforts :)

That's funny, because you're in the exact same situation. You sort of wish people would stop using your product so that you could move on to doing something else. But that isn't going to happen because people need what you're offering so badly that they search the web until they find your product. Weather you make that easy for them or not.

Then again a product called "Access" is probably the most likely of any to be a special case.


seems to be very similar to apple's strategy on HyperCard - which apple probably deemed to be too powerful for rapidly writing applications and thus pulled it from the market. maybe this is a common strategy among large IT companies?

I've also noticed the ms sql server management studio to be fairly unintuitive (adding backups as drives, etc.) and I'm wondering whether the big players in IT might realize that future profits may not come from better software, but from the consulting/maintenance services that accompanies the software (following IBM's tracks).

EDIT: looking at the success of HyperCard/Access, that would probably be a great open source project. taking HyperCards approach to the web.


The other player in the space is Filemaker Pro - a fair bit nicer to work with than Access, although still fairly horrible from an internals point of view (I think they've moved to using proper SQL and everything since I last used it though).


I used to work for a web developer who worked on a well known campground company's site. We had a process that built static html pages out of an old Filemaker database and then synced them to a cluster of Windows web servers. That process has been in place since 2000 and there's no rush to change it because, frankly, it works pretty well.

I believe it started that way because our original developer didn't know anything else.


I briefly had the pleasure of working on a similar thing: it sticks out in my memory because the stack that ran the web site was entirely obsolete.

OS: Mac OS 9; DB: Filemaker Pro; templates: Lasso; webserver 4D WebSTAR.

This was in 2011.


Absolutely. In fact, it was exactly this situation that got me my first break as a developer. Started working for a gas pipelining company (yes, urgh) as an 'administration assistant' and encountered a spreadsheet based tracking system that involved entering in the same data in six different places.

I threw together an Access database that everyone responded to very well- as you might imagine. The IT department ignored my repeated attempts to get them involved as it progressed from an Access DB to a .NET client front-end app with a MS SQL backend (that I managed to get from the company's external IT contractors).

Shortly before I left to get a "proper" job, they announced that the entire workflow system was moving to SAP, and that the work I'd done would be integrated into that system. That was five years ago... and apparently they're still using my system today.


    I mean, we are running Access in Wine in X11 on Linux in an isolated
    user account on our server slice that revision controls your Access
    database in git, and we're displaying it using VNC in your web browser
    in flash.
If you asked me to describe the glue-code hackery project I would want to work on least in the whole world, I would have come up with something like that. There are probably uglier projects around, but I don't have the creative masochism to think of them.


This is actually great. They had a bunch of problems to solve with limited interface options. They picked the simplest combination of options, found the solution worked and stuck with it. All too often I find myself looking at something and saying "ewww!" and writing my own code without exploring the other, less glamorous options.

This weekend I nearly slammed my head into the wall after spending half the day implementing and testing shared caching for my high volume distributed web scraper according to part 6 of the HTTP RFC (aside: I cannot believe how something so simple has been made so complicated)... only to remember I'd totally forgotten I could just stick a caching proxy server (e.g. nginx) in front of the scraper and have it do all the hard work. Later I remembered why I wanted to control caching, primarily to do with some future plans, but right now I could have gotten along just fine with a caching proxy server.

When I'm tired and not thinking straight I often dive straight into the next task without asking the right questions. I've been trying to get into the habit by creating tasks in PivotalTracker and looking over other stuff to remind myself of how much else there is to do so I'm forced to ask myself "is this really what I should be working on next?", but I still slip up even after YEARS of relearning this lesson. The annoying thing is I'm very good at planning and allocating tasks to a team, then managing their progress, but very inconsistent when I'm both the manager and developer.


I love the continuation even more.

    People can't possibly want that. But they need it.


Necessity is the mother of invention, indeed. But, the invention can be heinous.

My favorite encounter with such hacky tech was at an insurance broker's office. It was 3 years ago, and this company first of all still used 15" CRT screens (with burned-in images because they ran no screensavers). They opened up a remote Citrix session (which I only know because it took over a minute for the splash screen to disappear) which connected to a Windows 2000 server that displayed a vt100 terminal window that then connected to their insurance company mainframe. They couldn't use the mouse to navigate the interface, of course, only tab/alt-tab.

Somewhere there is disruption, but, really, where does one even begin? It's the technology equivalent of a court-appointed cleanout of a hoarders home.


You tell me. I work for an insurance company that bet the farm on Forté's 4GL technology (then bought by SUN and renamed Unified Development Server, and then killed).

http://en.wikipedia.org/wiki/Forte_4GL

so I'm stuck maintaining a few hundred applications which are hacks and workarounds on top of the obsolete enterprise software.

The pay is very good for my country but I'm looking for my way out.


An even worse project would involve automating a GUI-only application by moving the mouse in recorded patterns, for sure.


Fun fact: back in the day, Bingo Card Creator's "CMS" was a script which drove a mouse around the BCC swing applet, printed the card to a software PDF driver, opened it in Acrobat Reader, hit "Print Screen" on the keyboard, opened MS Paint, used dragging the mouse to hardcoded screen coordinates to crop the card, saved it as a GIF, and then put the GIF and PDF in subversion to be uploaded to the website later.

It worked well enough to justify learning Rails and Prawn to rewrite the whole thing server side, after which point I rm -rf'ed the directory that script was in and deleted the backup so that its vengeful shade would never trouble humanity again.


How much time did you lost over figuring out the recipe and scripting it?

Before I heard about configuration management, I had bash scripts to configure my server from clean state. I tried it locally hundreds of times to get every command right.


$150 for the script from somebody who did that thing for a living (apparently banks do this sort of hinge with regularity) and probably five hours lost to babysitting it and manually restarting when it hiccuped.


One of my daily tasks is to ssh into a remote Windows server, port forward the RDP port, connect to the local forward using remote desktop, close any dialogs about crashed drivers, open a java application that displays a KVM terminal connected to a serial port, and through the KVM control two separate devices with a series of magic key-presses to log into two accounts at the start of each day.

I automated this with Sikuli (http://sikuli.org/); the ninja feeling while watching it run was awesome (now it runs before I come into the office).

Sikuli made the job perversely fun, and the entire script is less than 100 lines of python. Highly recommended for any sort of GUI automation (though watch the dependency on Java 1.6; 1.7 doesn't work).


How about doing just that, to automate entering a password in Outlook so that emails can be collected from Exchange (which would require the mail user's password to be entered if Exchange was restarted) with Outlook running in a Windows VM on a Linux box, and a custom app processing the emails and POSTing them to an API on the Linux host, where they were processed to extract structured data out of attachments?

That wasn't a fragile system AT ALL.


That's a funny coincidence. I worked at one of the author's earlier startups and wound up working on a project doing exactly this.


When looking at silent install files for ANSYS, I discovered that the install file simply records the keystrokes the user makes when a setup is recorded. That means that it also records any typos and backspaces. I must have really had trouble typing that license server name!


Like recording a macro in Excel? It is, I admit, uncomfortable.



Nah, there are tools to script that!


Okay, but isn't the actual opportunity to write a compiler that would transform Access apps into web apps (Rails or whatever)?

That would be very high level and should interest many (very) talented hackers; and you could either licence it -- or sell each transformation for a very hefty price (the price of writing it again).


Yeah, but by definition these customers don't know anything about Rails. And the way apenwarr describes it, these little Access projects are often done without anybody's approval, so getting an outside vendor involved will never happen.

What the customer needs is exactly what apenwarr provided: they continue to update their Access db as they always have, but, BAM, it's also on the web now.


Yes, yes, it would.

But it would take a lot of developers a lot of time to build a reliable product. And nobody can afford that.


Am I the only one who was surprised that of all the pieces in that sentence, the author italicized "web browser" and "flash?" Those actually seem like the least hacky pieces to me, but I guess he highlighted those because they're at the end?


It was just a "rising emphasis" sort of thing. I hate all technology about equally.


Nice article.

I had a similar experience with my first company in college selling software to corporate customers who previously used excel to track everything.

Theres millions of opportunities like that to optimize a terribly inefficient corporate process with software.

But the problem is always

a) the problem set is rarely easily reproduced, meaning you're not really selling a product, but instead selling a product + consulting - since every process is slightly unique

and

b) you're dealing with terribly inefficient systems and the type people who would put together a terribly inefficient system (like you said, not necessarily meaning they are bad people but likely are not a good customer)

I still believe its the most straightforward way to make money in software outside of getting a job at Google/facebook. But its certainly not fun and not a traditional "high growth" startup.


A fantastic read - thanks. Quite simply I'd hire someone to do the dirty work, pay them a decent salary and split the remainder of the income between you and your co-founder.


Sounds like a pragmatic solution. The product exists, people keep signing up, and maintenance is minimal. I can completely understand why the Pennarun dropped out, though, since I would loathe working in a career where my job was to fill deficiencies in a product I don't like. It sounds like the real problem is the web of hacks. Nobody likes hacky fixes--they can wear a person out.


It sounds like he'd love to do this, but is unable to (he and his cofounder might be the only people on the planet with the requisite skillset).


I wouldn't be so sure, I can think of a bunch of people I know in real life that could handle it.


I think his article may have been a desperate plea for such people to come out of the woodwork. I implore you to contact the author and refer those people.

I've actually had the pleasure of meeting both of them in real life and they're great people -- honest, smart, fun to talk to -- I imagine they would be great to work with.


Google for the following phrase to find the site: "Run unmodified Access applications, right inside a web browser."

(Edited to remove link to not affect Google juice)


Sounds to me like you're good at creating software small-businesses, but you're not built for the tedium of running them.

I find it's often hard to discuss the business of software because of a variety of preconceptions people in the field have so let's turn this into a restaurant. You wanted to start an amazing new french restaurant, but to get started initially you decided you're going to be more of a cafe where costs are lower and it's easier to attract customers. In that cafe you start offering a few sandwiches, and some baked goods. Some of your baked goods were vegan. Soon you were inundated with vegans seeking baked goods, and you basically turned your whole operation into a pretty successful self-sustainable single-store vegan bakery. This wasn't the french restaurant you wanted to start (and could have sold to a restauranteur as some do once they show promise), nor is it a true "startup" success which would have had you opening in many cities, perhaps franchising, etc.

Nevertheless, you were to able to create a business (as you've done before), direct it into a market where you're not just making a profit, you're solving a true need. I don't know if it would help your self-fulfillment (because it sounds like you're lamenting the lack of it), but that's the aspect I would concentrate on, and just be aware that it is more likely than not that you'll get bored or disgusted with what you've created and move on to the next part. Just work out an efficient way to do this "moving on" portion, and I think you'd be able to do something you love and something you're good at. And in a few more of these, perhaps you'll create that one company that'll be your baby, not just your creation. (The trick is of course to treat them all as your baby at first)


It is a beautiful illustration of how just making money is not worth it for some people.

Loved it, thanks for sharing.


How about trying to sell the company to http://www.red-gate.com, who IIRC built most of their business on making tools to deal with the awful parts of database systems?


Aren't Red Gate an SQL Server shop though?


They do all kinds of things these days including Exchange, .NET, Oracle, and other stuff. (Full Disclosure: I'm friends with some people working there.)


It sounds like the plight of one of my coworkers. He started on a team that had to know significant details about how dial up worked. This was great work when dial up was big. One aspect to dial up were fax calls, and nobody wanted to work those issues. He stepped up, learned fax, and earned quite a bit of notoriety due to his specific fax knowledge. Wrote a book, some RFCs, promotions, etc.

The only problem? He 10 years later he still sits around fax machines, listens to users who still use faxes (problem customers), and talks about one of the true dragging edges of technology. And it doesn't port well to other newer technologies.


Isn't he in a great position to disrupt, or at least provide information that other people need to disrupt?

People know the reasons why faxs suck. What they're not so sure about is why fax is still being used; what problems fax solves.


> why fax is still being used; what problems fax solves.

You give someone a telephone number, then you hear a noise and a document magically appears.

Sure, there's software solutions. People can email documents to an email address, and your computer can automatically send it to the printer. Unless your computer is off, or the app isn't running, or someone queued up a hundred pages in the printer, or the printer is out of ink, or the person sends you a file in an incomrehensible file format.

These issues are all solvable, but it's less effort to buy a fax machine and plug it into a phone line.


If I were you, I would hire a good entrepreneurial hacker who's looking for some autonomy and upside, give him/her 20% and a small salary, and leave me and my partner on the board.


Now if only there was a way to convert those Access databases into modern web apps based on MySQL and an HTML interface, that could really be a cash cow...


I think it could be done. The plugin itself is actually very clean code and doesn't need a lot of maintenance (well, someone has to update it to Access 2010, but that's probably not so bad). The server is crazy because it has Wine+Access+X11+VNC+Flash. Mostly the problem there is Access. But because the plugin has already done the insane work of exporting an Access database structure as text (and incidentally, re-importing it), all you need is a replacement server.

The actual form layout files (now that they're in text form) are pretty straightforward. Someone who really loves javascript + html5 could probably make this happen with a few months of work, at least for databases that don't get too fancy. (And the users are actually willing to modify their apps to make them work; slightly modifying Access apps is okay, rewriting them on a new platform is not.)

It would probably be much less gross that way, but I just don't have enough patience left to try it. I'd totally recommend it to a fresh hacker with some free time though :)


You just described "a couple decent developers + ruby on rails + custom development".

The interesting bit is trying to create the next Excel or Access. The most recent (aborted) attempt was DabbleDB (http://en.wikipedia.org/wiki/Dabble_DB).


Well, no, rewriting from scratch in Ruby is not, in any way, equivalent. If you're going to rewrite from scratch, by all means, Ruby seems to be the way to go. But that's way too expensive and risky. What people want is to take their current app and make it happen.

DabbleDB failed because it was trying to completely replicate Access, without being compatible with Access. And no, uploading your data is just not good enough. Access is not a crappy mysql; it's an IDE with a GUI editor and a programming language and a crappy mysql. DabbleDB was just a crappy mysql.

The huge business opportunity here is not to compete with Access. It's to let people leverage their existing huge - monstrously huge, seriously - investments in Access databases, adding just enough functionality to keep it alive.

Once you have the cash cow taken care of, then sure, see if you can add some features and replace Access entirely. But then Microsoft will probably not like you as much.


Your story sounds like a classic reason people sell small businesses: not excited about it anymore. Either figure out how to get excited again (perhaps by hiring people who like to do the parts you don't like), or sell the business to somebody who can be excited about it.


I feel the same way about cross-browser testing. Everybody knows that this is the worst part about web development. A number of startups already exist to solve it, but there's easily room for more solutions. This is a blindingly obvious startup idea.

But I get indigestion just thinking about it, because this means taking on the worst aspects of everyone else's job. There's no major win that I can offer for being clever, just for being more willing to focus on unpleasantness.

Or am I kidding myself? Is that focus on something unpleasant the very soul of entrepreneurship?

Sometimes you luck out, and something that everybody hates is something that you love. But there are some things which are probably universally unpleasant to even think about.


Be careful. I'd see this as a blindly obvious technical need, but not so much a market need. Do business decision makers know they want to pay for it?

I love Steve Blank's "hierarchy of needs" concept when evaluating business value for a new product. Look for the diagram in http://steveblank.com/2010/03/04/perfection-by-subtraction-t... .


Thanks for the link, that's a great way to evaluate ideas.

As for this specific problem - it's hard to say without doing surveys, but I'd say every webdev shop in the world does all 5 to some degree, unless they are developing for a corporate intranet with a controlled client base.

The spread of browser platforms right now is the largest it's ever been. There are many shops who have to fight IE6 on the rearguard while trying to be iPad-compatible in the vanguard.

The higher end shops do some formal or scripted testing, but my intuition is that what the average person really want is to do a quick check by hand in one browser, and somehow magically know that it works on all the other ones.


Why not try and sell the company?

There are people who would love a revenue stream and don't mind boring work. People who would rather be in a position to work in Thailand, or India, for example...


    > Customers need this so badly that they're willing to pay a lot for it.

    > It's just about providing something people want and are willing to pay for.
Truth.

As for your problem, I'd recommend open-sourcing the code and stepping away if you really can't stand it anymore.


I shuddered when I saw the word Access Database. (bad memories of porting those evil things). I don't care how much money is in those things, I will leave it to someone else :) Much like these guys are trying to do.


That is quite amazing. I am writing this as someone who has just rewritten an Access piece of crap for a construction company. I charged them $15k for 4-5 months of part time work. It is not a rewrite of their original app, it is an order of magnitude better, but if they saw your git versioning WINE running frankenstein, i wonder if they would have chosen that. Now i am off to sell this software again now that i have been paid to write it once. I wonder if you could get a nice income stream by referring companies who want to upgrade their application slowly. Now I am living in the USA, if I lived in another country I could take that $15k and live for 1 year. Here it is hardly much at all. I would love to take US jobs for $15k and finish them in 2-3 months. Low end US freelancer rates, but do a high quality job.


Sounds like a valuable contribution that increases the value of MS Access for its users by allowing it to become accessible for web-based applications. I'm searching for the company website that he did not want to mention in the article (if that says anything).

Great ideas alone do not make great entrepreneurs or profitable companies. At the end of the day, the way a person thinks is where they will eventually return. An employee can have a great idea and make an attempt at being an "entrepreneur" for a while only in the end to return to being an employee. Entrepreneurs think differently about how money is generated from someone with the mind of an employee- thinking on different sides of the quadrant. It takes a different set of skills and a different mindset. "Safe & secure" can be alluring.


What would I do? Delegate the care and feeding of this beast to two hires and get a marketing guy to continue the push for new business/deals. In short: hire up until I had outsourced all of the objectionable aspects of the technology to someone else. YMMV.


How about value adds and referrals? Seems your company has no google juice. I did a few google searches and didn't find your because the term 'web hosting' is so SEM'ed to death. In order to compensate you could offer referral bonuses. It seems a lot of people would want your technology but talk to regular hosting companies. I wonder if you could get money that way. As for value adds, what about regular backups? People don't know about your git backed system, you could charge them for regular offsite backups. Any congrats on a good business.


Ha! Awesome! There is nothing more satisfying than doing something "successful", hating it and leaving it behind (well, if you really don't like it). It's like a beautiful, perfect counterpoint to all the success-oriented, soul-sucking, BORING corporate-speak we hear so much of.

Of course, one could adopt a more practical self-improvement oriented outlook and say, well, someone with a talent for figuring out what people need (or stumbling on it) would surely come up with something even better that you'd like doing, but hey, just enjoy your liberation.


Wow. That was the longest "do ONLY what you love, here's why" article ever.


The Moby Dick of blog posts. Boring, but that's that point.


I can think of no way in which this post is like Moby Dick.


Assuming it makes enough money... why not just hire someone else to maintain it? As long as you're very careful about your documentation, even a pile of hacks like that could be interesting to work on.


"They are up to something" Never saw it from that perspective (working in a company running at least one too much Access app). Got me to rethink my quite fundamental possition on that. Yes, that actaully is a huge business opportunity, one you could eexpand upon. And yes, I can completly understand why you don't want it anymore. My answer to the last question: Do everything you can to get that pharma deal and maybe another one and then sell it!


If you're going to do something you don't want to do, just get a job. It's less risky, and usually better paid than a startup.

Starting a company is about working on something you're passionate about. I couldn't imagine starting something that wasn't really meaningful to me.


"Most small business owners don't own their own business, they own their own job." (paraphrasing The E-Myth Revisited)

My POV is this person needs to start delegating as much of the business as possible and manage growing the business to support that.


Why don't they just keep increasing the price to insane levels and see how deep the rabbit hole goes?



remove the link please. He started off mentioning he didn't want the google juice messed up. If his blog gets a lot of hits, think about how much more HN gets.


HN links are nofollow, at least.


Did anyone figure out what is the company behind it?


Dude, just charge more. like 5x 10x more.


This is an incredible piece. Brutally honest, and I can absolutely see myself in the author's shoes.


This is the Moby Dick of blog posts. Incredibly boring, but that's the point: he's illustrating how boring his startup was, so he quit, just as HM illustrated the monotony of whaling by making Moby Dick beautifully written but unreadable.

Still, I hated reading it, and I didn't mind MD.


> "That is, C is little more than a portable assembly language."

I know it's not really the point, but this is a pet peeve. No, C is nothing like assembly language. I seriously question whether anyone who repeats that frequently has actually done any assembly programming. It's an order of magnitude more tedious than writing C.


"Back in the day", pre-RISC, with fancy macro assemblers (like Turbo Assembler), programming in assembly was actually not that bad. C is obviously better, but not actually as much better as people seem to think.

Nowadays life would be a lot worse in assembler, because CPUs are designed to be programmed mostly by compilers, not humans.

(Also, the ways in which C is better than assembly roughly correspond to the ways in which our query language would be better than SQL. Like, say, variable scoping.)


One of the best assembler IDE was Turbo Pascal 6.0. One would write begin/end, stick asm block in the middle and the rest will be in assembly. Full debugger support too, it was very convenient.


You probably didn't mean any harm, but these "well, actually" comments are doing serious damage to HN usefulness and atmosphere.


right. We need to stay with the grain and not express our differing opinions.


If the opinion in question is a technical quibble that does nothing to further the discussion at hand, then we're in agreement.


It's a relative thing -- wouldn't you agree that C is closer to assembly than it is to, e.g., lisp?


Failure is rambling on and on, tangentializing without purpose or value, puking endless stream ofwords mostly devoid of meaning.

Learn to get to the point, man!


This is not documentation. It is a narrative. A well written one at that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: