I personally build workflow software. My current employer sells vertical systems to small government and nonprofits, and we built a dynamic workflow engine (along with a dynamic ORM layer and a dynamic presentation layer). This allows us to only hire programmers to build the underlying engine, and hire only moderately technical people as project managers and deployment specialists to actually customize the system for existing clients.
The programmers get to focus on building a pretty interesting code base, and the project teams get to focus squarely on the customer. This turns out to be a very effective division of labor. We're also in the process of rolling out our tools to VARs and clients.
Summary of lessons learned so far: a workflow engine can be a big win, but you really need to understand what you're going to use it for and how it's going to integrate with the rest of your tools to define requirements for one. For us, it slots right in at the business logic layer, and we just build UI on top of it as needed. A standalone workflow engine may or may not do what you want.
I think what you're describing may be in a different class than the products the OP is castigating. The latter are not vertical systems. They're middleware that the client expects to integrate with existing systems (paying huge consulting bills in the process, though they may not know it yet). And once that integration is done, then magic happens and the productivity boost is supposed to kick in.
What you're describing sounds more like an in-house framework that is used to boost your company's efficiency in delivering systems to customers. That's different. When you're doing a lot of similar projects, abstracting out the common parts can be a big win. I've seen this succeed, up to a point.
I say "up to a point" because I've noticed a pattern. Sooner or later the vendor decides that their in-house framework is so great that they've really solved the problem of enterprise software development in general. Now they "just" need to package it up as a platform.
The problem with this is that the value is not in the framework, it's in the team that's built up around the framework. The framework may make you more efficient ("you" meaning the programmers, managers, analysts who know it, identify with it, and belong to an organization based on it), but it doesn't follow that it will make others more efficient, because the framework abstracted away from you is not the same thing at all. The overhead for anybody else to use it is considerably greater and may (and usually does) exceed the benefit.
Vendors do this because they want to shift away from being a consulting company (where the product is you, and the framework is your secret weapon) to being a product company (where the product is the framework and the "you" part is vastly diminished, ideally to zero). Of course even in the success stories (SAP?) the consulting role is still huge. And I can't shake the impression that the successes are mainly in sales and marketing rather than technical substance.
Anyway, these are just things I've observed. I don't claim they apply to your situation.
> The latter are not vertical systems. They're middleware that the client expects to integrate with existing systems (paying huge consulting bills in the process, though they may not know it yet).
Agreed. Tools to abstract workflow can be useful. My impression is that whether middleware tools to abstract workflow can be useful depends largely on how well they integrate with the overall toolset being used. Ours works because it's fully embedded in the tools. Standalone solutions may or may not work. I'm hearing good early returns from a friend using Microsoft WF in an all-Microsoft shop, but (1) it's still too soon to tell whether it's really a success, and (2) solving the all-MS solution is a much smaller problem than a universal workflow engine or even a general-purpose Java workflow engine.
> Sooner or later the vendor decides that their in-house framework is so great that they've really solved the problem of enterprise software development in general. Now they "just" need to package it up as a platform.
Actually, we're exactly at the point of trying to figure out whether our tools useful to VARs and client-level users, or whether it's really only useful to us. And, yes, figuring out whether we can increase product revenues as a percentage of our overall revenue mix.
> Of course even in the success stories (SAP?) the consulting role is still huge.
Unequivocally. However, if we succeed in working with VARs and technically savvy clients, we would be offloading most of the consulting work to them by building expertise outside our organization.
I won't pretend that I think it's a lock to succeed at a problem a lot of people have failed at, but since we're not really risking our core business model to try, it seems like a reasonable gamble as a way to grow the business faster.
There is a long and ignoble tradition in the software industry of vendors who prey on the ignorance of managers and executives to sell expensive tools that allegedly eliminate or reduce the need for programming. Executives fall for it because they know that programming is hard and expensive but not why it is. The prospect of a tool that will eliminate their dependency on programmers is very seductive. What programmers produce is usually late and over budget and works like crap. They themselves tend to be weird, geeky, and erratic. Who wouldn't want to be rid of them? Meanwhile, the vendors with the magic programmer-replacement tool are wearing nice suits and know all about the problem. They are full of stories about how great it would be if business people - normal people - could "just" make the computer do what it should. "Defining business processes" shouldn't have to require programming, after all. Normal people know about business processes.
This is a sales job made in heaven. At no point in the process - the sales process - is it required for the tool to actually work. All they need is enough of a demo to convince the stakeholders - who already want to believe - that it will take away their pain.
At this point the well-chosen trivial visual example inevitably makes its triumphant appearance. Who can look at the diagram the post cited (http://secretgeek.net/image/NXTCode_small.jpg) and take anything associated with it seriously? Yet people do. Important people.
In this dysfunctional dynamic, a new crop of tools that will work "this time" makes a predictable appearance every 5-10 years. One reason this persists is that it's in no one's interest to publicize the failures. When such tools are purchased, rolled out, forced on organizations, and result in an even greater clusterf--k than what was there before, who's going to say so out loud? Obviously neither the vendors nor the executives. And only programmers listen to programmers.
There is a long and ignoble tradition in the software industry of programmers who prey on the ignorance of managers and executives to convince them in uselessness of productivity enhancing technologies so that the programmers can enjoy building their own productivity enhancing technologies that are subjectively (and very rarely objectively) superior to alternatives.
At least we agree on the ignorance of managers and executives :)
It is difficult to pin down your argument, since it isn't clear if you are talking in anecdotes about specific instances of product failures or if you are describing the entire marketplace. Assuming your claim is about the latter, you are missing the context of the general trends: yes, most of the "new crop of tools" that come out every few years fail. However that is in line with most new products -- over 90% of new software products that are released every year fail, so it shouldn't be surprising that tool products fail at a similar rate.
However, at what point are you throwing baby out with bathwater? Just because most of the workflow products are crap, doesn't mean there aren't a few innovative ones that truly improve productivity and deserve to be used.
Finally, you are confusing programming and writing workflows or designing business processes. Very much like BNF grammars are an effective way of describing parsers, flowcharts (or workflows) are an effective way of describing invariant, sequential operations. Neither are an effective general purpose programming language.
At least we agree on the ignorance of managers and executives :)
Heh.
programmers who prey on the ignorance of managers and executives to convince them in uselessness of productivity enhancing technologies so that the programmers can enjoy building their own productivity enhancing technologies that are subjectively (and very rarely objectively) superior to alternatives
I'm not sure I agree with that. It's true that programmers often want to build something instead of buy off-the-shelf solutions. But in my observation, they're right as often as wrong. Then again, it's true many programmers prefer their own intellectual gratification above all else.
You're quite right that I haven't surveyed all recent workflow systems and my generalization shouldn't be applied to all of them. I'm talking about a pattern I've observed and am highly skeptical of.
A very shallow argument. The author doesn't take into account realities of most companies out in the marketplace. I'm exaggerating of course, but not every company is like Google which can afford to hire Ph.Ds to write high performance code for distributed systems. In a typical mid-market business (500-5k employees), if the company can afford it, the IT dept. is staffed with "Bob"s from accounting who started programming using scripts in Excel and then moved on to basic Java. These aren't the people who know about lambda calculus, Petri nets or even nuances of ORM. So for them, a workflow product that works well in a distributed environment, e.g. supports a range of databases, web service integration, long running processes, asynchronous invocation is a huge time saver and productivity boost.
You're assuming your conclusion. Obviously, a product that dramatically simplified programming to the point where anyone (or "Bob") could do most of what they need, would be valuable. The question is whether these products really do that, or merely appear to.
>Obviously, a product that dramatically simplified programming to the point where anyone (or "Bob") could do most of what they need, would be valuable.
You are attacking a strawman :) My claim pointed out specific types of problems that are solved by workflow software, and solved in substance, not in form.
>supports a range of databases, web service integration, long running processes, asynchronous invocation
those are specific examples of problems that a workflow product would address and consequently a programmer wouldn't need to spend time on building from scratch.
>>The class of programs encompassing those things is enormous.
Heh. Welcome to the real world.
If you'd like to call them buzzwords, I don't care, it is your prerogative. They are what they are. People who have done workflow software know what they mean. If you were to spend some time on Google you'd get a sense of the meaning too.
A long-running web 2.0 startup for business process management: Thingamy http://www.thingamy.com/ by Sig Rinde http://thingamy.typepad.com/ . I'm not yet sure if this approach makes sense, the blog sounds intriguing but vaporware-y.
We experimented with a workflow engine at my last job (jBPM), and it was a colossal waste of time. I have a colleague who uses it very effectively at another job.
I think it's important to use workflow engines only in situations where it makes a lot of sense. In our case, we were using it to control the order in which network components were provisioned, and it was a lot more difficult than just implementing it in code. In his case, they're modeling complex sign-off business processes at financial institutions.
So I think it's most useful in modeling business processes in software in the rare cases where you'd want to do such a thing, especially if those processes can be dynamic (EDIT: and need to be validated by non-programmer types). But it's not a suitable replacement for "if statements" when good old code will do. We threw out flowcharts years ago for a reason.
It helps to think of a workflow as a domain specific language, then it becomes natural to ask yourself "which part of the system that I am building should be described using this language?". By "natural" I mean in the same way that if you were to build a parser, you'd use BNF as a domain specific language to describe it.
1. "Changing code is difficult. Our organization has all these 'change management' processes in place. If we can just make the change in the workflow engine, we can deploy it straight to production."
Perhaps a workflow engine keeps you from shooting yourself in the foot by constraining kinds of things you can modify in your system. However, change control processes also protect your organization from all sorts of things: fraud, business rule errors, etc. I once worked for an organization where project managers wanted to put business logic in the database because they were allowed to change it at will! Just because changes don't fall within the scope of your change control processes doesn't mean that those changes are safe to do at will. This is the same fallacy that led people to believe that SOAP was great because it could get through firewalls. The only reason admins would punch large HTTP holes in their firewalls was because they believed the risk of serving up web pages was relatively low, monitored, and contained.
2. "Coding is done by expensive software developers. Business uses can make these changes if we implement a workflow engine."
It's not syntax that keeps non-coders from becoming decent developers. It's the ability to think in a structured, logical way. And, for anything but the most trivial uses of a workflow engine, one must be able to think through business processes in a way that only software development trains you for. What are the holes in this process? What happens if an employee quits who has active workflow items? How are they re-assigned? For years, business rule vendors (iLog, Blaze, etc.) have been making similar claims. In practice, I have seen that business rule system users are only allowed to make configuration changes (15% discount on Tuesdays instead of 20%) rather than more dynamic changes to rules (which are just declarative code).
With all this said, I still like the concept of jBPM and other workflow engines. State machines are a powerful way of ensuring correctness, and I would hate to throw the baby out with the bathwater.
Confirmed. Most discussions I've encountered of applications "needing" workflow software, could be easily served by simple state machine. I was very confused for a while before I realized this.
The programmers get to focus on building a pretty interesting code base, and the project teams get to focus squarely on the customer. This turns out to be a very effective division of labor. We're also in the process of rolling out our tools to VARs and clients.
Summary of lessons learned so far: a workflow engine can be a big win, but you really need to understand what you're going to use it for and how it's going to integrate with the rest of your tools to define requirements for one. For us, it slots right in at the business logic layer, and we just build UI on top of it as needed. A standalone workflow engine may or may not do what you want.