It's silly to focus entirely on the fee aspect: the point of using Wealthfront is not because it's lower-cost, it's because you expect it to be better-performing from a total-return perspective. They may oversell themselves, and that's a valid criticism, but the OP fails to really analyze the raison d'etre of Wealthfront as a service. Comparing it to a single Vanguard ETF is not a proper comparison.

The Vanguard target retirement fund for 2035, for example, includes four underlying ETFs (US stock, international stock, US bonds, international bonds), whereas Wealthfront portfolios typically have more like six or seven asset classes (differentiating between developed and emerging international markets, and adding in natural resources and real estate). Left to my own devices, I don't have the time or inclination to do the research necessary to do that sort of additional asset diversification myself, determining the ideal allocation and then avoiding too much drift while not incurring too much tax. I also don't have the time to deal with tax loss harvesting, which might not matter for retirement accounts, but does make a difference for taxable accounts: you're likely incurring some taxation issues when it comes to portfolio rebalances, for example, since that necessarily involves asset sales. If you have $100k in wealthfront, you're paying $250/year in fees. If tax loss harvesting can only harvest $1k in losses during the year that offset (or at least avoid) $1k in capital gains, that still pays for the wealthfront fees by itself (if you assume 15% federal and 10% CA state tax on long-term capital gains). So, sure, the fees you pay wealthfront compound over time, but so does the money you pay in taxes. (You do eventually pay the taxes on that gain, of course, but the money you save now compounds over time, so you still come out significantly ahead).

So the question is: does additional asset class diversification plus tax optimization yield at least a 0.25% increase in your total return (net of taxes and fees) in an average year? I believe it does in my case, hence why I have my money with them. It's not because I'm some idiot taken in by slick marketing who can't do math and doesn't know about Vanguard ETFs: Wealthfront's core market is really people who can do math and understand what they're getting for their 0.25%.

Similarly, you can quibble over asset-based fees over fixed fees, but that also misses the point: as long as they make me more money than I pay them, I come out ahead. If I come out ahead, why would I be complaining? If I don't come out ahead, then there's still no point in complaining: just don't use the service. Capitalism at its finest.

Again, it's sad that the OP and most of the comments in this thread don't even attempt to tackle the real value proposition here. Just saying "Stick it in a Vanguard ETF and you'll pay less in fees" is not at all addressing whether or not the core value proposition is valid.

Why does increasing the number of asset classes guarantee a higher overall return? Is there any proof or intuition? Will increasing the number to 10 asset classes guarantee that you will consistently outperform VOO or VTI or VUG?

For a simplified model, check out Kelly gambling

https://en.wikipedia.org/wiki/Kelly_criterion

So with that logic, shouldn't Betterment with its 12 asset classes and 0.15% fees for >$100k invested be more optimal?

Re. this question, it is addressed in the Wealthfront FAQ: https://pages.wealthfront.com/faq/

> What’s to keep me from investing $10,000 with you and then mimicking trades for my $250,000 portfolio at a discount broker?

> Nothing. But that kind of sounds like a pain in the neck when we only charge you an annual advisory fee of 0.25% (on assets over $10,000) to take care of all the trades in your account, as well as the periodic rebalancing and daily tax-loss harvesting. That being said, you are welcome to copy anything we do if you would rather do it yourself.

The legit disagreement seems to be between WF's approach and the Bogleheads argument that you only need three funds. I lean toward the former but haven't been satisfied with any of the analysis around this honestly.

I personally have kind of moved away from TDD over the years, because of some of these reasons: namely, that if the tests match the structure of the code too closely, changes to the organization of that code are incredibly painful because of the work to be done in fixing the tests. I think the author's solution is a good one, though it still doesn't really solve the problem around what you do if you realize you got something wrong and need to refactor things.

Over the years I personally have moved to writing some of the integration tests first, basically defining the API and the contracts that I feel like are the least likely to change, then breaking things down into the pieces that I think are necessary, but only really filling in unit tests once I'm pretty confident that the structure is basically correct and won't require major refactorings in the near future (and often only for those pieces whose behavior is complicated enough that the integration tests are unlikely to catch all the potential bugs).

I think there sometimes needs to be a bit more honest discussion about things like: * When TDD isn't a good idea (say, when prototyping things, or when you don't yet know how you want to structure the system) * Which tests are the most valuable, and how to identify them * The different ways in which tests can provide value (in ensuring the system is designed for testability, in identifying bugs during early implementation, in providing a place to hang future regression tests, in enabling debugging of the system, in preventing regressions, etc.), what kinds of tests provide what value, and how to identify when they're no longer providing enough value to justify their continued maintenance * What to do when you have to do a major refactoring that kills hundreds of tests (i.e. how much is it worth it to rewrite those unit tests?) * That investment in testing is an ROI equation (as with everything), and how to evaluate the true value the tests are giving you against the true costs of writing and maintaining them * All the different failure modes of TDD (e.g. the unit tests work but the system as a whole is broken, mock hell, expensive refactorings, too many tiny pieces that make it hard to follow anything) and how to avoid them or minimize their cost

Sometimes it seems like the high level goals, i.e. shipping high-quality software that solves a user's problems, get lost in the dogma around how to meet those goals.

If you have the cash, spring for Gary Bernhardt's Destroy All Software screencasts. That $240 was the best money my employer ever spent on me. Trying to learn TDD on your own is asking for a lot of pain, and all you'll end up doing is reinventing the wheel.

There are a lot of subtle concepts Gary taught me that I'm still learning to master. You learn what to test, how to test it, at what level to test it, how to structure your workflow to accommodate it.

(I ask because there's no way I'm going to have time to watch/absorb all those things).

(Apologies in advance as I can't figure out how not to sound snarky here.)

Isn't that called "the design? And is there any meaningful way in which, if "test-driven design" fails if you don't already have the design, it's worth anything at all?

As a point of semantics: TDD generally stands for "test-driven development," not "test-driven design," though the article here does make the claim that TDD helps with design.

To reduce my personal philosophy to a near tautology: if you don't design the system to be testable, it's not going to be testable. TDD, to me, is really about designing for testability. Doing that, however, isn't easy: knowing what's testable and what's not requires a lot of practical experience which tends to be gained by writing a bunch of tests for things. In addition, the longer you wait to validate how testable your design actually is, the more likely it is that you got things wrong and will find it very painful to fix them. So when I talk about TDD myself, I'm really talking about "design for testability and validate testability early and often." If you don't have a clue how you want to build things, TDD isn't going to help.

If you take TDD to mean strictly test-first development . . . well, I only find that useful when I'm fixing bugs, where step 1 is always to write a regression test (if possible). Otherwise it just makes me miserable.

The other thing worth pointing out is that design for testability isn't always 100% aligned with other design concerns like performance, readability, or flexibility: you often have to make a tradeoff, and testability isn't always the right answer. I personally get really irked by the arguments some people make that "TDD always leads to good design; if you did TDD and the result isn't good, you're doing TDD wrong." Sure, plenty of people have no clue what they're doing and make a mess of things in the name of testability. (To be clear, I don't think the author here makes the mistake of begging the question: I liked the article because I think it honestly points out many of the types of mistakes people make and provides a reasonable approach to avoiding them.)

One point I'd like to draw out. If you don't have a clue how you want to build things, TDD isn't going to help.

This is exactly right. If you find yourself completely unable to articulate a test for something, you probably don't really know what it is you're trying to build. I think that's the greatest benefit to TDD: it forces you to stop typing and think.

Sounds crazy, but this is how I do everything I don't understand. And my second implementation is usually better than my first.

Unfortunately I find all too often that spike project finds its way into production for one reason or another. Now I only spike in Befunge.

I don’t buy this argument. How would you write tests to drive the development of a graphics demo, say rendering a Mandelbrot set? Or a tool to convert audio data from one format to another? Or any other kind of software where the output doesn’t consist of readily verifiable, discrete data points?

The problems you describe are very high level, but we could design an acceptance testing scheme for them. For the Mandelbrot set it might involve comparison to a reference rendering, for the audio tool a reference recording. In both cases you'd allow a delta relevant to the application, and probably also benchmark for acceptable performance.

But my point was more aimed at unit testing. When you set out to write a function you should know something about that function before starting. If you know enough to write the function signature, you can first write a failing test. If you can write a bit of code in that function, you can write a bit expecting the behavior of that code.

I suppose what I’m really asking is how you would go from not having software to having software that does those things, using TDD. I think in practice its fail-pass-refactor cycle is normally applied at the level of unit tests, but in any case, how would using TDD help to drive a good design, to ensure testability, or otherwise, in that kind of situation?

(I’m asking this rhetorically. I don’t think TDD is a very helpful process in this context. I’m just trying to demonstrate this with practical examples rather than bluntly stating it without any supporting argument.)

A great example for your point (which might have been what you were getting at with the audio thing) is a test for creating files in a lossy audio format. The acid test is if it sounds right to a human being with good ears, I've got no clue how you would write a pure computer test for that.

In my own work, a great example is finding a NURBS approximation of the intersection of two surfaces. There are an infinite number of correct answers for a given pair of surfaces, and testing that the curve you've generated fits the surfaces is a distressingly hard problem.

Indeed it does, but in order to verify them I see only two options.

One is that you have to choose test cases where the answer is trivially determined. However, with this strategy, it seems you must ultimately rely on the refactoring step to magically convert your implementation to support the general case, so the hard part isn’t really test-driven at all.

The other is that you verify non-trivial results. However, to do so you must inevitably reimplement the relevant mathematics one way or another to support your tests. Of course, if you could do that reliably, then you wouldn’t need the tests in the first place.

This isn’t to say that writing multiple independent implementations in parallel is a bad thing for testing purposes. If you do that and then run a statistical test comparing their outputs for a large sample of inputs, a perfect match will provide some confidence that you did at least implement the same thing each time, so it is likely that all versions correctly implement the spec. (Whether the spec itself is correct is another question, but then we’re getting into verification vs. validation, a different issue.) However, again for a simple example like rendering a fractal, you could do that by reimplementing the entire algorithm as a whole, without any of the overheads that TDD might impose.

I don't have any problem imagining myself developing a Mandelbrot set program using TDD.

I’m genuinely curious about how you’d see that going.

1) Establish tests for the is-in-set function. You're absolutely right that the most obvious way to do this meaningfully is to reimplement the function. A better approach would be to find some way to leverage an existing "known good" implementation for the test. Maybe a graphics file of the Mandelbrot set we can test against?

2) Establish tests that given an arbitrary (and quick!) is-in-set function, we write out the correct chunk of graphics (file?) for it.

3) Profit.

Observations: 1) I absolutely would NOT do the "write a test; write just enough code to pass that test; write another test..." thing for this. My strong inclination would be to write a fairly complete set of tests for is-in-set, and then focus on making that function work.

2) There's really no significant design going on here. I'd be using the exact same overall design I used for my first Mandelbrot program, back in the early 90s. (And of course, that design is dead obvious.)

In my mind, the world of software breaks down something like this: 1) Exhaustive tests are easy to write. 2) Tests are easy to write. 3) Tests are a pain to write. 4) Tests are incredibly hard to write. 5) Tests are impossible to write.

I think it's pretty telling that when TDD people talk about tests that are hard to write, they mean easy tests in hard to get at areas of your code. I've never heard one discuss what to do if the actual computations are hard to verify (ie 4 & 5 above) and when I've brought it up to them the typical response is "Wow, guess it sucks to be you."

The latter is what I’d expect most developers who like a test-first approach to do. I don’t see anything wrong with it, either. I just don’t think it’s the same as what TDD advocates are promoting.

I think it's pretty telling that when TDD people talk about tests that are hard to write, they mean easy tests in hard to get at areas of your code. I've never heard one discuss what to do if the actual computations are hard to verify (ie 4 & 5 above) and when I've brought it up to them the typical response is "Wow, guess it sucks to be you."

Indeed. At this point, I’m openly sceptical of TDD advocacy and consider much of it to be somewhere between well-intentioned naïveté and snake oil. There’s nothing wrong with automated unit testing, nor with writing those unit tests before/with the implementation rather than afterwards. Many projects benefit from these techniques. But TDD implies much more than that, and it’s the extra parts — or rather, the idea that the extra parts are universally applicable and superior to other methods — that I tend to challenge.

Thus I object to the original suggestion in this thread, which was that a developer probably doesn’t know what they are doing just because they can’t articulate a test case according to the critic’s preferred rules. I think those rules are inadequate for many of the real world problems that software developers work on.

I think you get to the nub of it here. TDD lets you develop a spec that is consistent with requirements (the subset so far implemented) and the code at all times.

Writing a comprehensive suite of tests before any production code is like writing a complete spec without any clue as to its applicability. Writing tests afterward would be analogous to writing a spec for an already shipped product.

Tests work both ways in TDD: you are checking both that the code behaves as intended and that your expected behavior is reasonable. If it were only about the former it wouldn't be very valuable.

This is another TDD-related argument that I just don’t understand.

A specification might say that the function add returns the sum of its arguments.

A unit test might verify that add(1,1) = 2.

One of these describes the general case. One of them describes a single specific case. Unless your problem space is small enough to enumerate every possible set of inputs and the expected result for each of them, no amount of unit tests can replace a full specification of the required behaviour. Unfortunately, not many real world problems are that convenient.

That said, i do still find that while test-driven development doesn't itself create good design, it is a useful tool to help me create good design. I have a bite-size piece of functionality to write; i think about what the class should look like; i write tests to describe the class; i write the class. The key thing is that the tests are a description of the class. The act of writing down a description of something has an amazing power to force the mind to really understand it; to see what's missing, what's contradictory, what's unnecessary, and what's really important. I experience this when i write presentations, when i write documentation, and when i write tests. The tests don't do the thinking for me, but they are a very useful tool for my thinking.

I do gather some places do things differently though. Must be nice.

The author of the post basically seems to treat any isolation level below serializability as some sort of sham perpetrated on the development community, and that's not the case: they're still useful, and they're still something that you can use to build the sorts of application guarantees you want. The mere fact that pretty much every database vendor gives you a choice as to what isolation level to use should be a pretty obvious clue that there's no one-size-fits-all answer there, so harping on the non-existence of a serializable isolation level is somewhat missing the point.

Consider a banking application that uses linked accounts such that transactions fail if the combined balance of linked accounts is below zero. If I implement this in Oracle using Snapshot Isolation (the highest isolation level Oracle offers) in the obvious manner, I'll get silent constraint violations: transactions that should fail will succeed because MVCC can't stop two competing transactions debiting the same account pair (i.e., I have two linked accounts A & B with $100 in each and I launch two simultaneous transactions to transfer $150 away from both: afterwards, I'll end up with -$50 in both or some garbage data).

Now, you can fix if you recognise the problem in advance by doing a select-for-update or changing your schema to materialize the constraint (say by creating a linked-balance table that holds the combined balance for all pairs of linked accounts).

But it is really hard to even notice the problem, especially if you've got a few dozen tables with multiple applications writing to your database (RDBMS advocates insist that this is a good thing). And there are no automated methods for determining when this will be a problem: you just get silent data corruption or silent constraint violations in your extremely expensive "ACID-compliant" (but not really) Oracle database.

Pretty much every ORM solution has some mechanism for dealing with this sort of a problem, as well. For example, the ORM we've built internally uses optimistic locking in most cases, which at least simplifies the problem space that our engineers have to worry about (i.e. they only have to worry about conflicts that can happen due to updates to two different rows, but any conflicts due to an update to the same row are automatically detected).

I'm also firmly in the "having multiple applications writing to the same database is sheer madness" camp, due to these (and other problems).

Again, I'm not saying that these things aren't annoying to think about, just that I think lots of people are used to thinking about them at this point, and it's a pretty well-understood and often well-communicated problem space with a set of known solution patterns, many of which are well-supported by ORM frameworks.

That isn't true. In a pure MVCC world you could detect such issues, specifically that two trxs were opened and they both tried to update the same row. This would generate a conflict and one transaction could just get shot. Snapshot-isolation/MVCC describes this issue.

However what InnoDB/mysql and Clustrix actually do is use MVCC for read-write conflicts and two-phase-locking for write-write conflicts which trivially fixes your proposed problem.

That said, Snapshot Isolation can lead to anomalies like Example 1.2: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.142... If you perform additional synchronization (e.g., 2PL, as you describe), then you can get serializability.

You may not care for this schema design, but I think most people would be surprised that Oracle silently corrupts data and violates constraints here.

you have to think about race conditions even if the database implements serializability. First of all, you have to have enough of a clue to do steps 2 and 3 inside the same database transaction, or else the isolation level doesn't help you in the least. Everyone who deals with an RDBMS has to worry about transaction boundaries all the time anyway to avoid race conditions, so it's not too much of a leap to assume that people who are used to worrying about transaction boundaries will have to have some idea of what kind of problems those transaction boundaries will prevent for them. Most databases in production systems don't do full-on serializability as an isolation level (and if they do, it's too expensive to turn on), so pretty much every engineer's real-world experience is that if they do a query against table A, then an update against table B based on that data, that they can have a race condition and they need to lock A or materialize potential conflicts.

I guess it's possible that my personal experience or the engineers that I work with are not a representative sample, but if you're working with sensitive issues like money transfers you tend to be really, really super-paranoid about it, and there's no way that something like that is overlooked by a halfway competent team.

What do you mean by consistency? I agree that there are many ways to ensure that application integrity constraints are not violated--without using serializability. My point is that, without ACID serializability, you'll have to do some extra work to ensure them in general [1].

The author of the post basically seems to treat any isolation level below serializability as some sort of sham perpetrated on the development community, and that's not the case

Weak isolation has a long tradition spanning decades [2] and is hardly a "sham." It's well known that weak isolation doesn't guarantee the ACID properties as traditionally defined. My point is that many databases don't provide ACID as they promise.

It's still a different world than trying to mimic ACID properties in a NoSQL database

In terms of current offerings, I'm sympathetic to this viewpoint, but you might be surprised how cheaply we can get properties like the transactional atomicity you mention.

In general, I'm curious how easily anyone's able to take their application level requirements and map them down into an isolation like "read committed," especially given how awkwardly they're defined [3] and how many corner cases there are.

[1] e.g., http://www.bailis.org/blog/when-is-acid-acid-rarely/#arbitra...

[2] "Granularity of Locks and Degrees of Consistency in a Shared Data Base," Jim Gray et al., 1976 http://diaswww.epfl.ch/courses/adms07/papers/GrayLocks.pdf

[3] e.g., http://www.bailis.org/blog/when-is-acid-acid-rarely/#weak-no...

For example, if I'm writing a process that merely queries a table to pull back the user's account balance in a single query, read committed isolation might be enough for that particular use case to have "consistency:" I know I'm always seeing some consistent balance, even if it might not reflect transactions that are currently in flight (thus giving me a different answer if I run the query it again in 2 seconds). That's still a better consistency guarantee than if I have a read dirty isolation level (or effectively no isolation), so it's still useful.

If I'm doing an actual update to account balances, however, that level of consistency is no longer good enough, obviously: if all updates hit the same rows in the same tables then snapshot isolation level might be good enough to avoid problems. And if that's not good enough, then I can acquire explicit update locks and such, if the conflict has a risk of update locks across different tables. Even in that case, though, I'll need to worry about application-level invariants (like "a person can only withdraw up to $300 per day.").

So my point is that even without serializable isolation the database can still guarantee some invariants for me, even if it can't guarantee all of them, and that the database can never really guarantee preservation of all of the invariants that matter to me no matter how strong it's guarantees, so I always have to think about what I'm going to handle at the application level anyway.

In the case of my company (which makes applications for insurance companies), we do have to think about those sorts of things, but again, we have to think of a ton of things anyway, and the division of labor between the app tier and the database tier is always something we have to worry about. We do things like build optimistic concurrency into our ORM layer to make most common cases easier to think about, and we have pretty well-defined transaction lifecycles, but for the most complicated cases we have to think about what the potential for race conditions in the database would be, just like we have to think about them at the application level, and then we have to decide how to handle them. Again, even a "true" ACID database wouldn't prevent us from having to do that work, because many of the invariants we want to preserve in the data aren't expressable in the database anyway.

One thing is conventions around things like where curly braces go, naming conventions for classes and methods and different types of variables, how much whitespace to use where, and so forth. Most of these things don't actually matter at all, but code certainly is easier to read if everyone on a team follows the same ones (so that I know that Foo.bar() is a static method while foo.bar() is a method call on a local variable and _foo.bar() is a method call on an instance variable on the class, without having any other deep knowledge of the code). The point of conventions is basically to get everyone to stop bikeshedding about things that don't matter (like which line a curly brace goes on in an if statement), so that they can shut up and go back to writing code. There's no rational way to decide trivial questions like "how much white space should we use," so rather than having people engaging in pointless holy wars over two versus four spaces, we just establish a convention and stop talking about it. It really does make life easier on a team if people are on the same page there, and there are plenty of tools that make it trivial for people to format their code so it conforms to the conventions.

Then there's the issue of "standards," which is a much larger, amorphous topic that probably shouldn't just be trivialized. Trying to encode them formally is probably madness, but when some person on your team writes a 245-line-long function or calls a method "getFoo" when it in fact writes a file to disk as a side effect, you do probably want to have a little chat during a code review to say, "Hey, yeah . . . so that's not really okay, even if the code functions correctly." That's what I think of as "standards:" the assumption that just getting things to work isn't the only thing that matters. And just throwing that out the window and saying "Anything goes so long as it works" is pretty suicidal if you're working on a team.

For Java specifically our "standards" also incorporate the book "Effective Java (2nd edition)" http://www.informit.com/store/effective-java-9780321356680 .

First of all, cost absolutely 100% has to factor into prioritization decisions. That doesn't require absolute estimation, but it does demand relative estimation (which he mentions tangentially at the end of the post). If Feature A will deliver $100,000 in revenue but take 18 months and Feature B will deliver $10,000 in revenue but take 1 week, the choice is pretty obvious. What matters is never "return" but always "return on investment." If you don't know anything about the I side of the ROI equation, you're doomed to make bad decisions. With no estimate at all, and just a snarky "it'll take as long as it takes, shut up and let me work" response, you'll inevitably focus on the wrong things.

Secondly, many of us do in fact have deadlines, and they're not just total BS. If you have a customer that you're building something for, they have to be able to plan their own schedule, and just telling them "well, maybe we'll ship it to you in 10/2012 or maybe in 6/2013, you'll get it when you get it" doesn't fly. And it's totally reasonable that it doesn't fly: if they have to, say, install the software, or buy hardware to run it on, or train users, or migrate data from an old system, or roll out new products of their own that are dependent on the new system, they clearly can't plan or budget those activities if they have no clue whatsoever when they'll get the new system.

And if you do have a deadline, you kind of want to know, to the extent that you can, if you're going to make it or not so you can take corrective action (adding people, cutting scope, pushing the date) if need be. You can't do that if you have no estimates at all.

Relative estimation of tasks with empirical measurement of the team's velocity works pretty well; it doesn't work in all cases, but it's pretty much always better than absolutely nothing.

There's a huge, huge difference between doing relative point-based estimation and date-driven, pointy-haired-boss estimation, and it's a total disservice to the software community that so many engineers seem to not really understand that difference, and seem to think that the only two options are "unrealistic date-based estimates" and "no estimates."

TL;DR - Don't just rant for 3000 words about how estimation is horrible and then add in one sentence about relative estimation. You'll do the world much more good if you just help educate people how to do things the right way and spare the ranting.

This was my beef with the article too. Basically on the one hand he proposes a strawman composed of known-worst practices (estimate-by-deadline, estimate-by-gut, ad hoc estimation and so on) and thereby tars all estimation with the brush ... except for the one alternative he approves.

This is the fallacy of dichotomy.

The root problem is the concept that estimates have to be accurate. Well, duh, they can't be. The bigger the project, the more people, the longer the timeframe, the less likely the project is to meet the estimate.

That's why you don't perform one estimate.

That's why you have confidence bands on estimates.

The whole blog article feels like a pastiche of criticism cribbed from agile books and not from a direct, thoughtful engagement with the primary or secondary literature on software project estimation.

I'm only 31. By any measure I'm still a young man. Why do I feel like such a curmudgeon all the time? Because apparently nobody reads books or papers any more. It's all blogs citing blogs who echoed the summary of the notes of the review of a single book.

One more thing. There's a difference between a plan and an estimate. Plan-and-control is not the same thing as performing an estimate; DeMarco's self-criticism is not directly applicable.

http://www.amazon.com/Software-Estimation-Demystifying-Pract...

Every time I estimate for clients I always talk about the Cone of Uncertainty.

Of course accurate estimates have tremendous business value. But in reality they often come at the expense of what the client really needs which is delivery of features. I have seen estimation and tight project control taking up substantially more time than delivering actual features. And it was exactly as the OP stated:

> Software projects that require tight cost control are the ones delivering little or no marginal value.

The lesser the project value the tighter the control leading to a vicious cycle of developers cutting corners and increased micro-management.

It sounds to me that what you saw was a conflation of estimates and plans. Which is a common error.

It's unfortunate that so much of the literature on relative estimation/story points/velocity/planning poker ends up intertwined in agile-development-consultantese, so sometimes reading some of these things, you have to take it with a serious grain of salt and weed out all the BS and dogma to get to the useful and important bits. The important bits there are really pretty simple: * Estimate in terms of points, not days, and estimate tasks relative to one another * Use planning poker (or something like it) within the team to get more accurate estimates * Empirically measure how many points you do in a given period of time to come up with your "velocity". To do that, you have to be pretty honest about things being "done done" before you move on to the next task; otherwise it's too easy to fudge and your numbers are worthless. "Oh, yeah, we did 10 points of work, well, except for writing the tests or doing the styling . . ."

Remember that velocity is NOT a productivity measure, it'll change over time, and it'll change if the team members change or if other external factors intervene, like an increase in support requests or something. So this technique kind of only works if your organization isn't already dysfunctional: as soon as velocity is seen as a productivity measurement, you're pretty screwed.

That's pretty much it. The relative estimates let you prioritize work appropriately (i.e. "I'd rather have these five 1-point stories than this one 5-point story, so we'll do those first"), and the velocity lets you track how fast you're actually going and about when you'll be done with a given chunk of work, so you can adjust plans as needed.

Note that relative estimation doesn't work nearly so well for large-scale estimation, or for greenfield development where you don't know what you're doing yet. For large-scale planning, my company will generally just give a SWAG in terms of points (we think this feature is 100 points) to give us at least some initial idea of expected relative sizes of things, then we'll compare that initial SWAG to the actual points as we break things out into more bite-sized stories that we can estimate more accurately. If we feel like we're half way through the feature and we've already done 70 points of work, that's a signal that we need to up our estimate for the remainder of the work. Steve McConnell's book is good as well, though honestly we don't really do much in the way of confidence-based estimates at this point. My experience is that out of every 10 projects we do, 8 or 9 will be within 20% of our initial SWAG and 1 or 2 will blow out to 2 or 3x the initial estimate. Of course, we never know which ones will blow out, we just know something will. In other words, we don't bother with confidence intervals at the individual feature level, we just do it at the macro level. So if a team has a velocity of 10 and we have 26 weeks to deliver a release, giving us a hypothetical total capacity of 260 points, we'll "commit" to maybe 2/3 of that. So we say "Okay, this first 170 points we're pretty sure we can get done. Anything after that will be a bonus."

Basically all that agile really changed is how often folks take a sounding. Velocity is basically a productivity metric, it's a first derivative of project size (hence ... velocity).

But I mean you could always do that with traditional estimates.

That's pretty much what an Earned Value chart does.

Don't get me wrong: agile is an improvement. But it is an improvement that is evolved from what came before; not a categorical change. Looking at the historical literature still pays dividends to thoughtful engineers.

You're right. This is one of the primary advantages of startups with technical founders that talk to customers. Among many other flaws of the "OfficeSpace" work environment, one of the biggest is the fact that no single person in the organization knows 1) what is possible to build 2) how much it will cost to make and 3) what is the value of product to the customers.

At my last corporate job before I founded a startup, the CEO didn't know which software was possible to write, and didn't know which new features were easy vs. hard. The marketing people rarely told engineers what the customer wanted, and the engineers didn't talk to customers, and didn't know the cost (or value) of their time.

If there are any well-tested rules for producing good software, they're 1) small teams, 2) technical people with authority and responsibility, 3) talk to the customer.

They'd be better off telling the client the truth that "We don't know a honest date and we don't want to lie and come up with an arbitrary one. I can only say it's unlikely that the project would take more than six months and you should at least be prepared to order the hardware by the early summer. We'll keep you posted."

Emphasis on the last part.

Schedules are more accurate later on when some work has been done already. Instead of creating an arbitrary point in the future keep the customer posted with: "We've already finished X and Y in the first month, so we'll soon start working on Z which is a big task but on the other hand we noticed that we don't have to do Q at all since we can just build it in terms of X and Z combined which won't take more than a week instead of the month that was originally planned."

If you tell this every week then everybody has an up-to-date view of the runway ahead. You can also give a "deadline", like: "If you need a deadline for higher-ups or administrative planning, use October 31st but tell them to prepare to plunge in earlier. We all know we'll be done then by a far margin."

Most customers that I've ever had have intuitively understood this is how it works, even if they had to have a "deadline". YMMV.

If you are then... "We'll keep you posted." "up-to-date view of the runway ahead."

Where do these things come from if you're not estimating effort?

The only thing you can really confirm though, is what is already completed, and that is what you should keep the stakeholders posted on. And a description of work left, not necessarily the time it will take.

This is the work left, we think it's this size*, this is how much work we've been getting done. It looks like we might be done at this date but that's dependent on a) our estimation being correct and b) our pace continuing.

I think it's more about managing expectations than it is about managing the information you let stakeholders have.

Even if there are no external customers waiting for the release, a business' other departments have to plan their activities. When will marketing start their pre-release activities without engineering's estimates? When will sales start talking to customers about the new release? There is just too many things that need engineering's estimates.

There are a key pieces to making that work. First of all, you give your estimates in terms of "points", not hours or days. Secondly, you estimate tasks relatively to one another, not absolutely: a 3-point task is about 3-times as much work as a 1-point task. Third, you have multiple people give the estimates, often using something like "planning poker" (where everyone on the team selects their estimate and reveals them simultaneously, then discusses if they're different), which ensures they're more reliable. Fourth, you measure the velocity of the team (not individuals, the team) over time. It often takes a while for it to get anywhere close to stable, as people get used to the project, the technology stack, each other, and so forth.

That's the only thing I've ever seen work, and it can actually work pretty well. It prevents the developers from having to account for all the non-coding things: you don't have to think "Well, this will take 2 hours of coding, 2 hours of test-writing, 1 hour of debugging, 1 hour of docs . . . but I also have 2 hours of meetings a day, and I get pulled off to firefight customer problems periodically, so really it'll take 4 calendar day." You just measure all that stuff empirically: if you have a bunch of meetings or firefighting, your velocity slows down, but your estimates don't have to change to account for it.

It also avoids wishful thinking, especially if you're rigorous about what counts as something being "done". It's harder to lie about whether or not you're going to hit your dates if your measurements say you're going too slowly; it's far easier to be in denial when all you have are time estimates, since those are easier to fudge than relative estimates of difficulty or empirical measurements of velocity.

(Of course, as with any measure of productivity, it's subject to abuse by idiots. But as a scheduling and measurement tool, it can be invaluable.)

I'm sure you're aware of Apple's fairly disastrous entry into the world of Mac clones back in the day. What you're arguing now is essentially the same theory back then: clones would benefit them by increasing the marketshare for their OS. Instead, clones just cannibalized Apple's own sales and diluted their brand with a bunch of crappy knock-offs. I'm not sure why anyone would expect it to play out differently now.

Sure, it might be in your interest as a consumer to have choice in hardware, but I don't see how it's in Apple's interest to do so. They seem to be doing pretty well precisely because of the choices they've made to lock things down, not in spite of them.

That's not what OP is saying:

If the Apple hardware ecosystem were open like that of Windows you'd have major companies all over the world evolving the platform in wonderful ways. Performance would go up and prices would come down.

This isn't "moar market share," it's "the end user would benefit greatly if Apple hardware opened up."

Today is much different than the Mac clone era for two reasons:

(1) Apple clearly has no intention of offering a high-end desktop solution. Instead of losing those users to high-end Windows or Linux machines, they could keep them in the ecosystem by licensing OS X for use in boutique-style desktops.

(2) Apple holds much more power now. In the 1990s they licensed Mac OS from a point of weakness; today they could license to high-end, boutique manufacturers who meet their design standards.

Is this the right way forward? Maybe, maybe not. But OP's suggestion does not necessarily lead to the Mac clone days of the 90s.

All this Adam Smith idealism is great until we look at the real world, where the barriers to entry, integration costs, barriers to accurate consumer knowledge and platform compatibility hurdles are too high for competition to actually produce ideal outcomes.

If people wanted a competitive platform with a propitiatory OS they'd all be developing on Windows. There is no reason for Apple to try to compete in the same space, especially not when they are winning everywhere that matters for consumer electronics.

Apple prints money, and could afford to choose only the cream of the licensing crop (which, to be fair, will be pretty curdled). Here's where we violently agree:

All this Adam Smith idealism is great until we look at the real world where [stuff is] too high for competition to actually produce ideal outcomes.

In this hypothetical I'm not considering "competition." I don't know if this was obvious from my last post, but I'm wondering what would happen if Apple licensed OS X to a high-end desktop manufacturer that would make Mac Pros on its behalf.

Let me be clear: there is a 0% chance this happens. At the same time, it is clear Apple is not interested in the high-end desktop game. So it's fun and interesting to consider the possibility that Apple -- a company with negotiating power in spades -- could pick from any manufacturer on this planet and choose a licensing partner.

More important, in my view, is the reverse side of that luck equation: if you assume that input A leads deterministically to outcome B, then if you didn't get outcome B, obviously you didn't put in input A. Replace "A" with "hard work" and B with "economic success" and you have a nice justification for killing the social safety net, for example: obviously people who aren't successful must not be working hard enough.

So accepting that luck plays a role in success doesn't just affect your view of someone's success, it affects your view of other people's potential lack of success, which is an even more important thing to have if you want to have empathy for your fellow human beings.

Unfortunately, cognitive dissonance being what it is, a desire to attribute your own personal success to hard work rather than to luck makes it harder to attribute other people's failures to bad luck, and inclines you to assume that they must "deserve" their situation in life.

So I think it's less important to play the "what if" game there with specific situations, and more important to realize that people who haven't been successful might have been unlucky (or less lucky), rather than to try to decide whether someone successful was lucky or not.

The article looks at the relative employment rate of people who have attended college vs those who haven't. People who have attended college should be more qualified for more jobs than those who haven't. They should be more employed, but they aren't. The simplest explanation(and the one that matches what I've seen/experienced) is that people who have more education are too picky about the jobs and careers they are willing to consider and are therefore finding it harder to get a job.

In fact they are more employed, according to the article, if they actually graduated (as opposed to merely attending). The article gives, for people 25 and up, the following unemployment rates:

    Overall: 6.6% unemployed
    High school, no college: 7.7% unemployed
    Some college, no degree: 8% unemployed
    2-year college degree: 6.2% unemployed
    4-year college degree: 4% unemployed

So I guess it depends on what problem you want to solve, but if you're worried about the federal deficit, then higher taxes (even in the form of just eliminating the bush tax cuts) will absolutely help to solve that problem.

If I tell you, "Fred is always running out of money; in fact, he's deep in credit card debt," your immediate response is not going to be "He should get a raise so he can pay off that debt" or "Here, give this money to Fred."

I'm not sure where you're going with that analogy. It all depends on whether Fred's spending his money wisely or not.

So your analogy doesn't just do work for your position; it clearly cuts both ways.