My job as a consultant is to tell you "your data fits in RAM" and charge you $20k. While it may seem expensive, the ROI is about 14 days, because now you don't need to hire that data scientist.
You nailed it. People don't get promoted for leading pedestrian projects, they get promoted for leading challenging, innovative, groundbreaking projects.
Sometimes tech reminds me rich, bored, stay at home SO's that are constantly redecorating their house. Not because they need it, but because they are bored and the next trendy design looks cool anyway.
It comes from both top and bottom. At the top managers want to justify their salaries to their managers so they always redesign / rebuild / reorganize, even if things work pretty well as is. At the bottom new programmers fresh out of college want to assert themselves. The best way to do is to propose that everything existing is old and shit and needs to be rewritten. So they volunteer of course.
How does someone get into a position like that? Are you an independent consultant or do you work for a company? Is this exclusively what you consult on?
His comment while accurate was probably a bit dramatized for effect.
My guess is he is a consultant hired to do a typical project (i.e., help me move X into the cloud, help me re-architect our data model for big data, help me implement AI for our dog walking app) and at that point he just shows them they aren't ready for it or flat out don't need it.
It's just my guess, but it's what consultants do. The bad ones are happy to take on your project and charge you $400 bucks an hour. The good ones unfortunately, deal with the dilemma of turning down lucrative work in the spirit of doing what's right.
My guess is that at some point you get tired of it and take solace in the fact that you're at least helping them to do it right.
After all, if they think they are going to hit big data scale and want the tools to handle it, you aren't completely a bad guy if you help them do it right, especially if you've already advised them not to do it.
I regret to inform you that my original comment was sarcasm. I'm not a consultant, but would be perfectly willing to do the job as I described. The hardest part would be keeping a straight face while demanding the $20k.
It's amazing how many startups don't re-evaluate their infrastructure requirements every couple of years and get stuck with numbers in their heads reflective of hardware prices five years ago.
No need for clever optimizations - Moore's Law will bail you out a lot of times.
