"NoSQL databases are not free to operate. Each NoSQL Databases come with the cost of hardware, administration of the hardware, and support of the software."
Hey Oracle, if you want to bring this point up, please at least pretend to compare the above costs between NoSQL and your own solution. Otherwise it is just pure FUD: "NoSQL databases eat your CPU cycles, disk and network badndwith! Booooo!"
Well yes but AFAIR MS at least provided some data that could be critically analyzed (whether it stood up to the analysis is a different matter.) I am all for using TCO as an argument in tech advocacy, just as long as it has some backup in verifiable data.
A discussion on TCO is very dependent on the context in which the analysis was made, such as application type, volume of data stored, volume of data indexed, type of data (e.g. is it a tree? is it a graph?), queries per second/hour, number of concurrent connections, analysis on further scaling possible (vertical / horizontal), analysis on the requirements for real-time ad-hoc queries, entry-level costs, and so on and so forth ...
Basically, the variables are so many and so complex that any TCO analysis (regardless of validity) won't apply to your case.
That said - I rarely feel the need for NoSQL, simply because I can fine-tune PostgreSQL to whatever needs I want. But I'm not running neither Facebook, nor Twitter.
Also include the skills of the people operating everything. A MCSE trying to administer various Linux boxes will have terrible TCO, and a RHCE trying to optimize an NT box will have similarly bad TCO.
As someone dealing with Oracle's "support of the software.", let me be blunt: Oracles own flagship software's support is... severely lacking when you are hitting bugs.
I believe their point is that deploying, for example, four types of NoSQL database creates four, separate sets of hardware etc.
i.e. it is not that NoSQL systems are inefficient individually, it's that it is not efficient to maintain four separate resource pools.
(This is still possibly FUD nonsense: Oracle has to exist as an independent resource pool only because their virtualization licensing requires all potential CPUs to be licensed. Are any/all of the commercial NoSQL implementations licensed this way?)
It takes less time to configure an HBase cluster than a single Oracle server under Linux, and even then you can just create a single HBase image to deploy in hundreds of servers.
I guess this boil down to the following four points:
- NoSQL is good in scaling: but do you really need to scale? Is your business so great to handle 1M users? Simple MySQL (and PostgreSQL when mysql is not ok) will go a long way for many many installation.
- NoSQL are in good in scaling but it is pain to maintain it: it hard to hire somebody who knows it good, there are no good management tools, etc.
- NoSQL requires you to make your application logic more complex and not portable
- There is no clear NoSQL leader and clear mainstream knowledge. And a lot of these NoSQL implementation requires some secret sauce known only to engineers maintaining and developing big NoSQL installations at Facebook, Google and Amazon.
In short, for startups, NoSQL is one more variable introducing risks and unknowns. So I think startups should not go NoSQL unless that is the core of their offering.
"In short, for startups, NoSQL is one more variable introducing risks and unknowns. So I think startups should not go NoSQL unless that is the core of their offering."
Well said.
NoSQL may have a lot to offer in some cases, but I think "easy" is an illusion when you look at the full picture.
And perhaps a few leaches to get the bad humors out, as well. (financial) "haircut", indeed.
I actually agreed with much of the data integrity problems the paper pointed out, but found it a bit too condescending, or perhaps even a bit threatening, around the "You are not Google" part. Big Brother loves me!
This document is nothing more than an embarrassment for Oracle. "One size DOES fit all - and that size is ORACLE!" Really? Even the most entrenched, old school DBA stuck in the bowels of some drab insurance company in Indiana wouldn't buy that. I'm not a NoSQL guy by any means (check my profile) but the desperation in this doc is way more palpable than a guy like Ellison would approve of.
You truly underestimate the zealotry and kool-aid drinking capacity of the average Oracle DBA. When your entire career is based on the training and certifications you've received from a single company, every database solution magically becomes Oracle...
(Previous Oracle Appreciation Concert performers have included Elton John, Billy Joel, Aerosmith, Roger Daltry, Don Henley, the Black Eyed Peas, Devo, and Joan Jett.)
"While in theory, NoSQL databases can be deployed on hundreds or thousands of
machines, in practice the deployments are much smaller and can typically easily fit within a
single Oracle database with fewer and more reliable components. The largest production
Cassandra cluster has over 100 TB of data in over 150 machines."
I find this interesting. I wonder if there any real TCO comparisons have been done to this point.
Sure, this paper is full of FUD. But it does raise a valid question for me: What's the difference between paying for Oracle and paying for MySQL, Datastax, or Cloudera support/enterprise to get the added features needed to run my business and keep away from reinventing the wheel?
Truth is, most of the Oracle (same applies do SQL Server) databases I've met could run just fine under MySQL or PostgreSQL. A good deal of them would be unable to significantly stress SQLite or MS Access.
Unless you start with a million users generating data, NoSQL (or Oracle RDBMS) is premature optimization. You want to architect for growth just in case, fine, but don't waste your time before you know you will have a problem.
I see this sentiment a lot. It annoys me. NoSQL is not a performance operation. It's about choosing a representation that is appropriate for your data.
It becomes one, but only when your dataset reaches sizes almost nobody will ever see.
If your business model implies a scale-or-die condition, then I'd say it's fair to design your solution to accommodate NoSQL, but not waste time implementing it in production. It's a horrible waste of time to design something around a Cassandra cluster that will be hit once every second.
And, BTW, if you hit the performance architecture's limit (actually, you should have detailed monitoring to tell you how many weeks you have until you hit it), then you should consider what you'll do to solve your particular problem (which may not be what you anticipated initially). You should always invest a little time evaluating your options for different scenarios.
It's a horrible waste of time to design something around a Cassandra cluster that will be hit once every second.
While I would probably agree that Cassandra was designed for scalability, there are hundreds of other NoSQL databases designed for completely different problems that have nothing to do with scaling. CouchDB, for instance, is really good at synchronizing data between separate disjoint databases (think temporary offline use); a problem some have with databases both big and small.
Every NoSQL database was born out of the need to solve a problem that was not well solved using existing technologies. Some of those problems are related to scaling, but scaling problems are just the cusp of what is now available to you to use. Definitely use SQL when it is the right tool for the job, but don't try to shoehorn it into the wrong job. There are other tools that just might be a better fit no matter how big your database is or how many people are going to be using it.
It is overkill to implement a full fledged SQL database and write a bunch of SQL queries, if all I need is a persistent key/value store.
I've been playing with Redis lately, and it is really incredibly easy to use, and it maps quite well to the data-structures that I already have in my code.
There is certainly use for both types of database, but a lot of the NoSQL products are genuinely good tools (completely ignoring scalability), and make using SQL look like hammering screws for certain applications.
If all you need is a persistent key/value store then why do you need more than one table and one each of SELECT, INSERT, UPDATE and DELETE statements?
I'm all for using the most appropriate tools, but sometimes path of least resistance is worth considering too. Android bundles SQLite as a basic service, while many/most organisations will have existing RDBMS SQL servers as part of their infrastructure which are well understood and maintained. In either of those scenarios SQL may well have more power than is required, but it's also already in place and well understood and on the principle of minimal deviation from the known standards, I'd probably still use it.
The path of least resistance for me, BY FAR, was CouchDB.
What I wanted:
* Easy interface to a key/value store.
* Reliability. Once something is stored in the database, I want a reasonably high level of guarantee that it won't be forgotten.
* Trivial replication so if I server goes down and takes the database with it, I can restore it quickly and have a backup server in the mean time.
* High speed on minimal hardware, so I wouldn't NEED to scale under likely usage scenarios.
* Easy path to scaling if it should become necessary.
Every SQL database I know takes nontrivial effort to shard. CouchDB has built-in master-master replication that takes two seconds to set up.
SQL imposes performance penalties that I simply don't need, and so NoSQL is going to be faster on the same hardware, and require less scaling.
CouchDB makes things so simple and low-maintenance that I don't need to hire people who understand it, because I can keep it maintained in my free time. I haven't needed to do anything with it after I spent a couple days to deploy it, and it's happily replicating to a second CouchDB instance that I can fall back on if needed.
And if I get to the point where I need to scale, well, it will work fine to create more CouchDB instances (my data is read FAR more often than it's written, so simple replication should be workable).
Even with MySQL, I feel like I would NEED an expert to be sure to achieve all of my goals above -- not because I couldn't, but because it would waste too much of my time. With CouchDB I got something that Just Worked in exactly the way I needed it to.
Well, one reason that forces itself is, that the 'value' part is not homogenous, primitive type. It is often kind of tree or graph, with possibly different branches for each key.
A lot of it really is a matter of what you know. I think SQL is incredibly easy to use. ESPECIALLY if all you are doing is select, insert, update, delete. Many people never use the full power of their RDBMS, and in that sense it may be the case that they don't need an RDBMS.
I recently built a view in postgres that uses windowed aggregate functions (what Oracle would call "analytics") to present a certain summary of statistical data. The beauty of this is that it's declarative, and took me a LOT less time than coding up a manual aggregation over the specific windows and grouping that I needed, the unit testing surface area is a lot smaller, and it's written once and available to any client platform/language/framework/tools that can talk to Postgres.
Postgres and even moreso Oracle gives you this kind of power out of the box. If you're not using it, you're not getting the full value out of those products.
Let's say my business is real time analtics---tracking mouse movements on the page.
This is tackled by quite a few startups, so it's not out of the realm of things we HNers do ourselves.
Mouse tracking can generate potentially millions of data points per day for even a small number of users. However, we only store a set amount of data per client.
If for example, you think that every point should be tracked individually you can easily run into the case where Postgres will be non-optimal under the read/write load.
Something like Redis on the other-hand might be optimal.
This is a case where its not a premature optimization, its just the right tool for the right job.
I once had to track user actions in a pretty active online game - I configured the Apache servers to pipe their logs to a Perl script. The Perl script was parsing the requests, then pushing that data to a buffer (i.e. an array).
Then when the buffer reached 5000 requests, it would push the data in PosgreSQL using a COPY FROM STDIN. On committing the transaction, we also specified synchronous_commit == off
As far as DB tables go, each day a new table was created with a timestamp in its name, then a cron script would take care of tables older than one week, aggregating data and getting rid of junk.
This setup was handling tens of thousands of writes per second without a sweat on a pretty modest server. Of course, it's less than what Redis can do, but then again I trust PostgreSQL more than I trust Redis.
Out of curiosity, were you doing a high number of reads at the same second? And if so, was it the same batch processing method you were using, or was it random reads?
I will readily admit that I am biased, A good deal of my experience has been in large high load web systems with a good deal of legacy environments. When it comes to the final end point for my data I like relational structures. Personally, I find them more adaptable to the unforeseen as far as business insight is concerned, and in the environment, that I have worked in, the unforeseen occurs daily.
For example, a marketing manager wants to aggregate data set X against Y to see the outcome. The more data I have in structures that support these unforeseen and ad-hoc requirements the more insight my organization obtain. So personally for your situation I would front cache the data in an NoSQL type structure and bulk transfer it into a relational structure at set intervals to avoid having to write logic in an application layer for each use case that comes through the door. I know that there are emerging tools in this space for the NoSQL databases, but I still find analytical and reporting easier to do in the relational world, relational models seems to lend themselves better to discovering links between data sets after the fact.
Why not store the data points in a Stable Bloom Filter and use the filter to automatically tell you the frequency of mouse movements? Could this work well? What does the data for mouse movements look like?
I am curious if your hypothetical question is a reality. :)
> If your business model implies a scale-or-die condition, then I'd say it's fair to design your solution to accommodate NoSQL, but not waste time implementing it in production. It's a horrible waste of time to design something around a Cassandra cluster that will be hit once every second.
It sounded to me like he was saying that sometimes it's actually more work to figure out how to stuff your data into a relational db than a non-relational one?
As the paper mentions, the relational model is mathematically more general than object or column one, but it is very difficult to implement fully and efficiently. Yet Oracle almost did it!
W.E.Wolfengagen, famous russian CS professor, likes to note that he doesn't know of any complete relational computation system.
Oops... Sorry. I probably projected my expectations on your answer.
Sure. When your data is not relational (or, representing it as relations doesn't help you getting to it) you should consider other forms of representing it.
I am very fond of Zope and its underlying ZODB storage (and the connectors that allow it to store documents on mostly anything else), but it's very Python-centric and Python, as much as I love it, is not the perfect solution to every problem.
Agreed, there are things I can do with CouchDB that make me absolutely giddy. And I'm not talking about performance, and I don't even want to think about what the relational model would look like.
I'm currently sitting at 2 million users generating data, and that number has grown from 1 million users in the last month and a half (100% growth in under two months).
I suppose I should've clarified that sooner. Having had some experience scaling MySQL at growing businesses, I am now trying to anticipate an explosion of analytics requirements alongside an explosion of user growth and activity caused by feature growth.
That sounds bogus. A 100 TiB Oracle database is doable but incredibly expensive. 1 TiB with the appropriate redundancy will cost you around 10,000 € if you go Oracle. Perhaps you can get a good discount if you order a large cluster right away. So we're looking at 1,000,000 € just for disks here.
Then you would have to pay for the CPUs, the licenses and the consulting (lots of it, tuning a 100 TiB system is a huge task). Add another million or two...
How much would it cost with a NoSQL solution? Don't want to turn this answer into self promotion, but I'm confident it's well under the million with better performance (assuming you don't need a relational database).
They probably give you volume discounts, you know. And would a 150 machine NoSQL cluster really compare so favorably? An enterprise-grade server with an array of 10k-15k RPM drives or SSD's and appropriate redundancy is not going to be cheap.
What about the cost of manpower to operate one vs. the other? If you can hire one novice sysadmin @ $40k, labor costs will be small, but for a 150-machine cluster you aren't going to get far with a $40k novice.
The price I gave you is what one of our customers pay, after negotiation. True, they didn't order for 100 TiB.
In 2011, you don't need 150 nodes to build a 100 TiB system.
Don't take this as an official estimate, but I think we would advise on 7200 drives and more RAM and it would probably end up with a system one (several?) order(s) of magnitude faster than what you would get with a regular RDBMS.
Good luck with that. At $100 per 8GB of ECC RAM, that's $1mil in RAM, and with 4-slot machines you'll need 3,200 boxes. 1,600 boxes if you go for dual-CPU machines with 8-slot.
First, 8GB ECC sticks are actually a bit under $100... retail. You're not going to buy almost 13,000 sticks of RAM retail, are you?
Second, server motherboards are not limited to a mere 8 slots. 16's are widely available, even 32's can be found at retail, and there are more options out there than what Newegg or whatever you're shopping at has.
In my own experience, deploying applications on traditional relational databases is a pain. Most of the applications I have worked on were applications for State government. The applications get heavily tied to the database, requiring the database to be populated to test the logic. The table structures are unnecessarily convoluted. These applications process client data (eligibility, certifications, enrollment) sometimes in the context of state data.
I have built solutions that allow me to collect data from running applications, and later inject data from these collected scenarios into the business logic for testing and debugging.
All of this to say that any objective analysis of these systems would reveal no need for a relational database. In fact, imposing the relational transaction model onto these applications is clearly detrimental to the development of these systems, the cost of these systems, the testability of these systems (though this isn't really the fault of the database technology itself), and the performance of these systems.
We should be processing the information through an application designed solely according to the needs of the application, and then selective pushing data to a data warehouse for reporting (which can certainly be a relational database).
It is almost like Ford claiming all your transportation needs require a truck, and forcing all supermarkets to accommodate driving trucks up and down the grocery aisles...
Nobody would say you don't need trucks. But the fact that trucks are sometimes needed isn't proof you should always use a truck.
This is a Google cache link to an Oracle whitepaper on NoSQL databases that jakewins posted to the programming subreddit. On page 13 the paper asks "Do you really want to be contributing to an open source effort?".
That entire paragraph (paper?) is an interesting bit of FUD:
>>The NoSQL database has been sponsored by large internet companies. Such sponsorship has added credibility to internet scale claims of the databases and an aura of "coolness." The large internet companies have been motivated by having free database software that is under their complete control. These companies have hired top developers for their scalable database implementations and to keep them running. Is your company prepared to make such an investment in an area which is not your company's core competence?
Do you really want to be contributing to an open source effort? Do you really want to bet your project on something that's on the bleeding edge? Do you want to build the superstar team required to make the NoSQL vision a reality? You are not Google.
"These companies hire top developers for their scalable (NoSQL) database implementations and to keep them running...Do you really want to build the superstar team required to make the NoSQL vision? You are not Google."
Wow - a salient remark about the state of enterprise software and what message Oracle sees resonating with management.
The reality is running software (proprietary or FOSS) is costly...A large DB operation will be an expensive overhead regardless.
I wasn't commenting on SQL vs NoSQL and which will add more value. I was making a remark on Oracle's insinuating tone "Your developers are probably poorly skilled, your not going to change that, using new tools require new skills, you'll fail at that because your not Google - don't even try"
Regardless of my personal opinions, what happened to striving for excellence?
> I was making a remark on Oracle's insinuating tone "Your developers are probably poorly skilled, your not going to change that, using new tools require new skills, you'll fail at that because your not Google - don't even try"
That "insinuation" is mostly true.
> Regardless of my personal opinions, what happened to striving for excellence?
Striving for excellence in everything is bad business.
Excellence is expensive. In many cases, the expense far outweighs the benefits.
There is some good information in this article if your ignore the Oracle marketing message. It is basically a synopsis of the past 3 years of NoSQL discussion.
What it doesn't mention is that everyone who picks NoSQL generally acknowledges these downsides. So it doesn't do a good job explaining why people still pick NoSQL.
TCO is probably the biggest factor I would have in picking NoSQL. As a (Microsoft) SQL Server customer, we (developers) all hate the cost of scaling SQL Server for each customer. Each customer needs their own resource governance. This means either separate servers or a virtual machine, both tied to a SAN. Virtualization MSSQL licensing costs are large without much material benefit due to odd licensing strategies that negate most if not all financial benefit to virtualization for small IT shops like us. By comparison, the better we understand why customers use our product and the more engineering decisions we can make (get away with?) then the cheaper technologies we will be able to use (NoSQL, etc.).
I left out the part where most of our fetch operations operate on just one primary key. The data structure is very stable and the need for different kinds of analysis has disappeared after several years of refining what the product does and should be. It's an analysis app that sets market prices to hypothetical values and simulates that alternate reality. It provides very fine-grained maximum likelihood explanations for why revenue would drop or rise.
Yes, this document is FUD. Frankly, I'm surprised they released this... The tone is: desperation.
That said, the NOSQL movement is so full of hype... It really annoys me that whenever anything new comes around, a large amount of people want to use it for EVERYTHING! NoSQL is inconvenient for a lot of what it is getting used for today... In 5 years, we'll be past that and we'll see it used more appropriately.
I think that's a bit of a misread. Sometimes ravenous adoption comes from something that legitimately makes 90% of cases easier. Once I tried MongoDB, it effortlessly became my default for projects large and small going forward - SQL would be a lot of extra work. NoSQL scaling well explains companies adopting it, but it doesn't explain the masses of individual developers and tiny startups jumping to it. That's explained by it being really, really pleasant.
Oracle really doesn't understand FOSS in general or NoSQL in particular. One of their points is that NoSQL databases don't have all of the advanced features of Oracle database completely missing that the point is not to have PL/SQL or secondary indexes but for simplicity and speed.
Oracle must be miffed: they built the first relational database (the first implementation of Codd's paper), which largely replaced hierarchical databases - i.e. NoSQL.
> Oracle must be miffed: they built the first relational database
No way. What about System R? And there were probably others of which I am unaware. Many of the now-standard RDBMS implementation techniques were pioneered in System R. This paper has a good overview: http://www.cs.berkeley.edu/~brewer/cs262/SystemR.pdf
Ah, I see; it was just the first commercially available relational database - at least according to wiki, it beat the commercial version of System R (System/38) "by a few weeks" http://en.wikipedia.org/wiki/SQL#History I read somewhere that it was previously thought of as nice theoretically but not fast enough to be practical.
I'm actually very interested in what made the relational model so commercially successful, if you happen to know more about it (it's hard to judge now, because so many other features have been added and market standardization etc). The mathematicians say it was because it's based on a mathematical model - but that doesn't sound like it would be very convincing to pragmatic benefit-oriented business buyers.
EDIT that pdf looks pretty good - I'll have a proper read of it when I get the chance - thanks
And yet at the same time, Oracle is trying to fold in NoSQL to its product line. The idea of them trying to stymie NoSQL growth and using it at the same time is laughable. They're starting to feel the pressure a little too late now.
This is a marketing piece targeted at decision making managers or architects who have to justify their decision to activist decision making managers. Which is ok, Oracle knows who pays the bills.
Sometime soon, Oracle will develop or acquire a NoSQL product and this whitepaper will be buried. It will then be replaced by a new whitepaper to get you excited about how Oracle's new product will revolutionize price and performance for your web application.
The problem with NoSQL databases is that developers still do not understand how to use them well. NoSQL requires relearning how to use a database. Many developers misuse NoSQL databases, relying on the SQL patterns they know so well. I think that's Oracle's main point here. For example, it's easy to have a lot of round trips if you use NoSQL badly, but if you adapt and take advantage of its strongpoints, NoSQL can easily reduce the query complexity.
The point of this paper is to scare Oracle's customers from switching to NoSQL, and I agree that companies should not blindly switch from SQL to NoSQL without the willingness to relearn and adapt.
Everything in computing requires learning and relearning. If you hired, or work with, developers who can't learn or adapt you have far more serious problems than database choices to worry about.
Some of the points are definitely worth considering. It's also a great resource for links on NoSQL issues. I can't help feeling this was a project for an intern however. It wasn't proofread very well.
"embrace asynchrony" - really?
"Clearly the traditional 3-tier application development model scales well and the desired development model for established non-internet companies." - there's a missing word here somewhere to make this a sentence
Oracle is correct when the bemoan NoSQL's lack of joins and relations ... so I'm using CouchDB as my operational store and exporting the data to PostgreSQL so that I can issue arbitrary queries. It took a little bit of work to make the exporter understand that there might be missing or extra fields in the CouchDB documents, but it was well worth it.
Why don't they just create a key value style API for Oracle? If people are prepared to a lot pay for a database, maybe the same people will pay a lot for NoSQL.
nice FUD (strange resemblance to the Sun's old FUD on Sparc vs. x86)
With all my respect to Oracle (and gratitude for that part of living i make from it), they would better spend these resources on implementing a distributed query execution (the parallel query implementation they have is a joke really), and it would make their position vs. NoSQL much more stronger.
Wow, this is a huge fail. It should have started with graphs based on studies of major customers that tried to make the switch and failed and why on the first 20 pages, but instead it starts off by naming the leaders in the space that use NoSQL. They've convinced me to use NoSQL now, because they couldn't just shut up. Oracle, you screwed Sun, and now you are screwing yourself.
Hey Oracle, if you want to bring this point up, please at least pretend to compare the above costs between NoSQL and your own solution. Otherwise it is just pure FUD: "NoSQL databases eat your CPU cycles, disk and network badndwith! Booooo!"