Hacker News new | past | comments | ask | show | jobs | submit login

>Theres no rhyme or reason to it at the API level. You have to look under the hood...

Every abstraction is leaky. I have found that the best engineers are those who understand how things work under the hood. They can solve hard problems when they come up. People who rely upon a mountain of libraries are stumped and begin trying random things until something works.

Honestly, what is the problem with understanding how your tools work? I don't know of any other engineering profession that thinks this way.




>Honestly, what is the problem with understanding how your tools work? I don't know of any other engineering profession that thinks this way.

There is actually a huge problem with this. Human intelligence is limited and therefore much of the art and science of computer programming is targeted towards managing complexity. The fundamental concept that humans use to reduce complexity in everything from math, physics, language, chemistry and everything out there is abstraction. When look at a wall you are looking at an abstraction. To truly understand the nature of the wall you must understand that it is composed of plant cells, (wood) which is composed of carbon based molecules which at is center is all serialized as a single hereditary molecule called DNA which is itself composed of atoms that are governed by complex differential equations and on and on and on...

You don't need to know any of this to understand the wall because of abstraction. You encapsulate all that complexity into a concept called a wall. Programming is largely the same thing, with abstraction made explicit. To move forward, to produce things of greater complexity we have to move in the direction of good, non-leaky abstractions. Limitations in our intelligence will inevitably force us to become ignorant of the underlying implementations of many technologies so the abstractions around these things must be excellent. It is the only way forward and it is also arguably the reason why humans are different from other animals. We have the ability to abstract.

This concept is everywhere in computer science. The web programmer doesn't need to understand assembly language even though it's the underlying implementation of all his tools. But you are right in the sense that it doesn't doesn't mean a programmer can't learn assembly. But this still illustrates my point. HUmans have a limited ability in handling complexity and therefore we must use abstraction to handle it.

In fact, there is an area of computer science where this has become startlingly obvious. It involves machine learning. Take a visualization algorithm that can recognize objects. Currently the de facto way of doing this training a black box of neural networks. The complexity is so high that we have no idea how to program such algorithms by hand and the only way forward is to rely on these black box abstractions where we "train" the system. Even if the programmer pries open the neural network and tries to understand what each neuron is doing in the context of other neurons is impossible.

In short. To build systems of greater complexity developers must use tools to abstract complexity.

SQL has a different if not tangential problem. It's not leaking enough complexity. It's hiding too much, and producing an interface that is expensive in terms of performance. I'm proposing an abstraction that is one to one. Where if I learn the language and I understand how to optimize the language then a bijection allows me to optimize the underlying implementation as well without understanding it. To borrow the term from Rust and C++: Zero Cost abstractions.


A database is not on the same level as e.g. the CPU. Almost none of us could write assembler for a modern chip without a lot initial of research. That's ok because we have compilers, the OS, and other layers to handle that for us.

However, even for your bog standard web dev, a database is up at the top of the list of systems you need to understand in order to do your job. If you have no clue how to scale it out, how to optimize queries, hell, even how to write non-trivial queries (!), you're not doing your job. You sit there and write an N+1 on every page via your favorite ORM and no one knows why everything is so slow.

A carpenter doesn't have to be a blacksmith, but they do need to know how a saw works. Not knowing the basics of how a DB works makes you disposable, just another JS/HTML/CSS monkey. Can't solve the hard problems, but has no problem bloating every page they create beyond reason.


>That's ok because we have compilers, the OS, and other layers to handle that for us.

This is the essence of my argument. Why can't the database API occupy a layer similar to the OS or compilers?

If I don't understand the underlying implementation of compilers and OSs' why do I need to understand the underlying implementation of my database? Why is the OS abstraction different from the DB sql abstraction? Can't we write abstractions that are zero cost?

The answer is we can, however we haven't and therefore the only solutions available to us is to use SQL and understand the lower level details of the database.

>However, even for your bog typical web dev, a database is right up there in the list of systems you need to understand in order to do your job.

I'm not talking about your responsibilities on the job. I'm questioning the current state of tech. A web developer uses an API exposed by an OS and doesn't have to understand the OS. A web developer uses an API exposed by the compiler and doesn't have to understand Assembly. A web developer uses an API exposed by a database but he MUST understand the underlying features of a database. See the difference? I'm pointing out an issue that could be improved not what you need to know for your job.

>A carpenter doesn't have to be a blacksmith, but they do need to know how a saw works.

SQL is a saw that breaks in half on every use. In order to use such a saw a carpenter must have the knowledge of a blacksmith to reforge the metal and combine the two halves of the saw into a single metal blade. A leaky abstraction. That is a proper analogy for SQL.


>This is the essence of my argument. Why can't the database API occupy a layer similar to the OS or compilers?

>If I don't understand the underlying implementation of compilers and OSs' why do I need to understand the underlying implementation of my database? Why is the OS abstraction different from the DB sql abstraction? Can't we write abstractions that are zero cost?

Because it's closer to your application. An OS doesn't need to know how e.g. a Customer relates to a Business in order to read a file. Some of us do need to know the ins and outs of the CPU and OS because our application requires it. Yours doesn't, but it does require knowledge of how the DB works.

>I'm not talking about your responsibilities on the job. I'm questioning the current state of tech. A web developer uses an API exposed by an OS and doesn't have to understand the OS. A web developer uses an API exposed by the compiler and doesn't have to understand Assembly. A web developer uses an API exposed by a database but he MUST understand the underlying features of a database. See the difference?

Not really, no. As a web dev you are about 100 layers away from the OS most of the time, but your application logic and architecture is very much dependent upon the information store. I spent years working in digital imaging and automation and had to know what my compiler was doing and how my OS implemented all sorts of things because my application depended on it.

>SQL is a saw that breaks in half on every use. In order to use such a saw a carpenter must have the knowledge of a blacksmith to reforge the metal and combine the two halves of the saw into a single metal blade. A leaky abstraction. That is a proper analogy for SQL.

This is what irks me. I hear stuff like this all of the time from people who solve relatively trivial problems day to day, but for whatever reason think things should be better than they are. They, of course, have no idea what that would look like, but damn it if they don't have a strong opinion on the matter.

I'm not saying that's you necessarily, but if you hadn't noticed, these things are hard. This is the best we have come up with so far. Planes still crash, cars don't drive themselves, and my laptop crashed literally 30 minutes ago.

In my experience, the people who complain most loudly about poor abstractions, those who harp on you for not following "modern design patterns", and generally complain about everything write the worst code. I eagerly await your proposal for a better data store and application protocol that solves all of the problems SQL currently does without any of the warts.


>Because it's closer to your application. An OS doesn't need to know how e.g. a Customer relates to a Business in order to read a file. Some of us do need to know the ins and outs of the CPU and OS because our application requires it. Yours doesn't, but it does require knowledge of how the DB works.

SQL is 100% trying to achieve the goal opposite of what you say. Look at the query language. The literal intention of SQL is to hide everything from the user. EXPLAIN ANALYZE is tacked on later when people started using SQL in the wrong way causing the godawful and impossible to remove mess we have today.

I'm arguing for a language that's one to one with the implementation. Zero cost abstractions like C, C++ and Rust.

>Not really, no. As a web dev you are about 100 layers away from the OS most of the time, but your application logic and architecture is very much dependent upon the information store. I spent years working in digital imaging and automation and had to know what my compiler was doing and how my OS implemented all sorts of things because my application depended on it.

No to you dude. You're not 100 layers away from the OS. All the languages that you use to develop a web app have system calls, you are essentially 1 layer away from the OS. The reason why we can use high level languages like ruby or python is because the the database is a bottleneck and it doesn't really matter how slow the app is... But even these languages expose OS level system calls, so it's essentially the same experience as a "low level language" sans memory management.

Also the fact that you needed to understand what you're compiler was doing is important. Did you have to do SQL level hacks in order to get your compiler to do what you wanted it to do? I'm sure your compiler was DESIGNED to be low level so that you didn't see BS like UNION vs. OR.

>Because it's closer to your application. An OS doesn't need to know how e.g. a Customer relates to a Business in order to read a file.

You realize the entire initial purpose of SQL was for business questions by laymen. The things we do with it a lot nowadays are optimizations via hacks and query string construction via meta programming. I'm not saying don't understand your toolset implementation (even though in the ideal world you shouldn't need to) I'm saying there's a problem when an optimization involves a high level api hack. Literally a week ago I wrote a SQL optimization hack that improved speed by at least 3x, it involved serializing a data-structure into a string and doing string manipulation to change the data structure into something easier to deal with... before casting it back down... What other API in the universe requires you to do this? A bad one.

>In my experience, the people who complain most loudly about poor abstractions, those who harp on you for not following "modern design patterns", and generally complain about everything write the worst code. I eagerly await your proposal for a better data store and application protocol that solves all of the problems SQL currently does without any of the warts.

I have no proposal, I'm not trying to design a solution I'm saying there's a problem. I'm not a database developer. You trying to imply that I shouldn't talk unless I'm a database developer with a solution in mind? Are you a database developer?

Also I hate design patterns, I avoid design patterns like the plague.

>This is what irks me. I hear stuff like this all of the time from people who solve relatively trivial problems day to day, but for whatever reason think things should be better than they are. They, of course, have no idea what that would look like, but damn it if they don't have a strong opinion on the matter.

What's your opinion on "trivial problem"? I'm literally doing that everyday. I'm working on building SQL views for a database littered with technical debt and JSON typed columns. Literally all I'm doing is writing SQL views to cover up all this crap up so customers can query our database as if everything was properly sequestered into tables rather then JSON blobs. Day in and day out I try to poke at a query trying to coax more speed out of it using non-intuitive and illogical hacks. All I do is SQL (currently). Trivial as hell.

What do you do? Must be some super crazy dream project for solving some ultra hard problem. You talk as if you're elite.


>What do you do? Must be some super crazy dream project for solving some ultra hard problem. You talk as if you're elite.

I don't have any more time for this at the moment, but I promise you that your views are not nearly as complex as designing and maintaining a database engine. I'm sorry if that hurts your feelings. Remember; I'm not the one bemoaning "poor abstractions" and offering exactly zero ideas as to how it could be improved.


>I don't have any more time for this at the moment, but I promise you that your views are not nearly as complex as designing and maintaining a database engine. I'm sorry if that hurts your feelings. Remember; I'm not the one bemoaning "poor abstractions" and offering exactly zero ideas as to how it could be improved.

No time or more like nothing left to say?

Why be sorry? My feelings aren't hurt when you're completely and utterly wrong. You should be the one with your feelings hurt not that you've been shown how off-base you are.

Additionally who said I'm talking about a database engine?? I'm talking about the API man. I never mentioned anything about the engine itself. I'm talking about how a USER accesses lower level calls that conduct JOINS and SCANS. We don't have explicit control over the individual commands you see in an EXPLAIN, we can manipulate those calls via a SQL query. I'm literally only referring to SQL.

Man when you started speculating about what I do with "non-trivial problems," that's when you took this argument into no mans land.

The very idea you have to be an expert in something to comment on a problems is literal mental stupidity. I don't restrict myself and neither do you because clearly YOU aren't a database developer either.

Think about it this way, I'm not an expert but am I allowed to comment on the environment? About global warming? Am I not allowed to support either of these causes without a PHD in environmental science? Of course I'm allowed. As are you.

You're even allowed to speculate on my coding practices, patterns and career... "trivial problems" as you've already implied. But it's a dirty low blow when you do stuff like that, I tend to avoid it.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: