Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I wish I read this article a year ago before I started developing my latest app. I have that exact problem, where adding a column to a 10M+ row table takes over an hour. So instead I end up with "things" tables all over the place.

Sigh.




The last company I worked at made an application that used the Entity-Attribute-Value pattern in the database. The stated reason was to be dynamic, so we didn't have to worry about adding new columns and the associated downtime (assuming the DB got huge, which of course it would because this app would surely be a huge success). We had that problem on our main app (with 10s of millions of rows) where adding a column was always tricky, so I think management over-corrected. The other supposed win was that since the model didn't change, the code didn't need to be updated.

The data that was being stored fit into the relational model pretty well. But thanks to E.A.V. it was very difficult to query. The kinds of questions we often looked at (how many records from this zip code) would have been trivial without the E.A.V. Today you might use a NoSQL database (which were just starting to get noticed at the time), but in reality it fit into MySQL just fine.

The real sad part is, we never used that functionality in the 2-3 years after it was developed while I was there. The app wasn't big enough for adding columns to take much time at all. All that "flexibility" we needed? We didn't use it, because it would have taken additional time to implement the additional front-ends and update the other backend systems.

Even if we wanted to keep things smaller, we could have gone with a table per type of record (record_type_one, record_type_two, etc) instead of one big records table. That would have made schema changes easier.

And of course, the code did need to be updated. Always. Sure there were no new columns that might cause problems if they didn't have default values, but you can never make changes to an app without code changes. We still had to implement the new interface. We had to implement the code to post that new kind of record to the systems it got processed by. Making the code handle the DB changes would have taken less time than the day or two a designer might work on the front end. It never would have been a bottleneck.


Looks like you guys did a bit of a premature optimization there. I know how painful it can get :(


That's exactly what it was.

It was designed to replace a growing set of systems that were all tiny forks of the same basic code base over a couple of years. Managing all that had become a mess, and it did need to be replaced. But there wan an opportunity and we ended up trying to reach for the stars when we should have aimed a bit lower.

The irony is that after the system had been in production for a few months, we noticed it had terrible performance that was getting worse with load.

It turned out the programmer who had written some parts of the system had it recalculating way too much data, things that didn't apply to what was going on and that couldn't have changed. It was probably an artifact from initial development (I'll do it this way to get it going, then cleanup later...). Once we caught and fixed that, it was much much faster.

That was the only optimization I remember it needing while I was there.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: