Hacker News new | past | comments | ask | show | jobs | submit login

Depending on your specific requirements, querying the data dictionary isn't always sufficient; in particular, when there is a chance for changes to the schema.

An example: in case of Debezium's MySQL connector (a change data capture platform), we need to know the table schema in order to interpret incoming change events from the MySQL binlog. Due to the async nature of this process, such event may adhere to an earlier schema version, which is different from how the table looks like in the data dictionary now. For instance, a column's type may have changed between the point in time when that change event was created and now. That's why indeed we also implemented a full MySQL DDL parser for Debezium, as DDL events show up in-stream and thus allow us to react to schema changes at the right moment and update our model of the table structures accordingly.

Now, ideally, data dictionaries were revisioned, so that indeed we could query history dictionary entries (with actual change events containing a schema revision id), but I'm not aware of any database which supports that. Touched on this briefly in a talk I did earlier this year for Andy Pavlo's database group at CMU [1].

[1] https://speakerdeck.com/gunnarmorling/open-source-change-dat...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: