The evolution of technologies goes something like this:

1. Generation 1 is statically typed / schemaful because it's principled and and offers performance benefits.

2. Everyone recoils in horror at how complicated and over-designed generation 1 is. Generation 2 is dynamically typed / schemaless, and conventional wisdom becomes that this is generally more programmer-friendly.

3. The drawbacks of schemaless become more clear (annoying runtime errors, misspelled field names, harder to statically analyze the program/system/etc). Meanwhile the static typing people have figured out how offer the benefits of static typing without making it feel so complicated.

We see this with programming languages:

1. C++

2. Ruby/Python/PHP/etc.

3. Swift, Dart, Go, Rust to some extent, as well as the general trend of inferred types and optional type annotations

Or messaging formats:

1. CORBA, ASN.1, XML Schema, SOAP

2. JSON

3. Protocol Buffers, Cap'n Proto, Avro, Thrift

Or databases:

1. SQL

2. NoSQL

3. well, sort of a return to SQL to some extent, it wasn't that bad to begin with given the right tooling.

If you are allergic to the idea of schemas, I would be curious to ask:

1. isn't most of your data "de facto" schemaful anyway? Like when you send an API call with JSON, isn't there a standard set of keys that the server is expecting? Isn't it nicer to actually write down this set of keys and their expected types in a way that a machine can understand, instead of it just being documentation on a web page?

2. Is it the schema itself that you are opposed to, or the pain that clunky schema-based technologies have imposed on you? If importing your schema types was as simple as importing any other library function in your native language, are you still opposed to it?

http://rubygems.org/gems/classy_hash

https://github.com/deseretbook/classy_hash

I wrote it for an internal backend system at a small ecommerce site with a large retail legacy.

Edit: Ruby Hashes (the base "language" used by Classy Hash) aren't easily serialized and shared, but if there's enough interest, it would be possible to compile most JSON Schema schemas to Classy Hash schemas.

I'd argue that it's not.

Your schema is implicitly defined somewhere in the business logic, and you have to first learn the schema description language in order to translate your application code into schema description code. And when the application code changes, you won't be very excited to adjust the schema again.

Sometimes it's worth the effort and makes development easier, often it's the opposite. An error message saying `error: articleId missing in sale object` is more informative than `schema error in line 4282`.

  {
    "level": "error",
    "schema": {
      "loadingURI": "#",
      "pointer": ""
    },
    "instance": {
      "pointer": ""
    },
    "domain": "validation",
    "keyword": "required",
    "message": "object has missing required properties ([\"targetUrl\"])",
    "required": [
      "targetUrl"
    ],
    "missing": [
      "targetUrl"
    ]
  }

For databases, weren't there other reasons people got excited about NoSQL databases? Not having a schema was one aspect of it, but it mostly had to do with scaling. Now people realize SQL scales just fine for pretty much most of the use cases that were getting replaced with NoSQL. And also that most data(in webpages at least) is relational in nature.

Neither is C++. But languages being designed these days don't look like Python or Ruby, they look like Swift, Dart, and Go.

Essentially JSON gives you numbers, strings and nulls so when accepted on the other side it obviously knows what's a number, what's a string, etc.

Honestly if JSON could be expanded to essentially be the same but add additional types along with bolting on new types (extensible) then I think it would be perfect for the job.

At least in my opinion.

- encoding the schema along with the message makes encodings like this less efficient.

- without an ahead-of-time schema, you don't have any canonical list for all the fields that can exist and their types. Instead this gets specified in ad-hoc ways in documentation. For example, like this: https://developers.facebook.com/docs/graph-api/reference/v2....

That URL describes a schema for groups. The schema exists, it's just not machine-readable! That means you can't use it for IDE auto-completion, you can't reflect over it programmatically, and you can't use it to make encoding/decoding more CPU/memory efficient. It's so close to being useful for these purposes, why not just take that final step and put it in a machine-readable format?

- https://github.com/balanced/balanced-api - https://helloreverb.com/developers/swagger

You can have self-describing messages without having to embed the schema in the instance. Instead you embed a reference to the schema in the message. We came up with an approach to this called self-describing JSONs: http://snowplowanalytics.com/blog/2014/05/15/introducing-sel... The Avro community do something similar.

AKA XML DTD

The usefulness of schema is inversely proportional to the rate of change. They are great for getting it right, but what's the point if it all has to change before you are done?

Rate of change is a question of fact, not personal preference.

NB. I like getting things right, choose static typing, and am developing a tooling technology to further this.

What exactly would one gain from using schemas, if I can send the value (state) of any of my static types to another application using Transit?

Interoperability with other languages, for one. The static type you defined in your language can't be used with any other languages. Schemas are static types that can be used across languages.

That's the whole point of Transit, interoperability with other languages. Having a good set of scalar types, basic composite types, and the ability to extend it with your own semantic types built recursively from the base types.

Fressian [2] supports the same types and extensibility as EDN, has a compact binary encoding, and the serializer/writer can choose its own caching strategy (so-called domain-specific caching[3]). I believe it was created to provide a serialization format for Datomic.

Transit sounds like an evolution of EDN and Fressian: make the bottom layer pluggable to support human-readable/browser-friendly JSON or use the well-established msgpack for compactness. Caching is still there, but it can only be used for keywords/strings/symbols/etc. instead of arbitrary values like Fressian -- probably a good trade-off for simplicity.

[1]: http://edn-format.org [2]: http://fressian.org [3]: https://github.com/Datomic/fressian/wiki/Caching

+ whatever internal stuff big software companies have cooked up etc.

What was so special about your use-case that demanded a totally new standard?

I hate to bring up that xkcd but it's actually relevant here.

Is it the higher-level semantics on top that allow abstraction over the underlying serialization format?

The "caching" doesn't seem to be that big of a win where network latency is high and some of the other formats can be directly mmapped, but it looks intriguing however it seems like something that could be added in a versioned binary format that some of the others provide.

Go find a format that offers everything transit does, and when you don't find a perfect match for all the goals, you'll understand why this library was created.

Cross-platform (without writing in C), self describing, schema-less, extensible, support for caching, etc.

[1]: http://avro.apache.org/docs/current/

"minimize the effort required to maintain or extend legacy systems" and "reduce the need for users and developers of a system to consult secondary documentation sources" are fitting here.

I really hate comments like this.

These guys took the time to show the world this thing they created to fill a need they had, and this comment takes a dump on it without its author first getting any experience using the system. As though the author understands Transit's purpose better than Transit's authors do.

Hey, I get it: Transit /does/ (at first glance) seem largely redundant with all the other serialization libraries out there. But before we assume that its authors spent all this time on their project because they're "stupid", it behooves us to try to understand their motivations.

In the end they're not hurting anyone by releasing this thing they built. If it's bad, you don't have to use it. There's no need to be mean or get upset.

Being mindful isn't hard.

I upvoted only after some hesitation, though, because of the unnecessary snark about stupidity. I think that's what you were downvoted for.

As an aside, transit seems dramatically faster in v8 than in firefox, at least in the versons of browsers I'm using, despite the fact that JSON.parse and hydrate are faster in firefox. Has it been specially optimised for v8?

Look at JSPerf for other serialization formats vs. JSON. They all look pretty much like this. Compare these results to http://jsperf.com/json-vs-transit/2 where in some cases we beat JSON.

I still am not seeing why the other formats fail, especially with the very limited compression that you have baked into the spec (!)

Where are the benchmarks on ie6 era browsers? And why should I let ie6 era perf direct my future data format design?

It sounds like Transit may help with this via its caching etc.? Can someone from Cognitect comment on whether this is a suitable use?

http://blog.cognitect.com/cognicast/060-tim-ewald

This means that you can for example, transmit an array of dates and not need to worry about parsing the dates in the right location when you receive them.

Additionally, it's extensible, so you can define formats for any domain specific data that you're dealing with, if you need to.

The only thing missing for me was native support for JS (but I quickly found 3rd party libraries).

I don't quite understand how transit, since it's schema-less, addresses that problem. From transit-java docs:

Object data = reader.read();

I might be missing something, but it seems I have to manually create the native classes on both endpoints and cast to those classes. Either that or I still have to manually extract values using the reader API.

What's the equivalent for JSON/Transit etc? When parsing and validating the correctness of an input, what is the standard protocol for propagating error messages laced with contextual domain information?

The two solutions I've found was: - use XSLT - use a domain specific language

[1] https://thrift.apache.org/

Also unless you can see electrons bouncing on the wire, JSON is readable because there is a program that decodes and shows it to you. It would probably take a couple of lines of code in python to cat a msgpack file.

As Alan Kay suggested, it the total and deliberate ignorance of lessons learned from our history that has made modern programming into a pop-culture phenomenon.

EDIT: I mean as opposed to something like json that has what the authors describe as out-of-band schema.

http://en.wikipedia.org/wiki/Lean_manufacturing

[["^ ","~:district/region","~:region/e","~:db/id",["^ ","~:idx",-1000001,"~:part","~:db.part/user"],"~:district/name","East"],["^ ","^2",["^ ...

  [{"~:district/region": "~:region/e",
    "~:db/id": {"~:idx": -1000001, "~:part": "~:db.part/user"},
    "~:district/name": "East"}, ...]