This looks neat, but man. I'm not a fan of code generation magic. It's part of the reason I moved away from the Rails ecosystem - even though I knew what Rails was doing under the hood, the magic started to give me the heebie-jeebies. Maybe I'm weird, but I'd rather spend 10-15 minutes writing some boilerplate for a service and control the code than trust a third-party tool to do it for me.

I also feel like this is doing too much to be a framework, in a way. Like, if it just did the code generation, cool. But add in the secret management, cloud deployment and migration stuff, and it starts to feel like a platform - which, coincidentally, it is. With pricing and everything. I can't use this to build a service that runs on AWS/GCP/my own servers - at least it doesn't look like it from first glace.

So if this were actually just a framework that could be separated from the platform stuff, I'd be willing to try it out. But as is, I'd rather write boilerplate and be able to 'go run' my service and deploy it wherever I want.

EDIT: Just found that I could deploy to another cloud provider, which is nice. I wish I didn't have to manage that via a web interface, and that it supported just building a container I could deploy anywhere.

Thanks for the feedback. I think there are many different types of magic. In my (biased) opinion I think the code generation we do is much more on the side of "how to do this is obvious but annoying" rather than on the "I don't know what's happening" side.

As for platform vs framework, my realization is that the traditional separation of frameworks/libraries from infrastructure means that it's almost impossible to improve many parts of the developer experience.

The annoying parts of building backends and distributed systems come from the integration between the two. Setting up databases, secrets, and so on all require bridging that gap. Another big area that we're investing in is providing insights about how your application works at runtime, in production, directly at your fingertips when you're writing code. That's also not something you can do without integrating across that gap.

As you pointed out in the edit, you can absolutely use Encore with your own cloud. We're working on making the deployment options much more flexible, but it's very much not another "let's make an infrastructure tool and then charge a premium on top of your cloud bill to manage it for you".

Our business model is much more about aligning the incentives with the developers. We want to provide the best developer experience out there, and by charging for that as opposed to for infrastructure, we'll happily invest time to reduce our users' and customers' cloud bills, rather than just selling them more servers they don't really need.

I will say I do like the focus on telemetry and tracing - honestly that's the thing I have the hardest time setting up in applications, distributed or otherwise.

I agree regarding magic, but I've struggled to find the balance because if you have copy/pasted boilerplate, it's much harder to upgrade major versions because you have a lot of stuff to update/review. If you've modified the boilerplate, it may be impossible.

How do you balance that?

The language should have more features and less boiler plate. In Golang code, there is first class code generation support.

Compared to Ruby or Python, there is minimal code generation outside of the initial framework setups.

One thing I absolute hate about magic in code is that it's often done at runtime. Considering the idiomatic way for go is actually have code generated and checked into git makes a huge difference.

I've been working on Encore for the past 3 years.

I was at Spotify for the past 8 years as a Staff Software Engineer. We grew frustrated with the disproportionally large effort needed to build even simple backend applications, and created Encore to solve this problem.

Encore uses static analysis and code generation to reduce the boilerplate you have to write, resulting in an extremely productive developer experience. It's a combination of a framework that combines with a cloud platform in order to simplify all aspects of building distributed systems, from how you write code to shipping it to production.

I shared some more background on Encore here [1]. Feedback very much appreciated :)

[1] https://twitter.com/_eandre/status/1381668476662734849

I've built my fair share of Go microservices and have never felt the need to develop an entire framework like this.

We have everything like CI/CD, distributed tracing and one line project generation.

It's just based on some foundation code and some simple bash scripts to find and replace some service specific things.

And then it's just the basic routing and HTTP handlers to unmarshal and marshal responses.

When all your services are built off the same foundation then creating a new one is pretty much and copy and paste job in my experience.

It's a cool project but it seems like unnecessary complexity, something I strive to avoid.

If that works for you, all the better! There are quite a few things that Encore does beyond what you mentioned:

- Automatically sets up dedicated Preview Environments for your Pull Requests

- Provisions all the infrastructure you need, both for local development as well as in production (and other environments like staging or testing)

- Provides auto-completion and compile-time type safety for making cross-service API calls, with a single line of code

- Integrates secrets management, API documentation, frontend client generation, etc :)

> it seems like unnecessary complexity, something I strive to avoid.

This. Adding complexity to reduce boilerplate seems like the wrong direction to go in (pun intended).

I like boilerplate. I can overwrite it with special cases whenever I need to, and it takes up very little mental space (as opposed to the rules needed to generate it).

I hated working in Rails because of all the magic. I love working in Go because of the lack of magic.

But y'know. I'm sure it'll suit someone. Good luck with it OP.

Curious - how do you end up handling transactions that span services?

I'm getting into go, but there doesn't seem to be a great pattern for this. Used to @Transactional on the service, so feels annoying in having to build that.

I think the answers requires more context, but for example one well known pattern is called “sagas” for implementing long lived distributed transactions.

The gist is you timeout but provide a way for transactions to resume.

Also obviously there’s 2PC but again it depends on what unite trying to solve.

How does @Transactional span services? With a JtaTransaction manager?

If you are saying about micro services here which talk grpc/http then @Transactional does not help.

If you are talking about inside one service then you have write all that boilerplates, and it will not look as simple as Java

isnt code generation just smarter copypasta? since there will be no difference in complexity during runtime i feel like it's equivalent, just maybe more powerful

what i mean is that you have complexity in your existing setup too... its just that you understand the ins and outs. and so the same applies to encore.

No it isn't.

If you did down enough, you'll find some form of code generation in your environment of choice. If nothing else, compilers, but it's not limited to them.

The largest difference between that and copypasta is - generated code can be easily updated, by regenerating it.

It's also an implementation detail. I've read elsewhere in this thread that code generation is bad and language supported features are good. I'm not sure it's so clear cut. It shouldn't matter if the generated code is mostly invisible (when it's baked inside your binary by the compiler, due to language support), versus code that is generated in the source language and then compiled.

One thing that's always bad is mixing generated and non-generated code. You have all the disadvantages and little benefit.

I don't think it is. It's fundamentally taking the code out of your control. With my services all the code is there, we can tweak it if necessary, we're not tied to a framework, it's just a barebones foundation of a Go application.

Nothing is hidden away from you with code generation. There's no in-between steps from what you write to what you execute other than Go compilation.

I think it's the wrong mentality to require a whole new set of third-party tools to maintain a Go application. It then starts feeling like you're writing Javascript, and does anyone really want that?

What is your setup for distributed tracing?

Elastic APM. As far as I am aware it's an implementation of the OpenTracing specification.

We just use the Elastic client libraries. Tell it where our APM server is, do the little bit of application set up and then view all our application traces in Kibana.

All applications follow the same basic setup. Probably less than 20 lines of code and then wrapping some HTTP handlers and HTTP clients. Then we have some application specific spans.

Any plans to switch to Opentelemetry?

Not that I know of

Personally, I am using Opentelemetry to create and export my distributed traces. At first sight it looks like they have reinvented the wheel for this project.

Hi André. Encore looks really cool! It must have taken a huge amount of work to put this together. I think streamlining from IDE to production is the future of tooling and frameworks, and it looks like you are going in that direction. I feel like that is the entire hype behind "serverless", but no one has really been able to pull it off yet imo.

I wonder what you think about K8s integration? Spotify is all K8s right? I'm guessing that is tougher to monetize. I tried at a prior startup to work on something like this, offering a PaaS like experience on K8s with development libraries (it's actually still in progress - check out https://cloudstate.io/). It is a large effort to build such a thing in a nice way.

Anyways, I'm looking forward to seeing how Encore develops. Good luck!

Hey thanks a lot for sharing this, Could you recommend a couple of resources for someone to get into distributed systems? I read Data Intensive Applications, and have been finding such sources since.

A lot of the seminal research papers are surprisingly readable! I really recommend reading them to get a good feel for the sort of problems you face at scale and good ways of handling them.

Off the top of my head: a lot of stuff from Google (Dapper, Stubby, Zanzibar). Dynamo from Amazon. Leslie Lamport's guide on TLA+. The two generals problem, distributed consensus, and so on.

Most distributed systems don't need all that fancy stuff, but it's useful to know regardless. Even the simplest systems will get weird and interesting bottlenecks with scale.

And then a lot of just doing it in practice! Find a job at a startup with some scale, and learn on the job.

> We grew frustrated with the disproportionally large effort needed to build even simple backend applications, and created Encore to solve this problem.

I'm going to sound naive but where did running Docker daemon on a VPS and orchestrating deployment with `docker-compose.yml` or `Terraform` fall short?

This seems very similar to https://goa.design/? Did you know it exists?

Did you look at whether Go was part of the problem or were you using something else and you found that Go was part of the solution?

I'm sure this is awesome but do we really need phrases like "state of the art developer experience" giving you "unmatched productivity"? Is this the new normal? Do people feel the need to add this type of crazy hyperbole to stand out? I'm sure most people who read this are technical enough to appreciate this framework for what it can achieve technically, there's no need for an open source framework presentation to use the same language as a brochure for selling a sports car.

"a blazing fast library [rocket emoji] [fire emoji]"

This has escalated from the claims of being "simple" and "fun to use", and a bunch of other highly subjective, and distracting, terms.

I think the either the term distributed systems has been watered down or perhaps my definition was too strict vs what most people think it means.

To me something is only a distributed system if it infact has stateful (even ephemerally) components that interact with each other through some sort of distributed consensus or eventual consistency algorithm.

This on the other hand seems more like an application framework for traditional top-down scale out web applications where requests flow in from the top, through stateless application tier and into a stateful, centralised database.

Arguably it's "distributed" because it's made of multiple processes on multiple machines but it does really feel like a watering down of the term.

Things I would consider distributed systems: Zookeeper, etcd, Kafka, Druid, Hadoop, Bookkeeper, Pulsar, RabbitMQ, crypto stuff, bittorrent, Hazelcast, etc. Things that distribute work and storage across components and are aware of their peers.

Not to diminish the work here, just that I think the wording is a bit rich.

A distributed system doesn’t require consensus at all. Consensus is just a famous category of problems within the field of distributed systems, and there are many different types of consensus requirements.

I like Leslie lamport’s definition of a distributed system. I’m paraphrasing but it’s something to the effect, “a distributed system is when your program crashes because of a computer you never even heard of”.

So yeah, I mean a multi-tier web app can definitely be a distributed system.

Typically people don’t think of it like that because those systems are very centralized around a database for example, but in the most technical sense a multi tier application is still a distributed system.

So is an app consisting of composition of docker containers. Message passing in general can kind of blur the lines but yeah this too.

I didn't say it requires consensus but rather that it's often present. Absence of it doesn't disqualify it because eventually consistent systems are definitely "distributed systems". I guess my quandary is between what I would consider embarrassing parallel scale out problems vs distributed systems that often do some sort of coordination even if that is loose coordination in the case of things like p2p protocols ala BTC, bittorrent, etc.

You actually did say it requires consensus or eventually consistency algorithm.

Your definition is too strict. Distributed systems are systems where the components of a system are distributed across a network. There are more issues than just consistency of data. You could have a system that doesn’t store data at all but still is considered distributed due to the nature of it being spread across a network.

> To me something is only a distributed system if it infact has stateful (even ephemerally) components that interact with each other through some sort of distributed consensus or eventual consistency algorithm.

Well you're going to learn something new! Here's the short definition from Wikipedia:

Distributed computing is a field of computer science that studies distributed systems. A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another from any system.

https://en.wikipedia.org/wiki/Distributed_computing

Any system that is distributed between multiple machines over a network and communicate with each other is a distributed system. The term is purposefully generic and not required to be constrained.

I think one typical "distributed systems problem" that comes in even with a SOA of stateless services is transactions and eventual consistency. That being said I sorta agree that a distributed system without state is at least a much simplified version of "true" distributed systems.

There is no "true" distributed system. Any system that is distributed across multiple machines over network qualifies to be labeled as a distributed system.

I'm unsure where this protection of a term for more "complex" systems come from. It's a rather simple term that doesn't require to be needlessly reserved.

We built a conference management system using an early release of Encore this past winter, and it was an amazing experience. I can absolutely recommend you check it out if you're building services in Go. The developer experience is so WELL thought out, it's a pleasure to use it.

I'd love to hear more about this. Did you deploy it to the encore managed environment, or private cloud? Which "escape hatches" (e.g., dropping down into plain http requests) did you end up using and where did you encounter friction?

Looks interesting. How do the infrastructure layer features (deployment, cloud resource management) compare to the open source alternatives? If I am already using Kubernetes and terraform, would it still make sense for me to build applications on Encore?

Currently Encore deploys to Kubernetes, and uses Terraform under the hood to provision the infrastructure. Over time we'll add much more flexibility in how you deploy the application and the infrastructure we support.

It's not another infrastructure offering but rather a new developer experience for building backend applications and distributed systems. The idea is to get the developer feedback loop to be about building your product and writing business logic.

Encore then uses static analysis to understand what your application does, the different backend services involved, and the infrastructure you need. It can then provision all the infrastructure, servers, databases, and hook everything together.

This looks pretty awesome!

I'd love to see an example using websockets. I've used Go for websockets in the past but find the boilerplate-to-custom code ratio isn't great.

Knowing Go has a solid webrtc library (https://github.com/pion/webrtc), another cool feature would be webrtc support. Tons you can build with that!

Thanks! Encore does support dropping down to plain HTTP requests which lets you use Websockets, see [1]. We hope to make it more ergonomic and less boilerplate-y in the future, but it is supported :)

[1] https://github.com/encoredev/examples/tree/main/websocket-ec...

Good to know! Thanks for the example. Looking forward to giving this a spin!

Looks quite interesting to me. The telemetry and tracing aspects are cool I think. That sort of thing is a 'must have' really but I find it is quite a pain to manage the tools for it yourself, especially on a small team.

You mentioned on another comment that you would like to introduce a pub/sub element to this. Could you talk a bit more about that? I can see how that work for communicating between services managed by Encore but could it be integrated with existing event sources?

In general, I guess it's hard to see how some of this would integrate with existing systems nicely. If you're starting from a blank slate I'm sure the devex is excellent, but what would the workflow be like if only a small percentage of my infrastructure was done this way? (Which is how I imagine all of your customers would start out).

The page didn't mention retries, exponential back off, etc. Is there somewhere where it shows how that sort of thing is handled?

I kinda like this a lot. Raw SQL in the examples. And generated and consumable api services. I will surely play with this later.

Looks pretty cool! Couple of questions:

- Is there a story behind the name?

- Out of curiosity, does “Encore Cloud” also run on AWS?

- I think a lot of developers have a mentality of “rather write boilerplate than be tied to a platform not endorsed by $bigCompany”. Do you have a strategy to mitigate this?

Good questions! "Encore Cloud" is currently deployed to AWS and DigitalOcean under the hood.

As for being tied to us, that's a big part of why we're open sourcing it. We see that Encore can rather reduce lock-in, as the code you end up writing with Encore is very cloud-agnostic, but also quite agnostic of Encore itself (by virtue of using static analysis, you end up writing mostly regular business logic).

And when you use it you can deploy the applications to your own account at any major cloud provider, so you can use your existing trust relationships with your cloud provider of choice.

As for the name, it's a bit of a wordplay :). "en-" for "to bring about [something]", and "core" for "the part of something that is central to its existence or character" (in our case backend development).

I have a very minor issue with it: The API endpoints naming scheme. /greet.Person seems quite unusual - wouldn't something like /greet/person/abc be better?

It's like gRPC in the sense that it's endpoint-centric rather than URL-centric.

At least for a public facing API i would like some more customization possibilities, but this can maybe be done with a seperate BFF / API Gateway / GraphQL gateway etc. Thanks for the answer.

Fair enough :) We're working on adding GraphQL support as an alternative approach to exposing your API, and what you suggested wouldn't be that difficult to add either.

Is it possible to integrate this with event based systems or message queues or is encore not made to be integrated into existing service landscapes anyways?

Since Encore supports deploying to your own cloud, it's definitely possible to integrate with existing systems you already have in place.

We're also working on expanding the set of infrastructure primitives supported, so that you get the Encore experience for doing more types of things. Things like Pub/Sub and message queues are the next things we'd like to add support for.

It looks like the secret management is just pulling env variables into a map? Maybe it's a language thing but this isn't really how I interpret people describing the need for "secrets management", which is more to do with secure storage and access control. A map of key/values pulled from the environment doesn't really handle any of that, right?

No, we do support storing secrets and securely delivering them to the running application. See `encore secret set --help`.

Is this something like https://dapr.io/?

I’d be interested in OP’s answer here as well.

I know dapr injects a sidecar so you can code in any language and then use their SDK to call other microservices. Encore looks to be Go centric and uses code generation to create actual Go functions, which provides much better IDE support.

I did not see this in the docs but I’m not sure if Encore supports retries with or without exponential backoffs; dapr supports both.

Any ideas how this compares with [go-kit](https://github.com/go-kit/kit)?

I like how go-kit's transport not just JSON and HTTP (e.g. gRPC and protobufs are supported).

If you look at the docs you can see that encore does much more than go-kit, like provisioning infrastructure, autoscaling etc. It seems like go-kit + kubernetes environment.

They definitely aren't 1:1 comparison.

Nice to see elixir get a but of competition in this space.

They are fighting an uphill battle to beat the OTP + lib-cluster + horde combo that elixir developers have but that autogenerated api stuff looks neat.

I don't think this is competition to Elixir/OTP. To me the USP of OTP is in-memory state shared across a cluster and scheduled dynamically, persisted through code updates. This is more like automation of the traditional stateless service - database combo.

It might be like rails in the sense of prioritizing the developer experience, but the goals are quite different. Rails is a web framework designed to build websites. Encore is a backend framework designed to build distributed systems. In practice that means the feature sets are extremely different, with very little overlap.

Apologies to the Encore team. A bad comment. /A bitter coder

What are the more modern technologies you are referring to?