The library can be re-used in other apps or services.
Then the whole damn library is unit-testable with any arguments you throw at it. Got an OS where argv may be null? You can unit test that. Got a user who decided to use --iamstupid instead of --iamawesome? You can unit test that too. Want to set up environment variables? Well that's not thread safe, but your test harness can do it before it instantiates the library object.
Want to use semver? You can. I use git commit checksums for versions and automatic tagging to semver. It's more annoying but superawesome.
We run unit tests with ASAN. Good test coverage gives us good confidence in safety.
I recently picked up a subscription to undo.io and I figure the next time I see any problem then I'll take that for a spin. I've seen trouble with gRPC and trying to debug it is infuriating.
I just used AMQP routing to handle loads, and small limited-run programs that cache credentials like netflix biological inspired systems. We were never convinced gRPC could efficiently handle the periodic traffic spike-nature of our data streams (more of n! edges in fault tolerant mode concern). i.e. we run our intake like an insect colony to handle the various architecture roles, and each process instance is only handling a few network links at a time (i.e. gets rid of threading cleanup, busy credential store hits, and error routing.)
I wish they used Erlang/Elixir/Phoenix channels to reduce the system complexity,
Used to do this in .NET for personal projects then got pulled into various dotnet projects that had their own approaches. But it was really nice. What I had hoped to eventually achieve is being able to hotswap the library somehow so I can have a server with zero downtime for updates, but never found the time to do so. I would assume I would do it via a microservice type of architecture instead.
You can just do this with nginx or whatever you have in front (IIS if you're into Microsoft stuff I guess?). Run service at port xxx1, this is your "live" port. When you wanna upgrade, launch service to port xxx2, do graceful reload of config, swap the ports around so "live" port now points to right service, graceful reload of config again and done!
In .NET for plugin management there are unloadable assemblies so you can dynamically load and unload e.g. plugins at runtime. However, it requires care, has caveats and the industry generally gravitates to other techniques for zero-downtime and/or rolling deployments where the replicas (nodes) are at first drained, then shut down, and replaced one by one or in groups but never in a way where there are none to serve the requests.
CI has a step with a bash script which takes the most-recent tag, bumps it to a new semver, and pushes the updated tag pointing to the commit going through CI.
It's all fun and games until you hit version n+1 or n+2, and often realize how slow many customers are to upgrade the library. Then there are the potential conflicts in your own dependencies. And let's not forget the occasional breaking change you introduced.
You are now sacrificing the money saved by not hosting to maintain what will likely be a growing matrix of possible versions, underlying assumptions, and my personal favorite: weird customer deployment scenarios that break your library functionality. Bonus points if you are developing in a language without strong typing, or need to integrate with a dependency manager.
Plus, if there was anything about your service that leveraged unique algorithmic improvements or some other proprietary tech, it is now at the mercy of anyone with a decompiler and sufficient time.
I suspect the author may change their mind if exposed to scaled up solutions and technologies.
> It's all fun and games until you hit version n+1 or n+2, and often realize how slow many customers are to upgrade the library.
How is that different with services? I don't develop services, but I can imagine that before breaking the services, you have to poll with your biggest customers and you don't break until they are ready to move to the new version. The alternative I can imagine is to keep providing the old services with a grace period (e.g. /v1 will be available until dec next year).
The idea is to link a thin client library into the user’s code.
Then the thick client is controlled server side by the devs.
If you design the thin and thick client intelligently, you can make a lot of changes to the system by merely modifying the thick client (which you control), without needing to update the thin client.
Problems tends to come in a few flavors, namely that this increases complexity and that some changes will always require modifying the thin client.
The biggest pitfall, though, is that you have to actually put thought into the thin client design. “Thin” is an emergent property of a well thought out client. You can’t just “touch thin_client.java” and think because you named it “thin” that it’s inherently decoupled from the service. You have to actually put thought into it.
Your example is exactly right. The service can support either v1 or v2 depending on the version of the thin client.
The main other technique is to have the thin client communicate in a relatively abstract and generic manner. For instead a “write()” endpoint that accepts an “options” dictionary.
“write()” is so generic that it probably won’t have to change, and changes in behavior can be modified by shoving stuff into dict, perhaps with a “version” field to instruct the server how to interpret the call.
When you add a new feature, you can bump the thin client to v2 and shove new options in the dict. Then the server can support v1 and v2. But also, you can modify the server to handle v1 differently. E.g the absence of a “disable_new_feature” key automatically opts v1 callers into the new feature.
I think the details are are really coupled into what you are engineering and what you want to roll out. There’s no magic bullet, and this increases the complexity of your code. Good engineering in my opinion is deciding when these approaches are worth it and how they should be implemented.
As per the article, the advantage of a library over a service is that you don't have the burden of maintaining the said service. How having a thin client and a server helps?
I also don't see how "shoving new options in the dict" can help. From my POV, if the client needs to be updated to benefit from the new features, then there is no way around it, work has to be done. From there, I much prefer having sensible name, parameters, etc. from a library that can be leveraged by static typing rather than a documentation for a JSON API (I know API-as-specs exist but I've never used, maybe that's why I think that).
I understand it may be different if the software is distributed through the package manager of linux distribution because you have to follow what's provided by the distribution. But for a commercial software you can probably bundle your own libraries in executable or compile it statically.
But I am not familiar with the difficulties of the above process so I am probably missing something.
Many changes are believed to be non-breaking and so you can be running only one version in prod. (Most of the changes believed to be non-breaking are non-breaking.)
With a library, you end up with many different minor or point versions running without control over it.
You have the same problem with services and libraries when you’re introducing a breaking change. With services you can make non-breaking changes like security patches on the server side, without needing to coordinate with a customer.
That's not a great example given OpenSSL versions are famously backwards-incompatible. Older versions get ABI-compatible security patches because people put in the time to backport all of them to every version still supported, in many cases by distro package maintainers. It's exactly the situation libraries should generally avoid, because there are maintainance costs for older versions of the library as well as migration costs for all of its users.
The point is that someone is paying the cost of maintaining the library's ABI surface area even if it's not you. In a thread specifically for developers considering how to offer an interface to their software, I think it's only fair to recognize the costs of those different approaches. In that regard, OpenSSL is an extremely bad example, or an extremely good example of what not to do.
Simple answer is YOU control the deploy of a server in miroservices, so YOU decide when it goes live. You aren't playing spreadsheet telephone at scale to get everyone even im an internal org to update. If you're smart you have your callers send version information and you usually just add fields, only removals or renames cause version issues. If you're having everyone use a library they control when they update the logic and you have less telemetry when some wierd corner of your company is using out of date logic.
You are wrong by pushing your changes on "slow to update" customers. Customers know when to update much better than you do. If you are trying to update before they want to, you are doing them a disservice.
This attitude is inspired by the desire to sell more, and often times, it means to sell more unwanted crap, where customers are trapped by the "package deal", where they are either forced to update to gain useless features and the headache resulted from various inconsistencies and defects coming from the last update, or they are unable to get the product at all (because the provider cancels support or wouldn't sell older versions etc.)
In other words, you with straight face describe some really shady practice and you don't even realize how bad it sounds.
> This attitude is inspired by the desire to sell more, and often times, it means to sell more unwanted crap, where customers are trapped by the "package deal", where they are either forced to update to gain useless features and the headache resulted from various inconsistencies and defects coming from the last update, or they are unable to get the product at all (because the provider cancels support or wouldn't sell older versions etc.)
Selling more is how service providers are able to sell their generic service at a lower price than the cost you would incur by building it in-house. In other words, being forced to upgrade to accommodate features that other users want is the price you pay for sharing the development cost with those other users. That's not shady.
You assume I wanted to make the update in the first place.
I invite you to review roll outs of security updates. These are rarely about pushing new features, and may have nothing to do with my own code at all. It might just be a version bump to my dependencies config (if a dep manager with shared libraries is involved), or just a refresh of my flatpak equivalent. But, either way, shit breaks in weird and wonderful ways, and there is little I can do but wave my arms frantically at customers.
"But this assumes that slow-to-upgrade users can have negative effects on everyone else. If one user can't have a negative impact on other users, then you don't care if some users are slow to upgrade; they're only hurting themselves."
There's still the support issue, I agree. If a customer paid you money and they are on version n-10, they still expect support.
> Plus, if there was anything about your service that leveraged unique algorithmic improvements or some other proprietary tech, it is now at the mercy of anyone with a decompiler and sufficient time.
This is a valid point. My answer would be: it's all tradeoffs, but if your secret sauce is so valuable that it would be worth decompiling and can't be protected with decent pricing, license terms and lawyers, run a service.
There are plenty of technologies that are not worth decompiling for your average business customers.
> It's all fun and games until you hit version n+1 or n+2, and often realize how slow many customers are to upgrade the library.
Given how frequently needless breaking changes are made, or features are removed and paywalled, I would consider this a feature rather than a bug. Sometimes I have higher priorities than working to support someone else's breaking changes.
Two years ago, I was using an official library for interfacing with a video chat service, and they decided to break the underlying API without updating their library, so I had to rewrite the library myself.
Yes you need to have n versions you support in parallel in the wild. One per released incompatible binary API (so in semver 1.x, 2.x etc). We support about 30 libraries with 5 ppl spread around a fortune 1000 org here and there. It’s quite tenable, we are just super strict regarding ticket policy and release notes, so we always know what’s released where. Just have a system. Stick to it. Keep things organized. Works like a charm. If someone wants a service out of those someone just packages them to docker and puts them to backemd somewhere. We can focus on pure business logic, someone else maintains services etc. Really nice setup.
Services usually depend on databases. Libraries usually don’t. Either you need to support every storage backend your users might have, require them to write an integration layer from your generic hooks, or expect them to provision and manage new storage when using your library. In any case you are asking them to do a lot more work (manage the data) and in some sense breaking encapsulation by making them responsible for this.
It's only an unreasonable amount of work if you assume that the user is managing a separate storage backend for each library. If you take the Tim Berners-Lee approach (re: https://solidproject.org/) then each user is only managing one storage backend: the one that stores their data. The marginal cost of hooking in one more library to the existing backend is low.
We just have to get a little more fed up with all of these services and then the initial cost of setting it up in the first place will be worth it. Any day now...
I think most interesting web services are providing structured access to the same data for multiple people. A private, individual data silo wouldn't get the job done unless combined with some kind of message-passing. A silo to which users can invite peers is interesting, but it's an important characteristic of many web services that the specific read and write transactions allowed are application-defined... you don't actually want to give your collaborators general read or write access at the storage level.
For example, it's important that I can add this comment, and I can't delete your comment, but the moderators can. The "storage" software would have to know something about the business logic of a web forum to make that happen.
I think we just need smarter browsers which can be configured to know who we trust in which dimension.
If I want to leave a comment on an article and then delete it, I can publish the comment in my pod, and I can also publish the deletion. If you've got your browser in a mode where it's interested in my comments, it can pull in the data from my pod and render it in context with the article--whether or not the article's author cared to provide a comments section.
If you drop the idea that anyone is authoritative about how it all comes together on the viewer's screen, you can also dispense with the headaches of being that authority (e.g. services).
Cloud providers are (counter to intuition about their lock-in incentives) improving the space of this. Lots of tools now allow configuring storage by just pointing to various cloud stores, most often a S3 compatible api, not exclusively though.
K8s PersistentVolume is another decent shot at storage abstraction, only a bit raw.
Finally, more and more tools expect you to have a Postgres they can plug into as backend.
All above assumes you want to treat the library data as a big unknown blob. Once data start being corrupted and need bespoke repair, things are less fun. Access and retention is another fun rabbit hole.
This is interesting, makes me wonder if a "dockerised" database is something people could use. I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform.
> I mean a database frontend with its own language/protocols/whatever that allows you to define the data structure but leaves the specific storage engine or format as a backend detail that can change from platform to platform.
Nah. It's not that. We lack a concept that can organize storage. Let me illustrate this.
So, until some years ago there was complete nonsense and anarchy in Linux networking management. That is until we got the "ip" program. There's still nonsense and anarchy, because the "ip" program doesn't cover everything, but it's on the right track to organize everything Linux knows about networking under one roof. So, vendors today, like, say, Melanox (i.e. NVidia) choose to interface with "ip" and work with that stack rather than invent their own interfaces.
When it's extendable in predictable and convenient ways, user will extend and enrich functionality.
Now, compare this to Linux storage... I want to scream and kill somebody every time I have to deal with any aspect of it because of how poorly mismanaged it is. There's no uniformity, plenty of standards where at most one is necessary, duplication upon duplication, layers... well, forget layers. Like, say, you wanted a RAID0, well, you have MD RAIDs, you have LVM RAIDs, you have ZFS RAIDs, you have multipassing with DM (is that a RAID, well sorta' depends on what you expected...) also, well, Ceph RDB are also kind of like RAIDs, DRBD can also sort of be like a RAID...
Do you maybe also want snapshots? How about encryption? -- Every solution will end up so particularly tailored to the needs of your organization that even an experienced admin in this very area your org is specializing will have no clue what's going on with your storage.
Needs can be studied, understood, catalogued, rolled into some sort of a hierarchy or some other structure amenable to management. We haven't solved this problem. But we have an even bigger one: no coordination and no desire to coordinate even within Linux core components, forget third-party vendors.
Adding two numbers together takes on the order of a nanosecond. Doing the same thing using a rest / http service (like an idiot) in the same datacenter takes on the order of a millisecond. Six orders of magnitude actually.
I'm pretty sure the parent comment was about storage media, not about network hops and service boundaries. Also, extremely basic REST/HTTP services definitely do not take 1 millisecond even if you are bad at software - the overhead of that stack is in the tens of microseconds if you are doing nothing.
For the comparison being referenced here, if you want to compare RAM, the storage medium that backs compute, to modern persistent storage, here it is:
* 40 GB/s per DIMM vs 5-10 GB/s per NVMe SSD. At most one order of magnitude off, but you can pack enough disks into a computer that the throughput ratio is almost 1:1. AWS EBS is about 1 order of magnitude different here, and that is with network-attached storage.
* 100-200 ns latency (RAM) vs 10-50 us (fast SSD) - about 2 orders of magnitude, but also possible to hide with batching.
I blame the SQL "standard". It's a massive, unnecessary abstraction layer that only complicates attempts to build bridges between code and relational databases (which I believe is the most general-purpose paradigm).
Personally, I am working on a modern Python ORM for PostgreSQL and PostgreSQL alone.
Isn't this what SQL is supposed to be? You bring the DBI for your database and plug it into the app. Shame that it doesn't work out so well in practice.
The title of the article literally says "where possible". You found a case when it's not possible, and decided to argue against that...
No, not all services come connected with a database. Alternatively, often times a database is an artifact of tenancy and the need to manage users which would not be needed, had the functionality be exposed as a library.
More importantly, whether users realize this or not, a library is more beneficial for them than a service in majority of cases. Much in the same way how it's almost always better to own something than to rent it.
Just to give some examples of the above: all the Internet of crap stuff, all sorts of "smart" home nonsense which requires that you subscribe to a service, install an app on your phone and send all your private data unsupervised to some shady Joe Shmo who you know nothing about. To be more specific, take something like Google Nest thermostat. There's no reason this contraption should ever go on the Internet, nor should it require to know your street address, nor your email etc. In fact, the utility it brings is very marginal (saves you few steps you'd have to make to reach for the boiler's controls to program it). It absolutely could've been designed in such a way that it doesn't connect to the Internet, or, at least, to never leave the local area network, and yet it's a cloud service...
I'm maybe naive, but is it not possible to supply a repository interface for the user to implement? Bring your own glue?
The library uses only the interface to work with whatever orm/db connector exists in the client project.
If services at any given company all use a standard db library, it could even directly interface assuming your using that. I don't think we're talking about public apis and packages here.
Great point. In my opinion it is possible and maybe even ideal to do both: make it easy for anyone to run their own service while also running your own service so that users have the option to not have to manage the data, patching and ops side.
SQLite alone doesn't help you get away with any of that. If you already know that a single machine is enough for you, then sure, SQLite is a fine choice. Availability needs alone often force you to run multi-node setups.
Most private software is, but there's an awfully large amount of publicly served software that can't fit into this model (also, any software that has network effects like twitter).
Is this really a common scenario where there's a choice between these 2 options that isn't obvious? I've never considered libraries and services to be two equal options of distributing functionality, and you just pick one of them. It's usually a function of practicality and monetization.
In my experience a new micro-service considered the default option for anything which could be done as a micro-service nowadays. Library API design is the lost art (almost). And the choose is not always obvious for non-technical reasons. Consider a following example: you have 3 micro-services X, Y, Z which need to interact with a platform G but this interaction requires non-trivial chunk of code. It can be done as a library or as a new adapter micro-service A which will encapsulate knowledge about platform G and will interact with X, Y, Z the way it would be easier for X, Y, Z developers to integrate. Micro-service will add network latency but will allow to make/deploy all changes only to service A. With a library one would have to test and release a new version and then ask X, Y, Z maintainers to switch to this new version. In some organizations it will be a very slow process because X, Y, Z can put an update request (from the library team) at the bottom of the backlog. With a micro-service A teams X, Y, Z would have much less power to stop/slow development.
This is making technology objectively worse to solve people problems, and this only expands.
More software needs iron clad leadership and control. Any organization that lacks this can’t help but produce shit software. There has to be a single person with real decision making power than can force upgrades and prioritization of work, and they need to be able to axe people or teams who can’t hack it.
OK but most business compete in a market. And the people they employ are in a labour market.
It seems that worse is better.
What we see is just another manifestation of the current economic paradigm. Waste is winning.
Making too many pairs of jeans so we burn tons of perfectly good new pairs daily wins.
Throwing hundreds of engineers on problems bashing out hundreds of thousands of lines of code sending megabyte messages between dozens of service instances in multiple kubernetes clusters appear to be a winning move. Otherwise someone would beat them, right?
Most engineering orgs don’t understand or feel “the market” until mass layoffs. They exit their companies with beautiful resumes touting micro services and k8s.
Worked a place where we switched from having a client library to making the other teams use our APIs directly themselves. Of course, we had to do the switch for them because they were too busy... Pretty much just copy-pasted the library code directly into their projects.
If you're distributing something publicly, it's fairly obvious which to pick, yeah.
It's less obvious for internal systems and architecture. For example, your company wants to add domain-specific auditing to all of your existing services. You could have every service add a library dependency that lets them just call `auditor.log(...)` and the library internally writes to storage. Or you could add an auditing service with a full HTTP/GRPC API. Or you could go halfway and build an auditing service but provide a library that acts as an interface.
There's no right answer for this IMO, all those approaches have pros and cons.
This was a really common dilemma at Amazon. The prevailing wisdom was opposite to the advice in the article though. Unless you had an exceptional reason, your functionality and data should be exposed as service, not a library.
Oh, absolutely. Service brings convenience to the backoffice and money! Libraries suck to support and hard to sell.
Of course users want libraries but vendors want services. There are plenty of examples where something could've been a library, if the vendor had user's interest at heart, but instead it's sold (or rather rented out) as a service.
Go to Amazon marketplace, for example. Virtually everything there is a product that should've been a library but is sold as a service...
Exactly. I've never run into a situation where there was even a choice.
Is it something that relies on a private database, queue, massive processing, dedicated hardware, shared state, something geographically distributed? It's a service out of necessity.
Or is it just a bundle of quickly executing code? Then it's obviously a library.
I've never seen anybody try to turn leftPad() into a service.
I've seen plenty of engineers try to turn trivial functionality like that into a microservice "to avoid version upgrade hell" (quote from one person in a previous job), or bundle what should be a simple self-contained (i.e. no-dependencies) library of functions (i.e. an API) into a REST/gRPC interface service. Microservices fad-following is as bad as TDD in this industry.
You can also start calling your library - a "distributable, embedded, natively consumable micro-service" and then other folks start using it. (Believe it or not - this is what some teams in my org started calling a traditional library)
> You write a library, then wrap a thin service interface around it. Distribute the lib as needed. Publish the service as needed.
At $CURJOB, we did this but at a higher level of abstraction (an authentication architectural component, rather than a library). I think this is what the author means when they say "writing a standalone server reached through a network protocol".
We see a lot of folks who like the flexibility of consuming functionality as a service or library, as they see fit. We've even had customers who said "we chose you because now we want you as a service, but later will want you as library" or vice versa.
Flexibility isn't free, though. Versioning, support, backwards compatibility (features and performance), even offering the service all become more complex.
I take "where possible" to mean that what you describe should be considered an exception instead of a rule, which I agree with. I do have some thin services which are libraries, but above and beyond directly importing libraries is preferred. With edge functions becoming more popular, this also seems to be the preferred pattern, having "fat" edge functions with shared code, vs many small edge functions calling each other.
Libraries and services both have maintenance costs and upgrade impedance from clients. The costs might differ, but in my experience work out to about the same overall. The correct way to determine whether a piece of software should be a library or service is by examining its intended purpose and its dependencies.
- If the software is dependent on another service or a data store, it should be a service: this provides the owners freedom to include error handling and observability that is appropriate for the service and provides protection for its dependencies against unbounded access (via observability at minimum, or human-organized contracts, etc.). Examples: software to retrieve user data from a database, software that aggregates data from a user service and an inventory service to produce a purchase history
- If the software is self-contained, e.g., it does math or "pure business logic" algorithms, it probably should be a library: performance can be optimized for one or a small handful of common use cases, error handling and observability become the responsibility of clients, and neither owners of the library or clients of it have to concern themselves with the impact to transitive dependencies (e.g., load added to a database). Examples: software that transforms user input into internal serialization formats; software that validates data, encrypts or decrypts data, or otherwise is "purely functional"
It’s not an either or. As argued elsethread, it should arguably always be a library (though not necessarily a published one), and optionally (if needed) also a service that wraps the library.
If the code is included in the service, it's not a library. If it's packaged as a library and the service code is a (minimal) wrapper around it, it's a pointless additional complexity.
I think this sentiment is a partial cause of the trend towards self-hostable, downloadable software too.
The customer has a cost when they operate a library instead of consume a service, no doubt. They also get more control (no surprise upgrades, availability is their responsibility) and assurances (no worries about the service suddenly being end of lifed).
Yes. And if an updated library breaks you, you can fix it easily by rolling back to the earlier version of the library. If an outside service breaks you, you're hosed until they fix the service or you rework your code to route around the breakage.
Exactly. It'll break sometime (it's software). So when do you want it to break? And is the benefit of controlling/planning the timing of that breakage worth the cost of operating the software?
As always, it depends. Questions I'd ask:
* How critical is the software to your application's proper functioning?
* How big is the team?
* Who are your customers and what are customer expectations around your application's proper functioning?
* How often does the library change?
* What expectations does a service set around backwards compatibility? What commitments are made?
>When you use an outside service, they might break your system at a time of their choosing.
Well sure, but if this is important to you, you might consider a contract with the company that lays out cases and conditions where breakage is acceptable.
Not for any specific benefits, though these exist, but for architectural reasons. You can trivially wrap a library into pretty much everything else, but not the other way around.
In fact, on macOS/iOS I put all functionality in frameworks, as these have structure and can thus contain other frameworks and non-code resources, which is more difficult with a library.
Yeah but then who is responsible for maintaining the library? If you have a bunch of internal teams that depend on your library and there is an issue or a feature request, you're back to the same position of being the one that does the work to implement. Better to have a service IMO, you can get telemetry out of it and scale it out or replace internals without having to worry about who will get affected.
Because if you ship code to users (library), you lose control. You may want to change something, but your users will just tell you to go pound sand. Conversely, if you keep the implementation on your side (service), you get to control how things work and when things change, and your users don't have a say in this.
I don't see the issue here. If I release my_fancy_lib 2.0.0, and I have users who only ever want to stay on the last 1.x release, that's fine. It's no skin off my nose if users choose to stay on an old version forever.
I'm writing from the perspective of the user (which may be a developer using your product in their product). I don't really care if you view software as a knife or as a noose, I don't want to be coerced by the threat of either.
Many engineering organizations have "platform teams"--teams that have as their sole responsibility maintaining shared libraries and "core" services for the entire platform.
As opposed to being responsible for maintaining and doing operational support of the service...
I still don't get what advantage exactly you are expecting.
Scaling it is only ever a problem for the service; you don't even have to think about it in a library. You can get telemetry from a library just as well as from a service; your users may find that a bit invasive, but it's still way less invasive than calling your service. And you can replace the internals of whatever, without having to worry about who will get affected, that's what defines internals.
In most cases, I write as a library. It makes life sooooooo much easier and can be tested at different levels with unit tests.
I will always remember a relatively large GUI application... one big EXE project. All content was hard coded inside Windows/Forms, Button Click methods, etc.
I started to break the GUI app down into smaller components. OK.. It has (A), (B), (C), (D), etc. Started to build isolated libraries. I could then test it without having to run the GUI app and get to the respected Window, etc.
It made life sooooooooo much easier.
Now, it was a bunch of libraries which the GUI apps includes and wraps with presentation. The great thing is that we need to upgrade our tools, opening the door to using a more modern GUI -- as everything is in their own library.. this becomes so much easier to do.
This was a C# application using WinForms using .NET framework (v4 something). We upgraded to .NET Core. Once the libraries had upgraded, we decided to use WPF. It was a relatively pleasant experience. Now imagine trying to upgrade .NET version + WinForms to WPF as it currently stood -- all scattered inside the GUI app.
I also tend to advise people not to write so many services -- a generation of developers has entered the workforce thinking "cloud native microservices" are the only way to factor software of any kind -- but I also caution people not to make everything a library either. Libraries have maintenance costs that have to pay off overall, and I consistently see people underestimate those costs when first making a library.
Any library used by more than one project faces friction making almost any kind of changes, especially if the ideal form of the change breaks backwards compatibility even a little bit. Even if it's used by only one project (an unfortunate antipattern some teams fall into), it still introduces friction to individual changes, which is worse in some languages than others because even just testing against a WIP version of the library might be a whole tangle that everyone seems to reinvent "workspaces" to try to mitigate even a little.
I now urge people to keep closely related projects in a single repo with purely internal libraries, so you can make backwards incompatible changes because you fix them for all consumers within the same commit that made them. This is like a monorepo but only for closely related projects.
It cuts out a lot of the friction of updating shared code, so it's a good way to avoid tech debt and keep both the library and its users evolving. This was inspired by working in an actual monorepo for years, but avoiding the size of repo where you can no longer make every related change in the same commit.
When an internal library's API surface has proven to work well for several projects and for long enough to be considered stable, then it can be spun off as a standalone library, and whatever warts slowly form there are probably acceptable in return for the wider reuse.
I had the same opinion in the past. But after some more years of work experience - mostly in the managed services area - I don't think it's that clear anymore:
- If you offer your users a library, you are competing with tons of open source libraries which claim to offer the same thing. A lot of those will be incomplete, buggy or insecure. But most potential users will never know and try to get them work instead of looking at your offering.
- If you are offering a library, debugging and user support can at times be challenging. Do you expect user to look at the internals (source code) of your library? Provide core dumps?
- Having to support N different [major] versions of libraries can become challenging. It's hard to know when all users have upgraded. With a service, you can control the update schedule. Even though changes to public APIs of the service certainly are still problematic.
- Write a library - but in which language? You might prefer Rust, but your users might prefer Go, Java or Python. You could write the core in one language, and add wrappers in other languages, but some users will still be unhappy with it (e.g. because it makes compilation difficult, people don't want "unsafe C code" in their high level language project, or the wrapper might slow down performance).
- Libraries which are general-purpose and are not targetting a specific application/service can over time become feature bloated since things are added "just for the case that someone might need it". This makes them hard to maintain. And since there's no feedback/telemetry, it's also hard to say whether something can be safely removed.
Note that all of this doesn't mean "don't write libraries". Even if you write applications/services, its good to structure internal components into libraries. It's mostly about "what is preferrable to offer for users".
I hope GraalVM's Polygot runtime will eventually allow more companies to go with libraries over services in more situations for multi-language businesses.
And Polylith for better reuse across projects generally.
I wish I could nuke GraalVM from the earth. It ruined my life for 2 years and I will never forgive it. It's a very stupid idea, pushed by CTOs that think Java's "write once, run everywhere" is still relevant. It literally runs every language slower and introduces all kinds of build pains.
Ugh, I can't believe I had to read your comment this morning. smh
this sounds very nice on paper (it sounds common sense trivial utopia that everyone sets out to do anyway, but somehow life, entropy and deadlines get in the way), but ... is there a bigger real life project using something like this?
"where possible" is a keyword here and easy to argue about. For example:
* my logic requires a database
* my logic should be isolated for compliance reasons
* my logic accesses a service which is not available for everyone
* my logic should be processed async and needs to store state, how do I make sure library owners have that environment
* customers embedding my library are not upgrading it frequently, which leaves us to support 15 years of libraries
* ...
Imagine a scenario, your service is a low traffic, but service which embeds you as a library is a high traffic with many instances and always opens DB connection.
Why should you optimize your Database for high traffic use case, when your use case is really a low traffic?
And then repeat this for 10 other libraries and library owners. Everyone is optimizing for nothing.
make it even more difficult, 100 different types of services with different traffic patterns are embedding your library with different behaviours when it comes to managing DB connection state
Scenario: your service is accessing a service which exposes PII data and you only process them.
Service which embeds your service enabled audit logs of network requests and made it visible to everyone in the company. You have created a risk unintentionally
At an enterprise scale shop the service model will always win as it will better match the organization structure and avoids the maintenance burden of having to keep a dozen or so libraries in sync. Enterprise shops never standardize solely on a single language as they are filled with exceptions and the new hotness.
Everyplace else I'd say it will vary but I agree with the general consensus that shooting for the model of a library that can be easily run as a service is your best approach.
Those aren't "either or" things. Usually services are exposed through libraries. And a library requires a service if it has a canonical or centralized storage or processing, which can't be done locally.
Now, sure, we do stupid things as services, for sure. But people often do it to monetize the service, or control the users. And so that'll never change.
I recall someone making an "async HTML5 AJAX blink service" as a joke for this trend.
I think this is roughly true, but at a BigCo it’s not really feasible/easy unless you have a monorepo or otherwise extremely good build/integration tooling to deal with many repos (though Go can sort of deal with this)
The issue is coordinating changes (and, god help you, library releases) across repos is often an utter nightmare with multiple PR/merge builds
> it’s not really feasible/easy unless you have a monorepo or otherwise extremely good build/integration tooling to deal with many repos[...] The issue is coordinating changes [...] across repos is often an utter nightmare with multiple PR/merge builds
I'm coming up on a year out of $BIG_TECH_JOB (where the idea of a monorepo was horrifying) and transferred to $WAY_SMALLER_NON_TECH_COMPANY_WHO_USES_TECH_JOB (where the enthusiastically use a monorepo), and have been really confused by repeated claims like this. I really feel like I'm missing something here, because lots of obviously-smart-and-experienced folks repeat it. I'd really appreciate it if you could check my understanding and see what I'm missing.
(To be clear, here I'm assuming that a monorepo is a single repository which contains conceptually-distinct-but-related projects - things which _could_ justifiably be their own repos, but which are kept in one repo for reasons of maintenance and managements - and wherein the build system is such that every sub-project within the repo uses the same package dependency tree, i.e. if ProjectA and ProjectB in the monorepo depend on LibraryZ, then the versions of Z that A and B depend on must be identical for any given commit/build. If I've misunderstood that - if that's just straight-up not what a monorepo is, or if it _is_ but with some extra nuance or sauce - then I guess we can short-circuit the response pretty quickly :P )
Here's how a release of a breaking change of a library would work in a polyrepo world:
* I publish v2.0.0 of my library
* Consumers of that library are notified that a new version exists (via automated email notification, Dependabot, whatever)
\
* Anyone who _wants_ to update can do so (independently and at their own rate) - anyone who's comfortable staying on the old version can do so
** If we really want to make things easy for consumers, we can make automated PRs against their repos (this would be the "extremely good build/integration tooling to deal with many repos" you refer to, I suspect? Something like Spotify's FleetShift[0]) to make the change - though in practice this is probably way more trouble than its worth, I've only seen it done for serious security vulnerabilities where a) everyone in the whole damn company has to b) upgrade RIGHT THE FUCK NOW.
* Time goes by
* v1.x gets deprecated
* Anyone still using v1.x gets notified that they are using a deprecated version and strongly encouraged to update
* A little more time goes by (not much!)
* Consequences Occur for people still using the old version - this could be a visit from your friendly InfoSec enforcer, or automatically failing builds, or the removal of v1.x from the package repository (which will indirectly cause failing builds), or...
Conversely, with a monorepo, the situation seems to be:
* I publish v2.0.0 of my library
* Every single team whose code is in the monorepo must coordinate to make the changes to consume version 2. This change, by definition, proceeds at the pace of the slowest team - if one of them is underwater on oncall, or has their only competent engineer on PTO, or has some quirk of implementation (or dependency on old feature) which means they can't update for two months, then _the whole dang monorepo_ is staying on version 1 for two months; no ifs, ands, or buts
** But, yes, if you want to a wide-ranging refactoring to change all the in-monorepo consumers to use the new method, then yeah, modern IDEs are going to have _some_ functionality built-in for that within a single repo. But - by virtue of being a breaking change, it's pretty likely that the change-at-call-site is going to be more complex than simply changing the type or name of the method being called. Maybe the returned object needs different methods called on it. Maybe the method requires an extra parameter (which refactoring IDEs can add to the actual callsite, but cannot implement for you _fetching and providing_ that parameter). At this point, either humans are going to have to comb through the changes to finalize them (at which point, you lose the claimed advantages of being in a monorepo where changes can be done by tooling), or you're going to have to implement some sort of code-parsing system to make correct changes throughout (which, again, is approximately equivalent to the automation work required in the polyrepo case)
So: while, yes, it _would_ be a hassle to "coordinate changes across repos" (though _significantly_ less to coordinate across a single repo-to-repo boundary, especially if both are owned by the same team, than it would to coordinate a whole monorepo's worth of interactions), the joy of a polyrepo situation is that _you don't have to_. Consumers can update when they want to, at their own pace - no coordination necessary! So - yes, there will be a greater _volume_ of PRs required in a polyrepo situation, but a) each of them will be way simpler and the total volume of work will be , b) they are independent (so teams who don't want to update do not hold back those who do).
But - people keep saying "it's easier to make wide-ranging changes in a monorepo", so I _must_ be missing something. What is it?
What I’ve heard from friends working at places where the monorepo model has worked well, is that it also involved a culture shift such that the burden of upgrading the consuming services is put on the shoulders of the _providers_ of the library, rather than the consumers. This implicitly brings along some benefits, like making the providers not cause excessive/unnecessary version churn and ensuring easy upgrade paths, because they experience the pain rather than it being externalized onto the consumers :) Also, good tooling helps—and I think that this model also encourages investing in that.
Wow, that is certainly a _huge_ culture shift. It certainly explains why it's not sitting right with me - while I do think that "providers not caus[ing] excessive/unnecessary version churn and ensuring easy upgrade paths" is a good thing in isolation, it seems like it would be massively outweighed by the downsides of forcing a library provider to maintain familiarity with all their consumers' codebases and business areas. Seems to me like that would make it impractical to make any changes to a library that is consumed by more than a few other teams - which runs totally contrary to the intention of extracting _commonly_-depended-upon logic. The ideal situation would be to create a library that is _so_ popular that maintaining that level of familiarity is literally impossible.
There are already incentives in place to ensure that a library provider is doing right by their consumers (building features that they want, making upgrades not too arduous), because if they make a hard-to-use library then people won't use it which should show up in however their team's success is judged. Seems to me that going the extra mile from "you have to provide a good desirable library" to "and you must also be familiar enough with your consumers' services to do the integration/upgrade work _for_ them" gives no new upside but all downside - service teams _already_ know their domain area, why should someone else have to?
(I recognize that you're just reporting what you've heard, I'm not arguing _with you_, I'm trying to reason out the arguments in my head to get them straight)
> forcing a library provider to maintain familiarity with all their consumers' codebases and business areas
In my experience, this hasn’t been an issue—the library authors are inherently intimately familiar with how to accomplish transparent upgrades of any of their library’s API changes in a consuming codebase. Or if there are breaking API changes that make it impossible/infeasible to maintain the current consumer behavior, that’s a very useful signal that it’s time for the library producer to go back to the drawing board because the proposed library update has issues—before it’s “too late” to rethink because the library update was formally released :) Or at least, helps trigger a conversation between the producer and consumers to see if the breakage can be accommodated without too much heartburn, or if it’s a non-starter and a pun upstream solution that works for both parties needs to be discussed/implemented.
Overall, it really reduces the “library producer in their ivory tower releases changes without fully considering the downstream impacts, causing a crapload of burden/schedule slip for XX teams” issue. And when you multiply that by N library producers in a company of a given scale, it’s easy for app/service dev teams to have a lot of their time non-productively taken up just trying to keep up with the constant treadmill of poorly handled upstream changes.
So removing the externality effect by having the producers have some “skin in the game” can really help improve the experience for the consuming teams.
> I’m curious about this aspect that you mentioned:
>> forcing a library provider to maintain familiarity with all their consumers' codebases and business areas
> In my experience, this hasn’t been an issue—the library authors are inherently intimately familiar with how to [make changes] in a consuming codebase
(See my comments below where I'm not sure what a "transparent upgrade" means)
Wow - that's genuinely surprising (and impressive!) to me. This implies that, for all N consumers of the library, the library team are already familiar (without having to look it up - otherwise, that's added ramp-up burden) with the consumers' preferred code style, testing expectations, commit and branching structure, personal development environment setup, running integration tests against personal environments, how to announce PRs (if at all), and so on - all the "other stuff" that goes "around" making the actual code change. In an ideal world, a lot of those things _should_ be trivially clear/provided (code-style provided by auto-linters, testing carried out automatically during PR, etc.) - but practically speaking they very rarely are. This seems like a _lot_ of extra knowledge which is (in my opinion - though clearly not in a monorepo-mindset opinion) outside of their area of ownership and expertise that they have to carry around on a day-to-day basis, on the off-chance that they make a breaking change to their library (which, as I think we both agree, should hopefully be rare).
Even in the best-reasonable case, where all of those things are well-documented (and they _should_ be, because an outside contributor is indistinguishable from a New Hire), that's still a fair bit of reading, ramp-up, and workspace-setup that the library team needs to do - vs. simply telling the consuming team "here's the general shape of change that you need to make, apply this as appropriate to your own situation - ask us if anything's unclear"
OK, on to the direct reply:
---
So, to be clear about terminology, I think we're only really concerned with breaking changes, here - i.e. those for which, if the consumer moves from consuming v-<previous> to v-<new>, they will also need to change something about how their code calls the library. The most obvious ways I can think of for this to be necessary are:
* The type signature changes in a non-backwards compatible way - i.e. adding required parameters (including making previously-optional parameters required), changing the type of a parameter, or changing the type of the return value to something incompatible with the previous type ("incompatible" is hand-waving because different type systems think about this differently, but I think my meaning is clear in general)
* The name of the method itself is changed, or the entire call pattern is changed (e.g. instead of `foo.execute(command)`, you now have to call `foo.prepareCommand(command); foo.execute()`)
For any library change which _isn't_ a breaking change (i.e. one for which the consumers could keep their call-site code exactly the same, and expect the same behaviour) I believe this whole discussion is moot - both perspectives would agree that nothing would need to be done in the consuming code. My perspective would say that the consuming team has nothing to do, and the monorepo perspective would say that the library team has nothing to do - but, either way, `0 === 0`, even JavaScript agrees on that :)
So, then, I might need a bit of elaboration on what a "transparent upgrade" is, since my initial naïve interpretation (a response to a non-breaking change - one in which the call-site code can remain the same) cannot be the thing at issue, since - well, yeah, of course a library team knows how to make a no-op change to their consumers ;) might "transparent upgrade" instead mean "a change where all the data required to make the library call is available at the call-site, but it needs to be reshaped, retyped, or otherwise reorganized before being passed to the library function in v-<new>"? If so, then...yeah, I can see it being _nice_ if the library owners were to publish some deterministic code-diff tool which would transform all their consumers' call-sites to use the newly-reshuffled data, but honestly I'd expect them instead to do that "reshuffling" _internally_ to their function and not even publish the new "breaking but with reorganized data" version in the first place. Maybe there's a good case for this "data-reshuffling" that I'm missing, though.
Maybe a pseudocode example of a "transparent upgrade" would help me understand what it is?
---
> if there are breaking API changes that make it impossible/infeasible to maintain the current consumer behavior, that’s a very useful signal that it’s time for the library producer to go back to the drawing board because the proposed library update has issues[...]Or at least, helps trigger a conversation between the producer and consumers to see if the breakage can be accommodated without too much heartburn
Yeah, I think we're generally on the same page, here; just assigning differing weights, priorities, and likelihoods to breaking changes. Wherever possible, for sure, aim for non-breaking changes; and where breaking changes are necessary, make sure you have had a full discussion with consumers to see whether the churn you're going to create is justified (and maybe consider providing a long period of support for the previous version so that consumers have a long period to upgrade and to amortize that upgrade-churn) - but, if a breaking change _is_ truly justified (is worth more than the upgrade-burden it will cost), don't shy away from it!
> Overall, it really reduces the “library producer in their ivory tower releases changes without fully considering the downstream impacts, causing a crapload of burden/schedule slip for XX teams” issue.
Maybe I've just been very lucky to have barely run into that problem! :) In ~10 years in my previous role, I'm struggling to think of a handful of times where a major version upgrade was any more than a trivial process - and the few times that it _was_ a meatier change, I was perfectly content with being given a guide or walkthrough on how to migrate. But, yeah - if cavalier library teams making profligate and unjustified breaking changes _is_ a problem being faced, then I can see that that culture of "you break it, you fix it" would rein them in.
---
FWIW, this has _already_ really helped me to understand where a coworker was coming from in an (unrelated) proposal which, to me and all other colleagues, initially looked nonsensical. After chewing on it a bit, I realized he's coming at this from the same perspective - "people who make changes to things that other people depend on, should bear the pain of updating those other people's things to work with the new dependency-thing". Gotta say I am still extremely-unconvinced (I'm still on the side of "ceteris paribus, an owner making a change is more efficient than mandating that someone else make the change; so I'd prefer to find other ways of forcing library owners to confront the externalities of disruptive breaking changes that don't introduce that inefficiency"), but it does very much help to understand where he's coming from - thanks!
Unfortunately, the obvious economic incentive to effectively paywalling code through the Internet, is so strong that on this very website (that was supposed to have hacker mentality) most of the comments are in favour of it, because “you have to make a living”…
If you have any feasible ideas that could plausibly work where people don't "have to make a living" so that we can find ourselves and not be lost, we're all ears. Until then we're stuck in a place where people need money to pay for frivolous things like "food" or "rent" or "transportation". Still, in the face of that, there's a site called GitHub where people do freely share code, despite the economic misincentives, so I think the kids will be alright.
Of course the prevalent reason to make a service is that it can be monetized to Hell and back again. Does it fit the customer's needs? Who cares, as long as they have no clue what they will end up paying until it is too late to migrate away.
Meh. Yes that happens and yes it can be frustrating. But…
If users want something with long-lasting support and enhancements, that’s going to take work. The people who do this work are going to need to earn a living.
There are lots of ways to accomplish this. Advertising, altruism, one time purchases. But charging for a service can be a perfectly reasonable way of making it sustainable.
> By library, I mean any software that can be run by the user: shared objects, modules, servers, command line utilities, and others. By service, I mean any software which the user can't run on their own; anything which depends (usually through an API) on a service provider for its functionality.
this definition makes (open source) self-hosted services libraries too, and so i think it’s wrong and unusable. the distinction between a library and a service is clear enough and colloquial at this point that a redefinition probably obscures rather than clarifies. a library isn’t runnable (~has no binary), a service is runnable (~has a binary)
I have a hunch that a lot of people who have been spurred to think critically about this today are realizing that what they're actually interested in are services (in the economic sense) and not software.
- Libraries are suitable for providing building blocks and common functionality, that you'll build your system on top of, that systems will depend more heavily and execute more frequently.
- Services are a strong choice when there's a need to abstract high level operations and architectural complexity. Your system can then focus on making high-level calls without the burden of operating and managing all components and dependencies involved.
> By library, I mean any software that can be run by the user: shared objects, modules, servers, command line utilities, and others. By service, I mean any software which the user can't run on their own; anything which depends (usually through an API) on a service provider for its functionality.
These seem to be odd definitions and they make the article hard to reason about.
This is so obviously in your face true that it blows my mind that anybody would argue against it. You obviously write a service by linking a networking front-end to a back-end stand-alone library. It’s just basic software development decomposition. Software architecture 101.
Containers can run services without most of the overhead they talk about. And as everyone else is mentioning here, a service-oriented architecture is not dependent on any one tech stack, it’s just a way of designing applications.
I don't get it. Can anyone kindly share where this practice can be applied to a commercial software? I am struggling to wrap my head around "how run by user" works where they are using a commercial software service.
There's some confusion in this thread between "service" as out-of-process functionality called via IPC or over the network, and "service" as something done by a service-provider for money, such as emptying your bins.
Ultimately, it can't. Proprietary software has fundamental limitations that force proprietary software developers to choose technically inferior designs. It's why in the long run proprietary software is doomed.
It’s hard when you have N services that need a lockstep migration from v1 to v2 of the library. I tend to agree with much of this but it’s not a one size fits all thing.
i get the benefit of offloading the admin costs onto your consumers, but, as always, the devil is in the details. i've met both cases where a library should've been a service, and vice versa. this advice is way too broad and abstract to be practical
Not as a primary interface. By all means provide a wrapper around the library that does that, for users to use on the command line, but there should be an underlying library for other programs to use.
Problem with libraries are that they tie the user to a specific language/OS. A cli tool eating and spitting ASCII/UTF-8 is about as cross platform as it gets.
The difficulty is in maintaining your ability to send in the teeth of third-party block lists, which I think Stallman would likely rail against, so I concede the point.
``` #include "servicelib.hpp" int main(int argc, char argv) { return servicelib{argc, argv}.run(); } ```
The library can be re-used in other apps or services.
Then the whole damn library is unit-testable with any arguments you throw at it. Got an OS where argv may be null? You can unit test that. Got a user who decided to use --iamstupid instead of --iamawesome? You can unit test that too. Want to set up environment variables? Well that's not thread safe, but your test harness can do it before it instantiates the library object.
Want to use semver? You can. I use git commit checksums for versions and automatic tagging to semver. It's more annoying but superawesome.