Don't just write pure functions. Write point free combinators.
Have you guys ever wondered why no matter how much care or planning you use to organize your code when you begin a project, some time down the line you will always encounter a situation where the organizational scheme you chose is less than ideal or even flat out wrong? It's a sort of inevitable technical debt that occurs.
The industry spends an endless amount of time debating and coming up with all kinds of solutions to deal with the above problem. First it was OOP, the latest is microservices.
The actual solution to the above problem is to use point free combinators as much as possible in your code.
> some time down the line you will always encounter a situation where the organizational scheme you chose is less than ideal or even flat out wrong?
I'm with you here...
> The actual solution to the above problem is to use point free combinators as much as possible in your code.
You lost me here. Perhaps if you're working in a problem space with high complexity, little ambiguity and hard performance requirements (fairly rare in my experience), you might get mileage here.
Otherwise, functional programming techniques are somewhat orthogonal to Conway's law. Point-free combinators don't stop new information about customers from completely altering your domain model, business logic and how your data is stored at rest.
I think it's helpful to realize that some of the root causes of bit-rot in a codebase are more closely tied to not fully knowing what was going to evolve into ahead of time, which in and of itself isn't a bad thing. However, it means that it's a failure mode that is independent and often out of the hands of the engineer writing the code -- functional programming idioms or not.
I'm more referring to a type of technical debt. Of course if your product manager wants you to refactor your web app into a PS5 video game nothing can save you. I wouldn't call the fact that your code wasn't prepared to be refactored into a PS5 video game "technical debt."
As most software engineers know, there's tons of examples where the main issue was mostly organizational issues preventing the programmer from simply removing/replacing/moving logic to fulfill the main objective. This is the "technical debt" I'm talking about.
In that case if all your logic was combinators (and thus dependency free) then all you need to do is move things around, pull things out and/or put things in like lego blocks.
The main reason why people can't move code around like lego blocks is because most logic contains dependencies or is tied to state. Combinators are never tied to state or anything and thus if all your code was made up of combinators you would have no issues in moving logic around. The problem of organizational technical debt is solved with combinators.
If you take it a step further and use the point free style, you are eliminating state all together further protecting your program from ever being dependent on state.
The logic is pretty sound though most people can't put the benefits of FP into purely logical terms. They only talk about why they like it qualitatively without ever pinpointing why it's truly better for design/organizational issues.
> If you take it a step further and use the point free style, you are eliminating state all together further protecting your program from ever being dependent on state.
The problem is that the part of your program which processes state...is generally the useful part. This is part of the reason why I eventually came back down to earth after a couple of years of FP zealotry in my early career. I realized that it makes the easy parts easier. And there is something to be said about that -- there are a lot of codebases where just doing the table stakes refactoring that minimizes the surface area of stateful code does a ton to improve the codebase.
But in my experience, that's only table stakes. It doesn't make the hard parts (such as unfucking a broken data model and code + existing prod data which depends on those implicit assumptions) any easier. So it's a tool of limited use for most of the thorny issues I've come across in my career, across startups and BigCos.
You cannot write a program without state. I am obviously not talking about eliminating all state. I am talking about eliminating state from sections of code that don’t need it. Generally for web development the only required state is caching and the database.I am not saying that you eliminate this.
Additionally shortcuts can be taken. Graph algorithms tend to be easier with mutation.
> But in my experience, that's only table stakes. It doesn't make the hard parts (such as unfucking a broken data model and code + existing prod data which depends on those implicit assumptions) any easier. So it's a tool of limited use for most of the thorny issues I've come across in my career, across startups and BigCos.
Well your talking in terms of fuzzy feelings and experience. I am talking about a hard logical rule. A mathematical rule.
If you want to eliminate all technical debt that originates from organizational issues then where you want that technical debt eliminated you simply need to have all your code be combinators in that location.
This occurs because combinators have no dependencies and therefore can alway be moved shifted decomposed and reused. This is a logical statement different from some random anecdote from experience. Also this isn’t just fp. A combinator is different from a pure function.
Look you can’t eliminate IO and you can’t eliminate state in practice. So the formal logic I stated above cannot effectively be applied to your code to completely eliminate this form of debt.
It is however a logical statement and holds as a truth. Thus if you follow it as much as possible you will see that what ends up happening is a segregation and minimization of state along with a minimization of the form of technical debt I described. (Which of course cannot apply to the stateful part of your program)
Most of web development is a similar pattern that was developed independently from the above philosophy but coincidentally arrived at the same conclusion. Web applications are generally stateless with state segregated away to databases and caches. You will find that generally the technical debt lives in this state more then it lives in the web app, but if you follow the combinator rule, you can eliminate organizational technical debt from a good portion of your web app.
The purpose of this post is to elucidate a logical rule and pinpoint the hard origins of technical debt arising from organizational issues rather then walk around and talk about fuzzy feelings and intuitions that arise from experience. Of course state and io cannot be removed from your program. What this means logically is that organizational technical debt can also never be fully removed, but it can be segregated. Better to understand this concept as a science rather then some fuzzy art that comes from experience.
The industry often repeats history because of lack of formal understanding. It’s very hard to optimize something without formal understanding. You cannot prove one pattern is better than another with just fuzzy experience. Hence the reason why the industry jumps from pattern to pattern in an endless flat circle.
> Look you can’t eliminate IO and you can’t eliminate state in practice
Not only can you not eliminate IO nor state in practice, but it's literally the most important part. The useful things computers do are IO and state. The difference between using pure functions and combinators is the difference between which color of paint you're going to put on your car. The color of paint you put on your car has nothing to do with its drivetrain.
It depends on the application whether or not it’s more important or less important. Chat application vs. neural network. A neural network is mostly compute a chat application is mostly io.
If you work with a framework like nodejs the framework agrees with your assumption but nobody is going to write a neural network with it. Ironically JavaScript is the language showing a sort of resurgence for fp but nodejs is the worst platform for it due to its focus on io based concurrency.
Still though you will notice that despite the above caveats for node the framework still follows the classic pattern of segregating state away. Typical nodejs apps are stateless along with most web apps. If you worked with frameworks that have request handlers as the primary pattern you will see that this pattern tries to segregate io away as much as possible by turning the abstraction into a request/response combinator
So basically, The compute part of any application, no matter how small is a prime candidate for combinators via segregation of compute and side effects.... but of course if your application is io bound or highly stateful you can only go so far.
The above is not a combinator. addSomething relies on the line x = 1 and is forever tied to it. You cannot reuse addSoemthing without moving x = 1 with it. Therefore addSomething is not modular. This is the root of organizational technical debt. When logic depends on external factors it cannot be reorganized or moved.
This is also a big argument against OOP because OOP is basically the same exact thing with a bit of scope around it:
class B()
x = 1
addSomething(y) = y + x
divideSomething ...
b = 3
Now addSomething can never be used without taking x=1 and everything inside B with it. It is less modular as a result.
A combinator is like this:
addSomething(x,y) = x + y
fully modular and not tied to state.
The point free style is a bit long to explain. Another poster brought up readability and now I think I went too far with it as a recommendation, it's like going vegan basically if you employ that style. Just stick to combinators and you get 99% of the same benefits.
Suffice to say the point free style eliminates the usage of all variables in your code. You are building just pipelines of pure logic without any state.
When you get rid of state as much as possible, your logic will not have any dependencies on state, and thus will generally be free of technical debt caused by logic being tied to dependencies.
The downside to taking that combinator approach too dogmatically is that passing all state as parameters can get extra unwieldy, because now a simple change in data schema can result in you refactoring every single function call.
This dilemma has a name: The Expression Problem. A decent summary can be found here.
Functional programming is an amazing paradigm for most domains. However, some domains will take a seasoned functional programmer and make them want to jump off a cliff. UI programming, game programming, and simulation programming are some examples where pure functional approaches have never made a dent, and for good reason.
One more thing I should mention. UI programming and game programming are now currently the areas where functional programming techniques are sort of in vogue.
If you want to do FP in your job, becoming a front end developer is your best bet as React + Redux currently follow a paradigm called functional reactive programming (FRP) with react trying to go more and more in the direction of FP and trying to separate out all side effects from pure functions.
A popular pattern in game programming is called ECS, which isn't strictly FP but is similar in the sense that functions are separate from data rather then attached to data as it is in OOP. The game industry is definitely heading in this direction over OOP style techniques. It's actually rather similar to FRP.
Generally the solution that functional programmers arrive at when facing UI / game / simulation, is to have a consistent persistent data structure. Incidentally, this is also basically what SQL is: a functional language for transactionally querying the global state.
>The downside to taking that combinator approach too dogmatically is that passing all state as parameters can get extra unwieldy, because now a simple change in data schema can result in you refactoring every single function call.
This should happen with methods too. Whether a variable is free or a parameter doesn't change anything.
x = {b = 1, c = 2}
f(x) {return x.b}
g() return x.b
A change in x, say deleting b, will require a refactor for both the combinator and the method.
I'm not saying combinators are the solution to everything. Of course not. I'm saying combinators are the solution to technical debt caused by organizational issues. Of course there are trade offs, I never said otherwise.
Both of the issues above are separate from the expression problem though. Personally I don't think the expression problem is much of a problem. Whether you add a new function or a new shape to either paradigm in the example link you gave, the amount of logical operations you have to add is equal for both cases. The difference is the location of where you put those logical operations. In one case they can be placed closed together, in another case they have to be placed in separate scopes, but the total amount of logical operations written to achieve a certain goal is equal.
For example adding perimeter to either paradigm necessitates the need for you to define the perimeter of every shape no matter what. Neither paradigm actually offers a shortcut when new information is introduced into the system.
Classes and methods are just sugar around namespaces, functions with implicit "this" params, and some extra markup around design ownership (ie private members).
You don't gain or lose state with classes alone. Your examples didn't remove any state. X is still there, its just not B.x.
What you're fighting against is side effects and reducing what is in scope at any given time. One could argue that the goal of classes is the same!
Sadly one can write terrible, leaky code in either style.
I am not talking about leaky code. I am talking about code that is not modular.
Rest assured, I know you’re talking about a perceived isomorphism between a function with a struct as a parameter and the same struct with a method. There are some flaws with this direction of thought.
It is the usage of implicit ‘this’ that breaks modularity. When a method is used outside of a class the ‘this’ is no longer implicit thereby preventing the method from ever being moved outside of the context of the class. This breaks modularity. Python does not suffer from this issue.
Couple this with mutation. Often methods rely on temporal phenomena (aka mutations) to work, meaning that a method cannot be used until after a constructor or setter has been called. This ties the method to the constructor or setter rendering the method less modular as the method cannot be moved or used anywhere without moving or using the constructor with it.
My claim is that combinators can be reorganized without dragging context around thereby eliminAting technical debt related to organization and repurposing and reusing logic.
Note that when I say combinator, I am not referring to a pure function.
So basically, use functions but limit your use of closures? As in define your functions to be dependent only on parameters and not surrounding scope (even if the surrounding scope is immutable/pure)? If that’s the lesson, I’m all for it, with the exception of fully local closures that are used more for expressiveness than standalone functionality.
>So basically, use functions but limit your use of closures?
Not limit, terminate the use all together along with classes because methods in classes are basically closures.
>I’m all for it, with the exception of fully local closures that are used more for expressiveness than standalone functionality.
Sure I can agree with this... formally though. When you write a local closure you are preventing it from ever being reused. The philosophy of this style is to assume an unknown future where anything has the possibility of being reused.
When too much of your logic gets inserted into "local closures" your program will be more likely to hit the type of organizational technical debt I am talking about above.
It's not a huge deal breaker though, you can always duplicate your code to deal with it. I'm not against shortcuts but most programmers need to know when they are actually taking a shortcut while being aware of the formal rules that will actually eliminate organizational technical debt.
Many functional programmers are unaware of the the origins of organizational technical debt and mistakenly build technical debt into their programs with closures even when it wasn't their intention which is separate from your case as you are doing it intentionally.
I think we’re mostly in agreement. I’m a little looser than your absolute in practice, but I apply the same principles. Where I’m looser is basically an allowance for closures as a simple expression (and where languages with more expressiveness may not require a closure). If any local logic becomes more complex than that, I’m quick to parameterize it and move it out to its own function.
Yeah I'm already on board with modular functions and functional programming in general. I was wondering about the point free thing. I agree that's like going vegan.
functional programming still allows the usage of functions that are not combinators. So I'm referring to that specifically, not functional programming in general. The OP is recommending functional programming I'm taking it a step further.
It sounds like a combinator is roughly the same thing as a pure function. I'm more familiar with the term pure function, and OP does specifically advocate for pure functions.
No, they refer to different things but can intersect. Not all pure functions are combinators. Just look it up. Haskell is purely functional but it also promotes many patterns that are not combinatorial.
I think the easiest way to explain point-free is this bash snippet:
grok () {
grep -v DEBUG | sort -n | uniq | tail
}
Unlike a generic pure function, it forcibly abstracts away arguments. It's a bunch of other functions combined somehow, but it's impossible to know the type/structure/semantics of the input. And in most cases also of the output! Although it's less visible in my example.
Of course it’s possible. Just look at the type signature of each function to know what goes in and what comes out. Not much different then looking at the type signature of an assigned variable.
Debugging the point free style is different though as there are no points to put your breakpoint. Think of the identity function with a side effect as a replacement for print or breakpoints.
I could be wrong, but I take it to mean that the function shouldn't necessarily depend on the arguments that are passed in. Rather, you focus on a set of procedures that are independent of the number of arguments and can act on them.
I've been playing with Joy (concatinative, combinator-based) and I have to agree. So far everything I've translated into Joy has been more elegant and easier to understand. Conal Elliott's "Compiling to categories" is one way in: http://conal.net/papers/compiling-to-categories/
Any paradigm can look clean when you’re playing. Work on it for a year and then have management tell you that we must add a feature that breaks your core design because our big client needs it.
Many aspects of a programming language can make it nasty or hard to use under certain contexts. This is something most programmers are aware of through experience.
My point is, that the use of combinators and point free programming formally eliminates organizational technical debt. So for this specific issue, Joy should indeed be better by logic.
BTW, the term "organizational" usually refers to how you organize your people.
Even under your definition, though, point-free only helps with a limited definition of "how you organize your logic". How do you organize it into layers? How do you organize it into files? How do you organize it into processes? How do you organize it across machines? You can get real "organizational technical debt" on all of those.
> BTW, the term "organizational" usually refers to how you organize your people.
Then what, in your opinion, is a better adjective for "technical debt" that best conveys my point?
>How do you organize it into layers? How do you organize it into files?
Namespacing, files, and "layers" are aesthetic forms of organization that do not produce actual barriers in organization. They are for people to read and do not produce actual structured barriers.
Namespace MathLayer1 {
function mathlayer1combinator()
}
Namespace OtherLayer2 {
function otherlayer2combinator()
}
You will note because both functions are combinators, they can always be moved/called interchangeably into either namespace/file/layer provided that you handle circular dependencies (easily done in C++ by declaring everything in a header file first and making sure you don't have any sort of circular recursion in the definitions).
Thus "organizational" mistakes in namespacing/layers are actually trivial because none of the combinators in the namespace are tied to context. If you find you can't move a function out of a namespace it is always because of the fact that that function is not a combinator and is likely relying on some other state declared within the namespace/class. It is not the namespace itself that is causing this issue.
Another way to think about a namespace or layer is that all it does is put a prefix on your combinator for readability purposes.
Layer3.combinatorG as opposed to just combinatorG
A third perspective is to view the combinator as a concept that transcends namespacing or layers. A combinator is never limited by scope because it carries the scope with it rather then existing as an entity tied to scope.
Machine organization is a problem though. Machines are a physical limitation placed upon our virtual code and people exploiting this limitation as a feature makes code even less modular. A machine barrier is no different from a namespace or object with one difference: Moving or reusing code in a different machine requires data transfer. There is no reason to impose this limitation on our code unless we have no choice.
Thus the limitations of machines should only be utilized to optimize for performance, not as a feature to organize your code. Inevitably in practice this can cause organizational debt if you placed a function in machine A for optimization reasons and suddenly find that you need to use that function in machine B things will be inconvenient.
Hopefully, if that function is a combinator, moving it for use in machine B will be less of a pain. But keep in mind in the idealistic world of programming the machine barrier doesn't exist. Formally moving your combinator out of Machine B into Machine A is the same as if the Machine was called a Namespace. There is no intrinsic difference. It is the physical limitations of the real world that is making things inconvenient so my statement about combinators still holds logically.
That being said the physical barrier of machines can be abstracted away in a single project directory. There are strategies to handle this (imperfectly), docker or RPCs for example.
It's Good to have awareness of the exact formal and logical consequences of certain actions rather then rely on some fuzzy intuition of design. Clarity in the fundamentals of what's going on is key to developing a logical rule set so that optimal structure can be calculated rather then sub-optimally designed from intuition.
IME, the limitations of this recommendation appear when one needs to combine multiple functions taking multiple arguments, some being fixe and some other not.
Although they are mostly syntactic, they hamper the readability of the resulting code.
I agree with you. Readability is definitely worse when using this method. It doesn't mean it can't be circumvented with good naming.
Overall though this method basically solves the problem with technical debt I described above.
The root of all dependencies come from free variables. So if you get rid of free variables and turn your functions into combinators, then all your logic is modular.
If you get rid of all variables then all your code has to be combinators.
Perhaps the point free style goes to far in terms of readability. Maybe just switching to combinators is a good middle ground. In layman's terms for the readers who don't understand this means don't even use pure functions in classes. Just use functions.
Depends on the definition of tech debt. Do you mean structural/organizational or readability?
If you mean readability then yes, the point free style does not protect you from that imo.
But combinators and the point free style does protect you from structural and organizational issues, which is the type of technical debt I'm referring too.
Look up something called currying. It solves the issue. You can still write functions with multiple arguments and then "curry" them into functions of just one argument.
Of course this does not solve the readability issue.
Yeah technically it's not point free. But technically nothing can truly be point free just like nothing can ever be truly purely functional.
In purely functional programming your IO and state has to live somewhere just like how your points in point free programming still have to exist on the function call.
A function that takes multiple arguments introduces extra points to your program similar to introducing extra IO calls to your purely functional program. These points are inevitable additions to your program. The philosophy remains point free however.
Another way to think about it is that every parameter in a point free program is tied to a point somewhere up the pipeline. If you introduce a function with multiple arguments, that extra argument will still need to be tied to a point either right now or up some other pipeline.
So either you curry in a point into that extra parameter or you curry in the result of another pipeline.
A good way to think of the point free style is a series of pipes flowing in one direction, from left to right. All pipes segments must be eventually connected to a source and eventually flow to an output.
A function with 2 parameters is a T junction within this system with 2 inflows and one outflow. No matter how you configure your network of pipes; points need to live to the at the source and output of this network either as IO or actual state. There is no "point" in creating a network of pipes that isn't connected to a source and an output.
When you introduce a new T junction into this network of pipes, you will inevitably need to connect these inflows to points at the source of the pipe network. There's no way around it.
Don't get me wrong, I rather agree with you on the conceptual side of things; I just wish there was a pretty syntax to do it practically.
As you mention them, pipes, for instance, get close to that, and e.g. Elixir uses them to a great result. However, it requires unambiguous priority of the arguments and cooperation & discipline from the librairies authors, so that piping follow the intuitive (and hopefully unambiguous) understanding of the data flow.
I wasn't talking about Unix pipes. Apologies. I meant actual pipes. Like sewage piping.
Reread my pipe example but imagine a 2D diagram of physical pipes instead. This is the physical analog of the point free style and the origin of the term when used in unix. The diagram should get around the problem that is caused by the point free style (aka Unix piping) when you use text to represent the concept. In fact this is one of the few times where graphical representations of programming is clearly superior to text.
Isn't it ironic to complain about the industry coming up with different solutions for the problem, then to suggest the actual solution is X (whatever x is)?
Well that would be assuming I’m from the industry. What if I’m currently in some place outside of the industry? What if I have a lot of experience both in industry and outside of the industry? Would it still be ironic?
I think the bigger question is what is X and is it definitively correct? Because the irony of a situation has nothing to do with how correct X is.
You’re completely right. I’m talking about Technical debt arising from organizational flaws and issues and your statement made me realize this concept doesn’t exist. Nobody has ever dealt with this form of technical debt ever in the history of existence.
You’re right, everything I said is completely wrong. I’m not a biased person so I admit when I’m wrong.
Also note that this post is not sarcastic. I am 100% serious and being honest here. You are completely right. This is not a joke. It’s rare for people to do a 180 flip so I can see how this post can come off as sarcasm. I am not most people and this post is earnest.
I was talking about your proposed solution, not the problem itself. Point free code does not inherently solve any problems, and it certainly doesn't solve the entire notion of technical debt.
Yes by eliminating state you eliminate the possibility for a function to depend on state. When a function does not depend on state or anything it can be moved, reused and reorganized anywhere thus eliminating technical debt that comes from organizational issues. In other words, all functions in the point free style must be combinators because the point free style eliminates state as much as possible.
It's a forcing function to make your logic all combinators and not dependent on free variables.
Though I do agree with one replier. It does harm readability so now I'm thinking it may not be the best recommendation. Using combinators without the point free style is enough to actually fix the technical debt I'm talking about.
Don't just write pure functions. Write point free combinators.
Have you guys ever wondered why no matter how much care or planning you use to organize your code when you begin a project, some time down the line you will always encounter a situation where the organizational scheme you chose is less than ideal or even flat out wrong? It's a sort of inevitable technical debt that occurs.
The industry spends an endless amount of time debating and coming up with all kinds of solutions to deal with the above problem. First it was OOP, the latest is microservices.
The actual solution to the above problem is to use point free combinators as much as possible in your code.