>Any numerical computation language should have tensors generic in dimension, layout and type. That is the point I am trying to hammer home. If you think that Julia's tensors are better than PyTorch's and feel that my equal treatment of them is unfair - I honestly feel that is the wrong way to think about this. Rather than fighting to be the king of a very small hill, it would be better to try climbing a much larger mountain instead. Look inward and be unhappy at what the language has now.
It's still silly to compare Julia's tensors to PyTorch's because they are generic in dimension, layout, and type unlike PyTorch's and these seem to be exactly the pieces that you're ragging on. This means that your beef against PyTorch is fine, but it doesn't necessarily apply to Julia. I am curious if you can go into more detail about Julia's implementation instead of talking about possible Julia problems through an explanation of PyTorch.
>Wouldn't Julia be a much better language if it had its inlining guarantees?
Not necessarily. Julia is a dynamic interactive language which is made to be used through a REPL. Although you can increasingly statically compile packages, the standard way to use it is interactive and in this case you do have to watch your compile-times closely.
Julia does have opt-in inlining guarantees through the `@inline` macro. It's opt-in instead of opt-out in order to stop unnecessary compile-time growth (inlining isn't always that helpful). To decide when inlining should occur, Julia uses a cost model:
This then tries to only inline what's necessary to do for performance. Of course, using the fact that Julia can inline what it needs allows things like CUDANative.jl to force inlining and get the kernels it needs.
>And maybe add to that the intensional polymorphism that would allow it to implement them.
This sounds like a good idea. We should take this over to our community and see whether it would work well in the language.
I think the last thing to address is:
>Since it presents itself as a numerical computation language capable of replacing C++ and dominating machine learning and GPU programming I will hold it to the same standard I would take Spiral.
Not really. Julia is dynamic and interactive. C++ with its full templating is very powerful but can allow its compile-times to explode because its main use case is not with a REPL (and templated C++ code compile times are...). While Julia's generics and speed have increased the domain of what's possible with a dynamic language, it is still made for a much different domain than C++. I would say it's not really in direct competition with Spiral either since the interactive workflow is just so different. Julia's static binary generation is growing in power but will never be as strong as something made for static compilation, and C++/Spiral won't feel like an on-demand calculator where most users are punching commands and getting instant feedback.
To me, Spiral looks like a cool alternative to C++/Rust/Go in the scientific computing domain, and package ecosystems that mix Spiral+Julia could be quite powerful.
> It's still silly to compare Julia's tensors to PyTorch's because they are generic in dimension, layout, and type unlike PyTorch's and these seem to be exactly the pieces that you're ragging on. This means that your beef against PyTorch is fine, but it doesn't necessarily apply to Julia. I am curious if you can go into more detail about Julia's implementation instead of talking about possible Julia problems through an explanation of PyTorch.
Well, no, I can't go into more detail. It is the responsibility of those who take up the challenge I made, so why don't you go into detail instead, in the context of a GPU map kernel? I am genuinely curious about this, but not so much that I actually want to dive deep into the language.
Apart from that, I do like the rest of your reply. It is true that best performance lies somewhere between full inlining and full heap allocation, and that full inlining will kill compile times. One of the points I want to make with Spiral is that starting from the position of full inlining and then adding boxing and join points is a lot easier and much less error prone than starting from the other end and inlining by hand. I feel that this is important other programmers realize along with the fact that sophisticated compiler heuristics for when to inline are the only alternative.
> This sounds like a good idea. We should take this over to our community and see whether it would work well in the language.
I know that Odesky thinks that pattern matching on types directly only really works on languages with simple type systems such as Spiral's. If a language uses abstract types then that becomes more difficult. I definitely see a language having both intensional and extensional (parametric) polymorphism as a significant challenge. I'd love to see it met because extensional polymorphism is a lot easier on the IDEs and the way Spiral does it will make it difficult to create good tooling for it.
If Julia could support intensional polymorphism then inlining guarantees would just fall out from that. It is quite powerful - the inlining guarantees Spiral gives does not just apply to top level functions, but any function passed as an argument and captured by other functions.
> Not really. Julia is dynamic and interactive. C++ with its full templating is very powerful but can allow its compile-times to explode because its main use case is not with a REPL (and templated C++ code compile times are...).
If C++'s templating system was a car, then it would be missing wheels and a steering wheel. And possibly the brakes. How can one possibly do staging without join points, and all the other things Spiral has? As a language it misses several key concepts that would actually make such a system work well.
With the last paragraph I half agree with you.
It is true that right now Spiral uses a compile-run loop and acts as a sophisticated code generator, but it could be made into caching interpreter with some work. It probably will have to be if it gets bigger for the sake of compile times and library support.
Getting instant feedback in the REPL is hardly the domain of dynamic languages anymore - F# can do it well for example and so can various other static functional languages. Even C# can do it these days.
> To me, Spiral looks like a cool alternative to C++/Rust/Go in the scientific computing domain, and package ecosystems that mix Spiral+Julia could be quite powerful.
Julia does a lot of multi-stage compilation. It probably has had a lot of work done on its JIT by now. And language mixing, especially of languages with different strengths and weaknesses is a great idea. Staging gives the user access to it in a more direct manner so why not bring it out?
Spiral is quite a small language so if JITing abilities are flexible enough, it might be a good idea to implement it in Julia as a DSL and merge it with main language. That would combine all the advantages of Julia and Spiral as languages.
It's still silly to compare Julia's tensors to PyTorch's because they are generic in dimension, layout, and type unlike PyTorch's and these seem to be exactly the pieces that you're ragging on. This means that your beef against PyTorch is fine, but it doesn't necessarily apply to Julia. I am curious if you can go into more detail about Julia's implementation instead of talking about possible Julia problems through an explanation of PyTorch.
>Wouldn't Julia be a much better language if it had its inlining guarantees?
Not necessarily. Julia is a dynamic interactive language which is made to be used through a REPL. Although you can increasingly statically compile packages, the standard way to use it is interactive and in this case you do have to watch your compile-times closely.
Julia does have opt-in inlining guarantees through the `@inline` macro. It's opt-in instead of opt-out in order to stop unnecessary compile-time growth (inlining isn't always that helpful). To decide when inlining should occur, Julia uses a cost model:
https://github.com/JuliaLang/julia/pull/22210
This then tries to only inline what's necessary to do for performance. Of course, using the fact that Julia can inline what it needs allows things like CUDANative.jl to force inlining and get the kernels it needs.
>And maybe add to that the intensional polymorphism that would allow it to implement them.
This sounds like a good idea. We should take this over to our community and see whether it would work well in the language.
I think the last thing to address is:
>Since it presents itself as a numerical computation language capable of replacing C++ and dominating machine learning and GPU programming I will hold it to the same standard I would take Spiral.
Not really. Julia is dynamic and interactive. C++ with its full templating is very powerful but can allow its compile-times to explode because its main use case is not with a REPL (and templated C++ code compile times are...). While Julia's generics and speed have increased the domain of what's possible with a dynamic language, it is still made for a much different domain than C++. I would say it's not really in direct competition with Spiral either since the interactive workflow is just so different. Julia's static binary generation is growing in power but will never be as strong as something made for static compilation, and C++/Spiral won't feel like an on-demand calculator where most users are punching commands and getting instant feedback.
To me, Spiral looks like a cool alternative to C++/Rust/Go in the scientific computing domain, and package ecosystems that mix Spiral+Julia could be quite powerful.