As someone who did cross platform development for iPhone, Android and Windows Phone way back when, Windows Phone did actually have the superior dev experience by far (talking about WinPhone 7+ here). It wasn't free, but neither was iPhone development.
They didn't have any market share though, so there wasn't much money to be made making apps for them. I suspect they failed because they launched 2-3 years after Android and iPhone, so the other platforms had accumulated the network effects of an existing user base and app ecosystem that they couldn't catch up to. And they tried hard, IIRC, Microsoft offered to build a Snapchat client for Snap Inc, and to pay them to be allowed to do so, but were denied.
PyTorch is the most impressive piece of software engineering that I know of. So yeah, it's a nice interface for writing fast numerical code. And for zero effort you can change between running on CPUs, GPUs and TPUs. There's some compiler functionality in there for kernel fusing and more. Oh, and you can autodiff everything. There's just an incredible amount of complexity being hidden behind behind a very simple interface there, and it just continues to impress me how they've been able to get this so right.
BS. There's so much effort getting Pytorch working on TPUs, and at the end of it it's incredibly slow compared to what you have in Tensorflow. I hate this myth and wish it would die.
OTOH PyTorch seems to be highly explosive if you try to use it outside the mainstream use (i.e. neural networks). There's sadly no performant autodiff system for general purpose Python. Numba is fine for performance, but does not support autodiff. JAX aims to be sort of general purpose, but in practice it is quite explosive when doing something other than neural networks.
A lot of this is probably due to supporting CPUs and GPUs with the same interface. There are quite profound differences in how CPUs and GPUs are programmed, so the interface tends to restrict especially more "CPU-oriented" approaches.
I have nothing against supporting GPUs (although I think their use is overrated and most people would do fine with CPUs), but Python really needs a general purpose, high performance autodiff.
> I have nothing against supporting GPUs (although I think their use is overrated and most people would do fine with CPUs), but Python really needs a general purpose, high performance autodiff.
As someone who works with machine learning models day-to-day (yes, some deep NNs, but also other stuff) - GPUs really seem unbeatable to me for anything gradient-optimization-of-matrices (i.e. like 80% of what I do) related. Even inference in a relatively simple image classification net takes an order of magnitude longer on CPU than GPU on the smallest dataset I'm working with.
Was this a comment about specific models that have a reputation as being more difficult to optimize on the GPU (like tree-based models - although Microsoft is working in this space)? Or am I genuinely missing some optimization techniques that might let me make more use of our CPU compute?
For gradient-optimization-of-matrices for sure. Just make sure that you don't use gradient-optimization-of-matrices just because they run well on GPUs. There may well be more efficient approaches to your problems that are infeasible for the GPUs' wide SIMD architecture you may miss if you tie yourself to GPUs.
In general it's more that some specific models are easy for GPUs. Most models probably are not.
I really don't understand the GPUs are overrated comment. As someone who uses Pytorch a lot and GPU compute almost every day, there is an order of magnitude difference in the speeds involved for most common CUDA / Open-CL accelerated computations.
Pytorch makes it pretty easy to get large GPU accelerated speed-ups with a lot of code we used to traditionally limit to Numpy. And this is for things that have nothing to do with neural-networks.
For a lot of cases you don't really need that much performance. Modern processors are plenty fast. It seems that current push to use GPU also pushes people towards GPU oriented solutions, such as using huge NNs for more or less anything, while other approaches would in many cases be magnitudes more efficient and robust.
GPUs (or "wide SIMDs" more generally) have quite profound limitations. Branching is very limited, recursion is more or less impossible and parallelism is possible only for identical operations. This makes for example many recursion-based time-series methods (e.g. Bayesian filtering) very tricky or practically impossible. From what I gather, running recurrent networks is also tricky and/or hacky on GPU.
GPUs are great for some quite specific, yet quite generally applicable, solutions, like tensor operations etc. But being tied to GPUs' inherent limitations also limits the space of approaches that are feasible to use. And in the long run this can stunt the development of different approaches.
> For a lot of cases you don't really need that much performance. Modern processors are plenty fast. It seems that current push to use GPU also pushes people towards GPU oriented solutions, such as using huge NNs for more or less anything, while other approaches would in many cases be magnitudes more efficient and robust.
I still don't get the criticism of Pytorch. If anything, you can get the best of both worlds in many way with their API supporting on GPU and on CPU operations in exactly the same ways.
What do you mean by “seems to be highly explosive”? I have used Pytorch to model many non-dnn things and have not experienced highly explosive behavior. (Could be that I have become too familiar with common footguns though)
I get what you mean by the GPUs are overrated comment, which is that they're thought of as essential in many cases when they're probably not, but in many domains like NLP, GPUs are a hard requirement for getting anything done
Wait wat, jax and also pytorch is used in a lot more areas then NN's.
Jax is even consider to do better in that department in terms on performance then all of julia so wat are u talking about
GP makes a fair point about JAX still requiring a limited subset of Python though (mostly control flow stuff). Also, there's really no in-library way to add new kernels. This doesn't matter for most ML people but is absolutely important in other domains. So Numba/Julia/Fortran are "better in that department in terms on performance" than JAX because the latter doesn't even support said functionality.
I don't think this is a particularly accurate description of pytorch in 2021. Yeah, the original c++ backend came from torch, but I think most of that has been replaced. AFAIK, all the development of the c++ backend for pytorch over that last several years has been done as part of the pytorch project -it's not just python wrappers at this point.
What I like about PyTorch is that most of the functionality is actually available through the C++ API as well, which has 'beta API stability' as they call it. So, there are good bindings for some other languages as well. E.g., I have been using the Rust bindings in a larger project [1], and they have been awesome. A precursor to the project was implemented using Tensorflow, which was a world of pain.
Even things like mixed-precision training are fairly easy to do through the API.
It's the same in Norway, and it seems completely indefensible.
1) It's the opposite of progressive taxation, since the tax deduction is higher the larger your mortgage is (and there's no deduction if you don't have a mortgage).
2) It artificially inflates housing prices.
Overall, it's a shifting of cash away from those that want to enter the housing market towards those that are in it, i.e. taking from the poor and giving to the rich. The policy seems like an obviously indefensible mistake, yet no political party dears to touch it because the majority of voters are beneficiaries of it, and the topic is slightly too complicated for there to be an informed public debate about it.
I think it's relatively widely recognised as a bad idea now, but the challenge is that people have made long-term financial plans based on it, and if you were to suddenly cancel it altogether, lots of people would probably get in trouble. They're imposing further restrictions on it in the Netherlands, but it's all gradual and slooooow.
Denmark is in the process of raising the retirement age. This is done gradually to allow people to properly plan for it. This means that the retirement age is based on year of birth. For people born before 1. january 1954 the age is 65.5. For people born in 1996 or later the retirement age is 74. There is a scale between these two points, I was born in the early 80es and my retirement age is 72.
The argument behind this is that the current level of income/taxation can not support people on pension for more than ~12 years on average and as the life expectancy rises the retirement age must rise with it.
There is a fairly big debate happening on 'graduated pensions'. Should a bricklayer who has carried heavy loads all his life be forced to work to the same age as an academic who has spent a large amount of time behind a desk in an office?
No, and that means this is the best time to get this done. The least amount of people will be affected.
The interest rate will go back up at some point and we would have a fairer house market without this tax deduction. It's basically a tax break for people with enough resources to buy their house. And it drives up the housing prices making it harder for first-time buyers.
> the challenge is that people have made long-term financial plans based on it, and if you were to suddenly cancel it altogether, lots of people would probably get in trouble.
This could be solved by just cancelling it. People taking advantage of this scheme:
a) had no reasonable expectation that it would be long-term, and,
b) had to be wealthy enough to initially access the scheme to benefit from it, and have since continued to benefit.
Handling any exceptional cases (which would be relatively very few) would be a lot more effective, in every sense, than continuing the scheme.
Besides, the Netherlands already has a Mortgage Guarantee Scheme which protects people with mortgages and changed circumstances.
People who recently got one might have some reasonable expectation that it wouldn't be long term (although often their mortgage advisor would probably have given them the impression that it would be), but there are still many home owners who would reasonably have believed so. And even the ones who do expect it to not be long-term, would not have expected it to be cancelled overnight.
I would certainly love to be rid of it sooner rather than later, but I do acknowledge that that's going to be disadvantageous to a lot of people, and will have to take them into account as well.
The mortgage guarantee scheme doesn't really apply here, I think: it raises the buying power of people with low incomes, but once you've bought a house, it only means that their bank gets paid if you're forced to sell it - but you'd still be forced to sell it.
Is that so? Like which? Generally, I wouldn't be in favour of suddenly significantly cutting people's benefits without some compensation or other either. Luckily, I don't think that happens a lot.
For example, whereas we need to raise the retirement age, we don't suddenly increase it with a couple of years. It is raised slowly in increments, and less for people who are closer to the retirement age and thus have had less time to plan for it.
> the challenge is that people have made long-term financial plans based on it, and if you were to suddenly cancel it altogether, lots of people would probably get in trouble
This sounds much like the situation in Sweden including why things remain the way they have become. If it's brought to a political level, it use to be more about how house owners with very high mortages can be protected, haha... sigh
The UK managed to stop doing this. Also most deductions were limited to basic tax rate not higher rates. At some point the extra tax revenue becomes attractive.
Dutch political commentators suggest that there had been a tacit housing policy bargain between the left and right wing parties for many decades: the left delivers large rent subsidies while the right delivers mortgage subsidies for their constituencies. It just seemed much harder to win elections with supply side policies (build, baby, build!).
I've seen studies where they tracked wild Norwegian salmon, and found that 90% died before spawning. As far as I understood it, this was interpreted as "natural death", but it's the same figure as the one in the article. It seems like a very plausible explanation that those deaths also largely occurred due to the same poisoning from tires.
The difference is presumably the digital contract, that is written in a programming language that only a handful of people on earth can understand, where a programming error can make your money disappear, such that you have no recourse in the court system.
Another difference is the pyramid scheme incentives in the cryptocurrency tech, causing everyone who's bought into it to be incentivised to talk about how great it is.
> All it does is infringe on our rights to be able to do what we want on our own devices.
No, but this is exactly the point of DRM and the legal protections around circumventing it. It never was about copyright protection. Copyright infringement was already illegal before the DMCA, and the introduction of DRM didn't make a dent in the amount of copyright infringement.
The point of making DRM circumvention illegal is for me to be able to sell you a bunch of bits, but ensure that I don't have any commercial competition in regards to how you use those bits. You can't legally make a device that plays DVDs without the blessing of a cartel known as DVD FLLC. You can't legally make a device that plays music from iTunes without the blessing of Apple. Etc. It's about retaining monopolistic control over media distribution and use, by forbidding certain forms of competition in the market.
Getting a law passed that forbids market competition (in many countries! not just the US) under the guise of being about copyright protection, is one of the greatest cons I've ever heard of, but that is what has happened.
I don't think the example here is the best. There's a case to be made for extracting pure functions and organizing them like this, but I don't think this code makes it. The benefit of pure functions IMO is primarily in that the code becomes easy to reason about if it doesn't depend on state. But any app that does anything will have state, and the question is how you manage that. One guideline could be that individual code units should reduce the amount of state you need to worry about at higher levels of abstraction.
In the example, there is hardly any code that does anything different depending on state. There's no state being managed, so there isn't actually any architectural problem being solved here. Should the API go down or change its format, the code breaks. The pure pluck_definition() will still fail to parse the JSON if the format changes. The pure build_url() will stop working if the API changes its URL format. They will pass unit tests, but fail in practice.
An actual problem to be solved here is to abstract away the details of the REST API, formatting and network errors. One way to do this is to pack that into a component with a well defined interface. You can still do this stateful/non-stateful split within the component if you want, but on the application level you need to apply that heuristic recursively at different levels of abstraction.
There is absolutely a problem here. Having worked in disasters of a code base, the architectural pattern in the first example is probably fine... until the software grows. The first function truly is a thing-do-er which violates SRP. Then it will easily become a ball of mud.
Why is this so bad? Its not because its expensive, yes that is bad, but the largest issue with working in a ball of mud architecture, is that the code becomes so fragile and interdependent that changing any one thing can easily lead to breaking many other things. This leads to a culture of fear of change which grows tech debt. Then one day someone steps up and decides to actually refactor this ball of mud to have some semblance of logic to it, what a noble soul. That person is then subject to a barrage of bugs and issues from the refactor and is that the mercy of their supervisor.
Dealing with state and other side effect like issues is certainly something to consider in architecture, but it is a different argument entirely.
Yeah, but ResNet does not have sparse matrices, so how could it use them? Post ReLU activations may be sparse, but I don't think that helps when used with a non-sparse Conv2d.
I don't know if there are any white papers with hard details yet (if anyone knows of one, please share!), but nVidia's marketing material[0] for the Ampere architecture claims the following:
"Sparsity is possible in deep learning because the importance of individual weights evolves during the learning process, and by the end of network training, only a subset of weights have acquired a meaningful purpose in determining the learned output. The remaining weights are no longer needed.
Fine grained structured sparsity imposes a constraint on the allowed sparsity pattern, making it more efficient for hardware to do the necessary alignment of input operands. Because deep learning networks are able to adapt weights during the training process based on training feedback, NVIDIA engineers have found in general that the structure constraint does not impact the accuracy of the trained network for inferencing. This enables inferencing acceleration with sparsity."
So the idea seems to be that at the end of training, there's fine tuning that can be done to figure out which weights can be zeroed out without significantly impacting prediction accuracy, and then you can accelerate inferences with sparse matrix multiplication. They consider training acceleration with sparse matrices an "active research area."
I could see it being nice for the sake of running large language models on consumer, or really cool for the few edge computing applications that can actually demand and power conventional GPUs (e.g. self-driving cars.) It's probably not a great boon to the researcher who wants to reduce their iteration timeline though.
First I've heard of this. What is the software situation like on a device like this? I assume no comptability with Android, so apps have to be developed specifically targeting their OS?
They didn't have any market share though, so there wasn't much money to be made making apps for them. I suspect they failed because they launched 2-3 years after Android and iPhone, so the other platforms had accumulated the network effects of an existing user base and app ecosystem that they couldn't catch up to. And they tried hard, IIRC, Microsoft offered to build a Snapchat client for Snap Inc, and to pay them to be allowed to do so, but were denied.