The NEP (Numpy Enhancement Proposal) link for those more curious about the detai...

davidinosauro · 2024-10-23T20:28:54 1729715334

In particular from the article I was confused by this:

> NumPy doesn’t offer a way to store data outside of the array buffer—there’s no concept of “sidecar storage” in NumPy.

But then it goes on and say to he strings are stored on the heap (which clearly is also possible with dtype=object) with an arena allocator. Reading the NEP now

ngoldbaum · 2024-10-23T20:39:52 1729715992

Well, there was no concept of sidecar storage. Now we have the hack we came up with for StringDType to store data on the DType instance and also make it so StringDType arrays don't share StringDType instances, unless the array is a view.

EDIT: looking back at the NEP, I'm not sure it does a great job explaining exactly how the per-array descriptor works. Ultimately it's powered by a hook in the DType API: https://github.com/numpy/numpy/pull/24988. There is only one spot in NumPy where array buffers are allocated, so we hooked there and made sure any arrays with newly allocated buffers get a new DType instance.