>In the specific example above, however, I'd try to reduce that overhead by going even more C-style and allocating an array or two so you avoid the overhead of individual instances and instead have one Python object for many elements.

How exactly would you avoid the overhead of individual instances using a ctypes structure? Sure, you can allocate an array very easily, using Point * 10 and then putting your instances in the array. In fact, you'd have to do this if you wanted a collection of these structs. I'm pretty sure that this array is stored continguously, but you still have the overhead of the Structure base class. How are you proposing to avoid this in ctypes? Or were you referring to cffi?

If you really want to be serious about large, performance-intensive arrays in Python, it's time to pull out numpy. It actually would fit well here.

Re: cffi, I do love luajit but I'm not a huge fan of its C interface. I've not used cffi, but it seems to be inspired by, or at least shares a lot with, luajit's C ffi. Which means you have to basically paste all the C header files into a string in your Lua or Python code to use the corresponding shared library.

The beautiful thing about ctypes is that you don't have to declare all the function prototypes and type declarations when you use a shared library. You simply make sure to feed the functions the properly typed variables, and store the returned value properly, and it just works.

As a result, I find ctypes a lot easier to use for simple optimizations. YMMV.

> The beautiful thing about ctypes is that you don't have to declare all the function prototypes and type declarations when you use a shared library. You simply make sure to feed the functions the properly typed variables, and store the returned value properly, and it just works.

It just works... until the argument that you passed as int but is actually size_t makes the program crash on 64-bit. Or you have to use an API which includes a giant structure which you then must transcribe manually into Python. Or you need to use a system library located at different paths on different platforms.

In my opinion, ctypes isn't the worst, but a sane C FFI should allow you to specify some #includes and -ls and automatically extract the proper declarations and linkage. cffi only partially satisfies this, since you have to add declarations manually and then have cffi "verify" them using a C compiler, but it's still better than ctypes.

> How exactly would you avoid the overhead of individual instances using a ctypes structure?

This optimization only works if you have an upper bound for the total number of items but if so I'd at least try something like declaring point_x and point_y arrays so you'd have two Python objects accessed by indexes rather than many Point references. This technique can be particularly useful if you don't access all of the elements at the same time and you can get more efficient memory access patterns because you can access a subset of the elements in nice linear reads.

> If you really want to be serious about large, performance-intensive arrays in Python, it's time to pull out numpy. It actually would fit well here.

Very strong agreement - numpy is a treasure.

> As a result, I find ctypes a lot easier to use for simple optimizations. YMMV.

Ditto - I've found cffi to be faster in some cases but it's definitely more overhead to use.

Ah ok, so you're proposing a nested array, not an array of structure pointers. I thought you were saying there was some way to use ctypes structures and yet avoid the overhead of the class instances.

And yes, I agree that if you're going to create this type of data structure, then it's worth the trouble to pull out numpy if you can.