other projects in that space are: cython (need to manually do wrapping), xdress, SWIG, SIP, clif.
Does anyone have experience with using these and able to compare/contrast?
I'm afraid of C++, but if I can wrap it and learn how to use from a python REPL, I think I can handle... any recommendations/tuotirals/howtos would be much appreciated. (The library I want to wrap is https://github.com/openzim/libzim )
I built bindings for various C++ libraries in SWIG (not only for Python but also Lua and Ruby) and I was always very impressed with the result. What's amazing about SWIG is its ability to easily port very complex and rich code. I successfully exported very complex class hierarchies and code relying heavily on pointers and STL to Python and/or Lua. The process consists of writing several header-like configuration files that SWIG ingests and uses to generate bindings, which is often pretty straightforward. Often you can simply import the normal C/C++ header files in those configs and add some glue code / extra hints to help SWIG in cases where it can't figure out what to do with a particular type.
In my experience, the most difficult aspect of binding generation is not writing the glue code, but doing so in a scalable way. Manually writing bindings for any real-world C++ codebase would therefore be extremely tedious (IMHO), so having an automated system that does this work for you is a huge time saver.
Some larger libraries / frameworks have their own wrapping generators Btw, PyQt (which provides extremely good bindings to QT) for example has SIP which is worth looking at (it's open source).
Have you used pybind? I've used SWIG extensively so I'm familiar with that - you talked about SWIG but not how it compares to this new one.
Reading through some docs, it looks like pybind suits better for a python codebase that occasionally needs C++ pieces added in as you have to setup the C++ code to handle the py needs.
SWIG I was always impressed with exactly your point. You could sometimes just make 1 big header file in a facade pattern to expose some "start" buttons needed to run a huge c++ application and nothing else needed to be setup. No going back and forth.
Example usage here was a single heavy duty image processing application but using python has the distributor and work manager. C++ code was a "worker" and python and zeroMQ did everything else. So the only "connection" we needed was a start button and some metadata to the c++. Maybe 1 header file with 100-200 lines of code to make the bridge for SWIG.
(to anyone else reading) does that sound trivially simple to the do the same with pybind?
SWIG is amazing. I have usually taken a different approach so that I could get the scaling you talk about.
I point SWIG at the real headers in my projects and use ifdefs to block out things don't want SWIG to see. There are also a few rules that should be followed when doing this. Things like: don't let SWIG see overloads if the other language doesn't support them; don't return raw pointers that need to be deleted; don't let swig see std headers, etc...
Then I make SWIG (optionally) run as part of the build. Any warning it makes need to be resolved somehow because CI will reject commits causing warnings. It cost just a little more dev time, but prevents eons of debugging time.
I haven't used pybind yet, so I can't compare the two unfortunately. From how I understand the docs it seems you would have to write the interface definitions in C++ and the compiler would generate the bindings for you, which sounds like a nice approach that could be a bit more cumbersome though as (again, in my understanding) forces you to write and adapt all bindings by hand.
Ownership can be a tricky question, but usually SWIG is pretty explicit about which memory/objects it owns and which it doesn't. Like the other poster I also often thought that my program was leaking memory as the Python interpreter does not immediately release unused memory, so even if everything is fine it can seem like your program is consuming more and more memory (explicitly collecting garbage using `gc.collect()` usually helps here). In general, you should be really clear about who owns a given object (either the C/C++-side of your code or the Python side) and avoid passing ownership back and forth between the two. I already mapped some pretty intricate code using SWIG (including a parser generator that created lots of object instances in C++ and passed them to Lua for manipulation, and where the Lua side could also create objects and link them to the C++-created objects) and so far didn't have any problems with memory leaks due to the binding code.
I have used SWIG a great deal. Any time I have had leaks near it they were in my code or not really leaks and just quirks of the garbage collector in the Non-C language.
I have used swig with Ruby, Lua and Java and it the code it makes doesn't leak unless the C/C++ code is bad, its just too simple.
What's also important is to have a simple, neat API, ideally C API. If you do have it, then writing wrappers for any language, even manually, without SWIG (which is great, but does add complexity to your project's build process), is simple.
Yes that's a valid point, manually writing bindings for C++ is very hard though (IMHO), as there are many intricacies that you have to keep in mind, and SWIG is really good at automating all of these away. For C-code the story is a bit different because usually the code is much less complex.
I've used cython, swig, and boost.python. I prefer pybind11 to all of them. It requires the least extra tooling and allows you to seamlessly transition between C++ and Python using the idioms native to each language.
A proprietary piece of software we use at work has a Python API for which they are using Boost.Python.
I've written a piece of software to get some data from the API and to analyze the retrieved data. The Python API has a bit of a "C++ smell" to it and doesn't feel completely pythonic but it's good enough. The bigger problem is that the vendor's idea of documentation is a single PDF generated from the class definitions and everything is either very sparesly or not at all commented, heh.
In fact, pybind11 was heavily influenced by Boost.Python, to the point where some of the syntax may look quite similar. It has one obvious advantage though, in that it doesn't depend on Boost :)
I used pybind11 for my open source asteroid detection code. It's very slick, especially it's ability to seamlessly convert between stl and python lists- for example a vector of strings converts directly to a python list of strings. Also the interoperability with numpy. A class containing an stl of primitive types and the dimensions can easily be turned into a numpy array. Very neat
ROOT https://root.cern.ch/ provides Python bindings through its PyROOT functionality. It builds the bindings (ROOT calls them "dictionaries") by parsing your C++ using their "cling" interpreter based on LLVM. As far as usability, ROOT is a big package but once a given it's rather easy and non-intrusive to create Python bindings for your C++.
I've used SIP, which wasn't too painful. The code you write to do the interfacing looks quite like the C++ definitions. I mostly used it as was an application which used PyQt. It probably is harder to use as it's not the most popular option.
Does anyone have experience with using these and able to compare/contrast?
I'm afraid of C++, but if I can wrap it and learn how to use from a python REPL, I think I can handle... any recommendations/tuotirals/howtos would be much appreciated. (The library I want to wrap is https://github.com/openzim/libzim )