The author has a "lack of equal experience in the three evaluated languages", especially C, but is comparing them for development time and SLOC based on a sample size of one developer and one very small program.
He rewrites a distributed program as a threaded one, because Rust and Go didn't have a distributed computing library, but says Go and Rust have "similar performance" in HPC.
He says he was surprised at the C performance, but there's no investigation of a cause. Maybe just because C is slow...
I don't mean to be overly critical, but these results in the paper mean absolutely nothing. It's a waste of time.
> I do agree that the advisor should have pushed back lots harder on the methods; it's a weak paper, even as an experience report.
I agree - this seems to be mostly the advisors fault, by the looks of it it's not that the student didn't put a lot of work into it but methodological problems are what advisors are supposed to be there to address; I saw a lot of this back in college, at least with technical papers it's easy to tell what's wrong, with the humanities it's just a complete disaster.
I agree with your assessment only thing I did not understand is the Erlang integration with other languages. I am not sure if he is saying that NIFs are bad or he is saying there are is not support for other languages. I think non of these are true...
I agree that the paper isn't very convincing. One conclusion you might draw is that learning to write performant code in C is harder than learning to write performant code in Rust. That could just be a side-effect of Rust's much stronger standard library, though.
Yes, because the point is being able to write (on a resume): successfully presented thesis on "Evaluation of C, Go, and Rust in the HPC environment".
The advisors should have encouraged a smaller and more manageable scope with a more thorough methodology where the results would actually be worth something.
The goal is for the student to get some experience with performing their own research wherever that may lead them. This is actually a pretty impressive piece of independent work for an undergrad.
"Correct methodology" and "real research" are very subjective things. We often judge too harshly, and as long as the paper is not misleading, you can judge it's merits directly without resorting to non-science.
Peer review should sort out the bad science from the good. The above link is just laziness: they have correlated P values with low quality research, and have decided to use it as a crap filter in leu of better, but more expensive, peer review.
The article submitted was not peer reviewed, and could very well be bad science, but I think accusations of non-science are not very productive and do not provide feedback in how to do better.
> The above link is just laziness: they have correlated P values with low quality research,
My point was rather that there was so much low quality research they decided to try to filter it by banning a statistical tool. Are you saying the reasons for so much low quality research had nothing to do with methodological problems? Maybe it's not low-quality, since you know it's "subjective"...
> The article submitted was not peer reviewed, and could very well be bad science, but I think accusations of non-science are not very productive and do not provide feedback in how to do better.
You are addressing a point I never made. What I said was(quote): The advisors should have encouraged a smaller and more manageable scope with a more thorough methodology
Because an important part of undergraduate education is in fact learning to be able to tell what good research is. The advisor's aren't really benefiting them by not pointing out methodological problems and letting them do research ">>>>> wherever that may lead them" - because that only encourages methodologically sloppy work in the future.
The problem with reading this in BS thesis form is that it hasn't gone through the wringer enough. Most of the performance comparison results vs C are bogus because (to its credit, as noted), the author accidentally compiled some important support libraries for the C version without optimization and used those on the cluster. Oopsie.
As briefly mentioned in the previous chapter this performance regression might have been caused by the two unoptimized libraries that were compiled on the development laptop and copied to the cluster.
You are correct this was definitely an oversight and I would have redone the measurements if time allowed for it.
However I just want to state that the libaries were built with optimizations just not on the target platform which might not have been clear from the qoute.
Hey Floya, I'm writing this here hoping you'll see this comment. You're getting a lot of unjust responses in here. The commentors are reading your thesis with the eyes of a trained skeptic expecting peer reviewed, collaborative work that takes years of experience to produce. They're nitpicking flaws and mistakes we all make when we first start researching.
Your thesis is in fact really impressive and very nicely written! Your conclusions and future work section is excellent. Really very well done. Congratulations on finishing it. I hope you don't let the irrationally negative feedback here dissuade you from continuing your work, you certainly have a bright future ahead of you.
Hope the HN experience hasn't felt too bruising. The use of go and rust makes it quite interesting for those of us interested in correct code, not just fast code. And most of the criticism is that you didn't write the paper the commenter would have wanted to see, or didn't write a graduate level paper. And good on you for publishing it on github.
Keep in mind, this is a bachelors thesis. It's not a peer reviewed research article in a respected systems journal or conference. I mean this in both a positive and a negative sense. Positive in sense that the purpose of a bachelors thesis is to introduce undergraduates to an extended, self directed project with an element of research to it. As someone who has supervised many such projects, I think that this is a pretty good one. Great work! At the same time, since the author is an undergraduate, presumably with a full coarse load, this is not a high quality research publication. The results are probably a bit shaky and the methodology is perhaps a bit off. But that's ok. Every excellent piece of work has to start somewhere.
When you do not use the comma-ok form, you are specifically asking the map to return the zero value for non initialized elements (which is something fine to do in many cases).
Well, you're implicitely asking for it. In Rust you have to explicitely specify your intent, i.e. not by omitting something, but rather by specifying that you want to ignore it.
True. I agree it could be better if omitting the check looked like the exception and not the other way around. You could also argue that encoding non-presence on the type is more powerful/elegant/whatever than multiple return values. But what I wanted to point out is that the problem is not that "Go indicate[s] this error case with a return of zero". The language has an easy way to check this and he ignored it (he eventually used comma-ok in his code, although he makes the check first and the assignment later in different operations for no apparent reason).
I feel that it's a shame to test C rather than C++, because the latter offers so many advantages to developer productivity -- the paper cites memory management and type-related errors as being major causes of developer unproductivity, and C++ has automatic memory management and templates and such.
Also, it doesn't restrict the use of C libraries, so even if you've been writing C for HPC code all these years, there should be minimal technical difficulties in switching over, except for the time it takes to learn C++, which shouldn't be long if you know C well.
Looking through the links, and methodology makes me question a lot of things, most of them regarding his choices in how he allocated memory. First and foremost, he was using the glibc malloc, which belongs in a museum, not in a running system. One of rust's big advantages in performance comes from that alone, given all the memory that's being allocated and destroyed. Which leads to the second question of how he did allocate objects. He knew he was going to allocate a number of objects, yet he didn't pre-allocate a large chunk of memory ahead of time to make it so he wouldn't have to call malloc and friends as often. I wouldn't be surprised if spending a bit of time just tuning memory didn't cut significant time from how long execution takes. I really wonder how much time was spent in the system context of all three of those; I'd be willing to bet that the final part of the examination would probably fare much different had he used the same malloc for the C program that rust and go already use.
The programmer is by far the slowest part of HPC. It often makes sense to use the language you're fastest at instead of the language that will run the fastest.
It's very common for experiments to be written in languages not know for their performance (Java, Matlab, Perl, and Python for instance).
For reference, here are some of the packages commonly used in experiments[1]:
BLAS Fortran http://www.netlib.org/blas/#_software
NCBI BLAST C++ ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/ncbi-blast-2.2.30+-src.zip
BFAST C http://sourceforge.net/projects/bfast/
BioPerl Perl https://github.com/bioperl/bioperl-live
Bowtie C++, C http://sourceforge.net/projects/bowtie-bio/files/bowtie/
Clustal C, C++ http://www.clustal.org/omega/#Download
cp2k Fortran https://github.com/cp2k/cp2k
Gromacs C https://github.com/gromacs/gromacs
HTSeq Python https://pypi.python.org/pypi/HTSeq
MUSCLE C++ http://www.drive5.com/muscle/downloads.htm
MrBayes C http://sourceforge.net/p/mrbayes/code/HEAD/tree/
OpenFOAM C++ https://github.com/OpenFOAM/OpenFOAM-2.3.x
SAMtools C https://github.com/samtools/samtools
SNAP C http://korflab.ucdavis.edu/software.html
fftw3 C https://github.com/FFTW/fftw3
[1]: I went down this list and picked some of the ones I remember using (been out of the HPC world for a couple years): https://portal.tacc.utexas.edu/software It'll be a bit skewed towards genomics.
The list is not just skewed toward bioinformatics; a majority of them is bioinformatics software. I would not call BioPerl or HTSeq high-performance-computing software. Some of the software are written by academic programmers with all attendant baggages. I am not aware of any bioinformatics software written in Fortran, which is not popular in the bioinformatics community. But this is not to say Fortran might not be a good fit. The Broad Institute is rewriting some of its tools in C++ with help from Intel to leverage vector instructions. Who is to say if they had started with Fortran they might not have gotten vector instructions for free earlier.
... the reason is mostly legacy. All your libraries are written in fortran. Everyone learned fortran in the past and they don't want to switch.
As an example, I've seen physicists use fortran for parsing and transforming the output of a program. The cluster at our department has an awesomely updated ifort but no python2.7 (I think it has 2.2 or 2.3 or something). C++ isn't used directly either.
But yeah, there should have been fortran and some other languages there.
I'm a bit surprised that the author didn't mention julialang.org in the report - it's much more suited for technical / scientific computation, while Go and Rust seem better for systems-level concurrency/design.
My biggest criticism of this work is that it does not consider C++. Most of the big HPC projects today are written in C++, not C, and to the best of my knowledge this transition happened in the 90s.
> Most of the big HPC projects today are written in C++
My experience is limited to one single field, but all weather prediction models and climate models that I know of, are in Fortran. And I would count them as big HPC projects.
I have not done a census of the various libraries around, but all of the free finite element libraries in common use (deal.II, libmesh, and FEnICs come to mind first) are written in C++. I suspect Fluent and Abaqus are written in C (maybe Fortran?) but they are a bit older.
Trilinos is another C++ example, while PETSc and HYPRE are written in C.
My tentative conclusion from this is that newer projects tend to use C++ and slightly older projects use C.
Yes, there are things like ODEPACK, QUADPACK, and FFTPACK, but those are not under development anymore (as far as I know). The only widely used Fortran library still under development I can think of is LAPACK.
I did not count the occasional 'bespoke' code. There are still some Fortran applications in development for particular purposes (like MOM), but those are harder to survey.
I do not think free finite element libraries are representative of commercial software. They usually include a mix of different languages, and that includes lots of Fortran, which is used both in the core (in solvers and linear algebra libraries, for example) and at the highest level for user subroutines. I have seen people using either Fortran or C++ with Abaqus, LS-Dyna or MSC, but nobody writting C (except for some very experimental solvers).
Most users are usually fine with whatever the GUI allows and a bit of Python, but for those of us developing new models, Fortran is probably the most useful language, followed by C++.
There is still a lot of Fortran being written, but a lot of it is on legacy projects (scientific codes have a habit of lasting a long time). While there are some cool things like coarray Fortran and some groups require that everything be done in Fortran, C++ is a favorite for new projects afaik.
It's easier and more precise to say something like "run the quantum Monte Carlo code" than "run the quantum Monte Carlo model" or "run the quantum Monte Carlo software."
A quantum Monte Carlo code will of course include a model, but I think people don't want to call it "software" because it's so research-grade and janky. "Program" seems better, but I think that implies that it's a static thing (not in a constant state of development).
The plural "codes" is used because usually a research team has historically implemented a bunch of models into disparate codebases.
An interesting case study, though not really HPC or scientific computing related. It would be good to see some profiling to understand exactly why each implementation was slower in particular ways. From a HPC context, linking with Fortran and C codes, and exchanging data with them, is very important.
Would've been nice with some internal links in that pdf, especially in the table-of-contents (links to sections and subsections). This is simple with hyperref.
"In times where compilers are smart enough to basically rewrite and change code for
performance reasons it is completely inexcusable that the order of source arguments to
process is still that relevant."
After reading the comments on C and the difficulties the author had in the compilation process.... All I can say is Garbage in garbage out. I have worked in the HPC enviornment for a number of years using both Fortrash and C. The errors the author encountered would have been avoided by someone with any experience developing large softwares. I learned these things after working on my first project and now they barely, if at all, factor into my development time. Make sure you build and keep a dependency graph while you are developing and this becomes a non issue.
I don't disagree. Fortran is brilliant at what it does.... but I find many of its object oriented features to be very obtuse compared to C and C++. The way it uses pointers drives me insane. Building certain datastructures in fortran just takes too much time.
I kind of wish I could find a good way to make Fortran and C play nicely together. (There are some compiler dependent peculiarities that I have encountered that make it difficult.)
Fortran's intrisics and file handling capabilities make it indispensable in the work I do.... So it is really a love hate relationship.
It has some pretty basic flaws that are pervasive that the reader
should be aware of. First off is the "productivity" measure: the last
time I looked around, this was nearly impossible to quantify for
software development. The author chose SLOC and development time as
stand-ins for productivity. SLOC has a connection with code quality
and time of development, but development time is well known to vary[1].
In particular, development time in this thesis is linked to results in
a pretty classic "Psychologist's fallacy"[2]; the author generalizes
their experiences as conclusive. This implies in particular that
"development time" in this thesis should be thrown out for meaningful
conclusions, as the sample size is 1. It is, however, an interesting
experience report.
Another basic flaw is the connecting of 'modern' with 'good'. The
author remarks the main disadvantage of C and Fortran are their age;
this crops up here and there. Workflow tooling in Go and Rust are
major focuses by the developers: C workflow tooling is usually locally
brewed. This does not make C-the-language worse.
More on a meta level, my advisor in my Master's work drummed into me
that I should NOT insert my opinion into the thesis until the
conclusion. So I found the editorializing along the way very annoying.
And finally, and very unfortunately, the experience level of the
author appears to be low in all three languages; this is significant
when it comes to implementing high performance code.
---
Now, for the interesting / good parts of this experience report.
Standout interesting for me was the Rust speedup on the 48-core
machine. I did not expect that, nor did I expect Go to make such a
good showing here as well.
In general Go performance/memory made a surprisingly good showing
(to me) for this work. I shall have to revise my opinion of it upward
in the performance axis.
I am both surprised and vaguely annoyed by the Rust runtime eating so
much memory (Would love to hear from a Rust contributor why that is
and what's being scheduled to be done about it).
Of course the stronger type system of Rust catching an error that C
and Go didn't pick up is both (a) humorous and (b) justifies the type
community's work in these areas. I look forward to Rust 1.0!
One note by the author that is worth calling out stronger is the
deployment story: Go has a great one with static linking, whereas C
gets sketchy and Rust is... ??. I believe Rust has a static linker
option, but I havn't perused the manual in that area for some
time. For serious cloud-level deployments over time, static linking is
very nice, and I'm not surprised Google went that route. It's
something that would be very nice to put as a Rust emission option
"--crate-type staticbin".
Anyway. I look forward to larger sample sizes and, one day, a better productiv
> Would love to hear from a Rust contributor why that is and what's being scheduled to be done about it.
I haven't spent time with the code yet, so I can't comment. It shouldn't be the runtime, though, as our runtime is about as big as C or C++'s. My first thought would be that Vec's growth factor may be poor in this instance, but without reading the code, who knows.
> Rust is... ??
Rust statically links everything but glibc by default. Experimental support for musl was added last week.
So, basically _every_ non-assembly programming language has a runtime. Even C and C++. But not all runtimes are created equal. Languages like C, C++, and Rust have a very minimal, no setup required runtime. Other, higher level languages often have something significantly more heavy. People will sometimes say "no runtime" to mean the class of languages which have very small runtimes.
Well, Go and Rust can do better threading on this problem than a not-really-optimized C code with OpenMP. Thats a start. Message passing has to be shown next, and then one can decide whether the installation and runtime of Go/Rust/Python/Julia on something like an IBM Blue Gene/Q with its custom compute node OS is easier than a link flag.
80 pages for that, with one user and one implementation of one algorithm ? Sorry but the rate knowledge/size is very low, so don't be impressed by the page number, it harms the paper more than anything else. Even for a bachelor thesis, it's mostly "filling"
I often find bold text helpful for key points, or for the first mention of key terms to help orient me in a document. There was something about the mix in this document that felt inconsistent. I found myself pausing to try to figure out why the author wanted my attention on this particular word or phrase.
Wildly offtopic- reading papers like this bring to mind for some reason, that the next evolution of CPython, the one that deals with the Python2/3 rift should be written in Rust.
He rewrites a distributed program as a threaded one, because Rust and Go didn't have a distributed computing library, but says Go and Rust have "similar performance" in HPC.
He says he was surprised at the C performance, but there's no investigation of a cause. Maybe just because C is slow...
I don't mean to be overly critical, but these results in the paper mean absolutely nothing. It's a waste of time.