Academic writing suffers, albeit to a lesser degree, from the same problem that has made legalese so annoying to read: the need for specificity.
Also, a need for references. Not just citations, but verbal references. What is jargon, after all, if not a specific and functional set of keywords and referents used in common by a given group or field? Jargon is efficient in academia. It is a common standard. If I write something in my field's jargon, people in that field will know what I'm talking about, and precisely what I'm talking about. This is more efficient, sadly, than trying to turn a clever phrase to describe a phenomenon there's already a perfectly dreadful word for.
It's not just specificity. Take for example the physical sciences' tradition of using passive voice, exclusively, when writing about the methods used in an experiment. You never write "We poured the mixture" or "We measured the transmittance", but rather "The mixture was poured into a 250ml Erlenmeyer flask" and "Transmittance was measured using a model 500 spectrophotometer with a 580nm filter".
The reason, as anyone who has read and followed more than a few "Materials and Methods" sections can tell you, is that this style of writing makes it very easy to skim through and find the relevant bits. The last thing you want to have to do with a pipette in hand is wade through someone's flowery prose to find out how many microliters to dispense.
For what it's worth, in Psychology (by APA standards) 1st person voice is discouraged, but so is passive voice. In the passages you use, removing 1st person (almost) requires using passive voice, but I would say removing 1st person voice is what discourages a very informal, prosey style, while the frequent use of passive voice often feels indirect.
So, use of the first person, plural is
quite widely accepted in nearly all of
current and recent mathematics.
I wrote my Ph.D. dissertation this way,
and one professor said "When you
say 'we' maybe I don't agree?"
and thankfully a fellow student
spoke right up and defended me;
she explained that using "we"
was standard in mathematics.
Once I was trying to socialize with
a high school English teacher and
sent her a draft of a paper I
was about to publish in some
applied mathematics and asked her
to give the paper a critical reading.
Soon she asked me if using
"we" was standard in mathematics,
and I had to say yes. She
gave me no more feedback! Gee,
that's much better than what I
got from English teachers in
high school and college!
In the end, I first learned to write
in college and
by writing proofs in pure mathematics;
the reason I was able to soak up
the lessons was that such writing, as
English, is so darned simple. Later
I branched out from such
simplistic writing.
Later I was trying to socialize
with a woman who was a secretary
in a university. She confessed that,
in her experience typing, etc.,
the really clear writing was from
the professors of mathematics
and the physical sciences. Maybe
she was just trying to butter me up!
I use "we" all the time...but I try to make it mean "me and the reader", i.e. "we then look at ....". In that sense, a paper should be a conversation between you and the reader even though the reader is passive in the interaction.
I'm struggling to figure out how you would write lab instructions in neither 1st person or passive voice. Would you just credit all actions to an anonymous experimenter, described in third person?
That's fair. For psychology methods it's easier to avoid both, at times, since you can write "participants viewed...", etc.. For other types of methods, especially where the object becomes implicitly understood, or would be redundant to state, I can see where passive voice might be useful. For instructions, you can just leave off the implicit "you should", EG "put X in Y". The tradeoff between passive and first person is important, and which is appropriate likely depends on the circumstance. They're both discouraged, but not banned, I'd imagine because poor writing often uses one or the other too often.
Passive voice has a sense of "objectivity" to it, which would explain its prevalence. In my opinion, it is harmful in the sense that experiments (papers) are by no means truly objective, but rather attempts at raising subjectivity to a broadly intersubjective level.
In my experience, younger PIs and the field I work in in general is moving away from the passive voice. While your experience seems to be that passive voice is clearer, from my perspective it is clunky and inelegant.
I disagree that 1st person voice results in flowery prose. Bad writing remains bad independent of passive or first person style.
It can be much clearer and, well, direct to use first person voice ("To assess wether the effect data X suggests holds true when Y, we do Z1, Z2, and Z3 as they cover Y1, Y2, Y3". This is clean and clear and much nicer than the respective passive voice. But again, perhaps just my opinion.
Also, a need for references. Not just citations, but verbal references. What is jargon, after all, if not a specific and functional set of keywords and referents used in common by a given group or field? Jargon is efficient in academia. It is a common standard. If I write something in my field's jargon, people in that field will know what I'm talking about, and precisely what I'm talking about. This is more efficient, sadly, than trying to turn a clever phrase to describe a phenomenon there's already a perfectly dreadful word for.