TLDR Addendum for people like me who aren't immediately familiar with these terms
High
- Distributed practice: Implementing a schedule of practice that spreads out study activities over time
- Practice testing: Self-testing or taking practice tests over to-be-learned material
Moderate
- Elaborative interrogation: Generating an explanation for why an explicitly stated fact or concept is true
- Interleaved practice: Implementing a schedule of practice that mixes different kinds of problems, or a schedule of study that mixes different kinds of material, within a single study session
- Self-explanation: Explaining how new information is related to known information, or explaining steps taken during problem solving
Low
- Highlighting: Marking potentially important portions of to-be-learned materials while reading
- Imagery use for text learning: Attempting to form mental images of text materials while reading or listening
- Rereading: Restudying text material again after an initial reading
- Summarization: Writing summaries (of various lengths) of to-be-learned texts
- The keyword mnemonic: Using keywords and mental imagery to associate verbal materials
Problem is that people greatly prefer the low result study techniques since they requires less mental effort. It has been known for a while that they are bad, but since doing those allows people to study more hours for same amount of effort they do it anyway.
Not surprising highlighting has low impact. I never understood how people could thing that just putting color over something would help in anyway. Also that was one of the feature my former lab used to promote in its learning software and happily wrote cringe research papers about. I understand why: it's both easy to implement and track, which makes it a good feature for low-effort papers.
I've practiced highlighting on and off over the years, but it's never been for memorization. I've mainly used it to separate out critical passages to speed up and simplify future reading.
Absolutely this. I use it on my Kindle all the time for this same purpose, and come back to some of them when I know the Chapter they are in and I need to refresh a concept or phrase.
"Low" doesn't mean bad. Sure, practice is better than highlighting/rereading, but I expect that practice fails if the student makes a mistake that can be cured by re-reading. Also, practice takes a lot longer (costs a lot more), so it better have higher impact.
Highlighting is an indexing technique, and maybe a marginal improvement to reading. It's a piece of a solution.
One round of highlighting is almost worthless on its own. BUT. "Progressive summarization" (term coined by Tiago Forte) starts with broad highlighting, followed by 1-2 more rounds of increasingly refined highlighting, and finally rewriting material in own words based on the last round of highlights. Leads to deeper understanding and recall etc.
When I was in law school, I highlighted in different colors to mark different parts of cases (facts, procedural posture, legal holding, etc.). I found this to be very helpful, both when I was marking up cases, and when I was reviewing the cases and able to quickly find the different elements. This is especially important in law school, where professors cold-call students and expect them to know various facts about each case at a moments' notice.
I've made plenty of flashcards during undergrad and creation can definitely be both high effort and low effort. Low effort copy and paste from textbook, high effort when using a plugin for anki can mean obfuscating multiple parts of an algorithm/tree. I'm proud of my best flashcards - but is there any evidence that creation + studying is more effective than studying alone over the same timeframe? Anki suggests that on their site but I haven't seen much evidence for it.
Study -> create -> study -> create is effective for us programmers using google but is study -> create -> study -> study ... more effective than study -> study... ?
I'm not sure I understand what "study -> study" is. Is it studying without creating flash cards, or utilizing them? As in just studying the book, doing problems, etc?
If so, in my experience, then yes, integrating flash cards is much more effective. Especially for working professionals who often have to step away from what they are studying for many months at a time (except doing the cards for 15 minutes a day).
Yeah my comment wasn't clear. Creating the flashcards is a slow process but supposedly imparts 'deeper' knowledge for the creator. That is the claim I'm curious about. Anki has shared decks that anyone can download and study [1]. Say someone has an exam coming up that they need to prepare for. They have two choices: study flashcards given to them, or make and study their own flashcards. For the sake of this example the deck that would be given to them and the one they'd make are identical and contain sufficient information to achieve a perfect score. 3 hypothetical scenarios:
- A short length study period where a large proportion of time is used to create the cards allowing less time to 'test' their knowledge compared to if they had started reviewing right away.
- A medium length study period where the creation period is now a smaller proportion of the total time. Because they created the deck their self testing performance increases at a greater rate compared to testing with a deck given to them.
- A longer length study period where the creation period is now a negligible proportion of the total study time. The benefits of being the one who created the flashcards fade. Whether they created the flashcards or not matters less as knowledge of the deck is complete either way and studying the deck is done only for maintenence.
Either way this is just a question about the most effective way to 'download' knowledge. I'd always choose to make my own cards because I'd be able to make a more targeted deck and know how to use plugins to make the cards more interactive.
Making flash cards is in the low gain category, and that is very popular. Disciplined testing using those cards wasn't nearly as common from what I saw, people just made them and then forgot about them. Or just read through them without putting in effort trying to answer each one without looking at the answer. Doing those properly requires effort.
People do it all the time because it's fun and you get the initial kick. Like planning a budget, habits, hobby, exercise routine or spend a week collecting courses. Then the daily grind starts and you give up after a few days.
I, personally, just am reading a novel series Japanese, 8 books in. While reading, I have highlighted all the words/parts I didn't know in order to export them and make flashcards out of them. I haven't still started going through them, but the thought of doing so feels laborious, so procrastinating is bound to happen.
Finnish is a second language to me and what I've found better than flash cards is to just write down words I don't know as I'm reading. Initially this can seem terribkt slow because each book with have its own vocabularly set. As I continue if I read a word I don't know zi scan the list briefly. If it isn't there or I fail to see it I write it down again.
If you do not have time to study... I had to do long commutes while working part-time to get my CS degree.
Sometimes I was left with only 1 day of study for an exam and this was the best method for me.
The article is incredibly nuanced, and answers many more questions than only "is this method any good?"
I'd also say it's not correct to place rereading at low effectiveness. The result on the article is more complicated than that.
Also, the article is mostly about memorization learning, and entirely about things with correct/incorrect answers. How well this generalizes for learning the things you need outside of schools isn't there at all.
It's very surprising to me that elaborative explanation, summarizing, and self-explanation score lower than practice tests. I've found that trying to teach something to somebody else is the fastest way for me to determine if I really understand something.
So that method might be fast, but it isn't accurate, your explanations could easily be wrong without you knowing they are wrong. So even though you feel like you know it if you can explain it, many times people are wrong and give wrong explanations but still feel like they are experts, don't be one of those people.
It looks like the type of practice test matters a lot:
> a multiple-choice practice test increased intrusions of false alternatives on a final cued-recall test when no feedback was provided, whereas no such increase was observed when feedback was given
Sounds like practice tests are most useful when they are graded and the student gets that feedback. I suppose that makes sense. Flash cards and practice problems also fall under practice testing and it sounds like they were less effective.
It doesn't surprise me. Practice tests can show you you are unambiguously wrong about the details of things in a way writing my own summaries can't unless you go over them with a fine toothed comb anyway.
> ChatGPT seems to be very good at generating cards btw
So you can get very good at memorizing occassionally incorrect information? If I'm going to put in the effort to drill information into my brain with spaced-repetition, I have a high bar for the accuracy of that information.
I hand write all my cards, and think it's commonly accepted that the manual process of producing the cards is an important first step in the memorization process.
I'm not sure if this really saves time in the end. At least for me the part of taking a lot of information and thinking about condensing it in a few sentences is the most important part of making flashcards or similar summaries.
A lot of the time I end up never reading them, or reading them once because I already indirectly studied everything in the process of making them in the first place.
I read that “distributed practice” seems to be scheduling the learning activities to be spread out over time. I’m not sure if distributed practice has this, but “spaced repetition” seems to have some additional part to it where you don’t review the flash cards that you answer correctly as often as the ones you get wrong.
Personally I prefer coming up with them myself as I am studying the material because that's already the _first pass_ of studying. Then as I review I adjust and improve the cards.
No one really said anything about shortcuts. Building maps of knowledge that have logical progressions and connections is so much more fulfilling than rote memorization. I can either memorize that the Iron Age came after the Bronze Age or I can weave a story with as many details as I can find and understand the inner workings of how the transition from the Bronze Age to the Iron Age happened.
You can also identify patterns within domains, which is a pretty common way to begin tackling problems or questions that you don’t really know but can start making an educated guess about.
I think there's a very useless and useful way of memorizing, say, history or biology, and the useless way is probably better for passing exams, and the useful way is better for, everything else.
The useless way is: let me memorize all of the dates of the US Presidents. If the question is "When did Ulysses Grant become President?" you remember it's 1869.
The useful way is, let me learn about each Presidential administration and what baggage it inherited from the previous administration and passed onto the next one. Then, if the question is "When did Ulysses Grant become President?" you think, well Lincoln was assassinated shortly after the end of the Civil War in 1865, Johnson served out his term but definitely didn't get another, so Grant would have started in 1869.
This is just a guess: maybe thought experiments? In law, that might be randomly picking/generating cases and arguing the points to rehearse knowledge. In history, similarly it could mean role-playing historical scenarios and considering the individuals involved and the politics of the day.
I'm imagining these as group activities, not something one would do alone.
I also freely admit my naivety regarding these subjects.
Finally, I don't think these or similar approaches are "short cuts". Quite the opposite.
I dual majored in history in college in the 90s. The best professors made you write a paper every class that would fit on the front of a college ruled looseleaf paper that would be more comfortable to write on 3-4 pages.
The process of writing and editing taught you to think. Journalism writing before the shit-fest of online news was similar. 6 editors would wield the sword over whatever you wrote.
In my opinion, one of the most significant failures of modern education is the lack of emphasis on spaced repetition. It's really a shame that while we have lessons on a wide range of topics, including the latest social justice causes, there is one technique that can be truly life-changing, yet it's often not taught until people reach adulthood and have to discover it on their own.
I frequently hear this comment about spaced repetition, but I’m not sure how it would be different than modern education. In physics, we started with unit analysis, but it continued being a factor throughout the course. Then we learned kinematic, which didn’t disappear once we went into dynamics and both were constantly used with electrostatics. Similarly, in language courses, you start with a single conjugation in present indicative. You build out from there, but the present tense would still show up on every assignment, even after you’ve gone through pluperfect subjunctive. Needless to say, algebra courses freely called back to basic arithmetic and calculus called back to algebra.
I feel like I must be attacking a straw man here, as you’re not the only individual I’ve met who found spaced repetition revelatory, but I’m still not sure what I’m missing.
Spaced repetition is particularly effective when it comes to memorizing a large volume of information quickly and retaining it for the long term. This is applicable to a wide range of subjects, including history (facts and dates), physics (formulas and laws), mathematics (formulas and laws), and many others.
There is often a misconception that subjects like physics are solely about understanding and reasoning, but at the school level, it is necessary to have a solid grasp of the laws and formulas. Without this foundation, it is like attempting to write a program in a programming language without knowing the syntax. You have to stop and look up the syntax for each line of code, slowing down the entire process. Some things simply need to be memorized to achieve a certain level of proficiency.
People are already doing this though. If you are a historian, you are already having spaced repetition on your subject of interest, since every book you pick up from the field will cite the seminal information everyone knows in the field. You are constantly being refreshed on it. Likewise in physics or math, if you specialize in some niche, you are constantly reviewing the groundwork and in all likelihood reading a good deal of review articles as well. That being said, there's no point in the professor of history reviewing their calculus from undergrad, and there's no point in the physics researcher to review all the central dates and names of world history every few years. If you strive to have the breadth of an ocean you won't have time for much depth. There's just a finite amount of information one can intake a day.
> Without this foundation, it is like attempting to write a program in a programming language without knowing the syntax. You have to stop and look up the syntax for each line of code, slowing down the entire process. Some things simply need to be memorized to achieve a certain level of proficiency.
This is pretty much the opposite of my experience. I have learned many programming languages and never found it useful to memorize any syntax. I read the book/docs/tutorial once, and then reference it (or code I previously wrote) just-in-time whenever I need a reminder, and find that whatever I use repeatedly is quickly committed to memory. I don't see how memorization could make this faster.
This is definitely not my experience with space repetition. I really wanted it to work for me but I found it didn't help at all with retaining anything. I was using anki.
Curious: How many cards did you create? And how often do you test yourself? You really need to do it daily with few gaps for it to be very effective.
I would recommend not giving up but instead altering your approach. I tried spaced repetition multiple times in grad school and completely failed at it. I then tried again over 10 years later (and much older!) and it was really like attaining a super power. The success is even more remarkable because compared to my university days, I am much less actively using the material in the cards.
In my case, the difference between the two attempts was sticking to the daily schedule[1] and getting better at creating relevant cards.
And keep in mind: SR aids in recall, not in understanding. There have been times where I look at the cards on a certain topic and do very poorly (consistently). I then realize it's partly because some key understanding has been lost. I then need to do a proper study session (get the book out, read for understanding, perhaps do some problems). After that study session, I get good on the cards and stay good for a long time. In the last 4 years, I've needed to do this perhaps twice.
[1] Yes, I do stop from time to time, but those are the exceptions, not the rule.
I created cards as I was learning the topic. And I was doing it daily or twice daily.
I certainly wanted it to work, when I first learned about it I thought it was the answer. But I seemed to get worse and more stressed out because I wasn't learning anything.
I'm not too fussed about it not working for me. I don't think there is a one size style of learning.
This has been my experience as well (especially with Anki), though I've always felt like my ability to memorize information to be poor in general. In my experience, I find it easier to memorize things rather through brute force/frequent repetition. To use memorizing foreign language vocabulary as an example; repeating words to myself, listening, or writing them out for 30-60 minutes everyday or every few days (as opposed to spreading out fewer terms over, say, two weeks). The conjecture I have is that my brain commits to memory more heavily based on frequency/intensity during short intervals of time rather than long intervals of time, but it's just a guess.
Not sure if I'm just an outlier, but I think you're right that many learning heuristics aren't one-size-fits-all. Hope you can find something that's effective (and enjoyable) for you. :)
In all of my physics classes (high school, undergrad, grad), we were allowed to have a formula sheet with whatever we wanted written on it. Furthermore, important equations would just be given to us on the test, in case you forgot to write it. Even so, test scores were usually below 50% (corrected by curving after). Memorizing formulas to study would have been a laughable waste of time, the better approach is to practice a wide variety of problems.
If one has to look up a formula, can we say that they have a good understanding of it? If one had to look up a multiplication table each time they calculated something, how far would they have gone in learning?
I think you have it backward. All that time/energy wasted memorizing multiplications could have been used to understand more complicated topics. It's a cool party trick but it won't make you better at solving problems past high school level. I would prefer to work with someone who understand the concepts well and how to apply them than someone who can recall 32x9 or 10 digits of pi from memory.
When people complain about the difficulty of their college math courses or the math problems they solve at work, I can assure that it's not because they had a hard time remembering the answer to a multiplication. In a lot of cases, the equations they are trying to solve don't even have numbers in them. And when there are numbers, you won't be able to calculate them mentally because they won't simplify like they do in a no-calculator math exam.
You might not have realized, but a lot of the work when designing a no-calculator exam is to make sure people can compute the numbers mentally. When you step out in the real world, 32x9 becomes into 32.091x9.1^1.1. It turns ugly real fast when you have to deal with real numbers instead of a carefully crafted exam question.
I would have to look up Maxwell's equations right now if I needed them. Nevertheless, I am confident I have a good understanding of them. What I have in my head are important relationships (conservation laws, symmetries, wave solutions, etc.), not the exact mathematical formulas.
to add to this, I think spaced repetition is to learning, the way typing speed is to programming. It's good at solving for the minor frustrations and slowdowns that stand in between you & the complex reasoning to get to a solution.
When I can edit code via vim keybindings quickly, I can move things, sketch ideas, swap entire blocks, and so on, easily. That lets me check and look at things, and allow my whole brain to be spent on higher level things like looking at the control flow, applying aesthetic considerations to simplify, or whatever.
I failed Calc 3 as a college freshman because I had taken high school calc as a junior, leaving a full year between. I understood the ideas of calc 3, but couldn't remember the basic stuff like integrating tan() or whatever. (and I also didn't have any study discipline at the time to actually solve for my situation). So I couldn't answer the questions on a test even though I knew the steps I would need to take past a roadblock. Solvable with study, which would have been effective in the form of spaced repetition flash cards.
You could call the way the curriculum is organised "spaced repetition", but this isn't personalised and thus missing the feedback part of spaced repetition. Generally, children aren't taught the method of spaced repetition itself. "Learning how to learn" should be its own subject early on.
This seems to be changing a bit. I have two children in a US public elementary school. The teachers use various apps to have children review mathematics facts. The apps allow children to move at their own pace and also incorporates some spaced repetition. (Though teachers seem to pick whatever app they like and the apps are of varying quality.)
I really hope you're correct. It feels a lot like the debate between phonics and whole language to me, where how we teach makes it difficult for the average student to learn. People here are discussing Bloom's Taxonomy and higher levels of learning, but many students struggle with even the most basic concepts.
IMO you kind of do get spaced repetition in modern education. As you specialize, you frequently have major classes where its "first week or two we dig into what you already should know, then we really dive into the deeper stuff." Or you take up a job, and maybe you have to refer to that one topic from class years ago twice a year, that's spaced repetition.
In a sense, the nature of a specialized career forces you to engage in spaced repetition of the things you need to know, and allows you to safely ignore the stuff you never need to touch again. If we encouraged spaced repetition on each and everything, I would be wasting a ton of time on subjects that ultimately aren't relevant, at the expense of free time and maximizing time spent on the relevant bits.
This is the perspective many people have, but in my experience it falls short. To quibble with your examples:
> As you specialize, you frequently have major classes where its "first week or two we dig into what you already should know, then we really dive into the deeper stuff."
Yes, they do this. But I've often seen (especially at the undergrad level) instructors avoiding topics or challenging problems because it required material beyond just the basic math/calculus (e.g. some trig identities, differential equations, etc). So in practice, this is not happening.
> Or you take up a job, and maybe you have to refer to that one topic from class years ago twice a year, that's spaced repetition.
It's even worse at work, where you don't have a teacher to enforce things. Suggestions get shot down all the time because coworkers have forgotten some (even trivial) things they learned at university, and they do not want to review it. I've seen this happens where they avoid basic Calc I stuff, where they avoid stuff in data structures (e.g. union find). They'll implement a poor solution, and convince management it can't be done better. Or that it can but it's a long term risk because if that one person who knows "union find" leaves, we won't be able to hire someone to maintain it, etc.
> If we encouraged spaced repetition on each and everything, I would be wasting a ton of time on subjects that ultimately aren't relevant, at the expense of free time and maximizing time spent on the relevant bits.
Michael Nielsen did an analysis of this after using Anki for a bunch of years: The average amount of time spent on a card over one's lifetime is about 10 minutes. And so he used that as a metric when encountering any new piece of information: "If I don't learn it, do I think I will spend more than 10 minutes looking it up in the future?" If yes, he'll make a flashcard for it. If not, he won't.
But yes, I definitely have disabled cards on topics that I don't think I'll want to touch again in the future.
what do you call the two midterms, the final, the reading, the lecture, and the worksheets? all cover the same material at different times.
if anything, it's higher level education (4, 5, and 600 level college courses) where i saw a lack of spaced repetition, or any sort of learning beyond the professor presenting the material. I guess it's expected that at that point you have your learning technique figured out.
That's precisely my point. There should be a dedicated subject called "Effective Learning" or something similar, where students are equipped with the necessary tools to continue learning at a higher level after leaving school. The fact that individuals are unaware of these skills later in life is evidence of the education system's shortcomings.
Having an intensive course on a subject over 3-6 months doesn't really let you do much in terms of spaced repetition. If a course is a year long and material is briefly visited, then often revisited, that could work, but even with a year long course, if you see the material on a quiz, a test and (a little bit) in the final that's not even close to optimal.
In post-secondary education, the amount of material, typically due to the speed or density of delivery, may reveal that your learning technique isn't working. For example you may spend too much time on one topic rather than on a topic more heavily weighted in an exam.
I'm not sure but by spread out I would assume multi-year. What you described is still problematic. I believe this is why mixed grade classrooms worked out so well because this was inherently unavoidable.
Maybe I'm missing something, but it seems like spaced repetition is rather limited to the bottom two tiers of bloom's taxonomy. Can you flash-cards your way towards being effective at making the changes you want to see in the world?
Using spaced repetition is not the same as replacing all other learning methods with spaced repetition.
Just because spaced repetition isn't good for everything doesn't mean it shouldn't be used for what it is good at. It's good for remembering the specifics of things that you have already learned and understood.
No, you can not. But Bloom's taxonomy is hierarchical, so covering the first two rungs is very important and necessary for the steps after that. Spaced repetition systems allow you to make sure those bases are covered in an optimal way.
Pretty much this. Essentially, the bottom two tiers serve as the fundamental basis that ought to be robust and sturdy, but frequently end up being the place where students' academic progress comes to a halt.
His book "Why Students Don't Like School" is one of the best books on cognition I've ever read if you can transfer its knowledge from the pedagogical form in which it's presented.
It's so interesting to me that interleaved practice is an effective learning technique, that's probably the single method of study I used consistently throughout my entire schooling. Slightly bored of one subject? Okay switch to the next. Bored of this one? Switch to the next. CONSTANT context switching. I thought I was "cheating the system" by making my homework more interesting, who knew I was learning more effectively by employing a (moderately) effective study habit!
>What's the best way to learn a thing?
>By doing that thing, repeatedly.
Wow, really activates my almonds. That massive complex academic performance for a conclusion you could have received if you just asked your grandmother.
There are lots of examples of counter intuitive results and lots of examples of “obviously true” statements turning out to be false after a rigorous study. Here is an example of a study that reinforces the “obvious”. That doesn’t make the study without merit or that it was a waste of time.
There is an open question of what smaller things can you teach someone to do so they can do the next more difficult thing.
Music education is full of standard exercises like this that build skills and fluency in them until students can combine them in ways that seem magical to outsiders.
I'm confused by the overly concise title and humbly suggest:
"Improving Student's Learning with Learning Techniques That Improve Learning by virtue of being Effective".
It needs to be clear that not all techniques are the same. Only some are effective. Moreover, it's not clear that what it means for a learning technique to be effective is that it improves learning. And so—aha!—that is why the study is only about the effective ones.
From years of editing wikipedia the first sentence sets off alarms for using weasel words "many students", "some people". I hope the rest of the paper is more rigorous.
Teaching students is almost the same as mentoring junior developers on how to successfully onboard them. The goal is to let them actively control the project they're working on.
I hope not. Anki did not work for me at all. I genuinely got worse at remembering things trying that spaced repetition stuff. Definitely not the way I learn things.
Maybe. I created cards, and regularly reviewed them in the app, whilst creating more as I was learning new things. Is there a different way of using it?
All it did was stress me out because I wasn't recalling anything and all my cards were repeated basically all the time.
Is there a guide you can recommend that shows me how to use it properly?
But it honestly depends on the purpose, and the person. There are tunables about how often you see cards in the learning phase, how many new cards are released daily, and how impactful forgetting is on future reviews. Language learning cards focused on vocabulary, may need a longer "learning" stage than math or science topics. So a lot of this comes down to looking at your recall percent, and tuning / reducing new cards added cards to the review set until you are hitting 80-90 percent recall for a week straight. I don't even recall the defaults but especially if you are downloading other people's decks its easy to drown in new cards.
My other suggestion is to use Cloze deletion cards where feasible. It's much easier to learn the US Bill of rights as a set of clauses than trying to recall the seventh amendment on demand in its entirety, and Cloze makes it simple to produce these.
This is not the TLDR and the Anki worshipping is silly. You can’t use Anki to learn physics, or math. There’s only so much you need to memorize in those subjects, the rest you have to work through problems.
Some other comments had me doing some research on Bloom's taxonomy and I really think the principles apply in mathematics and physics.
Problem solving is great practice for learning, and covers the 3rd and 4th domain levels (application and analysis, respectively). However, what Bloom believed is that learning is hierarchal, and to apply and analyze you must complete the first two domain levels (memorization and understanding).
Anki in the terms of math and physics is great for that (and I will also rarely create a flashcard out of a exercise, but that is mostly because I think something similar will be tested on later).
Yes you can, check Michael Nielsen or "A mind for numbers". Anki is two highest highest utility learning techniques combined, so by pareto principle it is tldr.
High
- Distributed practice
- Practice testing
Moderate
- Elaborative interrogation
- Interleaved practice
- Self-explanation
Low
- Highlighting
- Imagery use for text learning
- Rereading
- Summarization
- The keyword mnemonic
// via: https://twitter.com/emollick/status/1630938305213628417 ( see tips How AI/ChatGPT can help )