How often do students guess their way through?

When designing online courses, like AlgebraX and GeometryX, a common cause for concern is that students can just "guess their way through" lessons and assessments. They're either totally lost, or not paying much attention, but they can still advance and get credit by brute-forcing their way.

This can be very problematic for the student (who is grinding away without really learning) and the course (which loses credibility when this type of student behavior is common). So here at School Yourself, we've been asking ourselves: 1) How often does this happen in our courses, and 2) What can we do about it?

To figure out how often students guess their way through lessons, our team developed a formula we call a student's guess index. It's a number between 0 and 1, where a greater number means the student is probably guessing. A student's guess index takes into account the question type (guessing on a multiple choice question vs. an algebraic question are handled differently), number of attempts, what students do between attempts, and a few other factors. Over time (and with lots of student data) we have found that a guess index of greater than or equal to 0.4 means a student probably guessed their way through a lesson. An index of 0.2 or 0.3 means they might have struggled through parts of the lesson, but they worked throughout without resorting to guessing. And an index of 0 or 0.1 means they confidently proceeded through the lesson.

So here's the distribution for the most recent 10,000 completed lessons:


Students appear to be guess their way through lessons 2% of the time. And we can do the same analysis for reviews (assessments):


Students seem to guess their way through reviews less than 1% of the time. It's a little comforting to know that students guess less on reviews than they do in lessons, since the reviews are what hold them accountable for their knowledge.

One or two guessing students out of a hundred may not sound too shabby. Indeed, these numbers are likely so low because 1) we prefer asking open-ended questions (with interactive graphs, algebraic expressions, numerical, or short answer) where possible, to ensure students understand what they're doing, and 2) our lessons are built around continuous, step-by-step user interaction, where the questions never get too far ahead of the students. But even though relatively few students are guessing, these are also the very students who need the most help! So now that we can precisely measure guessing behavior, what can we do to mitigate or prevent it?

First, it's helpful to know whether students are guessing at similar rates across all lessons and reviews (indicating a problem with the overall course design, question types, etc.), or if the guessing behavior is concentrated among certain lessons.


Half of the lessons exhibited zero guessing behavior, And guessing is concentrated among a handful of lessons, like Ambiguous case, Multiples, and Unsolvable equations. This is great news, because we can focus our attention on these lessons, see where students are guessing the most, and update the lessons (and add new branches providing feedback for common wrong answers).

When it comes to reviews, students don't guess as much:


The majority (69%) of reviews had zero guessing. And once again, it's a handful of reviews that are the most conducive to guessing. So that's where we'll focus our efforts.

So while students guessing their way through lessons doesn't seem to be a major problem for our content, it's something we can mitigate by focusing our effects on the most guess-prone lessons and reviews.

4 comments:

  1. Is there a way to check and see if my child is guessing he way through? Is there a tracking or other wise way to see his progress through in a teacher mode or something?

    ReplyDelete
    Replies
    1. THE MOST RELIABLE SPELL CASTER THAT CAN HELP YOU GET BACK YOUR LOST LOVE BACK WITHIN 24HRS. (DOCTOR0LOVESPELL@GMAIL.COM)
      My name is Tonyas Mary from Canada, I want to quickly tell the world that there is a real on line spell caster that is powerful and genuine, His name is Dr love0 He helped me recently to reunite my relationship with my husband who left me, When i contacted Dr love0 he cast a love spell for me, and my husband who said he doesn't have anything to do with me again called me and started begging me to come back. he is back now with so much love and care. today i am glad to let you all know that this spell caster have the powers to restore broken relationship back. because i am now happy with my husband. To anyone who is reading this article and needs any help, Dr love0 can also offer any type of help like, Curing of all types of Diseases, Court Cases, Pregnancy Spell, Spiritual protection and lot's more. You can contact him Via email (doctor0lovespell@gmail.com) check his blogs: https://doctorlovespell0.blogspot.com/  or check website:  https://doctor0lovespell.wixsite.com/lovespell or visit his FB page: https://www.facebook.com/Love-Spell-Doctor0-107453264273672/  

      Delete
    2. I have been married with my wife for 6 years and recently she broke up with me and it hurt me deeply when she told me to leave her alone and that she does not love me anymore when i was always faithful and honest to her. I tried all the ways to get her back buying her what she wants like i always did and she still left me heart broken and she even has a new boyfriend which destroyed me even more until a friend of mine from high school directed me to Dr Emu. This man changed my life completely. I followed everything he told me to do and my wife came back. I was stunned everything happened exactly like he told me. I had faith in everything he told me and everything was true. You can contact him via email: Emutemple@gmail.com He will help you in anything you need and quick to answer once you contact him through his whats-app number: +2347012841542
      For more destails about Dr Emu you can visit his Website: http://emutemple.website2.me/

      Delete
  2. I think this is the best way of the learning online

    ReplyDelete