REMARK: A PEN FOR STUDENT FEEDBACK

Overview:

Remark is an interactive pen attachment and system for use in high school classrooms. This device helps to alleviate communication issues between students and teachers. The Remark system helps teachers better gauge their classes' understanding, while allowing students to give immediate feedback regarding their levels of understanding. Remark is made to enhance overall classroom experiences and increase one on one communication between students and teachers.

Duration: One Semester

Team Size: A team of four M.S. Human-Computer Interaction students.

Team Members: Lu Dong, Tara Ramanan, Laurel Warrell, Samuel Weiller

Skills Exercised: Observations, Interviews, Affinity Mapping, Survey Designing, Arduino, Processing, 3D Printing, Sketch, Heuristic Evaluation, Focus Groups

Project Roles:

I was individually responsible for creating and coding the information visualization for the teacher's interface in Processing, coding to connect the Arduino to Processing software in order to transfer student data to visualization, and wiring Remark pens to our Arduino. My other teammates completed the Arduino coding, the 3D Printing, and the teacher interface mockups. As a group we all completed Observations, Analyses, Research, Brainstorming, and User Evaluations.

WHY REMARK?

User Research:

At the start of this project, our team decided to focus on education technology as the basis for our project. Given this, we sat in on high school classes, observed classroom interactions, and had face to face interviews with many different teachers to get an understanding for how technology was used in classes of different natures. Walton High School and Atlanta Country Day School are the two schools that we observed throughout a series of days. Throughout these observations we were able to note student habits, the regularity and flow within the classrooms of different subject areas, and the types of technology available in each class. For broader insight, we completed telephone interviews and video chat interviews with teachers outside the Atlanta area. We also created surveys to further investigate the needs of teachers in our target population. Our intent was to supplement the data we gathered through observations and interviews to strengthen our task analysis and inform our final design. 

After reviewing our notes and conducting data analyses, we found that different teachers preferred using different systems for different kinds of student-teacher communication based on their intent and content. Teachers also felt that they were juggling too many systems at once. This oftentimes led to confusion for both, students and teachers between classes. Due to the numerous educational technology platforms being used, teachers and other users of these technologies required a lot of training and effort to identify which combination of technologies to use. This also led to the fear of not having technical support or alternate backup methods of communication present, should any of these existing systems break down. From this, it was clear that teachers did not need just another educational platform to integrate with their classes.

Findings:

At the end of our observatory period we found that:

  • Teachers are overwhelmed with the amount of educational platforms that they must integrate for classroom use. (ex. Piazza, BlackBoard, Edmodo, Parlo, etc. )

  • Teachers are not fond of how much time they have to spend learning and adapting to new technologies each school year and would prefer to a set of a few effective ones.

  • Students have a hard time managing which platforms should be used for what (ex. Not knowing where to look for announcements and homework assignment posting, checking the wrong websites for specific information)

  • Most students will not ask questions in class because they are concerned about what classmates may think about them or view them

  • Students prefer to hold off on questions and then speak to teacher in private at a later date and time

  • Students would ask more questions if they knew that classmates were also confused

  • Teachers over and under explain material because there is no feedback from the students

  • Teachers have no way to gauge students’ understanding as the class goes on because students are often nonresponsive when teacher asks question (ex. Does everyone understand this concept? Any questions?)

Brainstorming Sessions

Following this, we collectively thought of and proposed potential design ideas that we composed together. We then went through and added features to them as a group. At the end of our brainstorming session, we had 30 design alternatives. After adding details to these 30 designs, we voted and decided on the last 10 standing ideas. The last 10 standing ideas were (1) Terminal Broadcast System, (2) In-Class Collaborative System, (3) Wearable Magic Bands, (4) Standalone Module System, (5) One Unified System, (6) Unified Front-end System, (7) Grade Capture, (8) Bring Your Own Device, (9) Physical Data Visualization of Progress, and (10) Modules for Existing System. We then voted on these 10 ideas and to come up with the final three. 

The Final Three Design Alternatives:

1. Module Based Online Course Management System

Pro: Intuitive Con: An additional system

  • Teachers can only use the modules they need
  • Using a central communication system, teachers interact with students and not devices
  • Comprehensive Onboarding
  • Unified interface, despite multiple modules

 

 

2. In Class Collaboration System

Pro: Reduce Interruptions Con: Costs and Limited Control of Response Types.

  • Teacher broadcast polls, lectures, and notes
  • Students can ask questions through Help Board publically
  • Students can ask questions to teacher privately
  • Virtual "hand raise"
  • Students broadcast hand drawn notes
  • Teacher has class data daily

 

 

3. Student Wearable

Pro: Quick Feedback Con: Infrastructure

  • Student Response System
  • Teachers are able to get quick feedback on students' understanding of lecture content
  • Elicits feedback from students
  • No login required

 

 


 

Design Decisions

After more brainstorming and consulting with panelist at event that we opened to the campus, we decided to expand upon design alternative #2, the In Class Collaboration System, and tackle the problem of there being an interesting communication gap between students and teachers in high school classrooms.  

 

In Class Collaboration System - Initial Design

Given this initial design, we modified the In Class Collaboration System to better fill our problem space, focus on two-way communication between teachers and students, and satisfy our findings.

  • This initial design required students to bring their own devices to class. This would have been a problem for students without devices.
  • We went away with the on desk interface to avoid distracting students.
  • We added and removed certain features to focus more on student to teacher communication, where students preserve their identity.
  • We removed the option for students to respond anonymously. This way students would be less inclined to use the system for joking matters and teachers can keep track of all student feedback and responses.
  • This initial design focused more on creating a new way for students to raise their hands and less on eliminating the communication gap between teachers and students.

Final Design: A PEN?

With more brainstorming, we decided to integrate our design ideas with something that students must have on their person everyday, a writing utensil. 

Student During User Testing

The solution that we prototyped is a pen students can use to give their teacher live feedback on their understanding of lecture content. The system is designed such that it supports eliciting feedback from students that is anonymous to everyone in the classroom except the teacher. This helps eliminate the existence of false or hidden feedback that students are able to understand that result from peer behavior or embarrassment.

Materials Used to Create Remark Pen:

  • Soft Potentiometers to provide swiping capability
  • Paper Mate Pens
  • NeoPixel LED's to provide feedback that user response has been registered,
  • 3D printed cases (for soft potentiometers and NeoPixel)
  • Arduino, which each pen plugs into
  • Code to receive input from pens
  • Code to send data to Processing  

Each student has a device that clips onto the back of his or her pen, which contains identification information for each student, and is hence unique to each student. With this type of device, students are better able to “speak up” if they are not understanding, which allows teachers to tailor the class to the actual speed of their students’ learning. There are only 2 actions that can be performed by the student – swipe up and swipe down.

  • Swiping up signifies that the student does follow or does understand what is currently being taught in class.
  • Swiping down signifies that the student does not follow or does not understand what is currently being taught in class.
  • Feedback light, a green blink, signifies that the user's response has been registered.

Teacher's Interface

This device is used to send binary feedback to the teacher to inform them if the student is able to understand what is being taught in the class or not. All this information is collected and presented to the teacher in the form of visualizations so that the teacher can get live feedback throughout the class. This helps make the teacher more efficient by helping them understand which concepts are easily understood and which ones are harder for students to grasp, and hence need more clarification or emphasis.

After teacher is logged in, real-time data visualization that is dynamically updated. Teachers can view more statistics in the software.They can also click on any point on the line chart or any pie chart to view more information about that point. Teachers can also view students’ statistics. The circles around students profile photos give a quick information about the average percentage of understanding during previous class period. Different colors represent different percentage. Teachers can also click on each student’s photo or name to see additional details.

Live Mode:

This is live visualization of the feedback from the students that is displayed for the teacher. This visualization provides the information that teacher will need while teaching. The live visualization features:

  • Comprehensible from just a quick glance  
  • Minimalistic display that can be read even from a distance
  • Shows a pie chart graphing the number of students who are following/understanding the class versus the number of students who are not, for a given instance
  • A dynamic visualization and reflects the current average status of understanding in the classroom at any given point in time
  • A numeric representation of the same percentage of students who are understanding 

Offline Mode:

The teacher can also interact with the system after class or when he/she is not teaching. This s in “offline” mode, where teachers can look through already recorded information for in-depth viewing. 

  • Log in for each teacher.
  • Per student performance view/details over time
  • Class performance view over time.
  • Superset of the essential information displayed to the teacher in live mode.

 

 

Quick Look at Live Visualization:

Final Design Workflow

Here are screen shots of the teacher's live visualization. I coded this visualization to be extremely easy to glance and simple. Such an interface allows for the teacher to quickly understand and capture feedback from the students on the spot. The percentage's display is also color coded to match the colored rings around students profiles. These colored rings act as a quick way to view where students' are placed in terms overall grade information for the semester so far.

  • Grades of 85% and above fall into the green category.
  • Grades between 85% and 70% (inclusive) fall into the blue category
  • Grades between 70%  and 50% (inclusive) fall into the yellow category.
  • Grades below 50% fall into the red category.

Evaluations:

User Testing

Subjects: Six high school students

  • Subjects were first introduced to the device’s background, purpose, and forms of interaction.
  • Two users at a time tried out the device, allowing them to experience its form factor and how it could be used in a classroom setting.
  • After all six subjects tested the device, we conducted a focus group to assess users’ opinions and thoughts on the device

After user testing, we debriefed our notes from the session and summarized key themes into three categories to conduct focus group evaluations. 

 

Focus Group Evaluation

We focused on three areas of evaluation:

  • Form Factor - the physical design of the device
  • Usability - could students use the device without issue or confusion?
  • Utility - would students actually use this device if it was employed in their classroom? 

Later, we asked subjects if they had any suggestions for improvements to the device. We received extremely helpful feedback and suggestions from our users. 

 

Heuristic Evaluation

Subjects: Five MS-HCI students

Evaluators were given two tasks to perform with Remark: at any time, indicate understanding of the lecture using the device and, if polled, indicate their individual response on the device.

We used Nielsen’s 10 Usability Heuristics for evaluation.

After the mini-lecture, we asked evaluators to write down their comments and evaluations on a sheet of paper with Nielsen’s Heuristics listed.

For each item, we also asked evaluators to give us a grade from 1 (does not meet heuristics) to 4 (completely meets heuristics), and explain their gradings.

The evaluation was set in an environment that simulates a real classroom situation, where the instructor gives a lecture while students take notes. Although all the evaluators were graduate students and our target users are high school students, evaluators provided usability feedback from students’ perspective. Both qualitative and quantitative data were obtained from this session. 

 

Future Plans:

Iterating, Revising, and Improving

Based on the findings that our group has collected throughout our development process, heuristic evaluation, and user testing session we have decided on a few changes and improvements that we would like to implement to our Remark pen. 

  • Having an additional student dashboard to go along with the teacher visualization interface, were student can login to track their individual points of understanding/not understanding throughout the semester. this feature will be useful when it comes to preparing for tests and exams. 
  • Adding a lock and unlock feature to the device 
  • Incorporating up and down swipe direction labels
  • Decreasing the size of the feedback light
  • Adding a function to show a student’s current swipe status
  • Adding an indication on the device which would state that it is powered, connected correctly, and ready to be used

Unlike the prototype, the final product would be Bluetooth enabled, battery powered, and wireless.