Sequential incoherence in a multi-party synchronous computer mediated communication for an introductory Health Informatics course
Abstract
Online courses will play a key role in the high-volume Informatics education required to train the personnel that will be necessary to fulfill the health IT needs of the country. Online courses can cause feelings of isolation in students. A common way to address these feelings is to hold synchronous online “chats” for students. Conventional chats, however, can be confusing and impose a high extrinsic cognitive load on their participants that hinders the learning process. In this paper we present a qualitative analysis that shows the causes of this high cognitive load and our solution through the use of a moderated chat system.
Introduction
The University of Texas School of Health Information Sciences (SHIS) offers graduate degrees and certificates in Health Informatics. As part of its degree programs SHIS offers several of its courses online. Online courses allow SHIS to reach remote students who would otherwise be unable to attend, and permits students to work asynchronously. Both features will be critical in the near future, as SHIS prepares itself to train thousands of students under the HITECH Act of 2009 (part of the American Recovery and Reinvestment Act, ARRA) (1).
Online courses have served SHIS well and generate the same outcomes as traditional classroom-based instruction (2). However, one particular deficiency of online courses mentioned both by students and instructors in class feedback is the lack of a sense of belonging to a community. The lack of belonging is a well-known problem of online classes that can lead to feelings of isolation and lower completion rates (3). The Foundations of Health Information Science I (F1) class at SHIS addresses this concern through the use of “chats” - synchronous weekly sessions in which students and instructors discuss the material and any ongoing concerns.
Shortly after incorporating chats into the F1 class we noticed that students continually interrupted each other and inserted comments that confused other participants. Multi-party synchronous computer-mediated communications (CMC) like the F1 chats have only recently become an object of study for linguists and psychologists (4–7). Despite the dearth of literature this particular phenomenon is already being studied in non-educational contexts like Internet Relay Chat (IRC). The simultaneous multiple threads of conversation and continuous interruptions are cognitively challenging (5), especially for novice users. In other words, multi-party synchronous CMCs impose an extrinsic cognitive load on the user. We therefore believe that understanding and avoiding the cognitive load of multi-party synchronous CMC is crucial to reduce confusion and improve their educational value.
In this paper we explore the phenomenon of multiparty synchronous CMC from the perspective of an online health informatics course at SHIS. We present a retrospective qualitative analysis of the changes the F1 chats went through and the lessons learned as we tried several multi-party synchronous CMC tools while we searched for a way to reduce the cognitive load the chats imposed on our students. Our analysis will help other institutions teaching online informatics programs benefit from our experience.
Setting
The University of Texas Health Science Center at Houston (UTHealth) is located in Houston’s Texas Medical Center. SHIS is one of UTHealth’s six schools, and the only freestanding school of Biomedical Informatics in the United States. SHIS offers certificates, Master’s and doctoral degrees in Health Informatics.
The Foundations of Health Information Sciences I (F1) course is an online three-credit survey course that introduces students who are new to biomedical informatics to its key concepts and fundamental readings. F1 is taught by an instructor of record (JRH) and a teaching assistant (JCG). Mastery of F1 content is a predictor of success in Informatics graduate programs (2). Most certificate, Master’s, and doctoral students take the F1 course in their first semester at SHIS. There were 31 students enrolled in the F1 course in the semester discussed in this paper.
A chat in F1 is a weekly hour-long synchronous session centered on a previously assigned topic. Students are encouraged to bring questions and comments about the designated topic to the chat, but are also allowed to discuss anything they wish.
Discourse analysis terminology
From here on this paper will adopt the terminology of the discourse analysis literature. Each participant in a multi-party synchronous CMC is an interlocutor. Each time an interlocutor posts a message, he or she is taking a turn and holds the conversational floor. If an interlocutor formally attempts to gain the floor, he or she is requesting the floor. Turns that are related topically to one another are coherent. Sequences of coherent turns are conversational threads. Successive coherent turns are called sequentially coherent.
Chat systems
Moodle Chat
SHIS uses the Moodle (www.moodle.org) open-source course management system to deliver its online courses. Moodle includes a chat module that uses HTML and JavaScript to deliver a simple chat experience. Users take turns by typing into a text box and hitting Enter to submit their messages to the Moodle server, which shows them to all interlocutors in a main conversation window that constitutes Moodle Chat’s floor.
Moodle Chat was our original choice for a chat system in F1. However, the class instructors felt that students were distracted by the chaos in the conversation. Smith describes this phenomenon and suggests etiquette as a way of addressing it (8). We implemented Smith’s etiquette by asking students to request the floor by posting “?”. Smith’s etiquette was almost universally ignored.
In practice Moodle Chat was uncomfortable to use. Broken connections were frequent, and every turn could take several seconds to appear on the floor. The lack of responsiveness was jarring to novice users, as evidenced by repeated comments from the students during chats.
Second Life
Second Life (www.secondlife.com) is a virtual reality environment that allows users to create avatars and navigate them through an immersive world (Figure 1). Second Life offers voice-based synchronous CMC, but it requires extra hardware and configuration. We therefore used its text system, which is functionally similar to Moodle’s but overlaid on the graphical environment. The Second Life client software also resulted in a fast, responsive chat experience.
We tried Second Life as an alternative to Moodle Chat. We hoped its immersive, embodied environment would make text-based chat more fun and appealing. Students liked Second Life, and let us know it. Unfortunately, the immersive environment was a problem too.
First, we had to contend with most students’ inexperience with Second Life. Questions like “how do I walk?” and “how do I dress my avatar?” were extremely frequent and quite disruptive. We asked students to raise their avatars’ hands to request the floor, and the request was again universally ignored. Second, at that point in time Second Life was not yet divided into “mature” (adults-only) and content-restricted areas as it is currently. We occasionally had to contend with uninvited and uninhibited interlocutors, judged Second Life to be a liability risk, and abandoned it.
CoverItLive
CoverItLive (www.coveritlive.com) is a commercial but free to use system to handle interactive Q&A sessions over the World Wide Web. CoverItLive is a text-based system. The key difference between CoverItLive and Moodle and Second Life’s CMC systems is that CoverItLive is moderated. The producer in CoverItLive is the creator of the chat session. In our case, the instructors assumed the role of producers. A producer can see interlocutors taking turns before they appear as part of the main discourse. The producer moderates by deciding when to allow a turn to become part of the main discourse. Producers can also save submitted turns for later, or simply ignore submitted comments. Students therefore only saw moderator comments and selected turns from their classmates.
Although students were initially confused by their turns not appearing in the common dialog in real time, they eventually adapted. The instructors (JRH and JCG) subjectively judged CoverItLive to be extremely appealing. Conversations carried on in CoverItLive flowed more fluidly than in Second Life or Moodle chat. It is this perception of conversational ease with the system that we explore here.
Methods
We retrospectively annotated chat logs from the F1 course at SHIS using ChatLinE 2.376-JRH (6). Since our version can only read a single input format (Moodle’s chat logs) we edited the chat logs for Second Life and CoverItLive with a text editor to reorganize their fields and make them match the format of Moodle Chat logs. We ignored all automated messages announcing the arrival and departure of interlocutors to the chat system. We also ignored turns that were never posted to the floor in CoverItLive.
We used a single semester to ensure that the chat participants were all drawn from the same pool. For each type of CMC we measured the coherence of the conversation (5, 9). In other words, we linked the same topics throughout the chat into threads, so when two conversations happened simultaneously we kept them in separate threads. Three of the authors (JRH, JCG and PBS) annotated the chat logs individually and then reviewed and discussed each other’s annotations until we reached consensus on all of them. We obtained basic descriptive statistics from ChatLinE.
We used ChatLinE’s vertical reference diagrams to evaluate the simultaneous occurrence of conversational threads. A vertical reference diagram is a visual representation of the flow of a conversation in which each message is visually connected by an arc to the next message in the same thread. Vertical reference diagrams allowed us to evaluate how sequential the flow of conversation was, and provide a visual representation of thread overlap. We also computed a weighted average, for each chat system, of the number of turns taken during periods where there was more than one overlapping conversational thread.
An isolated turn is a single turn that did not refer to a conversation thread and was not responded to by anyone else (see the second line in Figure 2). Isolated turns can be considered single-turn threads. Isolated turns interrupt the flow of conversation, as they are off-topic for the current discussion and do not shift the topic in a new direction. The total of isolated turns suggests the number of interruptions interlocutors had to process in the course of the session. It is therefore one measure of sequential incoherence. We eliminated social turns at the beginning and end of the session and then counted the number of isolated turns left, which we divided by the total number of threads to obtain a normalized measure of disruption.
We also used ChatLinE’s sociograms to study the interaction patterns within conversation threads. A sociogram is a graph where each node represents an interlocutor and the edges represent exchanges between interlocutors. Self-referring edges appear when an interlocutor takes two turns in succession within the same thread.
Results
We studied eight chat logs from the same semester of SHIS’ F1 course. During the time covered by the study the F1 course used three different multi-party synchronous CMC tools: Moodle Chat (two chat logs), Second Life (three chat logs), and CoverItLive (three chat logs). The number of interlocutors ranged from 11 to 16 (average 13.6, standard deviation 2.5). These interlocutors took between 71 and 336 turns (median 177) that were grouped into 11 to 88 threads (median 24). The number of threads was similar in Moodle Chat and CoverItLive, averaging 20 threads per chat. The Second Life chats had an average of 72 threads.
Second Life and Moodle Chat
Figure 2 shows a typical vertical reference diagram from a section of a Second Life session. There are up to three overlapping threads. The interlocutors take the floor and introduce new topics without waiting for an exchange to end. The single interlocutor who adheres to protocol and requests the floor by posting “?” is ignored.
Figure 3 shows a sociogram from the same Second Life chat. All but one student (number 12) participate in threads with the instructor or the TA. Only student 12 has an exchange with other students (4, 10, and 15). This pattern is representative of the Moodle chat and Second Life sessions. Few students took turns after other students, and the vast majority of the exchanges were between a student and the instructor or TA.
A sociogram from a Second Life chat. The instructor and teaching assistant (TA) are labeled. Line thickness increases with the number of turns between two interlocutors. Numbers that are not shown correspond to interlocutors that did not participate in threads.
In these unmoderated systems, the threads consist of questions and answers to and from the instructor and TA. In fact, despite the free form and lack of moderation, the instructor and TA wrote on average 51.3% of the words in these sessions. In other words, there is very little dialog between students, and the instructors provided the majority of the content.
39.1% of turns were taken during overlapping threads in Moodle, and 32.8% in Second Life. We also found that the proportion of isolated exchanges was, on average, 21% for Moodle, and 45% for Second Life.
CoverItLive
By comparison, CoverItLive sessions tend to be more linear and sparser than unmoderated sessions (Figure 4). Despite the presence of moderators and the ability to ignore comments, there are still overlapping conversational threads. 6.0% of turns were taken during overlapping exchanges using CoverItLive. There are fewer overlapping threads than in the unmoderated systems. The sequential coherence of the CoverItLive chats is therefore greater than that of Moodle and Second Life chats.
The sociogram for the same session (Figure 5) shows that the dialog pattern is still the same. Exchanges on CoverItLive are predominantly between instructors and students. The instructors provided, on average, 61.8% of the words ni CoverItLive sessions. On average, 20% of turns in CoverItLive sessions were isolated.
Discussion
Sequential incoherence is a well-documented phenomenon in other fields. People eventually adapt to the idiosyncratic style of overlapping threads (5). However, the SHIS F1 class is in many cases our students’ first online educational experience, and their first CMC experience as well. The confusion and extrinsic cognitive load placed on students by sequential incoherence is undesirable for our student population.
The perceived low sequential coherence of the sessions in Moodle and Second Life is a result of the large number of turns that can be part of more than one ongoing thread and must be disambiguated by the reader. CoverItLive, thanks to the moderation feature, had a much lower rate of thread overlap. 6.0% of messages could belong to more than one thread in CoverItLive, compared with 39.1% and 32.8% for the other two systems in our study. We believe that CoverItLive’s lower rate of thread overlap is the reason the instructors preferred it.
We were surprised at the similar number of isolated turns for Moodle Chat and CoverItLive (Table 1), as it did not match our subjective experience. We therefore conducted a postmortem analysis. The deleted arrival and departure messages turned out to be a large fraction of Moodle Chat logs. After removing the expected initial arrival and anticipated final departure messages, 18.4% of all turns in Moodle Chat logs were arrivals and departures. This was caused by the frequent broken connections and confused more than just the users whose connection dropped, because it added visual noise to the chat. The frequent disconnections also triggered discussions about the status of the chat system that detracted from the intended topics.
Table 1.
Description of the three chat systems. All numbers are per-session averages. All fields exclude initial and final social turns.
| System | Turns | Threads | Isolated turns |
|---|---|---|---|
| Moodle | 181 | 14 | 3 |
| Second Life | 289 | 62 | 29 |
| CoverItLive | 90 | 13 | 3 |
Moderated chat systems respect the interaction patterns we observed in unmoderated systems but allowed us to lower sequential incoherence. Floor control by the moderator maintains focus on the targeted topic. Therefore moderated chat systems impose a smaller cognitive load on new CMC users like our introductory students.
Interactions in F1 chats follow a simple pattern closely: instructors and students ask questions of each other and respond to them. CoverItLive facilitates this interaction pattern while decreasing the amount of noise in the conversation. However, the students, who preferred Second Life, stated repeatedly during chats that CoverItLive was less conducive to spontaneous discussion. Second Life was unquestionably chaotic and occasionally inappropriate, yet in the authors’ opinion its immersive environment and embodied avatars were the most conducive to foster the sense of belonging to a group so necessary in online classes. Even the interactions with scantily clad visitors were shared experiences fondly remembered by students after the course was over.
Two of the main limitations of this study are its retrospective design and its lack of outcome measures. We plan to address both limitations by designing experimental studies as we pursue this line of research in the future. SHIS is also in the process of building its own island in Second Life. Controlling our own environment will allow us to reevaluate Second Life as a CMC without the liability problems.
Conclusion
We evaluated three free multi-party synchronous CMCs for an online introductory health informatics class. We discovered that a moderated system (CoverItLive) had lower sequential incoherence than two unmoderated systems and therefore placed a lower extrinsic cognitive load on students.