About this case

We have created a short tutorial, with play-by-play screenshots of what learners and the Scenario Director will see when playing a Turk Talk case. 

http://demo.openlabyrinth.ca/files/606/Turk Talk screenshots for tutorial.docx 

Turk Talk is now featured briefly in our 'OpenLabyrinth Highlights' video. Here is a short snippet:

There are often concerns about the effects of cueing: most students are better able to pick from a list of predefined choices because it is so hard to avoid inserting a cue about the correct response when designing your question and context. See this article from 1966:

An_assessment_of_the_influence_of_cueing_items_in.10.pdf - 'An assessment of the influence of cueing items in objective examinations.' McCarthy, W. Journal of Medical Education, 41(3), 1966. http://journals.lww.com/academicmedicine/Abstract/1966/03000/An_assessment_of_the_influence_of_cueing_items_in.10.aspx

How strong is this effect? In the 1966 paper, it was thought be really quite significant but other studies since then have questioned this. It does depend on the topic, relevance of the cue and what you are actually testing: fact recall or clinical reasoning. 

To try and avoid the effect of cueing the learner with predefined responses, we are sometimes asked about whether OpenLabyrinth can handle natural language responses. That is, a typed-in answer as free text, with no cues as to what might be acceptable. 

In our research on this, we have found that careful design of responses, good distractors, and having a good range of alternatives to choose from (some would suggest 12-20 are needed to significantly allay this effect, but this is sometimes hard to do) all help to reduce this effect. But there are times when free-text input is useful. (This case attempts to illustrate such a situation). 

There are several ways in which one can handle free-text input. Indeed, there is a whole genre of online gaming called Interactive Fiction which revolves entirely around this concept. There are now some excellent tools out there which help authors in programming such learning designs. Inform 7 is well known in the Interactive Fiction world. ChatScript is a free open-source chat-bot which can be modified to such purposes. 

But beware of the effort involved in such an approach. Writing a script that can accommodate the majority of reasonable responses at such a juncture can be very, very time-consuming. As an example, we cite the very interesting Maryland Virtual Patient project. Although we have not seen any official figures, informal enquiries suggest that a single module in this very impressive virtual patient required six years and $11M to develop - which would not be so bad, if the language ontologies and maps were more extensible to other topic areas and cases. 

OpenLabyrinth has taken a much simpler approach. Firstly, we provide very simple text matching and logic rules in IF-THEN-ELSE style. This approach is demontrated in this case. But because the variety of reasonable responses can easily be quite large, it can be quite frustrating to learners to play "guess what I'm thinking" and find words that match the list constructed by the case authors. 

So we came up with a second approach, based on the principle of the Mechanical Turk. In this approach, we use a human-computer hybrid, where the actual parsing or interpretation of the entered text is provided by a human. Because a single human can typically handle multiple learners, and does not need to be collocated with the learners, it is a bit less resource intensive than other approaches such as the OSCE which require a 1:1 ratio between teacher and learner. 

Now, you still need to be quite careful about your learning design in such cases or else you can easily swamp your human Turker. But our research suggests that in a typical case, you only need to use natural language processing on 2-3 nodes out of the many in your case. Space these out carefully to provide a more steady flow of inputs to your Turker.

For those who want to get more of a sense of how the Turk Talk interface works, from the facilitator and learner viewpoints, we have this short video:

Lastly, it is also worthwhile creating a script for your Turkers, providing details on the objectives for the case, the concept map of the case, and the acceptable responses at the Turk Talk points in the case. Please refer to this example script here: http://demo.openlabyrinth.ca/files/606/Turker Script for Scenario 92.docx 

If you are interested in collaborating on research in this topic area, please contact us.