START

Situational Judgment Testing

SJT rank order preference items are currently scored uniformly across the set. i.e. you get as many points for being concordant with the expert panel in your preference order for the first two items as you are for the last two items. See here for more info on the normal SJT scoring method.

There are potentially some problems with this: 

1. One has to ask why there is no weighting given to the degree of concordance amongst the expert panel, as there is for SCT items. 
2. There is some suggestion that many candidates, and experts, pay much more attention to their first two preferences than to their last two. 
3. In some situations, where the range of responses is negatively skewed (none of the choices offered are very good, but some are worse than others), there would be particular interest in what the candidates' least favorite choice is.
4. There is some indication that outliers in such last choices are more useful in a 'selecting out' context. 
5. With the current method of indicating rank preference, there is no way to indicate how strong that preference is, whereas in real life there are many situations where some choices are near-equivalents, whereas others are distant also-rans, the OMDB choice (Over My Dead Body). 

 We have some suggestions on alternate scoring approaches that may be able to address some of these challenges.