The P-3 Game
In a series of papers, John Roberts and his coworkers used pile sorts and rating tasks to study how people perceive various kinds of behaviors in games (see, for example,
PILE SORTS WITH OBJECTS
Pile sorts don't have to be done with cards. James Boster (1987) studied the structure of the domain of birds among the Aguaruna Ji'varo of Peru. He paid people to bring him specimens of birds and he had the birds stuffed. He built a huge table out in the open, laid the birds on the table, and asked the Aguaruna to sort the birds into groups.
Carl Kendall led a team project in El Progreso, Honduras, to study beliefs about dengue fever (Kendall et al. 1990). Part of their study involved a pile sort of the nine most common flying insects in the region. They mounted specimens of the insects in little boxes and asked people to group the insects in terms of ''those that are similar.'' Some fieldworkers have used photographs of objects as stimuli for a pile sort.
Borgatti (1999:133), however, points out that physical stimuli, like images or objects, make people focus on form rather than function. In fact, when asked to sort drawings offish, fishermen in North Carolina sorted on shape—the long thin ones, the ones with a big dorsal fin, the small roundish ones (Boster and Johnson 1989). ''In contrast,'' says Borgatti (1999:133), ''sorting names offish allows hidden attributes to affect the sorting''—things like taste or how much of a struggle fish put up. ''If you are after shared cultural beliefs,'' says Borgatti, ''I recommend keeping the stimulus as abstract as possible'' (1992b:6).  
Inexperienced pilots rated ‘‘inducing an autofeather” as more serious than did highly experienced pilots. Inducing an autofeather is more embarrassing than it is dangerous and it’s the sort of error that experienced pilots just don’t make. On the other hand, as the number of air hours increased, so did pilots’ view of the seriousness of ‘‘failure to use all available navigational aids to determine position.’’ Roberts et al. suggested that inexperienced pilots might not have had enough training to assess the seriousness of this error correctly (Further Reading: pile sorts).
-  Roberts and Chick 1979; Roberts and Nattrass 1980). One ‘‘game,’’ studied by Roberts etal. (1980), is pretty serious: searching for foreign submarines in a P-3 airplane. The P-3is a four-engine, turboprop, low-wing aircraft that can stay in the air for a long time andcover large patches of ocean. It is also used for search-and-rescue missions. Making errorsin flying the P-3 can result in career damage and embarrassment, at least, and injury ordeath, at worst. Through extensive, unstructured interviews with Navy P-3 pilots, Roberts et al. isolated60 named flying errors. (This is the equivalent of extracting a free list from your interviews.) Here are a few of the errors: flying into a known thunderstorm area; taking offwith the trim tabs set improperly; allowing the prop wash to cause damage to otheraircraft; inducing an autofeather by rapid movement of power level controls. Roberts etal. asked 52 pilots to do a free pile sort of the 60 errors and to rate each error on a 7-point scale of ‘‘seriousness.’’ They also asked the pilots to rank a subset of 13 errors on four criteria: (1) how mucheach error would ‘‘rattle’’ a pilot; (2) how badly each error would damage a pilot’s career;
-  how embarrassing each error would be to commit; and (4) how much ‘‘fun’’ it wouldbe to commit each error. Flying into a thunderstorm on purpose, for example, could bevery damaging to a pilot’s career, and extremely embarrassing if he had to abort themission and turn back in the middle (when Roberts et al. did their research in the 1970s,all P-3 pilots were men). But if the mission was successful, then taking the risk of committing a very dangerous error would be a lot of fun for pilots who are, as Roberts calledthem, ‘‘high self-testers’’ (personal communication).