Thoughts on Surveys (10/3)

Our readings this week included a few overview articles on methodology and several example surveys. Each of the myth's presented offered some different perspectives (and underlying motivations) and had different outcome goals.

I thought that the methods outlined in the SIG-CHI article were interesting but a bit vague. The best practices included were somewhat obvious (plan well, adjust structure of survey to situation, keep the context/situation of the user in the center of the survey, be polite) but I was interested to see the list of study methods (heavy emphasis on surveys). They did also include ethnographic style Studies as well as focus groups.

In contrast, the Bauersfeld article included three contextual-centric methods which seemed to seek a balance between observation tools and user imposition. I would expect that the use of cameras in such obvious ways would seem a bit imposing for the participants. They also point to a major bias in their methods towards the end of the article, stating that these methods all focus on software/system use and process/feature models. They might not be particuarly useful for open ended thinking or satisfaction style surveys.

The three survey examples (Library, Postal, Pret) provided some interesting perspectives. Pret's approach reminded me of the comments made in Popwicz's article regarding customer comment cards (more useful for getting initial contact than gathering data) and was clearly oriented towards being a giveaway than a survey. The postal service survey was much more serious and focused. I noticed repeated themes (quality of service, speed of service, courtesy of staff) repeated throughout different interaction environments. The structure of the survey remained fairly consistent and they did not confuse user demographics with opinions.

The Philadelphia survey drew some interesting conclusions (for 1990) but upon review of the survey it seemed that in contrast to the Postal service used 'loaded' questions that both combined several actions/perspectives into a single question and included biases about the contexts in which those actions would be taken. For example, one of the questions is 'I am interested in learning about a general subject area on my own. . .I came to the library today to find answers' In addition, they have the patron fill out a scale (not a reason to important reason) for a question that seems to be a yes/no. . .Another issue I had with the Philadelphia survey was the inlcusion of opinion questions in the demographic section, this would seem to confuse the type of data that they are collecting at different points in the survey.

In contrast to these brief user interactions, Solomon's article on an ethnographic study that spanned 3 years utilizes a number of tools (none of them surveys) to gather data about/from individuals. Solomon describes an immersive experience which included iterative review & consideration of the study both by the observer & observed. I was curious to discover to what extent the study participants agreed with/corrected perceptions of the observer and what impact the observation process had on the actions of the participants. To what extent does the researcher have to hide their true motivations inorder to not 'corrupt' the sampling process? The article about Nickelodeon gets to this point a bit in describing different research context where opinions are solicited without explanation of use. Is this an appropriate step to take?
I found myself comparing these Studies to the Libqual + study run by ARL (http://www.libqual.org/). While it is a survey, it is focused on standardization across libraries, and attempts to measure 'impact factors.' I have found the survey to be intolerable. It takes entirely too long and imho results in a low return rate due to user frustration. It does this in an attempt to get high quality data. Can we, as researchers expect much tolerance from research subjects? How does a reward system play into the gathering of data? I'm not sure, I just think that having experienced the LQP survey that it has gone too far in the opposite direction of some of the others reviewed this week.