Sampling design and data collection methods

Sampling Design & Data Method

The intentions of the research article Walk the Talk? What Employers Say vs. What They Do, is to develop a method of testing self reported intentions of employers to hire applicants based upon race and/or criminal history. Sociologists have been fundamentally aware of a significant difference between reported behavioral intentions and actual actions and yet according to Pager and Quillian continue to rely heavily on surveys as a valid source of information, "with little effort to validate this assumption." (355) This near universal acceptance of survey/self-report results with regard to difficult to research social phenomena is what the authors consider a serious discrepancy in the manner in which understandings are developed about serious social phenomena.

The opening remarks of the paper are a brief synopsis of a research study conducted in 1930 by a sociologist who first traveled across the country (twice) with a Chinese couple to test establishment reactions to such a situation. The trio (a married Chinese couple, and the researcher) were refused service by only one establishment. The researcher then developed a brief survey and made contact with each establishment, asking them if they would serve people of Chinese heritage at their establishment, 90% reported that they would not, even though all but one had served Chinese people when faced with the option in real life. (355) the intentions of the research are much the same as Walk the Talk, to test the validity of survey results against real behavior on a social charged issue.

Pager and Quillian also effectively point out that race and criminal history are particularly troubling issues to look at, as peoples intentions or outward statements about both issues rarely match their actions when confronted with the situation. The authors point out that because of the relative difficulty of gaining truthful data in a natural setting regarding discriminatory feelings and/or potential actions many sociologists continue to rely almost exclusively on survey and the authors provide several examples of survey types and standards used and reported as accurate, despite known weaknesses in the method. (357)

Pager and Quillian first point out this potential discrepancy, between reported intentions (survey answers) and actual behaviors, by research support that indicates pervasive social and economic status disparities among those who have either or both a criminal record or are black. White or black men with a criminal record are far more likely to be unemployed, or working in jobs with limited reward and limited mobility, while white men with criminal records tend to fair slightly better than black men with criminal records both are obviously discriminated against in the labor market and in many other ways. Pager and Quillian use raw data regarding employment and wage earning and the disturbingly high and discriminatory incarceration rates among black men. (355-362) Finally the work points out the limited number of direct studies that test the correlation between survey and action. (360)

The work contends that because of the taboo regarding statements that are openly discriminatory information gathered via self-report survey; having to do with such information is likely to be flawed and inaccurate. To develop this idea further the authors create a research sample that attempts to test real behavior against reported behavior. The work has a duel sample method, with a sample method that tested real life job application experiences against survey reports of the same set of employers.

In the first sample (test) a matched pair of testers (actors) was sent to apply for 350 entry-level jobs in the Milwaukee area. The scenario of each tester was one of four alternatives, the tester was either white, and had fictional criminal record (150 jobs) or black (200 jobs) and again had a fictional criminal record or not. Applications were scripted as was resume material and every attempt was made to keep interchanges professional and similar in character at each situation. Data collection was developed in this half of the test by separate voicemail boxes for applicants. The employee preference was measured by the number of callbacks found in each of the four voicemail boxes as well as additional voicemail boxes set up for references which had been listed on the fictitious employee resume. According to the study, jobs were randomly selected from the classified ads in the Milwaukee Journal Sentinel and applicants were randomly assigned to different jobs. (362) One additional note on this half of the duel research study was that the pair of applicants with and without fictitious criminal records was rotated throughout the experiment to reduce the odds that a single applicant would alter results if assigned the rigid role of ex-con or clean record applicant.

In the second half of the research study the same set of potential employers was surveyed using a vignette method. The vignette described the scenario of applicants who matched the (tester) applicants. The employers who were screened by asking for the person in charge of hiring at the place of business were then asked to respond to the scenario by answering questions regarding if they would or would not hire or consider hiring the applicant in the vignette. Data was collected utilizing the responses to the survey questions, which avoided direct racial comparisons but simply stated the race of the vignette applicant in each scenario. The survey was also conducted in a split ballot method with none of the businesses being read the same vignette but all answering the same questions regarding the vignette, corresponding directly to the type of applicant that had applied for a job in the previous study. (362)

Data collection from both samples methods were then compiled individually and compared to one another to see if there was a significant statistical disparity between what employers reported they would do and what they actually did, when faced with a real applicant (tester) who was either black or white and had a criminal record or did not. Not surprisingly the comparison between the two tests supported the theory that more often than not employers acted in a way that was inconsistent with their reported intentions/hiring policies.

364) the work then goes on to analyze the results using different statistical methods, some resulting in greater disparity regarding race while still stressing the disparity in report vs. action. The results are in fact too conclusive to be completely dismisses based on any outside influences, though these should be discussed as point of critique.