Media Program Evaluation and Computer Analysis
Current technology provides the most sophisticated evaluation and research of media productions through the application of microcomputer technology. Electronic analyses can be made of the effects of any media presentation, documenting an audience’s reaction second by second. This technology can document the effects of minute changes in presentation on audience attitudes, skill, and knowledge. Program staff has the opportunity to manipulate covertly what the audience sees and to analyze responses instantaneously. It allows a program to collect instant feedback data on an infinite number of visual, graphic, and verbal configurations. It is a powerful technique to conduct formative media evaluations during the pre-production stage. Before investing in production costs that could run in the millions, agencies need to perform careful preliminary assessments.
Readability Testing
Readability is another important aspect of pre-testing written materials. Readability tests are available and easy to apply. Readability tests determine the reading grade level required of the average person to understand the written materials. Readability estimates provide evidence of only the structural difficulties of a written document: vocabulary and sentence structure. They indicate how well the information will be understood but do not guarantee comprehension. Many readability formulas exist (Dale and Chall, 1948; Flesch, 1948: Fry, 1968; Klare, 1974-1975), but the Simple Measure of Gobbledygook (SMOG) grading formula for testing the readability of educational material is one of the most commonly applied. Generally considered an excellent method of assessing the grade level a person must have reached to understand the text, it requires 100% comprehension of the material read. To calculate the SMOG reading-grade level, program staff should use the entire written work being evaluated using these steps:
- • Count 10 consecutive sentences at the start, in the middle, and the end of the text.
- • From this sample, circle all words with > 3 syllables, including repetitions of the same word.
- • Total the number of words circled.
- • Estimate the square root of the total number of polysyllabic words counted.
- • Find the nearest perfect square and calculate the square root.
- • Add a constant of 3 to the square root to calculate grade (reading- grade level) that a person must have completed to fully understand the text being evaluated.
Sentence and word length and difficulty affect the readability score. The SMOG formula ensures 90% comprehension; that is, a person with a tenth-grade reading level will comprehend 90% of the material rated at that level. This procedure can be applied to all texts prepared by a program for public consumption. “Making Health Communication Programs Work” (USDHHS, 2005) presents useful discussions of readability in general and in health-related literature.
Another readability method increasingly being used by health education specialists is the Flesch-Kincaid Readability tests. This method has a long history, dating to 1948, when Flesch published the Reading Ease test. The formula for the test produces a score ranging from 0 to 120. A higher score indicates material easier to read, while a lower number indicates material more difficult to read. For example, reading ease scores of 90-100 indicate the material can be read and understood by the average 11-year-old. Reading ease scores of 0-30 mean the passage is understood by university graduates. In 1975 Kincaid, working under a US Navy contract, developed the Reading Grade level. It translates the Flesch Reading Ease to actual grade levels in US schools. These formulae were combined and are now known as the Flesch-Kincaid Readability tests. They are widely applied by writers of self-help booklets, brochures, and pamphlets. The Flesch-Kincaid formulae can be accessed on a number of computer software programs and no longer require hand calculation of reading levels.
