Discussions on Higher Scores but Lower Skills
(September 9 and 10, 2009)
Huiming Wang
I received an inquiry to locate recorded empirical studies regarding ‘higher test scores but lower ability’. It would be even better if such reports/investigation was conducted in China. The requestor intends to develop a theoretical model (in economics).
Willard Hom
Hi. Good query. For clarity, are you referring to college entrance test scores, NAEP test scores, TIMMS scores, within-course exams, or others? Is the focus upon change in test-taking skill, change in test scoring/norming/equating, change in test content, or other factors (possibly even the effect of teaching to the test, coaching, and collusion/cheating, among other things)? If your review is totally wide open with respect to the above dimensions, then that will also clarify things.
Meihua Zhai
Actually, I have been trying to figure out how this is related with economics model. Low ability in what? In the testing area or in general?
Liqun Liu (Requestor)
I am interested in empirical studies or reports on the effects of college entrance exams (for example, Gao Kao in China and the SAT in the U.S.) on skill formation for high school students. In particular, I hope to find evidence that the cohorts with higher scores on the college entrance exam tend to have lower skills (in terms of high school graduates' productivity or college freshmen's learning ability, as judged by employers and college professors).
The theoretical argument for this "high scores but low skills" phenomenon goes like this. Because a college degree earns a college premium, high school students would spend too much time/effort preparing for the college entrance exam, but not enough time/effort for real skill building.
I suspect the "high scores but low skills" phenomenon, if it exists, is more prominent in East Asian countries where college admission is mainly determined by the scores on a college entrance exam.
I have searched the College Board archive and the economics literature without getting useful information.
Please let me know if I can clarify my query further.
Michael Tamada
Some possibilities which come to mind include: policy research, e.g. maybe a policy has raised test scores in a school, but resulted in students with lower ability -- this could include the "teaching to the test" possibility that Willard raised, or it could be an admissions issue: if a school seeks to maximize its first-year students' SAT scores, it can incorporate admissions policies which do so but at the likely cost of getting students who are lower in overall ability. Other economic research topics include the cheating teachers famously described in Levitt's "Freakonomics". Or test-takers' behavior: students can raise their scores by taking the test again and again and also taking test preparation courses; with the result being higher test scores but the students likely being no better off in terms of ability.
All of these are examples which economists might, or have already, studied. But as Willard says, there are so many different possibilities that it is hard to be specific. I suspect that there is much more of this type of research at the primary and secondary levels, in the US at least. It is not clear if the researcher is focusing on higher education or not.
P.S. Broadly speaking, a lot of these economic models are examples of "signaling models", where people have imperfect information so they have to rely on imperfect information -- i.e. signals -- to make decisions. E.g. the Admissions Office doesn't know how good an applicant really is, so test scores are a signal that the student produces and that the Admission Office uses. Signals however can produce problems: the signaler has incentive to "put the sizzle before the steak", and work to make his or her signals look superficially good, while neglecting substantive quality. (Better signals reduce this problem but all signals are imperfect.) E.g. job applicants will probably all have snazzy-looking resumes or CVs, sharply-dressed business suits, carefully-combed hair, etc. etc. because those are basic universal signals -- and thus of little use in actually selecting the best job applicant. (I'm talking about the formatting of the CV here, not the content.) Also signals can work against the good of society: some people might use race or nationality as a signal and automatically assume that a job applicant is inferior based on that.
Willard Hom
Thanks for the clarification. I sort of thought that labor researchers would have a concern about a "high scores/low ability" paradox because they frequently use measures of ability in wage modeling. If test scores no longer accurately proxy for individual ability, then wage models (i.e., wage equations) would probably need to employ a different measure for individual ability.
Aha. I just saw Michael T's message (so I don't need to say more). :)
Meihua Zhai
Thanks a lot, Michael, for the explanation.
What Ming was asking was studies done in China.
Chinese children practically are raised among tests and competitions. Taking TOFEL for example, in the mid-late 80's, it was very common to have Chinese students coming to study in the U.S. with near perfect TOFEL test scores, but could hardly speak fluent English nor write coherent English essays. However, we cannot say those students have low "ability" because one doesn't need good speaking skills to score high in TOEFEL. In this case, we can only say that the test needs improvement (and later many universities did add tests for spoken English).
In one of my graduate courses, I had a discussion with my then professor on the issue of learning to the tests. Of course, I was against it. The course instructor asked me one question and I couldn't answer. He said: “What’s wrong with the student being able committing the contents of the textbook into memory in order to have good scores for the tests?"
I still have difficulty to associate the disparity between test scores and actual "ability" (which also needs some type of measurement to be rated "high" or "low"), and the distribution of limited resources which is what economics is about. ;(
Meihua Zhai
Thank you, Michael and Willard.
I think I got it now: What people are looking for here is the loss/profit of using the results from a test that doesn't measure what it was supposed to. Correct? If that's the case, we will need Department of Commerce to work together with ED, using student unit level records to get empirical evidence. ;)
Fairfax VA is the proud home of Thomas Jefferson High School, No.1 HS in the US. Several years ago in one of my arguments with one of my son's HS counselors (a long story of cultural and generation conflicts in his HS years between me, him, his schools and etc.), the counselor asked: “Do you want your child to score high on tests and don't know how to walk across the road?!" Personally I think that's another type of bias against high score achievers.
Liqun Liu
Thanks a lot, Willard! I will look at wage models by labor researchers to see if they have references I am looking for. In any case, citing those wage equation studies would help motivate our own paper.
You are very resourceful! Those comments are excellent! Thanks!
Of course I want to share our paper with your very knowledgeable colleagues.
