Calculating Confidence Interval

Dongyang Li: Here is a quick question: If a survey is conducted among 500 students and 250 respond. Say for the question: "Are you satisfied with your freshman year?" 20% says "very satisfied". What is the confidence interval for this result? Or do we need other information to infer the confidence interval?

---

Beiling Xiao: Dongyang:

A 95% confidence interval of 20% of 250 said satisfied with freshman year is:

		 0.20 - 1.96*square root of {[0.20 * (1-0.20)]/250}, 
		 0.20 + 1.96*square root of {[0.20*(1-0.20)]/250} 
		 
		that is: 
		0.20 - 1.96*0.0253, 
		0.20 + 1.96*0.0253 (0.20 - 0.0496, 0.20 + 0.0496) 
		
		or (15.0% , 25.0%) 

You do not need other info to infer the confidence onterval. Hope this helps. Beiling Xiao

---

Willard: Dongyang, et al.,

I agree in essence with the calculation of the confidence intervals that you have just seen.

However, the 50% response rate still imposes a serious threat from nonresponse bias. The confidence interval for the target population (which includes those who did not respond)could be much wider than the confidence interval BX and XF suggested.

[William G.] Cochran, Sampling Techniques,3rd Edition (1977) recommends calculating two additional confidence intervals as a means of exploring the sensitivity (or risk) that the analyst incurs when nonresponse is high. In a "best-case scenario"---although we realize this is the unlikely state of nature in reality---the analyst would calculate the confidence interval under the assumption that all the nonrespondents would have marked "highly satisfied" if they had really done so. In a "worst-case scenario"---somewhat unlikely in the real world but more likely than the best-case scenario---the analyst would calculate the confidence interval under the asumption that none of the nonrespondents would have marked "highly satisfied" if they had really done so.

The bottom line is that severe nonresponse creates major problems in the interpretation of "standard" confidence intervals. We often hope that consumers of statistical analysis will consider the response rate when a report displays the confidence interval, but they don't (as a rule) pay that much attention. Cochran's sensitivity analysis drives home the risk factor inherent in the potential for nonresponse bias in such a survey as yours---even though most analysts feel that mention of the response rate is sufficient "disclosure." Some federal reports include Cochran's method (albeit in their appendices), and I have included them in my past analyses of survey data. Unfortunately, most analysts have not adopted this particular technique, and it does not get reported often (for obvious reasons). But if the costs for erroneous estimation are high, calculation for "internal use" is very informative to the analyst(although not necessarily helpful to administrators).

A special implication is noteworthy. If Donyang were finding the confidence interval for the "very dissatisfied" category, then I would strongly urge calculating the "worst-case scenario"---where we assume that all nonrespondents are assumed to feel that very dissatisfied would have been their response. I tend to believe that many nonrespondents avoid answers that may jeopardize them or conflict with the social-desirability response set....It is less likely for nonrespondents to withhold a response because they feel very positive towards a subject.

A final, perhaps superfluous consideration, concerns the sampling design. Dongyang's survey question does not specify this, but it's relevant. If the data come from a sample design that is a stratified or multi-stage design, then the analyst should apply the appropriate sampling weights in calculating the confidence interval for the population total. The adjustment resulting from the sample design weighting may or may not be material, depending upon the design, the population, and the responses. Since many surveys of institutional populations use classes as a sampling unit (that is, they draw a sample of classes and then survey all students, or a further sample thereof, within those selected classes), the confidence interval should consider that these samples are essentially cluster samples (implying that weighting for a cluster sample design is needed...).

Well, that's enough overkill from me, but it is an intriguing topic.

---

Willard: A while back, the astute John Lin tried out my suggestion below and noted that something did not seem right. I got occupied with my many duties here, but I finally found my error after I dug up my Cochran book.

On page 362, Cochran says the following: "In calculating the lower limit, assume that all sample nonrespondents would have given a negative response. In calculating the upper limit, assume that all sample respondents would have given a positive response...The limits are a little wider..." Cochran's didactic example quoted above pertained to just a 20% nonresponse rate. As the nonresponse rate increases, so does the width of the interval suggested in the quote. The error in my posting (see below in my Jan.9 e-mail)was my prescription to calculate two separate confidence intervals. Cochran's method really produces one confidence interval. It just uses the worst-case scenario to find the improvised lower limit and the best-case scenario to find the improvised upper limit---making just one confidence interval.

I apologize to the listserve readers for missing this. I wrote the Jan.9 message without referring directly to the Cochran text. It has been about four years since my use of the Cochran method in an analysis. (Yes, I must be exhibiting some memory decay here...) We should thank (as I do now)John for his keen observation here...(FYI, Shuqin Guo also questioned my Jan.9 message, so she deserves credit here too.)