IJE Advance Access originally published online on January 27, 2009
International Journal of Epidemiology 2009 38(2):361-368; doi:10.1093/ije/dyn356
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Commentary: Individual, ecological and multilevel fallacies
Division of Epidemiology, University of Minnesota, USA. E-mail: oakes007@umn.edu
Accepted 6 November 2008
The new paper by Drs Subramanian, Jones, Kaddour and Krieger (hereinafter Authors) contains many important and subtle insights about the fallacies of single-level research, be it at the individual or ecological level.1 The Authors urge epidemiologists to consider contexts and multilevel phenomena when investigating and explaining population health. They also criticize the late William S. Robinson and his classic 1950 paper, and methodological individualism (MI) as a research paradigm.2 Support comes from historical anecdotes, theory and a re-analysis of Robinson's data.
Assuming I understood it properly, I am in full agreement with the primary aim of the new paper. Epidemiologists, especially those interested in the effect of social forces on health, should consider contexts and multilevel phenomena. And as a general proposition, I also agree that critical examination of a scientist's culture, history and personal motivation can be enlightening. The Authors’ scholarship on these matters merits careful study.
On the other hand, I find the Authors’ critique of Robinson and his paper unhelpful and their critique of MI misguided. More importantly, the Authors conflate multilevel thinking with the so-called multilevel regression model and in so doing offer readers questionable advice. My goal here is to explain these disagreements in hopes of stimulating more critical thinking about multilevel phenomena and research for the improvement of population health.
 |
Robinson's paper in context |
|---|
 Top Robinson's paper in context Robinson's mistakeMI and multilevel thinkingMultilevel modelsConclusionReferences |
|---|
According to Robinson, he was motivated to write his paper because of the ‘impressive number of quantitative ecological studies’ that relied on ecological correlations to make inference about individual behaviour.2 He cites research on tuberculosis, voting, crime and fertility, and states that in each case the authors were not interested in ecological correlations but rather discovering something about the behaviour of individuals. Robinson thus aimed to determine whether ecological correlations could be validly substituted for individual correlations. He showed through some simple algebra and a worked example that they cannot. His example demonstrated that a correlation between illiteracy and race differed quantitatively by level of aggregation and that a correlation between nativity and illiteracy differed qualitatively—reversed signs—by level of aggregation. Robinson expressed concern that his finding would have serious consequences. But he wanted to prevent future mistakes and set researchers on a more fruitful path.
I believe Robinson's paper is so widely cited (by those who have read it, anyway) because of its elegant simplicity in answering one important question. It remains a delight to read, especially in light of the work it spawned on Simpson's Paradox, aggregation bias, the multiple-area unit problem, confounder control and so forth. I cannot help but associate Robinson's classic paper and the Authors’ work with two new papers by Gelman and colleagues and, separately, Hernán and colleagues.3,4
Gelman and colleagues illuminate the fallacy that citizens of wealthier American states tend to vote for Democratic candidates while those residing in less wealthy states vote for Republican candidates. This paper uncovers a striking example of Simpson's Paradox and presents a remarkably clear and accurate explanation of it. Recall that Simpson's Paradox is a situation where the relationship between two variables (e.g. income and political party) is reversed when a third variable (e.g. state) is considered. 5,6 Figure 1 is a caricature of the paradox addressed by Gelman and colleagues. The idea is that the within-state probability of an individual voting for a Republican party candidate increases with individual income, while the between-state mean probability of voting Republican declines with increasing mean state income. Gelman and colleagues fit a sophisticated multilevel model that nicely describes their paradoxical data.
|
Hernán and colleagues address the divergent effect estimates of hormone replacement therapy (HRT) on cardiovascular disease (CVD) in menopausal females between a large cohort study and a randomized clinical trial (RCT). Recall that inferences from several epidemiologic cohort studies suggested HRT was protective for CVD while a more recent RCT showed HRT deadly with respect to it. Hernán and colleagues showed the presumably mistaken inference from the observational data was not due to the observational design (e.g. confounding) or the data themselves, but to the improper conceptualization of desired effects and the inappropriate use of regression adjustment for time-dependent confounding in the observational study. By analysing cohort data in a way that mimics an experimental trial, Hernán and colleagues estimated effects nearly identical to those in the RCT.
Together, these two papers highlight the contemporary relevance of Robinson's problem and the vital importance of clear, causally informed thinking in epidemiology.
|
Robinson's mistake |
|---|
|
TopRobinson's paper in contextRobinson's mistakeMI and multilevel thinkingMultilevel modelsConclusionReferences |
|---|
The Authors take umbrage with Robinson's statements about the meaning of individual-level correlations. Yet the only time Robinson uses the word ‘meaningful’ is in the last sentence of his conclusion, where he wrote:
The purpose of this paper will have been accomplished, however, if it prevents the future computation of meaningless correlations and stimulates the study of similar problems with the use of meaningful correlations between the properties of individuals. (p. 357)
Unlike the Authors, I do not read this as Robinson saying researchers should only consider individual correlations or that only individual correlations are universally meaningful. In light of his stated aims, I interpret his concluding sentence as an admonition to readers of the American Sociological Review who mistakenly think ecological correlations may be substituted for individual ones and those who may believe that one correlation is as good as any other. That is, Robinson was ‘speaking to’ sociologists who misunderstood the meaning of ecological correlation and confounding.
It is difficult for me to accept the claim that a sociologist could be uninterested in how social context, including despicable laws, affects individual behaviour. Since its founding by Comte and Quetelet, sociology's raison d’être has been the impact of social context and social forces on behaviour. To this end, one must appreciate that Robinson was one of the first sociological methodologists and served the council of the American Sociological Association's methodology group from 1961 to 1965. He was interested in causal inference and contributed to debates about measurement error, induction, falsificationism, modelling and even counterfactual thinking. In none of his other papers (I tried to read everything he wrote) does he argue against consideration of contexts or anything even related.2,7–14 Consequently, I read Robinson's concluding sentence much more narrowly than do the Authors.
What about the impact of Robinson's paper on social and health science research? For those willing to heed Robinson's warning about improper substitution, there can be no question that his paper would have eliminated any enthusiasm for doing so. And it is true that the paper has been widely cited. But it is not clear how responsible Robinson's paper is for individual-level research in epidemiology and/or public health. While not diminishing the paper's import, it seems there is more myth than fact here. The trouble is that data on a ‘squelching effect’ are difficult to come bye: papers not published cannot easily be counted.
In his paper on the impact of sociological methodology on statistics, Clogg does not mention Robinson's work or the ecological fallacy.15 The same goes for Raftery's review of the impact of statistics on sociology, a paper that carefully addresses multilevel modelling.16 Further, my analysis of citations in articles published in the American Journal of Epidemiology during the period 1981–2002 showed Robinson's paper was cited just six times in actual research papers, a tiny fraction of even the social science citations of the period.17 Parallel but unpublished data on the citations in American Journal of Public Health (AJPH) articles shows Robinson's paper was cited just 10 times over the same time period. By comparison, Susser's 1994 paper on the fallacy of ecological fallacy was cited 63 times in AJPH alone.18 Finally, in an online supplement to his 2004 article about citations to papers published in the American Sociological Review, Jacobs shows that (ISI Thompson database) citations to Robinson's paper were relatively meager for a decade or so.19 I abstract the central result in Table 1 below. For comparison, also tabulated are citations to a paper by Leo Srole which addressed social integration from a Durkheimian (i.e. contextual) perspective.20 While social isolation remains central to social epidemiology, I doubt many today know Srole's work. If Srole's more popular paper is not responsible for contextual research how can Robinson's paper be responsible for individual-level research?
|
The Authors suggest a link between Robinson's paper and the 1954 legal case of Brown v. Board of Education (wherein the US Supreme Court ruled that the racial segregation of public schools was unconstitutional). While I agree that in hindsight Robinson should have done more to clarify the narrowness of his paper, I worry that readers will misunderstand things. Accordingly, I have tried to examine the major works on the case, paying particular attention to the social scientific research used to support or refute the plaintiff's claims. Among other literature, I reviewed the actual Court decision, the Appellants’ Brief signed by thirty-two social/behavioural scientists (including Prof. Robert K. Merton of Columbia University), subsequent law journal debates about the validity and propriety of social science in the related cases, and what is widely considered the most definitive history of the case.21–29 Neither Robinson, his paper, or the ‘ecological fallacy’ is mentioned or cited—not even once—in the reviewed literature. In fact, for reasons beyond the scope here, the only social science research that mattered in the case was a few (rather crude) psychological experiments with toy dolls and children.
Among the alternative hypothesis about the hidden motivation, if any, behind Robinson's paper, one merits special consideration. This one revolves around another major decision by the US Supreme Court: Korematsu v. United States of 1944.24,27,28 The issue here was the Constitutionality of the forced internment (in barbed-wire camps) of Japanese Americans at the start of World War II. Despite there being no evidence of individual treason or sabotage, and in the face of many individual Japanese-Americans trying to join the military to fight for the USA, the Court accepted claims about the ‘threatening and sneaky nature’ of the Japanese as a group. Based on this ‘sociological evidence’, in 1944 the Court upheld the internments. Robinson does not mention this case either but it seems entirely possible that the problem of relying on ecological correlations to detain innocent individuals was on Robinson's mind. That is, he and every other American sociologist must have been worried about xenophobic stereotyping. While I have found no hard evidence to support the claim, circumstantial evidence leads me to believe that, if he had one, Robinson's hidden motivation for his paper (also) included concerns about the use of such correlations as ‘evidence’ for political/legal activity.
The Authors also tie Robinson's work to the Cold War and all the negatives that accompanied it. Evidence for this claim is also lacking. First, suffice it to say that only some American sociology circa 1950 espoused the American ideology of freedom, choice and opportunity. But few would associate Robinson's work, or that of the ‘Columbia School’, to any such paradigm. In any case, it is illogical to associate the ideology of freedom with the oppressive fascism the late Senator Joseph McCarthy espoused. Second, it seems the idea for the ‘ecological fallacy’ paper came not from Robinson but his colleague/teacher, the late Paul F. Lazarsfeld of Columbia University.30 It is reported that a few years before Robinson's publication, Lazarsfeld was analysing census data at the tract level and became worried about the validity of his correlations. Robinson ended up working out the math and publishing the result. For purposes here, Lazarsfeld's role as initiator is important because Lazarsfeld was not only a leading sociological methodologist, but a Jewish political socialist who fled Vienna in the 1930s and author of the first comprehensive study on the impact of McCarthy-era intimidation on academic freedom—a text widely viewed as one of the first mulitlevel contextual analyses, ever.31–33
Absent new data about some hidden motivation, it seems to me Robinson's only mistake was in concluding his paper with a vague statement. But I am less concerned with his obtuse comments on the meaning of individual correlations than I am with his comments on the meaning and utility of correlations more generally. Here I am referring to the second half of his concluding sentence. From my perspective, Robinson should have articulated the meaninglessness of correlations, at any level, as compared with causal effects. By 1950 sociologists such as Stuart F. Chapin and Samuel A. Stouffer had already clarified the distinction.34 And statistician Ronald A. Fisher's masterpiece, The Design of Experiments, was available in 1935.35 In linking the ecological fallacy to correlation, Robinson contributed to the Pearsonian myth that correlation is superior to causation.6 Regardless, were he able to defend himself, I imagine Robinson would forcefully reject the Authors’ indictment.
|
MI and multilevel thinking |
|---|
|
TopRobinson's paper in contextRobinson's mistakeMI and multilevel thinkingMultilevel modelsConclusionReferences |
|---|
As part of their effort to promote consideration of multilevel contexts the Authors decry MI. But the Authors’ conception of MI is a ‘straw man’ that few contemporary MI scholars, even those on the political left,36–40 would accept. Consider, among many others, the work of the late sociologist and methodological individualist James S. Coleman.41 A student of Paul F. Lazarsfeld's, Coleman tried to better formalize how social change occurs by drawing a trapezoidal figure which I have affectionately called the ‘Coleman bathtub.’ The simple idea is that society is made up of individuals—indivisible objects in sociology—and that individuals make up society. Notice that Coleman's is a multilevel theory.
It is critical to understand that for methodological individualists like Coleman, societal or group-level change does not just happen mysteriously without the involvement of actual persons. Institutions and other social phenomena play a key role in analyses, but these phenomena must be grounded to the activity of individuals. Changes in smoking rates can therefore only be explained by understanding the actions of individual smokers and non-smokers, and their interrelationships in the context of laws and social norms. Group-level phenomena are never simple aggregations but rather complex dynamic and multilevel phenomena.
Figure 2 is my basic interpretation of Coleman's bathtub. For purposes here, the thicker (near) vertical arrows to the left and right are most important. The downward pointing arrow to the left, from the larger circle to the smaller, represents the impact or influence of society/contexts on individuals. This is the macro-to-micro transition and includes various aspects of socialization and resource constraints. The arrow to the right, from the smaller to the larger circle, represents the impact of individuals on society. This is the micro-to-macro transition and incorporates, among other things, collective action, social choice and social movements. Together, these ‘micro–macro’ transitions represent the most important but most difficult challenge for multilevel thinking in sociology and epidemiology alike.
|
Coleman's approach is important here for yet another reason. It was his landmark 1966 report on the equality of educational opportunity (in America) that gave birth to contemporary multilevel theory and modelling.42 On the heels of the aforementioned 1954 Brown decision, the Civil Rights Act of 1964 required the US Department of Education to conduct a study of the equality of educational opportunities in American schools. The result was one of the largest social science studies in American history: a cross-sectional survey that included some 645 000 school children (grades 3, 6, 9 and 12) nested in 60 000 teachers nested in 4000 public schools. Coleman aimed to estimate the independent effect of school funding and social contexts on student academic achievement. That is, Coleman aimed to conduct a multilevel study before the multilevel model was even recognized. His work initiated the enormous and still thriving academic industry that addresses the effect of schools, contexts and policies on educational attainment. Much could and should be learned from this body of work. But in any case, the fact is a methodological individualist laid the foundation for Bryk and Raudenbush's famous text, Hierarchical Linear Models, and subsequent multilevel modeling research.43
Multilevel theory without MI seems indefensible. To see this, appreciate that the Authors’ preferred ecosocial theory treats contexts and institutions as given. Absent some notion of individual agency and interaction, how can this theory explain the emergence, maintenance or change of social contexts? With respect to the Authors’ own typology of studies (their Figure 5), where does X come from? Is it not some complicated function of y and x? The fact is that MI is not only not consistent with multilevel thinking, it is the foundation of it.44
|
Multilevel models |
|---|
|
TopRobinson's paper in contextRobinson's mistakeMI and multilevel thinkingMultilevel modelsConclusionReferences |
|---|
In an effort to demonstrate the importance of the multilevel perspective, the Authors re-analyse Robinson's data with a sophisticated multilevel statistical model (MLM). They claim that only by incorporation of contexts (e.g. states, state law and state educational resources) can one properly understand the relationship between race and illiteracy. Further, the Authors believe their MLM reveals the pitfalls of Robinson's argument against ecological correlations. I am not so sure. What do their MLM results really mean? Even assuming the models yield unbiased parameter estimates, what scientific and/or political use is a finding that the odds ratio for black illiteracy changes when conditioned on state? And the variation of odds ratios across states means what, exactly?
Setting aside interpretative disagreements, I fully agree that contextual factors are often necessary to understand a given phenomena. But with respect to the Authors’ other point, I do not think the MLM is always necessary. My take on the social epidemiologic literature employing MLMs to observational data is that MLM analyses have increased confusion and distracted researchers. Why MLM advocates continue to ignore the vast literature on causal inference and the limitations of regression-based inference escapes me.
It is critical to draw a distinction between multilevel thinking and MLMs because there is nothing especially multilevel about MLMs. MLMs do not model cross-level processes if processes are properly understood to be the micro–macro transitions described above. And to yield statistically valid results, MLMs must meet a number of strong if not heroic assumptions that few appear interested in examining.45–51 In terms of causal inference, the principal benefit of MLMs lies in their ability to relax the ‘no-clustering within groups’ assumption and to borrow strength from relationships found in data-rich contexts so as to improve those in data-poor contexts. Yet I know of no methodologist who believes MLMs give special purchase to causal inference. Gelman, for example, is abundantly clear that his models should not be interpreted causally.3,52,53 And as early as 1995, the distinguished statistician, David Draper, expressed concern about the misuse and abuse of the model.54 Some of my own work has revealed the tendency for MLMs to yield inferences dependent not on data but model assumptions and extrapolations.47,55,56
An empirical illustration would seem helpful. I examine not tabular data,57 as Robinson and the Authors did, but person-level 1930 Census data. Such data is publically available from the Integrated Public Use Microdata Series (IPUMS) project at the Minnesota Population Center (www.umn.ipums.edu). For the 1930 census, IPUMS offers a 1% random sample of the actual Census Bureau person-level records. Like Robinson and the Authors, I limit my data to persons aged 10 years or older living in the 48 continental states. Unfortunately, only a handful of coarse measures are available, most of which are self-explanatory (e.g. black, native born). But the data do include two proxy measures of socioeconomic status (SES): having a radio and owning a house.
As a first step I replicated Robinson's and most of the Authors’ analyses, especially their multilevel logistic regression models and their state-wise correlations. There was no difficulty here; my results differ only slightly from theirs and such differences are surely due to slightly different data and estimation algorithms. It is worth noting that my results support the Authors’ methods for analysis of tabular (census) data.
My next step is to address a more fruitful multilevel question: what is the effect of Jim Crow laws on African American illiteracy? This question is meaningful since it suggests an (absurdly obvious) decision: repeal Jim Crow or not? The question conforms to the counterfactual framework because it asks: Absent Jim Crow, what would the 1930 illiteracy rate of African Americans have been? The question is multilevel because individual persons are nested within state law conditions.
The ideal data for answering the question include outcomes measured under counterfactual conditions, which are by definition unobservable. The second best data set would contain measures from a group randomized trial, wherein some randomly selected states (i.e. groups) would adopt Jim Crow laws while others did not. Group randomized trials are useful because, so long as sample sizes are large, they yield multilevel observations that may substitute for the desired but unobservable counterfactuals. But since it is obvious that such an experiment cannot be conducted we are left to look for a natural experiment or rely on a purely observational design. Nevertheless, the analytic goal remains the same: use observable data to mimic the ideal (unobservable) conditions.
Table 2 presents the percent persons illiterate by race and state Jim Crow status. It is easy to see that, for all 48 states, blacks in Jim Crow states have over four times the illiteracy rate as blacks in non-Jim Crow states. But before concluding that Jim Crow elevates illiteracy risk in blacks we must consider that, compared with other states, whites in Jim Crow states have over five times the illiteracy as whites in non-Jim Crow states. Why would Jim Crow inhibit the literacy of whites? Simply put, it did not. Instead, the enactment of Jim Crow laws emerged in the first place through some function of the illiterate and otherwise intellectually retarded views of whites in the US south. Unlike experiments, exposures in observational contextual effect studies are typically some function of group members. That is, exposures are endogenous.
|
Many epidemiologists would instinctively worry about confounding by SES. But confounder control in a multilevel study is anything but straightforward. In this case, if we adjust for SES in a blacks-only data set we ignore the fact that the meaningful variation in SES is not within blacks but between blacks and whites. Yet including whites in order to adjust for between race SES confuses things. Of special note is that only 3% of blacks in Jim Crow states owned a radio (i.e. were higher SES). Do we really want to employ a procedure (i.e. regression adjustment) that fictitiously removes this disparity from consideration? Further, since SES is most certainly a function of literacy the rationale for statistical adjustment of SES is dubious. For this and many other reasons, the upshot is that teasing out the independent effect of Jim Crow on black illiteracy is quite complicated and MLMs appear unable to help.
The way forward is to recall that the goal is to compare illiteracy in persons with otherwise identical characteristics residing in Jim Crow and non-Jim Crow states. Toward this end, it may be useful to restrict analyses to states that lie at the border of Jim Crow states (see Authors’ map). I hastily defined Jim Crow border states as: Arizona, Indiana, Kansas, Kentucky, Missouri, New Mexico, Oklahoma and West Virginia. And non-Jim Crow border states as: Colorado, Illinois, Iowa, Nebraska, Ohio and Utah. Presumably, these states have more exchangeable populations, leaving the key difference between them Jim Crow laws. The middle panel of Table 2 presents the corresponding results. It is readily seen that the impact of Jim Crow on black illiteracy declines to 2.29, but there remains wide disparities in white illiteracy. Historians would not be surprised. All of the border states were racially charged with many incidents of horrific racist acts in Jim Crow and non-Jim Crow states alike. Another problem (not shown) is that the disparity in radio ownership (i.e. SES) between Jim Crow and non-Jim Crow states in this border region remains large.
Yet another approach would be to seek out data akin to a natural experiment. For purposes here, I seek states that were not part of the confederation of states that attempted to succeed from the American Union during its racially motivated Civil War (circa 1861–65) but later ended up enacting Jim Crow laws. Such states would presumably have a diminished culture of racial animus that is confounding the desired effect. Again, hasty research suggests there were two such states: Kansas and Wyoming. Restricting analysis to a comparison between these states and other similarly situated states that did not enact Jim Crow (e.g. Colorado and Nebraska) would seem to better identify the effect of Jim Crow, now better disentangled from the background cultures so troubling in the 48-state and border state analyses.
The third panel in Table 2 presents these results. As with the border state analysis, the effect of Jim Crow seems to double black illiteracy but now there are negligible difference in white illiteracy and relatively little difference in radio ownership (not shown). This analysis would be considerably strengthened by evaluating data on Confederate states that did not enact Jim Crow, but I am unaware of any with these characteristics. It follows that the identification of Jim Crow effects rests on a careful analysis of anomalous but real cases, not the imputed fiction generated by a sophisticated regression model. A cross-tabulation tells the story, and illuminates the inferential limitations of it.
A final word on statistical inference seems necessary. With the tens of thousands of subjects analysed here there is no reason to present confidence intervals or P-values: all point estimates are extremely precise and most any difference is large enough to reject a null hypothesis of no difference. But several issues merit attention if one was interested in testing hypotheses from a frequentist perspective in a typical multilevel data set. Among these are Type I error rates. Critically, if my last analysis is considered to mimic a conventional group randomized trial about the impact of Jim Crow laws on illiteracy, the primary test statistic has only two degrees of freedom! This is because in group trials, the number of degrees of freedom for main effects equals the number of experimental arms times the number of groups in each arm (e.g. Jim Crow or not) minus one.58 The number of persons in total, per arm, or per group is not part of the calculation. Neither is the intraclass correlation coefficient. Because most MLM programmes default to evaluating test statistics against the asymptotic Z-distribution, the Type I error rates from analyses that include only a small number of groups are much larger than programme output implies. In other words, from the experimental perspective advocated here, P-values from MLMs may be artificially too small. If contexts are thought to be the driving force theoretically, they should be evaluated as such statistically.
|
Conclusion |
|---|
|
TopRobinson's paper in contextRobinson's mistakeMI and multilevel thinkingMultilevel modelsConclusionReferences |
|---|
I am in full agreement with the primary goal of the Author's new paper: epidemiologists should consider contexts and multilevel phenomena. I also agree with them that critical examination of a scientist's culture, history and personal motivation can be enlightening. Finally, we all agree that Robinson's analysis is technically correct and has withstood the test of time. But our agreement seems to end here.
First, existing data do not support the Authors’ claims about Robinson's motivation and/or impact. Second, abundant data show that the Authors’ claims about MI and multilevel research are wrong. Third, I do not see how estimated associations from a sophisticated MLM are an improvement over Robinson's simple correlations. There is a place for MLMs in social epidemiology but in too many cases researchers who employ them end up making a Faustian bargain (I am not against such transactions so long as researchers give informed consent and disclose accordingly). An experimental (thinking) approach, as originally advanced by Fisher, appears more promising. For proof of this, see the vast body of research on school effects. Finally, the two new papers by Gelman and Hernán demonstrate yet again that clarity in thought and a deep substantive understanding of the phenomena under investigation are the keys to scientific advancement.3,4 Whether our research is at the individual-, ecological- or multi-level, our analytic goal must not be reduced to a mere demonstration of model-fitting.
To sum up, I sincerely commend the Authors for writing a provocative paper. As this journal's editors know, discussion and debate are the life blood of scientific advancement. The purpose of this paper will have been accomplished if, however, it prevents the future computation of meaningless variance components and stimulates the study of multilevel causal inference in social epidemiology.
|
References |
|---|
|
TopRobinson's paper in contextRobinson's mistakeMI and multilevel thinkingMultilevel modelsConclusionReferences |
|---|
1 Subramanian SV, Jones K, Kaddour A, Krieger N. Revisiting Robinson: the perils of individualistic and ecological fallacy. In: Int J Epidemiol. (2009) 38:342–60.
2 Robinson WS. Ecological correlations and the behavior of individuals. Am Sociol Rev (1950) 15:351–57. Reprinted Int J Epidemiol 2009; 38:337–41.[CrossRef][Web of Science]
3
Gelman A, Shor B, Bafumi J, Park D. Rich state, poor state, red state,
blue state: what's the matter with Connecticut? Quart J Polit Sci
(2007) 2:345–67.[CrossRef]
4
Hernan MA, Alonso A, Logan R, et al. Observational studies analyzed
like randomized experiments: an application to postmenopausal hormone
therapy and coronary heart disease. Epidemiol (2008) 19:766–79. (with
discussion).[CrossRef][Web of Science][Medline]
5 Simpson EH. The interpretation of interaction in contingency tables. J Royal Stat Soc B (1951) 13:238–41.
6 Pearl J. Causality: Models, Reasoning, and Inference. (2000) New York: Cambridge University Press.
7
Lazarsfeld PF, Robinson WS. Some properties of the trichotomy ‘Like, No
Opinion, Dislike’ and their psychological interpretation. Sociometry
(1940) 3:151–78.[CrossRef][Web of Science]
8 Robinson WS. Bias, probability, and trial by jury. Am Sociol Rev (1950) 15:73–78.[CrossRef][Web of Science]
9 Robinson WS. A method for chronologically ordering archaeological deposits. Am Antiq (1951) 16:293–301.[Medline]
10 Robinson WS. The logical structure of analytic induction. Am Sociol Rev (1951) 16:812–18.[CrossRef]
11 Robinson WS. The motivational structure of political participation. Am Sociol Rev (1952) 17:151–56.[CrossRef][Web of Science]
12 Robinson WS. The statistical measurement of agreement. Am Sociol Rev (1957) 22:17–25.[Medline]
13 Robinson WS. The geometric interpretation of agreement. Am Sociol Rev (1959) 24:338–45.[CrossRef][Web of Science]
14 Robinson WS. Asymmetric causal models: comments on Polk and Blalock. Am Sociol Rev (1962) 27:545–48.[CrossRef][Web of Science]
15 Clogg CC. The impact of sociological methodology on statistical methodology. Stat Sci (1992) 7:183–207.[CrossRef]
16 Raftery AE. Statistics in sociology, 1950-2000: a selective review. Sociol Method (2001) 31:1–45.[CrossRef]
17
Oakes JM. An analysis of AJE citations with special reference to
statistics and social science. Am J Epidemiol (2005) 161:494–500.
18 Susser M. The logic in ecological: I. The logic of analysis. Am J Public Health (1994) 84:825–29.
19 Jacobs JA. ASR: yesteday, today, tomorrow. Am Sociol Rev (2004) 69:1–3.[CrossRef]
20 Srole L. Social integration and certain corrollaries: an exploratory study. Am Sociol Rev (1956) 21:709–16.[CrossRef][Web of Science]
21
Appellant's Brief. The effects of segregation and the consequences of
desegregation: a social science statement. In: Minnesota Law Rev.
(1953) 37:427–93.
22 Clark KB. The desegregation cases: criticism of the social scientist's role. Villanova Law Rev (1960) 5:224–46.
23 Hovenkamp H. Social science and segregation before Brown. Duke Law J (1985) 3–4:624–72.
24 Irons P. A People's History of the Supreme Court. (1999) New York: Viking.
25 Lucas SR, Paret M. Law, race and education in the United States. Ann Rev Law Social Sci (2005) 1:203–31.[CrossRef]
26
Van Den Haag E. Social science testimony in the desegregation cases - a
reply to Professor Kenneth Clark. Villanova Law Rev (1961) 6:69–79.[Medline]
27 DeGroot MH, Fienberg SE, Kadane JB (eds). Statistics and the Law. (1986) New York: John Wiley & Sons.
28 Rosen PL. The Supreme Court and Social Science. (1972) Urbana, IL: University of Illinois Press.
29 Erickson RJ, Simon RJ. The Use of Social Science Data in the Supreme Court. (1998) Urbana, IL: University of Illinois Press.
30 Selvin HC. On following in someone's footsteps: two examples of Lazarsfeldian methodology. In: Qualitative and Quantitative Social Research: Papers in Honor of Paul F Lazarsfeld.—Merton RK, Coleman JS, Rossi PH, eds. (1979,) New York: Free Press. 232–44.
31 Sills DL. Paul F. Lazarsfeld: 1901–1976. (1987) Washington, DC: National Academy of Sciences.
32 Coleman JS. Social theory, social research, and a theory of action. Am J Sociol (1986) 91:1309–35.[CrossRef]
33
Jerabek H. Paul Lazarsfeld—the founder of modern empirical sociology: a
research biography. Int J Pub Opin Res (2001) 13:229–44.[CrossRef]
34 Morgan SL, Winship C. Counterfactuals and Causal Inference. (2007) New York: Cambridge.
35 Fisher RA. The Design of Experiments. (1935) 1st. Edinburgh: Oliver and Boyde.
36 Bowles S. Microeconomics: Behavior, Institutions, and Evolution. (2004) Princeton, NJ: Princeton University Press.
37
Bowles S, Gintis H. The revenge of homo-economicus: contested exchange
and the revival of political economy. J Econ Perspect (1993) 7:83–102.[Web of Science]
38
Durlauf SN. A framework for the study of individual behavior and social
interactions (with discussion). Sociol Method (2001) 31:47–87.[CrossRef]
39 Przeworski A. Contextual models of political behavior. Polit Method (1974) 1:27–61.
40 Roemer J (ed). Analytical Marxism: Studies in Marxism and Social Theory. (1986) New York: Cambridge.
41 Coleman JS. Foundations of Social Theory. (1990) Cambridge, Massachusetts: Belknap Press of Harvard University Press.
42 Coleman JS, Campbell EQ, Hobson CJ, et al. Equality of Educational Opportunity. (1966) Washington DC: U.S. Government Printing Office.
43 Bryk AS, Raudenbush SW. Hierarchical Linear Models. (1992) Newbury Park: Sage.
44 Oakes JM. Invited commentary: rescuing Robinson Crusoe. Am J Epidemiol (2008) 168:9–12.
45 Berk R. Regression Analysis. A constructive critique. (2004) Thousand Oaks, CA: Sage Publications.
46 Freedman D. From association to causation: some remarks on the history of statistics. Stat Sci (2002) 14:243–58.
47
Oakes JM. The (mis)estimation of neighborhood effects: causal inference
for a practicable social epidemiology. Soc Sci Med (2004) 58:1929–52.
(with discussion).[CrossRef][Web of Science][Medline]
48 Vandenbroucke JP. Should we abandon statistical modeling altogether? Am J Epidemiol (1987) 126:10–13.
49 Vanderweele TJ. Ignorability and stability assumptions in neighborhood effects research. Stat Med (2007) 27:1934–43.[CrossRef][Web of Science]
50
Rosenbaum PR. Heterogeneity and causality: unit heterogeneity and
design sensitivity in observational studies. Am Statist (2005)
59:147–52.[CrossRef]
51
Sobel ME. What do randomized studies of housing mobility demonstrate?
Causal inference in the face of interference. JASA (2006) 101:1398–407.
52 Gelman A. Multilevel (Hierarchical) modeling: what it can and cannot do. Technometrics (2006) 48:432–35.[CrossRef][Web of Science]
53 Gelman A, Hill J. Data Analysis Using Regression and Multilevel/Hierarchical Models. (2007) New York: Cambridge.
54 Draper D. Inference and hierarchical modeling in the social sciences. J Educ Behav Stat (1995) 20:115–47.
55
Hearst MO, Oakes JM, Johnson PJ. The effect of racial residential
segregation on black infant mortality. Am J Epidemiol (2008)
168:1247–54.
56
Johnson PJ, Oakes JM, Anderton DL. Neighborhood poverty and American
Indian infant death: are the effects identifiable? Ann Epidemiol (2008)
18:552–59.[CrossRef][Web of Science][Medline]
57 U.S. Department of Commence. Fifteenth Census of the United States, 1930: Population Vol II: Chapter 13 Iliteracy. (1933) Washington, DC: US Government Printing Office.
58 Hannan PJ. Experimental social epidemiology: controlled community trials. In: Methods in Social Epidemiology.—Oakes JM, Kaufman JS, eds. (2006,) San Francisco: Jossey-Bass/Wiley. 335–64.
CiteULike 
Connotea 
Del.icio.us What's this?
Related articles in Int. J. Epidemiol.:
- Response: The value of a historically informed multilevel analysis of Robinson's data
- SV Subramanian, Kelvyn Jones, Afamia Kaddour, and Nancy Krieger
Int. J. Epidemiol. 2009 38: 370-373.[Extract] [Full Text] - Commentary: ‘Is the Social World Flat? W.S. Robinson and the Ecologic Fallacy’
- Glenn Firebaugh
Int. J. Epidemiol. 2009 38: 368-370.[Extract] [Full Text] - Ecological Correlations and the Behavior of Individuals
- WS Robinson
Int. J. Epidemiol. 2009 38: 337-341.[Extract] [Full Text] - Revisiting Robinson: The perils of individualistic and ecologic fallacy
- S V Subramanian, Kelvyn Jones, Afamia Kaddour, and Nancy Krieger
Int. J. Epidemiol. 2009 38: 342-360.[Abstract] [Full Text]
This article has been cited by other articles:
|
|
 |
|
  S. Subramanian, K. Jones, A. Kaddour, and N. Krieger Erratum Int. J. Epidemiol., June 1, 2009; 38(3): 891 - 894. [Full Text] [PDF]
|
|
|
|
|
|
S. Subramanian, K. Jones, A. Kaddour, and N. Krieger Response: The value of a historically informed multilevel analysis of Robinson's data Int. J. Epidemiol., April 1, 2009; 38(2): 370 - 373. [Full Text] [PDF]
|
|
|
|
|
|
J. Wakefield Multi-level modelling, the ecologic fallacy, and hybrid study designs Int. J. Epidemiol., April 1, 2009; 38(2): 330 - 336. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
