Using Google Earth to Conduct a Neighborhood Audit: Reliability of a Virtual Audit Instrument - PMC Skip to main content
Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Health Place. Author manuscript; available in PMC 2011 Nov 1.
Published in final edited form as:
Health Place. 2010 Nov; 16(6): 1224–1229.
Published online 2010 Aug 11. doi: 10.1016/j.healthplace.2010.08.007
PMCID: PMC2952684
NIHMSID: NIHMS229030
PMID: 20797897

Using Google Earth to Conduct a Neighborhood Audit: Reliability of a Virtual Audit Instrument

Philippa Clarke,a Jennifer Ailshire,a,1 Robert Melendez,a Michael Bader,a,2 and Jeff Morenoffa
Author information Copyright and License information PMC Disclaimer

Abstract

Over the last two decades, the impact of community characteristics on the physical and mental health of residents has emerged as an important frontier of research in population health and health disparities. However, the development and evaluation of measures to capture community characteristics is still at a relatively early stage. The purpose of this work was to assess the reliability of a neighborhood audit instrument administered in the city of Chicago using Google Street View by comparing these “virtual” data to those obtained from an identical instrument administered “in person”. We find that a virtual audit instrument can provide reliable indicators of recreational facilities, the local food environment, and general land use. However, caution should be exercised when trying to gather more finely detailed observations. Using the Internet to conduct a neighborhood audit has the potential to significantly reduce the costs of collecting data objectively and unobtrusively.

Characterizing Neighborhoods in Health Research

Over the last two decades, the impact of community characteristics on the physical and mental health of residents has emerged as an important frontier of research in population health and health disparities (Diez Roux 2001; 2004; O'Campo 2003; Sampson, Morenoff et al. 2002). The measurement of community characteristics is evolving, but strategies typically fall under one of three categories of measurement: secondary analysis of archival data sources, perceived (self-reported) responses in a community survey, and objective audit instruments (Brownson et al. 2009). Using secondary data from administrative sources (e.g. decennial census), both to define neighborhoods and as an aggregate measure of neighborhood characteristics, researchers have examined the relationship between various health outcomes and factors such as population density (Lopez 2004), land use diversity (Clarke and George 2005; Cervero and Duncan 2003), and block size (Boer, Zheng et al. 2007). These archival data are often enhanced using geographic information systems (GIS) to incorporate data on characteristics such as traffic volume (Tonne, Melly et al. 2007), street connectivity (McGinn, Evenson et al. 2007), and the availability of food (Bader et al. 2010) and recreational facilities (Diez Roux, Evenson et al. 2007) within local neighborhoods.

Tapping individuals' perceptions of their environments is another common measurement strategy (e.g. (Moore et al. 2008)), particularly in the research on physical activity and the built environment (Brownson et al. 2009). However, subjective reports from respondents are subject to same-source bias (e.g., those in poor health are more likely to report poorer neighborhood conditions) (Echeverría et al. 2008), and conflicting findings can arise when using both subjective and archival measures (McGinn, Evenson, et al. 2007). As an alternative, direct observation of neighborhood characteristics using an audit instrument relies on more objective measurement to capture many of the comprehensive and detailed environmental characteristics relevant for health (Clifton, Smith and Rodriguez 2007, Clarke, Ailshire et al. 2008; Schaefer-McDaniel, O'Brien Caughy, O'Campo, Gearey, 2010). While driving or walking through small-area respondent-centered neighborhoods, researchers observe and document neighborhood features using a standardized instrument (e.g. Pikora, Bull et al. 2002). The direct observational method known as systematic social observation (SSO) is a measurement strategy used in the social sciences (Reiss 1971; Raudenbush and Sampson 1999; Sampson and Raudenbush 1999) whereby survey interviewers or raters systematically rate each respondent's neighborhood block (e.g. condition of the street, presence of litter, and heavy traffic) during the survey period. However, these in-person audits are highly resource intensive and costly, making them prohibitive for many studies.

The development and evaluation of measures to capture community characteristics is still at a relatively early stage (Brownson et al. 2009; Sallis 2009), and only a few studies have explicitly compared measurement properties across different strategies (e.g. Bader et al. 2010). The purpose of this work was to assess the reliability of a neighborhood audit instrument administered using the Internet by comparing these “virtual” data to those obtained from an identical instrument administered “in person”. Using the Internet to conduct a neighborhood audit has the potential to significantly reduce the costs of collecting data “objectively and unobtrusively” (Brownson et al. 2009). Our objective in this work is to ascertain the reliability of this method by capitalizing on existing data that were collected as part of a study on neighborhoods and health in the city of Chicago.

Using the Internet for a Neighborhood Audit

Recently, there has been dramatic growth in internet capacities for observing and characterizing small area neighborhoods. Google Earth (Google Inc. 2005) is a free, internet-based software that displays satellite images of the earth's surface at a resolution of 15 meters or higher. Google Street View is a relatively new technology featured in Google Earth that provides 360° horizontal and 290° vertical panoramic views at the street-level (based on images taken at approximately 10 or 20 meter intervals) from a height of about 2.5 meters. Thus, Google Street View gives the viewer the feeling of virtually being on the street and the capacity to virtually walk down that street. Street View was launched on May 25, 2007 in several major US cities, and has been expanding to include coverage throughout the world.

The highly detailed imagery available in Google Street View raises the possibility of conducting a “virtual” neighborhood audit. Despite the widespread availability of visual data on community and built environments, few studies have utilized such electronic images on the Internet to characterize neighborhood environments (Curtis, Duval-Diop and Novak 2010; Doyle, Dodge and Smith 1998). In this paper we assess the level of agreement between street level characteristics documented by trained raters using SSO as part of a community-based survey in the city of Chicago, and data collected with an identical instrument using Google Street View. This is a case study that draws on existing data collected “in-person” in 2002, and collects comparable data using Google Street View when it became available 4 to 5 years later. While we would ideally like to have had more contemporaneous measurement occasions, cost considerations prohibited the collection of data solely for this purpose. Rather, this is an opportunistic study that draws on existing data to conduct a case study in Chicago, offering initial insight into the reliability of a virtual method. We hope this is a first step in considering the utility of this method and that other researchers will replicate such analyses in other settings with better temporal alignment of data.

Methods

Data

Data come from the Chicago Community Adult Health Study (CCAHS) which was conducted in 2002 through face-to-face interviews with a multi-stage probability sample of 3,105 adults aged 18 and over, living in the city of Chicago, and stratified into 343 neighborhood clusters previously defined by the Project on Human Development in Chicago Neighborhoods (Sampson, Raudenbush et al. 1997). CCAHS was specifically designed to examine the effects of neighborhoods on health and observational data were collected on the block around each sampled residence through the method of systematic social observation. Corresponding with each face-to-face interview, survey raters completed a standardized instrument for rating the block where the respondent lived. On the cover page of the instrument is a diagram of a typical city block on which the rater fills in the names of the streets s/he is coding (Figure 1). Each side of one of these streets is referred to as a block face, and a typical city block contains eight block faces. Each rater walked around the entire block two times while recording observations – the first time walking along the “inside” block faces and the second along the “outside” block faces. Inter-rater reliability of this method was demonstrated using a subsample of 80 blocks in a pilot study conducted in 2001 where two raters made separate, independent observations of the same block at the same time. Observed agreement ranged from .78 to 1.00 (κ =.27 to .91). Agreement tended to be higher for objective indicators (e.g. presence of high-rise housing; κ =.84) and lower for observations requiring a qualitative judgment (e.g. quality of street conditions; κ =.27).

An external file that holds a picture, illustration, etc. Object name is nihms229030f1.jpg
Chicago Community Adult Health Study: Systematic Social Observation Coding Sheet

Using this standardized instrument, observational data were collected on multiple neighborhood characteristics that have been shown to be related to health (see Table 1), including land use (e.g. housing type, commercial, institutional, industrial), recreational facilities (e.g. parks, playgrounds), food environment (e.g. supermarkets, fast food, restaurants, liquor stores), neighborhood physical and social disorder (e.g. garbage, litter, broken glass, graffiti, signs advertising alcohol), as well as built environment characteristics (e.g. presence of trees, quality of street conditions). Some questions are asked at the level of the block face, meaning that the rater must code each side of the same street separately (e.g., presence of graffiti on buildings, signs or walls). Other questions were asked at the street level where one observation was made for the entire street (e.g., condition of the street). For our purposes we focus on characteristics at the street level, aggregating the block face characteristics up to the street level where necessary.

Table 1

Street-Level Agreement in Observed Chicago Neighborhood Characteristics (N=244 Streets): Systematic Social Observation In-Person (2002) versus Google Street View (2007)

Observed AgreementKappa95% Confidence Interval
Recreational Facilities
Any recreational facilities (e.g. park, playground, sports fields)0.9230.499(.29, .70)
Any park0.9530.397(.11, .68)
Any playground0.9660.320(-.02, .66)
Any sports fields, playing fields, courts0.9700.573(.29, .86)
Food Environment
Any convenience store0.9240.064(-.12, .25)
Any supermarket/grocery store0.9410.099(-.13, .32)
Any fast food/take out0.9240.268(.03, .51)
Any restaurant/other eating place0.9030.412(.22, .61)
Any bar/cocktail lounge0.9320.252(.01, .49)
Any liquor store0.9620.381(.06, .70)
General Land Use
Any high-rise housing0.9730.713(.49, .93)
Any low-rise private housing0.7300.453(.35, .56)
Any detached single family houses0.8330.604(.49, .72)
Any commercial/industrial unit0.7750.475(.35, .60)
Any institutional land use (e.g. schools)0.8430.305(.13, .47)
Any church/religious center0.9070.449(.25, .64)
Any parking lots0.8180.360(.21, .51)
Commercial Land Use
Any bank0.9620.288(-.04, .62)
Any check cashing service0.9870.394(-.15, .94)
Any drug store/pharmacy0.9580.145(-.15, .43)
Indicators of Neighborhood Social and Physical Disorder
Any abandoned, burned out, or boarded up housing0.9240.147(-.07, .37)
Any garbage, litter, or broken glass in the street or on sidewalks0.3470.041(.01, .08)
Any vacant lots or open space0.8430.273(.11, .44)
Condition of residences (well kept vs. moderate/fair condition)0.6370.211(.10, .32)
No visible graffiti0.7970.095(-.05, .24)
Any second hand shop or pawn shop0.9270.160(-.07, .39)
Signs advertising alcohol0.9150.339(.12, .55)
Built Environment Characteristics
Street condition (Poor/Fair vs. Good)0.5980.032(-.10, .17)
Trees lining the street (trees on all, most or some of the street vs. none)0.9410.487(.25, .73)

For comparison, we used an identical instrument on a subset of 60 of these residential blocks (244 streets) to conduct a virtual SSO using Google Earth. These blocks were selected from a random sample of all blocks in the study and were spatially distributed throughout the city of Chicago (Figure 2), with somewhat greater density on the north side of the city. Using the Street View images for the city of Chicago, a trained rater did a virtual walk around the block where respondents lived and documented observed characteristics using the identical standardized SSO instrument. Google Street View images for the city of Chicago were dated around 2007 (about four to five years after the in-person SSO data were collected).

An external file that holds a picture, illustration, etc. Object name is nihms229030f2.jpg
Map Showing 60 Chicago Neighborhood Blocks (244 Street Segments)

Analyses

We examine the inter-source reliability of street-level characteristics observed in the virtual compared to the in-person neighborhood audit. Agreement between observed characteristics using the in-person SSO and the virtual SSO was assessed using the Kappa coefficient (Cohen, 1960). The Kappa statistic adjusts for the amount of agreement that could be expected to occur by chance alone (Landis & Koch, 1977), and ranges from 1.0 (representing perfect agreement) to 0 (representing agreement corresponding to that expected by chance). However, due to the sensitivity of the Kappa statistic to the underlying prevalence of the characteristic (Feinstein and Cicchetti 1990), we also report the observed agreement between the in-person and virtual SSO data. All analyses were conducted in SAS Version 9.2 for Windows.

Results

Observed agreement and Kappa statistics (with 95% confidence intervals) for the SSO data are presented in Table 1. Levels of observed agreement for the presence of recreational facilities and characteristics of the local food environment were high (>.90), indicating a high reliability between these types of observational data collected in person and using Google Street View. Corresponding Kappa coefficients tended to be lower (κ =.06 to .57), especially for aspects of the environment observed less commonly in residential areas (e.g. supermarkets). Observed agreement for indicators of general and commercial land use ranged from .73 (low-rise private housing) to .99 (check cashing services), with lower Kappa statistics obtained for less prevalent characteristics such as drug stores or pharmacies (κ =.15).

Similarly, indicators of the built environment and neighborhood social and physical disorder were assessed reliably using Google Street View, particularly for objectively observed conditions such as signs advertising alcohol (observed agreement =.92, κ =.34) or the presence of trees lining the street (observed agreement =.94, κ =.49). However, indicators requiring a finer level of observation (e.g. the presence of garbage, litter, or broken glass) were less reliably assessed using Google Earth (observed agreement=.35, κ =.04), as were those that were likely to have changed substantially over the five years between the in-person and virtual audit, such as the condition of streets and residential housing (observed agreement=.60-.64, κ =.03-.21).

Observed levels of agreement between characteristics collected using the in-person and virtual SSO instruments were comparable to the inter-rater reliability of the in-person audit conducted as part of the Chicago Community Adult Health Study (data not shown). For example, the inter-rater reliability for the presence of liquor stores on a street (observed agreement=.97, κ =.36) was similar to the level of agreement across the in-person and virtual audit instrument (observed agreement=.96, κ =.38).

Discussion

Compared to direct observational data collected as part of a face-to-face interview in the city of Chicago, we demonstrate in this case study that many neighborhood characteristics can be assessed reliably using a virtual audit instrument with the Street View feature of Google Earth. We found that the presence of recreational facilities and aspects of the local food environment are reliably captured using a virtual walk around with a standardized instrument. Observed agreement in the presence of parks, playgrounds and sports fields was over 92 percent, while observed agreement in characteristics of the local food environment (e.g. presence of fast food restaurants, bars, convenience stores) was over 90 percent.

While observed agreement in objectively rated characteristics was consistently high (>.70), corresponding Kappa coefficients tended to be lower, likely due to the low prevalence of many of these characteristics in a small subset of an urban residential area (Feinstein and Cicchetti 1990). In our data the overall prevalence of bars or signs advertising alcohol was around 3%, while the prevalence of abandoned buildings or graffiti was 10%. Because the expected chance agreement is inflated for these rare characteristics, the denominator of the Kappa statistic is minimized, resulting in a low Kappa value (Feinstein and Cicchetti 1990). Nevertheless, for most characteristics the level of agreement was not due to chance (Kappa statistic significantly different from zero), and many of the Kappas indicated fair (κ =.20-.39) to moderate (κ =.40-.59) or substantial (κ =.60-.79) agreement between the two rating methods (Landis & Koch, 1977).

Consistent with other work on the reliability of in-person audit instruments (Clifton, Livi Smith and Rodriguez 2007), agreement tended to be lower for characteristics requiring a qualitative judgment, such as the quality of street conditions, and also for those requiring highly detailed observations at the street level (e.g. presence of garbage, litter or broken glass) that may be less obvious using Google Street View images. Given the five year interval in our study between the in-person and virtual audit, reliability was also lower for more fluid neighborhood characteristics that are likely to change considerably over time (e.g. the presence of graffiti or the condition of residential housing). Given the general comparability between the observed agreement across the in-person and virtual audit and the inter-rater reliability of the in-person audit (.78 to 1.00; κ =.27 to .91), some of the variability in characteristics observed across modes of observation may in fact be due to inter-rater reliability or test-retest reliability over the five years between observations.

Limitations

Currently, coverage in Google Street View is not complete. While it tends to be more comprehensive in urban rather than rural areas, not all cities have Street View available for all streets (especially smaller streets). Moreover, ethical issues and controversies have been raised surrounding the use of Street View, and not all users have access to these data. In addition, the dates of the images in Google Street View are not always readily apparent. Using Google Street View for a virtual neighborhood audit is contingent upon a temporal alignment between the Street View images and the individual data to which researchers may wish to link them. Our study was limited by a five year lag between in-person and virtual assessments. Further studies are needed to replicate these analyses in other settings with more contemporaneous timing between the in-person and virtual audit. This line of research would also benefit from an assessment of inter-rater reliability in the virtual audit (using more than one rater for the Street View assessments).

However, our results indicate that a virtual audit instrument using Google Street View can provide reliable indicators of recreational facilities, the local food environment, and general land use at a fraction of the cost of an in-person neighborhood audit. Objective indicators of the built environment and neighborhood social and physical disorder are also reliably assessed. Caution should be exercised when conducting more qualitative observations (e.g. quality of street conditions or residential housing) or when trying to gather more finely detailed observations (e.g. garbage, litter or broken glass) that benefit from direct observation in the field. Researchers should also be aware that strong agreement between measurements is not necessarily indicative of valid measurement. As for all data collection methods, rigorous and standardized training of raters is important for the quality of both the in-person and neighborhood audit. However, there are opportunities for considerable cost savings with a virtual data collection instrument because raters are not required to travel to different locations to perform the neighborhood audit. The use of a virtual audit also allows researchers greater flexibility in the data collection phase. Similar to going back to a stored blood spot for biological markers on a respondent, it is possible to return to the Street View images at a later date (provided they have not been updated) if it becomes apparent that other aspects of the environment need to be documented.

Our study contributes to the growing literature on the development and evaluation of measures to capture community characteristics (Brownson et al. 2009; Sallis 2009). We empirically demonstrate the reliability of using Internet-based resources to conduct a neighborhood audit, providing evidence for researchers to consider during the design stages of a project when weighing issues such as the number of raters to employ or the number of city blocks to observe. Despite the widespread availability of visual data on community and built environments, few studies have utilized such electronic images on the Internet to characterize neighborhood environments (Curtis, Duval-Diop and Novak 2010; Doyle, Dodge and Smith 1998). Our hope is that future research will continue to examine the utility of the Internet for conducting a neighborhood audit across other settings.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  • Bader MD, Ailshire JA, Morenoff JD, House JS. Measurement of the Local Food Environment: A Comparison of Existing Data Sources. Am J Epidemiol. 2010;171:609–617. [PMC free article] [PubMed] [Google Scholar]
  • Boer R, Zheng Y, et al. Neighborhood design and walking trips in ten U.S. metropolitan areas. American Journal of Preventive Medicine. 2007;32:298–304. [PMC free article] [PubMed] [Google Scholar]
  • Brownson RC, Hoehner CM, Day K, Forsyth A, Sallis JF. Measuring the Built Environment for Physical Activity: State of the Science. American Journal of Preventive Medicine. 2009;36:S99–S123.e12. [PMC free article] [PubMed] [Google Scholar]
  • Cervero R, Duncan M. Walking, bicycling, and urban landscapes: Evidence from the San Francisco Bay area. American Journal of Public Health. 2003;93:1478–1483. [PMC free article] [PubMed] [Google Scholar]
  • Clarke P, Ailshire JA, et al. Mobility Disability and the Urban Built Environment. American Journal of Epidemiology. 2008;168(5):506–513. [PMC free article] [PubMed] [Google Scholar]
  • Clarke P, George LK. The role of the built environment in the Disablement Process. American Journal of Public Health. 2005;95(11):1933–1939. [PMC free article] [PubMed] [Google Scholar]
  • Clifton Kelly J, Livi Smith AD, Rodriguez D. The development and testing of an audit for the pedestrian environment. Landscape and Urban Planning. 2007;80:95–110. [Google Scholar]
  • Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37–46. [Google Scholar]
  • Curtis A, Duval-Diop D, Novak J. Identifying spatial patterns of recovery and abandonment in the post-Katrina Holy Cross neighborhood of New Orleans. Cartography and Geographic Information Science. 2010;37:45(12). [Google Scholar]
  • Diez Roux AV. Investigating neighborhood and area effects on health: Commentary. American Journal of Public Health. 2001;91(11):1783–1789. [PMC free article] [PubMed] [Google Scholar]
  • Diez Roux AV. The study of group-level factors in epidemiology: Rethinking variables, study designs, and analytic approaches. Epidemiologic Reviews. 2004;26:104–111. [PubMed] [Google Scholar]
  • Diez Roux AV, Evenson KR, et al. Availability of recreational resources and physical activity in a sample of adults. American Journal of Public Health. 2007;97(3):493–499. [PMC free article] [PubMed] [Google Scholar]
  • Doyle S, Dodge M, Smith A. The potential of Web-based mapping and virtual reality technologies for modelling urban environments. Computers, Environment and Urban Systems. 1998;22:137–155. [Google Scholar]
  • Echeverría S, Diez-Roux AV, Shea S, Borrell LN, Jackson S. Associations of neighborhood problems and neighborhood social cohesion with mental health and health behaviors: The Multi-Ethnic Study of Atherosclerosis. Health & Place. 2008;14:853–865. [PubMed] [Google Scholar]
  • Feinstein AR, Cicchetti DV. High agreement but low Kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology. 1990;43(6):543–549. [PubMed] [Google Scholar]
  • Google Inc. Google Earth. Mountain View, CA: 2005. [Google Scholar]
  • Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. [PubMed] [Google Scholar]
  • Lopez R. Urban sprawl and risk for being overweight or obese. American Journal of Public Health. 2004;94:1574–1579. [PMC free article] [PubMed] [Google Scholar]
  • McGinn AP, Evenson KR, et al. The relationship between leisure, walking, and transportation activity with the natural environment. Health & Place. 2007;13:588–602. [PubMed] [Google Scholar]
  • Moore LV, Diez Roux AV, Nettleton JA, Jacobs DR., Jr Associations of the Local Food Environment with Diet Quality--A Comparison of Assessments based on Surveys and Geographic Information Systems: The Multi-Ethnic Study of Atherosclerosis. Am J Epidemiol. 2008;167:917–924. [PMC free article] [PubMed] [Google Scholar]
  • O'Campo P. Invited commentary: Advancing theory and methods for multilevel models of residential neighborhoods and health. American Journal of Epidemiology. 2003;157:9–13. [PubMed] [Google Scholar]
  • Pikora TJ, Bull FCL, et al. Developing a reliable audit instrument to measure the physical environment for physical activity. American Journal of Preventive Medicine. 2002;23:187–194. [PubMed] [Google Scholar]
  • Raudenbush SW, Sampson RJ. Ecometrics: Toward a Science of Assessing Ecological Settings, With Application to the Systematic Social Observation of Neighborhoods. Sociological Methodology. 1999;29:1–41. [Google Scholar]
  • Reiss AJ. Systematic observation of natural social phenomena. Sociological Methodology. 1971;3:3–33. [Google Scholar]
  • Sallis JF. Measuring Physical Activity Environments: A Brief History. American Journal of Preventive Medicine. 2009;36:S86–S92. [PMC free article] [PubMed] [Google Scholar]
  • Sampson RJ, Morenoff JD, et al. Assessing “neighborhood effects”: Social processes and new directions in research. Annual Review of Sociology. 2002;28:443–478. [Google Scholar]
  • Sampson RJ, Raudenbush SW. Systematic social observation of public spaces: A new look at disorder in urban neighborhoods. American Journal of Sociology. 1999;105:603–651. [Google Scholar]
  • Sampson RJ, Raudenbush SW, et al. Neighborhoods and Violent Crime: A Multilevel Study of Collective Efficacy. Science. 1997;277:918–924. [PubMed] [Google Scholar]
  • Schaefer-McDaniel N, O'Brien Caughy M, O'Campo P, Gearey W. Examining methodological details of neighbourhood observatiosn and the relationship to health: A literature review. Social Science and Medicine. 2010;70:277–292. [PubMed] [Google Scholar]
  • Tonne C, Melly S, et al. A case-control study of exposure to traffic and acute myocardial infarction. Environmental Health Perspectives. 2007;115:53–57. [PMC free article] [PubMed] [Google Scholar]