Abstract
Many aspects of our society are reflected in how we have transformed the earth over time. However, the limited availability of highly detailed historical and spatial data made it difficult for us to deepen our knowledge of land use methods in the long term. Using a proprietary national database of housing and real estate, which is the result of extensive, industry-driven efforts to harmonize data, we have created publicly available mesh surface sequences that describe the progress of built-up land use in the neighboring United States with spatial accuracy (i.e. , 250 m × 250 m) and time resolution (i.e. 1 year – 5 years) in the years 1940-2015. Six land-use classes are represented in the data product: agricultural, commercial, industrial, residential, residential-income and recreational facilities, as well as complementary uncertainty layers informing users of measurable components of data uncertainty. The datasets are part of the US Historical Settlement Data Compilation (HISDAC-USA) and enable the generation of new knowledge on long-term land-use dynamics, opening new avenues of inquiry in many areas of research.
Background & Summary
Land use, land cover and settlement databases are typically remote sensing or combined products that have made a significant contribution to the scientific research of environmental and human systems but are limited in time and may suffer from poor classification accuracy and limited subject matter depth 1, 2,3,4. Moreover, the lack of processing infrastructure has created significant obstacles to the deepening of our understanding of the historical development of settlement5,6,7. With increasing data availability and technological advances, large-scale spatial-spatial data infrastructures are becoming more feasible and popular in the social and natural sciences8,9. As such, data products such as National Land Cover Dataset (NLCD) 1,10,11 or Global Human Settlement Layer (GHSL) 12 typically characterize the physical properties of surfaces as measured by remote sensing signals over time but cannot represent the thematic details of settlements (e.g. land use classes). Such data did not exist until the 1970s, when remote sensing Earth observation began to operate on a global scale. Accordingly, scientists are able to assess and quantify changes in built-up land, building intensity or the share of built-up land over several decades, but have limited understanding of the semantic and functional elements of building and real estate land. use and its changes. Moreover, existing datasets going back in time are usually model-based with unknown accuracy and low spatial detail13,14.
Significant progress in our understanding of rural and urban development can only be made if we are able to capture the underlying spatio-temporal processes that contribute to small-scale land change. However, so far significant obstacles to the availability of alternative data and the computational costs of obtaining relevant information15,16,17, low spatial detail contained in historical records18 and limited geographic scope19,20,21 have made it difficult for us to create high-resolution data layers. This may interest you : Is the United States Entering a Recession?. which represent different aspects of land use in an urban and rural environment over longer periods of time and over a large spatial extent.
The multi-temporal layers of land use described in this article are fundamentally different from the previous parts of the land cover / land use data due to a wealth of attributes, temporal scope, and excellent temporal and spatial resolution. We detail the development and properties of a novel meshed data product that represents the progression of built-up land use in the United States between 1940 and 2015. This product was created using the Zillow Transaction and Assessment Dataset (ZTRAX), which is a collection of over 200 million geocoded housing and property records collected from existing cadastral data sources22. These records were rasterized to generate two core datasets spanning the 1940-2015 period, most land-use layers with an annual temporal resolution, and class-specific property count layers with half-decade temporal resolution, with a spatial resolution of 250 µm for most of the contiguous United States (CONUS) , covering six different land use classes.
This unique dataset has the potential to change our understanding of how the composition of communities and urban centers in the United States has evolved over the course of 75 years. These data products will be of great use to social and life sciences scientists and will be applicable to research related to urban development, vulnerability and natural hazards. Although ZTRAX is a proprietary data source, the derived data described here are disseminated as public data to the scientific community. The historical land use data is published as part of the US Historical Settlement Data Compilation (HISDAC-US) 23.24 and will be available through the Harvard Dataverse (https://dataverse.harvard.edu/dataverse/hisdacus). HISDAC-US has been used in several recent studies on urban development and change25,26,27,28,29,30, landscape change analysis and modeling31, transport infrastructure analysis32, population modeling33 as well as natural risk assessment34,35.
Input Data and Methods
Semantic aggregation of ZTRAX land use types
The ZTRAX dataset contains information on over 200 million shipments using over 400 million public registers22. External vendors and internal initiatives were used to collect data from evaluator information and publicly available documentation. The wealth of attributes of this dataset offers unique opportunities to study land use progress and the built environment from novel and compelling perspectives. Recently, ZTRAX has gained more and more popularity in the life and social sciences36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54.
This land use data product contains six built environment thematic classes that represent the types of land use by building structures. The six thematic classes described in the presented data are: agriculture, trade, industry, housing, housing-income and recreational facilities. The six classes used in this document represent a subset of the rich land use classification used in ZTRAX (300+ land use classes in total) and were selected for their importance in studying urban dynamics and development55,56,57,58,59. There are 12 general subject classes in ZTRAX; agriculture, commercial, exempt, government, historical, industrial, miscellaneous, private, residential, recreational, transportation and vacant. Read also : How HBO Max replaced Netflix in our streamer power rankings. Due to the low overall representation and incompleteness of several classes and the importance of the 6 data included in the product described, 7 classes (exempt, government, historical, miscellaneous, private, transport and free) were omitted from the data and the housing class was divided into housing and housing income. . These omissions are reflected in the uncertainty shape files and mesh layers we provided and characterized with the cumulative county sum and mesh cell count attribute. In addition, we provide subclasses of the included and excluded categories of land use and their frequencies60.
The agricultural thematic class in ZTRAX includes 23 subclasses that define agricultural parcels in more detail. For our purpose, all unstructured (i.e. unbuilt) agricultural subclasses have been removed from the data prior to processing, retaining all structures such as farms, ranches, various structures and rural non-residential structure improvements. This step ensures that all classes are defined on the basis of built-up structures and not the general purpose of the soil. Examples of excluded uses of agricultural land are pasture, arable land and other uses that do not describe the physical structure. The reader is directed to the circular histogram60 to fully break down all 300+ land-use types and the frequency with which they appear in the ZTRAX database.
The commercial sector includes 65 subclasses, ranging from office and medical buildings to dry cleaners, casinos and gas stations – no data has been removed from this class. For the industrial theme, the ZTRAX data includes 44 subclasses that distinguish heavy industrial buildings such as labor camps, quarries, and slaughterhouses, as well as lighter industrial facilities such as assembly plants, recycling centers, and loft buildings. The residential (or residential) sector is divided into two basic categories 1) Residential owned buildings (RO) or residential buildings owned by a Housing Account Holder who owns the property at a registered address for the provision of services (https: // www. Lawinsider.com/ dictionary / residential-owner) and 2) Income Residential (RI) buildings or residential buildings that have been used as rented or rented (i.e. owner-occupied) housing 60. The housing sector contains 36 subclasses describing housing construction, all of which are included in the final grid product.
Finally, the recreational land use class includes recreational facilities that contain 32 subclasses, including bowling alleys, playgrounds, zoos, and dance halls. The Land Use Attribute can have three levels of detail. Down to the last detail, the attributes distinguish aspects of the subclasses, such as the quality of duplex enclosures. For the presented data products, we have included attributes limited to the basic thematic classes. Table 1 shows the progression of the records for these land use classes since 1940, and the circular histogram60 shows the subclasses included in each thematic class represented in the data.
Gridded surface creation
The ZTRAX database is based on the records of cadastral and tax plots obtained from the state and poviat records and contains over 400 million records in total22, of which approximately 200 million have spatial information. This spatial information typically consists of address points or approximate centroid locations of the plots. Due to differences in reporting practices across counties, areas of the country are under-represented, particularly in the Midwest and Louisiana. See the article : Meet the 2021-22 Boone County Sports Awards Athletes of the Year. According to Zillow’s documentation, home legal transactions are processed by the county registrar’s office, and it is quite common that county registrars fail to record the assessor’s address or plot number (APN) in the legislation. In such cases, it is not possible to systematically allocate these records to the specific parcels concerned (https://www.zillow.com/research/ztrax/ztrax-faqs/). The absence of an APN or address manifests itself in areas of missing data (eg, Wisconsin and Louisiana) in the dataset presented in this document.
We converted ZTRAX data (available as CSV data) into a set of relational databases for efficient query execution. We extracted property locations, land use and year of construction attributes, and assigned a two-dimensional spatial index (i.e., grid cell ID) to each record, referencing a 250 m × 250 m grid in the Albers Equal Area Conic projection (SR-ORG: 7480) (https: //spatialreference.org/ref/sr-org/7480/). This mesh is compatible with the spatial mesh used in other HISDAC-US data products. We then split the data into complete records and records with missing year of construction or land use attributes (Chapter 2.3). We rasterized complete data records to generate grid cell level land use statistics in annual and half-decade slices as defined for each record’s year of construction attribute23,24. In particular, these records have been grouped into space-time intervals (according to the grid cell identifier and the year of construction attribute) and processed to determine the most common land use per grid cell per year during the 1940-2015 period. These summary statistics (i.e. most common land use type per grid cell ID and year) were then used as inputs to the rasterization process. We used the Python packages Numpy61 and Rasterio62 to generate a mesh surface in GeoTiff format. We also used this process to calculate the abundance of each elementary land use class per grid cell for each half decade, starting in 1940. Data with missing spatial references and missing attribute values (ie year of construction, land use type) were then used. to calculate various uncertainty statistics using pandas63 and geopandas64 and Python65 base packages. FROM At the same time, this data product consists of three items: (a) annual dominant (majority class) land use strata, (b) half decade strata measuring the number of properties of each land use class per grid cell, and (c) associated uncertainty area.
Creation of uncertainty layers
The uncertainty in the data generated consists of several elements: (a) incomplete ZTRAX data due to missing attribute or missing geolocation (i.e. non-georeferenced records), (b) survival error due to historical land use changes not captured by ZTRAX, and (c) thematic uncertainty in the land use attribute. While the latter two types of uncertainty are attempted to be quantified using auxiliary data (see section 3), the incompleteness of the ZTRAX attribute can be quantified directly and is reported in the accompanying datasets as additional county and grid cell summary statistics. In addition, ZTRAX suffers from positioning inaccuracies due to the approximation of plot units to separate point locations and associated problems50. In previous work, we quantified and reported positional, thematic and temporal uncertainties in the existing settlement layers, and shared the uncertainty layers hosted in the HISDAC-US23.24 repository (https://dataverse.harvard.edu/dataverse/hisdacus). These uncertainty layers are highly recommended for users interested in using historical billing data products. The uncertainty layers described here focus on the data gaps in the sequencing of the temporal layers of land use in the grid at the county level and at the grid cell level.
We calculated the percentage of records with missing land use attributes and / or missing geolocation to quantify uncertainty at the county level. We determined the total number of records in each county and calculated the proportion of records with and without georeferencing. In addition, we compiled the missing attributes for the year built (“to”) and land use (“lu”) attributes in the cross table:
percentage of records that contained both the value of land use and the value of annual buildings (“by-lu”)
percentage of records excluding land use value and annual value (“nby-nlu”)
percentage of records with the use of land and without the year of construction (“nby-lu”)
percentage of records with year of construction and without land use (“by-nlu”)
To further characterize the lack of county-level attributes over time, we generated ten-year county-level shape files containing a proportion of records with a valid year of construction attribute but a missing land-use attribute per county. The result was seven county boundary files, one for each decade over the time span of our data. For each decade, we then calculated the proportions of the georeferencing records, the proportions of the records that had both the year of construction and the land use attribute, and the proportions of the records with the missing land use attribute for that decade. Additionally, we created uncertainty layers at the mesh cell level. To gain an accurate understanding of the uncertainties, we generated mesh uncertainty layers using georeferenced records in the ZTRAX data. The grid time sequences (ten years) were created using all georeferenced records that had a value for the construction year but no land use attribute to quantify the proportion of missing land use type entries at the grid cell level. Data users are advised to include these uncertainty layers in their analyzes in order to be able to take into account the varying quality of the data, both regionally and over time.
Data Records
Historical gridded land use layers
The datasets described in the following sections have been published to the Harvard Dataverse HISDAC-US repository at the following URL https://dataverse.harvard.edu/dataverse/hisdacus66,67,68. The multi-term land use areas are organized as sequences of georeferenced mesh layers (filenames include the year, e.g. LU_ThemeMaj_1985) covering most of the built-up areas in CONUS (excluding Hawaii, Alaska and uncovered counties) with a spatial resolution of 250 µm and a temporal resolution of 1 year for the data product majority grades and 5 years for the class-specific layers. In the main data product, each grid cell value represents the most common land use class of all ZTRAX records located in that grid cell for that year (for all geo-referenced records with construction year and land use). In addition, for each individual land use class, we have created a time sequence of mesh count layers representing the number of records for that land use class (e.g. industrial) in the grid cell for each half decade, starting in 1940. These layers have a land use class and year included in their filenames (eg LU_ThemeCount_RO_1975, for residential buildings in 1975). These data products cover the period 1940-2015. We provided the raster layers in GeoTIFF format with a spatial resolution of 250 m. We adjusted these layers to the existing layers in HISDAC-US to ensure the consistency of the billing data products included in this data compilation. We published all data in the HISDAC-US repository using the Albers Equal Area Conic projections for the neighboring United States (USGS version, SR-ORG: 7480).
Figure 1 illustrates the different aspects of the land use data package that offer new perspectives for urban development. The dataset allows the user to understand urban development in terms of land-use change, not only through the thematic majority, but also by counting the areas that characterize the increase in land use classes over time. The top three lines in Fig. 1 show the cumulative figures for commercial, residential, and residential land use classes at three points in time in Houston, Texas. In the bottom row in Fig. 1.The cumulative counts for all other Land Use, Agriculture, Industry, and Leisure classes are displayed. Additionally, we generated contemporary (i.e. 2016) counting surfaces for thematic classes and the accompanying uncertainty surfaces. These 2016 strata also contain those records that do not contain data for the year of construction, and therefore provide a more complete picture of newer land-use patterns.
For Houston, Texas, land use properties for 1945, 1985, and 2015 are counted. The top three lines display subject-specific meters. The bottom line shows the aggregate number of land use classes for agricultural, industrial and recreational land.
Uncertainty surfaces
As described above, we have created several uncertainty surfaces to provide information on the fundamental aspects of data quality. The completeness of the data and multivariable processing quickly creates a complex picture of uncertainty. There are two categories of uncertainty layers we have shared: vector files with multi-time data in the table of attributes aggregated to the county level (2010-boundaries) (https://www.census.gov/geographies/mapping-files/ time-series / geo / carto -boundary-file.html) and multi-term uncertainty layers in the grid consistent with the main data products.
Seven county boundary vector layers are provided, one for each decade. Each tier contains a proportion of records that are georeferenced and a proportion of records that are not georeferenced but reside in that county based on the county ID for each decade. Additionally, we have provided the proportion of records that have a year of construction and the proportion of records that have land use for that county and year for both geo-referenced and non-geo-referenced data. Due to the exclusion of poorly represented land use types (i.e. excluded, government, historic buildings), there are counties in uncertainty plots that have more cumulative structures listed in the year of construction attribute than those listed in the land use attribute column. Cases with more structures in a given year than included in the land use attribute represent those with the land use types not included in the land use data. We provided an additional ten-year layer of uncertainty in the grid to address structures that were excluded from the dataset. For each decade, we calculated the total number of excluded structures per mesh cell. There are 6 thematic classes excluded from the main data product and 12 agricultural subclasses that have been excluded because they do not characterize built-up structures. The six non-agricultural thematic classes represented by this lattice layer of uncertainty are: (1) Exempt, (2) Historical, (3) Miscellaneous, (4) Private, (5) Transport, and (6) Free. The data user is encouraged to use these layers and the detailed land use disaggregation60 to perform an analysis using the quality of the master data and completeness information.
Multi-temporal mesh uncertainty layers for all georeferenced data quantify gaps in land-use type entries with the same resolution as the main data product. Each surface only represents structures that have been explicitly geocoded in ZTRAX. There are five attributes characterizing uncertainty in shape files at the county level: (1) cumulative sum of all structures included in ZTRAX for each county and decade, (2) cumulative sum of all features containing land use attribute per county and decade, (3) cumulative sum of all buildings with the attribute of the year of construction by county and decade, (4) the proportion of buildings containing the attribute of land use to all buildings in the county for a decade, and finally (5) the proportion of buildings containing the attribute of the year of construction to all buildings in the county for a decade. The shapefiles containing the proportions represent the completeness of the land use or year of construction attributes. Data users are encouraged to use the uncertainty surfaces provided with the data presented in this document and the position uncertainty layers published in HISDAC-US23,24 to assess the suitability of the data for a given location and to take into account the inherent position uncertainty. The table below (Table 2) describes the files included in the spatial development data sets.
Technical Validation
ZTRAX is exposed to quality issues that include spatial, temporal and thematic uncertainties that spread to the mesh surfaces contained in HISDAC-US. Some of these uncertainties have been quantified in previous works23,24. In order to accurately assess the accuracy of the positioning of mesh surfaces in HISDAC-US, over time and in the rural-urban continuum, we refer the reader to Uhl et al. (2021a), who report regionally different levels of positional agreement. Uhl et al. (2021a) provide important information on the quantitative consistency of ZTRAX-derived grid cell aggregates on built-up inventories and locations, compared to land area, census and housing data. Such reported misunderstandings also spread to the land use data layers described herein, and the user of any HISDAC-US data products is encouraged to refer to these validation results to adequately reflect the accuracy of the data. From this validation it is known that although completeness in HISDAC-US is acceptable for post 1900 data layers, all products derived from ZTRAX will be underestimated due to difficulties in obtaining structured records from counties that have different reporting rules, incompleteness and inconsistency of attributes and the dynamic nature of development. The level of underestimation for housing records was assessed in Uhl et al. (2021a), who reported different levels of incompleteness of records along the village-city continuum compared to the number of housing units in the census.
In this document, we assessed the completeness of the land use and annual construction attributes in ZTRAX (section 4.1) and applied three supporting datasets to quantitatively and qualitatively address the uncertainties specific to the land use product. Specifically, we used land use data from voluntary geographic information (i.e. OpenStreetMap, OSM) (https://planet.openstreetmap.org) to evaluate the contract with the established (modern) land use layers (section 4.2) and compared our use layers to land cover / land use data from remote sensing (LULC) from the National Land Cover Database 200169 and 201670, as well as to the urban land use classes from the Local Climate Zone (LCZ) dataset 71 available to CONUS (Chapter 4.3). In addition, we used building demolition data to quantify the impact of a survival error as building replacements or demolitions are not recorded in ZTRAX (section 4.4). Finally, we used top-down images and a visual-analytical approach to assess the visual consistency of buildings at ZTRAX locations for different land use categories (section 4.5).
Attribute incompleteness
Grid cells with low structural number and low attribute completeness should be carefully considered as the cell value assignment in the main data product is based on the most common land use class within the grid cell coverage. In such cases, it is recommended that the user use the land use type counting layers in conjunction with the uncertainty layers to better understand the underlying reliability of the data. Table 3 summarizes the attribute missing statistics for the land use data product, showing that more than 98% of ZTRAX records contain correct land use information (lower levels are e.g. in Maine or Iowa, see Figure 2a) and more than 75% of ZTRAX records are contain up-to-date information on land use and year of construction. Most ZTRAX records contain correct location information (Figure 2b). About 25% of the data do not contain information on land use and annual buildings, and these are located in about 400 counties (see Figure 2c), which can also be identified from the completeness strata at the county level (section 3.2).
ZTRAX attribute completeness: Percentage of records per county with valid (a) land use attribute, (b) location attribute (i.e. latitude and longitude) and (c) year of construction attribute.
Comparison to OpenStreetMap land use data
While detailed and reliable land use data is scarce, OpenStreetMap (OSM) offers user-generated land use information and functional information at the building level. While the SSO is not expected to be highly complete in terms of the land use attribute, we assume that the reported land use information is accurate. We generated mesh surfaces, matched to HISDAC-US land use data grids, containing the number of buildings of a given land use type in OSM per grid cell, and performed a contract assessment at the cell level. We have mapped the relevant OSM land use types to the land use classification scheme of the HISDAC-US land use data presented. Moreover, due to the rarity of some land use classes in both datasets and the potentially heavy burden it introduced, we assessed only the three most common land use classes: residential, commercial and industrial.
Initial tests showed that a significant number of building outlines in OSM do not have a land use attribute, and therefore its completeness in OSM appears to be low in some CONUS regions, while the correctness of these attributes that exist should be high. Hence, only Type II errors (ie commission errors) in the land use data of ZTRAX can be quantified by comparison with OSM data. Please refer to section 4.3 (comparison with remotely read LULC data) and section 4.4 (survivability burden) to evaluate the commission error. It should be noted that this assessment was performed for the last time point of the presented series of land use layers (i.e. 2016) and for modern OSM data taken in 2021 to minimize the time gap. We performed these assessments on a county-by-county basis as well as across the village-city continuum using Rural and Urban Continuity (RUCC) codes created by the United States Department of Agriculture (USDA) 72.73. The RUCC define nine rural-urban classes, including three metropolises and six non-metropolitan counties, using criteria of population size, degree of urbanization, and neighborhood with an urban area (Table 4).
Given these constraints in the OSM reference data, we first extracted all grid cells containing at least one record from OSM and ZTRAX of the same land use class and evaluated the correlations of the grid cell count as a measure of quantitative consistency. Due to the structure of the ZTRAX data and the effects of spatial generalization, these distributions may contain outliers resulting from the large number of records in individual cells of the grid24, therefore we used the Spearman’s rank correlation coefficient for this assessment. In addition, we calculated a reminder (i.e. accuracy or manufacturer sensitivity) of the ZTRAX land use counts against the OSM to quantify the skip errors associated with the ZTRAX data. The latter was made based on the binarized absence-presence mesh surfaces, using a threshold of at least one record per mesh cell, thus allowing the measurement of the type II error term of positional consistency between surfaces from ZTRAX and OSM. We quantified both the correlations of the number of cells in the grid and the spatial reconciliation (i.e. reminders) for each of the three studied land-use classes for the entire CONUS and for the rural-urban continuum by performing stratified assessments for grid cells located in the counties of each RUCC (Table 4 ), and also for each poviat separately (Fig. 3).
Comparison to building-level land use classes from OpenStreetMap. (a) Distribution of the Spearman correlation coefficient based on the number of 250 × 250 m grid cells of residential, commercial, and industrial records; and (b) distribution of values recalled at the county level; Panels (c) and (d) show the correlation and recall distributions at the county level, disaggregated for each village-city continuum code.
First, we observe positive correlations between the number of buildings in the ZTRAX and OSM land use classes in CONUS (> 0.33 in all RUCCs for any class), and these correlations are highest for the residential class in highly urbanized environments (i.e. RUCC 1, c = 0.66). The correlations generally decrease towards more rural environments where both the completeness of ZTRAX and OSM may be poor. The completeness of the ZTRAX land use records appears to decline from residential to commercial and industrial grades, giving residual values for all RUCCs of 0.77, 0.61 and 0.34 respectively. Reminder values across the RUC are similar to the correlation, showing the highest values for the housing class in an urban setting (RUCC 1, recall = 0.88) and the lowest values in a rural setting for the industrial class.
While these general patterns illustrate the large-scale consistency between the ZTRAX and OSM-based land use data, we observe strong local fluctuations in uncertainty at the county level as suggested by the distributions of Spearman’s rank correlation coefficients and of the reminder calculated at the county level (Fig. 3a, b) . As indicated by the top tails of these distributions, there are a significant number of counties that show very high Type II quantitative and positional agreement between ZTRAX and OSM. When analyzing the distribution of poviat-level contract indices in the rural-city continuum, we note that while the overall contract indices in Table 4 decrease from urban to rural regions, this trend is less pronounced for poviat-level indices (Fig. 3c, d). . For example, a significant number of rural counties (RUCC 6-9) show high recall values for residential and commercial land use classes. We would like to emphasize that various factors such as spatial, temporal and semantic inconsistencies between the ZTRAX and OSM data, as well as the user generated nature of the OSM database and related uncertainty issues, influence the presented contract evaluation results, highlighting the difficulty of validating the data on land use in general. However, these results suggest that the areas representing modern land use are largely consistent with the independently collected and compiled OSM data and therefore provide a reliable and reliable approximation of the land use distribution in most CONUS regions.
Comparison to remote-sensing-based LULC datasets
We used grid NLCD land cover data for the years 200169 and 201670, as well as the grid LCZ urban land use data (temporarily referenced from 2016-2018) available for CONUS71. We implemented two approaches to this comparison: First, we implemented a record-based approach: we drew a layered random sample of the ZTRAX records, we took the land use zone / climate zone from the underlying NLCD and LCZ grids at the location of each record, and a cross-tabulation of the land use class of each ZTRAX record and the appropriate NLCD and LCZ grade labels located in the appropriate location. In particular, we randomly selected one county for each of the nine RUCCs, in each of the nine census divisions of US72,73. We then searched for ZTRAX property records in these counties and sampled n = 1000 records (with swap) from each of our six land use classes (see Table 1) per county. In this way, we obtained a sample of N = 486,000 ZTRAX records, located in 81 US counties evenly distributed throughout CONUS and on the rural-urban continuum and in equal proportions in the land use classes used here.
Second, we implemented a raster-based approach: we reduced the sampling of NLCD mesh surfaces from their native resolution of 30m and the LCZ data from 100m to a 250m HISDAC mesh using two resampling techniques: 1) Most Area Rule, and 2) Using 1-hot coding, i.e. creating a binary lattice surface of 250 µm for each NLCD and LCZ class, encoding the presence of each class with 1 and the absence with 0. In this way, we were able to evaluate the correspondence of our land use classes also with the underrepresented classes in NLCD and LCZ, which are likely to disappear when resampling with the majority area. Similarly, we created a binary area in our 250m grid, indicating the presence (1) or the absence of (0) of any of our six land-use classes, based on the property-counting surfaces for a specific land-use (see Figure 2). We compared these data layers by comparing our binary area based on land use with the binary areas of each LULC class with NLCD and LCZ, respectively.
We compared the 2016 NLCD and the LCZ with our 2016 tiers, and to minimize the effects of time inconsistencies, we compared the 2001 NLCD with our 2000 tiers. These different strategies (record-based and raster) allowed us to obtain a relatively objective a picture of the relationship between our land use classes and remotely detected LULC types. The record-based approach assesses the match between ZTRAX and LULC datasets without being affected by additional uncertainty due to resampling. However, it only evaluates the thematic agreement where ZTRAX records are available, disregarding skip errors. The raster approach may suffer from additional position uncertainty due to resampling from 30m (NLCD) and 100m (LCZ) to a target resolution of 250m, but allows quantification of class specific skip errors in regions where there are no records ZTRAX are available.
The record comparison (Figure 4, top) revealed very similar patterns for NLCD 2001 and NLCD 2016: The highest percentages of income and ZTRAX housing record hold are in the NLCD grades ‘Developed Low Intensity’ and ‘Developed Medium Intensity’, while industrial and commercial land use has the highest proportions in “developed, high intensity”, particularly in urban counties. Agricultural properties have the largest share in “pasture / hay” and “arable crops”. Compared to the local climate zones (Fig. 4, lower part), it can be seen that the highest shares of ZTRAX records for most land use classes are in the “open low development” class, with the exception of the agricultural land use class, which peaks in the LCZ class ” Low plants ”and“ Dense trees ”.
Record comparison of ZTRAX land use classes and LULC land use categories, for the full sample, for rural (RUCC 6-9) and urban (RUCC 1-5) counties. Values are shown in% of sample N = 486,000 of ZTRAX records used.
Some of these cross-tabulations appear improbable, such as ZTRAX records located in wetlands or open water. These are likely artifacts due to the resampling and spatial resolution of the HISDAC-US land use data layers. In the least optimistic scenario, we can consider these mismatches as commission errors (ie ZTRAX reports built structures that do not exist). In such a case, these commission errors quantified by the cross-comparison performed would only account for 4-5% of all ZTRAX records. At this point it is worth noting that commission errors in ZTRAX can also occur due to demolished buildings not removed or updated (ie set as “empty” land use) in ZTRAX. However, these cases are unlikely to exceed 1-2% of all ZTRAX records (see Section 4.4).
The raster approach reveals a complementary picture. As shown in Table 5, for most uninhabited and vegetation-dominated NLCD classes, only small proportions of area are covered by the grid cells containing one or more ZTRAX records. This trend is reversed for settlement-related land cover classes: For example, more than 82% of the ‘low intensity built up’ land in 2016 is geographically in line with the land use data described here. This agreement is lower for NLCD 2001 data because grid cells without time information count as “% not covered by HISDAC”. This trend continues with two different data resampling techniques. Larger differences in these proportions between majority-based resampling and 1-hot encoding indicate that the land cover classes (eg, “Developed, low intensity”) are underrepresented and / or spatially dispersed and therefore disappear with majority-based resampling. Moreover, by distinguishing between these cross tables into the proportions of HISDAC covered and non-covered area (Table 5, lower part), we have observed that the highest proportion of the vacant area is scrub / scrub (i.e. 23%) and the highest proportion of HISDAC areas is in ‘forest deciduous ”and“ pasture / hay ”(agricultural class, cf. fig. 4), and then in developed classes.
Raster-based crosstabs with LCZ classes follow a similar pattern: most settlement and built-up classes (e.g. compact and open skyscrapers etc.) are covered by HISDAC-US (Table 6, left), while most LCZ classes dominated by vegetation is not included. However, we observed some exceptions that deviated from this trend: for example, the ‘Open midrise’ class is largely not included in HISDAC-US. The reason may be public buildings, which are omitted from HISDAC-US24 and not included in the land use data presented here. Moreover, only 10-12% of the grid cells labeled “heavy industry” are covered by HISDAC-US. This may be due to spatial shifts as industrially used parcels can be very large, but also indicates relatively poor industrial land-use cover, which is also in line with the observations made when comparing our data with OSM (section 4.2). Conversely, the highest percentage of the area covered by HISDAC-US in LCZ is classified as “open low buildings”, “dense trees” or “low plants”. This is likely because dense urban settlements make up only a small fraction of the built environment in the United States, and suburban and rural settlements, as well as agricultural structures are typically spatially dispersed and thus re-sampling more than a single grid cell. developed properties in densely built-up urban areas.
It is worth noting that due to different properties of the compared data (i.e. discrete locations compared to categorical and density information contained on mesh surfaces), the position uncertainty in both LULC data (e.g. caused by the registration accuracy of the remoting data substrate) and in the ZTRAX data ( e.g. using plot centroids or address points instead of the location of actual built-up structures) may introduce additional uncertainty in these cross comparisons. However, it is assumed that aggregating the NLCD and LCZ datasets from fine resolutions to a target resolution of 250 µm will partially alleviate this burden.
Assessing survivorship bias in the historical data
The survival bias is a problem that occurs in several disciplines and is of particular importance for most types of settlement, land use, or building stock data74,75,76,77,78. This type of error occurs when entities such as a built structure are removed from the population but are not included in the data. For example, a structure built in 1930 may be rebuilt or may be demolished over time79. ZTRAX does not take directly demolished structures into account and therefore does not represent structures that no longer exist. The land use data product described has the same limitation as only surviving buildings are considered without taking into account possible structural losses. To demonstrate and measure the impact of this Colorado survival error, we used address-level demolition data over a 10-year period (2008-2017) obtained from the Colorado State Archives (https://spl.cde.state.co.us/artemis / heserials / he171017internet /).
We stratified the counties in Colorado according to their RUCC and found that demolitions were happening in the urban counties at a rate more than 10 times faster than in the rural counties; out of 28,403 possible demolitions, 26,011 took place in rather urban counties (ie RUCC 1–5 designations). We grouped RUCC 1-5 as city counties and RUCC 6-9 as rural for all analyzes using RUC codes. By comparing the total number of demolitions occurring in 2008-2017 with the total number of structures built in 2015, we estimate that approximately 1.1% of Colorado’s building stock (thus an annual average rate of 0.11%) was demolished during this period. On the poviat scale, it was found, both for rural and urban poviats, that the maximum percentage of demolished building stocks did not exceed 2.5% in a 10-year period. As mentioned earlier, this observation also provides an estimate of the upper limit of potential commission error in ZTRAX and derived land use datasets: There may be cases where demolished buildings are not rebuilt and the demolition is not updated in ZTRAX leading to a false positive (i.e. commission error). In addition, we refer to Uhl et al. (2021), in which the commission errors of ZTRAX-based settlement strata were quantified and a high level of precision was observed in contemporary urban conditions, dropping to about 0.7 in rural areas and in the early periods of time.
In addition, we matched the demolition records to the ZTRAX records based on the address information provided in both datasets and assessed the relationship between the demolition year and the year recorded in the ZTRAX, separately for urban and rural counties (Fig. 5). The scraped-off data from the demolition contained a total of 33,645 addresses, of which we were able to match 28,403 records to ZTRAX data, leaving 5,242 records unmatched. These mismatched records may represent structures that have completely disappeared and are not included in the ZTRAX data, or are an artifact of our address-based alignment process, and therefore may be prone to spelling errors in addresses. We have observed many cases in scraped demolition data that was misspelled or had inconsistent formatting. From the analysis of these data, we observed the following: (1) Most buildings that have been demolished do not have a valid annual construction attribute, and this percentage is higher in rural than urban areas. This indicates that the missing attributes built during the year may be the result of recent demolitions and possibly reconstruction, and reporting delays that appear to be higher in rural counties. (2) Only a small percentage of buildings demolished (around 10% in 2008) are built annually & gt; = Year of demolition. These records represent remodeling activities likely to result in replacement of a built in the previous year and represent the survival bias in building age information derived from ZTRAX. (3) About 20% in 2008 and 50% in 2016 demolitions failed to update the record construction year in ZTRAX (construction year
Comparison of ZTRAX records and Colorado building demolition records. The bar graphs show the proportions of demolished buildings in different categories determined by comparing the year of demolition and the year built in ZTRAX, separately for ZTRAX records reported as undeveloped and undeveloped. Urban counties have RUCC 1-5, rural counties have RUCC 6-9.
In conclusion, while we recognize the uncertainty due to the survival error contained in our data and generated by our modeling approach, it is clear that even in the most conservative scenarios using verifiable data, the survival error would have a minimal impact on the analytical results. As described above, we assume that the intended use of the land for construction was designated at the time the construction was constructed, as the building documents / permits were submitted to the county appraiser at that time. Therefore, some uncertainty remains unresolved if the buildings were built on plots of land that had changed and therefore their land use may have changed at some point. However, different types of land-use change over time have different transition probabilities. We illustrate this in Table 7 to provide the basis for identifying the land use classes that may be susceptible to this type of thematic uncertainty if no past land use change has been recorded in the database.
Qualitative comparison to overhead imagery
Finally, we used Bing aerial imagery (https://www.arcgis.com/home/item.html?id=8651e4d585654f6b955564efe44d04e5) to qualitatively assess the relationship between broad-scale earth surface patterns in Earth observation data from remote sensing and the different land use classes recorded in ZTRAX. To this end, we randomly selected one location for each land use class for each of the 3,019 counties covered, for a total of 18,114 sample locations. We then obtained RGB Bing images in a 100 × 100 meter bounding box around each ZTRAX location and generated a mosaic of these individual images by land use class. These mosaics are based on the method proposed and used by Uhl et al .80 that computes the color moments81 for each image, resulting in a 12-dimensional low-level descriptor summarizing the color content of each image. We then use the t-distributed stochastic embedding of neighbors (T-SNE) 82 to map these 12-dimensional descriptors to two-dimensional space. The T-SNE arranges the data points in a two-dimensional target space such that similar data points are close together. We then corrected the obtained two-dimensional point cloud and visualized each image at the appropriate location in the t-SNE space. This method groups similar images together and allows for the integrated visual evaluation of large numbers of images (Fig. 6). These visualizations characterize the widespread geographic context patterns found at ZTRAX locations across land use classes. For example, they illustrate the amount of vegetation-dominated habitats that are most common in the agricultural land use class. Small buildings are commonly found in agricultural and residential land use classes. Large bright objects represent (common e flat) roofs of large industrial, commercial or leisure facilities and appear to be common in the locations of these three land use classes. Remember that vegetation-only images are likely to be visible due to positional shifts of ZTRAX point locations from actual building locations within rural (often larger) cadastral plots50. However, we can assume that these shifts have little effect on the accuracy of the data due to the chosen spatial resolution of 250 × 250 m, as suggested by recent multi-scale accuracy assessments24. Thus, a visual inspection of the t-SNE plots in Fig. 6 reveals likely matches for most sample locations. This technique can be used to systematically refine samples on a larger scale to verify the building level for quantitative accuracy assessments, as long as the building function can be inferred from the photos from above.
Visual assessment of Bing advance images collected at the locations of a layered random sample of ZTRAX records for the six land use classes used in this document. (a) residential (ownership), (b) residential (income), (c) commercial, (d) industrial, (e) agricultural and (f) recreational. The images collected at each location in each land use class are ordered based on their color similarity, using color moments and t-decomposed stochastic neighbor transformation (t-SNE). The small spots to the right of each mosaic show sample magnifications, providing further details on the building characteristics of each ZTRAX location. The yellow rectangles show the location of the magnification (the upper magnification corresponds to the upper of the two rectangles per land use class.
In this analytical effort, we used OpenStreetMap, demolition data records, remote sensing land cover classification, and upfront images as the comparative data sources, being aware that none of these external data sources represented an optimal ground truth to assess the quality of the layer’s land use formed. While these comparisons do not quantify uncertainties in historical land use data, they highlight important aspects of data quality and properties that help to better understand the completeness and inherent bias in the data product.
Usage Notes
In the previous chapters we have described our efforts to quantify some systematic errors that are present in land use data. However, there are a few other limitations that the user should consider when using the land use datasets in the grid. First, ZTRAX relies heavily on county records to fill in land use attributes, and county reporting practices vary from site to site, which may not take into account all existing buildings. Likewise, the implemented land use classification procedure may vary from county to county, introducing some uncertainty regarding the type of building. We tried to alleviate this uncertainty by grouping the 300+ land use types into broad thematic classes, e.g. commercial or residential. A significant limitation of this dataset is due to the collection methods used to build the ZTRAX database; Public buildings such as universities and low income housing are generally not represented in the data presented here. The land use data in the grid will therefore typically characterize privately owned structures. We recommend that users integrate open source data to capture the presence of public buildings in your area to mitigate the error caused by excluding these buildings. Finally, we emphasize that these data were limited to land use by physical objects and thematic classes that were identified in the literature as relevant to urban development. Therefore, the data excludes land use which is not related to structure, e.g. arable land or pasture, and the data excludes other potentially important land use classifications such as tax exempt or b government structures. As there has historically been a lack of data that could directly describe the structural land use in built-up areas, the design of this data product deliberately gives preference to land use classes identified as drivers of urban development at the expense of other potentially important land uses. Users should be aware of this inherent systematic error and we encourage them to use layers of uncertainty to estimate the number of excluded structures in the research area under consideration.
Code availability
The ZTRAX dataset was stored in relational databases using Safe Software Feature Manipulation Engine (FME) (https://www.safe.com/). The code for this pipe is available at https://github.com/johannesuhl/ztrax2sqlite2csv.
References
Homer, C.G. and others National Land Cover 2001 database completed for the continental United States. Photo Eng. Remote Sens. 73, 337-341 (2007).
Bhaduri, B., Bright, E., Coleman, P. & amp; Urban, ML Landscan USA: A High Resolution Geospatial and Temporal Modeling Approach for Population Distribution and Dynamics. GeoJournal. 69 (1-2), 103-117, https://doi.org/10.1007/s10708-007-9105-9 (2007).
Article
Google Scholar
Nowak, DJ & amp; Greenfield, EJ Assessing Tree Crowns in the National Land Cover Database and Impermeable Ground Cover Estimates in the neighboring United States: Comparison with the estimates interpreted in the photos. J. Environmental management. 46 (3), 378-390, https://doi.org/10.1007/s00267-010-9536-9 (2010).
ADS
Article
Google Scholar
Wickham, J.D., Stehman, S.V., Fry, J.A., Smith, J.H. Homer, C.G. Thematic Accuracy of NLCD 2001 Land Cover for the Contiguous United States. Remote ambient sensor. 114 (6), 1286-1296, https://doi.org/10.1016/j.rse.2010.01.18 (2010).
ADS
Article
Google Scholar
Yang, C., Raskin, R., Goodchild, M. & amp; Gahegan, M. Cyberinfrastructure geospatial: past, present and future. Computer. Surround. Syst. 34 (4), 264–277, https://doi-org.colorado.idm.oclc.org/10.1016/j.compenvurbsys.2010.04.001 (2010).
Article
Google Scholar
Sengupta, A., Lemmen, C., Devos, W., Bandyopadhyay, D., Van der Veen, A. Construct a seamless digital cadastral database using colonial cadastral maps and images from an Indian perspective. Surv. Rev. 48 (349), https://doi.org/10.1179/1752270615Y.0000000003 (2016).
Dong, N., Yang, X., Cai, H. & amp; Xu, F. Research on matching grid size to population distribution in networks in an urban area: a case study in the urban area of Xuanzhou District in China. PLoS One. 12 (1), e0170830, https://doi.org/10.1371/journal.pone.0170830 (2017).
CAS
Article
PubMed
PubMed Center
Google Scholar
Trepal, D., Lafreniere, D. & amp; Gilliland, J. Historical infrastructures for spatial data for archeology: towards a big-data spatial-temporal approach to the study of a post-industrial city. inside J.Hist. Archeol. 54 (2), 424–452, https://doi.org/10.1007/s41636-020-00245-5 (2020).
Article
Google Scholar
Hosseini K., McDonough K., Van Strien D., Vane O. & amp; Wilson, DC Maps of the nation? Digitized weapon review for new historical research. J. Vic. Cult. 26 (2), 284-299, https://doi.org/10.1093/jvcult/vcab009 (2021).
Article
Google Scholar
Vogelmann, J.E. and others The 1990s National Land Cover dataset for the contiguous US was completed based on LandSat thematic map data and auxiliary data sources. Photo Eng. Remote Sens. 67, 650-662 (2001).
Homer, C. et al. The 2011 National Land Cover database for the continental United States – representing a decade of information on land cover changes – has been completed. Photogram. Eng. Rem. S. 11 (2015).
Pesaresi, M. et al. Global layer of human settlements based on optical HR / VHR RS data: concept and first results. IEEE J. Sel. Top. Regret. Earth Obs. Remote Sens. 6 (5), 2102-2131, https://doi.org/10.1109/JSTARS.2013.2271445 (2013).
ADS
Article
Google Scholar
Sohl, T.L. and others Spatial explicit modeling of land cover and age of stands in 1992–2100 for the neighboring United States. Ek. Regret. 24, 1015-1036, https://doi.org/10.1890/13-1245.1 (2014).
Article
PubMed
Google Scholar
Klein-Goldewijk, K., Beusen, A., Doelman, J. & amp; Stehfest, E. Anthropogenic land use estimates for the Holocene – HYDE 3.2. Earth Syst. Science. Data. 9, 927-953, https://doi.org/10.5194/essd-9-927-2017 (2017).
ADS
Article
Google Scholar
Kwon, Y.-B., Ogier, JM Graphics Recognition. New trends and challenges: 9th International Workshop, GREC 2011, Seoul, Korea, September 15-16, 2011 Revised Selected Papers (Springer-Verlag Berlin Heidelberg, 2013).
Heitzler, M. & amp; Hurni, L. Cartographic reconstruction of building traces from historical maps: elaboration on the Swiss map of Siegfried. Trans. GIS. 24 (2), 442–461, https://doi.org/10.1111/tgis.12610 (2020).
Article
Google Scholar
Uhl, J.H., Leyk, S., Chiang, Y.Y., Duan, W. & amp; Knoblock, CA Automated extraction of human settlement patterns from historical series of topographic maps using poorly supervised convolutional neural networks. IEEE Access. 8, 6978–6996, https://doi.org/10.1109/ACCESS.2019.2963213 (2020).
Article
Google Scholar
Reba, ML, Reitsma, F. & amp; Seto, KC Spatializing 6000 years of global urbanization from 3700 BC to AD 2000. Sci. Data. 3, 160034, https://doi.org/10.1038/sdata.2016.34 (2016).
Article
PubMed
PubMed Center
Google Scholar
Kaim D., Szwagrzyk M., Dobosz M., Troll M. & amp; Ostafin, K. Location of buildings from the mid-19th century in Galicia and Austrian Silesia under the rule of the Habsburg monarchy. Earth Syst. Science. Data. 13, 1693–1709, https://doi.org/10.5194/essd-13-1693-2021 (2021).
ADS
Article
Google Scholar
Lieskovský, J. et al. Historical land use data collection in the Carpathian region (1819–1980). J. Maps. 14 (2), 644-651, https://doi-org.colorado.idm.oclc.org/10.1080/17445647.2018.1502099 (2018).
Article
Google Scholar
K. Ostafin, D. Kaim, T. Siwek & amp; Miklar, A. Collection of historical data of administrative units with socio-economic features for Austrian Silesia 1837-1910. Science. Data. 7 (1), 1-14, https://doi.org/10.1038/s41597-020-0546-z (2020).
Article
Google Scholar
Zillow Group, Inc. Zillow Transaction and Assessment Dataset (ZTRAX). Available online. https://www.zillow.com/ztrax/ (2021).
Leyk, S. & amp; Uhl, JH HISDAC-US, Compilation of Historical Billing Data for the Continental US over 200 years. Science. Data. 5, 180175, https://doi.org/10.1038/sdata.2018.175 (2018).
Article
PubMed
PubMed Center
Google Scholar
Uhl, J.H. and others Fine-grained space-time datasets measuring 200 years of land development in the United States. Earth Syst. Science. Data. 13, 119–153, https://doi.org/10.5194/essd-13-119-2021 (2021a).
ADS
Article
PubMed
PubMed Center
Google Scholar
Leyk, S. et al. Two centuries of settlement and urban development in the United States. Science. Appr. 6 (23), eaba2937, https://doi.org/10.1126/sciadv.aba2937 (2020).
ADS
Article
PubMed
PubMed Center
Google Scholar
Boeing, G. Off the grid… and back again ?: the latest evolution in American street grid planning and design. J. Am. Schedule. dr hab. 87 (1), 123–137, https://doi.org/10.1080/01944363.2020.1819382(2021) (2021).
Article
Google Scholar
Uhl, J.H., Connor, D.S., Leyk, S. & amp; Braswell, A.E. The Age of Separation of the Size and Structure of Urban Spaces in the United States. Communion. Earth. Surround. 2 (1), 1-14, https://doi.org/10.1038/s43247-020-00082-7 (2021b).
Article
Google Scholar
McDonald, R.I. and others Tree coverage and temperature disparity in urban areas in the US: quantify the relationship with incomes in 5,723 communities. PLoS One. 16 (4), e0249715, https://doi.org/10.1371/journal.pone.0249715 (2021).
CAS
Article
PubMed
PubMed Center
Google Scholar
Salazar-Miranda, A. Micropermanence of Layouts and Designs: Quasi-Experimental Evidence from an American Housing Corporation. Reg. Science. Econo. 103755, https://doi.org/10.1016/j.regsciurbeco.2021.103755 (2021).
Li, X. et al. Global urban development 1870–2100 with integrated high-resolution mapped data and dynamic city models. Communion. Earth. Surround. 2 (1), 1-10, https://doi.org/10.1038/s43247-021-00273-w (2021).
ADS
Article
Google Scholar
Dornbierer, J., Wika, S., Robison, C., Rouze, G. & amp; Sohl, T. Prototyping a Long-Term (1680-2100) Historical to Future Landscape Modeling Methodology for the Continental United States. Ground. 10 (5), 536, https://doi.org/10.3390/land10050536 (2021).
Article
Google Scholar
Millard-Ball, A. Width and Value of Residential Streets. J. Am. Schedule. dr hab. 88 (1), 30-43, https://doi-org.colorado.idm.oclc.org/10.1080/01944363.2021.1903973 (2022).
Article
Google Scholar
Wan, H., Yoon, J., Srikrishnan, V., Daniel, B. & amp; Judi, D. Population reduction using high-resolution time-rich US real estate data. Kartogr. geographic Info. Science. 49 (1), 1-14, https://doi-org.colorado.idm.oclc.org/10.1080/15230406.2021.1991479 (2021).
Mietkiewicz, N. et al. In the Line of Fire: Implications of Human Fires for US Homes (1992–2015). Fire. 3 (3), 50, https://doi.org/10.3390/fire3030050 (2020).
Article
Google Scholar
Iglesias, V. et al. Risky Development: Increasing Exposure to Natural Hazards in the United States. The future of the Earth. 9 (7), e2020EF001795, https://doi.org/10.1029/2020EF001795 (2021).
ADS
Acknowledgements
Article
Author information
Authors and Affiliations
PubMed
PubMed Center
Google Scholar
Bernstein, A., Gustafson, MT & amp; Lewis, R. Catastrophe on the Horizon: Price Effect of Sea Level Rise. J. Finance. Econo. 134, 253–272, https://doi-org.colorado.idm.oclc.org/10.1016/j.jfineco.2019.03.013 (2019).
Contributions
Article
Corresponding authors
Ethics declarations
Competing interests
Google Scholar
Additional information
Boslett, A. & amp; Hill, E. Shale Gas Transmission Prices and Housing. Resource. Energy. Econo. 57, 36-50, https://doi-org.colorado.idm.oclc.org/10.1016/j.reseneeco.2019.02.001 (2019).
Rights and permissions
Article
About this article
Cite this article
Google Scholar
