ArcGIS Pro Lesson 3: Methods of Visualizing Temporal Data

This lesson is undergoing open peer review. Please use the comment section to review the lesson and highlight any areas that require more clarity or could otherwise use improvement. Contact Jim Clifford if you have any further questions about the process: jim.clifford@usask.ca

By Nolan Kressin and Joshua MacFadyen, University of Prince Edward Island 

October 7, 2020

This lesson will teach skills to help manage and visualize temporal datasets in ArcGIS Pro. The work focuses on the tools available in ArcGIS Pro, but similar tools are available in QGIS, and we plan to release a parallel lesson on that application soon.

For more information about how historians visualize temporal data, and why we chose to use this method, see the blog post by MacFadyen and Kressin “The Fir Trade in Canada” on NiCHE: Network in Canadian History & Environment.

Table of Contents

Introduction
Before We Begin: Understanding and Downloading the Lesson Data
Step 1: Adding Spatial Data to ArcGIS Pro
Step 2: Setting up the GIS to Join Historical Data
Step 3: Joining the Historical Firewood Statistics to the GIS
Step 4: Symbolizing Attribute Data
Step 5: QAQC: Correcting Historical Inaccuracies
Step 6: Creating an Exportable Map Layout
Step 7: Creating a Time Series

Introduction

In this article, we explain the process we followed to create an animated map of a specific commodity as it was hauled around all historical railways in Eastern Canada. We needed a firewood freight map that would help shed light on both the geographic distribution of biomass energy production and consumption and the temporal trends during a period of intensive economic growth (and hence energy expansion). We know that seems like a rather specific research question, but we argue that others could follow a similar process to map the movements of any commodity or passenger traffic on historical railways or any other routes (eg road or water transport) over time. The method we cover in the tutorial is a relatively new feature in Geographic Information Systems known as time mapping, or temporal GIS.

Before We Begin: Understanding and Downloading the Lesson Data

Data Development

The research required three general categories of sources, historical railway/freight statistics, historical railway GIS data, and general railway histories. We found most of the historical statistics in the Sessional Papers. The annual reports of the Canadian Minister of Railways and Canals usually contained a “Summary Statement of Description of Freight Carried” by railways in the previous year. These tables were also reproduced in the Canada Year Book, and in the later years (1900s), these were the only historical sources available to us. The historical railway/freight statistics data were reported inconsistently in both time and space. We were missing key dates in the 1890s because officials reported each company’s freight data in an aggregated form. We were also missing some spaces because certain railways existed but did not report their data accurately, or at all, in some years. 

The historical railway GIS data, a collection of line segments representing railways built between 1836-1922, were supplied by Byron Moldofsky and Marcel Fortin of the Canadian Historical GIS Partnership. They are available here in GIS-ready shapefile format. However, even these require data development, technical knowledge, and collaboration to use them to their full potential. Finally, general railway histories were required to provide historical context and to fill some of the gaps that appeared when trying to match the historical statistics to the historical GIS data. Filling these gaps requires extensive research in railway histories and other secondary sources such as Christopher Andreae’s Lines of Country (1997). In the cases where no match could be found, we had to decide whether the railways existed but failed to report their data, or whether the historical GIS files were incorrect. This is explained in the difference between “grey and white” lines in the tutorial, below.

Finding Spatial Data

Screenshot Caption: Downloading the zipped folder from the linked Google Drive page. Click image to enlarge.

First, we will import the data we need for our project. The historical statistical data can be downloaded in a zipped folder here, and the historical GIS railway files are available on dataverse here in GIS-ready shapefile format. In the zipped folder, you will find an Excel file titled “Cordwood in Sessional Papers of Canada 1876-1890, Firewood for Lesson3”, the Documentation_CordwoodSessionalPapersCanada1876-1903.rtf file, and the Documentation_HR_rails_NEW.doc file. In the folder you download from dataverse, you will find a number of zipped files (including a shapefile titled “HR_rails_new.shp”), as well as a .dbf titled “hr_codes”.

  • Download the zipped folder ArcGIS_Lesson3 to your computer’s Downloads folder.
  • Navigate to your computer’s Downloads folder, right-click the zipped folder and click Extract all…. 
  • Click Browse. 
  • Navigate to C:\Users\<username>\ArcGIS, and click Select Folder. 
  • Under “Files will be extracted to this folder,” it should now say C:\Users\<username>\ArcGIS.
  • Click Extract.
  • After you extract the zipped folder, the file path should be C:\Users\<username>\ArcGIS\ArcGIS_Lesson3
  • Add the folder of historical GIS railway files that you downloaded (it should be called dataverse_files) from Dataverse to the same ArcGIS_Lesson3 folder you just created

Key Concept: temporal GIS. As ESRI’s guide to “Working with Temporal Data in ArcGIS” explains, there are at least four types of temporal GIS data. Moving features are those that move over space, like a tornado or hurricane where the centre is often in a different location every hour. Discrete events can contain data about features that usually happen in a single place and time, like a car accident. Stationary features contain data about a single place over time, such as a sensor, and change/growth features represent a feature that changes in size, such as a forest fire, or in the case of this exercise, railways.

Step 1: Adding Spatial Data to ArcGIS Pro

Note: save your work in ArcGIS Pro early and often to avoid data loss if the program unexpectedly crashes. 

Open ArcGIS Pro (at the time of writing we were using ArcGIS Pro 2.5)

  • select New, and create a new project. 
  • Under the New heading, click Map
  • We recommend navigating to the same folder, as above (C:\Users\<username>\ArcGIS\ArcGIS_Lesson3\ProjectFiles) 
  • We recommend naming your “Lesson#3_VisualizingData_Date” where Date is the current date in YY-MM-DD format, e.g 20-10-19. 
  • Select map on the ribbon > Add Data > Navigate to wherever the data has been stored (C:\Users\<username>\ArcGIS\ArcGIS_Lesson3\dataverse_files)
  • Select HR_rails_new.shp > Click OK
  • Once the shapefile has been added to the map, it should now appear in the Contents pane as a layer. Right-click the layer and hit “Zoom to layer.” 
  • You should now see the extent of the shapefile on your screen!

Historical railway GIS data

Before you start modifying the dataset, explore it first. 

  • Open the ArcGIS Lesson 3 folder containing the downloaded data and documentation. Navigate to the file named “Documentation_HR_rails_NEW.doc”. This file contains the data about our data. It essentially explains all the core aspects of what our data is; such as a short explanation of the data, who made it, when did they make it, and so forth. Documentation is a very helpful resource, and you should always read it (assuming it has been given) before starting your work to fully understand the data you are working with.
  • Short explanation of the historical railway GIS data: This dataset is a compilation of railway lines from 1836 to 1990, sourced from the Historical Atlas of Canada project and modified by ESRI Canada. The data is composed of numerous line segments representing railways spread throughout Canada , each with unique attribute information. To view these unique attributes, we will have to open the attribute table of the HR_rails_NEW.shp file. To do this, right click the layer titled “HR_rails_NEW” in the Contents pane and select the Attribute table option.
  • A table should now be open on your screen, in ArcGIS Pro. The default location is at the bottom of the project, but these table preview windows can be moved around.
  • Each row is an individual feature (i.e a unique railway segment) on your screen. Each column describes information about each segment. These variables include:
    • FID: The unique identifier for each railway segment. ArcGIS Pro produces this unique FID number for each railway segment. This is helpful for organizational purposes, as we can identify a segment of railway by its number in the FID column.
    • Shape: The type of vector geometry of the feature. These should all be polyline. Polylines are lines with multiple vertices, hence they can travel in multiple pathways rather than just straight. 
    • BLDR_Code and INCRP_Code: These codes describe the railway companies that either incorporated or built this railway segment. An incorporated railway means that the railway was purchased or leased to another company. INCRP_Code is the railway company that originally built the railway. BLDR_Code is the railway company that incorporated the railway, if it was incorporated at all. If not, then both BLDR and INCRP codes will be the same.  
      • The first digit in the code i.e 1,2,3 describes the railway’s “parent” company. The parent company of the railway is the major company that eventually incorporated that railway segment into its own system. These major railways have single-digit codes only. These single digits should match to the first digit of the code you are looking at. For instance, any railway that starts with the digit “1” was eventually incorporated into the “1” railway system. You can see which major railway was the “1” in the zipped folder.
      • Add the hr_codes.dbf file to ArcGIS Pro. Simply find the file in the Windows File Explorer, and select and drag it into the ArcGIS Pro map. This should add a table under the “Standalone Tables” section at the bottom of the Contents pane.

  • Open the table by right clicking it in the Contents pane and selecting Open.
  • hr_codes is a very helpful resource as it matches the codes found in INCRP, BLDR and HACNo with the actual historical names for each railway found in the shapefile. For example, we can see the number/code “1” has the company name “Canadian Pacific”.
  • The “hr_codes” document contains the codes and names for all railways that appear in the HR_rails_NEW shapefile. Use it often! 
  • You may be wondering what the distinction is between BLDR_Code, and the written code, if they both describe who incorporated the railway. A good example of where this distinction is important is at FID #1732. In this railway feature, the INCRP_Code is 12Z, and the BLDR_Code is 12G.
  • Referring back to the hr_codes spreadsheet, the 12Z railway is named “Richelieu, Drummond & Arthabaska Ry Co”
  • 12G is named “South Eastern Counties Jct Ry Co“.
  • This means that the Richelieu railway was the original construction, and was incorporated into the South Eastern at some point in time. Then, the South Eastern was eventually incorporated into the Canadian Pacific, hence the actual code starts with “1”.  This can be further confirmed through some minute historical research: https://en.wikipedia.org/wiki/South_Eastern_Railway_(Quebec)
  • The 2nd digit in the code refers to some other qualifier, like the geographic region of the railway, chosen arbitrarily. If the railway has a 2nd digit, it was its own separate company at some point, and was later incorporated into the parent company. The third part of the code is a letter. The letter adds further distinction to railway segments within a certain region. Usually, railways with the same first 2 digits in their code were all incorporated as a single system at one point in time. 
    • CNSTRCTD: The date the railway was constructed.
    • ABNDND : The date the railway went out of use. 
    • TRANSCAN, TC1917, TC1992: The values in this field are either 1 or 0. They describe whether the railway was part of the Trans-Canada line in 1917, 1992, and/or today.
    • Shape_Leng: This field contains the length of each line segment. 
    • There are other attributes that exist in the shapefile, but at the moment they are not relevant to the work at hand.

Historical railway/freight statistics

Next, open the .xlsx file “Cordwood in Sessional Papers of Canada 1876-1890, Firewood for Lesson3” from the ArcGIS Lesson 3 folder. This dataset contains historical information about the firewood hauled on Canadian railways selected from 1876 – 1919. This is the data that is going to be joined to our railway segments. Make sure to look closely at this data, and compare it to the attribute table of the shapefile we added into the project. What similarities do you see? What about differences? Re-open the zipped folder containing the downloaded data and documentation. Open the file named Documentation_CordwoodSessionalPapersCanada1876-1903.rtf. Within the documentation for the data, also take note of how the document was created and which sources it used. How comparable is this document to our railway data?

  • Short explanation of data: This excel file is a spreadsheet of historical information about the quantities of firewood hauled on Canadian railways between 1876 – 1903. It is sourced from the Sessional Papers of Canada (1876-1903) with additional data from the Canada Year Book (1907-1919). It should be separated into multiple tabs, with each tab containing the historical information for each year recorded for this tutorial.
  • Attribute data:
    • Number in Sessional Papers: The number assigned to the railway in the Sessional Papers for selected year. 
    • HAC_No: The best estimation of the equivalent railway and its code in the Historical Atlas of Canada. 
    • Name of Railway: Name of the railway in the Sessional Papers.
    • Cords Firewood: Amount of firewood hauled on the railway, in cords. A cord is a unit of volume equal to 128 cubic feet, or a pile of wood 4 ft by 4 ft by 8 ft. Sometimes this field has been calculated from the tonnes of firewood reported, based on a conversion. The conversion was usually determined by creating an average ratio (tonnes/cord) from years where both variables existed. 
    • Tons per cord: Calculated amount of tons/cord based on the reported numbers in the Sessional Papers.

Step 2: Setting up the GIS to Join Historical Data

This step follows another stage of data development. We are going to modify the railway dataset so we can work with it more easily when joining it with our historical firewood data. 

Note: First, make sure we are not modifying the original data file. It is a good idea to create a backup of the Historical_Railways_allYears_HAC file before working with it. 

Splitting the data

We are going to split our dataset into individual years to join with the spreadsheet. Working with smaller, less diverse datasets can be easier to manage, as they are much more simple to visualize and manipulate. 

  • To perform this split, we need to think of a way to select only the railways that existed within the selected years in our spreadsheet. As an exercise, think of how you could discern each railway in the shapefile so that we know if it existed in each selected year in our historical firewood statistics.
  • To perform the split, we are going to use the date in CNSTRCTD and the date in ABNDND. If the railway was constructed before or on our selected year, and abandoned after that, then we can assume that the railway most likely existed during our selected year! 
    • This logic can be further written as mathematical expressions, where if our sample year is 1876 for instance, we can then select all features with a value in CNSTRCTD <=1876, and where ABNDND > 1876. 

Cleaning the data

If you noticed when you were analysing the dataset, there are odd values such as “0”, “-99”, “-98” and “-97”  that also exist within these fields. These are known as placeholders, which are used by historians and data analysts to substitute as values when the data for something is unknown, or not applicable. We are going to have to change these values, so that the expressions we made above become applicable. 

Key Concept: There are often multiple ways to prepare and modify the GIS data. For instance in this process of splitting and cleaning the data, there is a manual process and a more automated process using a python script. 

Here is the process for running an automated process with a python script:

  • Open the attribute table for HR_rails_NEW 
  • Go to Fields -> Calculate
  • Field Name: ABNDND, then CNSTRCTD 
  • You will have to write a little in Python script into the Code Block

def replaceInt(number): // def is how we declare a program. replaceInt is the name of our program. (number) is the value being inputted into the program

 if number == 0 or number == -99: // number refers to the values in ABNDND or CNSTRCTD field . If statement, if the value within our field is 0 or -99, then we replace with a 9999. 

 return 9999

else: // else statement that always proceeds an if statement. If there is no 0 or -99 value, replace the value with the same value 

return number

  • Under FIELDNAME= , put replaceInt(!FIELDNAME!) . Remember that replaceInt is the name of the program, and the field in the brackets are the values being inputted into the program
  • Now you can run the tool. Check the Attribute Table afterwards to verify that the values have changed so that all the placeholder values (eg. -99) have been removed from the dataset. 
  • Here is the process for manually preparing the data for a join:
    • Navigate to the ribbon > Map > Selection > Select by Attributes and choose the following values:
      • Input Rows: HR_rails_NEW
      • Selection type: New selection
      • Expression: Where CNSTRCTD is equal -99
      • Or Where …  Fill out for all placeholder values
  • Fill out for all placeholder values for CNSTRCTD and ABNDND 
  • Click Run
  • Open the attribute table for HR_rails_NEW 
  • Change the view to Show Selected Records
  • In the CNSTRCTD and ABNDND fields, manually click and change each value to 999. 

Completing the split

Back to the split, we are going to export the Historical_Railways_allYears_HAC shapefile into just the railways for each of our selected years. This will help us perform quality control on the data later on, after we have joined the firewood data to our railway data. 

  • Go to Map in the ribbon, and find Select by Attributes. Open the tool
  • Input Rows > Historical_Railways_allYears_HAC, Selection type > new selection > Expression > Remember the expressions we made earlier in Step 6. Using the expression builder, for the year you want to select, select all railways Where CNSTRCTD is less than or equal to Date, Add Clause, AND Where ABNDND is greater than Date.
  • Note: If the date you have selected isn’t discretely mentioned in the drop-down list, you can type it into the box directly. If you find that the tool is returning an “Expression invalid” error during execution, you can try changing the Expression to the SQL view. This can be done easily by clicking the SQL button on the top-right of the expression boxes. Write out the same expressions here that we practiced in the “Splitting the Data” section.
  • Once your features have been selected, right-click the HR_rails_NEW layer and navigate to Data > Export Features > Input Features: HR_rails_NEW > Output Location: The database we created for the lesson > Output Feature Class: SelectedYear_HAC
  • Repeat this process for all of the selected years. 

Step 3: Joining the Historical Firewood Statistics to the GIS

Preparing to Join Historical Data

Now we will start joining the historical statistics (the historical railway/freight statistics on firewood) to the historical GIS data (the historical railway shapefiles) in ArcGIS Pro. 

  • Recall the similarities between the data in the firewood spreadsheet and in our spreadsheet. To perform a join of our data, we need to identify a commonality between our two datasets. This commonality will be HAC_No and the INCRP and BLDR codes. 
    • We can select all the railways that existed during a certain time point, and then join the data based on their codes in HAC_No and INCRP_Code or BLDR_Code. 
  • Now that we understand what we need from our spreadsheet, we must do a little leg work to ensure that the spreadsheet is readable by ArcGIS. Note how the .xlsx document of firewood data is separated into multiple tabs for each date. For ArcGIS to more easily read our data, all our tabs should be combined into a single master sheet. 

Creating a Master Spreadsheet

To do this, we are going to compile our data from each tab into a single “master” tab to then create a Pivot Table. This is accomplished through the usage of a pivot table. A pivot table is a tool that helps compile unsorted data into a single table that can be more easily managed. 

  • If you opened the .xlsx file in Google Sheets, this can be most easily accomplished the following way:
  • Create a new tab in the spreadsheet by clicking the + sign on the bottom left of the sheet.
  • Name it “Master”. We will then fill in this sheet with data from each tab.
    • Again there are two ways to proceed: Automatic: If you are familiar with equations in Sheets, feel free to use formulae like QUERY to fill in the tab automatically. The format of the tab should be: Hac_No (column), Name of Railway (column), Cords of Firewood (column), Date (column) 
    • Manual: The most simple way to fill in the sheet is to copy and paste the data from each tab into the master.
      • Go into each tab in the spreadsheet, and copy the contents of the following columns: HACNo, Name of Railway, Cords Firewood. Paste them into the Master tab. Remember to not copy the headers of the columns into the Master, if they are already in there. Then, add in a column next to “Cords Firewood” titled “Year”. The Year column will be the date of the selected year you are copying from. Fill in this column with the selected year each time you copy/paste data from each tab into the Master. Repeat this step for every year in the spreadsheet. 
  • Once every year has been filled into the master tab, you can now create a pivot table. Highlight the data in the Master tab and go to Data > pivot table > Insert into new sheet. Call this new sheet “Master – Pivot Table”
  • Rows: HAC_No, Columns: Year, Values: Cords Firewood
  • Now you should have a compiled table of firewood on railways for every sample year in a single sheet. 
  • Download/Save As this version of the spreadsheet as a separate file.

Joining Historical Data

Now we will go back into ArcGIS and perform the actual Join process. First, we must select all railways within a certain year. 

  • Go to Map in the ribbon, and find Select by Attributes. Open the tool
  • Input Rows > HR_rails_NEW, Selection type > new selection , Expression > Remember the expressions we made earlier in Step 6. Using the expression builder, for the year you want to select, select all railways Where CNSTRCTD is less than or equal to Date, Add Clause, AND Where ABNDND is greater than Date. 
  • Click Run and now you should see some railways highlighted. The highlighted railways should only be the railways that exist during your selected year. To double check if your logic was correct, you can open the attribute table and select on the bottom left “Show Selected Records”, which will show you attribute information on only selected records. Make sure that there are no dates in the CNSTRCTD field that are higher than your selected date, and no dates in the ABNDND field that are lower than your selected date. 

Once you have all railways for a certain date selected and confirmed that they are valid, we can join the firewood data in our spreadsheet to those railways. 

  • Go to Analysis in the ribbon, Click on Tools, search for the Join Field tool 
    • Input table: HR_rails_NEW
    • Input Join Field: INCRP_Code 
    • Join Table: Spreadsheet > Select Master – Pivot table 
    • Output Join Field: HacNo
      • Note that for the Join tool to perform a Join, the input and output join fields must have matching values. If a railway in the GIS has an input join field of 1, and a railway in the spreadsheet has an output join field of 1, they are then joined by the GIS. 
    • Join Fields: Your Selected Date e.g 1876
    • Click Run.
    • The rows you have selected should now have a new field called 1876, with firewood data. 
  • Clear your selection after verifying, and repeat for every sample year

Make the Join Permanent

Once our firewood data is joined to the spreadsheet, we now can export each year as a separate shapefile. This process allows us to isolate only the railways that have data and look at them more closely. 

  • We will do another Select by Attributes. This time, select all railways for a certain date that are not null. This will return only the railways that have been joined to our firewood data, including railways hauling 0 cords of firewood. Run the tool with: Where 1876 is not null 
  • Once all railways have been selected, now right-click the layer. Go to Data > Export Features > Input Features: Our layer > Output location: geodatabase we made > Output feature class: title > Run 
  • A common error that can occur in this process is within the attribute field, is that the cord values of firewood become listed as Strings rather than numbers. One way to quickly verify if this has occurred or not is to  visually check the attribute table of the exported layer. In the values field, see if the values are oriented on the left or right side of the cell. If the values are oriented on the left sides of their respective cells, then they are actually Strings. If the values are oriented on the right sides of their respective cells, they are numbers. String values in your joined fields can be a problem, as ArcGIS does not read String values the same as numbers. When using Symbology later in the lesson, ArcGIS will not recognise the Strings and won’t be able to visualize the data. If you find your data has converted into a String value for whatever reason, there is a simple workflow that can be followed here (source: official esri support website):
  • Repeat this process for all of the selected years.

Step 4: Symbolizing Attribute Data

Key Concept: data symbology involves mapping the attribute data, changing layer parameters, and then mapping the attribute data again. Repeating this process can reveal spatial patterns.

Now that we have railways with firewood data for every year, we can modify the symbology of the layer to more easily visualize our data. 

  • Right-click the layer of your selected year > Symbology
  • A new pane should show up titled “Symbology”. In this pane, look for “Primary Symbology”. There should be a drop-down menu under this text with a list of options for which symbology to use to visualize the layer. For our railways, find and select the Graduated Colors option. Use the following values:
    • Field: We want to show the amount of firewood on the railways, so find and select the field containing firewood data for your selected year. 
    • Normalization: None
    • Method: The default should be “Natural Breaks (Jenks)”. Note: The Method field is where you can choose how you would like to create class breaks to represent your data. Class breaks are the points at which your data is divided. Establishing class breaks creates classes, which are ranges of data that are each assigned their own colour in the map features.There are multiple options for creating class breaks. Essentially, you can divide your data into ranges manually, or you can have ArcGIS Pro divide your data according to a particular method. Visit the ArcGIS Pro website to learn more about the various methods.
    • Classes: 5 is the default amount. You can increase the amount of classes to show a greater variability in colors, or decrease if it’s too variable. Note that the visualization process should allow for users and their audience to easily identify trends, hotspots, and anomalies that exist within the data. It is up to the user, however, to ensure that the data is at the point where it can be visualized. 
    • Color scheme: Choose a color scheme that feels intuitive to your data. For example, precipitation maps are often blue-to-purple to also represent their water attribute. Usually, the yellow-to-red heat map is sufficiently effective in representing quantities, but experiment and find what fits the best!
    • Below the options for symbology are several panes: “Classes”, “Histogram”, and “Scales”. It is a good idea to review these tabs to see how your data is being distributed in terms of their colours.  In the Classes tab, you can see which colours align to which “Upper value.” The upper value is the maximum quantity that a color will be assigned to a feature.  
      • In this example, the color yellow will be assigned to all railways hauling less than or equal to 12 cords of firewood. The next color, light orange, will be assigned to all values more than 12 but less than or equal to 1200 cords of firewood, and so forth. The last color will always be assigned to maximum value in the dataset. For your dataset, ensure that every layer has a class for all railways that have a 0 firewood value. This is an important distinction, as it shows which railways were joined but were reported as not hauling any firewood.
      • The Histogram tab is very useful in seeing the distribution of your data. In this pane, you will see a long, coloured bar travelling vertically with grey bars horizontally extending out from it. These grey bars represent the amounts of your data that fall under a certain quantity. In this case, it would be the amount of railways that have a certain quantity of firewood. The black arrows show the numbers at which your data is classified.  You can use them to modify how your data is visualized by simply clicking on any of the arrows and dragging them up and down. Use this if you think a colour is under, or overrepresented in the visualization. 
    • Using symbology, we can also visualize other attributes of the data in different ways. Navigate to “Vary Symbology by Attribute” in the Symbology pane. Here, you have other options listed to visualize your data. Open the drop-down menu for Size. Under Field, select the field that contains your firewood data. Check the Enable Size Range option. Experiment with the Minimum and Maximum sizes for your line features. 
    • Repeat these steps for all your joined layers. 
Symbology options for railway lines.

Step 5: QAQC: Correcting Historical Inaccuracies

Verifying the Railways with no data (white lines)

As we suggested, historical data can contain many inconsistencies and inaccuracies, and often those become most visible at this stage. After your data have been visualized to your liking, we can look over the results to spot any problematic areas. For instance, in our dataset we found that there were multiple railway segments that did not have historical freight statistics for firewood (we symbolized these as white), but there were also lines in the historical railway GIS layers that existed in that year, but did not match the list in the sessional papers (we symbolized these as grey). Another example, as follows, was a probable gap in the lines.

Verifying the Unmatched Railways (grey lines)

Physical gaps of missing railways

  • Turn on the layer for your selected time in Step 9. Give it a distinct, but easy to see colour. You can do this by modifying its symbology, or by clicking its drop menu in the Contents pane and clicking on its symbol. This will open the Symbology pane > Properties > Here, the user can change both the colour and width to easily differentiate it from the layer we are correcting. For QAQC, make the color of these “unmatched” railways grey. 
  • In the Contents pane, place this layer under the firewood layer you are correcting. In the map, you should now see your firewood railway with all the railways that existed in your selected time period under it. This will show you which railways did not join in Step 10. These railways did not Join because their codes were not mentioned in the firewood spreadsheet, which can happen due to a variety of reasons. The railway may be so insignificant that the census takers did not gather any firewood data for it, they might have forgotten about the railway, or perhaps it was misclassified when the dataset was being created. 
    • For our QAQC process, we will have to determine which of the unmatched railways should have been matched but are missing from the current datasets. These railways are the ones that we assume we missed due to human error; the railways that were not matched because the railway codes mentioned in the Sessional papers spreadsheet did not correspond to the actual railways in the HAC shapefile. If you recall in the documentation for the “Cordwood in Sessional Papers of Canada 1876-1890, Firewood 20-08-04”, the HACNo is a best estimation of the code of the railway in the Historical Atlas of Canada shapefile. This estimation is based on historical research and logical deduction only, and there is not a 100% certainty that the railways mentioned between these 2 resources are the exact same. In order to get the most complete dataset possible, we will have to supplement our work with error checking and more historical research. 
    • The easiest way to identify missed railways is to identify clear gaps between railways. This is an example from the 1878 dataset. Here, we see that there are 2 railways in our joined layer , one of which exists in isolation, not connected to anything else. The isolated railway’s code is 42D, and its closest segment of the same colour is 42D. Although it is possible that the railways may have been connected through the water, we should check if there is some other land railway system connecting the two lines that was not matched to our dataset. Turning on our grey layer, we can see that there are in fact no land railway systems connecting the two lines. 
  • Another example from 1878 is here, in south-western Ontario. The small line in the centre of the image is isolated from the rest of the railways. The code for that line is 43. The line beneath is also coded as 43, so we assume they must be connected in some way. Turning on our grey layer for QAQC, we can see a land-based railway system connecting our small red line to the main line. We can select the grey lines and identify their codes, and we see that the top grey line has a code 43G, while the other greys are of code 81B and 82A. We assume that the 43G line should be part of the 43 railway system, based on the similarities in code. Now, go back to the ‘Cordwood in Sessional Papers of Canada 1876-1890, Firewood 20-08-04” and “hr_codes” spreadsheets. In the Sessional papers spreadsheet, find the 1878 tab and the 43 code. We can see that it belongs to the “London and Gore RR Co”, as part of the “Great Western” railway system.
    • Note: we can see this because the London and Gore RR Co railway line does not have a corresponding number in the “Number in Sessional Papers” column. Also, it has the same amount of firewood as the Great Western railway system. This means that in the Sessional papers, the Great Western railway system was the only railway mentioned. However, the Great Western in the the Historical Atlas of Canada shapefile is NOT the exact same as the one mentioned in the Sessional papers. Because these gaps in lines exist, we assume that the HAC version of the Great Western is only a portion of the one mentioned in the Sessional papers. To fix this, we simply just add in the “gaps” into the spreadsheet. Add a new row below the London and Gore RR Co entry. Under HACNo, enter the INCRP_Code of the grey line, so 43G. Under name of railway, we will first search up the 43G code in the hr_codes spreadsheet. We can see it is named “London & Port Sarnia Ry Co”. Add this name into the “name of railway” column in your new row in the Sessional papers spreadsheet. For the “Cords Firewood” column, because we assume they all belong in the Great Western system, simply copy the amount from the Great Western row. Next, update the Master tab and Pivot table we created in Step 10 c). Make sure you also update the version of the spreadsheet that is being read by ArcGIS on your physical computer! Next, delete the joined version of 1878 from ArcGIS. 
  • In the ribbon go to View > Catalog View > Databases > Open the database for your project > Find the 1878 joined file > Delete. 
    • Re-do these steps, for 1878 only. You have successfully completed the necessary steps to QAQC a single year of the railway map! 

Key Concept: Additional Analysis and QAQC. Looking for outliers in the visualization is a useful way to conduct both analysis and QAQC. Because our symbology is a form of basic temporal analysis (freight hauled per distance on each line over time), the results allowed us to focus our historical research on railways and regions that “stood out” on the map. It also helps identify the areas that might be gaps or errors. Depending on the variables in each database, it is possible to look for other explanations for anomalies. By comparing firewood to other commodities hauled on each line we see that the Toronto-area narrow gauge railways dedicated up to 50 percent of their freight to firewood. Others like the Whitby and Port Perry were not known as fuel trains, but they still stand out on the relative map and they actually dedicated increasing amounts to firewood in the early years (rising from 8 to 12 percent). If some lines are missing entire freight classes and are nearly completely committed to one type, we might assume that there were errors in their data reporting.

Step 6: Creating an Exportable Map Layout

Key Concept: in ArcGIS Pro, there are two ways you can view your data. There is the way we have been using thus far, Map View, which appears in the first tab. There is also Layout View, which we will use to create an exportable map. Layout View opens in a new tab.

Layouts in ArcGIS Pro are an extremely useful tool for creating static images of your map. Each Layout you create forms a separate view or tab that allows you to add elements like legends and citation information on the map and then export the map as a static image. The Layout will maintain the same map frame (including the scale and extent you selected) for each image, even as you change the layers and other contents in the original project (ie. the “Map View”). For temporal GIS projects, it is usually essential to maintain a static map frame, as a sequence of images will appear as a single map with animated map features.

In this step, we will create a Layout for exporting our images. 

  • In the Map view, go to the ribbon > Insert > new Layout > Select ansi landscape letter 8.5 “ * 11”
  • This should open the Layout view. In the ribbon go to Insert > Map Frame > In the drop down menu, select the 2nd option in Map; it should resemble the railway map. Next, your cursor should turn into a target. Using this target, left-click once on the top-left corner of the white page on your screen and drag down to the bottom-right of the white page and release  the left click. An image of your map should now appear on your screen! 
  • Any updates you make to the map will appear in the layout, except if you move the map. The layout is a fixed extent, so it will always be looking at the same position on your map. To change the extent of your layout to one that is easier to see , you can “Activate” your Layout , which allows you to make edits in the Map within the Layout. This will also allow you to change the extent of the Layout, so you can zoom in or out to best capture the extent of your data. Navigate to ribbon, Layout > Activate. Experiment with the best view for the map! 
  • We are also going to add a nicer background for the map, as well as some labels for cities to help any potential viewers orient themselves. In the ribbon, navigate to Map > Basemap > Select the dark gray canvas. This will also add a layer called Dark Gray Reference, which controls the names of the cities, provinces and states that appear on the map. Make sure this is above all your layers so the names are visible. 
  • Next we are going to add a Legend. Once satisfied with the extent of the view, in the ribbon go to Layout > Close Activation. Now go to Insert > Legend. This will open another cursor with a target symbol, drag this wherever you want the legend to be on your map. There should be a clear box in your map titled “Legend”, and a Legend item in the Contents pane once you have created the Legend. To add items to the legend, you can select their layer in the Contents pane and drag them under the Legend group. This will create a new legend item in the Legend group. For our project, we want to drag all the joined railways with symbology, and the grey railways we used for QAQC into the Legend group. The order of layers in the Legend group determines their order of appearance within the legend itself, so make sure the grey railway layers are beneath the joined railways with symbology in the Legend group. 
  • Once your layers have been added into the Legend, we can modify some of the legend properties to make it look more visually pleasing. Double-click the Legend group in the Contents pane. This will open a “Format Legend” pane. Under Legend Items, hit the Show properties button. Now, click on Properties. Here, you can edit the appearance of the text in the Legend. Experiment with each setting and find what looks the best for your Legend. 
    • As a further note, if you wish to edit the amount of decimal places that appear in your Legend, go back to the Map view. Open the symbology for whichever layer you wish to present. Under Classes > Label, you can modify the data labels for each Legend item. Because we are representing a discrete quantity (cords of firewood), we can remove all the decimals for our numbers. Furthermore, we can also add in thousands-separators to help with the readability of our images. In the Symbology pane, go to Advanced Symbol Options > Format Labels > Rounding, Decimal Places: 0 > Check the Show thousands separators box. 
  • Back in the Layout, we are going to add a title for our image. Go to the ribbon > Insert > Rectangle > Drag and drop wherever you wish your title to be. A clear box should show on your map, as well as a new Text item in the Contents pane. To easily edit the text, double-click the Text item in the Contents pane. This will open a “Format Text” pane. You should see a text box; in it, write the following title: Cords of Firewood in Canadian Railways *Date*. Next, click on Text Symbol in the Format Text pane. Under Appearance, modify the text properties so it shows up well on the map. If the full text is not fully appearing on the map, you can click on the text box on the map and adjust its size by selecting and dragging on its vertices. 
  • Once you are satisfied with the appearance of your layout, you can now export it. Go to the ribbon > Share > In Export, Layout > File Type: your preference > Name: save the image wherever you like to save pictures, the file name should be the same as the Title. Click Export once satisfied. Here is what ours looked like:
  • Your static map has now been exported and can be shared across the web! 
  • Repeat this process for all of the selected years.

Step 7: Creating a Time Series

Using ArcGIS Pro’s Automated Time Tool

Now that we have each individual year, we can merge our features into a single shapefile that contains the matched railways for every sample year in the dataset. 

  • Go to Analysis > Tools > Search for Merge > Input datasets: select every sample year > Output Dataset: Merged_name, place in the geodatabase you made > Run 
  • This shapefile now contains every railway for each individual year that had firewood data inputted into it.

Once we are satisfied with the quality of the data we have, we can now begin the visualization process. Because the data we have is over a period of time, we can create a timeline, or Time Series, of it. 

  • To create a time series in ArcGIS Pro, our data must first be compatible with the Time tool. The Time tool in ArcGIS requires that the date is an actual value in the Attribute table, rather than it being a column heading. This step requires that we transpose our data into a different formatting. 
    • Go to Analysis in the ribbon > Open tools > Search for Transpose Fields > 
    • Input table: Merged_dataset, this is the dataset where we are changing the Date from headings to values
    • Fields to Transpose Field: Select every sample year you have. In the Value section, edit the text to only include the year. 
    • Output table: Merged_Transposed_dataset
    • Transposed Field: This is the field that is changing orientation i.e being transposed. As mentioned before, we are changing the Date column into an actual value in the Attribute table, therefore name this section “Date”. 
    • Value Field: This field will contain the values from the transposed fields. Recall the format of our data. The date is our column heading, and its values are the firewood cords. So here, we will name this section “Cords of Firewood.” 
    • Attribute Fields: These fields are additional fields we would like included in the outputted layer. For this, we’ll include Shape, BLDR_Code, INCRP_Code, CNSTRCTD, ABNDND, and HACNo
    • You can now run the tool
    • Check the attribute tool of the new feature class that has been created. You may notice some railways with null values in the Cords_of_Firewood field. Null values can be created when we transpose because in our merged table, one segment of railway has multiple values for each year, and transposition creates a new feature for every value. So if the segment has a value in 1876, 1878, and a null then three segments will be created after transposition. We have to manually delete these null occurrences. 
      • Select by Attributes > input: transposed dataset > Select Where cords of firewood is null . 
      • Delete all rows that have a null cords_of_firewood 
    • Now your data is ready for time series! 
  • Right click the transposed layer  and click properties. Navigate to Time. 
    • Layer Time: Each Feature has a single time field
    • Time Field: Date
    • Time format: This should default as YYYY. 
    • OK
  • There should now be a timeline on the top right of the map. Hovering your cursor over this timeline will reveal an interactive bar with a play/pause button, forward and reverse buttons, and a slider that the user can manipulate. 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s