Article Review Transportation Engineering

¶ … Longitudinal evaluation of a GIS laboratory in a transportation engineering course," Bham, Cernusca, Luna and Manepalli look at the effectiveness of a geographic information system-based tutorial in the teaching of transportation engineering. The authors studied students who were given this form of tutorial in conjunction with other learning techniques. They found that the students who received this tutorial performed better than those who did not. The paper serves to build the body of evidence with respect to GIS as a teaching technique for this subject. The authors sought to show that this technique is effective, and they accomplished that. The study was significantly rigorous to make a reasonable contribution to this field of study.

The article initially introduces the problem. They noted that "previous studies reveal that ... entry-level engineers lack significant exposure to transportation engineering methodologies" despite having hourly requirements for study in transportation-related courses in civil engineering programs (Bham et al., p.258). The use of geographic information systems in tutorials is one of the means by which educators in the field have sought to overcome this knowledge gap among entry-level engineers, and thereby raise the quality of education in this subject. GIS had received the lowest amount of coverage among education methodologies that were analyzed in another survey, which lead to the hypothesis that increasing the usage of GIS might contribute to resolving this knowledge gap. In particular, GIS would help civil engineering education become more focused on market needs (Bham, et al., p.258).

Working with this basic understanding of the problem, the authors developed a GIS laboratory that could be used to assist civil engineering students learn more about transportation engineering. The software was a "self-paced multimedia tutorial that introduced the steps associated with the use of ArcGIS for each stage of ... " eight different tasks (Bham et al., p. 259). The software was based on a set of highway crash data. Working with this laboratory, the authors sought to explore three key research questions. The first is whether or not this GIS laboratory is an effect means to help civil engineering students learn about traffic safety, the second is whether it produces a better learning experience and the third is whether students perceive it to be better (Bham et al., p.259). Arguably, only the first genuinely matters -- outcomes-based evidence is superior when analyzing whether the software improves engineering education. However, there is good reason for the authors to include the other two research questions as well, as the responses from instructors and students could help to develop improvements to the software going forward. In particular, if the results to the first research question were not positive, it would be important to know why. So the last two research questions deliver valuable feedback to the makers of the GIS tutorial.

The authors then outlined the research methodology. The project was divided into multiple phases. In the first phase, the GIS was implemented in 2009 on a limited basis. There were subsequent phases where adjustments were made to the original methodology. One of the objectives of the different phases was to test if different means of delivering the software impacted on the results. The authors determined that there was a significant effect from using the software. The authors set out a control group as well, but the control group was merely a group that did not have a 20 minute lecture before using the laboratory. All groups use the GIS lab. Thus, this experiment does not appear to effectively test what the authors say it tests -- there is no control group recorded that did not use the lab. The study therefore tests the 20 minute instructional prior to the lab at the independent variable, not the lab itself. Nevertheless, the researchers found that the non-control group outperformed the control group, even when accounting for GPA and other factors. The different phases were conducted at course levels spanning all four undergraduate years, and the findings held up across all four phases.

Another fault with the study was that although the four phases were conducted differently (adjustments were made in between each phase) and with different sample compositions, the authors lumped the first three phases in with each other, while holding the fourth phase separate for the purposes of their statistical analysis. This seems arbitrary, and contrary to best practice in statistical analysis. The findings of the authors may ultimately have been massaged by this decision. Understanding the analysis is also compromised by the manner in which the paper is written. The authors clearly understand what they are writing about, but in jumping around between four different phases, and different types of students (transportation and non-transportation), they obfuscate the results of individual variables. There are essentially eight different samples here (transportation vs. non-transportation, over four different phases) but they do not present eight different data sets. For this reason, the study is of limited actual value -- the authors are inconsistent in their treatment of the different samples, grouping some, not grouping others, and grouping samples that were not the same in terms of how the experiment was administered, or in their composition. Best practices for statistical rigor were not followed, and one has to wonder why that was.

There was also a qualitative dimension to this research, reflecting the final research question. The authors wanted feedback from students about the different techniques. While there is logic to the authors' choice of in vivo coding -- it is commonly used for this type of analysis -- in vivo coding is also substantially open to bias. The authors can cherry-pick key words or phrases to support whatever narrative they wish. For example, "hands-on activity" was cited as a positive response (Bham et al., p.263), when it is clearly a neutral statement about the nature of the activity. The students' responses were not provided in this paper (a few examples handpicked by the authors were provided in Tables 7 and 9). So ultimately any reader must take the authors' assertions at face value. This is not possible due to the fact that the authors were involved in the development of the GIS software, and therefore have significant person al interest in demonstrating its effectiveness and popularity with the students who would be using it -- the bias is too strong to ignore, and too strong to allow for the biased individuals to perform the in vivo coding themselves.