A machine learning approach for the discrimination of theropod and ornithischian dinosaur tracks

Abstract

Fossil tracks are important palaeobiological data sources. The quantitative analysis of their shape, however, has been hampered by their high variability and lack of discrete margins and landmarks. We here present the first approach using deep convolutional neural networks (DCNNs) to study fossil tracks, overcoming the limitations of previous statistical approaches. We employ a DCNN to discriminate between theropod and ornithischian dinosaur tracks based on a total of 1372 outline silhouettes. The DCNN consistently outperformed human experts on an independent test set. We also used the DCNN to classify tracks of a large tridactyl trackmaker from Lark Quarry, Australia, the identity of which has been subject to intense debate. The presented approach can only be considered a first step towards the wider application of machine learning in fossil track research, which is not limited to classification problems. Current limitations, such as the subjectivity and information loss inherent in interpretive outlines, may be overcome in the future by training neural networks on three-dimensional models directly, though this will require an increased uptake in digitization among workers in the field.

1. Introduction

Distinguishing between tridactyl (three-toed) dinosaur tracks of the herbivorous ornithischians and the predominantly carnivorous theropods is a complex and long-standing problem [1–9]. Broadly, ornithischian tracks are expected to be wider and more symmetric than theropod tracks, with digit impression III less projecting beyond digit impressions II and IV, and with digit impressions being broader, more splayed apart, and terminating in blunt hoof marks instead of sharp claw marks. However, any of these characteristics can be found in both groups, and which are the most important depends on the particular track type in question. Moratalla et al. [1] presented a quantitative approach to discriminate these groups, albeit limited to larger theropod and ornithopod tracks. Limitations of this approach include the small sample size, issues with the measurement scheme and omission of relevant shape characteristics [1,5,6]; despite this, the method has found wide application [3,9–12].

To overcome the limitations of previous statistical approaches, and to remove as much subjectivity as possible, we trained and then employed an artificial neural network to categorize outlines of tridactyl dinosaur footprints as theropod or ornithischian. Artificial neural networks are a type of nonlinear model that can learn from data, and a principal component of machine learning and artificial intelligence. Inspired by the structure of the human brain, such neural networks comprise interconnected nodes (or neurons), with each connection represented by a number (weight). The artificial neural network learns from its mistakes during multiple training iterations in which the weights are optimized to achieve an increasingly good fit to the data structure [13]. We here employ a deep convolutional neural network (DCNN) composed of multiple layers of neurons (deep learning).

We applied our model to an independent test set and compared its identifications with those of five human experts. We then used the neural network to assess outlines from a large trackway from Lark Quarry, Australia; a tracksite also known as the Dinosaur Stampede National Monument (Australian Natural Heritage List Place ID 105664). The identity of the largest trackmaker has been the subject of intense discussion, particularly in the past decade, namely as having been either a predatory theropod dinosaur or a herbivorous ornithischian [3,5,14–20]. The ambiguity of the footprint shapes and lack of agreement among track-workers make this trackway an ideal case study for our machine learning approach.

2. Materials and methods

2.1. Standardized track silhouettes

Neural networks can work with a variety of input data, such as simple measurements, categorical data, two-dimensional images and even three-dimensional models. For best performance, however, it is crucial to maximize relevant information while minimizing irrelevant information.

Moratalla et al. [1] present both a quantitative approach and a list of qualitative criteria to discriminate theropod and ornithischian tracks. Linear and angular measurements, as used by Moratalla et al. [1], may be selected to capture what are believed to be the most relevant shape features of a track. However, such measurements may not capture all relevant details, such as the shape of claw marks. Furthermore, such measurements can be difficult to apply consistently to a large number of tracks of varying shapes [21].

On the other extreme to linear measurements, three-dimensional models capture a maximum of shape information [22], but are highly complex and include much information that is not relevant to the identification of the tracks (e.g. the presence of cracks or ripple marks; what some authors term ‘extramorphological’ features). Although neural networks are able to become invariant to such irrelevant information, they require a larger sample size to do so.

In order to minimize irrelevant information and maximize sample size, we use black-and-white silhouettes of interpretive track outlines. Using silhouettes as input data avoids the need for measuring particular anatomical features, allowing for inclusion of any track shape. Furthermore, silhouettes may capture most of the information that is considered relevant for the identification of tracks, including features such as digit terminations and the number and shape of phalangeal pads. However, such outlines have been repeatedly criticized because of their inherent subjectivity and simplification of complex three-dimensional morphologies (e.g. [19,23]).

Published outline drawings can be complex and may consist of multiple and/or discontinuous lines, dotted lines to indicate uncertainty, internal shadings and other features. Such complexity does often bear the artistic signature of the individual ichnologist. By converging such drawings into simpler black-and-white silhouettes, the artistic signature is mostly removed, avoiding irrelevant detail that may deteriorate the performance of the model. Silhouettes may be represented as a continuous outline or as a set of separate outlines that may show individual pad impressions.

The conversion of published outlines into our standard silhouette format often required some degree of interpolation; e.g. dotted lines were treated as solid lines, and gaps in the outline were closed. We included tracks of functionally tridactyl ornithischian (basal ornithischians, ornithopods and stegosaurs) and non-avian theropod dinosaurs spanning from the Upper Triassic to the Upper Cretaceous. This included tracks that show the impression of digit I in addition to digits II–IV, as long as this digit is unlikely to have played a major role in locomotion in the trackmaker. We did not include tracks interpreted to have been made by partially buoyant animals, nor did we include tracks that show substantial incompleteness, but we include cases in which the track walls have collapsed, as in penetrative tracks [24–26]. Some of the included tracks contain little relevant anatomical detail, or even contain misleading features, due to unfavourable substrate conditions, trackmaker behaviour or post-formational alteration of the tracks. By training the model on a sufficiently large sample size, this ambiguity is hoped to be reflected by a decreased confidence of the model when classifying such tracks.

The majority of outlines were collected from the literature, building on a dataset previously collected by Lallensack et al. [7]. Multiple outlines from single trackways were included where available, and, in some cases, different interpretations of the same track were included. These data are supplemented with outlines drawn by one of us (J.N.L.) from three-dimensional models that are published as part of this contribution or have been published elsewhere. The silhouettes are rotated upright (relative to digit impression III) before analysis and downscaled to 100 × 100 pixels, with the shape centred in the image. The model was made invariant to subtle variations in rotation and position (see below).

Our original dataset consisted of 677 ornithischian and 959 theropodan examples (see electronic supplementary material). This imbalance between the categories is caused by the much greater abundance of theropod tracks in the fossil record, even though attempts were made to include as many ornithischian tracks as possible. Models trained on the full imbalanced dataset tended to perform well on large tracks, but appeared to be biased towards theropods when testing ambiguous small tracks similar to the ichnotaxon Anomoepus. We consequently applied undersampling (see e.g. [27]) to arrive at a balanced dataset by randomly removing theropodan examples until their number was equal to that of ornithischian examples.

2.2. Model training

The model training was done using the open-source machine learning library TensorFlow (www.tensorflow.org; version 2.9), which was controlled through the Keras interface (https://keras.io) using Python scripting.

The model architecture employed, and the parameters used, may drastically influence the performance of a model. Precise optimization of models by finding the best parameters is often impractical given the number of possibilities that can be set. We tested separate model architectures of increasing complexity, different numbers of epochs (training iterations during which the model sees all data in the training set) and batch sizes (the number of silhouettes the model will see at the same time) on test sets that were randomly separated from the training set. We used loss rather than accuracy to select models [28] and discarded those that showed a problematic degree of overfitting. Following these tests, we selected a modification of the VGG16 architecture with four convolutional layers of 32, 64, 128 and 128 neurons, respectively. Pooling layers were used in-between these layers, and a dense layer of 256 neurons was used for flattening. After each pooling layer and after the 256 neuron dense layer, we used a dropout of 0.2 [29]. We selected ‘Adams’ as optimizer and a batch size of 30. We used early stopping to select the model version at the epoch where loss was minimal [30]. For our selected model, this mechanism led to a training period of 74 epochs, after which the validation loss was at 0.37 and the validation accuracy at 0.83.

We used online data augmentation in order to make the model more invariant against slight variations in rotation, position and the exact configuration of pixels of the shape margins, all of which are not informative for classification purposes. This was achieved by random horizontal flipping; rotation by a random value between −30 and +30 degrees; and a slight random shift in x and y directions. These operations were performed on each image before each epoch—as our model was trained with 74 epochs, 74 versions per image were created.

For each tested silhouette, the neural network returns a numerical value that ranged between 0 and 1, to denote confidence of track affinities as ornithischian and theropodan, respectively. A value of 0.5 indicates an ambiguous outcome without any tendency towards either category. The neural network model, along with detailed step-by-step instructions, is provided in the electronic supplementary material.

The final model was validated on a test set of 36 tracks that were collected by J.N.L. from the literature (see electronic supplementary material). Tracks were selected to obtain a varied sample that includes both relatively obvious candidates and challenging ones. In a few cases, the silhouettes do seem difficult or even impossible to assign to their correct label, as they show characteristics of both groups. The same test set was given to five researchers who all are established experts on tridactyl dinosaur tracks. These researchers were asked to identify each track either as ornithischian (0), theropod (1) or ambiguous (0.5).

3. Results and discussion

3.1. Performance on test set and comparison with human performance

On the test set, the neural network outperformed human experts by a constant margin (figure 1, table 1). Eighty-six per cent of tracks were correctly classified while 14% were incorrectly classified. If all neural network decisions above 0.4 and below 0.6 are treated as ambiguous (22% of tracks), the neural network classified 67% of tracks correctly and 11% incorrectly. The human experts, on average, classified 57% correctly, 20% incorrectly and 24% as ambiguous. These results demonstrate the ability of big data applications to outperform human experience when trained on a sufficiently large sample. Most of the test set examples are significantly different from any of those the model was trained on, demonstrating the ability of the model to classify new distinct shapes.

Figure 1.

The test set was used to compare the performance of the neural network model with that of five human experts. The neural network returned values between 0 and 1, with values less than 0.5 indicating more ornithischian-like shapes and values greater than 0.5 indicating theropod-like shapes. Values between 0.4 and 0.6 are here considered as ‘ambiguous’. Human experts marked each track as either ‘ornithischian’, ‘ambiguous’ or ‘theropodan’. The ratio between correct, ambiguous and incorrect identifications of these five experts is shown for each track.

Table 1.

Performance of the individual human experts and the neural network on the test set. Note that high percentages of correct identifications are associated with high percentages of incorrect identifications, and vice versa.

identifier	% correct	% ambiguous	% incorrect
Expert 1	67	3	31
Expert 2	58	25	17
Expert 3	58	25	17
Expert 4	42	44	14
Expert 5	58	22	19
neural network (no ambiguous cases)	86	0	14
neural network (0.4–0.6 treated as ambiguous)	67	22	11

3.2. Limitations and strengths

Our neural network was trained to classify shapes without any context information. When an ichnologist decides how to assign a track, such context information (e.g. size, stratigraphy and shape of other tracks of the same trackway) plays an important role. While it is generally possible to provide a neural network such context information, this comes at a cost: the model will be more likely to overfit, or, in other words, may fail to make a correct assignment when the track in question was found outside of its expected context. This may lead to circular arguments: if, for example, information on size and geological age is provided to the model, large tridactyl tracks from the Upper Jurassic are more unlikely to be classified as ornithischians because no such examples are present in the training set. This may easily lead to false conclusions if the a priori assumption (no large ornithischians in the Upper Jurassic) is wrong. In this light, this apparent shortcoming of our neural network may therefore be its greatest strength: providing objective assessments of shape independent of context. Consequently, it has to be the job of the ichnologist to combine the neural network evaluation of the shape with all relevant context information to arrive at a meaningful interpretation of the track.

In the initial stages of the present project, the authors were undecided on whether to include size information. Size is undeniably important in some cases—e.g. a 50 cm long slender-toed track can confidently be assigned to a theropod trackmaker, even though the same shape at a much smaller scale might well be produced by an ornithischian trackmaker. On the other hand, we may argue that tracks that are unambiguously classifiable based on size do not require the judgement of our neural network to start with. In all ambiguous cases, evaluating shape independent of size can yet again be considered a strength, as it brackets off a priori assumptions about the possible size range of a trackmaker, which may be based on incomplete knowledge. This reasoning notwithstanding, we did collect size information along with all outlines of our dataset. Tests using an earlier model version that included scale information did not appear to produce better results than other models that did not include such information, and we consequently abstained from including this information. Another difficult decision is the identification of a track as belonging to the left or right foot. Tests using early model versions did not show clear improvements in performance when presented with uniform right shapes versus randomly mirrored shapes, and most tracks of unknown trackmaker identification cannot be unambiguously attributed to left or right to start with. Consequently, we decided to augment the data to randomly mirror the shapes, making the model invariant against left or right.

Another inevitable limitation is our reliance on previous identifications of tracks as either ‘theropod’ or ‘ornithischian’ to train the models. We assume that the vast majority of these a priori identifications are probably correct even when the shapes themselves are ambiguous, since the context usually offers valuable clues (most unambiguously the presence of manus impressions, which only occur in ornithischians). However, misidentifications cannot be ruled out especially for smaller tracks which are particularly ambiguous.

Finally, the greatest limitations of the presented approach are the interpretive outline silhouettes themselves. Advantages of using such silhouettes, namely their simplicity and wide availability, are compromised by their inherent subjectivity [19,22]. This subjectivity is highlighted by figure 2, where outlines produced by different researchers may be different in shape to a problematical degree [23,31]. Besides adding substantial noise, this subjectivity may shift the neural network evaluation towards one or the other category, as is the case with track 3 (figure 2, outline of P.L.F. and J.N.L. versus that of A.R.). Furthermore, given the loss of potentially valuable information, the reduction of information content can be considered an advantage and a drawback at the same time. Although extramorphological features tend to be excluded from interpretive outline drawings to arrive at a meaningful hypothesis about the shape of the foot, this is not always possible, and the degree to which outlines are ‘idealized’ varies greatly between separate ichnologists. Indeed, some authors (e.g. [32]) have argued that ‘extramorphological’ features should be considered part of the track itself (e.g. displacement rims and collapse structures), and that these features may contain information about the trackmaker that should not be excluded.

Figure 2.

Tracks 1–6, 9 and 10 of trackway LQ1 from Lark Quarry, Australia. Height maps of three-dimensional models are shown together with respective interpretive outlines produced by the three authors (P.L.F., J.N.L. and A.R.) and the objective outline algorithm of [23]. Note the marked differences in shape and size between outlines, in particular of track 3. Scores returned by our neural network indicate that most tracks with the exception of track 3 are ornithischian-like.

As we authors experienced with the Lark Quarry trackway (see below), the production of meaningful outlines was not always possible, and identifying which features were related to the foot morphology, and which to sediment movement, and later erosion, was difficult. Given these drawbacks, our approach using interpretive outlines can only be considered the first step, and more objective and comprehensive data formats, such as three-dimensional shapes, need to be considered for future applications of neural networks (see below).

3.3. Analysis of Lark Quarry tracks

Previous debate on the identification of the large tridactyl trackway Q1 from Lark Quarry was complicated by two points of disagreement: (i) which tracks best reflect the anatomy of the foot and (ii) how the outline of the individual track should be drawn. The interpretation of outlines of individual tracks has been even more controversial and has been argued to be significantly influenced by the researcher's opinion about the trackmaker responsible [3,5,16–20,23].

We here consider all tracks of the Lark Quarry trackway except for tracks 7, 9 and 11, as these tracks lack an obvious tridactyl morphology. Outlines were drawn by all three of us based on digital models made from the archival cast material, before damage and reconstruction occurred to many of the tracks in situ. It should be noted that in previous work, P.L.F. favoured a theropod affinity [19] and A.R. favoured an ornithopod affinity [3,16], while J.N.L. considered the tracks to be ambiguous. In addition, we produced objective outlines of all tracks using the algorithm of Lallensack [23], run with default settings. These objective outlines mimic the way humans interpret outlines but are by design devoid of a priori assumptions about the trackmaker.

For six of the eight tracks, the neural network tended to be in favour of an ornithischian affinity (figure 2). The mean score of P.L.F., who favoured a theropod affinity of the tracks, was slightly higher (0.32) than that of A.R. (0.26), J.N.L. (0.24) and the objective outlines (0.24). More substantial differences were found for track 3, which Thulborn & Wade [14] considered the best-preserved track: outlines by P.L.F. (0.99) and J.N.L. (0.93) where found to be very theropod-like, while outlines by A.R. (0.48) and the objective outlines (0.52) were ambiguous. This indicates that although a priori assumptions about the responsible trackmaker may indeed influence the result in some cases, general trends seem to be robust (i.e. this track was the most theropod-like regardless of bias).

However, the mean of scores of all considered tracks is not necessarily an informative measure, as in many cases only one or few tracks will convey the anatomy of the foot accurately. We therefore continue by taking a closer look at those tracks that have been identified as the best preserved by previous workers: tracks 3, 5 and 6. Track 3 is clearly different in appearance from all other tracks and is consistently recovered as the most theropod-like. Analysis with gradcam [33] of the outline by J.N.L. of track 3 (figure 3a) suggests that the curvature of digit impression III and the shape of the heel are the important features that led to its classification as theropod. The curvature of digit III is also indicated by longitudinal ridges at the floor and wall of the track that might represent pull-up structures, indicating that this curvature is original and not the result of erosion [16].

Figure 3.

Analysis of Lark Quarry track 3. (a) Gradcam [33] map superimposed on the interpretive outline of J.N.L., highlighting the areas that were important for the classification of this shape as theropod. (b) A (subjectively chosen) ‘minimum’ contour line superimposed on a ‘maximum’ contour line. The minimum contour line gives neural network scores close to 1, suggesting a theropod trackmaker, while the maximum contour gives an ambiguous score, suggesting that contours become more theropod-like with increasing depth. (c) Interpretive outline of track 5 superimposed on that of track 3, showing marked differences in width but not in length.

Tracks 5 and 6, which are unambiguously ornithopod-like, are similar to some of the other, more fragmented tracks. A direct overlay of interpretive ‘maximum’ outlines of tracks 3 and 5 (figure 3c) shows that the latter is distinctly wider with larger interdigital angles and a generally more robust appearance. Both tracks are, however, almost identical in length. If track 3 does reflect the foot anatomy more closely, it must follow that other tracks were broadened due to erosion or rotation of the foot. If, on the other hand, tracks 5 and 6 more closely match the foot anatomy, it follows that digits II and IV of track 3 are partially collapsed during or after track formation.

While morphologies of most tracks other than track 3 tend to suggest an ornithischian trackmaker, the long stride lengths and low pace angulation values of around 180° [16] are more typical for a theropod trackmaker. We may also be running into the limitations of using outlines for relatively deep tracks, where outlines created near the top of the track are broader and more rounded (and hence more ornithopodan) than outlines from deeper in the track, at the foot-sediment interface, as demonstrated by Falkingham [19]. This is confirmed by testing (subjectively selected) ‘maximum’ and ‘minimum’ contour lines of track 3 (figure 3b), where the maximum outline was ambiguous (score of 0.51) and the minimum outline very theropodan-like (score of 0.98). Future studies involving neural networks trained on three-dimensional models of tracks and/or trackway parameters may be able to resolve this ambiguity, as we shall come to in our conclusions.

3.4. Re-evaluation of other tracks and implications for trackmaker identification

Shapes attributed or similar to those of the ornithischian ichnogenus Anomoepus are often recovered as ambiguous by our model. Anomoepus tracks are typically identified based on the presence of manus impressions, inward rotation of the pes, and low relative projection of digit III beyond digits II and IV [34]. The ambiguity expressed by our model, however, suggests that Anomoepus cannot be reliably distinguished based on pes track shape alone when context data are not available. Such difficulties may be illustrated by an isolated track from the Hettangian of Poland that was described as a particularly large example of the ornithischian ichnogenus Moyenisauropus (=Anomoepus) [35]. Our neural network, however, suggests that a theropod affinity is more likely (0.72). A trackway of three tracks from the Lower Jurassic (Sinemurian) Razorback beds of Australia had been traditionally interpreted as theropodan in origin [36,37], while a more recent evaluation identified these tracks as ornithischian Anomoepus tracks [38]. Our neural network confirms the latter interpretations (scores of 0.42, 0.44 and 0.32 for the three tracks, respectively).

Large tridactyl tracks, such as those of trackway 1 from Lark Quarry, pose different problems. Large ornithopod tracks are primarily differentiated from large theropod tracks based on their overall wide shape and their broad and rounded toe impressions that lack claw traces. Similar ornithopod-like features can, however, occur due to unfavourable substrate properties and/or within-sediment rotation of the foot, while theropod-like features are more unlikely to result from such factors. Consequently, large tridactyl tracks may be more easily mis-identified as ornithopod than as theropod. Schulp & Al-Wosabi [11] attributed a large tridactyl trackway from the Upper Jurassic of Yemen to an ornithopod trackmaker, although its large size (track length greater than 50 cm) is unusual for ornithopods of that age. This was subsequently questioned by Piñuela et al. [8], who argued that features of some of the tracks are theropod-like. Our neural network indeed suggested an ornithopod affinity for most tracks but two (tracks 5 and 13), which were more theropod-like (0.59 and 0.71, respectively). As in the case of Lark Quarry, the identification of the trackway will ultimately depend on the question which of these tracks does more closely match the trackmaker's foot, which cannot be fully addressed without careful restudy of the tracks, or three-dimensional models thereof.

3.5. Future research

The value of fossil tracks as palaeontological data sources cannot be overstated, yet their interpretation can be challenging. The shape of tracks is influenced by multiple factors, including anatomy, substrate properties and behaviour, but also post-formational alteration [32,39]. Although tracks are organized into separate ichnotaxa based on shape features thought to reflect anatomy, these ichnotaxa in fact form a continuum of shapes rather than discrete classes [21]; such continua must be analysed using rigorous quantitative methods. Neural networks may overcome limitations of previous quantitative approaches and are able to adapt to the immense complexity of the data. The application of neural networks may go well beyond simple classification tasks as carried out here. By learning to distinguish relevant from irrelevant shape features, these methods could be used to constrain foot shape, movement and sediment properties for a given track or trackway, and even to produce outline drawings and measurements from three-dimensional models that capture the important shape features better than any current objective outline technique [19,23]. Neural networks, in principle, and as demonstrated here, have the ability to outperform human experts in any of these tasks.

Traditional ichnology has been based on interpretive outline drawings in order to document and communicate track morphologies. Such outline drawings are, however, deeply problematic for quantitative analysis because of (i) their subjectivity, (ii) the loss of three-dimensional information and (iii) the abstraction of three-dimensional shapes to a set of margins which often do not exist in reality. While outlines drawn by a single well-versed researcher might be considered to be consistent, outlines of separate researchers are certainly not. Subjective decisions during drawing outlines include the precise position of the track margin [19,23], but also the degree of ‘improvement’ to interpolate outlines, remove post-formational alterations, and to highlight anatomical features of interest. These subjective decisions can result in strikingly different outline tracings of a single track (e.g. [19,23]). Because such decisions may be influenced by a priori assumptions, in particular about the trackmaker taxon responsible, the danger of circular arguments when analysing these outlines is real. Despite this, there is still value in outline drawings for quantitative analysis, if carried out carefully, as demonstrated in the present approach. Using such drawings can only be the first step, and future work will soon need to switch to three-dimensional models to train such neural networks.

There has been a growing movement toward collecting and publishing three-dimensional data, particularly as part of a ‘standard ichnological protocol’ [22], using methods such as laser scanning and particularly photogrammetry [40–42]. Long-term digital preservation, however, requires not only the capture of sets of photographs suitable for photogrammetry, but also the storage of such data in curated repositories such as Figshare and Morphosource, or as supplemental data associated with published papers. If kept outside of such repositories, data are likely to be lost eventually after retirement of the researcher. Only if and when three-dimensional data is more widely available, quantitative methods such as neural networks will be able to fully unlock the potential of fossil tracks as palaeobiological data sources.

Data accessibility

Electronic supplementary material, data includes the neural network and detailed instructions how to use the neural network to classify unknown tracks (electronic supplementary material, file S1); metadata and references associated with the dataset (electronic supplementary material, file S2) and the full dataset used to train and test the neural network (electronic supplementary material, file S3) [43].

Authors' contributions

J.N.L.: conceptualization, data curation, formal analysis, investigation, methodology, project administration, resources, software, validation, visualization, writing—original draft and writing—review and editing; A.R.: resources, validation, visualization and writing—review and editing; P.L.F.: conceptualization, investigation, methodology, resources, supervision, validation, visualization, writing—original draft and writing—review and editing.

All authors gave final approval for publication and agreed to be held accountable for the work performed therein.

Conflict of interest declaration

We declare we have no competing interests.

Funding

This work was founded by the German Research Foundation (DFG) under grant LA 4611/2-1 to J.N.L.