Bayesian hierarchical models for the prediction of the driver flow and passenger waiting times in a stochastic carpooling service

Abstract

Carpooling is an integral component in smart carbon-neutral cities, in particular to facilitate home-work commuting. We study an innovative carpooling service which offers stochastic passenger-driver matching. Stochastic matching is when a passenger makes a carpooling request, and then waits for the first driver from a population of drivers who are already en route. Crucially a designated driver is not assigned as in a traditional carpooling service. For this new form of stochastic carpooling, we propose a two-stage Bayesian hierarchical model to predict the driver flow and the passenger waiting times. The first stage focuses on prediction of the aggregated daily driver flows, and the second stage processes these daily driver flow into hourly predictions of the passenger waiting times. We demonstrate, for an operational carpooling service, that the predictions from our Bayesian hierarchical model outperform the predictions from a frequentist model and a Bayesian non-hierarchical model. The inferences from our proposed model provide insights for the service operator in their evidence-based decision making.

1. Introduction

Providing ecologically sustainable transportation that is accessible for all is one of the major challenges in the transition to post-carbon societies. A key component of the solution is carpooling services which cater to the mobility requirements in marginalised peri-urban regions with sparser population and physical/digital infrastructure. These carpooling lines, which closely resemble traditional bus lines, connect the physical meeting points for drivers and passengers. The meeting points are placed strategically in highly frequented areas, which take into account various factors such as aggregated traffic flow, socioeconomic characteristics, pedestrian accessibility, local government regulations, etc. This concentrates the demand and the supply of carpooling so that they can reach a critical mass more quickly and more sustainably. These meeting points are where the passenger makes a carpooling request on an electronic console. Since a driver is not allocated in advance, this request is then displayed on a electronic sign on the roadside which informs all passing drivers of a passenger request to the specified destination. This is a real-time, stochastic matching between a passenger and a flow of potential drivers. This new type of passenger-driver matching, along with the aggregating effects of the highly frequented physical meeting points, enables carpooling to reach economical feasibility in peri-urban regions.

From a mathematical and technological point-of-view, it is vastly more difficult to provide a reliable waiting time of a driver arrival in stochastic matching than in deterministic matching. In the latter, a reliable waiting time requires only the tracking of a single assigned driver, whereas stochastic matching requires a more comprehensive understanding of the general driver flow. To assist in the construction of this understanding, the service operator can incentivise drivers to share their GPS locations in real-time. We focus on how to predict the driver flows and passenger waiting times from these GPS traces, which can be considered to be a form of crowd-sourced data collection [13]. Due to the novelty of stochastic carpooling, there is a scarcity of empirically verified predictive models, apart from simple frequentist approaches [16,17]. We propose a more sophisticated Bayesian hierarchical approach where we first build predictive models of the potential driver flow from the observed GPS driver traces, and which are subsequently employed to predict the passenger waiting times. At the time when a passenger request is made, we model this instantaneous driver flow as a moving average of previous driver flows. Then we model the passenger waiting time as a regression with covariates based on the driver traffic flow from the first stage. Our objective is to construct a Bayesian two-stage hierarchical model which is able to predict well the daily driver flows and the hourly passenger waiting times.

The empirical data in this paper are extracted from the ‘Lane’ stochastic carpooling service (www.lanemove.com), operated by the carpooling provider Ecov (www.ecov.fr), in conjunction with Instant System (www.instant-system.com), in a peri-urban region in south-eastern France. See Papoutsis et al. [16] for more details on its set-up. The data collection period is the 382 days from 2018-05-15 (service launch) to 2019-05-31 (beginning of the following year's summer holiday season in France). The daily driver flows in the Lane network are presented in Figure 1, where we enumerate each driver GPS trace, rather than each unique driver. The ordinary weekdays (ORD) are in orange, the school holidays (SCH) in green, and the public holidays/weekends (PWE) in blue. The classic temporal cycles of driver flow data indicate that a moving average is a relevant approach for prediction.

Figure 1.

Daily driver flow in the Lane carpooling service, from 2018-05-15 to 2019-05-31. The ordinary weekdays (ORD) are in orange, the school holidays (SCH) in green and the public holidays/weekend days (PWE) in blue.

The passenger waiting times cover the period from 2019-07-25 to 2020-02-17. This range of dates is different to those for the driver GPS traces above since, due to operational technical difficulties, the passenger waiting times were not reliably recorded from 2018-05-15 until 2019-10-21, so these dates are excluded from the analysis. In Figure 2 are the (approximately) 1500 observed passenger waiting times aggregated for each day. The Lane service is guaranteed only for ordinary weekdays, and whilst the passengers and drivers are not prevented from using the service on other days, there are far fewer carpooling requests on school holiday weekdays and no requests on public holidays/weekends.

Figure 2.

Passenger waiting times (in minutes) in the Lane carpooling service from 2019-10-22 to 2020-01-15. The ordinary work days (ORD) are in orange, and the school holidays (SCH) in green.

In Section 2, we describe the two stages of the Bayesian hierarchical model for the daily driver traffic flow and the hourly passenger waiting times. In Section 3, we carry out a validation of the proposed model with simulated data. In Section 4 we apply it to empirical data drawn from an operational carpooling service, and compare the predictions of driver and passenger behaviour with those from frequentist and Bayesian non-hierarchical models. We end with some concluding remarks.

2. Bayesian hierarchical modelling of driver flow and passenger waiting times

As the driver flow and the passenger waiting time are fundamental quantities in transportation research, their estimation and prediction are the subject of a vast field of active research so we cite only a few references. Historically the simplest models for the driver flow are the moving window averages [18]. More advanced methods draw from time series analysis, within a frequentist [5] or a Bayesian framework [9] have been posited. Established methods for waiting time prediction for stochastic carpooling tend to be frequentist approaches [16,17].

Due to the hierarchical relationship between the driver flows and the passenger waiting times in a stochastic carpooling service, it is natural to consider nested hierarchical models. A general introduction to hierarchical models is provided in Gelman [7] and Gelman and Hill [8]. Hierarchical models can be implemented with frequentist approaches, though we cite only industrial applications using Bayesian approaches here, e.g. image analysis learning [14], football results prediction [1] and electricity load forecasting [20]. Within the transport sector, examples include traffic accident prediction [4] and traffic flow modelling [21]. These latter approaches do not combine the driver traffic flow and the passenger waiting times and do not analyse data with differing time scales in the different stages in the hierarchical model, as we propose.

Our proposed Bayesian multi-level hierarchical model is composed of two nested stages, as illustrated in the flowchart in Figure 3. The input data (driver GPS traces) are preprocessed, as outlined in Appendix A.1, so that they are suitable as input into the hierarchical models. The first model is a multi-level moving average model. It combines the robustness and simplicity of moving averages with the targeted adjustments of the multi-level coefficients [9,18]. The coefficient $\theta \theta$ in this moving average model depends on the day types [2,12], and so the number of components of $\theta \theta$ depends on the number of different day types considered. The output from the first hierarchical model is the daily driver flow, which is the immediate input into the second hierarchical model. The latter is a Gamma regression, whose regression coefficient $\mathrm{\text{β}}\mathrm{\text{β}}$ has S components, with ${\mathrm{\text{β}}}_s\in (0,1)$ for $s=1,\dots ,S$, for each of the S time intervals of a 24-hour period. The role of $\mathrm{\text{β}}\mathrm{\text{β}}$ is to assign the daily traffic flow to these sub-daily time intervals. The output of this second hierarchical model is the temporal profile of the passenger waiting times $ww\in {\mathbb{R}}_{+}^S$ for these sub-daily time intervals. The scarcity of the driver GPS traces allows us to model the driver flow robustly only at a daily level, whereas a higher temporal resolution of the output passenger waiting times is required for a carpooling service. Bayesian hierarchical models offer an intuitive treatment of these differing temporal resolutions within a single workflow.

Figure 3.

Flowchart of Bayesian hierarchical model for driver flow and passenger waiting time prediction. The input data (driver GPS traces) are in grey, the hierarchical models in green, the model parameters in orange, and the model outputs in purple.

2.1. Multi-level moving average for driver flows

From a visual inspection of the daily driver flows in Figure 1, a standard moving average which ignores the day types would be unable to account for the abrupt differences in the driver flow when consecutive days are of different day types. Let the day type function of day i be

$$\mathrm{DT}\left(i\right)=\{\begin{array}{ll}\mathrm{O}\mathrm{R}\mathrm{D}& \mathrm{i}\mathrm{f}\mathrm{d}\mathrm{a}\mathrm{y}i\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{d}\mathrm{a}\mathrm{y}\\ \mathrm{S}\mathrm{C}\mathrm{H}& \mathrm{i}\mathrm{f}\mathrm{d}\mathrm{a}\mathrm{y}i\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{o}\mathrm{l}\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{y}\\ \mathrm{P}\mathrm{W}\mathrm{E}& \mathrm{i}\mathrm{f}\mathrm{d}\mathrm{a}\mathrm{y}i\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{p}\mathrm{u}\mathrm{b}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{y}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{d}\end{array}$$ (1)

where $i=1,\dots ,N$. Thus a suitable $K^{th}$ order recurrence relation of the daily driver flow $y_i$, for $i\ge K\ge 1$, satisfies

$$y_i={\alpha }_{\mathrm{DT}\left(i\right)}\sum _{k=1}^K{\eta }_{\mathrm{DT}(i-k)}{y}_{i-k}+{ϵ}_i$$ (2)

where ${\alpha }_{\mathrm{DT}(\cdot )}$ is the coefficient for the current day i, ${\eta }_{\mathrm{DT}(\cdot )}=\mathbf{1}\{\mathrm{DT}(\cdot )=\mathrm{O}\mathrm{R}\mathrm{D}\}+{\eta }_{\mathrm{S}\mathrm{C}\mathrm{H}}\mathbf{1}\{\mathrm{DT}(\cdot )=\mathrm{S}\mathrm{C}\mathrm{H}\}+{\eta }_{\mathrm{P}\mathrm{W}\mathrm{E}}\mathbf{1}\{\mathrm{DT}(\cdot )=\mathrm{P}\mathrm{W}\mathrm{E}\}$ are the coefficients for the past K driver flows, and $\left\{{ϵ}_i\right\}$ are a sequence of independent normal random variables $\mathcal{N}(0,{\sigma }_{ϵ}^2)$. To ensure the identifiability of ${\eta }_{\mathrm{DT}(\cdot )}$, without loss of generality, we set ${\eta }_{\mathrm{O}\mathrm{R}\mathrm{D}}=1$ for all days.

The model in Equation (2) has a moving average structure of order K, but with two additional multi-level coefficients that make the average adaptive to the day types for the current day i and the previous K days. The multi-level coefficients ${\eta }_{\mathrm{DT}(\cdot )}$ allows us to model the current driver conditioned on the previous day types, whereas the multi-level coefficients ${\alpha }_{\mathrm{DT}(\cdot )}$ re-scale these flows conditioned on the current day type. For example, if day i is a school holiday, then the right hand side of Equation (2) is

$$\begin{array}{rl}& {\alpha }_{\mathrm{S}\mathrm{C}\mathrm{H}}\sum _{k=1}^K\left[\mathbf{1}\right\{\mathrm{DT}(i-k)=\mathrm{O}\mathrm{R}\mathrm{D}\}+{\eta }_{\mathrm{S}\mathrm{C}\mathrm{H}}\mathbf{1}\{\mathrm{DT}(i-k)=\mathrm{S}\mathrm{C}\mathrm{H}\}\\ & +{\eta }_{\mathrm{P}\mathrm{W}\mathrm{E}}\mathbf{1}\{\mathrm{DT}(i-k)=\mathrm{P}\mathrm{W}\mathrm{E}\}]y_{i-k}.\end{array}$$ (3)

In the summand of Equation (3), the day type functions allow us to sum over the K previous days, even if they are of different types. If a previous day is a work day, then its contribution to the current driver flow is ${\alpha }_{\mathrm{S}\mathrm{C}\mathrm{H}}{y}_{i-k}$; if a previous day is a school holiday then it is ${\alpha }_{\mathrm{S}\mathrm{C}\mathrm{H}}{\eta }_{\mathrm{S}\mathrm{C}\mathrm{H}}{y}_{i-k}$; if a previous day is a public holiday/weekend then it is ${\alpha }_{\mathrm{S}\mathrm{C}\mathrm{H}}{\eta }_{\mathrm{P}\mathrm{W}\mathrm{E}}{y}_{i-k}$.

Our model in Equation (2) possesses a similar structure to an autoregressive model, though it does not strictly satisfy the definition of the latter. It cannot be defined with a back shift operator due to the action of the multi-level coefficients ${\alpha }_{\mathrm{DT}(\cdot )}$ and ${\eta }_{\mathrm{DT}(\cdot )}$, and the process $\{y_i\in [0,\mathrm{\infty }),i=1,2,\dots \}$ is non-stationary due to the drift in the driver participation rate after the launch of the carpooling service.

Let $\theta \theta =({\alpha }_{\mathrm{O}\mathrm{R}\mathrm{D}},{\alpha }_{\mathrm{S}\mathrm{C}\mathrm{H}},{\alpha }_{\mathrm{P}\mathrm{W}\mathrm{E}},{\eta }_{\mathrm{O}\mathrm{R}\mathrm{D}},{\eta }_{\mathrm{S}\mathrm{C}\mathrm{H}},{\eta }_{\mathrm{P}\mathrm{W}\mathrm{E}},{\sigma }_{ϵ}^2)$, where we fix ${\eta }_{\mathrm{O}\mathrm{R}\mathrm{D}}=1$ identically, and have ${\alpha }_{\mathrm{O}\mathrm{R}\mathrm{D}},{\alpha }_{\mathrm{S}\mathrm{C}\mathrm{H}},{\alpha }_{\mathrm{P}\mathrm{W}\mathrm{E}},{\eta }_{\mathrm{S}\mathrm{C}\mathrm{H}},{\eta }_{\mathrm{P}\mathrm{W}\mathrm{E}}\in (0,1)$ and ${\sigma }_{ϵ}^2\in {\mathbb{R}}_{+}$. Let the N days of observed daily driver flows be $y_i,i=1,\dots ,N$, where N>K. Since the error variables are independent Gaussian random variables, then the conditional likelihood of $yy=(y_K,y_{K+1},\dots ,y_N)$ is

$$L\left(yy|\theta \theta \right)=\frac{1}{(2\pi {\sigma }_{ϵ}^2{)}^{(N-K+1)/2}}\exp [-\frac{1}{2{\sigma }_{ϵ}^2}\sum _{i=K}^N(y_i-g_i\left(\theta \theta \right){)}^2]$$

where $g_i\left(\theta \theta \right)={\alpha }_{\mathrm{DT}\left(i\right)}\sum _{k=1}^K{\eta }_{\mathrm{DT}(i-k)}{y}_{i-k}$. This conditional likelihood is formed by the product of the conditional densities of $y_i$, given $y_{i-K},\dots ,y_{i-1}$, for $i=K+1,\dots ,N$.

In Bayesian analysis, the parameter of interest $\theta \theta$ is a random variable, and its prior distribution π represents our belief in its uncertainty. The posterior density $\pi \left(\theta \theta |yy\right)$ represents an update of the prior distribution by taking into account the observed data. In our case, we do not have access to existing knowledge that would provide an informative prior and thus we form a non-informative prior on $\theta \theta$, i.e. $\pi \left(\theta \theta \right)\propto {\sigma }_{ϵ}^{-2}$ [3, Chapter 1]. This leads to the following posterior distribution

$$\pi \left(\theta \theta |yy\right)\propto L\left(yy|\theta \theta \right)\pi \left(\theta \theta \right)\propto \frac{1}{{\sigma }_{ϵ}^{N-K+1}}\exp [-\frac{1}{2{\sigma }_{ϵ}^2}\sum _{i=K}^N(y_i-g_i\left(\theta \theta \right){)}^2].$$ (4)

For the inference on $\theta \theta$, Monte Carlo approximations are required since the posterior distribution (and its moments, quantiles etc.) cannot be calculated explicitly. The most widely used family of methods is the Monte Carlo Markov Chain (MCMC) which generates a Markov Chain $\{\theta {\theta }_0,\theta {\theta }_1,\dots \}$ whose equilibrium distribution converges to the posterior distribution $\pi \left(\theta \theta |yy\right)$.

The next stage is to predict a driver flow $\tilde{y}$ in the future from the observed past driver flows $yy$. Bayesian prediction is based on the posterior predictive distribution of $\tilde{y}|yy$. Its density $p\left(\tilde{y}|yy\right)$ is given by

$$p\left(\tilde{y}|yy\right)={\int }_{\mathrm{\Theta }}p(\tilde{y}|\theta \theta ,yy)\pi \left(\theta \theta |yy\right)\mathrm{d}\theta \theta$$ (5)

where $\mathrm{\Theta }=(0,1{)}^5\times {\mathbb{R}}_{+}$. Since $p\left(\tilde{y}|yy\right)$ is a compound probability distribution, we can simulate samples from this predictive distribution.

For the choice of an MCMC sampler, we use the NUT sampler [11], which is the default in the pyStan Python package (https://pystan.readthedocs.io). This package is an interface to the state-of-art platform for Bayesian computations Stan (https://mc-stan.org). To carry out the integration and then a random draw from the posterior predictive distribution of daily driver flows $p\left(\tilde{y}|yy\right)$ in Equation (4), we are only required to input the prior $\pi \left(\theta \theta \right)$, the likelihood $L\left(yy|\theta \theta \right)$ and the recurrence relation which generates the vector of simulated driver flows $yy$ (Algorithm 2 in Appendix A.2) into pyStan. The latter automatically simulates for the $j^{th}$ iteration, $j=1,\dots ,J$,

$${\widetilde{yy}}^{\left(j\right)}=\left[\begin{array}{c}{\tilde{y}}^{(j,1)}\\ ⋮\\ {\tilde{y}}^{(j,N)}\end{array}\right]\sim \left[\begin{array}{c}p\left({\tilde{y}}^{(j,1)}|yy\right)\\ ⋮\\ p\left({\tilde{y}}^{(j,N)}|yy\right)\end{array}\right],$$ (6)

and its final output is the sequence of posterior prediction vectors $\tilde{\mathbb{Y}}=\{{\widetilde{yy}}^{\left(1\right)},\dots ,{\widetilde{yy}}^{\left(J\right)}\}$.

2.2. Gamma regression for passenger waiting times

For simplicity, we assume that a passenger can only make one request at a time for themselves only at a carpooling meeting point, and the drivers can embark only one passenger in their vehicle in the order that the passenger requests are made. For day i, let $y_i$ be the daily traffic flow, and that $n_i$ passengers make carpooling requests at times $t_{i,1}<\cdots <t_{i,n_i}$. Let $t_{i,j}^{\mathrm{\prime }}$ be the driver arrival time for the passenger request at time $t_{i,j},i=1,\dots ,N$ and $j=1,\dots ,n_i$. The perceived waiting time $w_{i,j}^{\ast }$ and the pseudo waiting time $w_{i,j}$ for the passenger request at time $t_{i,j}$ are

$$\begin{array}{rl}{w}_{i,j}^{\ast }& =t_{i,j}^{\mathrm{\prime }}-t_{i,j}\\ w_{i,j}& =t_{i,j}^{\mathrm{\prime }}-max(t_{i,j},t_{i,j-1}^{\mathrm{\prime }})\end{array}$$

with the convention $t_{i,0}^{\mathrm{\prime }}=t_{i,1}$ for the first passenger on day i. Figure 4 illustrates the difference between the perceived and the pseudo waiting times for two passengers A, B who are both not the first passenger of the day. Passenger A arrives first and is the $j^{th}$ passenger of day i, and makes a carpooling request at time $t_{i,j}$. Passenger B arrives immediately afterwards and is the $(j+1{)}^{th}$ passenger with request time $t_{i,j+1}$. Suppose that there are at least two drivers en route to embark these passengers, and they have not received any passenger requests before passenger A's request. The first driver arrives at $t_{i,j}^{\mathrm{\prime }}>t_{i,j+1}$ (i.e. after passenger B's request time) and the second driver at $t_{i,j+1}^{\mathrm{\prime }}>t_{i,j}^{\mathrm{\prime }}$. The perceived waiting time for the passenger A is $w_{i,j}^{\ast }=t_{i,j}^{\mathrm{\prime }}-t_{i,j}$ (the blue brace in Figure 4) and for the passenger B it is $w_{i,j+1}^{\ast }=t_{i,j+1}^{\mathrm{\prime }}-t_{i,j+1}$ (the green brace). The pseudo waiting time for passenger A is $w_{i,j}=w_{i,j}^{\ast }$ (the blue brace) since they are at the front of the queue, and for passenger B it is $w_{i,j+1}=t_{i,j+1}^{\mathrm{\prime }}-t_{i,j}^{\mathrm{\prime }}$ (the grey brace). The pseudo waiting time $w_{i,j+1}$ for passenger B is the difference between their departure time and the departure time of the previous passenger A, and this is shorter than the perceived waiting time $w_{i,j+1}^{\ast }$.

Figure 4.

Perceived and pseudo waiting times for the case of two passengers at a carpooling meeting point. Passenger A is at the head of the queue so their perceived waiting time (blue brace) coincides with their pseudo waiting time. For passenger B, their pseudo waiting time (grey brace) is the difference between their departure time and the departure time of passenger A, which is shorter than their perceived waiting time (green brace).

From Figure 4, we observe that the perceived the waiting times $w_{i,j}^{\ast }$ and $w_{i,j+1}^{\ast }$ for passengers A and B overlap, whereas the pseudo waiting times $w_{i,j}$ and $w_{i,j+1}$ do not overlap by construction. The overlapping nature of the interval processes that determine the perceived waiting times renders the problem of their unconditional prediction to be non-identifiable. Thus we focus on the pseudo waiting times, and we wish to formulate sub-daily predictions of them.

2.2.1. Modelling for pseudo waiting times

Let the 24 hour period of a day be divided into S equal intervals $I_1<\cdots <I_S$. The fraction of the daily driver flow $y_i$ on each interval $I_s,s=1,\dots ,S$ is $y_i{\mathrm{\text{β}}}_s$, where ${\mathrm{\text{β}}}_s\ge 0$ and $\sum _{s=1}^S{\mathrm{\text{β}}}_s=1$. Conditional on the traffic flow $y_i$ and the passenger request times $t_{i,j}\in I_s$, we suppose that the pseudo waiting times $w_{i,j}$ are independent Gamma random variables with parameters ν and ${\mathrm{\text{β}}}_s{y}_i$, i.e.

$$w_{i,j}|(y_i,\mathrm{\text{β}}\mathrm{\text{β}},t_{i,j}\in I_s)\sim \mathrm{\Gamma }(\nu ,{\mathrm{\text{β}}}_s{y}_i)$$ (7)

for $i=1\dots N$ and $j=1,\dots ,n_i$. This Gamma regression model assumes that the pseudo waiting times depend on the time of day and on the daily driver flow. Furthermore, it ensures that the conditional mean pseudo waiting time is

$$\mathbb{E}\left[w_{i,j}|\right(y_i,\mathrm{\text{β}}\mathrm{\text{β}},t_{i,j}\in I_s\left)\right]=\frac{\nu }{{\mathrm{\text{β}}}_s{y}_i}$$

which is consistent with our intuition of the inverse relationship between the driver flow and the waiting time. Since $\mathrm{\text{β}}\mathrm{\text{β}}$ is constant for all i, then the model assumes that the relative proportions of the driver flow in the intervals $I_1,\dots ,I_S$ remain unchanged for all aggregate daily driver flows.

A Dirichlet distribution is a natural choice as a prior distribution on the coefficients $\mathrm{\text{β}}\mathrm{\text{β}}$: $\mathrm{\text{β}}\mathrm{\text{β}}\sim \mathrm{D}\mathrm{i}\mathrm{r}(S,\alpha \alpha )$ where $\alpha \alpha =({\alpha }_1,\dots ,{\alpha }_S)$ are the concentration parameters, since it imposes the constraint $\sum _{s=1}^S{\mathrm{\text{β}}}_s=1$ on the coefficients. The corresponding Dirichlet density is $p\left(\mathrm{\text{β}}\mathrm{\text{β}}\right)=\left[\prod _{s=1}^S{\mathrm{\text{β}}}_s^{{\alpha }_s-1}\right]/\mathrm{B}\left(\alpha \alpha \right)$ where $B\left(\alpha \alpha \right)=\prod _{s=1}^S\mathrm{\Gamma }\left({\alpha }_s\right)/\mathrm{\Gamma }\left(\sum _{s=1}^S{\alpha }_s\right)$ and $\mathrm{\Gamma }\left(x\right)={\int }_0^{\mathrm{\infty }}{u}^{x-1}{e}^{-u}du$. The $\mathrm{\text{β}}\mathrm{\text{β}}$ vector allows us to rebuild the temporal distribution of the sub-daily traffic flow from an aggregated daily driver flow.

Let $t{t}_i=(t_{i,1},\dots ,t_{i,n_i})$ be the vector of the $n_i$ observed passenger carpooling request times for the day $i\in \{1,\dots ,N\}$, and $tt=(t{t}_1,\dots ,t{t}_N)$ be all observed passenger carpooling request times. Likewise for the passenger pseudo waiting times $w{w}_i$ for day i, and $ww$ for all days. Let $yy=(y_1,\dots ,y_N)$ be the observed driver flows for all days. It is reasonable to assume that the waiting times are mutually independent given $(\mathrm{\text{β}}\mathrm{\text{β}},yy,tt)$. The conditional likelihood of the pseudo waiting times is thus given by the joint density of $ww$ given $(\mathrm{\text{β}}\mathrm{\text{β}},yy,tt)$

$$L(ww|\mathrm{\text{β}}\mathrm{\text{β}},yy,tt)=\prod _{i=1}^{N}p(w{w}_i|\mathrm{\text{β}}\mathrm{\text{β}},yy,tt)=\prod _{i=1}^{N}p(w{w}_i|\mathrm{\text{β}}\mathrm{\text{β}},y_i,t{t}_i)$$

since $p(w{w}_i|\mathrm{\text{β}}\mathrm{\text{β}},y_i,t{t}_i)=\prod _{s=1}^S\prod _{\{j:t_{i,j}\in I_s\}}({\mathrm{\text{β}}}_s{y}_i{)}^{\nu }{w}_{i,j}^{\nu -1}\exp (-{\mathrm{\text{β}}}_s{y}_i{w}_{i,j}\left)/\mathrm{\Gamma }\right(\nu )$. Then we obtain the posterior density of $\mathrm{\text{β}}\mathrm{\text{β}}$, using a non-informative prior on $\mathrm{\text{β}}\mathrm{\text{β}}$, as

$$\pi (\mathrm{\text{β}}\mathrm{\text{β}}|yy,tt,ww)\propto \prod _{i=1}^N\prod _{s=1}^S\prod _{\{j:t_{i,j}\in I_s\}}\frac{({\mathrm{\text{β}}}_s{y}_i{)}^{\nu }}{\mathrm{\Gamma }\left(\nu \right)}{w}_{i,j}^{\nu -1}\exp (-{\mathrm{\text{β}}}_s{y}_i{w}_{i,j})\mathbf{1}\{\mathrm{\text{β}}\mathrm{\text{β}}\in {\mathbb{R}}_{+}^S\}.$$

Let ${\tilde{w}}_s$ be the pseudo waiting time for a future day for a passenger who makes a carpooling request in the time interval $I_s$. If we observe a new daily driver flow $\tilde{y}$ on this future day, then the posterior predictive distribution of the waiting time ${\tilde{w}}_s$ is

$$p({\tilde{w}}_s|\tilde{y},yy,ww)={\int }_0^{1}p({\tilde{w}}_s|\tilde{y},{\mathrm{\text{β}}}_s)\pi ({\mathrm{\text{β}}}_s|yy,ww,tt)\mathrm{d}{\mathrm{\text{β}}}_s,$$ (8)

which is then collated into an S-vector $\left(p\right(\text{∼}{w}_1|\tilde{y},yy,ww),\dots ,p(\text{∼}{w}_S|\tilde{y},yy,ww\left)\right)$ for all time intervals $I_1,\dots ,I_S$.

To carry out the integration and then a random draw from the posterior predictive distribution of $p(\text{∼}{w}_s|\tilde{y},yy,ww)$ in Equation (8), we are only required to input the posterior predicted value of the driver flow $\tilde{y}$ Equation (5), the recurrence relation which generates the vector of simulated driver flows $yy$ (Algorithm 2 in Appendix A.2), and the recurrence relation which generates the vector of simulated passenger pseudo waiting times $ww$ (Algorithm 3 in Appendix A.2) into pyStan. The latter automatically simulates this $N\times S$ matrix distribution, for the $j^{th}$ iteration, $j=1,\dots ,J$,

$${\widetilde{WW}}^{\left(j\right)}=\left[\begin{array}{ccc}{\tilde{w}}_{1,1}^{\left(j\right)}& \dots & {\tilde{w}}_{1,S}^{\left(j\right)}\\ ⋮& & ⋮\\ {\tilde{w}}_{1,N}^{\left(j\right)}& \dots & {\tilde{w}}_{N,S}^{\left(j\right)}\end{array}\right]\sim \left[\begin{array}{ccc}p({\tilde{w}}_1^{(j,1)}|\tilde{y},yy,ww)& \dots & p({\tilde{w}}_S^{(j,1)}|\tilde{y},yy,ww)\\ ⋮& & ⋮\\ p({\tilde{w}}_1^{(j,N)}|\tilde{y},yy,ww)& \dots & p({\tilde{w}}_S^{(j,N)}|\tilde{y},yy,ww)\end{array}\right]$$

and its final output is the sequence of posterior prediction matrices $\stackrel{\mathbb{~}}{\mathbb{W}}=\{\tilde{W}{\tilde{W}}^{\left(1\right)},\dots ,\tilde{W}{\tilde{W}}^{\left(J\right)})\}$.

2.2.2. Modelling for perceived waiting times

As for now, we have focused on the pseudo waiting times $w_{i,j}$, though for the applications the perceived waiting times $w_{i,j}^{\ast }$ are more pertinent, since the latter are the true waiting times from the passenger point-of-view. The current set-up of the Lane carpooling service is not able to collect reliable perceived waiting times since the passenger arrivals are not reliably tracked. Also, as alluded to earlier, the unconditional prediction of the perceived waiting times is a non-identifiable problem.

However, we can provide a framework for their analysis based on conditioning on simulated passenger arrival processes. Since the complete posterior predictive distribution in Equation (8) is available, we can integrate it with respect to the passenger arrival distribution to obtain the pseudo waiting time distribution. Then we are able to reconstruct the perceived waiting times using $w_{i,j+1}^{\ast }=w_{i,j+1}+[w_{i,j}-{\zeta }_i|(w_{i,j}>{\zeta }_i\left)\right]$ with ${\zeta }_i=t_{i,j+1}-t_{i,j}$. Let the passenger arrivals be a Poisson process, for t>0, $N\left(t\right)=max\{n:\sum _{k=0}^n{A}_k\le t\}$ where $N\left(0\right)=0$, $A_0=0$, and $A_k\sim \mathcal{E}\left(\lambda \right)$ and $\lambda >0$ are independent exponential random variables. The parameter λ is the rate of the passenger arrivals, and it measures the mean number of arrivals over a unit of time. An application of this methodology is given in Section 4.3.

3. Model validation with simulated pseudo waiting times

We choose parameter values to produce simulated data which are comparable to those observed in the Lane carpooling service. We set the initial day i = 1 to be 2018-01-01, and the weekdays (ORD), school holidays (SCH) and public holidays/weekends (PWE) to be those observed in south-eastern France. For the simulation algorithms, the number of days is N = 365, the day types coefficients are $\theta \theta =(0.333,0.33,0.331,1,1,1)$, the autoregression order is K = 3, the error variance is ${\sigma }_{ϵ}^2=5$, the 24 hour period is divided in S = 8 equal intervals of 3 hours, the first Gamma shape parameter is $\nu =7$, the Gamma regression parameters are $\mathrm{\text{β}}\mathrm{\text{β}}=$ (0.012, 0.01, 0.011, 0.013, 0.018, 0.016, 0.017, 0.019), and the number of replicates (waiting times per time interval) is J = 10.

We generate one simulated data set of N = 365 days, each with one daily driver flow $y_i,i=1,\dots ,N$ (Algorithm 2), and J = 10 passenger pseudo waiting time $N\times S$ matrices $\mathbb{W}=\{W{W}^{\left(1\right)},\dots ,W{W}^{\left(J\right)}\}$ (Algorithm 3), and the corresponding $N\times S$ posterior prediction matrices $\tilde{\mathbb{W}}=\{{\widetilde{WW}}^{\left(1\right)},\dots ,{\widetilde{WW}}^{\left(J\right)}\}$. The data from these N = 365 days from 2018-01-01 to 2018-12-31 form the reference training data set. With the same parameters, we simulate a further $\tilde{N}=5$ days (2019-01-01 to 2019-01-05) as the oracle test data set of $\tilde{N}\times S$ matrices ${\mathbb{W}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}=\{W{W}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\left(1\right)},\dots ,W{W}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\left(J\right)}\}$. Furthermore, from the training data only, for these same extra $\tilde{N}$ days, we generate the corresponding $\tilde{N}\times S$ posterior prediction matrices ${\tilde{\mathbb{W}}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}=\{{\widetilde{WW}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\left(1\right)},\dots ,{\widetilde{WW}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}^{\left(J\right)}\}$. For brevity we have omitted the comparison of the driver flows and focus on the passenger waiting times for these simulated data: we make a more thorough comparison of both driver flows and passenger waiting times for the empirical data in the sequel.

From a passenger point of view, whilst the magnitude of the waiting times are important as a perception of the service quality, it is equally important that these posterior predicted waiting times be as close to the observed ones, whatever their magnitude. For example, suppose that a driver arrives 12 minutes after a passenger makes a carpooling request. In this case, a prediction of 15 minutes is better than 5 minutes since the former is closer to the observed waiting time than the latter (which is too optimistic). Therefore we propose the following metric to measure these discrepancies for a given threshold δ:

$$\mathrm{P}\mathrm{E}(\mathbb{W},\tilde{\mathbb{W}};\delta )=\frac{1}{J\tilde{N}S}\sum _{i=1}^{\tilde{N}}\sum _{s=1}^S\sum _{j=1}^J\mathbf{1}\{|{\overline{\tilde{w}}}_{i,s}-w_{i,s}^{\left(j\right)}|<\delta \}$$ (9)

where ${\tilde{w}}_{i,s}=\frac{1}{J}\sum _{j=1}^J{\tilde{w}}_s^{(j,i)}$ is the mean of the posterior predicted waiting times distribution for day i, and time interval $I_s$.

We focus on the temporal profiles, over the S = 8 periods of a day, of the waiting times. In Figure 5 are the quantiles of the waiting times for all time intervals $I_s,s=1,\dots ,8$, for all days $i=1,\dots ,\tilde{N}$ in the test phase. The grey box plots are of the observations ${\mathbb{W}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t},i,s}$ and the light, medium and dark purple circles superimposed over the box plots are the 50%, 75%, 95% quantiles of the posterior predictions ${\tilde{\mathbb{W}}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t},i,s}$. For operational purposes, short term prediction for the coming week is sufficient, and this is verified by the close agreement of the quantiles of the posterior predicted pseudo waiting times with their observed values for all $\tilde{N}=5$ prediction days. The advantage of an MCMC approach here is that we are able to reproduce the entire sampling distribution of the predicted waiting times, which is more comprehensive than point or interval predictions.

Figure 5.

Predictions of simulated pseudo waiting times (in minutes) for 3-hourly intervals, for all $\tilde{N}$ prediction days. The observed waiting times are the grey box plots, and the 50%, 75%, 95% quantiles of the posterior predicted waiting times are the light, medium and dark purple circles.

The PE metric from Equation (9), as a function of δ, illustrated in Figure 6, during both the training phase $\mathrm{P}\mathrm{E}(\mathbb{W},\tilde{\mathbb{W}};\delta )$ (blue curve) and the test phase $\mathrm{P}\mathrm{E}({\mathbb{W}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}},{\tilde{\mathbb{W}}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}};\delta )$ (red curve). The test predictions are more accurate than the training predictions for small values of $\delta <4$ minutes since red PE curve is above the blue PE curve in this interval. This reverses for δ greater than 4 minutes, and after 12 minutes, both curves level off at 1. Thus the posterior predictions from our proposed Bayesian hierarchical model can have robust prediction performance according to this PE metric.

Figure 6.

Evolution of the PE metric of simulated and posterior predicted pseudo waiting times, as a function of the threshold δ. The blue curve is for the training phase, and the red curve for the test phase.

4. Model validation with the lane carpooling service

Our objective is to employ the two-stage Bayesian hierarchical model to predict the daily driver flow distribution and the passenger pseudo waiting time distribution for the hourly intervals $I_s,s=1,\dots ,S$, with S = 24 for the upcoming week. These are then compared to the observed driver flows and the pseudo waiting times from the same period.

4.1. Daily driver flows

We have approximately 5000 GPS traces for the 382 days from 2018-05-15 to 2019-05-31. We first apply the preprocessing, as outlined in Appendix A.1, to convert the driver GPS traces into a format suitable for computing the daily driver flows $y_i$. For the driver flow moving average model in Equation (2), the $\theta \theta$ coefficient has a different value for each day type, since these day types are a key determinant of home-work daily commutes. This is verified empirically by the box plots of the daily driver flow in Figure 7. The daily driver flow for an ordinary weekday (ORD) approaches 150 trajectories, which is about double the driver flow on school holidays (SCH), and more than 20 times larger than on the public holiday/weekends (PWE).

Figure 7.

Aggregate driver flow by day type, for the Lane carpooling service from 2018-05-15 to 2019-05-31. The ordinary weekdays (ORD) are in orange, the school holidays (SCH) in green and the public holidays/weekend days (PWE) in blue.

We divide the observed driver flows $y_i$ into 6 different pairs of training phases, starting from 2018-05-15 and with varying N, and test phases with $\tilde{N}=7$. In each case, we select a test week with certain characteristics as outlined in Table 1. The first column are the dates of the test week, the second column are the day types in the test week, the third column are the dates of the training weeks, and the fourth column is the number of training days (N). For these training-test scenarios, in addition to our proposed Bayesian hierarchical multi-level (BHML) predictions, we compute predictions from a baseline frequentist model (BASE), and a Bayesian Prophet model (PROP). The details of these competing models are described in Appendix A.3. We input the daily driver flows into the first hierarchical model from the Bayesian hierarchical multi-level model BHML to produce the posterior predicted daily driver flows ${\tilde{y}}_i$, as well the corresponding predictions/estimations from the frequentist baseline model BASE and the Bayesian Prophet model PROP.

Table 1.

Training-test scenarios for daily driver flows. The first column are the dates of the test week ( $\tilde{N}=7$), the second is the day types in the test week, the third are the dates of the training weeks and the fourth column is the number of training days N.

	Test week	Test week day types	Training weeks	#training days (N)
#1	2019-01-14 – 2019-01-20	[l]All ORD after holiday period (PWE/SCH)	2018-05-15 – 2019-01-13	244
#2	2019-02-25 – 2019-03-03	All SCH	2018-05-15 – 2019-02-24	286
#3	2019-04-29 – 2019-05-05	[l]All ORD except 1 PWE (2019-05-01)	2018-05-15 – 2019-04-28	349
#4	2019-05-06 – 2019-05-12	[l]All ORD except 1 PWE (2019-05-08)	2018-05-15 – 2019-05-05	356
#5	2019-05-13 – 2019-05-19	[l]All ORD except 1 PWE (transport strike 2019-05-16)	2018-05-15 – 2019-05-12	363
#6	2019-05-20 – 2019-05-26	All ORD	2018-05-15 – 2019-05-19	370

For the Test scenario # 6, the training phase covers the dates 2018-05-15 to 2019-05-19. In Figure 8 is the evolution of the goodness-of-fit of the three different models for daily driver flow estimation (leaving out the first week 2018-05-15 to 2018-05-21 which serves as the ‘burn-in’ period). The goodness-of-fit is measured by the MSE of the estimated and the observed daily driver flows, aggregated per week. Visually the BHML tends to have the best goodness-of-fit (smallest MSE) for most weeks. The sum of these weekly MSEs are BASE: 421.9, PROP: 816.9, BHML: 297.2, which confirms our visual impression that the BHML achieves the best overall estimation accuracy.

Figure 8.

Evolution of the goodness-of-fit of the daily driver flow estimations over the training period (2018-05-15 to 2019-05-19, test scenario # 6). Goodness-of-fit is measured by the weekly aggregated estimation MSE. Bayesian hierarchical multi-level BHML is in purple, frequentist baseline BASE in black, and Bayesian Prophet PROP in green.

Therefore we can be confident that the Bayesian hierarchical multi-level moving average model has good estimation accuracy/goodness-of-fit, but this good performance does not necessarily translate to prediction [15]. So for each scenario described in Table 1, we compute the BHML, BASE and PROP models for the training phase, and then the days of the test phase are input into each these training models to yield the daily driver flow predictions. In Figure 9 are the MSEs between the observed and predicted daily driver flows: the frequentist baseline model BASE in black, the Bayesian Prophet PROP in green, and the Bayesian hierarchical multi-level BHML in purple. The predicted driver flows themselves are presented in Figure A1 in Appendix A.4. Overall the BHML has the best prediction accuracy for all test week scenarios. PROP is the uniformly the worst of these three models for all test weeks. BASE is the best for the test scenario #1 (all ORD after PWE/SCH period) and #6 (all ORD) with almost zero prediction MSE, though the difference with BHML is not so large. These two test scenarios are where all days in the test week are the same day type. For the other test week scenarios #1, #3, #4, #6, BHML has the smallest prediction MSE, some times by a large margin. These test week scenarios include a day which is a different day type to the other days within the test week, which the BHML handles the best.

Figure 9.

Prediction MSE of the daily driver flow predictions for the six test week scenarios. Bayesian hierarchical multi-level BHML are in purple, frequentist baseline BASE in black, and Bayesian Prophet PROP in green.

For the service operator, the sharp differences in the driver flow for different day types within the same week has operational repercussions. For example, since the driver flow is consistently low for all public holidays, the service operator must communicate to passengers that the service quality on a public holiday is not the same as that for an ordinary weekday. This is analogous to a public holiday schedule in lieu of a usual weekday schedule provided by a bus operator. On a more positive note, since the driver flow for weekdays on either side of the public holiday is similar to other weekdays further away, then the service operator can also communicate that this temporary reduction in service quality is limited to the public holiday itself and the usual weekday service level can be assured on the preceding and following weekdays.

4.2. Temporal profiles of passenger pseudo waiting times

For the passenger pseudo waiting time Gamma regression, the $\mathrm{\text{β}}\mathrm{\text{β}}$ coefficient, which determines the intra-day distribution of the waiting times, is considered to be constant for all days (see Figure A2 in Appendix A.4 for an illustration of the mean observed daily traffic flows for each weekday from the Lane carpooling service). We observe that each week day has a similar shape so this gives some empirical justification for supposing a constant $\mathrm{\text{β}}\mathrm{\text{β}}$ for all days. For the service operator, this means that it can treat all non-public holiday weekdays as similar to each other.

Before we examine the predictions from this Gamma regression model, we provide some heuristic justification of the model itself, namely concerning the choice of the Gamma distribution and the conditioning of the waiting times with respect to the driver flow (see Figure A3 in Appendix A.4 for a visual justification of the fits of these empirical waiting times to Gamma distributions). As expressed in Equation (7), the pseudo waiting times are represented by different Gamma distributions for each hourly interval, which implies that the mean pseudo waiting times are decreasing functions of the driver flows. In Figure 10, we have divided the observed daily driver flows into three categories: low (<100 vehicles per day), medium (100–200 vehicles per day), and high (>200 vehicles per day). Overall we observe that the mean waiting time is inversely proportional to the driver flow level.

Figure 10.

Mean pseudo waiting times per hourly intervals as a function of driver flow from 2019-07-25 to 2020-02-17. Grey: low (<100 vehicles per day), brown: medium (100–200 vehicles per day), and green: high (>200 vehicles per day).

Now that we have verified that a Gamma regression model is suitable for the data observed in the Lane carpooling service, we proceed with the BHML to form predictions of the passenger pseudo waiting times. Since there are insufficient passenger carpooling requests to robustly compute observed hourly waiting time profiles over an entire day for the school holidays (SCH) and the public holidays/weekends (PWE), we restrict ourselves to forming predictions for the weekdays (ORD). In Figure 11 are the box plots of the weekly number of observed pseudo waiting times for each hourly interval for the weekdays from 2019-07-25 to 2020-02-17. Although there are S = 24 hourly intervals, only those 6 which correspond to the service operating hours (06:00–09:00 and 16:00–19:00) contain any observed passenger waiting times.

Figure 11.

Box plots of the weekly number of observed pseudo waiting times for each hourly interval for weekdays from 2019-07-25 to 2020-02-17.

There are a maximum of around 40 observed waiting times per hourly interval per week, which are not sufficient to infer robustly their distribution within each interval. To remedy this data sparsity, we aggregate a moving window of test data so for time interval $I_s$ on day i, we combine its observed pseudo waiting times $w_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t},i,s}$ with those for the same time interval from the previous 5 weeks with the same day of week and same day type, i.e. $\{w_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t},i-k,s}:\mathrm{DT}(i-k)=\mathrm{DT}(i),\mathrm{DN}(i-k)=\mathrm{DN}(i),k=1,\dots ,35\}$. These days added to the test data are correspondingly removed from the training data. We aggregate the final 5 weeks to be a single test phase, so the Test scenario # 7 is composed of training weeks (2019-07-25 – 2020-01-12) with 1289 observed training waiting times, and test weeks (2020-01-13 – 2020-02-17) with 520 observed test waiting times. We make predictions for only the last test week (2020-02-10 – 2020-02-17), so the number of prediction weekdays remains $\tilde{N}=5$.

In Figure 12 are the box plots of the observed pseudo waiting times and the quantiles for the posterior predictions, for the hourly intervals for the Test scenario # 7. The observed pseudo waiting times are displayed as the grey box plots, and the 50%, 75%, 95% quantiles of the posterior predicted waiting times are the light, medium and dark purple circles. The advantage of the BHML is that we have the entire sampling distribution of the predicted waiting times, which is more comprehensive than point or interval predictions of the usual regression models. The median and upper quartile of the predicted pseudo waiting times tend to track those for the observed waiting times, especially for the 06:00–07:00, 17:00–18:00 and 18:00–19:00 intervals. From anecdotal evidence provided by Ecov, 15 minutes corresponds roughly to the maximum time that passengers are willing to wait for a driver to arrive if a pre-arranged meeting time has not been made. With the BHML predictions, we can assert that 95% of the waiting times for passenger requests do not exceed this 15 minutes threshold during most of the operating hours. Whilst this information could also be established with the empirical quantiles of the observed waiting times, the advantage of the BHML is that it gives a more solid basis that this performance will continue into the future.

Figure 12.

Predictions of passenger pseudo waiting times (PP PWT) for hourly time intervals for the Test scenario # 7. The observed waiting times are the grey box plots, and the 50%, 75%, 95% quantiles of the posterior predicted waiting times are the light, medium and dark purple circles.

Lastly we consider our custom PE metric from Equation (9) on the BHML posterior predictions. This metric is illustrated in Figure 13, for both the training phase $\mathrm{P}\mathrm{E}(\mathbb{W},\tilde{\mathbb{W}};\delta )$ (blue curve) and the test phase $\mathrm{P}\mathrm{E}({\mathbb{W}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}},{\tilde{\mathbb{W}}}_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}};\delta )$ (red curve). The blue curve dominates the red curve for most values of δ. This implies that the posterior predictions are more accurate during the test phase than in the training phase. This gives us confidence that the BHML posterior predictions are robust and are not based on over-fitting on the training data.

Figure 13.

Evolution of the PE metric of observed and BHML posterior predicted passenger pseudo waiting times, as a function of the threshold δ. The blue curve is for the training phase, and the red curve for the test phase.

For the service operator, the BHML implies that the key factor in determining the passenger waiting time (i.e. the output from the second stage) is the driver flow (i.e. the output from the first stage). It is thus imperative that a consistent level of driver participation in the carpooling service is maintained so that consistent waiting times can be provided to passengers. Since they are non-professional drivers, then non-monetary incentives are crucial in maintaining their participation in a stochastic carpooling service [22,23].

In contrast to the comparison of the predicted driver flows from the BHML to those from competing models in the previous section, the comparison of the Gamma regression with other possibilities in the second stage of the BHML is not considered here. According to Papoutsis et al. [16], the main predictor for the passenger waiting times is the driver flow, and so we conjecture that the choice of the passenger waiting times prediction model is of secondary importance with respect to the choice of the driver flow prediction model.

4.3. Temporal profiles for passenger perceived waiting times

Following the methodology described in Section 2.2.2 we propose the method below in order to reconstruct the perceived waiting times. In our case, the unit of time is one hour and we focus on the opening hours of the service (I₁ = 6:00–7:00, I₂ = 7:00–8:00, I₃ = 8:00–9:00, I₄ = 16:00–17:00, I₅ = 17:00–18:00, I₆ = 18:00–19:00). The base passenger arrival rate is denoted by ${\lambda }_1$, as shown in Table 2. The two other scenarios involve ${\lambda }_2$, an increase by 50%, and ${\lambda }_3$, an increase of 100%.

Table 2.

Poisson passenger arrival rates per hourly intervals. ${\lambda }_1$ is the base passenger arrival rate, ${\lambda }_2$ is an increase by 50% and ${\lambda }_3$ is an increase of 100%.

	Morning intervals			Evening intervals
	06:00–07:00	07:00–08:00	08:00–09:00	16:00–17:00	17:00–18:00	18:00–19:00
${\lambda }_1$	8	6	4	6	4	4
${\lambda }_2$	12	9	6	9	6	6
${\lambda }_3$	16	12	8	12	8	8

For the base passenger arrival scenario, there are few situations where the passenger requests overlap each other, and so there is little difference between the pseudo and the perceived waiting times. The second and third scenarios with increased rates of passenger arrivals lead to increased overlapping passenger requests. This is of intense interest to the service operator because it informs them how the passenger perceived waiting times respond to the increased passenger requests whilst maintaining the current driver flow. In Figure 14 are the box plots for the pseudo and perceived waiting times for the three passenger arrival scenarios from Table 2 with a constant driver flow. The pseudo waiting times of the left panel are similar for the three scenarios since they do not account for overlapping passenger arrivals. On the other hand, we observe that the perceived waiting times on the right panel tend to increase as the number of passengers arriving increases. For ${\lambda }_2$ with a 50% increase in the passenger arrivals, the perceived waiting times remain acceptable for a stochastic carpooling service (median less than 15 minutes). However for ${\lambda }_3$ with a 100% increase in the passenger arrivals, the perceived waiting times exceed 15 minutes for many passengers. For the service operator to reduce the waiting times, the driver flow must be increased by increasing the driver participation rate.

Figure 14.

Evolution of passenger waiting times as a function of increased passenger arrival rates. Perceived waiting times (light grey), pseudo waiting times (purple). ${\lambda }_1$ is the base passenger arrival rate, ${\lambda }_2$ is an increase by 50% and ${\lambda }_3$ is an increase of 100%.

5. Conclusions

The main contribution of this paper is the prediction of the daily driver flows and the hourly passenger waiting times using a nested two-stage Bayesian hierarchical model. The first stage is a multi-level moving average model of the daily driver flows, where the multi-level coefficient depends on if the current day is a work day, a school holiday or a public holiday/weekend. The second stage is a Gamma regression where the covariates are the daily driver flows from the first stage, and the response variables are the hourly passenger waiting times. The predicted driver flows and passenger waiting times are robust going into the future, since we demonstrated that they are not due to over-fitting. Furthermore, since we analyse the data from an operational carpooling service, we are able to provide operational advice. For the service operator, the baseline frequentist model is the simplest to implement, and so may be sufficient under certain cost-benefit scenarios. However only the more complex BHML can be utilised for more in-depth data analysis, such as quantiles and confidence regions of passenger waiting times, and for forward planning with the effect of increased passenger requests on waiting times.

We focused on modelling the driver arrival processes and assumed the passenger arrivals to be non-random in the first stage, and on pseudo waiting times in the second stage. One main advantage of the Bayesian hierarchical framework is that it is straightforward to generalise any of the models in the constituent stages (i) to allow the passenger arrivals to also be a random process, and (ii) to predict both the pseudo and perceived passenger waiting times. These perceived waiting times are of intense operational interest to stochastic carpooling service providers.

Appendix 1.

A.1. GPS traces pre-processing

A GPS trace is an ℓ-sequence of triplets $XX=\left\{\right(X_i,Y_i,T_i){\}}_{i=1}^{\ell }$ where $(X_i,Y_i)$ are the longitude, latitude coordinates of the GPS sensor at the $i^{th}$ timestamp $T_i$. The m pick-up/drop-off locations in the carpooling network are represented by their GPS coordinates $M{M}_1,\dots ,M{M}_m$. Around each of the m pick-up/drop-off locations $M{M}_1,\dots ,M{M}_m$, a ball of 1 km radius is drawn to obtain $B\left(M{M}_1\right),\dots ,B\left(M{M}_m\right)$. The intersection of these balls and the GPS trace, $XX\cap B\left(M{M}_1\right),\dots ,XX\cap B\left(M{M}_m\right)$, is m sub-sequences of the GPS points of $XX$. For those pick-up/drop-off locations with non-empty intersections, we consider that the driver is able to collect a passenger at these points without onerous detours.

This only considers the spatial proximity of the driver to a passenger at a pick-up/drop-off location. For the carpooling to succeed, they also need to be also in temporal proximity. Among the spatial intersections $XX\cap B\left(M{M}_1\right),\dots ,XX\cap B\left(M{M}_m\right)$, we examine the corresponding timestamps and retain only those in a suitably restrained time interval. If this set of spatio-temporal intersections is non-empty then we proceed to compute the closest GPS points in $XX$ to the pick-up/drop-off locations $M{M}_j$, as defined by $X{X}_{M{M}_j}=\left\{\right(X_k,Y_k,T_k):k={\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{i}\mathrm{n}}_{1\le i\le \ell }\Vert ((X_i,Y_i)-M{M}_j\Vert \},j=1,\dots ,m$. From this closest point $X{X}_{M{M}_j}$, we extract the corresponding timestamp $T_k$ to be an estimate of the driver arrival time at $M{M}_j$.

As an example, suppose that there two pick-up/drop-off points $M{M}_1,M{M}_2$ at which the GPS trace $XX$ has well-defined estimated arrival times. Then the ℓ points of $XX$ can be reduced to the sequence of 4 points $\widetilde{XX}=\left\{\right(X_1,Y_1,T_1)>X{X}_{M{M}_1}>X{X}_{M{M}_2}>(X_{\ell },Y_{\ell },T_{\ell }\left)\right\}$ where $(X_1,Y_1,T_1)$ is the driver origin and $(X_{\ell },Y_{\ell },T_{\ell })$ is the driver destination. With this simplified trace $\widetilde{XX}$, we are still able to determine if the driver can fulfil a passenger request at $M{M}_1$ for a trip going to $M{M}_2$ at time t. The complex topology of $XX$ is simplified by retaining a small number of key derived indicators [13].

A.2. Simulation algorithms

Algorithm 1 simulates a driver flow for a single day with no day types. Equation (2) with no day types simplifies to $y_i=\alpha \sum _{k=1}^K{y}_{i-k}+{ϵ}_i$, which is a true autoregressive model. The inputs are the day i, the coefficient α, the autoregression order K, and the error variance ${\sigma }_{ϵ}^2$. The output is a single driver flow for day i. The repeat loop ensures that the simulated driver flow is strictly positive. To simulate a sequence of N driver flows, we initialise the values generated by Algorithm 1 for $i=1,\dots ,K$ days, and then iterate Algorithm 1 sequentially for $i=K+1,\dots ,N$.

With Algorithm 1 defined, it is straightforward to define one with day types (i.e. Equation (2)) in Algorithm 2. The latter has similar inputs: the day i, the day type coefficients $\theta \theta$, the autoregression order K, the error variance ${\sigma }_{ϵ}^2$, and the vector coefficients $\theta \theta$. The output is the daily driver flow for day i, accounting for the day types before day i.

Algorithm 3 simulates the passenger pseudo waiting times in Equation (7) for a sequence of days. The inputs are the number of days N, the day type coefficients $\theta \theta$, the autoregression order K, the error variance ${\sigma }_{ϵ}^2$, the first shape parameter for the Gamma distribution ν, the S regression parameters $\mathrm{\text{β}}\mathrm{\text{β}}$, and the number of replicates of the waiting times J. The output are J replicates of a pseudo waiting time for each time interval $I_s,s=1,\dots ,S$, for each day $i=1,\dots ,N$. The TrafficFlowDT procedure (Algorithm 2) is called outside of the replicates loop since all waiting times on a given day are simulated from the same daily driver flow.

An iteration of the nested loop in Algorithm 3 in the Appendix results in a single $N\times S$ matrix of pseudo waiting times drawn from the appropriate Gamma distributions

$$W{W}^{\left(j\right)}\sim \left[\begin{array}{ccc}\mathrm{\Gamma }(\nu ,{\mathrm{\text{β}}}_1{y}_1)& \dots & \mathrm{\Gamma }(\nu ,{\mathrm{\text{β}}}_S{y}_1)\\ ⋮& & ⋮\\ \mathrm{\Gamma }(\nu ,{\mathrm{\text{β}}}_1{y}_N)& \dots & \mathrm{\Gamma }(\nu ,{\mathrm{\text{β}}}_S{y}_N)\end{array}\right].$$

for $j=1,\dots ,J$. These are collated into the sequence

$$\mathbb{W}=\{W{W}^{\left(1\right)},\dots ,W{W}^{\left(J\right)}\}=\{\left[\begin{array}{ccc}{w}_{1,1}^{\left(1\right)}& \dots & w_{1,S}^{\left(1\right)}\\ ⋮& & ⋮\\ w_{N,1}^{\left(1\right)}& \dots & w_{N,S}^{\left(1\right)}\end{array}\right],\dots ,\left[\begin{array}{ccc}{w}_{1,1}^{\left(J\right)}& \dots & w_{1,S}^{\left(J\right)}\\ ⋮& & ⋮\\ w_{N,1}^{\left(J\right)}& \dots & w_{N,S}^{\left(J\right)}\end{array}\right]\}.$$

As Equation (8) generates only a single posterior prediction ${\tilde{w}}_s$ for a time interval $I_s$, we collate these ${\tilde{w}}_s$ for $s=1,\dots S$ into an S-vector, and in turn collate N of these S-vectors of posterior prediction distributions row-wise into a $N\times S$ matrix.

A.3. Competing models for driver flows

In addition to the multi-level moving average model for driver flows, we consider a baseline frequentist model and a Bayesian Prophet model. The baseline frequentist model has multi-levels like our model, but without the Bayesian moving average structure. To account for the school/public holidays, as proposed by [10], if day i is not a school/public holiday then the average is calculated over all previous days with the same day of week as day i; and if day i is a school/public holiday, then the average is over all previous school/public holidays. That is,

$$y_i=\frac{1}{|T_d\left(i\right)|}\sum _{k\in T_d\left(i\right)}{y}_{i-k}\mathbf{1}\{{\mathrm{DT}}^{\mathrm{\prime }}(i)\ne \mathrm{H}\mathrm{O}\mathrm{L}\}+\frac{1}{|T_{\mathrm{H}\mathrm{O}\mathrm{L}}\left(i\right)|}\sum _{k\in T_{\mathrm{H}\mathrm{O}\mathrm{L}}\left(i\right)}{y}_{i-k}\mathbf{1}\{{\mathrm{DT}}^{\mathrm{\prime }}(i)=\mathrm{H}\mathrm{O}\mathrm{L}\}+{ϵ}_i$$ (A1)

where the day type function is

$${\mathrm{DT}}^{\mathrm{\prime }}\left(i\right)=\{\begin{array}{ll}\mathrm{O}\mathrm{R}\mathrm{W}& \mathrm{i}\mathrm{f}\mathrm{d}\mathrm{a}\mathrm{y}i\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{r}\mathrm{y}\mathrm{w}\mathrm{o}\mathrm{r}\mathrm{k}\mathrm{d}\mathrm{a}\mathrm{y}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{w}\mathrm{e}\mathrm{e}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{d}\mathrm{d}\mathrm{a}\mathrm{y}\\ \mathrm{H}\mathrm{O}\mathrm{L}& \mathrm{i}\mathrm{f}\mathrm{d}\mathrm{a}\mathrm{y}i\mathrm{i}\mathrm{s}\mathrm{a}\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{o}\mathrm{l}\mathrm{o}\mathrm{r}\mathrm{a}\mathrm{p}\mathrm{u}\mathrm{b}\mathrm{l}\mathrm{i}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{d}\mathrm{a}\mathrm{y};\end{array}$$

$T_d\left(i\right)=\{k:k<i,\mathrm{DN}(i-k)=d\}$ is the set of days with the same day of week before day i; $T_{\mathrm{H}\mathrm{O}\mathrm{L}}\left(i\right)=\{k:k<i,{\mathrm{DT}}^{\mathrm{\prime }}(i-k)=\mathrm{H}\mathrm{O}\mathrm{L}\}$ is the set of school/public holidays before day i; and DN is the day of week number function, $\mathrm{DN}\left(i\right)=1$ if day i is a Monday, $\mathrm{DN}\left(i\right)=2$ if day i is a Tuesday etc.

The Bayesian Prophet model, devised by Taylor and Letham [19]; Facebook Core Data Science Group [6], is an additive model with three components:

$$y_i=g\left(i\right)+s\left(i\right)+h\left(i\right)+{ϵ}_i$$ (A2)

where $g\left(i\right)$ is the trend, $s\left(i\right)$ is the seasonality, and $h\left(i\right)$ is the holiday effect. The linear trend is $g\left(i\right)=(k+aa(i{)}^{\mathrm{\top }}\delta \delta )i+(m+aa\left(i{)}^{\mathrm{\top }}\gamma \gamma \right)$ where k is the growth rate, m is the offset, $aa$ is the change point indicator, $\delta \delta$ is the growth rate adjustment, and $\gamma \gamma$ is the piece-wise continuity adjustment to ensure that g is continuous. The seasonality component is a Fourier decomposition $s\left(i\right)=\sum _{\ell =1}^L[{\alpha }_{\ell }\cos (2\pi \ell i/P)+{\mathrm{\text{β}}}_{\ell }\sin (2\pi \ell i/P\left)\right]$ where $({\alpha }_{\ell },{\mathrm{\text{β}}}_{\ell })$ are the Fourier coefficients, L is the number of Fourier coefficients and P is the period (in days). The holiday effect is $h\left(i\right)=hh(i{)}^{\mathrm{\top }}\kappa \kappa$ where, say, $hh\left(i\right)=\left(\mathbf{1}\right\{\mathrm{DT}\left(i\right)=\mathrm{S}\mathrm{C}\mathrm{H}\},\mathbf{1}\{\mathrm{DT}\left(i\right)=\mathrm{P}\mathrm{W}\mathrm{E}\left\}\right)$ is the vector of indicator variables of the type of holiday of day i, and $\kappa \kappa$ is the weight vector, usually equal to the all-ones vector. Taylor and Letham [19] provide the details for the construction of the change point function $aa\left(t\right)$ and the continuity adjustment parameter $\gamma \gamma$. These authors set the number of Fourier coefficients to be L = 10 for yearly cycles and L = 3 for weekly cycles. What remains is to estimate the trend growth rate k, the offset m, the growth rate adjustments $\delta \delta$ and the Fourier coefficients $\alpha \alpha$.

A.4. Supplementary figures for the model validation of the lane carpooling service

Figure A1 illustrates the predictions for each scenario described in Table 1: the Bayesian hierarchical multi-level BHML in purple, the frequentist baseline model BASE in black, the Bayesian Prophet PROP in green, and as well as the observed daily driver flows in blue. The PROP predictions are mostly too low on week days and too high on weekends for all six test weeks in comparison to the observed driver flows, whilst the BHML appears to have marginally better prediction performance than the BASE.

Figure A1.

Predictions of daily driver flows for the six test week scenarios. Observed daily driver flows are in blue. Bayesian hierarchical multi-level BHML are in purple, frequentist baseline BASE in black, and Bayesian Prophet PROP in green.

Figure A2 displays the mean observed daily traffic flows for each weekday from the Lane carpooling service, where the day is divided into 15 minute intervals. Since the service operating hours are 06:00–09:00 and 16:00–19:00, there are few drivers outside them. Each dot in the figure is the mean number of drivers for each 15 minute interval for each week day from 2018-05-15 to 2019-05-31.

Figure A2.

Mean driver flows for 15 minute intervals for each weekday for the Lane carpooling service, from 2018-05-15 to 2019-05-31. Monday is in blue, Tuesday in orange, Wednesday in green, Thursday in pink, Friday in purple.

In Figure A3 are the histograms of the observed pseudo waiting times for each hourly interval for weekdays from 2019-07-25 to 2020-02-17. We observe that these empirical distributions resemble Gamma distributions and have a different rate parameter ${\mathrm{\text{β}}}_s$ within each different hourly interval.

Figure A3.

Histograms of observed pseudo waiting times for each hourly interval for weekdays from 2019-07-25 to 2020-02-17.

Disclosure statement

No potential conflict of interest was reported by the author(s).