ASPECT OF THE MODEL TO BE EVALUATED BY THE RISK ASSESSOR – Models for primary producers		Yes	No
0. Evaluation of the growth model All models for primary producers consist of two parts: The physiological growth model describing growth as a function of temperature, irradiance, nutrient and carbon availability, etc. The TKTD part, describing the toxicant effect on growth This checklist is adapted to the evaluation of the TKTD part of the primary producer models (See Section 2.4 and Chapter 6), but the first thing to check is whether the physiological growth part was evaluated beforehand for the chosen species. Without that check, the use of the whole model cannot be suggested, even if the TKTD model part is evaluated positively.
(a)	Was the evaluation of the (physiological) growth model part conducted by a regulatory authority or a group delegated to this at the EU level?
(b)	If (a) is yes, did the above evaluation of the physiological DEB part conclude that it is suitable for use in risk assessment?
(c)	If (a) and (b) are yes, can the physiological DEB part of the model describe the behaviour of the control data?
1. Evaluation of the problem definition The problem definition needs to explain how the modelling fits into the risk assessment and how it can be used to address the specific protection goals (Chapter 3). Please check if due attention is paid to:
(a)	Is the regulatory context for the model application documented?
(b)	Is the question that has to be answered by the model clearly formulated?
(c)	Is the model output suitable to answer the formulated questions?
(d)	Is the choice of the test species clearly described and justified, also considering all the available valid information (including literature)?
(e)	Is the species to be modelled specified? – Is it clear whether the model is being used with a Tier‐1 test species i.e. Tier‐2C1 or with one or more relevant species (which might include the Tier‐1 species) i.e. Tier‐2C2?
2. Evaluation of the quality of the supporting experimental data In this part of the evaluation, it is checked whether the experimental data with which the model is compared (both calibration and validation data sets) have been subjected to quality control. The focus is on the data quality, i.e. the laboratory conditions, setup, chemical analytics and similar. Additional specific criteria for the suitability of the data sets for model calibration and validation are evaluated later in more detail (Sections 7 and 9 of this checklist).
(a)	Are growth conditions (temperature, irradiance, nutrient media composition, handling and thinning, etc.) and growth calculations (frequency and type of measurements, calibration between surface and weight data, etc.) described and documented?
(b)	Has the quality of the used data been considered and documented? (see the list of OECD test guidelines Chapter 7, Table 6).
(c)	Have all available data been used (either for calibration or for validation)? If not, is there a justification why some information has not been used?
(d)	Is it checked whether the actual exposure profile in the study matches the intended profile in the test (+/− 20%); if not, are then measured concentrations used for the modelling, instead of nominal ones?
3. Evaluation of the conceptual model The conceptual model provides a general and qualitative description of the system to be modelled. The conceptual model for physiological part is assumed to have been evaluated and approved beforehand ‘see Section 0 of this checklist); but the TKTD part, describing the toxicant effects on the physiological model needs to be documented.
(a)	Is the reference to the evaluation of the growth (physiological) model part given?
(b)	Is the mode of action of the toxicant specified (effects on e.g. assimilation or growth)?
(c)	Are the links between the external or internal concentration and the growth model logical?
(d)	In case environmental factors (e.g. nutrient levels, temperature) are explicitly considered, is their influence on the physiological and/or TKTD processes documented in the conceptual model?
(e)	Are the modelling endpoints relevant to the specific protection goal?
4. Evaluation of the formal model The formal model contains equations used in the model. For each model application, all equations that are used for the modelling should be documented:
(a)	Are all mathematical equations described, including the equations of the toxicokinetic (TK) part, and the equations relating the used internal concentration with the growth (physiological) model parameter(s)?
(b)	Is there a list or summary of the variables and parameters including their meaning and unit?
(c)	Are both the deterministic part (equations describing the mean tendency of the data) and the stochastic part (the probability law describing the variability around the mean tendency) of the model fully described? See Section 5.1 for an example with DEBtox.
5. Evaluation of the computer modelThe computer model corresponds to the implementation of the formal model that can run it on a computer. Please check the following items:
(a)	Is the computer code available, including explaining comments for the most important functions?
(b)	Is enough information provided to allow any user to re‐run the model independently (e.g. parameter sets, input data)?
(c)	Is it demonstrated that the mathematical model is correctly implemented (model verification), e.g. by checking and documenting the internal consistency of the model results based on a set of default or extreme cases (see e.g. Section 4.1.2.3 for an example with GUTS, ‘reality check’ in chapter 10.4 of the EFSA PPR Panel (2014))?
6. Evaluation of the regulatory model – the environmental scenarios For the application of primary producers TKTD models for the prediction of toxicological effects in a regulatory context, the environmental scenario need to be fixed to the laboratory conditions of the experiments used for calibrating the TKTD part of the model. Please check the following items:
(a)	Are all the relevant conditions, recorded in the experiments used for calibrating the TKTD part of the model, used consistently for the simulations (i.e. for generating EPx) used in the risk assessment?
7. Evaluation of the regulatory model – parameter estimation Parameter estimation requires a suitable data set, the correct application of a parameter optimisation routine, and the comprehensive documentation of methods and results (see Chapter 3 for background information). Model parameters are always estimated for a specific combination of species and compound. Supporting data for primary producers have to be of sufficient quality (Section 2 in this checklist), be relevant to the risk assessment problem and fulfil a set of basic criteria. Typical supporting data for the TKTD part of primary producer models are toxicity test data for growth measured either on the basis of frond number, surface area or biomass over time (see Chapters 2.4 for background information; Chapter 6 for examples). Specific requirements on the time resolution and temporal scale for the calibration data cannot be given, but in general the number of time‐points has to be sufficient to provide enough degrees of freedom for the model calibration. Please check the following items to evaluate the calibration data, the parameter optimisation and the results (see Sections 4.1.3 and 7.6.2):
(a)	Is it clear which parameters have been taken from literature or other sources or which have been fitted to data? If used, are values from literature reasonable and justified?
(b)	Are the calibration data sufficient to provide enough degrees of freedom for the model calibration, i.e. are endpoints reported at several18 intermediate time‐points?
(c)	Does calibration data span from treatment levels with no effects up to strong effects, ideally up to full effects (e.g. no growth)?
(d)	Have all data available for calibration been used? If not, is there a justification why this information has not been used?
(e)	Has attention been paid in terms of adjusting the time course of the experiment to capture the full toxicity of the pesticide?
(f)	Has the model parameter estimation been adequately documented, including settings of optimisation routines, and type and settings of the numerical solver that was used for solving the differential equations?
(g)	If Bayesian inference has been used, are priors on model parameters reported? If a frequentist approach has been used, are starting values for the optimisation reported?
(h)	Are the estimated parameter values reported including confidence/credible limits?
(i)	Is the method to get these limits reported and documented?
(j)	Are the optimal values of the objective function for calibration (e.g. Log‐likelihood function) as the result of the parameter optimisation reported?
(k)	Are plots of the calibrated TKTD models in comparison with the calibration data over time provided, and does the visual match appear of acceptable quality?
(l)	Has a posterior predictive check been performed and documented?
8. Evaluation of the sensitivity and uncertainty analysis It is assumed that sensitivity and uncertainty analyses for the physiological model part have been evaluated and approved beforehand. Sensitivity analysis identifies the influence parameters have on the model outputs and can hence identify the most relevant model parameters. Uncertainty analysis aims at identifying how uncertain the model output is regarding the uncertainty in parameter estimates. Please check the following items for the TKTD part:
(a)	Has a sensitivity analysis for the TKTD part been performed and been adequately documented? (The range of parameter variation in the sensitivity analysis should be justified by an analysis of the expected variation of model parameters).
(b)	Are the results of the sensitivity analysis presented so that the most sensitive parameters can be identified?
(c)	Is the parameter uncertainty for the most important TKTD model parameters propagated to the model outputs and the results of the uncertainty propagation been documented?
(d)	Are the model outputs reported including confidence/credible intervals?
9. Evaluation of the model by comparison with independent measurements (model validation) The model performance is usually evaluated by comparing relevant model outputs with experimental measurements (often referred to as model validation). Independent measurements (or validation data sets) are used to test the model performance in predicting the chosen endpoints under time‐variable exposure profiles. These data have not been used for the model calibration. For models for primary producers, relevant outputs are biomass or biomass‐proxies such as cell number or chlorophyll content for algae at a specific point in time. Please check the following items:
(a)	Are effect data available from experiments under time‐variable exposure?
(b)	Are sufficient endpoint measurements provided in order to cover at least before, during and after each pulse exposure? Are these time‐points enough to allow for evaluation of changes in growth rates different from the control treatment?
(c)	Are two exposure profiles tested with at least two pulses each, separated by no‐exposure intervals of different duration length?
(d)	‐ point not relevant for algae ‐ Is the individual depuration and repair time (DRT95; see Section 4.1.4.5) calculated based on toxicokinetic parameters, and is the duration of the no‐exposure intervals defined accordingly? [Ideally one of the profiles should show a no‐exposure interval shorter than the DRT95, the other profile clearly larger than the DRT95]
(e)	Is each profile tested at least at 3 concentration levels, in order to obtain low, medium and high effects at the end of the respective experiment?
(f)	Has attention been paid to the duration of the experiments considering the time course of development in toxicity of the specific pesticide?
(g)	Does the visual match (‘visual fit’ in FOCUS Kinetics (2006); see Section 4.1.4.5) of the model prediction quality indicate acceptability of the model predictions in comparison with the validation data?
(h)	Do the reported quantitative model performance criteria (e.g. PPC, NRMSE, SPPE) indicate a sufficient model performance? For primary producer models, the adequacy of the quantitative criteria listed above (set on basis of GUTS models (see Section 4.1.4.5)) needs still to be fully tested, and may need future adaptation. However, for the time being, this set of performance criteria is also suggested for primary producer models (see Section 7.7.2 ‘Model performance criteria’).
(i)	Has the performance of the model been reported in an objective and reproducible way?
10. Evaluation of model use When using a TKTD model for regulatory purposes, the inputs of species‐ and compound specific model parameters and of exposure profile data are required to run the model under new conditions. In this stage, it is important that the model is well documented and that it is clear how the model works. Please check the following items:
(a)	Is the use of the model sufficiently documented? Is a user manual available? See items provided in Section 10.7 of the EFSA PPR Panel (2014).
(b)	Is an executable implementation of the model made available to the reviewer, or Is at least the source code provided?
(c)	Has a summary sheet been provided by the applicant? The summary sheet should provide quick access to the comprehensive documentation with sections corresponding to the ones of this checklist.
(d)	Does the exposure profile used with the TKTD model come from the same source as the PEC used with the Tier‐1 effects data? For example if FOCUS Step 3 maximum values were used at Tier‐1, are the exposure profiles used Tier‐2C from the same FOCUS Step 3 modelling? If the exposure profile comes from any other source (e.g. different scenarios, different inputs, different model) has this been checked?
(e)	In case parameters of the growth (physiological) model were changed from calibration to validation, in order to better describe control data:‐ Is the difference in the parameters inducing considerable difference in the model predictions? And, if yes:‐ Are model simulations relevant for the risk assessment carried out with the parameter set producing the more conservative (worst‐case) predictions?