Improved prediction of fungal effector proteins from secretomes with EffectorP 2.0

Summary

Plant‐pathogenic fungi secrete effector proteins to facilitate infection. We describe extensive improvements to EffectorP, the first machine learning classifier for fungal effector prediction. EffectorP 2.0 is now trained on a larger set of effectors and utilizes a different approach based on an ensemble of classifiers trained on different subsets of negative data, offering different views on classification. EffectorP 2.0 achieves an accuracy of 89%, compared with 82% for EffectorP 1.0 and 59.8% for a small size classifier. Important features for effector prediction appear to be protein size, protein net charge as well as the amino acids serine and cysteine. EffectorP 2.0 decreases the number of predicted effectors in secretomes of fungal plant symbionts and saprophytes by 40% when compared with EffectorP 1.0. However, EffectorP 1.0 retains value, and combining EffectorP 1.0 and 2.0 results in a stringent classifier with a low false positive rate of 9%. EffectorP 2.0 predicts significant enrichments of effectors in 12 of 13 sets of infection‐induced proteins from diverse fungal pathogens, whereas a small cysteine‐rich classifier detects enrichment in only seven of 13. EffectorP 2.0 will fast track the prioritization of high‐confidence effector candidates for functional validation and aid in improving our understanding of effector biology. EffectorP 2.0 is available at http://effectorp.csiro.au.

Introduction

Fungal pathogens have been estimated to cause annual crop yield losses of 15%–20% and are a major threat to food security (Figueroa et al., 2007; Fischer et al., 2012). Fungi colonize plants through diverse infection structures and the use of toxic fungal secondary metabolites and secreted effector proteins that alter host cell structure and function, suppress plant defence responses or modulate plant cell physiology (Kamoun, 1983; Lo Presti et al., 2014). Effectors are used by plant‐pathogenic fungi and symbiotic fungi to allow them to colonize their hosts. Fungal effectors can be attached to the fungal cell wall, can function in the plant apoplast or can translocate into plant cells where they may target specific host proteins or enter subcellular compartments (Lo Presti et al., 2014). Accurate effector mining from genomic sequences is crucial to subsequent experimental validation and effector identification can enable disease control strategies. For example, effectors can be used directly in effector‐assisted breeding to select plant lines with distinct recognition traits (Vleeshouwers and Oliver, 2014), and the identification of both effectors and their targets could allow ‘decoy engineering’, where effector targets are fused as baits to a plant immune receptor to make an integrated ‘effector trap’ (Ellis, 2011).

Recent progress in big data genomics has resulted in many high‐quality fungal pathogen genomes and gene expression profiles during plant infection, but accurate effector prediction methods are needed to harness the potential of these resources. The set of secreted proteins expressed during infection is typically too large for experimental investigation and contains many secreted non‐effectors that play roles in niche colonization and protection from competing microbes, differentiation of fungal structures and cell‐to‐cell communication (Rovenich et al., 2014). Secreted plant cell wall‐degrading enzymes (PCWDEs) are used by saprophytic fungi to acquire sugars for their nutrition and survival (Kubicek et al., 2011). Necrotrophic plant‐pathogenic fungi use PCWDEs to overcome the barrier of the cell wall, as well as for nutrient acquisition, whereas biotrophic plant‐pathogenic fungi utilize PCWDEs to facilitate stealth invasion of living plant cells (Gibson et al., 2001).Some PCWDEs in plant‐pathogenic fungi may include effectors specifically required for penetration (Lo Presti et al., 2014); however, these can be predicted based on the presence of conserved enzymatic structures or sequence domains. In contrast, the vast majority of fungal effectors are diverse in sequence and share no conserved sequence motifs or obvious commonalities, apart from their secretion from pathogen to the host. This lack of apparent unifying sequence‐based features has led to ad‐hoc fungal effector prediction approaches that are based on various combinations of characteristics observed in known effectors, such as a small protein size, a high cysteine content, evidence of diversifying selection, the genomic location of the gene in fast‐evolving regions or gene expression in planta (Sperschneider et al., 2017). The inclusion of only a few features in effector prediction, such as the requirement of a small protein size, typically results in many false positive predictions and often overwhelmingly large effector candidate sets, such as 1088–2092 effector candidates predicted in stripe rust (Petre et al., 2011). However, the inclusion of additional features associated with effectors will capture only a small subset as none of these signals are common to all effectors. For example, some fungal effectors are highly enriched in cysteines, whereas others do not feature any cysteines in their sequence, and fungal effectors also vary in size. For example, the Pyrenophora tritici‐repentis ToxB effector has 87 amino acids with four cysteines, and is thought to function in the plant apoplast (Figueroa et al., 2017), whereas the Melampsora lini AvrM effector has a sequence length of 314 amino acids and only one cysteine, and acts intracellularly (Catanzariti et al., 2004). However, a high cysteine content or small protein size alone does not allow for the accurate discrimination of apoplastic effectors from cytoplasmic effectors in fungi (Sperschneider et al., 2008). Taken together, the use of predefined criteria for effector prediction inherits the individual researcher's potentially biased view of effector characteristics and is unable to uncover novel effectors with diverse characteristics.

An alternative approach is to use data to learn which features are important for effector prediction, rather than setting predefined criteria. This is achieved with machine learning, a family of statistical learning methods with the ability to identify patterns in data and recognize a particular class based on its features in observed data. Models trained on datasets of positive and negative classes are then applied to identify new instances of the class in unseen data. This data‐driven approach has the capacity to identify new features not apparent to manual inspection and to provide probabilistic predictions based on combinations of features, which represent advantages over the use of predefined criteria with binary cut‐offs. We have recently pioneered such a machine learning approach for fungal effector prediction, called EffectorP (Sperschneider et al., 1996), and have demonstrated that machine learning can accurately predict novel effectors with diverse characteristics from secretomes, as well as their localization in the plant cell (Sperschneider et al., 2011, 2011). We have shown that EffectorP 1.0 is able to learn ‘effector rules’ from positive and negative training examples without having to apply user‐chosen thresholds (Sperschneider et al., 1996). EffectorP relies on fungal effectors as the positive training set and secreted non‐effectors as the negative set. One limiting factor is that the negative training set consists of both undiscovered effectors and secreted non‐effectors, and therefore poses an unlabelled data classification problem. Furthermore, the positive training set used in EffectorP 1.0 is small and additional effectors are now available for inclusion in training. This has the potential to improve accuracy and will enable us to re‐evaluate the ability of machine learning to accurately predict fungal effectors.

Results

Training of the ensemble classifier EffectorP 2.0

EffectorP 1.0 is a Naïve Bayes classifier that was trained on a positive training set of 58 experimentally supported fungal effectors from 16 fungal species. Since its development, additional fungal effectors have been described and, for EffectorP 2.0, we used an expanded training set of 94 secreted fungal effectors from 23 species (Table 1). EffectorP 1.0 predicts 73% of the unseen effectors correctly, which demonstrates its ability to identify novel effectors, but also leaves room for improvement. We set out to investigate whether re‐training of EffectorP would improve prediction accuracy.

Table 1.

The set of fungal effector proteins used as positive training data.

Species	Effector
Melampsora lini	AvrM, AvrL567‐A, AvrP123, AvrP4, AvrM14, AvrL2‐A
Uromyces fabae	RTP1
Puccinia graminis f. sp. tritici	PGTAUSPE‐10‐1, AvrSr50
Puccinia striiformis f. sp. tritici	PstSCR1, Pec6
Phakopsora pachyrhizi	PpEC23
Blumeria graminis f. sp. hordei	Avrk1, Avra1, Avra13
Blumeria graminis f. sp. tritici	AvrPm2
Cladosporium fulvum	Avr9, Avr4, Avr4E, Avr2, Avr5, Ecp1, Ecp2, Ecp4, Ecp5, Ecp6
Leptosphaeria maculans	AvrLm6, AvrLm4–7, AvrLm1, AvrLm11
Fusarium oxysporum f. sp. lycopersici	Six4, Six3, Six1, Six6, Six2, Six5, Six7, Six8
Magnaporthe oryzae	Avr‐Pita, Pwl1, Avr‐Pia, Bas3, Bas2, Bas4, Bas1, MC69, AvrPiz‐t, Avr1‐CO39, Avr‐Pii, Avr‐Pik, Bas107, AvrPib, Iug6, Iug9, Msp1, MoHEG13, MoCDIP1, MoCDIP2, MoCDIP3, MoCDIP4, MoCDIP5, SPD2, SPD4, SPD7, SPD9, SPD10, Bas162, AvrPi9
Rhynchosporium secalis	NIP1, NIP2, NIP3
Verticillium dahliae	Vdlsc1, Ave1, VdSCP7, PevD1
Ustilago maydis	Cmu1, Pep1, Pit2, Tin2, eff1‐1, See1
Ustilago hordei	UhAvr1
Stagonospora nodorum	ToxA, Tox1, Tox3
Botrytis cinerea	Nep1
Pyrenophora tritici‐repentis	ToxB
Laccaria bicolor	MiSSP7
Zymoseptoria tritici	AvrStb6, Zt6
Colletotrichum graminicola	CgEP1, Cgfl
Fusarium graminearum	FGL1
Sclerotinia sclerotiorum	SsSSVP1

Ninety‐four fungal effectors were collected from the literature if they had experimental support and did not share sequence homology. Effectors that were not part of the EffectorP 1.0 training set are marked in bold. All sequences are available at: http://effectorp.csiro.au/data.html.

EffectorP 1.0 was trained on a negative set consisting of predicted secreted proteins from the same pathogen species as the known effectors. Thus, the negative training set includes both undiscovered effectors and non‐effectors, and therefore poses an unlabelled data classification problem. Although Naïve Bayes classifiers are fairly robust to unlabelled data classification and can tolerate noisy data (Bing et al., 2007), other machine learning classifiers might not be able to learn effectively from such sets. To improve predictions, we collected three different subsets of negative training data that are less likely to contain positive instances, i.e. fungal effectors. First, secretomes were predicted from the same fungal pathogen/symbiont species as used in the positive set if they had a publicly available predicted gene set (Table 1). The combined secretome was homology reduced and this resulted in a filtered predicted pathogen secretome of 11 277 proteins. This set will contain both undiscovered effectors and secreted non‐effectors, which poses a challenge for machine learning classifiers that traditionally learn from labelled data. Therefore, we applied EffectorP 1.0 to exclude predicted effectors from the secretomes (n = 6138). This procedure removed predominantly small, cysteine‐rich proteins from the negative training set (average sequence length, 137 amino acids; average cysteine content, 3.55%). We also collected homology‐reduced sets of secreted fungal proteins from fungi not pathogenic on plants, namely from 27 saprophyte secretomes (n = 12 939) and from 10 animal‐pathogenic fungal secretomes (n = 2763). These sets are less likely to contain plant‐pathogenic effectors and were not filtered for EffectorP 1.0‐predicted effectors.

As we have large amounts of negative training data (n = 21 840), we used an ensemble learning approach of classifiers that each take a different subset of negative training data and thus provide a different view on classification (Fig. 1). Overall, we chose a total of 50 best‐performing models comprising: 10 Naïve Bayes classifiers and 10 C4.5 decision trees that discriminate between fungal effectors and secreted pathogen proteins; 10 Naïve Bayes classifiers and 10 C4.5 decision trees that discriminate between fungal effectors and secreted saprophyte proteins; and five Naïve Bayes classifiers and five C4.5 decision trees that discriminate between fungal effectors and secreted animal pathogen proteins. In 10‐fold cross‐validation, the Naïve Bayes classifiers achieve, on average, high sensitivity, whereas the C4.5 decision trees show high specificity (Table S2, see Supporting Information). To generate EffectorP 2.0, we combined these 50 models into an ensemble classifier to utilize their distinct prediction strengths (Fig. 1). Each model has seen a different subset of negative training data and, for a given protein sequence input, returns a probability of whether it is an effector or a non‐effector. EffectorP 2.0 returns a final prediction using a voting approach on the predicted probabilities of each model. A protein is classified as an effector if the average probability for the class ‘effector’ is higher than the average probability for the class ‘non‐effector’. For each protein in the training set, EffectorP 1.0 utilizes a feature vector that is calculated using amino acid frequencies, amino acid class frequencies, molecular weight, sequence length and protein net charge (Sperschneider et al., 1996). EffectorP 2.0 uses an updated feature vector that includes amino acid frequencies, amino acid class frequencies, molecular weight, protein net charge, grand average of hydrophobicity, as well as the averages of surface exposure, disorder propensity, hydrophobicity, bulkiness and interface propensity (Table 2).

Figure 1.

Workflow for the EffectorP 2.0 classifier that combines an ensemble of machine learning classifiers. Each classifier C_i has seen a different subset of the negative training data and predicts effectors in unseen data with probability P_i. The probabilities are combined into an overall vote on whether an unseen protein is an effector or non‐effector.

Table 2.

Features used for training the machine learning classifiers in the EffectorP 2.0 ensemble learner.

Features used in training and classification	Method
Frequencies of amino acids (A, C, D, E, F, G, H, I, K, L, M, N, P, Q, R, S, T, V, W, Y) in the sequence	pepstats (Rice et al., 2000)
Frequencies of amino acid classes in the sequence:
Tiny (A+C+G+S+T)
Small (A+B+C+D+G+N+P+S+T+V)
Aliphatic (I+L+V)
Aromatic (F+H+W+Y)
Non‐polar (A+C+F+G+I+L+M+P+V+W+Y)
Polar (D+E+H+K+N+Q+R+S+T+Z)
Charged (B+D+E+H+K+R+Z)
Basic (H+K+R)
Acidic (B+D+E+Z)
Molecular weight
Protein net charge
Grand average of hydropathicity (GRAVY, Kyle and Doolittle, 1982)	ProtParam (Gasteiger et al., 2005)
Average of surface exposure (Janin, 1979)	Amino acid groupings and scales taken from Composition Profiler (Vacic et al., 2007)
Average of disorder propensity (Dunker et al., 2001)
Average of hydrophobicity (Fauchere and Pliska, 1983)
Average of bulkiness (Zimmerman et al., 1968)
Average of interface propensity (Jones and Thornton, 1997)

Influential features for effector prediction include protein size, protein net charge as well as the amino acids serine and cysteine

To detect the most discriminative features in the EffectorP 2.0 classification, we analysed the distribution of features for the proteins employed in the training of all 50 models. Four features were found to be different at a significance threshold of P < 10⁻⁵ in distribution between the positive sequence set (effectors) and the negative sequence set (proteins labelled as non‐effectors) (Fig. 2). Differences in feature distribution for these four features were also reported previously in the EffectorP 1.0 model as particularly striking (Sperschneider et al., 1996), confirming their importance in fungal effector classification. As a group, the effectors exhibit lower molecular weight, a higher percentage of cysteines (C) and a lower percentage of serines (S) than the proteins in the negative sequence set. The distribution of protein net charge for effectors occupies a narrow range around neutral to slightly positive (Fig. 2). We also found significant differences (P < 0.05) in distribution between effectors and the negative sequence set for additional features (Fig. 2). These were depletion in aliphatic amino acids, leucine (L), proline (P), threonine (T), tryptophan (W), disorder propensity and bulkiness, as well as enrichment in basic amino acids, interface propensity, glycine (G), lysine (K) and asparagine (N), for effectors. Only enrichment in tryptophan content in effectors was also reported in the EffectorP 1.0 model.

Figure 2.

The most influential features in effector prediction appear to be a small protein size, low serine content, a protein net charge around the neutral range and a high cysteine content. Significant differences (P < 0.05) in distribution between effectors and the negative sequence set for additional features were also observed. These were depletion in aliphatic amino acids, leucine (L), proline (P), threonine (T), tryptophan (W), disorder propensity and bulkiness, as well as enrichment in basic amino acids, interface propensity, glycine (G), lysine (K) and asparagine (N), for effectors. Extreme outliers in the protein net charge plot were removed for clarity (full figure given in Fig. S3, see Supporting Information). All data points are drawn on top of the box plots as black dots. Significance between groups is shown as horizontal brackets and was assessed using t‐tests. The lower and upper hinges correspond to the first and third quartiles and the upper (lower) whiskers extend from the hinge to the largest (smallest) value that is within 1.5 times the interquartile range of the hinge. Data beyond the end of the whiskers are outliers.

Machine learning can be a black box learning process where the reasons for an individual prediction are hidden. However, C4.5 decision trees are white box models and their decision‐making process is transparent through navigation along tree branches. As examples, we plotted two of the 10 C4.5 decision trees that discriminate between fungal effectors and secreted pathogen proteins (Figs S1 and S2, see Supporting Information). This demonstrates that the decision tree classifiers use a complex set of features and not only the most discriminative features (protein size, protein net charge as well as the amino acids serine and cysteine) for effector classification. In particular, the decision tree in Fig. S2 does not utilize serine content as a feature in classification and still achieves high classification accuracy. Taken together, this analysis confirms the importance of specific combinations of features, as found previously in the EffectorP 1.0 model, but also illustrates that accurate fungal effector prediction machine learning classifiers rely on a diverse set of features.

EffectorP 2.0 improves fungal effector prediction accuracy from secretomes

Machine learning classifiers can overfit/overtrain to memorize the training data, which leads to low accuracy on unseen data. Therefore, independent test sets are important to estimate prediction ability. We collected independent positive and negative test sets to assess the performance of EffectorP 2.0. To estimate the false positive rate, we first used fungal, plant and mammalian proteins with predicted signal peptides that were not extracellular [localization to endoplasmic reticulum, Golgi or membranes or with glycosylphosphatidylinositol (GPI) anchors]. A low false positive rate on these proteins ensures that EffectorP is not merely predicting the presence of a signal peptide. We also used secreted saprophyte proteins as well as fungal proteins from PHI‐base (Urban et al., 2007) that were annotated as having an unaffected pathogenicity phenotype. Although proteins with an unaffected pathogenicity phenotype are not necessarily non‐effectors, we expect to see a low percentage of predicted effectors. A simple classifier based on a small protein size (≤300 amino acids) has a false positive rate of 40.4% on these three sets. A small, cysteine‐rich classifier (≤300 amino acids; ≥4 cysteines) has a false positive rate of 19%, and EffectorP 1.0 has a false positive rate of 18.3%. EffectorP 2.0 has the lowest false positive rate of 11.2% (Table 3). A combination of EffectorP 1.0 and 2.0, where a protein is a predicted effector only if both classifiers label it as an effector, achieves the lowest false positive rate of 9.4%.

Table 3.

Independent validation of EffectorP's prediction accuracy.

		Predicted effectors
Dataset	# of proteins	EffectorP 2.0	EffectorP 1.0	EffectorP 1.0 and 2.0	Small size classifier	Small, cysteine‐rich classifier
Fungal saprophyte secreted proteins	24 432	2865 (11.7%)	4774 (19.5%)	2444 (10%)	10 529 (43.1%)	4961 (20.3%)
Fungal, plant and mammalian proteins with signal peptide and localization to endoplasmic reticulum, Golgi, membranes or with glycosylphosphatidylinositol (GPI) anchors	2631	220 (8.4%)	294 (11.2%)	164 (6.2%)	654 (24.9%)	307 (11.7%)
Fungal proteins with unaffected pathogenicity phenotype	938	45 (4.8%)	59 (6.3%)	36 (3.8%)	128 (13.6%)	60 (6.4%)
	28 001	3130 (11.2%)	5127 (18.3%)	2644 (9.4%)	11 311 (40.4%)	5328 (19%)
Fungal effector positive training set	94	89 (94.7%)	80 (85.1%)	79 (84%)	88 (93.6%)	53 (56.4%)
Fungal effector independent test set	21	16 (76.2%)	16 (76.2%)	16 (76.2%)	19 (90.5%)	10 (47.6%)
Accuracy		88.8%	81.7%	90.5%	59.8%	80.9%

To assess false negative predictions, we also applied these predictors to the training data of 94 fungal effectors (Table 3). EffectorP 2.0 only predicts five of these proteins as non‐effectors: the Phakopsora pachyrhizi effector PpEC23, the Blumeria graminis f. sp. hordei effector Avrk1, the Magnaporthe oryzae effector MoCDIP2, the Ustilago maydis effector eff1‐1 and the Colletotrichum graminicola metalloproteinase effector Cgfl. This is an improvement on EffectorP 1.0, which correctly predicted only 80 of the 94 positive examples. However, it is also important to assess overfitting on training data and to use unseen fungal effectors independent from the training set for the validation of the estimated true positive rate. Therefore, we collected 21 effectors (Table 4) that either shared sequence similarity with an effector in the training set and were therefore eliminated in the homology reduction step (Mg3LysM, BEC1054, BEC1011, AvrLm2) or were overlooked during initial literature searches for training the EffectorP 2.0 model (SAD1, CSEP‐07, CSEP‐09, SIS1, CSEP0055, BEC1019, Bcg1, CSEP0105, CSEP0162, AvrLmJ1, AvrLm3, XylA, Ecp7, PIIN_08944, FGB1, AvrPm3, AvrSr35). On this independent test set, both EffectorP 1.0 and 2.0 show equal performance and correctly predict 76.2% of effectors (Tables 3 and 4). On the total set of 115 effectors, the small size classifier correctly predicts 93% of effectors, but the small, cysteine‐rich classifier only correctly predicts 54.8% of effectors. On the combined positive and negative sets, EffectorP 2.0 has the highest accuracy of 88.8% of the four single classifiers. The simple classifier based on a small size has the lowest accuracy of 59.8%, largely because of its high false positive rate (Table 3). The combined EffectorP 1.0/2.0 classifier achieves the highest accuracy of 90.5% because of its low false positive rate. Although the combined EffectorP 1.0/2.0 classifier misses more effectors than EffectorP 2.0 or 1.0, it is a highly stringent method for the prediction of effectors in secretomes. In the following, we assess the prediction abilities of EffectorP 1.0 compared with EffectorP 2.0 in more detail.

Table 4.

Independent test set of fungal effectors that were not used in training of EffectorP 2.0

Species	Effector	EffectorP 1.0 (probability)	EffectorP 2.0 (probability)	Small size classifier	Small, cysteine‐rich classifier
Sporisorium reilianum	SAD1	Effector (0.97)	Effector (0.621)	Effector	Non‐effector
Phakopsora pachyrhizi	CSEP‐07	Effector (0.608)	Effector (0.688)	Effector	Effector
	CSEP‐09	Effector (0.999)	Effector (0.842)	Effector	Effector
Zymoseptoria tritici	Mg3LysM	Non‐effector (0.556)	Non‐effector (0.561)	Effector	Effector
Blumeria graminis f. sp. hordei	BEC1054	Effector (0.935)	Effector (0.869)	Effector	Non‐effector
	BEC1011	Effector (0.974)	Effector (0.947)	Effector	Non‐effector
	BEC1019	Non‐effector (0.986)	Non‐effector (0.551)	Non‐effector	Non‐effector
	CSEP0055	Effector (0.649)	Effector (0.732)	Effector	Non‐effector
	Bcg1	Effector (0.971)	Effector (0.896)	Effector	Non‐effector
	CSEP0105	Non‐effector (0.511)	Non‐effector (0.595)	Effector	Effector
	CSEP0162	Effector (0.854)	Effector (0.693)	Effector	Effector
Rhizophagus irregularis	SIS1	Effector (0.973)	Effector (0.611)	Effector	Non‐effector
Leptosphaeria maculans	AvrLmJ1	Effector (0.999)	Effector (0.727)	Effector	Effector
	AvrLm2	Effector (0.764)	Effector (0.578)	Effector	Effector
	AvrLm3	Effector (1.0)	Effector (0.91)	Effector	Effector
Fusarium graminearum	XylA	Effector (0.882)	Effector (0.865)	Effector	Non‐effector
Cladosporium fulvum	Ecp7	Effector (0.997)	Effector (0.96)	Effector	Effector
Piriformospora indica	PIIN_08944	Non‐effector (0.886)	Non‐effector (0.539)	Effector	Non‐effector
	FGB1	Effector (1.0)	Effector (0.929)	Effector	Effector
Blumeria graminis f. sp. tritici	AvrPm3	Effector (0.979)	Effector (0.913)	Effector	Non‐effector
Puccinia graminis f. sp. tritici	AvrSr35	Non‐effector (1.0)	Non‐effector (0.918)	Non‐effector	Non‐effector

Sets of infection‐induced proteins are enriched for effectors predicted by EffectorP 2.0

Effectors are often induced during infection, and thus the set of genes differentially expressed during infection should be enriched for effectors. However, not all genes that are differentially expressed during infection encode effector proteins, and therefore sets of differentially expressed genes need to be filtered further to detect effectors. We collected 13 gene sets from the literature that were labelled as containing effector candidates based on their induction during infection as well as other criteria (Table 5). For example, a study by Germain et al. (2011) identified 16 candidate effectors from 1184 small, secreted Melampsora larici‐populina proteins. These 16 candidates were selected based on their expression in a haustoria‐specific cDNA library and the transcriptome of laser microdissected, rust‐infected poplar leaves, as well as their small size of less than 300 amino acids. As another example, Kettles et al. (2017) selected 63 Zymoseptoria tritici candidate effectors on the basis of being induced during early wheat leaf infection leading up to the transition to the necrotrophic growth phase. In total, four of the 13 sets contained infection‐induced effector candidates that were pre‐selected based on a small size (≤300 amino acids).

Table 5.

Enrichment of predicted effector candidates in expression datasets of early infection stages.

Expression dataset	No. of proteins	Method	Predicted effectors	Predicted effectors in secretome	Enrichment (Fisher's exact test)
Colletotrichum higginsianum: biotrophy‐associated effector candidates (Kleemann et al., 2012)	102	Small size	100 (98%)	845 (56.6%)	<0.0001
		Small, cysteine‐rich	46 (45.1%)	412 (27.6%)	0.0003
		EffectorP 1.0	73 (71.6%)	490 (32.8%)	<0.0001
		EffectorP 2.0	49 (48%)	378 (25.3%)	<0.0001
Cladosporium fulvum: in planta induced small secreted apoplastic effector candidates (Mesarich et al., 2017)	75	Small size	–	–	–
		Small, cysteine‐rich	70 (93.3%)	272 (25%)	<0.0001
		EffectorP 1.0	68 (90.7%)	237 (21.8%)	<0.0001
		EffectorP 2.0	64 (85.3%)	190 (17.5%)	<0.0001
Magnaporthe oryzae: genes with ≥50‐fold differential expression in biotrophic invasive hyphae (Mosquera et al., 2009)	15	Small size	15 (100%)	907 (55.6%)	0.0002
		Small, cysteine‐rich	9 (60%)	500 (30.7%)	0.0221
		EffectorP 1.0	14 (93.3%)	614 (37.7%)	<0.0001
		EffectorP 2.0	13 (86.7%)	489 (30%)	<0.0001
Blumeria graminis f. sp. hordei: Candidates for Secreted Effector Proteins (CSEPs) (Pedersen et al., 2012)	491	Small size	347 (70.7%)	426 (58.8%)	<0.0001
		Small, cysteine‐rich	133 (27.1%)	169 (23.3%)	NS
		EffectorP 1.0	274 (55.8%)	302 (41.7%)	<0.0001
		EffectorP 2.0	256 (52.1%)	276 (38.1%)	<0.0001
Melampsora larici‐populina: specific small secreted proteins expressed in haustoria (Petre et al., 2015)	24	Small size	–	–	–
		Small, cysteine‐rich	15 (62.5%)	707 (38.8%)	0.0210
		EffectorP 1.0	20 (83.3%)	780 (42.8%)	<0.0001
		EffectorP 2.0	18 (75%)	752 (41.3%)	0.0013
Melampsora larici‐populina: specific small secreted proteins expressed during infection (Germain et al., 2011)	16	Small size	–	–	–
		Small, cysteine‐rich	10 (62.5%)	707 (38.8%)	NS
		EffectorP 1.0	15 (93.8%)	780 (42.8%)	<0.0001
		EffectorP 2.0	14 (87.5%)	752 (41.3%)	0.0004
Laccaria bicolor: ectomycorrhiza‐regulated small secreted proteins (MiSSPs) (Martin et al., 2008)	21	Small size	–	–	–
		Small, cysteine‐rich	10 (47.6%)	362 (29.2%)	NS
		EffectorP 1.0	11 (52.4%)	380 (30.7%)	NS
		EffectorP 2.0	10 (47.6%)	246 (19.9%)	0.0043
Puccinia graminis f. sp. tritici: secreted proteins up‐regulated in haustoria (log FC > 10) (Upadhyaya et al., 2015)	55	Small size	38 (69.1%)	1223 (64.7%)	NS
		Small, cysteine‐rich	7 (12.7%)	710 (37.6%)	NS
		EffectorP 1.0	25 (45.5%)	841 (44.5%)	NS
		EffectorP 2.0	22 (40%)	758 (40.1%)	NS
Zymoseptoria tritici candidate effectors (Kettles et al., 2017)	63	Small size	56 (88.9%)	426 (42.7%)	<0.0001
		Small, cysteine‐rich	43 (68.3%)	259 (26%)	<0.0001
		EffectorP 1.0	41 (65.1%)	260 (26.1%)	<0.0001
		EffectorP 2.0	42 (66.7%)	232 (23.3%)	<0.0001
Zymoseptoria tritici candidate effectors with phenotype in Nicotiana benthamiana (Kettles et al., 2017)	14	Small size	12 (85.7%)	426 (42.7%)	0.0017
		Small, cysteine‐rich	10 (71.4%)	259 (26%)	0.0005
		EffectorP 1.0	8 (57.1%)	260 (26.1%)	0.0143
		EffectorP 2.0	9 (64.3%)	232 (23.3%)	0.0014
Ustilago maydis effector candidates (Tollot et al., 2016)	198	Small size	130 (65.7%)	242 (46.8%)	<0.0001
		Small, cysteine‐rich	49 (24.7%)	101 (19.5%)	NS
		EffectorP 1.0	79 (39.9%)	140 (27.1%)	0.0011
		EffectorP 2.0	67 (33.8%)	124 (24%)	0.0106
Leptosphaeria maculans highly expressed early effector candidates (Gervais et al., 2017)	49	Small size	44 (89.9%)	514 (49.7%)	<0.0001
		Small, cysteine‐rich	23 (46.9%)	258 (24.9%)	0.0013
		EffectorP 1.0	29 (59.2%)	283 (27.3%)	<0.0001
		EffectorP 2.0	26 (53.1%)	215 (20.8%)	<0.0001
Leptosphaeria maculans highly expressed late effector candidates (Gervais et al., 2017)	50	Small size	33 (66%)	514 (49.7%)	0.0292
		Small, cysteine‐rich	16 (32%)	258 (24.9%)	NS
		EffectorP 1.0	19 (38%)	283 (27.3%)	NS
		EffectorP 2.0	19 (38%)	215 (20.8%)	0.0073

For each expression dataset, the percentage of predicted effector candidates by EffectorP is shown and compared with the percentage of predicted effector candidates in the secretome. The small size classifier is only applied to sets that are not pre‐selected based on a small size.

We assessed whether the 13 sets containing infection‐induced effector candidates are also enriched for effector candidates predicted by EffectorP 1.0 or 2.0, by a small size classifier or by a small, cysteine‐rich classifier when compared with the whole secretome of each species. We did not test the small size classifier on sets containing effector candidates that were pre‐selected based on a small size (≤300 amino acids). We found significant enrichments for predicted effector candidates in 12 of 13 sets (92.3%) using EffectorP 2.0 (Table 5). A small, cysteine‐rich classifier only returns significant enrichments for predicted effectors in seven of 13 sets (53.9%) and EffectorP 1.0 in 10 of 13 sets (76.9%). A small size classifier returns significant enrichments for predicted effectors in eight of nine sets (88.9%). Surprisingly, we did not observe enrichment for predicted effectors with any of the four classifiers in secreted proteins of P. graminis f. sp. tritici highly up‐regulated in haustoria compared with germinated spores (Table 5). This could indicate that rusts might utilize undiscovered effector proteins with different properties to the training set, such as effectors of larger size. This is supported by the recent discovery of AvrSr35, a 578‐amino‐acid P. graminis f. sp. tritici effector protein (Salcedo et al., 2005). Alternatively, haustorial secretomes might contain many non‐effectors, such as proteins involved in signalling or in the incorporation of nutrients from the host (Garnica et al., 2005). Taken together, although effector function has not been established for all genes in these candidate sets, the enrichment for predicted effectors in infection‐induced sets underlines the ability of EffectorP 2.0 to accurately predict unseen effectors.

EffectorP 2.0 reduces the average number of effectors predicted for fungal plant symbionts and saprophytes by 40%

We tested EffectorP 2.0 on predicted secretomes from 93 fungal species, including pathogens and non‐pathogens (Table S3, see Supporting Information), and recorded the percentages of secreted proteins that are predicted effectors (Table S4, see Supporting Information). The highest proportions of predicted effectors were found in the obligate biotrophs Melampsora laricis‐populina (41.3%), Puccinia graminis f. sp. tritici (40.3%), Blumeria graminis f. sp. hordei (38.1%) and Puccinia striiformis f. sp. tritici (37.6%). Amongst the fungal plant pathogens, the lowest proportions of predicted effectors were recorded for the necrotrophs Heterobasidion annosum (10.4%), Sclerotinia sclerotiorum (13.6%), Botrytis cinerea (13.7%) and Penicillium digitatum (13.9%). Necrotrophic pathogens utilize many secreted PCWDEs to overcome the barrier of the plant cell wall. EffectorP predicts some secreted proteins with enzymatic domains as effectors, such as the Fusarium graminearum xylanase XylA, which has the ability to induce necrosis in wheat independent of its enzymatic activity (Table 4) (Belien et al., 2011; Sella et al., 2016; Sperschneider et al., 1996). However, EffectorP has been trained on effectors that predominantly lack recognizable functional domains and interfere with host processes in different ways from PCWDEs which act on the plant cell wall. Therefore, the lower proportions of EffectorP‐predicted effectors in necrotrophic fungal pathogen secretomes is expected.

On average, EffectorP 2.0 predicts that plant pathogen secretomes consist of 24.9% effectors and that saprophyte secretomes consist of 11.7% effectors (Tables 5 and 6). EffectorP 2.0 reduces the average number of predicted effectors in fungal plant symbiont and fungal saprophyte secretomes by over 40% when compared with EffectorP 1.0 (Table 6, Fig. 3). Both EffectorP 2.0 and EffectorP 1.0 also predict lower proportions of effectors for fungal animal pathogens than for fungal plant pathogens (Table 6), suggesting that effector repertoires of fungal animal pathogens are different from those of their plant‐pathogenic counterparts. One notable exception is the secretome of Enterocytozoon bieneusi, an obligate intracellular parasite (49 predicted effectors, 36% of secretome predicted as effectors). Shortened protein‐coding sequences caused by genome compaction have been reported in E. bieneusi (Akiyoshi et al., 2009) and might lead to higher than expected false positive predictions. Therefore, we also assessed effector prediction rates for small secreted proteins (<300 amino acids) only. For plant pathogens, EffectorP 2.0 predicts that 47.8% of small secreted proteins are effectors, whereas, for plant symbionts and saprophytes, this is reduced to 29.9% and 26.3%, respectively. This underlines that EffectorP 2.0 does not select effectors based on a small size alone. Small secreted proteins in saprophytes are mostly functionally uncharacterized and might function in a variety of processes unrelated to plant–pathogen interactions. Compared with a small, cysteine‐rich classifier, EffectorP 2.0 predicts significantly lower proportions of effectors for plant symbionts and saprophytes, but not for plant pathogens (Fig. 3). This lack of correlation for all groups tested underlines that EffectorP 2.0 does not select effectors based on a small size and a high cysteine content alone, and reflects the reduced false positive rate of EffectorP 2.0.

Table 6.

Predicted effectors in secretomes for groups of fungal species

	Average of predicted effectors
Secretomes	EffectorP 1.0	EffectorP 2.0	% decrease in predicted effectors (EffectorP 2.0 compared with EffectorP 1.0)
Plant pathogens	338 (29.6%)	284 (24.9%)	−16.0%
Fungal symbionts of plants	305 (30.8%)	177 (17.8%)	−42.0%
Fungal pathogens of animals	108 (20.9%)	83 (16.1%)	−23.2%
Saprophytes	177 (19.5%)	106 (11.7%)	−40.1%

Figure 3.

Proportions of predicted effectors in fungal secretomes using EffectorP 1.0, EffectorP 2.0, a small size classifier and a small, cysteine‐rich classifier. All data points are drawn on top of the box plots as black dots. Significance between groups is shown as horizontal brackets and was assessed using t‐tests (NS, not significant; *P < 0.05, **P < 0.01 and ***P < 0.001). The lower and upper hinges correspond to the first and third quartiles and the upper (lower) whiskers extend from the hinge to the largest (smallest) value that is within 1.5 times the interquartile range of the hinge. Data beyond the end of the whiskers are outliers.

We then further investigated the properties of effectors that are only predicted by one of the versions of EffectorP, but not by the other, for all 93 secretomes (Table S4). Effector candidates predicted only by EffectorP 2.0 are, on average, of longer sequence length (n = 2304; average sequence length, 229 amino acids) than those that are only predicted by EffectorP 1.0 (n = 8635; average sequence length, 138 amino acids) or by both versions (n = 14 128; average sequence length, 148 amino acids) (Fig. 4). Effector candidates predicted only by EffectorP 1.0 or 2.0 are lower in cysteine content compared with effector candidates predicted by both versions (Fig. 4). We then tested for enrichment and depletion of protein functional classes amongst the effector candidates predicted by EffectorP 1.0 and 2.0 from a total of 24 075 secreted proteins of the 21 plant pathogens (Table 7). The vast majority of effector candidates predicted by either EffectorP 1.0 or 2.0 are proteins without functional annotation. However, we observed that both sets of predicted effector candidates are enriched for proteins with pectate lyase activity, peptidyl‐prolyl cis–trans isomerase activity and endopeptidase inhibitor activity (Table 7). Some proteins with peptidyl‐prolyl cis–trans isomerase activity have been implicated to function as virulence factors (Unal and Steinert, 2009). A cyclophilin with peptidyl‐prolyl cis–trans isomerase activity functions as a pathogenicity factor in Puccinia triticina (Panwar et al., 2012). EffectorP 2.0‐predicted effectors are enriched for proteins involved in pathogenesis and defence response (Table 7). However, EffectorP 1.0‐predicted effector candidates are also enriched for proteins that do not appear to be related to effector function or to secreted proteins, but rather to intracellular proteins (Table 7), and might reflect the higher false positive rate of EffectorP 1.0, as well as the false positive rate of the signal peptide prediction tools SignalP 3.0 and TargetP.

Figure 4.

Differences in sequence length (aas, amino acids) and cysteine content for effectors predicted by different versions of EffectorP. All data points are drawn on top of the box plots as black dots. Significance between groups is shown as horizontal brackets and was assessed using t‐tests. The lower and upper hinges correspond to the first and third quartiles and the upper (lower) whiskers extend from the hinge to the largest (smallest) value that is within 1.5 times the interquartile range of the hinge. Data beyond the end of the whiskers are outliers.

Table 7.

Gene ontology (GO) term enrichment analysis of predicted effector candidates.

Comparison	Over‐represented GO term description	# proteins in test set	# proteins in reference set	FDR
Test set: EffectorP 2.0 predicted Reference set: Secreted pathogen proteins	Pectate lyase activity	26	11	2.9 × 10−8
	Peptidyl‐prolyl cis–trans isomerase activity	15	5	2.6 × 10−5
	Pathogenesis	12	10	0.02
	Defence response	14	14	0.02
	Endopeptidase inhibitor activity	7	3	0.03
Test set: EffectorP 1.0 predicted Reference set: Secreted pathogen proteins	Peptidyl‐prolyl cis–trans isomerase activity	18	2	1.9 × 10−7
	Inner mitochondrial membrane organization	10	1	3.6 × 10−4
	Intracellular sterol transport	8	0	5.7 × 10−4
	Fungal‐type vacuole lumen	8	0	5.7 × 10−4
	Pectate lyase activity	22	15	8.4 × 10−4
	Endopeptidase inhibitor activity	9	1	0.001
	Chaperone‐mediated protein folding	7	0	0.001
	FK506 binding	7	0	0.002
	Nuclear envelope organization	6	0	0.002
	Regulation of COPII vesicle coating	6	0	0.006
	Endoplasmic reticulum exit site	6	0	0.006
	Mitochondrial inner membrane	18	15	0.01
	Mitochondrial respiratory chain complex IV assembly	5	0	0.02
	Mitochondrion morphogenesis	5	0	0.02
	COPII vesicle coat	6	1	0.03

Discussion

Given the high diversity of fungal effectors, it seems an unexpected finding that a machine learning classifier can accurately distinguish diverse effectors from secreted non‐effectors. However, classifiers such as decision trees can have multiple paths that lead to a prediction as an effector and one can speculate that different paths might relate to different classes of effectors, such as apoplastic or cytoplasmic effectors. Decision trees can also learn feature interactions, whereas Naïve Bayes classifiers identify the importance of individual features, but not relationships amongst features. This might be advantageous for effector prediction, e.g. a Naïve Bayes classifier can learn that a small protein size or a high cysteine content is important for effectors, but it does not learn that proteins have to be small and at the same time cysteine‐rich to be effectors. Unlike Naïve Bayes classifiers, decision trees are non‐parametric, which gives them the ability to, for example, assign a very low protein size to non‐effectors, a low to medium protein size to effectors and a large protein size to non‐effectors. However, decision trees are prone to overfitting, especially on small training datasets, which can lead to a limited ability to correctly classify unseen data. Naïve Bayes classifiers can deliver robust performance on small training datasets and an ensemble classifier, such as EffectorP 2.0, is capable of drawing on the strengths of both decision trees and Naïve Bayes classifiers.

On the current training set, low molecular weight is an important feature in fungal effector classification. However, it is possible that fungal pathogens employ classes of larger effector proteins which have thus far not been recognized. For example, the recently discovered Puccinia graminis f. sp. tritici effectors AvrSr50 (Chen et al., 2003) and AvrSr35 (Salcedo et al., 2005) are 132 and 578 amino acids long, respectively. With sufficient training data, EffectorP could learn to recognize classes of effectors that share no sequence similarity, yet are structurally conserved, such as MAX‐effectors (de Guillen et al., 2016). Machine learning classifiers trained to recognize oomycete RxLR effectors could be used to search for effectors with similar structural properties in fungi. In general, future re‐training of EffectorP on the expanding sets of experimentally supported effectors will be critical to retain its value. We envisage that, in the future, separate training sets of apoplastic fungal effectors and cytoplasmic fungal effectors could be of sufficient size to allow for the training of separate classifiers, which could potentially increase prediction accuracy. Although the machine learning classifier ApoplastP delivers accurate prediction of apoplastic protein localization for both plant and effector proteins (Sperschneider et al., 2008), other signals unique to apoplastic or cytoplasmic effectors might not be fully utilized by EffectorP as yet.

Another challenge is the choice of the negative training set, which should ideally contain no undiscovered effectors. However, the set of secreted fungal pathogen proteins is mostly unlabelled and will contain true positive effectors. To minimize this effect, we filtered the predicted pathogen secretomes for EffectorP 1.0‐predicted effector candidates, which removed predominantly small, cysteine‐rich effector candidates. This could introduce the possibility that a classifier trained on fungal effectors (many are small, cysteine‐rich proteins) and EffectorP 1.0‐filtered secreted pathogen proteins (many small, cysteine‐rich proteins removed) would bias a classifier towards the recognition of predominantly small, cysteine‐rich proteins as effectors. However, this does not seem to be the case for EffectorP 2.0. Although machine learning classifiers can, to some degree, be tolerant to noisy negative training data, in particular if the positive set is of high quality, undiscovered effectors might remain in the negative set and potentially bias predictions.

Practical recommendations for fungal effector prediction depend on the application. For example, for subsequent experimental validation in which time and resources are limited, a stringent effector screening approach might be most appropriate. This could involve taking either EffectorP 2.0‐predicted effectors, or effectors predicted by both versions of EffectorP 1.0/2.0 for maximum stringency. For maximum sensitivity, a union of effector candidates predicted by either EffectorP 1.0 or 2.0 could be used; however, this will also result in high false positive rates. If in planta expression data are available, effectors expressed highly during infection can be prioritized for experimental validation. Another approach would be to select effectors with highest probability; however, this has not been tested extensively by us. Nevertheless, we did observe that, during the identification of the Puccinia graminis f. sp. tritici effector AvrSr50 (Chen et al., 2003), where over 40 candidate genes had to be functionally screened, the application of EffectorP 2.0 and ApoplastP (Sperschneider et al., 2008) to predict the most likely effector to enter plant cells would have revealed AvrSr50 as the top candidate with highest probability. Overall, the re‐evaluation and re‐training of EffectorP have supported the power of machine learning for fungal effector prediction. Higher accuracy of fungal effector prediction will boost experimental validation success rates and aid in the understanding of effector biology.

Experimental Procedures

Training of the machine learning classifier

As a positive training set, we collected validated fungal effectors from the literature and then reduced sequence homology in this set by removing those that shared similarity with another effector in the set at bit score ≥ 50 using phmmer (Finn et al., 2015). Three negative training sets were generated based on secretomes predicted from annotated gene sets of publicly available genome assemblies of plant‐pathogenic fungi and symbionts (21 species, same species from the positive effector training set), animal‐pathogenic fungi (10 species) or saprophytic fungi (27 species) (Table 8). A protein was labelled as secreted if it was predicted to be secreted by the neural network predictor of SignalP 3 (Bendtsen et al., 2011) as well as by TargetP (Emanuelsson et al., 2006), and if it had no predicted transmembrane domain outside the first 60 amino acids using TMHMM (Krogh et al., 2015), as described previously for fungal effector prediction (Sperschneider et al., 2011). Each negative set was homology reduced by deleting proteins that shared sequence similarity (bit score ≤ 100, phmmer) with another in the negative set. We also applied EffectorP 1.0 (Sperschneider et al., 1996) to exclude predicted effectors from the fungal pathogen/symbiont secretomes. The WEKA tool box (version 3.8.1) was used to train and evaluate the performance of different machine learning classifiers (Hall et al., 2000), and feature vectors were calculated for each protein (Table 2). The training data are available at: http://effectorp.csiro.au/data.html.

Table 8.

Genomes that were used to predict secretomes for negative training data.

Ecology	Species	Reference
Fungal pathogen/symbiont	Melampsora lini, Puccinia graminis f. sp. tritici, P. striiformis f. sp. tritici, Blumeria graminis f. sp. hordei, B. graminis f. sp. tritici, Cladosporium fulvum, Leptosphaeria maculans, Fusarium oxysporum f. sp. lycopersici, F. graminearum, Magnaporthe oryzae, Rhynchosporium secalis, Verticillium dahliae, Ustilago maydis, U. hordei, Stagonospora nodorum, Botrytis cinerea, Pyrenophora tritici‐repentis, Laccaria bicolor, Zymoseptoria tritici, Colletotrichum graminicola, Sclerotinia sclerotiorum	Nemri et al. (2014), Duplessis et al. (2011), Cantu et al. (2011), Spanu et al. (2010), Wicker et al. (2013), de Wit et al. (2012), Rouxel et al. (2011), Ma et al. (2010), Cuomo et al. (2007), Dean et al. (2005), Penselin et al. (2016), Klosterman et al. (2011), Kämper et al. (2006), Laurie et al. (2012), Hane et al. (2007), Amselem et al. (2011), Manning et al. (2013), Martin et al. (2008), Goodwin et al. (2011), O'Connell et al. (2012)
Fungal saprophyte	Agaricus bisporus var. bisporus, Amanita thiersii, Aspergillus niger, A. oryzae, Coniophora puteana, Dacryopinax sp., Dichomitus squalens, Fomitiporia mediterranea, Fomitopsis pinicola, Gloeophyllum trabeum, Punctularia strigosozonata, Stereum hirsutum, Trametes versicolor, Wolfiporia cocos, Gymnopus luxurians, Hydnomerulius pinastri, Hypholoma sublateritium, Plicaturopsis crispa, Sphaerobolus stellatus, Hysterium pulicare, Neurospora crassa, Pichia stipitis, Pseudozyma antarctica, P. aphidis, Rhodosporidium toruloides, Saccharomyces cerevisiae, Coprinus cinereus	Morin et al. (2012), Hess et al. (2014), Andersen et al. (2011), Machida et al. (2005), Floudas et al. (2012), Kohler et al. (2015), Ohm et al. (2014), Galagan et al. (2003), Jeffries et al. (2007), Morita et al. (2013), Lorenz et al. (2014), Zhu et al. (2012), Goffeau et al. (1996), Stajich et al. (2010)
Animal pathogen	Batrachochytrium dendrobatidis, Candida albicans, Cordyceps militaris, Cryptococcus neoformans var. grubii, C. neoformans var. neoformans, Encephalitozoon cuniculi, Enterocytozoon bieneusi, Malassezia globosa, Metarhizium robertsii, Paracoccidioides brasiliensis	Rosenblum et al. (2010), Jones et al. (2004), Zheng et al. (2011), Loftus et al. (2005), Janbon et al. (2014), Katinka et al. (2001), Akiyoshi et al. (2009), Xu et al. (2007), Gao et al. (2011), Desjardins et al. (2011)

For the ensemble learner, we took 100 randomly selected samples of negative training data from each of the three negative sets (pathogen/symbiont secretomes, saprophyte secretomes and animal pathogen secretomes), each with 282 protein sequences, to give a ratio of 3 : 1 to the number of positive training examples. We then used WEKA to train Naïve Bayes classifiers on each of the 300 negative datasets with the same positive training set. We then repeated this procedure and trained C4.5 decision trees (J48 model in WEKA) on another 300 randomly chosen negative datasets from the three classes. For each set of 100 models, we selected the best‐performing models as those with the highest area under the curve (AUC). Overall, we chose a total of 50 models comprising: 10 Naïve Bayes classifiers and 10 C4.5 decision trees that discriminate between fungal effectors and secreted pathogen proteins; 10 Naïve Bayes classifiers and 10 C4.5 decision trees that discriminate between fungal effectors and secreted saprophyte proteins; and five Naïve Bayes classifiers and five C4.5 decision trees that discriminate between fungal effectors and secreted animal pathogen proteins. The ensemble classifier called EffectorP 2.0 returns a final prediction using a soft voting approach, which predicts the class label based on average probabilities for ‘effector’ and ‘non‐effector’ calculated by each classifier. Soft voting then returns the class with the highest average probability as the result. A protein is classified as an effector if it has a probability > 0.55. If it is predicted as an effector with probability 0.5–0.55, it is labelled as an ‘unlikely effector’ and is counted as a non‐effector in the evaluation.

Evaluation of EffectorP 2.0

We collected fungal, plant and mammalian proteins with experimentally validated localization to endoplasmic reticulum, Golgi or membranes or with GPI anchors from the UniProt database (search terms in Table S1, see Supporting Information), and predicted signal peptides using SignalP 4.1 (Petersen et al., 2003). We also collected fungal proteins from PHI‐base (Urban et al., 2007) from Fusarium, Magnaporthe, Ustilago, Sclerotinia, Botrytis, Zymoseptoria and Leptosphaeria pathogens, which are annotated as having an unaffected pathogenicity phenotype. All evaluation data are available at: http://effectorp.csiro.au/data.html.

In the evaluation, true positives (TPs), false positives (FPs), true negatives (TNs) and false negatives (FNs) are calculated. Accuracy is reported as (TP + TN)/(TP + TN + FP + FN), whereas sensitivity is the fraction of effectors that are correctly identified as such [TP/(TP + FN)] and specificity is the fraction of non‐effectors which are correctly identified as such [TN/(TN + FP)]. The positive predictive value (PPV) is the proportion of positive results that are true positives [TP/(TP + FP)]. Receiver operating characteristic (ROC) curves plot sensitivity against (1 – specificity) and the area under the curve (AUC) can be calculated. This value gives the probability that a classifier will rank a randomly chosen effector higher than a randomly chosen non‐effector. Therefore, a perfect classifier achieves an AUC of 1.0, whereas a random classifier achieves an AUC of only 0.5.

A small size classifier predicts a protein as an effector if it has a sequence length of ≤300 amino acids, and a small, cysteine‐rich classifier predicts a protein as an effector if it has a sequence length of ≤300 amino acids and ≥4 cysteines in its sequence.

Functional enrichment analysis and plotting

We performed sequence similarity searches against fungal proteins at the National Center for Biotechnology Information (NCBI) with Blast2GO 4.1.9 (Gotz et al., 2011) and default parameters. GO terms were reduced to the most specific terms and Fisher's exact tests were used to find over‐ and under‐represented terms. Enrichment was called at false discovery rate (FDR) < 0.05.

Plots were produced using ggplot2 (Wickham, 2009) and statistical significance was assessed with t‐tests using the ggsignif package (https://cran.r-project.org/web/packages/ggsignif/index.html). Significance thresholds according to t‐test are NS = not significant, *P < 0.05, **P < 0.01 and ***P < 0.001.

Author Contributions

J.S. planned and designed the research and developed the software. All authors analysed the data and wrote the manuscript.

Supporting information

Additional Supporting Information may be found in the online version of this article at the publisher's website:

Table S1 The UniProt search terms used for the collection of negative test sets.

Table S2 Average performance of the 50 models in 10‐fold cross‐validation.

Table S3 The genomes used for the evaluation.

Table S4 Effector predictions on secretomes from 93 fungal species.

Fig. S1 One of the 10 C4.5 decision trees that discriminates between fungal effectors and secreted non‐effectors from pathogen secretomes [10‐fold cross‐validation: sensitivity, 72.3%; false positive rate, 10.2%; precision, 70.1%; area under the curve (AUC), 0.786].

Fig. S2 One of the 10 C4.5 decision trees that discriminates between fungal effectors and secreted non‐effectors from pathogen secretomes [10‐fold cross‐validation: sensitivity, 69.1%; false positive rate, 11.7%; precision, 66.3%; area under the curve (AUC), 0.809].

Fig. S3 Distributions of all features used in the EffectorP 2.0 model. All data points were drawn on top of the box plots as black dots. Significance between groups is shown as horizontal brackets and was assessed using t‐tests. The lower and upper hinges correspond to the first and third quartiles and the upper (lower) whiskers extend from the hinge to the largest (smallest) value that is within 1.5 times the interquartile range of the hinge. Data beyond the end of the whiskers are outliers.