Optimizing support vector machine analysis in low density biological data sets

Abstract

We explore the effectiveness of Support Vector Machines (SVM) for classification in a sparse data set. Non-human primate models are utilized to analyze Alcohol Use Disorders (AUDs); however, the resulting data have a limited sample size. The challenge of low sample numbers and low replicates are explored using a variety of optimization strategies for feature extraction, including correlation, entropy, density, linear support vector machines for regression (SVR), backward SVR, and forward SVR. We investigate these approaches against the backdrop of the relationship between alcohol consumption and tibial bone mineral density. The results indicate that machine learning (ML) can effectively be used in cases of low and diverse biological data sets. The best relevance feature ranking strategies are correlation, SVR forward, and SVR backward.

I. INTRODUCTION

The rapid adoption of Machine Learning (ML) approaches to address issues in life science that do not respond well to classic statistical methods opens tremendous opportunities in data analytics. However, certain types of data sets are not optimally shaped to support soft learning due to high variance, low density, high dimensionality, and/or small sample size.

The interdependent characteristics that underlay Alcohol Use Disorders (AUD) spans numerous biological and behavioral disciplines and require deep understanding of influencing factors, such as sex, income, genetics, behavior, and co-morbidity with other disorders [1]. In human subjects these individual characteristics have uneven prevalence, intermittent responses, and low density, and represent the inherent difficulties with establishing robust longitudinal models [2].

Animal models have been used to compensate for these deficiencies, including the construction of a non-human primate (NHP) macaque model of oral alcohol self-administration. This model has the benefit of approximating human physiology and behavior. In addition, recent work has identified categorical levels of drinking severity, i.e., Low Drinkers (LD), Binge Drinkers (BD), Heavy Drinkers (HD), and Very Heaver Drinkers (VHD) [3], that are a reflection of AUD severity in humans. This is particularly true with respect to HD and VHD categories since they are associated with problems of dependence and other brain pathology [4], [5], [6]. Unfortunately, one consequence of the NHP model is that it has low throughput, which impacts sample sizes and the effectiveness of ML-mediated analysis.

II. CLASSIFIER CHOICE IN SPARSE, UNDER POWERED DATA SETS

While a range of ML applications have been used to address non-linear, binary, and multi-class classification problems in life science, there is a gap in the literature about the robustness of ML approaches in complex traits with limited sample sizes. Indeed, we have successfully used ML classification to identify strong behavioral contributors to drinking categories, but only after aggregating numerous animal cohorts [7]. There continues to be unknown effects of low replicate number in high degree feature space given behavioral data. We therefore examine the use of support vector machines for regression as a control model for prediction [8]. We use correlation, entropy, density, and forward-backward selection of features as indicators of the relationship between features and their relevance in modeling bone damage in primates [9]. This includes a ranking of feature selection methods to further identify which features are more predictive. In addition, we have chosen a single clinically relevant parameter to interrogate: bone mineral density (BMD). The study of the relationship between alcohol consumption patterns and BMD is an important initial step for understanding the effects of alcohol on bone.

In this paper we demonstrate that SVM can be employed to effectively classify parameters in low-density space. Whether the data is abundant or scarce, the mining and modeling of biological data is non-trivial because there is a risk of finding under-determined solutions or ill-posed models. This work illustrates potential solutions to feature selection under these constraints, and, consequently, demonstrate that the amount of alcohol consumed in the first two hour period of ethanol open access is the largest determining factor of BMD in monkeys intoxicated early in life.

III. METHODS

A. Primate Subjects

Postmortem tibial BMD was studied in 68 monkeys from the Oregon National Primate Research Center (ONPRC): 14 females and 54 males, representing Cynomolgus and Rhesus monkeys. Each animal was assigned a drinking category based on drinking behavior, see Table I [3].

TABLE I.

Summary of monkey cohorts.

Cohort Name	Total	Females	Males	BD	LD	HD	VHD
INIA Cyno 2	12	0	12	0	10	0	1
INIA Cyno 9	11	0	11	0	6	2	0
INIA Cyno 8	3	3	0	0	0	0	0
INIA Rhesus 7a	8	0	8	1	3	2	2
INIA Rhesus 7b	5	0	5	1	3	1	0
INIA Rhesus 4	10	0	10	4	5	1	0
INIA Rhesus 5	8	0	8	1	0	3	4
INIA Rhesus 6a	6	6	0	0	0	0	6
INIA Rhesus 6b	5	5	0	0	3	1	1
Total	68	14	54	7	30	10	14

B. Ethanol Self-Administration

All animals followed previously established procedures for schedule-induced polydipsia [10], [11]. After an induction period where all monkeys drank to levels that saturated metabolic capacity elevating their Blood Ethanol Content (BEC) over 50 mg/dl, equivalent to the 30^th session of 1.5 g/kg ethanol, animals had concurrent access to 4% w/v ethanol and a 22 h/d open-access to water and food in the form of 1g of banana flavored pellets (Noyes), provided at least three times a day during meals, and at least two hours apart. The open-access phase comprised 12 months.

C. Drinking Features for Analysis

Five attributes are analyzed to explore optimal SVM conditions for sparse data sets: drinking category, sex, age at first intoxication [12], [13], maximum bout volume, and ethanol consumption during induction. These produce a total of 14 features, see Table II. For example, from the maximum bout volume occurring during the first 120 minutes of daily open-access, five features are extracted: average maximum bout volume (μ), standard deviation of the maximum bout volume (σ), median of the maximum bout volume, maximum of the maximum bout volume (max), and minimum of the maximum bout volume (min). This amount is also measured as a fraction of what the animals drink. From this data six features are extracted: average percentage of ethanol consumed during induction (μ), median percentage of ethanol consumed during induction, total sum of the percentage of ethanol ingested during induction (Σ), standard deviation of the percentage of ethanol ingested during induction (σ), minimum percentage of ethanol consumed during induction (min), and maximum percentage of ethanol consumed during induction (max).

TABLE II.

Ranking of Informative Features. Each column shows a ranking methodology. The critical difference, δ_CD = 1.82 with α = 0.01, suggests that the top five ranked features are significantly better than the rest.

$ℱ$	Feature	Pearson	Entropy	Density	Linear SVR	SVR Backward	SVR Forward	Avg. Rank
1	Drinking Category	14	11	4	8	6	12	9.16
2	Sex	2	13	5	13	13	1	7.83
3	Age at First Intoxication	1	6	2	7	7	2	4.16
4	μ of Maximum Bout Vol.	5	3	14	1	5	4	5.33
5	σ of Maximum Bout Vol.	4	2	12	3	3	11	5.83
6	Median of Maximum Bout Vol.	3	4	10	2	4	3	4.33
7	max of Maximum Bout Vol.	11	8	7	10	8	9	8.83
8	min of Maximum Bout Vol.	7	5	3	12	14	14	9.16
9	μ % of EtOH During Ind.	13	10	13	6	1	13	9.33
10	Median % of EtOH During Ind.	9	12	8	5	9	10	8.83
11	Σ % of EtOH During Ind.	6	7	9	4	2	8	6.00
12	σ % of EtOH During Ind.	10	1	11	9	10	6	7.83
13	min % of EtOH During Ind.	8	9	6	11	11	5	8.33
14	max % of EtOH During Ind.	12	14	1	14	12	7	10.00

D. Estimation of Importance and Feature Ranking

Feature ranking is an approach to feature selection that reduces the risk of poor performance and high complexity [9]. Three methods are used to evaluate features based on individual relevance ranking: Pearson correlation, entropy, and density. Three methods are also used to evaluate combined relevance rankings based on the well-known support vector machines for regression (SVRs) [8]: linear SVR, backward SVR, and SVR forward.

1). Individual Relevance Ranking using Entropy:

Entropy is a measure of the amount of information that is contained in a random variable or feature [14]. If a feature is categorical, e.g., drinking category or sex, we expect entropy based on the distribution of those variable. Sex represents a binary category and low entropy, indicating high predictability. On the other hand, the entropy of drinking category is more evenly distributed and therefore has higher entropy and lower predictability. Note that calculating entropy requires no knowledge of the target variable Y , which makes this process faster to compute, in comparison with Pearson correlation coefficient, but of the same overall complexity: $𝒪\left(\mathit{mn}\right)$.

2). Individual Relevance Ranking using Density:

High density features are highly correlated with many other variables, although not necessarily redundant [9], and may be calculated using the differences of Pearson correlation coefficients. This is important in the study of complex trait disorders, such as AUD, where numerous factors impact outcome [10], [3], [15], [7]. High density features may be calculated as a cost of increased complexity, $𝒪\left(m^{2}n\right)$, but using computed averages and centered vectors leads to a complexity of $𝒪\left(\mathit{mn}\right)$ in the average case.

3). Combined Relevance Ranking:

SVRs have been extensively developed to reduce the high complexity of learning algorithms when kernel mappings of data are used [8] and in problems related to classification [16], [17], [18]. Similarly, they can be used for regression tasks with minor adjustments. The complexity of finding a solution to the SVR optimization problem, using the SMO algorithm with a linear kernel, is $𝒪\left(m^{2}n\right)$, without considering the cost of the grid search.

E. Feature Selection

We explore two major strategies for ranking sets of features: forward selection and backward selection [9]. Backward selection considers the set of all features first and removes features that produce the smallest contribution. It is also computationally expensive, pushing the complexity of SVR to $𝒪\left(m^{3}n\right)$ due to the kernel function. And since the process of having sets of n−1, n−2, … , 1 features selected corresponds to a series that converges to $n(n-1)/2=𝒪\left(n^2\right)$, then the total overall complexity becomes $𝒪\left(m^3{n}^2\right)$. This complexity does not consider the cost of grid search and leave-one out cross validation.

The forward selection of features begins with an empty set and adds individual features one by one, selecting the feature that provided the best score. This process is fast at the beginning since it starts with an empty set, however, it could miss holistic relationships among features, and is computationally expensive, with a complexity of $𝒪\left(m^{3}n\right)$.

F. Computational Resources for Analysis

All computations were executed on Monkey Alcohol and Tissue Research Resource (MATRR) [19] servers (four Intel Xeon E5620 processors at 2.4 GHz, with 47 GB of RAM and 1.7 TB storage). Statistical analysis and data processing was completed using Python’s Pandas, NumPy, and SciPy packages. For all ML algorithms we used Python’s Scikit-Learn package.

IV. RESULTS AND DISCUSSION

The identification of features associated with behavioral drinking patterns in a NHP open access drinking model are used to explore the effectiveness of SVM and associated feature selection models in low data space. Given several cohorts of monkeys and data in the MATRR, the identification of features that impact BMD are explored using a variety of methods.

A. Tibial Bone Mineral Density

Average BMD of all samples is found to be μ = 0.376 g/cm², with a standard deviation of σ = 0.049g/cm². Figure 1 depicts BMD broken down by gender and drinking category, and suggests no apparent BMD differences between drinking categories. Sex differences are also within the standard deviation. There is likewise no statistical correlation between BMD and other examined single features.

Fig. 1. Tibial bone mineral density by drinking category and sex.

Drinking categories are displayed along the horizontal axis: low drinker (LD), binge drinker (BD), heavy drinker (HD), and very heavy drinker (VHD). No females are reported as BDs. There is no significant difference between drinking categories or sex.

B. Feature Ranking

Each of the fourteen available features in our data set are ranked using each of the ranking methodologies, see Table II. However, given the small sample size, average rank may be misleading. To establish a more robust ranking approach, a Friedman test is performed [20]. The Friedman statistic is determined to be ${\chi }_F^2=17.3143$ with p = 0.1853, with a level of significance of α = 0.05. Furthermore, the critical difference δ_CD using the Nemenji’s test was calculated to determine if two features are significantly different if the corresponding average ranks differ [21]. The value of the critical difference for a confidence level of α = 0.01 corresponds to δ_CD = 1.82. Using the δ_CD value we observe that the top four features are not significantly different from each other, but are significantly better than the rest of the features, suggesting that the most contributing features are Drinking Category, Sex, Age at First Intoxication, μ of Maximum Bout Volume, σ of Maximum Bout Volume.

Feature Testing

Relevant features are tested based on size-increasing sets according to rank. The input to the algorithm is the data set $𝒟$ and a vector r = [r₁, r₂, … , r_n], where r₁ is the index of the best ranked feature and r_n is the index of the worst ranked feature. Note, however, that r is different for each ranking methodology. E.g., consider the column entitled “Pearson” in Table II, the corresponding r would be: r = [3, 2, 6, 5, 4, … , 1]

We perform a traditional grid search to find the best set of hyper-parameters {C, γ, ν} that produce the best 10-fold cross validation R² score. The search is conducted in the logarithmic spaces C ∈ {2⁻⁵, 2⁻⁴, … , 2¹⁵} and γ ∈ {2⁻¹⁵, 2⁻¹⁴, … , 2³}; and the linear space ν ∈ {0.05, 0.10, … , 0.95}. Then, with the best set of hyper-parameters, ν–SVRs are trained with a leave-one-out (LOO) cross validation strategy, which is proven to provide more accurate error estimates in smaller data sets [22].

Results for all of the relevant feature ranking methods are depicted in Figure 2, which plots all error vectors starting with one feature. Note that since the first feature added, i.e., the best, is distinct for every method, the initial score varies. However, all converge to the same score once all fourteen features are considered together. Ideally, we want the set of features that achieves the highest score with the smallest number of features. SVR forward and Pearson correlation coefficient achieve their highest score with only three features, while SVR backward performs highest with six features.

Fig. 2. Performance of feature sets using the R² score.

Pearson correlation coefficient, SVR backward, and SVR forward score the highest among all of the methods.

In order to determine which features appear the most frequently, we analyze which features are in the set that produced the highest score for each method. Table III presents a summary of which features are frequently part of the best set of features. The table also presents a weighted frequency count using the rank of each method. The rank was determined as the best overall score of a given method regardless of the number of features. From Figure 2 it can be determined that the best score corresponds to SVM backward and the worst corresponds to Density.

TABLE III.

Analysis of sets of features producing the best score. Column data is formatted as b/r, where b =1 if feature is part of the best scoring set; r is the method rank.

$ℱ$	Feature	C	H	D	Lin. SVR	SVR Bwd.	SVR Fwd.	Freq. (wFreq.)
1	Drinking Category		1/4	1/6		1/1		3 (1.42)
2	Sex	1/2.5		1/6			1/2.5	3 (0.97)
3	Age at First Intoxication	1/2.5	1/4	1/6			1/2.5	4 (1.22)
4	μ of Maximum Bout Vol.		1/4		1/5	1/1		3 (1.45)
5	σ of Maximum Bout Vol.		1/4		1/5	1/1		3 (1.45)
6	Median of Maximum Bout Vol.	1/2.5	1/4		1/5	1/1	1/2.5	5 (2.25)
7	max of Maximum Bout Vol.		1/4					1 (0.25)
8	min of Maximum Bout Vol.		1/4	1/6				2 (0.42)
9	μ % of EtOH During Ind.		1/4			1/1		2 (1.25)
10	Median % of EtOH During Ind.
11	Σ % of EtOH During Ind.		1/4		1/5	1/1		3 (1.45)
12	σ % of EtOH During Ind.		1/4					1 (0.25)
13	min % of EtOH During Ind.		1/4					1 (0.25)
14	max % of EtOH During Ind.			1/6				1 (0.17)

Careful inspection of Table III reveals that, considering the frequency count of the features, the two features that are frequently part of the best sets of features are Median of Maximum Bout Volume and Age at First Intoxication. On the other hand, if we consider the weighted frequency count which utilizes the rank, we observe that the top features are Median of Maximum Bout Volume with a w_Freq. = 2.25 and three ties with w_Freq. = 1.45.

Figure 3 depicts how two features, namely Median Maximum Bout Volume and Age at First Intoxication, interact with each other in predicting BMD. The figure suggests that lower BMD is associated with a median maximum bout volume between 10 and 150 mL for young monkeys whose age is between 1400 and 1700 days. The risk of having low BMD is less for older monkeys of ages greater than 2400 days. These two features do not appear to have a linear relationship in a two-dimensional plane; however, using the kernel method in SVRs, the possibility of having a linear relationship in a higher-dimensional space is often assumed.

Fig. 3. BMD values using Median Maximum Bout Volume and Age at First Intoxication as predictors.

Predicted BMD suggests that young monkeys are at higher risk of low BMDs if their Median Maximum Bout Volume is in the range of 10 to 150 mL.

A similar analysis is shown in Figure 4. In this case the two features analyzed are Median Maximum Bout Volume and Total Sum of the Percentage of Ethanol Consumed During Induction. The relationship between these two features appears to be quasi-linear, suggesting a more evident dependence or correlation. From the figure, we can observe that if the Median Maximum Bout Volume is less than 100 mL then there is a higher risk of low BMD for almost any sum of the percentage of ethanol, while the risk decreases in the opposite direction.

Fig. 4. BMD values using Median Maximum Bout Volume and Total Sum of the percentage of Ethanol During Induction.

Results suggest that Median Maximum Bout Volume provides a higher correlation to BMD than the percentage of Ethanol intake during Induction.

V. CONCLUSIONS

Several predictive features for bone mineral density were identified using relevance feature ranking techniques. These attributes are both categorical and quantitative, and include Median of Maximum Bout Volume, μ of Maximum Bout Volume, σ of Maximum Bout Volume, Age at First Intoxication, and Total Sum of the Percentage of Ethanol Consumed During Induction

We also analyzed the complexity of the feature ranking algorithms and observed that the Pearson correlation coefficient is significantly less expensive and performs well for up to three features; after that, Pearson does not consider the interdependence of the data. The SVR Backward selection strategy was the best method, since it considers all of the features at once and their relationship as a group; however, it is more expensive than Pearson’s correlation coefficient.

Encountering low sample sizes in certain biological data sets is inevitable, and careful application of optimization techniques can ensure that these sets maintain their utility. This approach also demonstrates an alternative to sentiment analysis [23].