Establishment of the benchmarking tool for evaluating the operation of biorepositories for pathogenic resource using a modified Delphi method

Highlights

• •Scientific questions: Given considerations to the significance of bio-resource, the sustainable, stable, and high-quality operation of biorepositories attracts high attention from governments around the world. However, during the practices of evaluating the operation of biorepositories for pathogenic resource, there is still a lack of quantitative and specific benchmarking tools.
• •Evidence before this study: To better standardize the professional operation of biorepositories, many international organizations and industry associations have formulated guidelines and carried out investigations. For example, the Organisation for Economic Co-operation and Development (OECD), World Federation for Culture Collections (WFCC), International Society for Biological and Environmental Repositories (ISBER), Common Access to Biological Resources and Information (CABRI) framework, and so on.
• •New findings: The Delphi method is an iterative and structured group discussion procedure for experts to evaluate complex issues with uncertainty and incomplete knowledge system. It is frequently applied to formulate the best practices or guidelines especially in the field of medicine and health care.
• •Significance of the study: This study, by adopting modified Delphi method, first established a benchmarking framework for evaluating biorepositories for pathogenic resources, and developed a benchmarking tool for their comprehensive evaluation and improvement of the management and operation.

Abstract

In recent years, as the infectious diseases caused by pathogens such as novel coronavirus and mpox (formerly called monkeypox) spread globally, the significance of identification, preservation, and sharing of pathogenic resources become prominent. Along with the rapid development of biorepositories, it is imperative to evaluate their operation in a scientific manner. By using the literature review and modified Delphi method, this study develops a benchmarking tool for the comprehensive evaluation of the operation of biorepositories for pathogenic resources. The effective response rates of both rounds of expert surveys were 100 %. The authority coefficients (Cr) were 0.82 and 0.85, respectively, manifesting the reliability of consultation results. In the second-round survey, the Kendall’s coefficient of concordance (Kendall’s W) of all indicators ranged from 0.09 to 0.31 (P < 0.001), the comprehensive score ranged from 4.02 to 4.94, the standard deviation ranged from 0.21 to 0.77, and the coefficient of variation (CV) ranged from 0.04 to 0.22, indicating that the expert opinions reached consensus. The final benchmarking tool was composed of 4 primary indicators, 12 secondary indicators, and 65 tertiary indicators. The weights of the four primary indicators allocated through the rank-sum ratio method, namely organizational structure, management requirements, biobanking capacity, and sharing capacity, were 30.50 %, 30.08 %, 25.45 %, and 13.97 %, respectively. The benchmarking tool established in this study provides references for the comprehensive evaluation of the operation and puts forward advice for the sustainable development of biorepositories for pathogenic resources.

1. Introduction

A biorepository, biological resource center (BRC), or biobank refers to a structured institution that stores biological samples and related data for current and future research purposes. Their main activities include the collection, processing, storage, and distribution of biological samples and related clinical data and information [1], [2], [3].

A biorepository for pathogenic resources is a type of institution that carries out biobanking activities for pathogenic bacteria, fungi, or viruses. Pathogenic resources are crucial for various fields such as infectious disease prevention and control, medical treatment, testing and quarantine, scientific research, education and production, etc. It is a significant prerequisite for ensuring experimental quality and results in primary scientific research. Proper preservation of pathogenic resources can maintain the purity, activity, and integrity of genetic information, avoid mutation and degradation, maintain a high survival rate and genetic stability for a long time, and thus enable its role in supporting long-term scientific research and production [4]. On the contrary, the accidental or intentional release of highly pathogenic strains may pose hazards to human and / or animal health and have a significant impact on biosafety and security [5]. Therefore, the sustainable, stable, and high-quality operation of biorepositories attracts high attention of governments around the world.

After years of practice, the foundation of global biobanking of pathogenic resources has been established in terms of legal requirements and actual practices. However, there are still many aspects that need to be continuously emphasized and improved, such as sufficient funds for operation, complete organizational structure, well-rounded regulatory documents, full implementation of the quality management system, consistent quality of pathogenic resources, high degree of informatization, regular international exchange and cooperation, and appropriate utilization and sharing of resources [6], [7], [8], [9], [10], [11].

In the past few decades, diverse biorepository systems have been established around the world according to the pragmatic needs of each country, and many international organizations have developed special guiding principles for the operation and development of biorepositories. For example, the Organisation for Economic Co-operation and Development (OECD), World Federation for Culture Collections (WFCC), International Society for Biological and Environmental Repositories (ISBER), Common Access to Biological Resources and Information (CABRI) framework, etc. have proposed a series of guidelines for the organization, operation, and quality management standards [12], [13], [14], [15]. However, during the practices of evaluation of biorepositories, there is still a lack of quantitative and specific benchmarking tools.

On the basis of literature review, and study of the above-mentioned international best practices related to biobanking and China's domestic laws, regulations and technical standards, this study adopted the modified Delphi method to initially establish a benchmarking framework to evaluate biorepositories for pathogenic resources, and provide a benchmarking tool for comprehensive evaluation and improvement of their operation.

2. Materials and methods

2.1. Literature review

Through retrieving “biobank” “biorepository” combined with “evaluation” “assessment” and “management” at English databases including PubMed, SpringerLink, and Elsevier Science Direct, and at Chinese databases including the China National Knowledge Infrastructure, Wanfang Data Knowledge Service Platform, and National Social Sciences Database, 45,056 articles were collected from 2004 to 2023. Another 26 Chinese and international guidelines were also included. 39,827 and 5,132 articles were excluded in two rounds by screening the published time and research type, and by examining the titles and abstracts, and another 11 articles were further excluded through detailed reading. Sixty Chinese articles and 52 English articles were ultimately selected. After summarizing and sorting out the current research status, existing problems, guideline documents, and other related influencing factors of the biorepositories, the benchmarking framework was initially formed.

2.2. Modified Delphi method

The Delphi method is an iterative and structured group discussion procedure for experts to evaluate complex issues with uncertainty and incomplete knowledge system [16], [17], which is conducive to integrating evidential knowledge, practical knowledge, and even daily life experience [18]. It is frequently applied to formulate the best practices or guidelines, especially in the field of medical and health care [19], [20], [21]. The modified Delphi Method, based on the classical Delphi method, involves meetings for experts to discuss or rate the results, or the use of progenerated items from the literature in the first-round survey [20].

2.3. Selection of consultants

By combining target sampling and convenience sampling, experts in the fields of biorepository management, laboratory management, biological resource sharing, information technology for biorepositories, biobank policies, scientific research on biorepositories and health policies research were selected, and an advisory group was set up. The inclusion criteria of consultants are as follows: (1) Long-term work in the professional field of preservation for more than 5 years; (2) Holding a master's degree or higher, or an associate senior title or above; (3) The capability to make constructive comments on research topics; and (4) Persistence in completing the surveys.

2.4. Design of questionnaire

The first-round questionnaire is made based on the benchmarking framework, which includes four primary indicators on organizational structure, management requirements, biobanking capacity and sharing capacity, as well as 12 secondary indicators and 71 tertiary indicators. The questionnaire mainly includes the form-filling introduction, basic information table of the expert, definitions of the benchmarking indicators, scoring table for the importance and accessibility of the indicators. The second-round questionnaire was revised based on statistical analysis and consultation on the advisory group. The comprehensive score and coefficient of variant (CV) of all indicators in the first-round questionnaire is marked in the scoring table during the second-round questionnaire to provide references for experts. The CV is calculated by the standard deviation and comprehensive score of the indicator as showed in formula (1). The two rounds of surveys were carried out respectively in August 2023 and October 2023.

$$\mathrm{C}V=\frac{{\mathrm{\sigma }}_j}{{\overline{X}}_j}$$ (1)

2.5. Expert scoring

The importance and accessibility were scored using the Likert 5 scale, with 5 points for “important” and “easy to access”, 4 points for “quite important” and “accessible”, 3 points for “moderately important” and “not easy to access”, 2 points for “not so important” and “not all available”, 1 point for “not important” and “not available”. A comprehensive score, which is the mean of importance and accessibility, was used to represent the final score. Experts evaluate their own authority coefficient (Cr), which is the mean of judgement coefficient (Cα) and familiarity coefficient (Cs) as showed in formula (2). The Cα consists of theoretical analysis, work experience, understanding from domestic and foreign counterparts and intuitive judgment. The Cs consists of five grades: familiar, quite familiar, moderately familiar, not so familiar, and not familiar. Experts also assigned weights to the importance and accessibility by scoring integer from 1 to 100.

$$\mathrm{C}\mathrm{r}=1/2(\mathrm{C}\mathrm{\alpha }+\mathrm{C}\mathrm{s})$$ (2)

2.6. Standards for screening indicators

According to the results of the first-round survey, the threshold value method was used to select the indicators that had more consistent opinions among the experts. The screening criteria for indicators were as follows: (1) Mean importance score > 3.50, and mean accessibility score > 3.50; and (2) CV < 0.20. When meeting none of the above conditions, an indicator should be deleted. When meeting one of the above conditions, an indicator should be revised upon the experts suggestions or deleted following internal discussion of the advisory group. When meeting all of the above conditions, an indicator should be retained [22], [23].

The Microsoft Excel 2019 software was used to establish the database, and SPSS 22.0 software was used for statistical analysis. The positive coefficient of experts was measured by the effective response rate of questionnaires. The Cr was used to measure the authority degree of experts.

When Cr ≥ 0.70, it is indicated that the consultation results are reliable. The comprehensive score and standard deviation were used to represent the experts' overall assessment of the indicators. The CV and Kendall’s coefficient of concordance (Kendall’s W) were used to measure the coordination of expert scoring. Smaller CV stands for higher coordination of experts’ opinion. When CV < 0.20, the coordination is considered good. Bigger Kendall’s W means higher coordination. Kendall’s W needs to be tested for consistency, and the result has statistical significance when P < 0.05. The rank-sum ratio is used to determine the weight of the indicators [24].

3. Results

3.1. Summary of consultants

A total of 26 experts from public institutions, universities, and scientific research institutions, who have been engaged in biorepository management, laboratory management, biological resource sharing, information technology for biorepositories, biobank policies, scientific research on biorepositories, and health policies research for more than 5 years, were invited in this study. The years of working experience, academic degrees and professional titles of the experts are presented in Table 1.

Table 1.

Basic information of experts.

Item	Number	Percentage (%)	Cumulative percentage (%)
Years of working experience
≤10	3	11.54	11.54
11–15	9	34.62	46.16
16–20	7	26.92	73.08
＞20	7	26.92	100.00
Academic degree
Undergraduate	4	15.38	15.38
Postgraduate	9	34.62	50.00
Doctorate	13	50.00	100.00
Professional title
Senior	14	53.85	53.85
Associate senior	7	26.92	80.77
Intermediate	4	15.38	96.15
Junior	1	3.85	100.00

3.2. Expert positive coefficient and Cr

Two rounds of surveys were carried out among the same 26 experts, with 26 valid questionnaires recovered in both rounds. The response rate of experts was 100 %. In the first and second round of surveys, the Cα were 0.90 and 0.92, respectively, the Cs were 0.73 and 0.77, respectively, which made the Cr 0.82 and 0.85, respectively. The Cr of both rounds exceeding 0.70 indicated the authority of scoring.

3.3. Expert scoring results

In the first-round survey, the comprehensive score of indicators ranged from 4.10 to 4.85, the standard deviation ranged from 0.27 to 0.76, the CV ranged from 0.06 to 0.23, and the CV for 83.91 % of the indicators was less than 0.20. All indicators’ mean importance score and mean accessibility score were > 3.50. However, 15 indicators failed to meet the criterion of CV < 0.20, and were either deleted or revised upon the discussion of the advisory group. In the second-round survey, the comprehensive score ranged from 4.02 to 4.94, the standard deviation ranged from 0.21 to 0.77, the CV ranged from 0.04 to 0.22, and the CV for 96.30 % indicators was less than 0.20. More details are provided in Table 2.

Table 2.

Indicator statistics of the second-round survey.

Indicator	Importance		Accessibility
	Score ($\overline{x}$ ± s)	CV	Score ($\overline{x}$ ± s)	CV
1 Organizational structure	4.96 ± 0.20	0.040	4.69 ± 0.47	0.100
1.1 Departments and personnel setup	4.85 ± 0.46	0.096	4.81 ± 0.40	0.084
1.1.1 Management layer	4.92 ± 0.27	0.055	4.77 ± 0.43	0.090
1.1.2 Departments and positions	4.85 ± 0.37	0.076	4.65 ± 0.49	0.104
1.1.3 Full-time personnel	4.58 ± 0.76	0.166	4.50 ± 0.65	0.144
1.1.4 Depositories	5.00 ± 0.00	0.000	4.65 ± 0.56	0.121
1.1.5 IT specialist	4.46 ± 0.71	0.158	4.38 ± 0.64	0.145
1.1.6 Biosafety specialist	4.77 ± 0.51	0.108	4.81 ± 0.40	0.084
1.1.7 Maintenance technician	4.58 ± 0.76	0.166	4.54 ± 0.65	0.143
1.2 Funding source	4.58 ± 0.58	0.126	4.19 ± 0.75	0.179
1.2.1 Earmarked funds	4.65 ± 0.69	0.148	4.12 ± 0.82	0.198
1.2.2 Supplementary funds	4.12 ± 0.91	0.221	3.92 ± 0.85	0.216
1.2.3 Annual budget and medium/long-term development plan	4.58 ± 0.64	0.140	4.12 ± 0.82	0.198
2 Management requirements	4.96 ± 0.20	0.040	4.65 ± 0.49	0.104
2.1 Biobanking management	4.88 ± 0.43	0.088	4.73 ± 0.45	0.096
2.1.1 Procedures for out-put and in-put depository	4.92 ± 0.39	0.080	4.77 ± 0.43	0.090
2.1.2 Procedures for sharing	4.73 ± 0.60	0.128	4.58 ± 0.58	0.126
2.1.3 Procedures for transportation	4.81 ± 0.49	0.102	4.85 ± 0.37	0.076
2.1.4 Procedures for regular checkup	4.73 ± 0.53	0.113	4.69 ± 0.47	0.100
2.1.5 Procedures for duplicates preservation	4.92 ± 0.27	0.055	4.77 ± 0.43	0.090
2.1.6 Procedures for destruction	4.96 ± 0.20	0.040	4.81 ± 0.40	0.084
2.1.7 Procedures for documentation	4.92 ± 0.27	0.055	4.77 ± 0.43	0.090
2.2 Laboratory management	4.73 ± 0.53	0.113	4.77 ± 0.43	0.090
2.2.1 Lab certifications	4.81 ± 0.40	0.084	4.77 ± 0.43	0.090
2.2.2 Procedures for risk assessment	4.88 ± 0.33	0.067	4.81 ± 0.40	0.084
2.2.3 Procedures for sterilization	4.88 ± 0.33	0.067	4.88 ± 0.33	0.067
2.2.4 Emergency response plan	4.88 ± 0.43	0.088	4.88 ± 0.33	0.067
2.2.5 Procedures for biosecurity	4.81 ± 0.40	0.084	4.69 ± 0.47	0.100
2.2.6 Operational requirements for experiments	4.81 ± 0.49	0.102	4.85 ± 0.37	0.076
2.2.7 Procedures for procurement of experimental materials	4.35 ± 0.69	0.158	4.62 ± 0.50	0.107
2.3 IT management	4.69 ± 0.55	0.117	4.50 ± 0.76	0.169
2.3.1 Catalogue management	4.88 ± 0.43	0.088	4.58 ± 0.64	0.140
2.3.2 Confidential information management	4.77 ± 0.51	0.108	4.46 ± 0.71	0.158
2.3.3 Database management	4.77 ± 0.51	0.108	4.27 ± 0.67	0.156
2.4 Capacity building	4.77 ± 0.43	0.090	4.46 ± 0.58	0.130
2.4.1 Training for technical specialists	4.77 ± 0.51	0.108	4.54 ± 0.58	0.128
2.4.2 Procedures for updating management documents	4.65 ± 0.56	0.121	4.65 ± 0.56	0.121
2.4.3 Annual report mechanism	4.46 ± 0.71	0.158	4.46 ± 0.58	0.130
2.4.4 Academic exchange and meetings	4.58 ± 0.64	0.140	4.58 ± 0.50	0.110
2.4.5 Application of scientific research	4.46 ± 0.71	0.158	4.42 ± 0.58	0.131
2.4.6 International cooperation	4.42 ± 0.58	0.131	4.35 ± 0.63	0.145
2.4.7 QMS certification	4.73 ± 0.53	0.113	4.54 ± 0.58	0.128
3 Biobanking capacity	4.88 ± 0.43	0.088	4.42 ± 0.70	0.159
3.1 Site layouts	4.85 ± 0.37	0.076	4.65 ± 0.56	0.121
3.1.1 Overall functional zoning	4.96 ± 0.20	0.040	4.81 ± 0.49	0.102
3.1.2 Receiving and distributing zone	4.96 ± 0.20	0.040	4.92 ± 0.27	0.055
3.1.3 Experimental zone	4.96 ± 0.20	0.040	4.88 ± 0.33	0.067
3.1.4 Preservation zone	4.96 ± 0.20	0.040	4.81 ± 0.49	0.102
3.1.5 Office area	4.69 ± 0.62	0.132	4.88 ± 0.33	0.067
3.2 Facility and equipment	4.81 ± 0.40	0.084	4.54 ± 0.58	0.128
3.2.1 Experimental equipment	4.92 ± 0.27	0.055	4.88 ± 0.33	0.067
3.2.2 Preservation equipment	4.92 ± 0.39	0.080	4.85 ± 0.37	0.076
3.2.3 Ventilation and gas supply equipment	4.85 ± 0.46	0.096	4.81 ± 0.40	0.084
3.2.4 Power supply and lighting facility	4.85 ± 0.46	0.096	4.81 ± 0.40	0.084
3.2.5 Monitoring and communication facility	4.77 ± 0.51	0.108	4.65 ± 0.56	0.121
3.2.6 Identifier equipment	4.81 ± 0.49	0.102	4.73 ± 0.53	0.113
3.2.7 Security facility	4.73 ± 0.53	0.113	4.58 ± 0.58	0.126
3.2.8 Equipment for professional personnel	4.77 ± 0.51	0.108	4.73 ± 0.45	0.096
3.3 Physical resource	4.96 ± 0.20	0.040	4.38 ± 0.57	0.130
3.3.1 Resource type	4.92 ± 0.39	0.080	4.77 ± 0.51	0.108
3.3.2 Number of original isolations	4.65 ± 0.69	0.148	4.42 ± 0.76	0.171
3.3.3 Number of preservations	4.85 ± 0.46	0.096	4.46 ± 0.76	0.171
3.3.4 Number of type cultures	4.58 ± 0.58	0.126	4.54 ± 0.71	0.156
3.3.5 Preservation methods	4.85 ± 0.37	0.076	4.65 ± 0.69	0.148
3.3.6 Resource duplicates	4.92 ± 0.27	0.055	4.46 ± 0.65	0.145
3.3.7 Regular checkup	4.92 ± 0.27	0.055	4.46 ± 0.58	0.130
3.4 Data resource	4.92 ± 0.27	0.055	4.19 ± 0.69	0.166
3.4.1 Standardized description of data resource	4.96 ± 0.20	0.040	4.50 ± 0.51	0.113
3.4.2 Operational management of database	4.88 ± 0.33	0.067	4.54 ± 0.51	0.112
3.4.3 Service management of database	4.77 ± 0.43	0.090	4.46 ± 0.58	0.130
4 Sharing capacity	4.65 ± 0.56	0.121	4.00 ± 0.85	0.212
4.1 Sharing amount	4.58 ± 0.58	0.126	4.12 ± 0.65	0.158
4.1.1 Amount of supply	4.58 ± 0.58	0.126	4.46 ± 0.51	0.114
4.1.2 Service objects	4.42 ± 0.58	0.131	4.35 ± 0.56	0.129
4.1.3 Purposes of utilization	4.77 ± 0.43	0.090	4.42 ± 0.58	0.131
4.1.4 Amount of receipt	4.62 ± 0.57	0.124	4.38 ± 0.57	0.130
4.1.5 Source of receipt	4.62 ± 0.57	0.124	4.19 ± 0.69	0.166
4.2 Sharing mechanism	4.62 ± 0.50	0.107	4.19 ± 0.63	0.151
4.2.1 Sharing agreement	4.73 ± 0.45	0.096	4.31 ± 0.62	0.143
4.2.2 Sharing time limit	4.42 ± 0.58	0.131	4.12 ± 0.65	0.158
4.2.3 Standardized sharing process	4.73 ± 0.45	0.096	4.38 ± 0.50	0.113

Note: The score of importance and accessibility were represented as mean ± standard deviation. Abbreviations: CV, coefficient of variation; QMS, quality management system; IT, information technology.

In the first-round survey, the Kendall’s W for the accessibility of biobanking capacity and the importance and accessibility of sharing capacity did not show statistical significance. In the second-round survey, the Kendall’s W for all indicators ranged from 0.09 to 0.31 with P < 0.01. More details are shown in Table 3.

Table 3.

The coordination degree for two rounds of surveys.

Primary indicator		Kendall’s W		χ2 value		P value
		First round	Second round	First round	Second round	First round	Second round
Organizational structure	Importance	0.18	0.17	138.51	68.34	＜0.001	＜0.001
	Accessibility	0.18	0.31	140.07	97.15	＜0.001	＜0.001
Management requirements	Importance	0.19	0.17	156.28	126.37	＜0.001	＜0.001
	Accessibility	0.15	0.16	124.51	115.16	＜0.001	＜0.001
Biobanking capacity	Importance	0.11	0.09	32.79	61.04	＜0.001	＜0.001
	Accessibility	0.04	0.20	11.97	136.79	＞0.05	＜0.001
Sharing capacity	Importance	0.05	0.09	12.07	61.04	＞0.05	＜0.001
	Accessibility	0.02	0.10	5.63	25.79	＞0.05	＜0.001

Abbreviation: Kendall’s W, Kendall’s coefficient of concordance.

3.4. Deletion and revision of indicators

According to the results of the first-round survey, six tertiary indicators were deleted based on their CVs following discussion of the advisory group, including the specialist for legal affairs, security, medium and long-term strategic goals, personnel health-monitoring, resource transfer system, and public popularization. The definitions of 12 tertiary indicators were revised in response to consultants’ advice after discussion of the advisory group. The names of four indicators out of these 12 tertiary indicators were edited and altered to better align with practical requirements, while the remaining eight indicators were either modified in their descriptions or added with more detailed and specific requirements.

3.5. Assignment of weights

The weight and composite weight of all indicators were calculated using the rank-sum ratio method. Among the primary indicators, the weights for organizational structure, management requirements, biobanking capacity and sharing capacity were 30.50 %, 30.08 %, 25.45 %, and 13.97 %, respectively. The top five secondary indicators in terms of weight are departments and personnel setup, biobanking management, laboratory management, physical resources, and data resources. The weight and composite of each indicator are presented in Table 4.

Table 4.

The weight and composite weight of each indicator.

Primary indicator	Weight (%)	Secondary indicator	Weight (%)	Composite weight (%)	Tertiary indicator	Weight (%)	Composite weight (%)
1 Organizational structure	30.50	1.1 Departments and personnel setup	66.71	11.43	1.1.1 Management layer	16.70	1.85
					1.1.2 Departments and positions	15.36	1.70
					1.1.3 Full-time personnel	12.93	1.43
					1.1.4 Depositories	16.84	1.87
					1.1.5 IT specialist	10.01	1.11
					1.1.6 Biosafety specialist	15.67	1.74
					1.1.7 Maintenance technician	12.49	1.38
		1.2 Funding source	33.29	5.70	1.2.1 Earmarked funds	38.87	1.05
					1.2.2 Supplementary funds	24.63	0.67
					1.2.3 Annual budget and medium/long-term development plan	36.51	0.99
2 Management requirements	30.08	2.1 Biobanking management	29.67	11.14	2.1.1 Procedures for out-put and in-put depository	15.19	1.90
					2.1.2 Procedures for sharing	11.98	1.50
					2.1.3 Procedures for transportation	14.86	1.85
					2.1.4 Procedures for regular checkup	12.53	1.56
					2.1.5 Procedures for duplicates preservation	14.96	1.87
					2.1.6 Procedures for destruction	15.53	1.94
					2.1.7 Procedures for documentation	14.97	1.87
		2.2 Laboratory management	27.15	10.20	2.2.1 Lab certifications	14.00	1.70
					2.2.2 Procedures for risk assessment	15.18	1.84
					2.2.3 Procedures for sterilization	16.21	1.97
					2.2.4 Emergency response plan	16.43	1.99
					2.2.5 Procedures for biosecurity	13.47	1.64
					2.2.6 Operational requirements for experiments	15.23	1.85
					2.2.7 Procedures for procurement of experimental materials	9.48	1.15
		2.3 IT Management	21.53	8.09	2.3.1 Catalogue management	38.55	1.66
					2.3.2 Confidential management	33.43	1.44
					2.3.3 Database management	28.03	1.21
		2.4 Capacity building	21.65	8.13	2.4.1 Training for technical specialists	17.08	1.50
					2.4.2 Procedures for updating management documents	16.70	1.47
					2.4.3 Annual report mechanism	12.56	1.10
					2.4.4 Academic exchange and meetings	14.64	1.29
					2.4.5 Application of scientific research	12.02	1.06
					2.4.6 International cooperation	10.86	0.95
					2.4.7 QMS certification	16.12	1.42
3 Biobanking capacity	25.45	3.1 Site layouts	28.77	10.04	3.1.1 Overall functional zoning	19.96	1.98
					3.1.2 Receiving and distributing zone	21.09	2.09
					3.1.3 Experimental zone	20.82	2.07
					3.1.4 Preservation zone	20.19	2.00
					3.1.5 Office area	17.95	1.78
		3.2 Facility and equipment	25.51	8.90	3.2.1 Experimental equipment	13.66	1.97
					3.2.2 Preservation equipment	13.70	1.97
					3.2.3 Ventilation and gas supply equipment	13.12	1.89
					3.2.4 Power supply and lighting facility	13.12	1.89
					3.2.5 Monitoring and communication facility	11.62	1.67
					3.2.6 Identifier equipment	12.47	1.79
					3.2.7 Security facility	10.88	1.57
					3.2.8 Equipment for professional personnel	11.43	1.65
		3.3 Physical resource	25.13	8.77	3.3.1 Resource type	17.52	1.89
					3.3.2 Number of original isolations	12.78	1.38
					3.3.3 Number of preservations	13.98	1.51
					3.3.4 Number of type cultures	12.15	1.31
					3.3.5 Preservation methods	15.47	1.67
					3.3.6 Resource duplicates	14.10	1.52
					3.3.7 Regular checkup	14.00	1.51
		3.4 Data resource	20.59	7.19	3.4.1 Standardized description of data resource	34.76	1.58
					3.4.2 Operational management of database	34.28	1.56
					3.4.3 Service management of database	30.96	1.41
4 Sharing capacity	13.97	4.1 Sharing amount	47.14	4.90	4.1.1 Amount of supply	25.56	1.19
					4.1.2 Service objects	20.37	0.94
					4.1.3 Purposes of utilization	28.66	1.33
					4.1.4 Amount of receipt	25.41	1.18
					4.1.5 Source of receipt	22.81	1.06
		4.2 Sharing mechanism	52.86	5.50	4.2.1 Sharing agreement	36.23	1.14
					4.2.2 Sharing time limit	25.11	0.79
					4.2.3 Standardized sharing process	38.66	1.21

Note: Abbreviations: IT, information technology; QMS, quality management system.

4. Discussion

In recent years, with the global prevalence of infectious diseases caused by novel coronavirus, monkeypox virus, and others, the identification, preservation, and sharing of pathogenic resources have become particularly important in supporting the study of the genome evolution mechanism of mutant strains. With the rapid development of biorepositories, it is an important content to evaluate their operation and management.

This study, grounded in a literature review, takes references of biobank-related policies and guidance documents from both China and international sources. It selects a series of key indicators for the operation of biorepositories, aiming to initially establish a benchmarking framework for evaluating biorepositories for pathogenic resources, and to create a benchmarking tool for their comprehensive evaluation and improvement of operation. The fact that the response rate was 100 %, and the Cr > 0.70 in both rounds of surveys ensures the reliability of the results. Following the second-round survey, the P value of Kendall’s W for each indicator was less than 0.001, indicating statistical significance, which manifests that the experts’ opinions are coordinated, affirming the scientificity, stability, and feasibility of the benchmarking framework. In the final determined benchmarking framework, there are four primary indicators, 10 secondary indicators, and 65 tertiary indicators, which are conducive to guide comprehensive evaluation of all aspects of the operation of biorepositories for pathogenic resources, including organizational structure, management requirements, biobanking capacity, and sharing capacity. This benchmarking tool brings forward a series of specific requirements regarding the departments and personnel setup, funding source, quality management system, site functional layouts, facilities and equipment, physical resources and data resources, and sharing amount and mechanism. By assigning clear weight to each indicator, it facilitates quantitative scoring during evaluation, enhancing the comparability, either between years or across different institutions.

The World Health Organization (WHO) established the Hub for Pandemic and Epidemic Intelligence (WHO Pandemic Hub) on 1 September 2021, which aims to strengthen global surveillance of emerging public health threats and build a global collaborative intelligence ecosystem [25]. On 20 May 2023, The WHO Pandemic Hub launched the International Pathogen Surveillance Network (IPSN) with the goal of uniting global efforts to accelerate progress in improving systems for collecting and analyzing samples, using these data to drive public health decision-making, and sharing that information more broadly [26]. What is more, to better standardize the professional operation of biorepositories, a lot of international organizations and industry associations formulated guidelines and carried out investigations. The College of American Pathologists (CAP) developed the Biorepository Accreditation Program (BAP) in 2012, establishing 273 standards with customized requirement lists based on the scope of activities performed by a biorepository. From May 2012 to December 2016, a total of 90 inspections were completed and 527 deficiencies were identified in the areas of Equipment / Instrumentation, Information Technology, Specimen Handling and Quality Control (QC), Quality Management, Personnel, Safety, Facilities, and Regulatory [27]. The Biobanking and BioMolecular Resources Research Infrastructure-European Research Infrastructure Consortium (BBMRI-ERIC) established the Quality Working Group and launched two online self-assessment surveys (SAS) from fall 2017 to fall 2018. The SAS tools included nine standards / guidelines including International Standard Organization (ISO) 9001:2015, ISO 20387:2018 and ISBER Best Practices. Twelve biobanks in the BBMRI.be network (https://www.bbmri.be) participated to report on the status and setup of their current QMS and future development. These online proficiency testing initiatives improved the international accreditation for those biobanks attending them [28]. In China, a study was carried out to comprehensively investigate and analyze the operation of biorepositories for pathogenic resources. This study also put forward suggestions from the perspectives of operational capacity, biobanking capacity, and sharing capacity [29].

In recent years, audit plans have been established and promoted for the biobanking network to emphasize work principles and operational procedures. It is recommended to establish a national network in China for both physical and data resource sharing, to enhance enforcement of policies and rules, supervise performance, quality, certification, inspection, and standardization nationwide, and coordinate relevant funding, evaluation, and promotion of regulations [8], [30]. Research show that constant participation in proficiency testing will benefit biorepositories in promoting their overall operational performance [31], [32].

This benchmarking framework, established using of modified Delphi method, indeed reflects the relative importance of current laboratory and biobanking aspects and provides an explicit, detailed, and measurable tool for both domestic and international biorepositories for their systemic evaluation.

5. Conclusion

The establishment of the benchmarking framework in this study offers a measurement tool for more systematic and comprehensive evaluation of the management and operation of biorepositories for pathogenic resources in China, and provides references for comprehensive audit and proficiency testing for international counterparts. However, the results of modified Delphi method have certain limitations. For example, the consulting results may be limited by the subjective cognition of experts. In the two round surveys, the results showed relatively low scoring and coordination in indicators of biobanking ability and sharing ability, which to some extent implies that China still needs to continue to strengthen the practices of pathogenic resource sharing. Although this benchmarking tool has strong adaptability by adequately combining international experience and domestic laws and regulations in China, it may exist some discrepancies with the international practices during application.

In the next step, investigational questionnaires can be formulated according to the benchmarking tool, which might help to provide a quantitative assessment for biorepositories completing the investigation. The application of the benchmarking tool is conductive to find out the influencing factors that limit the sustainable development of China's biorepositories for pathogenic resources, and put forward specific suggestions for strengthening the construction of China's national network for biorepositories. In the meanwhile, ongoing improvement can be made to this benchmarking tool during its long-term application.