Skip to main content
[Preprint]. 2025 Aug 25:rs.3.rs-7216581. [Version 1] doi: 10.21203/rs.3.rs-7216581/v1

Table 2.

Evaluation of dynamic prompting strategies (5-shot, 10-shot, and 20-shot) using GPT-4 and Llama 3 across five biomedical datasets. The table presents F1-score, precision, and recall for each retrieval method: Base Prompt, TF-IDF, SBERT, ColBERT, and DPR. The row “Base” represents using static prompts we proposed in the former section.

Reddit_Impacts BC5CDR MIMIC III NCBI Med-Mentions
P R F1 P R F1 P R F1 P R F1 P R F1
GPT-4
5-shot Base 18.87 52.01 27.60 68.62 90.32 78.03 63.06 64.12 63.58 45.02 49.02 46.93 27.26 60.06 37.49
TF-IDF 19.71 51.25 28.47 82.31 89.76 85.88 74.43 78.14 76.24 56.86 63.68 60.08 27.22 62.68 37.96
SBERT 24.31 55.00 33.72 76.63 91.41 83.37 72.63 74.27 73.44 55.05 60.30 57.56 28.05 64.65 39.12
ColBERT 22.66 56.79 32.39 78.64 81.03 79.82 74.14 77.02 75.56 50.43 54.48 52.38 28.14 68.69 39.93
DPR 22.60 58.79 32.64 79.39 88.24 83.58 69.77 70.00 69.89 46.67 52.39 49.37 27.90 65.49 39.13
10-shot Base 22.25 56.66 31.92 75.33 88.31 81.27 66.38 74.24 70.09 53.23 52.13 52.67 26.67 59.20 36.74
TF-IDF 21.53 56.25 31.14 83.81 89.67 86.64 73.85 77.29 75.53 58.81 65.66 62.05 28.14 71.42 40.37
SBERT 25.41 58.75 35.47 83.94 87.99 85.92 72.73 75.08 73.89 58.79 63.02 60.83 28.32 70.26 40.37
ColBERT 23.86 58.02 33.81 83.49 88.05 85.71 74.69 78.06 76.34 55.12 59.56 57.25 28.15 71.99 40.48
DPR 22.96 56.25 32.61 85.16 84.42 84.79 71.84 72.42 72.13 56.82 60.72 58.70 28.25 70.04 40.25
20-shot Base 27.74 58.75 37.67 74.57 89.18 81.15 70.65 71.32 70.98 51.68 52.29 51.98 28.10 60.78 38.39
TF-IDF 27.72 62.20 38.35 85.41 88.98 87.16 75.81 79.61 77.66 61.80 67.13 64.36 28.20 77.30 41.32
SBERT 28.44 59.50 38.22 85.37 89.57 87.42 73.79 76.54 75.14 60.89 63.59 62.21 26.81 74.09 39.37
ColBERT 31.19 66.67 42.49 82.09 83.94 83.00 75.27 78.19 76.70 56.13 59.35 57.69 27.70 75.47 40.53
DPR 28.55 60.75 38.84 85.81 85.40 85.60 71.82 72.74 72.28 59.00 61.74 60.34 27.16 69.37 39.23
Llama3-70B
5-shot Base 13.16 57.86 21.43 68.97 78.36 73.32 59.30 67.27 62.94 35.81 34.71 34.80 25.89 67.05 37.26
TF-IDF 18.89 58.62 28.57 78.49 81.78 80.11 66.48 74.84 70.41 48.93 50.70 49.80 26.46 72.06 38.68
SBERT 23.20 66.67 34.42 77.26 83.79 80.39 64.04 72.21 67.88 50.66 49.59 50.12 26.15 68.92 37.91
ColBERT 22.05 65.12 32.94 71.21 72.33 71.76 68.37 75.32 71.68 44.93 46.08 45.50 26.68 72.38 38.99
DPR 19.20 59.26 29.00 74.47 76.91 75.67 65.74 72.54 68.97 41.06 48.66 44.54 26.51 71.38 38.66
10-shot Base 22.37 59.94 32.50 72.56 77.91 75.15 59.13 71.63 63.77 39.67 31.49 35.60 25.57 64.33 36.50
TF-IDF 23.53 62.65 34.21 80.82 80.32 80.57 55.79 55.34 55.56 49.59 49.41 49.50 24.03 68.00 35.51
SBERT 22.27 59.76 32.45 77.72 84.94 81.17 67.67 76.09 71.63 52.84 49.94 51.35 27.61 66.88 39.08
ColBERT 22.58 60.50 32.89 78.40 82.37 80.34 69.65 76.37 72.85 38.72 38.81 38.77 26.49 67.58 38.06
DPR 24.37 57.83 34.29 85.16 84.42 84.79 65.85 73.68 69.54 47.60 45.04 46.28 25.80 70.97 37.85
20-shot Base 24.52 53.81 33.67 75.42 75.58 75.50 62.01 62.12 62.05 40.71 42.58 41.62 26.57 64.79 37.67
TF-IDF 27.62 66.95 39.11 74.64 82.47 78.36 55.95 58.51 57.66 45.39 49.83 47.50 27.80 64.39 38.83
SBERT 29.93 68.06 41.43 75.04 80.75 76.85 65.90 64.77 65.35 42.09 46.40 44.14 25.48 61.36 36.01
ColBERT 23.57 65.48 34.66 73.74 70.70 72.19 58.25 57.03 57.63 47.08 49.88 48.44 25.41 66.98 36.85
DPR 26.15 65.04 37.30 72.58 77.15 74.80 62.72 69.19 65.80 37.18 44.13 40.36 26.10 62.88 36.89