Evaluation results at the message level (Evaluation Metric 1) for structured records extracted using the fine-tuned LLaMA-2-7B model. For each record in the test set of USPTO-ORD-100K, an ORD-formatted JSON record is extracted from the unstructured text and evaluated against the ground truth using Evaluation Metric 1. The “Path” column denotes the root path of the corresponding messages in a reaction message. * These values were calculated using a more lenient routine detailed in the main text.
| Message type | Path | Accurate | Removal | Addition | Alteration | Total |
|---|---|---|---|---|---|---|
| Compound | Inputs | 38 470 (85.6%) | 2242 (5.0%) | 1015 (2.3%) | 4242 (9.4%) | 44 954 |
| 41 138* (91.5%) | 1574* (3.5%) | |||||
| ProductCompound | Outcomes | 7450 (71.3%) | 345 (3.3%) | 58 (0.6%) | 2656 (25.4%) | 10 451 |
| 9105* (87.1%) | 1001* (9.6%) | |||||
| ReactionConditions | Conditions | 9524 (95.7%) | N/A | N/A | 433 (4.4%) | 9957 |
| ReactionWorkup | Workups | 44 165 (90.7%) | 1713 (3.5%) | 1719 (3.5%) | 2807 (5.8%) | 48 685 |