Table 3:
Results on Long Range Arena (LRA) benchmark using our PyTorch implementation. We report classification accuracy for each individual task and average accuracy across all tasks. Our Nyströmformer performs competitively with standard self-attention, and significantly outperforms Reformer, Linformer, and Performer.
| Model | ListOps (2K) | Text (4K) | Retrieval (4K) | Image (1K) | Pathfinder (1K) | Avg |
|---|---|---|---|---|---|---|
| Standard | 37.10 | 65.02 | 79.35 | 38.20 | 74.16 | 58.77 |
| Reformer | 19.05 | 64.88 | 78.64 | 43.29 | 69.36 | 55.04 |
| Linformer | 37.25 | 55.91 | 79.37 | 37.84 | 67.60 | 55.59 |
| Performer | 18.80 | 63.81 | 78.62 | 37.07 | 69.87 | 53.63 |
| Nyströmformer (ours) | 37.15 | 65.52 | 79.56 | 41.58 | 70.94 | 58.95 |