Official websites use .gov
A
.gov website belongs to an official
government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you've safely
connected to the .gov website. Share sensitive
information only on official, secure websites.
Transformer models’ sizes and configurations.
| Model | Layers | Attention heads | Embedding dimension | Parameters (millions) | Pretraining corpus size (GB) |
| CamemBERT-base-CCNETa | 12 | 12 | 768 | 110 | 135 |
| FlauBERT-base-cased | 12 | 12 | 768 | 138 | 71 |
| BelGPT2 | 12 | 12 | 768 | 117 | 57.9 |
| GPTanam | 12 | 12 | 768 | 117 | 58.6 |
aCCNET: criss-cross attention for semantic segmentation.
