TY - GEN

T1 - A Warm Start and a Clean Crawled Corpus - A Recipe for Good Language Models

AU - Snæbjarnarson, Vésteinn

AU - Símonarson, Haukur Barri

AU - Ragnarsson, Pétur Orri

AU - Ingólfsdóttir, Svanhvít Lilja

AU - Jónsson, Haukur Páll

AU - Porsteinsson, Vilhjálmur

AU - Einarsson, Hafsteinn

N1 - Funding Information: We thank Prof. Dr.-Ing. Morris Riedel and his team for providing access to the DEEP super-computer at Forschungszentrum Jülich. We also thank the Icelandic Language Technology Program (Nikulásdóttir et al., 2020). It has enabled the authors to focus on work in Icelandic NLP. Finally, we thank the anonymous reviewers for their helpful feedback. Publisher Copyright: © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0.

PY - 2022

Y1 - 2022

N2 - We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error detection and constituency parsing. To train the models we introduce a new corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection of high quality texts found online by targeting the Icelandic top-level-domain.is. Several other public data sources are also collected for a total of 16GB of Icelandic text. To enhance the evaluation of model performance and to raise the bar in baselines for Icelandic, we manually translate and adapt the WinoGrande commonsense reasoning dataset. Through these efforts we demonstrate that a properly cleaned crawled corpus is sufficient to achieve state-of-the-art results in NLP applications for low to medium resource languages, by comparison with models trained on a curated corpus. We further show that initializing models using existing multilingual models can lead to state-of-the-art results for some downstream tasks.

AB - We train several language models for Icelandic, including IceBERT, that achieve state-of-the-art performance in a variety of downstream tasks, including part-of-speech tagging, named entity recognition, grammatical error detection and constituency parsing. To train the models we introduce a new corpus of Icelandic text, the Icelandic Common Crawl Corpus (IC3), a collection of high quality texts found online by targeting the Icelandic top-level-domain.is. Several other public data sources are also collected for a total of 16GB of Icelandic text. To enhance the evaluation of model performance and to raise the bar in baselines for Icelandic, we manually translate and adapt the WinoGrande commonsense reasoning dataset. Through these efforts we demonstrate that a properly cleaned crawled corpus is sufficient to achieve state-of-the-art results in NLP applications for low to medium resource languages, by comparison with models trained on a curated corpus. We further show that initializing models using existing multilingual models can lead to state-of-the-art results for some downstream tasks.

KW - co-reference resolution

KW - corpus

KW - IceBERT

KW - Icelandic

KW - language model

KW - named entity recognition

KW - natural language understanding

KW - parsing

KW - part of speech

UR - http://www.scopus.com/inward/record.url?scp=85137484516&partnerID=8YFLogxK

M3 - Article in proceedings

AN - SCOPUS:85137484516

SP - 4356

EP - 4366

BT - 2022 Language Resources and Evaluation Conference, LREC 2022

A2 - Calzolari, Nicoletta

A2 - Bechet, Frederic

A2 - Blache, Philippe

A2 - Choukri, Khalid

A2 - Cieri, Christopher

A2 - Declerck, Thierry

A2 - Goggi, Sara

A2 - Isahara, Hitoshi

A2 - Maegaard, Bente

A2 - Mariani, Joseph

A2 - Mazo, Helene

A2 - Odijk, Jan

A2 - Piperidis, Stelios

PB - European Language Resources Association (ELRA)

T2 - 13th International Conference on Language Resources and Evaluation Conference, LREC 2022

Y2 - 20 June 2022 through 25 June 2022

ER -