David Ifeoluwa Adelani

Ibrahim Ahmad

Nedjma OUSIDHOUM

Abinew Ayele

Saif Mohammad

Meriem Beloucif

2023-04-13

ArXiv (prépublication)

Varepsilon kú mask: Integrating Yorùbá cultural greetings into machine translation

Idris Akinade

Jesujoba Oluwadara Alabi

Clement Odoje

Dietrich Klakow

This paper investigates the performance of massively multilingual neural machine translation (NMT) systems in translating Yorùbá greetings… (voir plus) (kú mask), which are a big part of Yorùbá language and culture, into English. To evaluate these models, we present IkiniYorùbá, a Yorùbá-English translation dataset containing some Yorùbá greetings, and sample use cases. We analysed the performance of different multilingual NMT systems including Google and NLLB and show that these models struggle to accurately translate Yorùbá greetings into English. In addition, we trained a Yorùbá-English model by fine-tuning an existing NMT model on the training split of IkiniYorùbá and this achieved better performance when compared to the pre-trained multilingual NMT models, although they were trained on a large volume of data.

2023-03-31

ArXiv (prépublication)

ε KÚ <MASK>: INTEGRATING YORÙBÁ CULTURAL GREETINGS INTO MACHINE TRANSLATION

Idris Akinade

Jesujoba Oluwadara Alabi

Clement Oyeleke Odoje

Dietrich Klakow

This paper investigates the performance of massively multilingual neural machine translation (NMT) systems in translating Yorùbá greetings… (voir plus) (ε kú ), which are a big part of Yorùbá language and culture, into English. To evaluate these models, we present IkiniYorùbá, a Yorùbá-English translation dataset containing some Yorùbá greetings, and sample use cases. We analysed the performance of different multilingual NMT systems including Google and NLLB and show that these models struggle to accurately translate Yorùbá greetings into English. In addition, we trained a Yorùbá-English model by finetuning an existing NMT model on the training split of IkiniYorùbá and this achieved better performance when compared to the pre-trained multilingual NMT models, although they were trained on a large volume of data.

2023-03-03

ICLR.cc/2023/Workshop/AfricaNLP (publié)

AfriMTE and AfriCOMET: Empowering COMET to Embrace Under-resourced African Languages

Jiayi Wang

Sweta Agrawal

Ricardo Rei

Eleftheria Briakou

Marine Carpuat

Marek Masiak

Xuanli He

Sofia Bourhim

Andiswa Bukula

Muhidin A. Mohamed

Temitayo Olatoye

Hamam Mokayed

Christine Mwase

Wangui Kimotho

Foutse Yuehgoh

Aremu Anuoluwapo

Jessica Ojo

Shamsuddeen Hassan Muhammad

Salomey Osei … (voir 37 de plus)

Abdul-Hakeem Omotayo

Chiamaka Ijeoma Chukwuneke

Perez Ogayo

Oumaima Hourrane

Salma El Anigri

Lolwethu Ndolela

Thabiso Mangwana

Shafie Abdi Mohamed

Ayinde Hassan

Oluwabusayo Olufunke Awoyomi

Lama Alkhaled

sana Sabah al-azzawi

Naome Etori

Millicent Ochieng

Clemencia Siro

Samuel Njoroge

Eric Muchiri

Wangari Kimotho

Lyse Naomi Wamba

Daud Abolade

Simbiat Ajao

Tosin Adewumi

Iyanuoluwa Shode

Ricky Macharm

Ruqayya Nasir Iro

Saheed Salahudeen Abdullahi

Stephen Moore

Bernard Opoku

Zainab Akinjobi

Abeeb Afolabi

Nnaemeka Casmir Obiefuna

Onyekachi Ogbu

Sam Brian

Verrah Akinyi Otiende

CHINEDU EMMANUEL MBONU

Toadoum Sari Sakayo

Pontus Stenetorp

Despite the progress we have recorded in scaling multilingual machine translation (MT) models and evaluation data to several under-resourced… (voir plus) African languages, it is difficult to measure accurately the progress we have made on these languages because evaluation is often performed on n -gram matching metrics like BLEU that often have worse correlation with human judgments. Embedding-based metrics such as COMET correlate better; however, lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with a simplified MQM guideline for error-span annotation and direct assessment (DA) scoring for 13 typologi-cally diverse African languages. Furthermore, we develop A FRI COMET—a COMET evaluation metric for African languages by leveraging DA training data from high-resource languages and African-centric multilingual encoder (AfroXLM-Roberta) to create the state-of-the-art evaluation metric for African languages MT with respect to Spearman-rank correlation with human judgments ( +0 . 406 ).

2023-01-01

arXiv.org (prépublication)

AfriMTE and AfriCOMET: Empowering COMET to Embrace Under-resourced African Languages

Jiayi Wang

Sweta Agrawal

Ricardo Rei

Eleftheria Briakou

Marine Carpuat

Marek Masiak

Xuanli He

Sofia Bourhim

Andiswa Bukula

Muhidin A. Mohamed

Temitayo Olatoye

Hamam Mokayede

Christine Mwase

Wangui Kimotho

Foutse Yuehgoh

Anuoluwapo Aremu

Jessica Ojo

Shamsuddeen Hassan Muhammad

Salomey Osei … (voir 37 de plus)

Abdul-Hakeem Omotayo

Chiamaka Chukwuneke

Perez Ogayo

Oumaima Hourrane

Salma El Anigri

Lolwethu Ndolela

Thabiso Mangwana

Shafie Abdi Mohamed

Ayinde Hassan

Oluwabusayo Olufunke Awoyomi

Lama Alkhaled

sana Sabah al-azzawi

Naome A. Etori

Millicent A. Ochieng

Clemencia Siro

Samuel Njoroge

Eric Muchiri

Wangari Kimotho

Lyse Naomi Wamba Momo

Daud Abolade

Simbiat Ajao

Tosin P. Adewumi

Iyanuoluwa Shode

Ricky Macharm

Ruqayya Nasir Iro

Saheed Salahudeen Abdullahi

Stephen E. Moore

Bernard Opoku

Zainab Akinjobi

Abeeb Afolabi

Nnaemeka Casmir Obiefuna

Onyekachi Ogbu

Sam Brian

V. Otiende

CHINEDU EMMANUEL MBONU

Toadoum Sari Sakayo

Pontus Stenetorp

Despite the progress we have recorded in scaling multilingual machine translation (MT) models and evaluation data to several under-resourced… (voir plus) African languages, it is difficult to measure accurately the progress we have made on these languages because evaluation is often performed on n -gram matching metrics like BLEU that often have worse correlation with human judgments. Embedding-based metrics such as COMET correlate better; however, lack of evaluation data with human ratings for under-resourced languages, complexity of annotation guidelines like Multidimensional Quality Metrics (MQM), and limited language coverage of multilingual encoders have hampered their applicability to African languages. In this paper, we address these challenges by creating high-quality human evaluation data with a simplified MQM guideline for error-span annotation and direct assessment (DA) scoring for 13 typologi-cally diverse African languages. Furthermore, we develop A FRI COMET—a COMET evaluation metric for African languages by leveraging DA training data from high-resource languages and African-centric multilingual encoder (AfroXLM-Roberta) to create the state-of-the-art evaluation metric for African languages MT with respect to Spearman-rank correlation with human judgments ( +0 . 406 ).

2023-01-01

arXiv.org (preprint)

AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages

Shamsuddeen Hassan Muhammad

Idris Abdulmumin

Abinew Ayele

Nedjma OUSIDHOUM

Seid Muhie Yimam

Ibrahim Ahmad

Meriem Beloucif

Saif Mohammad

Sebastian Ruder

Oumaima Hourrane

Alipio Jorge

Pavel Brazdil

Felermino Ali

Davis David

Salomey Osei

Bello Shehu-Bello

Falalu Lawan

Tajuddeen Gwadabe

Samuel Rutunda … (voir 7 de plus)

Tadesse Belay

Wendimu Baye Messelle

Hailu Balcha

Sisay Adugna Chala

Hagos Gebremichael

Bernard Opoku

Stephen Arthur

2023-01-01

EMNLP (publié)

Findings of the 1st Shared Task on Multi-lingual Multi-task Information Retrieval at MRL 2023

Francesco Tinner

Chris Emezue

Mammad Hajili

Omer Goldman

Muhammad Farid Adilazuarda

Muhammad Dehan Al Kautsar

Aziza Mirsaidova

Müge Kural

Dylan Massey

Chiamaka Ijeoma Chukwuneke

CHINEDU EMMANUEL MBONU

Damilola Oluwaseun Oloyede

Kayode Olaleye

Jonathan Atala

Benjamin A. Ajibade

Saksham Bassi

Rahul Aralikatte

Najoung Kim

Duygu Ataman

Large language models (LLMs) excel in language understanding and generation, especially in English which has ample public benchmarks for var… (voir plus)ious natural language processing (NLP) tasks. Nevertheless, their reliability across different languages and domains remains uncertain. Our new shared task introduces a novel benchmark to assess the ability of multilingual LLMs to comprehend and produce language under sparse settings, particularly in scenarios with under-resourced languages, with an emphasis on the ability to capture logical, factual, or causal relationships within lengthy text contexts. The shared task consists of two sub-tasks crucial to information retrieval: Named Entity Recognition (NER) and Reading Comprehension (RC), in 7 data-scarce languages: Azerbaijani, Igbo, Indonesian, Swiss German, Turkish, Uzbek and Yorùbá, which previously lacked annotated resources in information retrieval tasks. Our evaluation of leading LLMs reveals that, despite their competitive performance, they still have notable weaknesses such as producing output in the non-target language or providing counterfactual information that cannot be inferred from the context. As more advanced models emerge, the benchmark will remain essential for supporting fairness and applicability in information retrieval systems.

2023-01-01

MRL (publié)

A FRI S ENTI : A B ENCHMARK T WITTER S ENTIMENT A NALYSIS D ATASET FOR A FRICAN L ANGUAGES

Shamsuddeen Hassan Muhammad

Idris Abdulmumin

Abinew Ayele

Nedjma OUSIDHOUM

Seid Muhie Yimam

Meriem Beloucif

Saif Mohammad

Sebastian Ruder

Oumaima Hourrane

Pavel Brazdil

Felermino D M A Ali

Davis David

Salomey Osei

Bello Shehu-Bello

Falalu Lawan

Tajuddeen Gwadabe

Samuel Rutunda

Tadesse Belay

Wendimu Baye Messelle … (voir 5 de plus)

Hailu Balcha

Sisay Adugna Chala

Hagos Gebremichael

Bernard Opoku

Stephen Arthur

Hausa, Igbo, Kinyarwanda, Moroccan Arabic, Mozambican Portuguese, Nigerian Pidgin, Oromo, Swahili, Tigrinya, Twi, Xitsonga, and Yoruba) from… (voir plus) four language families (Afro-Asiatic, English Creole, Indo European, and Niger-Congo). We describe the data collection methodology, annotation process, and related challenges when cu-rating each of the datasets. We also build different sentiment classification baseline models on the datasets and discuss their usefulness.

2023-01-01

(publié)

www.semanticscholar.org

MasakhaNEWS: News Topic Classification for African languages

Marek Masiak

Israel Abebe Azime

Jesujoba Oluwadara Alabi

Atnafu Lambebo Tonja

Christine Mwase

Odunayo Ogundepo

Bonaventure F. P. Dossou

Akintunde Oladipo

Doreen Nixdorf

Chris Emezue

sana Sabah al-azzawi

Blessing Kudzaishe Sibanda

Davis David

Lolwethu Ndolela

Jonathan Mukiibi

Tunde Oluwaseyi Ajayi

Tatiana Moteu Ngoli

Brian Odhiambo

Abraham Toluwase Owodunni … (voir 42 de plus)

Nnaemeka Casmir Obiefuna

Shamsuddeen Hassan Muhammad

Saheed Salahudeen Abdullahi

Mesay Gemeda Yigezu

Tajuddeen Gwadabe

Idris Abdulmumin

Mahlet Taye Bame

Oluwabusayo Olufunke Awoyomi

Iyanuoluwa Shode

Tolulope Anu Adelani

Habiba Abdulganiy Kailani

Abdul-Hakeem Omotayo

Adetola Adeeko

Afolabi Abeeb

Aremu Anuoluwapo

Olanrewaju Samuel

Clemencia Siro

Wangari Kimotho

Onyekachi Ogbu

CHINEDU EMMANUEL MBONU

Chiamaka Ijeoma Chukwuneke

Samuel Fanijo

Jessica Ojo

Oyinkansola Fiyinfoluwa Awosan

Tadesse Kebede Guge

Toadoum Sari Sakayo

Pamela Nyatsine

Freedmore Sidume

Oreen Yousuf

Mardiyyah Oduwole

USSEN ABRE KIMANUKA

Kanda Patrick Tshinu

Thina Diko

Siyanda Nxakama

Abdulmejid Tuni Johar

Sinodos Gebre

Muhidin A. Mohamed

Shafie Abdi Mohamed

Fuad Mire Hassan

Moges Ahmed Mehamed

Evrard Ngabire

Pontus Stenetorp

African languages are severely under-represented in NLP research due to lack of datasets covering several NLP tasks. While there are individ… (voir plus)ual language specific datasets that are being expanded to different tasks, only a handful of NLP tasks (e.g. named entity recognition and machine translation) have standardized benchmark datasets covering several geographical and typologically-diverse African languages. In this paper, we develop MasakhaNEWS -- a new benchmark dataset for news topic classification covering 16 languages widely spoken in Africa. We provide an evaluation of baseline models by training classical machine learning models and fine-tuning several language models. Furthermore, we explore several alternatives to full fine-tuning of language models that are better suited for zero-shot and few-shot learning such as cross-lingual parameter-efficient fine-tuning (like MAD-X), pattern exploiting training (PET), prompting language models (like ChatGPT), and prompt-free sentence transformer fine-tuning (SetFit and Cohere Embedding API). Our evaluation in zero-shot setting shows the potential of prompting ChatGPT for news topic classification in low-resource African languages, achieving an average performance of 70 F1 points without leveraging additional supervision like MAD-X. In few-shot setting, we show that with as little as 10 examples per label, we achieved more than 90\% (i.e. 86.0 F1 points) of the performance of full supervised training (92.6 F1 points) leveraging the PET approach.

2023-01-01

AfricaNLP (publié)

MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African languages

Cheikh M. Bamba Dione

Peter Nabende

Jesujoba Oluwadara Alabi

Thapelo Sindane

Happy Buzaaba

Shamsuddeen Hassan Muhammad

Chris Emezue

Perez Ogayo

Aremu Anuoluwapo

Catherine Gitau

Derguene Mbaye

Jonathan Mukiibi

Blessing Kudzaishe Sibanda

Bonaventure F. P. Dossou

Andiswa Bukula

Rooweither Mabuya

Allahsera Auguste Tapo

Edwin Munkoh-Buabeng

Victoire Memdjokam Koagne … (voir 24 de plus)

Fatoumata Ouoba Kabore

Amelia Taylor

Godson Kalipe

Tebogo Macucwa

Vukosi Marivate

Tajuddeen Gwadabe

Mboning Tchiaze Elvis

Ikechukwu Onyenwe

Gratien Atindogbe

Tolulope Anu Adelani

Idris Akinade

Olanrewaju Samuel

Marien Nahimana

Théogène Musabeyezu

Emile Niyomutabazi

Ester Chimhenga

Kudzai Gotosa

Patrick Mizha

Apelete Agbolo

Seydou Traore

Chinedu Uchechukwu

Aliyu Yusuf

Muhammad Abdullahi

Dietrich Klakow

In this paper, we present AfricaPOS, the largest part-of-speech (POS) dataset for 20 typologically diverse African languages. We discuss the… (voir plus) challenges in annotating POS for these languages using the universal dependencies (UD) guidelines. We conducted extensive POS baseline experiments using both conditional random field and several multilingual pre-trained language models. We applied various cross-lingual transfer models trained with data available in the UD. Evaluating on the AfricaPOS dataset, we show that choosing the best transfer language(s) in both single-source and multi-source setups greatly improves the POS tagging performance of the target languages, in particular when combined with parameter-fine-tuning methods. Crucially, transferring knowledge from a language that matches the language family and morphosyntactic properties seems to be more effective for POS tagging in unseen languages.

2023-01-01

ACL (1) (publié)

MphayaNER: Named Entity Recognition for Tshivenda

Rendani Mbuvha

Tendani Mutavhatsindi

Tshimangadzo Rakhuhu

Aluwani Mauda

Tshifhiwa Joshua Maumela

Andisani Masindi

Seani Rananga

Vukosi Marivate

Tshilidzi Marwala

Named Entity Recognition (NER) plays a vital role in various Natural Language Processing tasks such as information retrieval, text classific… (voir plus)ation, and question answering. However, NER can be challenging, especially in low-resource languages with limited annotated datasets and tools. This paper adds to the effort of addressing these challenges by introducing MphayaNER, the first Tshivenda NER corpus in the news domain. We establish NER baselines by fine-tuning state-of-the-art models on MphayaNER. The study also explores zero-shot transfer between Tshivenda and other related Bantu languages, with Setswana, chiShona and Kiswahili showing the best results. Augmenting MphayaNER with Setwana data was also found to improve model performance significantly. Both MphayaNER and the baseline models are made publicly available.

2023-01-01

AfricaNLP (publié)

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

Zheng Xin Yong

Hailey Schoelkopf

Niklas Muennighoff

Alham Fikri Aji

Khalid Almubarak

M. Saiful Bari

Lintang A. Sutawika

Jungo Kasai

Ahmed Baruwa

Genta Indra Winata

Stella Biderman

Dragomir R. Radev

Vassilina Nikoulina

The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the be… (voir plus)nefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling.

2022-12-19

ArXiv (prépublication)