IsROBERTa
The First Icelandic Large Transformer Language Model
Technology
PythonPytorchONNXWandBPoetryNumPyPandasFlaskTokenizerLLMDockerOpen APICloud Technology
AzureAzure FunctionsAzure storageGitHubGitHub actionsIndustry
Open SourceOverview
In the age of artificial intelligence, we proudly introduce "IsROBERTa" — our ground-breaking achievement as the first Icelandic large transformer language model. Developed in 2020, IsROBERTa was designed to understand and generate Icelandic text, pushing the boundaries of language processing in AI for Icelandic.
Challenges
Creating a large transformer language model, particularly for a language as unique and complex as Icelandic, is no small feat. The field of training large language models in 2020 was still in its infancy. Some challenges we faced were:
- Data Scarcity
- Language Complexity
- Computational Resources
- Lack of established libraries
About the model
IsROBERTa was trained on the OSCAR corpus using a Masked Language Model (MLM) downstream task, boasting the following hyperparameters:
- Per-device training batch size: 48
- Number of epochs: 1
- Vocabulary size: 52,000
- Maximum position embeddings: 514
- Number of attention heads: 12
- Number of hidden layers: 6
- Type vocabulary size: 1
- Learning rate: 0.00005
For detailed insights into the model, visit our GitHub repository.
Your model
We believe that the journey towards breakthrough AI solutions is best embarked on together. When you choose to work with us, you're not just selecting a service provider, you're choosing a partner dedicated to bringing your AI vision to life.