Vanuatu local languages · corpus in collection
Vanuatu local languages · ≈340k across 138 languages speakers · fine-tuning on a community-curated corpus.
The same thought, double the cost.
Bislama (creole) shown here as a sample; individual Vanuatu languages will each have their own tokenizer twin.
The world's densest language geography
Vanuatu has more languages per capita than any country on Earth — 138 languages for roughly 340,000 speakers. Some are documented; many are not. Some communities want their languages modeled; others do not.
This page is a cluster hub. It says so plainly. The base model itself is dashed on this page: under evaluation, pre-selection.
Pre-selection is not indecision
Choosing which languages to work on is the first act of the project, and it belongs to local partners — not to us.
Provisional honesty is the design.
We work with open source models. Gemma-4 (and SEA-LION )is being used for Burmese, Karen, and Jinghpaw as well as; Llama-3.1-8B may be used for Rohingya, for byte-level fallback where Hanifi is unrepresented.
Community participation and consent for curation and capture. Provenance recorded and fair compensation provided.
Converting legacy encodings and non-standard scripts into universal digital formats (e.g., Zawgyi to Unicode) is an act of cultural and historic preservation. Working alongside local linguists and community members, we systematically standardize text to be machine-readable while ensuring cultural context, orthographic nuances, and historical integrity remain intact.
Adapting foundational base models to local linguistic realities through targeted, data-efficient techniques calibrated to each language’s structure and context. Training logs and loss curves are published directly to the public archive with complete technical transparency.
Community-annotated benchmarks. Evals and benchmarks designed with local community and experts. Annotators acknowledgements and compensation. Inter-annotator agreement reported alongside every score.
- Q01Which languages within the cluster to serve first, and by whose invitation.
- Q02How to model without extracting — the whole question.
The corpus belongs to the communities that made it. Consent is per item and revocable. Revocation is honored in the weights, not just in the archive.
Model weights reflect the terms and governance frameworks chosen with the community.