Nearly half of the world’s roughly 7,000 identified languages 4 in ten of them exist with out an accompanying written element. These unwritten languages pose a singular downside for contemporary machine studying translation methods, as they usually must convert verbal speech to written phrases earlier than translating to the brand new language and reverting the textual content again to speech, however one which Meta has reportedly addressed with its newest open-source language AI development.

As a part of Meta’s Universal Speech Translator (UST) program which is working to develop real-time speech-to-speech translation in order that can extra simply work together (learn: ). As a part of this mission, Meta researchers checked out Hokkien, an unwritten language spoken all through Asia’s diaspora and considered one of Taiwan’s official languages.

Machine studying translation methods usually require intensive labelable examples of the language, each written and spoken, to coach on — exactly what unwritten languages like Hokkien don’t have. To get round that, “we used speech-to-unit translation (S2UT) to convert input speech to a sequence of acoustic units directly in the path previously pioneered by Meta,” CEO Mark Zuckerberg defined in a Wednesday weblog submit. “Then, we generated waveforms from the units. In addition, UnitY was adopted for a two-pass decoding mechanism where the first-pass decoder generates text in a related language (Mandarin), and the second-pass decoder creates units.”

“We leveraged Mandarin as an intermediate language to build pseudo-labels, where we first translated English (or Hokkien) speech to Mandarin text, and we then translated to Hokkien (or English) and added it to training data,” he continued. Currently, the system permits for somebody who speaks Hokkien to converse with somebody who speaks English, albeit stiltedly. The mannequin can solely translate one full sentence at a time however Zuckerberg is assured that the method can ultimately be utilized to extra languages and can enhance to the purpose of providing real-time translation.

In addition to the fashions and coaching knowledge that Meta is already open-sourcing from this mission, the corporate can be releasing a first-of-its-kind speech-to-speech translation benchmarking system based mostly on a Hokkien speech corpus referred to as Taiwanese Across Taiwan, in addition to “the speech matrix, a large corpus of speech-to-speech translations mined with Meta’s innovative data mining technique called LASER,” Zuckerberg introduced. This system will empower researchers to create speech-to-speech translation (S2ST) methods of their very own.