Voice Subtitles

Everyone at the same table

A roleplay server is a story told out loud, and somebody who does not speak the language it is told in is standing outside it. This release translates what they say and what they write, in both directions between Russian and English, on your own machine and without a word of it going anywhere.

Mark whoever speaks something else and the mod does the rest:

/vsub lang Gizmons_ en

From then on that player's voice goes into an English model rather than a Russian one, and what it produces is drawn translated under the original. Everybody else carries on through the model they always did. Text chat, /me, /do, /try and the rest are translated in place, under the line they belong to.

Four settings, and the difference between them is how much is loaded and when:

Off	nothing. Everybody's voice goes into your own model, which is what a Russian model does to English speech: writes it out in Cyrillic, phonetically, and it reads as nonsense
Recognise only	a second recognition model for the people you marked, so at least their words come out in their own alphabet. Nothing is translated and no translation model is downloaded
Automatic	recognise and translate, loading both the moment one of those people is heard and letting go of them once they have been quiet for five minutes
Always	the same, loaded from the moment you join, for somebody who hears another language all evening

The original stays under the translation, so a name a translator mangled can still be read as it was said. Names, and anything the server put in brackets, are held back from the translator entirely and put back afterwards: a translated character name is a different character.

It runs on OPUS-MT by the University of Helsinki, quantised to eight bits, on the copy of ONNX Runtime that sherpa-onnx has already put on disk. A hundred and thirteen megabytes per direction, a few hundred milliseconds a sentence, no network at any point.

Two downloads now

voice-subtitles-1.2.1.jar is the ordinary one: 3 MB, and it fetches the engine and the model the first time it needs them. Take this one.

voice-subtitles-1.2.1-full.jar is 177 MB and is the same mod with the recognition engine and the Russian GigaAM model already inside it. On the first run it copies them onto disk in about a second and never asks the network for anything.

It exists because for some people the download is the problem rather than the size. GitHub and Hugging Face are reached slowly from some places and not at all from others, and an evening spent watching a progress bar crawl is an evening not spent playing. The engine in it is built for Windows x64; on anything else the model is still free and the engine is a 7 to 30 MB download.

A crash that took the game down without a word

Two players lost their game to it, both within a third of a second of the recognition library loading. Handed a file it cannot open, sherpa-onnx throws a C++ exception, and a C++ exception crossing back into Java kills the process outright: no crash report, no message, nothing but a line saying it died in native code.

The file could not be opened because something else had it. Antivirus software is the usual reason: two hundred megabytes that have just appeared on disk are exactly what it wants to look at, and while it is looking, everything else is turned away.

So the mod now reads the whole model file itself, in Java, before the library is given the path, and waits and tries again if it cannot. Either the read succeeds, in which case the scanner has finished and the library will be let in too, or it fails here, where a failure is a caption that never appears rather than a game that dies.

Nobody is asked to make an exception for this mod in their antivirus. Being told to switch off the thing protecting your machine is what malware asks for, and a mod that needs it asked for is a mod with a bug in it.

Alongside that: a model that is only half downloaded is now recognised as half downloaded rather than loaded and trusted, whatever the game was opening when it died is set aside instead of opened again, and /vsub model repair throws that away and fetches it afresh.

Captions

• Up to 32 speakers and 32 lines at once, for a crowded room.
• The feed can sit on the right as well as centred or on the left.
• The letters really do appear one at a time now, all the way through a sentence rather than for the first two words, and the default fade is gentler.
• Recognition holds five speakers at once by default rather than three.

Fixed

• Translations of chat and of roleplay commands were missing from the conversation log, and from the chat itself. Exports carry them now too.
• Picking a language is a list rather than a text field.
• The settings screen could refuse to open. It was the new alignment setting having no value to be the default of.