Thanks to WebAssembly, the Vocalizer Text-to-Speech engine can now run locally inside the user's browser. This avoids the need of a complex, expensive (and often slow) cloud-based solution for your web-based applications.

With our Vocalizer for WebApps SDK you will be able to create true multiplatform applications that can run on Windows, OSX, iPhone and Android with the same code base.

Click here to play with our our online demo.

What is WebAssembly

From The WebAssembly Official Site:

WebAssembly is the resulting work of members representing the four major browsers. Currently, Chrome, Edge, Firefox and WebKit-based browsers (i.e. Safari) fully support WebAssembly.

           

Some WebAssembly Highlights:

• Efficient and fast: WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms.


• Safe: WebAssembly describes a memory-safe, sandboxed execution environment. When embedded in the web, WebAssembly will enfore the same-origin and permissions security policies of the browser.


• Part of the open web platform: WebAssembly is designed to maintain the versionless, feature-tested, and backwards-compatible nature of the web. WebAssembly modules will be able to call into and out of the JavaScript context and access browser functionality through the same Web APIs accessible from JavaScript.

In addition to running inside a standard browser, you can also develop desktop and mobile applications using frameworks such as Electron or Ionic.

New oportunities

Nuance and Code Factory have once again partnered to bring the Vocalizer Embedded Text-to-Speech engine to this new technology which opens the door to a wide range of possibilities to our customers.

This SDK will provide you:

• A simple and intuitive JavaScript API to make your web application speech-enabled.
• Extensive Documentation and Examples (Web, Electron, Ionic and Node).
• Access to the best Text-to-Speech engine with more than 70 voices in over 45 languages.
• Optional integration support from Code Factory.

Browser and Framework Versions Required

• Chrome: 57
• Firefox: 52
• Safari: 11
• Safari Mobile: 11
• Microsoft Edge: 16
• Opera: 44
• Samsung Internet: 7.0
• Node.js: 8.0.0
• Electron: any with Chromium >= 57
• Ionic: 3

Vocalizer for WebApps SDK Version History

In addition to your application, your server must also be configured to supply the Text-to-Speech modules.

These modules include:

• A JavaScript file with the API that you will use to interact with the Text-to-Speech engine (webtts.js)
• An additional WASM file (webtts.wasm) that contains the Text-to-Speech engine in WebAssembly format.
• A file containing the metadata information of the data files (voices, languages, etc.) needed by the engine.
• The data files containing voice and language data.

Directory Structure

Here is an example directory structure of a typical web application that uses the Text-to-Speech engine:

Here is a description of each component:

• /index.html is the application's main page.
• /webtts.js contains all the functions that you will call to interact with the Text-to-Speech engine.
• /webtts.wasm contains additional components needed by the engine.
• /data/files.metadata contains information (url, size, etc.) of all the files that are needed by the Text-to-Speech engine. As far as the engine is concerned, any file that is not listed in the metadata does not exist.
• /data/languages/* contains the data files directly extracted from the voice ZIPs supplied together with the SDK.

You can choose any other directory structure that best suits your application needs. For example, you may want to store the engine JavaScript and WASM file in a separate directory, and store the metadata in another folder, or even server.

How the engine code is loaded into your Application

Like all the code of your application, the Text-to-Speech engine files also reside in your server. When your application is loaded by the browser, all the dependencies will be resolved, and the script files related to the Text-to-Speech engine will be automatically downloaded. (Note that this does not include the voice data files, which will be covered in a further section)

Typically, your application's main HTML page will load the Text-to-Speech API script like this:

The rest of the loading process happens transparently in the background. The WASM and other JavaScript files will be downloaded and processed by the browser when needed.

Once the HTML is completely loaded, the Text-to-Speech engine is ready to be used.

How data files are handled

The Text-to-Speech engine has two parts: code and data. As explained in the previous section, the engine code is loaded just like a normal JavaScript file by the browser, but the voice data still remains in the server.

The engine knows about what files are available to it from the information contained in the metadata file. The metadata provides a list of files that will be needed by the Text-to-Speech engine during synthesis.

For example, a simple metadata file might look like this:

For each file, the metadata provides information of its URL relative to the application, the file name, size and a hash that uniquely identifies the contents of the file.

In order to tell the engine where to find the metadata, its URL must be passed to the initialization function. Here is an example:

Optionally, you can pass the metadata string itself instead of a URL to the file. The documentation for the engine initialization provides more information about this.

Generating a metadata file

The SDK comes with a Python command-line utility to automatically build the metadata from a given directory. The basic usage is as follows:

The example above will recursively scan the ./data directory in search for .dat and .hdr files (the default wildcards). Each file will be processed and added to the metadata file named files.metadata

This utility also has some additional command-line options:

• --silent: silent execution. No output is shown.
• --include wildcard: specifies the wildcards. For example --include *embedded-pro*
• --baseurl urlstr: prepends the given URL string to the URL of each file in the metadata. For example: --baseurl http://dataserver.foo.com/voices

As a general rule, it's best that the data files reside in a directory relative to your application. If this is not possible, the --baseurl option will allow you to specify any URL for the data files. However, beware of Cross Origin Resource Sharing (CORS) restrictions.

See examples/pack_data.sh for a few examples on how to generate metadata files for various scenarios.

How Data Files are loaded

Through the metadata file, the engine is informed of the data files that are available. The engine can be configured in two possible ways, each handling data files differently:

• local: All the data files reside in the local computer (Node and Electron environment). The URL property of each file in the metadata list points to a directory in the local file system.


• remote: Data files reside in the server and will be downloaded during the engine's initialization process.

The operating mode can be specified during initialization as follows:

Caching Data Files

When the data files are downloaded from the server, they will automatically be cached in the browser's Indexed DB. This prevents further launches of your application from re-downloading the data files.

If you want to avoid this default behavior, you can initialize the TTS like this:

The user can, at any time, clear the data files if storage space is needed. Also, the Text-to-Speech API provides a function to programatically remove all data files that may be stored in the browser's Indexed DB.

The Vocalizer for WebApps SDK ships with a several example that demonstrate the functionality of the API in various of the supported environments.

Web Example

This is example is meant to run in a web browser and is the one that tries to cover the widest range of functions in the TTS API. It's also the one with the simplest UI, because we want this example to show only how to use the API, and remove any any code that is not strictly necessary.

This example shows the following functionality:

• Monitor progress of voice data download by using a progress bar.
• Control audio playback (stop, pause and resume)
• Cursor tracking by highlighting the word that is currently being spoken.
• Real-time audio signal display through an oscilloscope.
• Monitor engine state changes.
• Real-time lip synchronization through a minimalistic face.

The example is written in HTML and JavaScript and can be found in the folder examples/web.

Demo Example

This example is very similar to the Web Example, but it has a more elaborated UI, showing process in a circle bar. It also uses the Compact voices (as opposed to the Embedded Pro voices of the Web Example), so it's muct faster to load.

A real life implementation of this demo can be found in the Code Factory page

The example is written in HTML, JavaScript and CSS and can be found in the folder examples/demo.

Electron Example

This example targets the Electron environment, which allows you to create Windows and OSX desktop application.

Electron is a very interesting environment because it has the advantages of a Node.js environment (that is, it allows access to the local file system) and also the Web environment (because the UI runs inside Chromium).

Therefore, the TTS in an Electron application can be initialized in either local or remote modes depending on where you want the voice data to reside.

In this particular example, the Text-to-Speech voice data in bundled in the application's package itself (in a folder called data). Therefore, the metadata file looks something like this:

Note how the URL property of each file in the metadata list contains a directory name rather than a URL. To determine the folder from which each voice data file directory is resolved, the engine uses the parameter localroot specified during initialization.

For example:

dirname is a Node variable that points to the current module's folder inside the local file system.
For example, if dirname is /Users/Alex/Example/app and the entry in the metadata file is ../data/voicedata/languages/enu/speech/ve/ve_pipeline_enu_zoe_22_embedded-pro_2-2-1.hdr the file will be loaded from /Users/Alex/Example/app/../data/voicedata/languages/enu/speech/ve/ve_pipeline_enu_zoe_22_embedded-pro_2-2-1.hdr

The example can be found in the examples/electron folder.

Ionic Example

This example shows you how to integrate the Text-to-Speech engine into an Ionic application that will run in both iOS and Android mobile phones.

It's not very different from the Demo Example: the data is downloaded from the server and cached inside the application for later use, but the main difference is how the engine is initialized from the applcation:

In this case, we tell the engine where webtts.js and webtts.wasm are located through the scriptsRoot property.

This example can be found in the examples/ionic folder and most of the code can be found under examples/ionic/src/app/home

Node.js Example

This example shows how the TTS can be integrated into a Node application.

Node does not by default support audio streaming, so this example shows how to get the raw PCM audio from the TTS and create a wav file.

This example can be found in the eamples/node folder.

The WebAssembly Text-to-Speech SDK for Vocalizer Embedded is copyrighted and propietary software of Code Factory, SL

Nuance, Vocalizer and their respective logos are registrered trademarks of Nuance Communications, Inc. All rights reserved.

Chrome, Safari, Firefox, Microsoft Edge, Android, iPhone, OSX, iOS, WebAssembly, Ionic, Electron and WebGL are registered trademarks of their respective owners.

Contains code generated by Emscripten. Copyright © 2010-2014 Emscripten authors. Used under the MIT and the University of Illinois/NCSA Open Source Licenses. You can see the full license here.

Uses vc_vector, a fast and simple C vector implementation. Copyright © 2016 Skogorev Anton. Used under the MIT license.

Uses portions of code inspired by the Fetch Progress Indicators. Copyright © 2018 Anthum, Inc. Used under the MIT license. See full license here.

The Demo example uses and bundles the Code Mirror text editor. Copyright © 2017 by Marijn Haverbeke and others. Used under the MIT license. See full license here.

The Demo example uses and bundles the CSS Percentage Circle by Andre Firchow.

The Node.js example uses and bundles the Web Audio API library. Copyright © 2013 Sébastien Piquemal. Used under the MIT license. See full license here.

The Node.js example uses and bundles the node-speaker library. Copyright © their authors. Used under the MIT license.

The Electron and Ionic examples may contain dependency modules automatically generated by their respective build environments. Check the LICENSE.md file in the Electron and Ionic example folders respectively.

Code Factory, SL claims no ownership and provides no warranty of the third-party components and libraries listed above.

The information in this document is subject to change without notice and should not be construed as a commitment by Code Factory SL, who assumes no responsibility for any errors that may appear in this document. In no event shall Code Factory, SL be liable for incidental or consequential damages arising from use of this document or the software described in this document.

The use of the WebAssembly Text-to-Speech SDK for Vocalizer Embedded is not free and must always be integrated into an application after a proper legal contract has been signed with Code Factory. The contract will define the nature and scope of the application that integrates and uses the SDK.

Please visit our Legal Information page for more details about Code Factory and its Code of Conduct.

Taling 3D Head courtesy of Javier Arevalo Baeza (@TheJare). Developed in three.js and used under the MIT license.

Developed in the winter of 2019 by Eduard Sanchez Palazon.

Initializes the Text-to-Speech engine.

The initialization parameters are passed through an object with key/value property pairs. Here are the supported initialization parameters:

• env: Environment. Possible values are: web, node, electron and ionic.
• metadata: URL of the metadata file that contains information about all the data files available during the Text-to-Speech session. Optionally, you can pass the metadata string itself instead of a URL.
• data: Where the voice data files reside. Possible values are: local, and remote.
• cache: If the voice data is downloaded from the server, this parameter tells the engine where the files should be cached. Possible values are: browser (files will be cached in the browser's Indexed DB) and none (data files will not be cached).
• scriptsRoot: This parameter specifies the path to the webtts.js and webtts.wasm files. Note that the path can also be specified in the <script> tag of the HTML page.
• localroot: If the voice data files are located in the local file system (data = local) this parameter tells the engine the root folder that contains the data. The 'URL' parameter of each file in the metadata is relative to this folder.

During initialization, if a progress function has been provided, it will be called back and given information about the status of the initialization process. This is mostly used to provide information to the user about the download of the data files, as it's the most time-consuming task during initialization.

Note that the initialization can be canceled by a call to ttsRelease

initProgressFunc=function(msg) {
  if ( msg.task == "download" ) {
    if ( msg.name != null )
      document.getElementById('status').innerHTML='Downloading ' + msg.name;
    else document.getElementById('status').innerHTML='Downloading data...';

    // Update progress bar
    if ( msg.totalFilesProgress > 0 ) {
      document.getElementById('progress').value=msg.totalFilesProgress;
    }
  }
  else if ( msg.task == "save" ) {
    document.getElementById('status').innerHTML='Saving ' + msg.fileName + "...";
  }
  else if ( msg.task == "load" ) {
     document.getElementById('status').innerHTML='Loading ' + msg.fileName + "...";
  }
};

initCompleteFunc=function(success) {
  if ( success ) {
    // Show the engine version information if it's available
    var vi=ttsGetVersionInfo();

    if ( vi != null ) {
      document.getElementById('status').innerHTML="Engine version: "+vi.major+"."+vi.minor+" ("+vi.buildDay+"/"+vi.buildMonth+"/"+vi.buildYear+")<br>"+vi.sdkProvider;
    }

    // Set the current voice
    var voiceList=ttsGetVoiceList();

    if ( voiceList.length > 0 )
      ttsSetCurrentVoice(voiceList[0]);
  }
  else {
    document.getElementById('status').innerHTML='Error during initialization!';
  }
};

...

ttsInitialize({
    env: 'web',
    metadata: '/data/files.metadata',
    data: 'remote'
}, initProgressFunc)
.then((success) => initCompleteFunc(success));

Uninitializes the Text-to-Speech engine previously initialized by ttsInitialize

Speaks the given string. The function can optionally be passed callback functions that receive information about the progress of the synthesis and playback process.

If synthesis of a previous text is in progress, this function will add the text to the speech queue and so that it is processed later. If ttsStop is called, Promises will not be resolved for neither the current or any of the queued texts.

Compatibility Note

As a result, this won't work:

Instead, do something like this:

Speech Events

The speech callback functions are always synchronized real-time with the audio device, so that events always occur at the time when the user hears them.

The Promise returned by ttsSpeak will resolve when the audio device is done playing all audio belonging to the given text string. The Promise is fulfilled with a boolean value indicating the success of the operation. Failure to synthesize text is likely due to ttsSpeak being called with a wrong engine state (such as "uninitialized"). See ttsGetState for more information about engine states.

The progressFunc function will be called with data of events (such as bookmarks, words, lip information, etc.) that occur during the synthesis process. The function is called with an object as a parameter, with a property named "type" that contains the type of the event.

Here is a list of events that occur during synthesis:

• "word": Word event. Speech has reached a word in the input text.

  Properties of this event type:

  cntSrcPos: Character index of the beginning of the word.
  cntSrcTextLen: Length of the word in characters.


• "bookmark": Speech has reached a bookmark event. Bookmarks are embedded in the input text by control sequences.

  Properties of this event type:

  cntSrcPos: Character index of the beginning of the bookmark.
  cntSrcTextLen: Length of the bookmark in characters.
  name: The name of the bookmark.


• "lipsync": This even is fired to inform of the mouth position of the phoneme being synthesized.

  Properties of this event type:

  cntSrcPos: Character index of the beginning of the phoneme.
  cntSrcTextLen: Length of the phoneme in characters.
  cntSrcPos: Character index of the beginning of the phoneme.
  sJawOpen: Opening angle of the jaw on a 0 to 255 linear scale, where 0 = fully closed, and 255 = completely open.
  sTeethUpVisible: Indicates if upper teeth are visible on a 0 to 255 linear scale, where 0 = upper teeth are completely hidden, 128 = only the teeth are visible, and 255 = upper teeth and gums are completely exposed.
  sTeethLoVisible: Indicates if lower teeth are visible on a 0 to 255 linear scale, where 0 = lower teeth are completely hidden, 128 = only the teeth are visible, and 255 = lower teeth and gums are completely exposed.
  sMouthHeight: Mouth height on a 0 to 255 linear scale, where 0 = minimum height (mouth and lips are closed) and 255 = maximum possible height for the mouth.
  sMouthWidth: Mouth or lips width on a 0 to 255 linear scale, where 0 = minimum width (mouth and lips are puckered) and 255 = maximum possible width for the mouth.
  sMouthUpturn: Indicates how much the mouth is turned up at the corners on a 0 to 255 linear scale, where 0 = mouth corners turning down, 128 = neutral, and 255 = mouth is fully upturned.
  sTonguePos: Indicates the tongue position relative to the upper teeth on a 0 to 255 linear scale, where 0 = tongue is completely relaxed, and 255 = tongue is against the upper teeth.
  sLipTension: Lip tension on a 0 to 255 linear scale, where 0 = lips are completely relaxed, and 255 = lips are very tense.
  LHPhoneme: Matching L&H+ phonetic symbol.


• "audio": This event is fired to pass the audio samples currently being played.

  Properties of this event type:

  buffer: A Float32Array that contains the audio samples (-1.0 to 1.0 per sample).

Control Sequences

A control sequence is a piece of text that is not to be read out, but instead offers the possibility to intervene in the automatic pronunciation process. In this way the user can alter the way in which a text will be read, and acquire full control over the pronunciation of the input text. Control sequences can also be used to insert bookmarks in the text.

Control sequences are always preceded by an escape character (Hexadecimal 0x1B). You can directly embed control sequence in the input text like this:

Setting the language of the text

Example:

Note that it depends on the multilingual capabilities of the voice whether the voice can take the language of the text into account.

Setting the type of prosodic boundary

// Example of the ttsSpeak function. As text is being spoken, highlight the words in the text area, update the lips position and
// show the audio wave in an oscilloscope window.

speakProgressFunc=function(msg) {
  if ( msg['type'] == 'word' ) {
    var textArea=document.getElementById('input_id');

    // Highlight words as they are spoken.
    textArea.setSelectionRange(msg['cntSrcPos'], msg['cntSrcPos']+msg['cntSrcTextLen']);
  } else if ( msg['type'] == 'lipsync' ) {
    updateLips(msg);
  }
  else if ( msg['type'] == 'audio' ) {
    updateOsci(msg);
  }
};

var str=document.getElementById('input_id').value;
ttsSpeak(str, speakProgressFunc).then((success) => {
  if ( success )
    console.log("Text spoken successfuly");
});

Stops synthesis of the current text, clears the speech queue and resets the audio device.

Promises returned by ttsSpeak will not be fulfilled for neither the current or pending speech requests.

ttsStop().then((success) => {
   resetUi();
});

Returns the list of available voices.

For each voice, the following properties are returned:

• name: The name of the voice. For example 'Zoe'.
• language: The language of the voice. For example 'US English'.
• vop: The voice operating point (quality). For example: embedded-pro
• age: The age of the speaker.
• type: Voice type of the speaker (male, female, or neutral).

Sets the voice to be used for speech synthesis.

The voice must be one returned by the ttsGetVoiceList function.

Gets the voice currently used for synthesis.

Sets speech parmeters to be used during speech synthesis.

The parameters are set via an object of key/valus pairs. Possible key values are:

• speed: Speech rate level, which is a scale factor (in %) on the default speech rate of the current voice. The valid range is [50..400], with 50 having the voice speak 2x slower, and 400 having the voice speak 4x faster. Default value: 100 (%)


• pitch: Pitch level, a scale factor (in %) on the inherent pitch of the current voice. The range is [50..200]; with value 50 the voice speaks one octave lower (pitch :2), with value 200 the voice speaks one octave higher (pitch x2). Default value: 100 (%)


• volume: Volume level on a 0 to 100 scale. For each 10 points on the scale the volume changes by 3 dB. Default value: 80


• waitFactor: Wait period inserted between two text units (e.g. sentences), on a scale from 0 to 9. Each unit is equivalent to 200ms of silence. Default value: 1

ttsSetSpeechParams({
  speed: 120,
  pitch: 100,
});

Possible values are:

• "uninitialized": This is the default state. The engine is in an uninitialized state and can't synthesize any text. Call ttsInitialize to begin the initialization process.


• "initializing": The engine is currently being initialized.


• "initialized": The engine is initialized and ready to synthesize text. When text is done synthesizing, the engine returns to this state.


• "speaking": The engine is currently synthesizing text and streaming it through the audio device.


• "paused": The engine is currently paused in response to a call to ttsPause


• "uninitializing": The engine is performing the uninitialization process.

Adds a listener function that will be called whenever the state of the engine changes.

updateButtonState=function(state) {
  if ( state == "initialized" ) {
    // Enable the button to speak some text
    speakButton.disabled=false;
  }
}

...

ttsAddStateChangeListener(updateButtonState);

Removes a state change listener function previously registered by ttsAddStateChangeListener

ttsRemoveStateChangeListener(updateButtonState);

Pauses the synthesis process.

Operation failure is likely due to the engine not being in a "speaking" state (see ttsGetState)

Resumes the synthesis process previously paused by a call to ttsPause

Operation failure is likely due to the engine not being in a "paused" state (see ttsGetState)

Checks whether or not the engine is currently paused.

Checks whether or not the engine is currently speaking.

Checks whether or not the engine is initialized.

Gets the version information of the Text-to-Speech engine.

• ttsProvider: Name and/or copyright information of the Text-to-Speech provider. For example: Nuance Communications, Inc.


• sdkProvider: Name and/or copyright information of the WASM SDK provider. For example: Code Factory, S.L.


• major: Major version of the engine. For example: 3


• minor: Minor version of the engine. For example: 3


• maint: Maintenance (build) number of the engine. For example: 4


• buildInfoStr: String containing additional information of the build. For example: Vocalizer Embedded 3.3.4


• buildDay: Day of the month when the build was made. For example: 1


• buildMonth: Month when the build was made. For example: 6


• buildYear: Year when the build was made. For example: 2019

Sets the audio device volume.

Deletes all files and data that have been stored in the Indexed DB for later use.

This function effectively removes all cached data. Therefore, in the next Text-to-Speech session all the files will have to be downloaded from the server again.

Note that this function can only be called when the engine is in an "uninitialized" state (See ttsGetState)

Gets information about the amount of space currently being used by the cached voice data in the browser's Inexed DB.

The object returned contains the following properties:

• assetCount: The number of files currently stored.
• assetsSize: The total size (in bytes) of the stored data.

Gets a string representation of the last error that occured in the engine. You must call this function whenever an asynchronous Text-to-Speech function call is resolved with an error.