Talking to your browser

(doesn't make you crazy if your browser can respond)

Chen Hui Jing / @hj_chen

🌈 </ RK> 🌈

🇲🇾 👾 🏀 🚲 🖌️ 👟 💻 🖊️ 🎙 🦊 🥑 🧗 🏳️‍🌈

@hj_chen

Must have some backstory…

🇸🇬 SingaporeCSS 🇸🇬

What is CSS? 🤔

Nah… I'm just kidding. I'm going make a bold assumption that even if you don't write CSS, you at least heard of it before and know what it does.

Hopefully.

If you also like colours…

*insert random caption here*

Where did CSS named colours come from?

🏴󠁧󠁢󠁥󠁮󠁧󠁿 English! 🏴󠁧󠁢󠁥󠁮󠁧󠁿

Your browser can speak English

For the most part.

Cue bo liao idea…

Singlish lesson #1: Bo liao

Hokkien for “nothing better to do”. Dangerously idle.
In Mandarin, it's “无聊 (wú liáo) ”
“What for he go and do that sort of thing? Must be damn bo liao.”

Source: The Coxford Singlish Dictionary

Change background colour with your voice!

Ah-maz-ing

Web Speech API

Speech Recognition

provides the ability to recognise voice context from an audio input and respond appropriately

SpeechRecognition
SpeechGrammar

Speech Synthesis

a text-to-speech component that allows programs to read out their text content

SpeechSynthesis
SpeechSynthesisVoice
SpeechSynthesisUtterance

Make your browser listen to you 👂

Bo liao also must plan

Have web page
Set background colour with CSS custom property
Have button to trigger microphone
Capture voice and process with SpeechRecognition
Use result to update background colour

Have web page

<!doctype html>
<html class="no-js" lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">

    <title>Let's talk CSS colours</title>
    <meta name="description" content="Playing around with the WebSpeech API, CSS custom properties and CSS named colours">
    <meta name="author" content="Chen Hui Jing">

    <link rel="stylesheet" href="styles.css">

    <!--[if lt IE 9]>
    <script src="http://html5shiv.googlecode.com/svn/trunk/html5.js"></script>
    <![endif]-->
  </head>

  <body>
    <main></main>
    <script src="scripts.js"></script>
  </body>
</html>

Set background colour with CSS custom property

:root {
  --bg-colour: transparent;
}

main {
  /* Of course got other styles la… */
  /* You think magic meh… */
  background-color: var(--bg-colour);
}

/* Moar styles not shown here */

What is a CSS custom property? 🤔

Defined in CSS Custom Properties for Cascading Variables Module Level 1

Introduces cascading variables as a new primitive value type that is accepted by all CSS properties, and custom properties for defining them

var( <custom-property-name> , <declaration-value>? )

Have button to trigger microphone

<body>
  <main>
    <h1>CSS Colours</h1>
    <p>How well do you know CSS named colours? Test both your knowledge as well as your browser's ability to recognise your accent when you speak English <span class="kaomoji">¯\_(ツ)_/¯</span></p>

    <button type="button" id="activateMic" class="btn-speak">Speak</button>

    <pre><code id="consoleLog">Click the button then say a colour…</code></pre>
  </main>

  <script src="scripts.js"></script>
</body>

Capture voice and process with `SpeechRecognition`

But first...

Feature detection.

Browser support for SpeechRecognition 😩

((window, undefined) => {
  const document = window.document;
  const docElement = document.documentElement;

  const speechRecognition = window.webkitSpeechRecognition || window.mozSpeechRecognition || window.msSpeechRecognition || window.oSpeechRecognition || window.SpeechRecognition;
  const speechGrammarList =  window.webkitSpeechGrammarList || window.mozSpeechGrammarList || window.msSpeechGrammarList || window.oSpeechGrammarList || window.SpeechGrammarList;

  function addClass(className) {
    docElement.className = `${docElement.className} ${className}`;
  }

  docElement.className = docElement.className.replace(/(^|\s)no-js(\s|$)/, '$1js$2');
  if (speechRecognition !== undefined) {
    addClass('speech');
    
  } else {
    addClass('no-speech');
  }
})(window);

Feature detection by Cătălin Mariș

To activate in Firefox 🦊

Must be newer than 72.0a1 (2019-10-22)

type about:config in your address bar
search for the media.webspeech.recognition.enable and media.webspeech.recognition.force_enable preferences
set them to true

How come browser can speak English?

Who's doing the processing?

Google.

Google Cloud Speech-to-Text, with speech recognition in 120 languages.

Mozilla is currently developing their own service called Deep Speech, hopefully can be validated in 2020 as a replacement for Google, at least in English.

Data and grammar stuff…

const colours = ['maroon', 'darkred', 'brown', 'firebrick', 'rosybrown', 'indianred', 'lightcoral', 'red', 'snow', 'mistyrose'.../* the rest of the 148 named CSS colours */];
const grammar = '#JSGF V1.0; grammar colours; public <colour> = ' + colours.join(' | ') + ' ;';

JSpeech Grammar Format (JSGF)

#JSGF V1.0; states the format and version used. Must be included first.

grammar colours; public <colour> indicates the type of term we want recognised, followed by list of items separated by pipe character.

/* Define speech recognition instance */
const recognition = new speechRecognition();
/* Create new speech grammar list */
const speechRecognitionList = new speechGrammarList();
/* Add grammar to the list */
speechRecognitionList.addFromString(grammar, 1);
/* Add speech grammar list to speech recognition instance */
recognition.grammars = speechRecognitionList;
/* Set language of the recognition */
recognition.lang = 'en-US';
/* Can choose to return interim results or final results */
recognition.interimResults = false;
/* Set number of alternative potential matches */
recognition.maxAlternatives = 1;

const micBtn = document.getElementById('activateMic')
const consoleLog = document.getElementById('consoleLog')

micBtn.addEventListener('click', function() {
  recognition.start(); /* Start speech recognition service */
  consoleLog.innerHTML = 'Ready to receive a colour command.'
}, false)

recognition.onresult = function(event) {
  const last = event.results.length - 1;
  const colour = event.results[last][0].transcript;
  const sanitiseColour = colour.replace(/\s/g, '');
  consoleLog.innerHTML = 'You probably said: ' + sanitiseColour + '.\nConfidence: ' + event.results[0][0].confidence;
  docBody.style.setProperty('--bg-colour', sanitiseColour);
}

Returns SpeechRecognitionResultList object with SpeechRecognitionResult objects, which can be accessed like an array

[last] returns the SpeechRecognitionResult at the last position

Bonus CSS thing

Don't need an extra HTML element for the warning message

.no-speech body::before {
  content: 'Tragically, your browser does not support the Speech Recognition API. When I wrote this, only Chrome supported it, so maybe try this out with Chrome.';
  font-family: sans-serif;
  line-height: 1.3;
  font-size: 85%;
  padding: 0.5em;
  background-color: #ab3c3c;
  color: white;
  text-align: center;
}

Make your browser talk back 🗣️

Browser support for SpeechSynthesis 😐

Moar planning…

Add select dropdown and play button
Populate select with device voice options
Change voice based on selected option
Play response when button clicked

Add requisite elements

<form id="hearResponse" class="response">
  <select id="pickVoice"></select>
  <button id="playResponse" class="btn-response">Hear response</button>
</form>

Populate the select list

const select = document.getElementById('pickVoice');
voices = speechSynthesis.getVoices();
voices.forEach(function(voice) { 
  const option = document.createElement('option');
  option.textContent = voice.name + ' (' + voice.lang + ')';
  if(voice.default) {
    option.textContent += ' -- DEFAULT';
  }
  option.setAttribute('data-lang', voice.lang);
  option.setAttribute('data-name', voice.name);
  select.appendChild(option);
});

Firefox voice list

Chrome voice list

const responseForm = document.getElementById('hearResponse')
responseForm.addEventListener('submit', function(event) {
  event.preventDefault();
  const select = document.getElementById('pickVoice');
  speechSynthesis.cancel(); /* Needed to clear the previous result */
  /* create a new SpeechSynthesisUtterance() instance */
  const utterStuff = new SpeechSynthesisUtterance(result);
  const selectedVoice = select.selectedOptions[0].getAttribute('data-name');
  voices.forEach(function(voice) { 
    if(voice.name === selectedVoice) {
      utterStuff.voice = voice;
    }
  });
  speechSynthesis.speak(utterStuff); /* Start the utterance being spoken */
}, false)

Do the live demo thingy