Building a Home Automation Virtual Agent for Smart Home Control

A walkthrough on building a bot to receive commands and control a connected home with IBM Watson and PubNub

The virtual agent, interactive command-receiving, action-taking entities, have found itself a staple feature of any home automation solution. From Tony Stark and Jarvis to Mark Zuckerberg’s Jarvis of his own, home virtual agents aren’t just a Hollywood dream, but a reality. And with now cloud-based technologies, it’s not so hard to build anymore.

In this tutorial, we’ll build a virtual home automation agent that responds to natural language to carry out commands and monitor state of the smart home. We’ll use IBM Watson Conversation to power our NLP and PubNub BLOCKS to execute our Watson Conversation functions on the data in motion. That’s right, this app is serverless!

Project Components and Source Code

We are going to build this exact replica of the simulated home automation demo. For your reference, the source code of this demo is available here in GitHub.

Here are the components of the demo:

Home (Repo)

We have used an SVG image of a home that has the floor plan with each room having a light bulb. Since we can’t ship a real working home automation system to you, I have created a virtual smart home with an SVG image and an icon for each room’s light bulb. This home understands commands to toggle the light in each room, and we will see how can we do it via the Watson Conversation API.

Virtual Agent (Repo)

The virtual agent runs on the PubNub BLOCK. Its job is to decipher the commands sent by the homeowner and trigger the specific action on the home. For interpreting the commands, it relies on the Watson Conversation service.

Command Window (Repo)

The UI that the homeowner can use to instruct the virtual agent.

IBM Watson Conversation

IBM Watson Conversation allows you to build natural language interfaces that can interact with machines and virtual agents. This enables us to retain our most natural way of communication while the conversation service does the heavy lifting at the backend to translate our messages so that the machines can understand.

Watson Conversation service provides a drag and drop dialog builder. This dialog builder is the key to designing dialogs that are intuitive for human users and, at the same time, can be deciphered by machines.

A typical human-to-machine conversation is very different from human-to-human conversation. Machines do not usually engage in a conversation flow like we humans do. They take commands and respond with an action. The Watson Conversation service allows us to build specific dialogs that convey commands and actions between humans and machines. The image below illustrates how this is achieved via intents and entities:

As you can see above, the Watson Conversation service breaks down each human message into an intent and an entity. The intent specifies the command to be executed, and the entity specifies the particular machine or device that should execute the command.

In the home automation scenario, this boils down to sending commands to the lights and appliances to either turn on or turn off. So imagine you have an Internet gateway at home which also hosts a virtual agent that is accessible through Whatsapp. If you hook up that gateway to your home automation system, then you can chat with the virtual agent to control your appliances (see the example illustration below).

Building the Home Automation Virtual Agent

Let’s go ahead and build this home automation virtual agent. We have already unraveled the complexity of deciphering the conversation thanks to IBM Watson. Now comes the next challenge, that is, building the virtual agent.

Contrary to the usual practice of building a virtual agent using a server-based application, we can have an agent up and running on the fly with the help of PubNub BLOCKS. PubNub BLOCKS is a serverless computing module which can be programmed easily and with no infrastructure overhead. With BLOCKS, this is how the system architecture of our home automation virtual agent will look (see below).

Now we have a system deployed using PubNub BLOCKS as a virtual agent. It performs three responsibilities:

1. Receive messages from the human user.
2. Translate the message with the help of Watson Conversation API and break them down into command (intent) and device (entity).
3. Send commands to the respective device.

Virtual Agent in Action

Once built and deployed, we could command our virtual agent and control our home.

Cloud Components

To power this virtual agent, we rely on two backend cloud services.

1. Watson Conversation API: Available as a service in the IBM Bluemix catalog, Watson Conversation service provides us with all the tools to build a context-aware, dialog system that can be used to power chatbots. At PubNub, we have used it to power our virtual agent that is hosted on PubNub BLOCKs.
2. PubNub Data Stream Network: PubNub provides an ultra-fast, real-time data streaming solution over the Internet which powers many messaging applications across the world. PubNub BLOCKs is a component of PubNub that enables us to deploy microservices over the PubNub data streaming service. We have built a lightweight Javascript Application for the virtual agent that executes within the BLOCK and interacts with the user, the home as well as the Watson conversation API.

Refer to the README file to setup your conversation service and PubNub BLOCKs service.

For building this app, you will need to create a Bluemix and PubNub account. Visit the IBM Bluemix signup page and PubNub service page to create your respective accounts. Both of the services offer a free tier account to play around with their offerings.

Operations Performed by the Virtual Agent

There is no limit to the kind of dialogs that you can build with the Watson Conversation service. However, to keep things simple and to show you the most basic novelty of this concept, let’s define the most common commands that we usually perform upon our home appliances.

[Turn on bedroom]

Here, Turn on is the intent and bedroom is the entity. When we issue this command, we are instructing the virtual agent (PubNub BLOCK) to switch on the bedroom light. Below is an illustrated example of how the messages flow across the system components to make this work.

In a similar way, we can send control commands affecting other rooms for turning on / off the lights:

• turn on childrenroom
• turn on livingroom
• turn on kitchen
• turn on portico
• turn off bedroom
• turn off childrenroom
• turn off livingroom
• turn off kitchen
• turn off portico

We have also defined an “is” intent used for monitoring purposes. This helps us to query the present state of the room light.

[is bedroom on]

The virtual agent’s JSON response for this command is:

{ "Type": "monitor", "Entity": "bedroom", "Status": "ON", "Message": "Status bedroom" }

This way, we can clearly decipher the chat commands and instruct the home gateway to either perform an action or report the status on a specific device/appliance inside the home.

Extending the Capability of the Virtual Agent

All these intents and entities are predefined in the Watson Conversation service workspace instance that we have created for this app. So in case we want to extend its capability, we can easily do so by adding more intents and entities.

Now that we have a virtual agent that can understand our desires to turn on or turn off the lights in our home, what more can we do? As it turns out, with Watson Conversation service’s dialog builder, we can build more complex dialogs and program our agent to perform more tasks.

So here is a challenge for you. How about dimming the lights of your home?

You can write a “dim on” and “dim off” intent for the individual rooms or you may choose to define a “dim all” intent that affects all the lights, as shown above.

Conclusion

With the Watson Conversation service, the cognitive computing wave has reached a new level. From mere number crunching and predictions based on data, we have moved on to the realm of natural language interaction with machines.

One can imagine all those device control dashboards with countless buttons and widgets, getting replaced by a plain and simple chat interface. And, along with PubNub BLOCKS, developers can create rich and intuitive experiences for the users just like this home automation virtual agent.

So buckle up and give it a shot. Follow the docs and learn to build the dialog for Watson Conversation service to add these new intents. And, please do share your experience with us!

Originally published at www.pubnub.com.