InSignum

  • About:
    InSignum is an
    interactive system to assist deaf people in their daily life at home. It consists of a smartwatch application and connected directional lights around the house, warning the user of the nature and the location of sounds at home. When something noisy is happening in the house, the smartwatch starts to vibrate, and the app displays what is happening. At the same time, the directional lights turn on guiding the user to the correct location of the sound. The color of the lights indicates the seriousness of the cause of the noise.
    The combination of the smartwatch app and the directional lights is to don't force users to always have the smartwatch and to have a backup system in case one of the two is not working properly.
    We tested the system by building a doll-house prototype with Arduino and emulating a smartwatch with Android Studio.

  • What: Group project

  • Where: Université Paris-Saclay

  • Duration: September 2020 - October 2020

Problem

Consciously or not, people able to hear constantly analyze sounds around them, to be able to orientate and to understand what is happening. Deaf people can only rely on visual and haptic stimuli, so their awareness is usually limited to the room in which they are.
Our target user is often frustrated or anxious to know what is happening inside the house. Things as simple as someone knocking at the door, or it’s raining outside, are all things we take for granted, without considering that not everyone has this luxury.

Target user

Deaf

Hard of Hearing (DHH)

  • Often frustrated or anxious.

  • Invest money in technologies from which to benefit (wearable smart tech, smart lighting solutions).

  • May live alone or not.

Brainstorming

Sounds categories (to which we associate a color in the prototype):

  • Alarms: fire alarm, carbon monoxide detector.

  • Doorbell: doorbell, phone ring, ring of a video call.

  • Water: opened taps, water leaks, water sounds inside the house.

  • Electronic devices: TV sounds, computers, music from speakers, completed dryer.

  • Heating devices: oven. They might involve fire hazards or a fast reaction.

  • Outside noise: rain, automatic watering system, lawnmower.

Ideas for smart house systems based on visual clues:

  • Full light system: lights on a surface area (ceiling, wall, floor).

    • Directional lights: to lead the user to the sound.

    • Ripples: to model the intensity of the sound.

  • Wearable device: the user would be notified of a sound by a wearable device (such as bracelets, smartwatch, smart glass).

  • Central portable board: digital panel the user could put anywhere in the house and on which information would be displayed.

    • Map of the house on which sound icons are drawn.

    • Colored grid, where rows are rooms and columns are types of sound.

    • Something more abstract: a colored grid or a painting changing depending on the current sounds in the house.

Interviews

Numbers

2 Deaf

1 Hard of Hearing

Questions

General:

  1. What was the last time they had difficulties communicating (during their trip for example) or encountered a problem because they couldn’t hear something?

  2. How did you deal with the situation?

  3. What do you think is your biggest struggle that comes from not hearing sounds? And what is your biggest struggle from not being able to communicate verbally?

  4. What kind of problem do you encounter and that you think hearing people are absolutely not aware of?

  5. Do you use any kind of device to help you in your daily life? Does it work well? Do you remember a time it couldn’t help you?

  6. How have the masks for COVID impacted your daily interaction and communication with other people?

  7. Tell us about a bad experience while you tried to communicate with people wearing masks.

  8. If there was an alternative medium to warn you of a sound going on around you, which one do you think you would prefer, a vibration or a light?

Specific:

  1. Tell us about a situation where you were annoyed by something in the house that you couldn’t hear.

  2. Tell me about a situation you realized there was something happening outside you would have liked to know earlier.

  3. Tell me about a situation in which you had a problem with not understanding where a sound was coming from.

  4. Have you developed any specific habits at home to compensate for the hearing loss?

  5. What sounds are you the most concerned about because you can’t hear them?

  6. How would you prefer to be alerted? Examples: vibration on the wrist, or lights, or messages on screens

  7. Where would you like to receive an alert of a sound? From the room where the sound is happening or from a central place? Or other?

  8. On which occasions would you prefer to use an automated system that closes something after a certain time? (for example, an automated system that closes fridge doors).

  9. On which occasions would you prefer to use a detection system that alerts you to close something?

  10. For people with hearing aids (not completely deaf people): Would you prefer for nonurgent alarms to be louder or replaced with lights?

  11. Do you find yourself more often in dangerous situations at home or outside your home?

Findings

Annoying situations because of something that they could not hear

No knowledge about their own sounds.

Unnecessary careful to don't make noise.

Daylight conditions don't allow them to realize the warning lights.

Situations related to sounds happening outside the house

Fear that someone sneak in through the windows when they fell asleep with the window open.

Neighbors ring the bell when something bad is happening (e.g. flooding).

Where they would like to receive the warning for the sounds

A handheld portable device (like a wristband), since a device for every room could be expensive, and an object to carry all the time is uncomfortable.

Warning lights on wall or floor (no ceiling because it's harder to notice). In general, at eye level.

Arrows or light movements to indicate the location of the sound, and colours represent sounds (e.g. blue lights directing to the kitchen could mean open tap).

No central place, to don't check in that place all the time.

Development of any specific habits at home to compensate with the hearing loss

Actively paying close attention to everything, making sure that lights, stoves, or home appliances are off, and taps and windows are closed.

Linking visual cues to sounds (e.g. turn on the extraction hood light every time she turns it on).

Checking often the smartphone, especially if they are waiting for someone.

Asking people to insist ontexting and calling several times, when they don't answer.

Possibility to use automatied systems

Liked the idea of having everything automated, but concerned about the expense to install such systems, since they require AI.

Examples:

  • A turned-on oven with no food inside could be automatically turned off.

  • A timer that turns off the TV when the user is sleeping.

  • A system that tells the user if everything is ok when is going outside home or when the user is on holiday.

  • Automatic tap closure.

Way in which they prefer to be warned

Vibrating accessories (e.g. bracelet, belt), useful when they are sleeping. It can convey information (for example by color).

Warning lamps (e.g. a lamp close to the TV that glows accordingly to its volume).

Something appealing depending on the room (e.g. a lamp with decorative finish, painting canvas, a sculpture).

Light in the skirting boards.

Combine all the options listed above as their pleasure.

Sounds are they most concerned about

Sounds that convey some degree of danger (e.g. burglars sneaking in the house, broken boiler, fires, or water running).

Sounds in the kitchen are more important as they are more dangerous.

Loudness of the TV.

Where do they find themselves more often in dangerous situations

They believe that accidents happen more often at home rather than outside.

Outside there are occasions where they can just “follow” the rest of the people. For example during an evacuation of a place.

Situations they didn't understand where a sound was coming from

Very annoying.

Feeling vibrations, but don't know how to interpret them.

Neighbor moving a sliding door making noise.

Questionnaires

Sent it to deaf associations, Facebook pages targeting deaf people, friends of deaf people, in 3 different languages (English, French, and Spanish).

Questions

  • What sounds in their home our users find most crucial to be alerted about.

  • Whether the sounds they themselves make are relevant to them or not.

  • Whether a qualitative evaluation (ON/OFF) of the sounds in their environment is enough or they would value a quantitative approach (Loudness).

  • How the user interprets colors: location of the sound, type of sound.

  • What colors they instinctively correlate with what kind of sound.

  • What colors they instinctively correlate with what room in the house.

  • Whether the user would find most valuable in a centralized information panel somewhere in their house or some sort of portable/wearable system.

  • For both options above, how the user reacted and interpreted our specific solutions.

  • If our system used lights to alert its user, where they would prefer to have them placed.

Findings

Ranking of importance of sounds:

  1. Doorbell (93%)

  2. Water Taps (77%)

  3. Smoke/Fire Detector (72%)

  4. Major Appliances (66%)

  5. Ventilation and Heating (55%)

  6. Small Appliances (55%)

  7. Broken Glass (50%)

  8. Alarm Clock (50%)

  9. Sounds from people in the house (44%)

  10. Phones & Faxes (38%)

  11. Consumer Electronics (38%)

  12. Rain/Wind outside (38%)

  13. Street Sounds (16%)

  14. Neighbors Sounds (16%)

Location of lights in each room:

  • ceiling (44%)

  • molding (61%)

  • walls (50%)

  • skirting board (38%)

  • floor (50%).

Wearable component to:

  • access cameras in the house (44% strongly agreed)

  • snooze/dismiss notifications (70% strongly agreed)

  • know which object is making sound (66% strongly agreed)

  • know where the sound is coming from (55% strongly agreed)

Preference for haptic feedback through such wearable device:

  • vibration (55%)

  • squeeze (0%)

  • taps (0%)

  • a combination of all 3 modes (33%)

  • 61% want to be aware of their own sounds
    38 % said they had no interest in that
    include their own sounds.

  • 72% want to have insights into the intensity of the sounds
    22% had no interest in that
    include volume.

  • 68% correlated sound with the type of objects that are producing it
    32% correlated it with the type where it is coming from
    set the first one as the default, but allow customization

  • Some users would prefer this system to be unrecognizable by visitors as a hearing aid
    Others did not seem to mind at all
    bridge this gap and design for transparency.

  • Users know what object is making a sound mostly for House Map and Decorative Abstract object
    They are aware of object’s location only with House Map.
    Grid representing different rooms did not test well for either question.
    access to a centralized House Map could be very fruitful but not necessarily the main attraction of the ideal solution.

Solution

01

Vibrating wearable smartwatch

WearOS application that alerts to any sound in the house.

02

House-wide smart lightning

Indicates the direction to the noise source in case the smartwatch is for some reason not accessible.

Easy to set up:

  • Replace ceiling light bulbs with smart-home alternatives

  • Install a light bulb on top of each door.

2 different alert channels: less anxiety when not paying attention to one of them.

    1. The light in the room the user is currently in changes color.

    2. A notification pop-up in the smartwatch containing location and nature of the sound.

Accessible technology: smartwatch, ceiling lights, some optional sensors in the rooms.

Design

Storyboard

“Diana is a deaf person who lives alone. She is feeling bad at her home, she usually has strong migraine attacks. After having breakfast and try to do some cleaning she has to go to the sofa to lie and rest a bit because she cannot stand the headache anymore. While sleeping on the sofa, she feels a vibration in her smartwatch that slightly wakes her up, then she sees a blue light moving toward the other part of the house. Then she checks the screen of the smartwatch and sees a drop icon with the text, water tap running in the big bathroom. She stands up and goes to close it right before it starts to overflow.”

“Diana is still in the shower. The light turns red. She get out of the shower to check her smartwatch. She sees a fire alarm icon appear on the screen and takes all of the screen (as it is an emergencies all other notifications are secondary). She clicks on it and see that the kitchen fire alarm is on then she gets there and sees that the radiator started to catch fire”

Smartwatch application

Navigation map

  • Home screen: shows what is currently happening in the house. “You have everything under control” when there are no notifications.

    • Notifications screen: list of all received notifications.

    • Actions screen: status of the house. It shows:

      • map of the house with icons on locations of noise sources;

      • views of the cameras that are around the house (if present);

      • sensors status.

    • Setting screen:

      • editing house map or names of rooms;

      • configuring sensors and lights;

      • define notification settings (priorities or colors of lights)

Development

The dollhouse

The smart lighting system

Scenario (demo)

  1. The user is cooking in the kitchen. Suddenly, the light of the room becomes green and the smartwatch starts to vibrate. They have to check the smartwatch to see what is happening.

(A person is ringing the bell to deliver a package to the user, so they are testing the notification function and the camera view from the smartwatch).

  1. The user decides to have breakfast in the living room watching the TV. When they turn on the TV, the light of the living room turns on as well to indicate that the TV is turned on. The user does not want to have this permanent notification, so he decides to deactivate it for 20 minutes.

(The user is testing the function of deactivating the light and smartwatch notifications for a given amount of time).

  1. While the user is watching the TV, the light of the room becomes red to indicate that an emergency is happening. Unfortunately, the smartwatch is out of battery, so the user has to follow the path designed by the lights around the house and discover what is happening.

(The user can test the path built from the lights around the house. Following them, he is able to find that there is a problem in the kitchen: they forgot to close the oven properly).