AI News·15 Nov 2026·via Dev Community

    Unmute: reading silent lips for ICU patients who can't make a sound (offline, on a laptop)

    A project developed at HackUMass, named Unmute, provides a system for Intensive Care Unit (ICU) patients unable to speak due to conditions like tracheostomies. The system uses a standard laptop camera to read silent lip movements, converting them into spoken sentences. Developers state Unmute operates offline, requires no special hardware, and learns from individual patient lip patterns.

    Unmute addresses a communication challenge for ICU patients who are awake but cannot vocalize. Nurses often resort to guessing patient needs, which can lead to fatigue. The system aims to mitigate this by presenting patients with three likely sentences from a list of 20 bedside phrases, which they can confirm using movements like a touch, nod, or blink.

    The system's core functionality relies on a blend of two technologies. It incorporates Auto-AVSR, an open-source lip reader with 250 million parameters trained on over 3,000 hours of video. Rather than attempting free-text transcription, Unmute uses this model to score the likelihood of fixed sentences. This is combined with a Dynamic Time Warping (DTW) matcher that compares new lip recordings against existing templates, adjusting for variations in speaking speed.

    A key aspect of Unmute is its adaptive learning mechanism. Each time a patient confirms a sentence, that specific recording of their lip movements is saved as a new template. This process allows the system to personalize its understanding to the individual, accommodating unique physiological challenges like swollen or lopsided lips, thereby improving accuracy over time. Urgent messages, such as "I cannot breathe," trigger immediate alarms to alert nursing staff.

    Share this article

    Original reporting by Dev CommunityWe don't republish, read the full story →

    Related reading

    6 stories