Event-Coupled Hidden Markov Models

T. T. Kristjansson, B. J. Frey, T. S. Huang

Abstract:
Inferences from time-series data can be greatly enhanced by taking into account multiple modalities. In some cases, such as audio of speech and the corresponding video of lip gestures, the different time-series are tightly coupled. Weare interested in loosely-coupled time series where only the onset of events are coupled in time. We present an extension of the forward-backward algorithm that can be used for inference and learning in event-coupled hidden Markov models and give results on a simplified multi-media indexing task where the objective is to detect an event whose onset isloosely coupled in audio and video.

Event Coupled Hidden Markov Models