MEDIA MINING INDEXER If you are looking for a high end speech recognition product that can automate the process of indexing live news feeds and multimedia archives in real-time, your search ends here.

• Best-of-breed speech recognition technology enables accurate conversion of spontaneous speech to text, even with high levels of background noise
• Speaker Change Detection segregates audio segments based on speaker for focused information retrieval
• Speaker Identification is your answer to "who" has been saying "what" in the news
• Topic Detection helps locate stories based on your favorite topics with ease
• Named Entity Detection makes finding of relevant information simpler as words belonging to categories like persons, locations, organizations etc are highlighted within the text output
• Keyword Translation gives immediate access to information contained in foreign language news content
• Scalable Architecture supports single machine to multi machine configuration, providing a cost effective solution for small and medium sized TV stations and media monitoring companies
• Industry standard XML output facilitates integration with complementary technologies

HARDWARE PREREQUISITES

Media Mining Indexer is designed to perform in real time or faster with common off-the-shelf PC hardware. We support hardware running Windows XP (32 bit version), Windows Server 2003 (32 bit version) or Linux on Intel or AMD CPUs.

The Media Mining Indexer requires a sound card to feed input other than PCM audio files.

To feed video from Cable-TV, Video recorders or other input devices using Composite input or S-Video input, Media Mining Indexer provides the ability to use a WDM compatible analog TV card as input. SAIL LABS uses the following TV cards:

• Hauppauge WinTV
• Hauppauge WinTV USB

Configuration hints for real-time indexing:

• INTEL Core 2 Duo E4500 2.0 GHz
• 2GB-DDR 2 RAM
• Disk space: 60 GB recommended for production use on indexers; more for machines acting as DMC; Minimum: 30GB (between 350 and 800 MB per Language Feature plus temporary disk space)

SOFTWARE PREREQUISITES

• Microsoft Windows XP SP2 or Windows 2003 Server
• DirectX 9.0 or later is required for indexing mpeg, mp3, wmv, avi files using the optional Media Feeder program.
• ActivePerl-5.8.7.813 from ActiveState. Can be downloaded from http://ftp.activestate.com/ActivePerl/Windows/5.8/ActivePerl-5.8.7.813-MSWin32-x86-148120.msi.
  Note: Only the mentioned version of ActivePerl is guaranteed to work with Media Mining Indexer.

Languages supported by Media Mining Indexer are:

• Arabic
• English (US)
• English (US/UK)
• French
• German
• Greek
• Norwegian
• Russian
• Spanish

The open architecture of the Media Mining Indexer enables easy integration with complementary technologies for diverse applications. Some of the products which have integrated the Media Mining Indexer are Fast's Media Miner®, Oracle 9i interMedia, NOA's Dactylo, pluggd's HearHere, Mediaclipping's mediaclipping.de platform, blue order's media archive®, Virage Videologger® etc. The integration can be done at two levels:

• Using command line tools provided by Sail Labs
• By linking with the API C++ library and directly accessing Indexer components

The command line tool is currently available in two different formats: precompiled as a win32 binary (asrsample.exe) and in source code form together with a Microsoft® Visual Studio 2003 solution. The command line tool works network transparent, allowing for speech recognition across hardware boundaries.

The API facilitates communication with the Media Mining Indexer. It provides a set of C++ classes. These classes are defined in an API Library Header file. The API allows a client application to:

• Connect to one of several Indexers running on the same or other machines
  
• Register a result handler which will receive transcription results and status info
• Initialize an Indexer instance for the desired language and domain
• Send audio data to the Indexer
• Wait for transcription results, live (phrase by phrase) and/or final (after processing all audio, in xml format)

The Media Mining System harnesses the synergies of some of the best speech processing and language technologies produced or currently in development. Technologies such as Automatic Speech Recognition, Speaker Identification, Named Entity Detection, Topic Classification, and Story Segmentation have been integrated in our system, and together produce comprehensively indexed text files from the media stream input.

Automatic Speech Recognition

Automatic Speech recognition is performed in a sequence of steps; it first processes the incoming audio, then segments the audio into sections of speech and non-speech, and then applies speech-recognition to those segments identified as speech. This is done in real-time, for large vocabularies (>>64K entries) and for 8 and 16kHz (but is not limited to these) audio data.

Our speech recognition engine is language independent (modulo changes on acoustic front end, e.g. for tonal languages) and has been run in a variety of languages such as: English, French, Spanish, German, Arabic, Russian, Norwegian or Polish.

Front-End: Standard MFCC coefficients, energy+ 5/3 frame regression, deltas, delta-deltas, cepstral mean subtraction, various types of normalization and active noise cancellation.

Acoustic Models: Speaker- and Gender-independent acoustic models, 3- and 5-phone context models, with Gaussian Prototypes tied at the prototype as well as mixture weight levels.

Language Models: word-based n-gram models employing a Witten-Bell-like back-off. The Language Model and Acoustic scores are combined for maximum accuracy.

Decoder Search: We use a multi-pass, time-synchronous search. During processing, increasingly detailed models are used at each step. After a forward pass and a backward pass, the resulting N-best list is re-scored using the most detailed acoustic models.

Speaker ID / Clustering

The Speaker Identification (SID) system identifies speakers or the speakers' gender. The incoming audio is first split up according to speech / non-speech regions and speaker turns are hypothesized on these chunks. Speaker clustering (SC) and Speaker ID (SID) are run on the resulting chunks. Typically from about 20 to 100 speakers can be identified; for non-target speakers the gender is detected and the unknown speaker's segments are labeled accordingly. Speaker ID/Clustering is language independent.

Speaker Identification: Gaussian Mixture Models (GMM) for a number of pre-selected target speakers and an additional number of cohort speakers who serve for normalization and gender detection purposes.

Speaker Clustering: clusters pre-determined chunks (from initial segmentation) into a number of clusters. As quality measure the within-class dispersion is used. Segments are clustered using a variant of the generalized likelihood ratio criterion.

Speaker Change Detection

Speaker Change Detection (SCD) performs a phone-level decoding stage, which employs a set of broad phonetic classes of speech sounds as well as non-speech sounds. Using the information produced by the decoder (i.e. the "transcript"), the SCD system sequentially hypothesizes speaker turns at phoneme boundaries. A generalized likelihood ratio test is used to determine whether a change should be made.

Story Detection and Topic Classification

Story Detection consists of several phases. First, an episode is partitioned into homogenous speaker turns. The speaker turns are initially classified and adjacent speaker turns are merged and re-classified in a subsequent step.

Support Vector Machines, one model per topic and one model to model general language (all those filler words which really aren't specific to any topic) are used. Each classifier represents the topic and its topic dependent words.  At decoding time, the most likely set of topics given the recognized text is determined.