| http://www.radarnetworks.com/enrichment#convertedBinaryContent
| -
Live Staging of Media Events – LIVE
Contact: www.ist-live.org
Email: info@ist-live.org
State of the Art Report
Intelligent Media Framework
Deliverable D7.1
Project Ref. No. Integrated Project FP6 / IST 27312
Deliverable/WP/Task D7.1 – State of the Art Report (IMF)
Delivery Date 27.11.2006
Author(s) Georg Güntner / SRFG / gguentner@salzburgresearch.at (ed.)
Wernher Behrendt / SRFG / wbehrendt@salzburgresearch.at
Tobias Buerger / SRFG / tbuerger@salzburgresearch.at
Alberto J. Cruz / ATOS / alberto.cruz@atosorigin.com
Christian Eckes / FhG / christian.eckes@iais.fraunhofer.de
Gerhard Stanz / ORF / gerhard.stanz@orf.at
Rupert Westenthaler / SRFG / rwesten@salzburgresearch.at
Janez Zaletelj / UoL / janez.zaletelj@fe.uni-lj.si
Felix Zielke / FhG / felix.zielke@iais.fraunhofer.de
Filename D71_SOTA_Report_20061127_final.doc
Publication Level PU=public
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 2 / 63
Copyright Notice: © LIVE Consortium. 2006. All rights reserved.
This document contains material, which is the copyright of certain LIVE con-
sortium parties, and may not be reproduced or copied without permission.
The commercial use of any information contained in this document may re-
quire a license from the proprietor of that information.
Disclaimer: Neither the LIVE consortium as a whole, nor a certain party of the LIVE con-
sortium warrant that the information contained in this document is capable of
use, nor that use of the information is free from risk, and accepts no liability
for loss or damage suffered by any person using this information.
Neither the European Commission, nor any person acting on behalf of the
Commission, is responsible for any use which might be made of the informa-
tion in this document.
The views expressed in this document are those of the authors and do not
necessarily reflect the policies of the European Commission.
Full project title: Live Staging of Media Events
Project Co-ordinator: Joachim Koehler / FhG
Project Information:
Project ID: FP6-27312
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 3 / 63
Table of Contents
1 Executive Summary..........................................................................................5
2 Introduction .......................................................................................................7
2.1 Scope ....................................................................................................................... 8
2.2 Relationship with other Deliverables ...................................................................... 9
3 Related Technologies .....................................................................................10
3.1 Asset Management Systems.................................................................................. 10
3.1.1 Media Asset Management (MAM)............................................................ 10
3.1.2 Semantic Media Asset Management ......................................................... 15
3.2 Recommender Systems ......................................................................................... 17
3.2.1 Collaborative Recommender Systems....................................................... 17
3.2.2 Content-based Filtering ............................................................................. 18
3.2.3 Hybrid Recommender System................................................................... 19
3.3 Metadata Generation Systems............................................................................... 20
3.3.1 Manual Annotation of Content Metadata .................................................. 20
3.3.2 Automatic Annotation of Content Metadata ............................................. 22
3.4 Video Conducting Systems ................................................................................... 23
3.5 Interface Technologies .......................................................................................... 24
3.5.1 Remote Procedure Invocation ................................................................... 24
3.5.2 Message Oriented Solutions ...................................................................... 27
4 Related Standards...........................................................................................28
4.1 Standards Related to the Broadcasting Domain and Media .................................. 28
4.2 Knowledge Representation Standards................................................................... 33
4.2.1 Knowledge Representation Languages ..................................................... 33
4.2.2 Knowledge Representation Models (Ontologies) ..................................... 34
4.2.3 Foundational Ontologies ........................................................................... 36
4.3 News and Journalism Domain Standards.............................................................. 40
5 Requirements for the Intelligent Media Framework.....................................43
5.1 Architectural Requirements................................................................................... 43
5.2 Combined Requirements of Knowledge-, Metadata- and Asset Management ..... 44
6 Assessment of Models for Intelligent Content .............................................46
6.1 Metadata Annotation in the Web........................................................................... 46
6.2 Approaches to Intelligent Content Models............................................................ 47
6.2.1 Early Models - from Office Automation to Media Presentation............... 47
6.2.2 The MPEG Family of Content Models ..................................................... 48
6.2.3 Semantic Web and Multimedia Standards ................................................ 49
6.2.4 Active Media Objects for Mobile Communications.................................. 50
6.3 Knowledge Content Objects (KCOs) .................................................................... 51
6.4 Comparison of Selected Knowledge & Content Models ...................................... 54
7 Conclusions for the Design of the Intelligent Media Framework................55
8 References.......................................................................................................57
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 4 / 63
History
Version Name Date Remark
V0.01 gg 2006-08-25 Definition of basic structure
V0.05 gg 2006-09-06 Draft structure finished for discussion with the
WP7-team
V0.16 gg (ed) 2006-11-10 Start of final revision based on the integration of
the inputs of the partners
V0.20 gg 2006-11-21 Start of consortium QA
final gg 2006-11-27 Final version sent to OWL
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 5 / 63
1 Executive Summary
The integrated project “LIVE Staging of Media Events” (LIVE; FP6-27312) aims at the crea-
tion of novel intelligent content production methods and tools for interactive digital broad-
casters to stage live media events in the area of sports, such as the 2008 Olympic Games.
This report presents the state of the art of the concepts, technologies and standards related to
one of the core subsystems developed in the LIVE project: The “Intelligent Media Frame-
work” provides a robust framework for the creation, management and delivery of so called
“Intelligent Media Assets” under real-time conditions.
In order to achieve the envisaged “intelligence” of both, the media assets and the framework,
the various descriptive schemes for the content objects (i.e. archival video material and live
broadcast streams), the staging concepts (i.e. a description of how an event is broadcasted),
the consumer profiles (individual or filtered group based profiles) and the professional user
profiles (characteristics of the video conducting team) have to be integrated on a semantic
level. The semantic tagging “tagging of meaning” has to be done in a consistent way through-
out the system and will lead to the definition of an intelligent content model for the purpose of
the LIVE project. Intelligent content models describe and define the structure of content and
knowledge associated with the content and moreover describe the relations between content
related items and the knowledge base (modelling the domain knowledge independently of the
assets).
D7.1 State of the Art Report
Intelligent Media Framework
Conclusions
Introduction
References
Related Technologies Related Standards
Requirements
Assessment of Models for Intelligent Content
Figure 1: Structure of D7.1 “State of the Art Report (IMF)”
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 6 / 63
The Intelligent Media Framework in its role as an integrating middleware for different knowl-
edge sources within the LIVE system is highly dependent on standardization. This applies to
both, interfaces with external systems and to internal communication with the metadata gen-
eration system, the video conducting system and the recommender system. Our research will
therefore be devoted to the development of a knowledge base that can be used and accessed
by various components of the LIVE system to integrate content and knowledge. Moreover,
the Intelligent Media Framework is defined in a way that allows interoperability with underly-
ing production systems, such as playout or/and asset management systems.
The objective of this report is an investigation of standards and technologies with respect to
the requirements of the Intelligent Media Framework’s interoperable and open approach. The
structure of the report is shown in Figure 1:
• Section 2 defines the scope of the report and describes its relationship with other de-
liverables.
• Section 3 provides a report on the basic technologies related to the Intelligent Media
Framework: Asset management systems, recommender systems, metadata generation
systems and video conducting systems. The section closes with a survey of interface
technologies.
• Section 4 gives an overview of standards related to the broadcasting and media do-
main, knowledge representation standards and standards in the journalism and news
domain.
• Based on the related technologies and standards we define the architectural require-
ments and the combined requirements for content-, metadata and asset management in
section 5.
• In section 6 we assess models for intelligent content models with respect to the re-
quirements for the Intelligent Media Framework: After an introductory note on meta-
data annotation in the Web, we describe recent approaches to intelligent content
models. After an introduction to knowledge content objects (KCOs) as a promising
model for the LIVE project, we compare selected knowledge and content models.
• Finally, we summarize the major findings and draw initial conclusions in section 7 and
section 8 lists the references.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 7 / 63
2 Introduction
The central idea of the “LIVE Staging of Media Events” (LIVE) project is to create
• Novel intelligent content production methods, and
• Tools for interactive digital broadcasters
to stage live media events such as the 2008 Olympic Games.
In the terminology of the project, “staging live media events” is a notion for the creation of a
non-linear multi-stream video show in real-time, which changes due to the interests of the
consumer (end user). From a technical viewpoint, this requires a transformation of raw audio-
visual content into “Intelligent Media Assets”. LIVE will develop a knowledge kit and a tool-
kit for an intelligent live content production process including dynamic human annotation and
automated real-time annotation. As a consequence novel iTV video formats for live events
will evolve.
From a scientific perspective, LIVE will address the key question: How can a video become
an intelligent media asset matching the needs of the interested consumer? The arising techni-
cal question is: Which tools allow for the creation of content (including commercial videos
and advertisements) that finds its way to an interested consumer on its own?
The Intelligent Media Framework (IMF) which is the topic of this deliverable is the central
part of work in workpackage 7 (WP 7) of the LIVE project. This workpackage is responsible
for the combination of cognitive based content knowledge, which is delivered by the to be de-
veloped annotation system, with social based consumer knowledge, which is part of the per-
sonalisation and feedback module (WP 6). To realise the novel idea of live staging of media
events, objects encapsulating knowledge about the content have to meet objects that bear
knowledge about the user (consumer) or/and groups of users, which is created by user feed-
back mechanisms. An open media framework for broadcasting environments will be devel-
oped, where the two types of media objects can live and interact.
The Intelligent Media Framework will be responsible for the combination of content based
knowledge with social based consumer knowledge. Content knowledge is provided by the
LIVE annotation system and the automatic knowledge extraction and detection components.
Consumer knowledge is derived from the consumers’ feedback by the personalisation and
feedback components of the LIVE system.
In order to design and implement the IMF, methodologies and tools from several work pack-
ages have to be integrated into a consistent and robust framework that allows for the live stag-
ing of media events. The methodologies for staging and content research (WP 4), the content
oriented detection, extraction and annotation of video material (WP 5) and the personalisation
of the users (WP 6) have to work together and enforce each other in an intelligent way. The
central item of the IMF are so called Intelligent Media Assets (IMA) that allow for this inter-
action of content based knowledge with social based consumer knowledge (connect item
based knowledge with user based knowledge).
In order to become Intelligent Media Assets, media objects have to be enriched by (semi-)
automatically extracted metadata. This input comes from tools developed in LIVE workpack-
age 5 (WP 5). From that perspective, the human annotation tool is an important component,
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 8 / 63
capable of catching human interpretations about content within the staging process. Personal-
isation and (hybrid) recommendation algorithms developed in WP6 will be crucial in order to
realize the envisioned “intelligence”. To interact with the environment and the consumer the
IMAs necessarily have to be a part of the video conducting system, another LIVE system
component processing and presenting the IMAs.
The following two sections define the scope and the relations to other deliverables of the
document.
2.1 Scope
The scope of the document is to provide a state of the art report on technologies, standards
and concepts for LIVE’s Intelligent Media Framework. This will include the following topics:
• A description of technologies and selection criteria for (semantic) media asset man-
agement systems relevant for the Intelligent Media Framework (see section 3.1).
• A description of technologies and selection criteria for recommender systems relevant
for the Intelligent Media Framework (see section 3.2).
• A description of technologies and selection criteria for metadata generation systems
(manual and automatic annotation) relevant for the Intelligent Media Framework (see
section 3.3).
• A reference to video conducting systems (see section 3.4).
• A description of technologies and selection criteria for interface technologies relevant
for the Intelligent Media Framework (see section 3.5).
• A description of standards related to the broadcasting and media area, knowledge rep-
resentation standards and standards in the news and journalism domain (see section 4).
• An introduction to intelligent content models comprising the requirements, current ap-
proaches and a conceptual approach, called “Knowledge Content Objects” (KCOs, see
section 5).
The following topics are not covered by this document to avoid redundancies with other LIVE
deliverables:
• Standards and technologies related to interactive digital TV technologies and plat-
forms (e.g. the Multimedia Home Platform, MHP): These are covered by [LIVE D3.2]
(“Technology Market Watch”).
• Standards and technologies related to staging concepts. They are covered by [LIVE
D4.3] (“Methods, design guidelines, workflows for online staging”).
• Standards and technologies related to storage and playout systems for broadcast me-
dia. They are partly covered by [LIVE D8.1] (“Description of the overall implementa-
tion and integration plan”).
• Standards and technologies related to the description of consumer models are subject
of [LIVE D6.1] (“First specification of the personalized content recommender sys-
tem”).
The document does not aim to provide in-depth introductions to all of the covered technolo-
gies, standards and topics: This information is provided and public accessible at other sources
which are in detail covered in section 8 (“References”). The document rather emphasises to
point out the relevance of the selected topics in the context of the LIVE project in general and
in the context of the Intelligent Media Framework in particular.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 9 / 63
2.2 Relationship with other Deliverables
The following public LIVE deliverables are related to this document and provide further in-
formation on the overall LIVE system:
[LIVE D3.2] “Technology Market Watch”
Reviews technologies with a potential impact on the new market segment
developed in LIVE: “intelligent television services”. The report focuses on
technologies supporting the access of consumers to the content.
[LIVE D4.3] “Methods, design guidelines, workflows for online staging”
Addresses concepts and methods for conceivable TV formats entailing a
multitude of live audio-visual material, which shall be transformed into an
interactive live TV event for the consumer at home.
[LIVE D9.1] “Results from initial User Requirement Analysis”
Defines the scenarios and requirements for the personalized services and user
feedback.
[LIVE D9.3] “Public synopsis document on basic system architecture”
Contains a summary of the basic system architecture and the components of
the LIVE system defined in the (not public accessible) D9.2 “Basic system
architecture”.
The following LIVE deliverables are (respectively will be related) to this document, however
access to the documents is restricted to the LIVE consortium:
[LIVE D5.2] “Report on live human annotation”
Contains additional information about annotation tools and introduce the an-
notation tool developed in LIVE.
[LIVE D6.1] “First specification of the personalized content recommender system”
Specifies the recommender system and introduces standards and technolo-
gies to model the consumer profiles.
[LIVE D6.3] “Report on content selection methods”
Describes the content selection and personalization methods.
[LIVE D7.2] “First specification of intelligent media framework”
Specifies the Intelligent Media Framework.
[LIVE D8.1] “Description of the overall implementation and integration plan”
Defines the first prototype of the LIVE system, including the services and in-
terfaces provided by the RS, and the implementation plan for the prototype.
[LIVE D9.2] “Basic system architecture”
Defines the basic system architecture and the components of the LIVE sys-
tem.
Throughout the document references to other LIVE deliverables are marked with [LIVE
Dn.m] (“Dn.m” denoting the number of the deliverable, e.g. “D5.2”),
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 10 / 63
3 Related Technologies
This section investigates technologies related to the Intelligent Media Framework with respect
to the objectives of LIVE. A full coverage of the technologies is not intended in this docu-
ment, rather hot spots in the area of the technologies are identified that have to be considered
to establish the “intelligence” of the Intelligent Media Framework. Section 3.1 gives an over-
view of asset management systems and extensions by the use of semantic technologies (se-
mantics media asset management). Section 3.2 covers the area of recommender systems and
provides an overview of technologies applicable in the scope of the LIVE project. Section 3.3
deals with metadata generation systems and covers aspects of manual and automatic annota-
tion and extraction of knowledge. Section 3.4 creates a reference to video conducting systems.
Finally section 3.5 gives an overview of interface technologies applicable for the communica-
tion of the system components of the LIVE system.
3.1 Asset Management Systems
3.1.1 Media Asset Management (MAM)
Digital Asset Management (DAM) is a set of coordinated technologies and procedures that
allow the efficient storage, retrieval, and reuse of digital files that are important to an organi-
zation. By employing the descriptive information attached to the assets, DAM can provide
and support the business rules and processes needed to acquire, store, index, secure, search,
export and transform them. Digital Asset Management is sometimes referred to as Media As-
set Management. Media Asset Management (MAM) is defined as the technologies used to
locate and retrieve specific content objects from analog to digital media.
In general, a digital asset is any digital media file that has value. Digital assets often include
rich media such as video, audio and graphics, but this is not a requirement. Images, graphics,
logos, video and sound files, web pages (HTML, XML), PDF documents, Quark and Illustra-
tor files, MS Office files, free text files, ads, marketing collateral, brochures, product packag-
ing designs, etc, all qualify as digital objects. Most companies and organizations have many
kinds of digital assets such as text, photos, logos, music, and video that will have potential
value in the future, if (and only if) they can be relocated and reused.
Media Asset Management Systems are systems that create a centralized repository for digi-
tal files that allows the content to be archived, searched and retrieved. The digital content is
stored in databases or on file systems called asset repositories while metadata such as photo
captions, article key words, advertiser names, contact names, file names or low-resolution
thumbnail images are stored in separate databases called media catalogs and point to the
original items.
According to popular definitions the difference to Content Management Systems (CMS)
and Document Management Systems (DMS) can be located in the type of managed assets.
MAM is targeted in managing rich media assets, such as video, audio, 3D, graphics, images
and animations, in a media independent way.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 11 / 63
Functionalities of content management systems for broadcast production
The challenge for large broadcasters in the switch from tape to file is to find and then imple-
ment systems, that firsthand simply replace all the implicit capabilities of physical media for
conventional workflows. The characteristic advantages of file-based workflows need of
course to be considered additionally at least to justify the substantial expenses.
Looking at tapes compared to files, one can easily enumerate a bunch of implicit methods that
are applicable to cassettes in the conventional workflow which are not trivial to be substituted
in the file domain. One can:
• organize tapes easily and intuitively on desks, shelves, rooms according to actual or-
ganizational needs
• write things on labels, apply stickers, plug documents into banderoles (e.g. to be sub-
stituted by file-attachments)
• intuitively organize the changeover of responsibilities by putting tapes form one’s
onto another’s desk
• hide confidential information by putting tapes into a drawer
• speed up the transfer by e.g. running to a studio if necessary
• easily transport large amounts fast over substantial distances (a production with 100
hrs of raw-material can by far faster change edit suites by putting 100 tapes into a
shopping cart than copy 100 hrs of video from one server to the other)
• prevent tapes from accidental deletion
All these and some more challenges need to be addressed somehow by a video-CMS for a
large broadcaster.
Typical functionalities and according software instances in this sense would be:
• Management of Media Ingest (Scheduled, Manual, Looped)
a software module and front end that in practice is often realized as a combination of a
grid of ingest channels and a timeline combined with a “crash record” video control in
order to manage video routers and encoding devices of various vendors
• File Import/Export (incl. Re-Transcoding, Rewrapping)
a software module with necessary controls for file “up- and download” often inte-
grated with a transcoding solution of a 3rd party provider and “watchfolder” function-
alities
• Metadata Import/Export/Throughput (incl. Re-Formatting/Embedding/Debedding, At-
tachment Handling)
a software module relying on a database not surprisingly with similar user controls as
for file im- and export, as the process on a meta-level is a form of “semantic transcod-
ing” and of course representation, also embedding and debedding of metadata into rsp.
from files is often a functionality in transcoding suites
• Unique Identification
a software module to provide unique-ids in a definable range (e.g. world-, company
workflowwide) to be used as filenames or data-tags
• Management of Versions
a feature of the data-model of the underlying master database with according function-
alities in maintenance and representation
• Integration with planning and other administrative systems
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 12 / 63
an implementation of the interface functionalities of the MAM
• Distributed Storage Management (Housekeeping, Reporting, Allocation)
a software module to manage metadata and essence in accordance with server-space
and usage, often combined with a rule-based delete-engine based on properties as free-
space, last use, file size and similar
• Management of Access-Rights
a software module to economically manage access-rights on various levels, with an in-
tegration to the authentification at the operating system
• Management of Workflows
a software module to organize human and technical resources of various kinds, and to
schedule the sequence and trigger the start of distinct tasks to be performed automati-
cally or manually. Often with graphical modeling interfaces. Often with powerful re-
porting functionalities
• Preview (Low-Resolution Video, Keyframes, Sound-Envelopes)
a software module usually highly integrated with transcoding/encoding as well as
video-server and video-player technology for preview files of different 3rd party ven-
dors to provide - in best case frame accurate – preview and rough-cut functionalities to
low res proxies of the managed hi-res files
• Rough-Cut and “Shopping Basket” functionalities
a functionality to organize and personalize selections of the preview-material
Some typically discussed features are deliberately not mentioned above, because they rather
belong to “broadcast archive CMS” or are not yet in a state of development for productive
use. Such features would be:
• all kinds of automatic analysis tools
• License and Rights Management
• Thesauri and other advanced classification systems
Introduction of some distinguishing terms
To asses the state of the art of “video broadcast production content management” and its dy-
namics, some distinguishing terms need to be introduced.
Production vs. archive CMS
An important differentiation is to be found between production- versus archive-CMS. A “pro-
duction CMS” covers the actual TV production therefore:
• it needs only weak metadata capabilities compared to an archive CMS (most of the
material gets deleted again, a lot of implicit information is known by the producing
staff)
• it needs only weak copyright management compared to an archive CMS (same reason
as above)
• it is deletion-oriented (a lot of material needs to be deleted every day – partly archived
of course - in order to get space for new material)
• supports strong versioning
• very fast access is necessary (all material is more or less close to immediate transmis-
sion)
• frame accurate preview in order to overcome manual hi-res editing
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 13 / 63
An “archive CMS” compared to it rather focuses on:
• strong metadata (essence selected to be archived deserves intensive documentalist’s
attention in order to facilitate its frequent re-use)
• strong rights documentation (same reason as above)
• orientation towards preservation
• economical storage of huge amounts with according cut backs in fast access
• preview of large amounts efficiently, not necessarily frame accurate
Essence vs. Content vs. Asset
As explained, the terms „media asset management“, „content management“, “media manage-
ment” describe functionalities necessary for the collaborative work with digital media espe-
cially if this work is based on the division of labour. Which clearly is the case in broadcast
companies.
In business communications – as at trade fairs - there is so far neither a clear differentiation
between these terms within the broadcast domain so that “content management” would have a
distinct and mutual denotation in relation to say “media management”. Nor is there a clear
differentiation between the usage of these terms within the broadcast domain towards
neighbouring domains.
If we take a closer look at the “product locator” of one of the most important broadcast trade-
shows, the IBC 2006 in Amsterdam, we find the following results:
• 87 matches for 'Archiving Systems'
• 104 matches for 'Content Management'
• 59 matches for 'Digital Asset Management'
• 76 matches for 'Media Asset Management'
• 102 matches for 'Media Management'
• 103 matches for 'Video Content Management'
Obviously there is a lot of background information necessary to understand, what the scope of
a particular product really is. On most systems one only can judge after a substantial hands on
experience.
In the broadcast/media domain we can find a widespread and commonly accepted definition
of basic content-management related terms that at least give some clue where a differentiation
between media-, content- and asset-management is to be found. Unless there is still the prob-
lem, that vendors do not stick to it in their denominations.
In this simple “hierarchy of media & metadata” the term “essence” (“media”) refers to the
media itself - which is essential for media production but of low value without any descriptive
information. Combined with descriptive metadata this essence is referred to as “content”. If
there is furthermore copyright rsp. licensing-metadata available it is called an “asset”.
In this sense it would be the complexity of metadata to be dealt with, ranging from simple to
complex descriptive information and further on to legal/commercial metadata, which would
build up a hierarchy ranging from Media- to Content- and further to Asset-Management Sys-
tems.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 14 / 63
A comparison with neighbouring areas which are already well developed, allows insights into
the dynamics of “content-managing-systems” as such, from which we can draw some conclu-
sions regarding the state of the art in the broadcast area.
As two areas with such a well developed state of “content management systems” we will now
take a look at document-management and web-content-management systems and how they
evolved up to now in order to further compare them to broadcast-CMS
Primary vs. Secondary Systems
Before we do this, a last distinction needs to be made. In the content-management domain we
need to differentiate between primary- and secondary-systems.
Typical primary-systems are the applications that generate and modify essence-files. In the
office domain this could be MS-Word, MS-Excel, PDF-Writer, Star-Office a.s.o..
In the Web-Domain we could e.g. look at HTML-Editors like Front-Page, Dreamweaver but
also Image-Editors, Flash-Editors, Low-Resolution Video Editors.
In this sense in the broadcast domain, primary-systems are edit-systems, playout-systems, in-
gest- en- and transcoding systems, sound-editing-systems a.s.o.
Content Management Systems clearly are Secondary Systems and set up onto rsp. integrate
with primary systems. To be efficiently and economically used, they rely as a key factor on a
certain basic interoperability of primary systems.
Here we see a first interesting development to which we will have to come back later. Many
would consider a software formerly just used as a html-editor like Adobe’s -Dreamweaver, to
be nowadays already a small Web-CMS by itself. This development that particular primary
systems try over time to also cover the CMS-functionalities seems to be a common develop-
ment in this area.
Next we have to look at the file formats, as they are a core element of interoperability.
Just to mention them by their file extensions in the document-management domain we know
*.doc, *.xls, *.tif, *.jpg. In the web domain this would be *.html. *.gif, *.jpg, *.swf.
The related extensions for the broadcast domain would be e.g. *.omf, *.m2v, *.avi, *.mxf and
many more. The extent of file-formats being common standards, indicates rsp. facilitates the
extent of overall management systems to be operatively introduced in the respective area.
Capable content management systems in the domain of office- rsp. web-documents have
evolved as soon as a few boundary conditions were reached like:
• a critical penetration of digital documents in workflows
• large companies affected whose organization is highly based on the division of labor
• practical interoperability of primary systems
• common file formats - as at least de-facto - standards
If we now take this criteria and compare it to the broadcast area, we see significant lags, that
give a clue to the state of the art of broadcast CMS:
• even though almost all large broadcasters have implemented file-based production is-
lands already for years, the switch to a substantial vertical diffusion is just under way
and will take some more years
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 15 / 63
• almost no large broadcaster has a digital archive so far. The respective islands in this
area are mostly short news contributions, which are archived already digital and online
• specially in the production of “long forms” (feature films, documentaries in compari-
son to short news contributions) and their typical workflows, the workflow integration
is still weak
• The interoperability of systems of different vendors is not to be taken as sure, it is hard
to be achieved from case to case and can take a lot of engineering effort
• The same is the case with file formats. Not only if “exotic” features as embedded
metadata are concerned
Trends as seen at IBC 2006
The trends described here, refer to systems suited for large broadcasters who produce a broad
range of genres. In this sense such a system has to be:
• extensive (ranging from ingest to playout)
• genre-spanning (support the production workflow from news to long-forms like
documentaries)
• primary-system spanning (not just suited for a e.g. particular edit suite or storage sys-
tem)
Of the ten to twenty industry providers on the European market, that potentially fit into this
descriptions each has a – not easy to compare – more or less unique and individual arrange-
ment of functionalities. But still there are some basic common trends:
• Even though one still can see it shining through, the mere newsroom orientation of the
CMS-systems is decreasing
• Flexible modeling of various workflows starts to be shown – at least in theory
• Support of various hard- and software-systems in theory and practical references is in-
creasingly mentioned.
• Web-Service orientation as a clue to general openness is implemented more often then
last year
• The integration and support of mxf as an common file-format also shows up signifi-
cantly as well in theory as, as real reference between different primary systems
To summarize the state of the art of production broadcast content-management, we see a lot
of promising features to cover the necessary workflows, but a industry-customer is well ad-
vised to carefully look at interoperability and future-proof openness at the basic level, rather
than taking cutting-edge features into close consideration.
This recommendation is even more important in the case of the LIVE project, in which an in-
tegration of the subsystems developed in the course of the project into the broadcasting pro-
duction systems is planned without a loss of the generality of the selected approach: This is
only possible when open standards are used at the interface and protocol level to obtain the
desired interoperability.
3.1.2 Semantic Media Asset Management
The term “semantic media management” denotes an extension of “classical” media manage-
ment through the application of semantic technologies. This section discusses the potential
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 16 / 63
benefits and research questions arising from the application of semantic technologies in the
area of media asset management.
Semantic organisation of media
Semantic Organisation of media aims to overcome the issue of having to sort everything un-
der predefined categories, which are often static and mapped to folder structures. Some of
these approaches like the one described in [Kha2003] enables users to organize media items
according to mental models: the authors use semantic regions to organize media items based
on two facts: “the variety of users’ mental models toward personal media data and the effec-
tiveness of spatial organization of information.” Media can there be organized according to
“timelines, locations, events, people, etc. depending on users’ current mental models.” In
[Ben2001] the authors describe another system that allows the organisation of multimedia
items using perceptual knowledge.
Semantic search & retrieval
Search and retrieval in multimedia databases is heavily dependent on the metadata that is pro-
vided together with the multimedia items. Metadata for this purpose is either generated manu-
ally or semi-automatically through the use of tools that are described in section 3.3 of this
document. Metadata forms the basis for search in these databases as most of them only allow
keyword-based searches. The problem of keyword-based search is, that in most cases you
have to deal with ambiguities: the people assigning keywords to media items use a different
terminology as the people searching for the media. The use of ontologies to overcome the
limitations of keyword-based search has been one of the motivations of the Semantic Web as
sketched in [Ber2001]. Most semantic search solutions give the possibility to search by pre-
defined concepts that were assigned to media in the indexing phase and thus allow to query
the database through the use of them. Some semantic search engines allow searching using
concepts of ontologies which also allows to see concepts in context and through the use of
these ontologies allow the exploration of a database around topics. The advantages of using
ontologies for information retrieval are manifold as described for example in [Din2004],
[May2003], [Par2003], [Val2005], or [Bür2005].
Semantic indexing support
Large digital repositories require efficient tools for search and retrieval of digital assets. Cur-
rent media asset management systems are highly tuned to index textual content of any form to
make it searchable and, for this purpose, provide the possibility to either manually or auto-
matically annotate audiovisual content with metadata. However, this manual annotation of
content is time-consuming and in many cases subjective. Some digital asset management sys-
tems, e.g. Virage’s VS Archive™1, allow to index audiovisual content by extracting key-
frames and by applying speech recognition modules to transcribe speech to text which is later
full text indexed. However this type of indexing does not meet all requirements: Keyframes
only support quick human recognition and possibly allow to search by example (QBIC). On
the other hand speech recognition is still very error prune, applicable only in well trained
situations and, moreover, cannot solve the problem, that not everything that is depicted in a
video clip is in any way directly referred to in the audio track. Search by the semantic features
of the content is a common requirement in media asset management systems: This require-
ment can only be met by a semantic indexing process with the ability to identify the semantic
1 Virage: http://www.virage.com/ - last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 17 / 63
features of the content. There are many approaches to this problem and the reader is referred
to a selection of solutions in the literature: [Nap2004], [Bar2001], [Chan1998], [Ada2003],
[Leo2002] and [Sno2006]. Section 3.3 also includes references to these topics.
3.2 Recommender Systems
Recommender systems are a field of research that is expanding rapidly and are quickly be-
coming more and more diversified. A lot of different techniques are being developed in dif-
ferent research groups. The two most widely used approaches are the collaborative based
recommender systems and content based recommender systems. The techniques that are rele-
vant or that can be used in the LIVE project fall into three categories. Collaborative recom-
mender techniques can be mostly used in the off-line recommendation phase and could
therefore be used in a situation where a professional user (i.e. the “LIVE video conductor”) is
preparing a re-run show. The content-based recommender system can be used in a wide vari-
ety of situations such as archive recommender systems, consumer recommender systems and
live event recommender systems. The final approach – hybrid recommender system presents a
way of integrating these two approaches into a common working system. [Kob2001] and
[Han2001] provide a good overview of recommender systems.
3.2.1 Collaborative Recommender Systems
The goal of a collaborative recommender system is to generate plausible recommendations for
the user based upon his ratings and those made by his nearest neighbors (people who have
similar taste).
The first stage of collaborative recommender system is selecting nearest neighbors. In order to
find user’s nearest neighbors the system must run a series of algorithms that compare the
main user and the compared user. After the system finds the required number of neighbors, it
can calculate recommendations for the current user. This approach gives good results, but also
requires a significant amount of processing time. Alternative solution is to find clusters of us-
ers, which should provide the system with less accurate list of nearest neighbors, but also sig-
nificantly reduces the amount of processing time required.
When designing a collaborative system the required input is very simple. All that is required
is user’s identification (unique identification key for example) and a database containing all
items and their ratings provided by users that are registered in the system. When the system
processes this data and generates the required recommendations it can produce two types of
output. It can directly recommend items to the user by sending him the recommended item’s
id (crid) or it can store this information into a database structure thus enabling access at a later
time and its reuse by the hybrid recommender.
When using collaborative recommendation one of the main advantages is that the system does
not need to use any content descriptors and is therefore platform independent. Because of this
it is also possible to use the system in an environment containing different content types, for
example movies, documents and images. This makes the system much more flexible and pre-
sents the user with a wider choice of recommended items. The downside of a collaborative
system is that it is based on searching for people with similar taste and therefore does not
work very well until there are enough users in the database. Integrating new items into the da-
tabase is another problem because the system processes only those items that have received
ratings from users, so until the item has been rated it is invisible to the system.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 18 / 63
Figure 2 presents the architecture of a collaborative system.
Figure 2: Architecture of a collaborative recommender
A good overview of collaboration recommender approaches can be found in [Kur2002] and
[Bre1998].
3.2.2 Content-based Filtering
Content-based recommenders on the other hand, do not interact with profiles from other us-
ers. The principle of content based recommendation is tracking the meta-data of items that the
user rates. If a user prefers a certain type of genre this will show in positive ratings given to
that genre description. The advantage of such a system is that it is independent of other users
and that integration of new items is very simple provided the necessary meta-data is available.
The problem of using this type of recommender is that it can only work with the selected
meta-data standard and therefore cannot include items that do not have the necessary meta-
data or are of a different type. Content based recommenders are therefore optimized to work
with a specific type of content, but cannot be easily made to work with a system that includes
different content types or meta-data standards.
The classification is done based on the history of content ratings given by the consumer and
the description (metadata) of a particular content item. The required input to such a system is
a training data set, containing descriptions of content items and the corresponding ratings.
Content descriptions used in our system contain the following attributes:
• List of genres
• List of actors
• List of directors
• List of screenplay writers
• List of awards given to the movie
• Additional metadata attributes like (country of origin, production year, synopsis, etc.)
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 19 / 63
The classification of content items (videos) is based on calculations of similarity between the
content description and the user model. Similarity is calculated separately for each of the de-
scription attributes (genre, keywords, actors, director awards, etc.). The calculated attribute
similarities are then combined into a total similarity measure using the Support Vector Ma-
chines (SVM) or other classification methods.
The output of the system is a list of recommended content items (movies). If certain condi-
tions are met (usage of regression methods for classification) the output can contain an or-
dered list of content items, including the indication of content suitability (a number ranging
from 0 to 1)
The system is designed in such a way that it is possible to combine it with a collaborative re-
commender system, which should improve the recommendation efficiency.
A good overview of content based recommender systems can be found in [Bez2002],
[Zim2004], [Buc2002], [Dif2002] and [Kur2001].
3.2.3 Hybrid Recommender System
Following the optimization of both stand alone recommender systems mentioned above, the
recommender results can be further improved by combining both approaches. There are many
possible ways for constructing hybrid recommender system, for instance a parallel design
where both approaches are run at the same time and the final decision is based on both out-
puts, a serial design where approaches are run one after another or more complex hybrid de-
sign, for example a cascade design where the output of one recommender represents an input
for the second recommender system.
Figure 3: Architecture of a hybrid recommender system
From a number of above mentioned hybrid recommender systems we suggest the parallel ap-
proach. The main reason for this decision is the nature of the parallel recommender, which al-
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 20 / 63
lows for arbitrary weighting of both recommender approaches. This means that the classifica-
tion decisions for individual content items made by both recommender approaches can by ar-
bitrarily combined based on the decision parameters. These parameters include the prediction
accuracy (precision and recall), availability of content recommendations from one or the other
approach, etc. In addition, the parallel hybrid recommender is easier to implement than other
hybrid recommenders. The architecture of a parallel hybrid recommender is presented in
Figure 3.
A good overview of hybrid recommender systems can be found in [Bur2002], [Uch2002],
[Sal2006] and [Pog2004].
3.3 Metadata Generation Systems
The task of metadata generation systems with respect to the LIVE project is to add descriptive
information to audiovisual essences (i.e. archival video material and live streams) in order to
give the Intelligent Media Framework relevant information on how to process, recommend
and interrelate the material.
At the “meta level” (see section 6 for a detailed description of this concept) the information
about the material can be describing features of the content object (e.g. features related to the
creation or to the encoding of an essence). On the “subject matter level” (see section 6) the in-
formation describes, at different levels of abstraction, the “meaning” of the material. Such
meaning is usually created manually to ensure a further reuse of the audiovisual material:
Broadcasters let experienced archivists assign labels to the audiovisual material: The labels
are taken from a controlled vocabulary in order to enable efficient search and retrieval of the
stored material. The controlled vocabulary or classification scheme has been optimized by the
archive experts to cover the most important use cases for search and retrieval within their
usual workflow.
Metadata generation systems can be classified into manual annotation tools, semi-automatic
annotation tools and automatic annotation tools. Manual annotation systems help the user to
label the data by some useful classification schemes which are important for finding relevant
material later on in an achieve. Free-text annotation is used in combination with terms stem-
ming from classification schemes. The user selects those classification labels which are most
helpful for later search and retrieval.
3.3.1 Manual Annotation of Content Metadata
In many user scenarios broadcast material is manually annotated: Major news companies,
such as Reuters or Sportsticker, perform online annotation of LIVE sports events and produce
real-time feeds to consumers, e.g. via RSS. Media observation companies monitor various
broadcast feeds and annotate the live material. This information is used in many scenarios,
e.g. in order to generate edit lists to cut out commercials, to validate whether a spot has been
broadcasted correctly, or, as a broadcaster, to facilitate reuse of the AV material in the future.
The following list highlights a few annotation tools in the academic area:
• VideoAnnEx from IBM is based on MPEG-7, supports only temporal labelling and
contains automatic temporal segmentation of the video content
(http://www.research.ibm.com/VideoAnnEx/ - last accessed: 10.11.2006)
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 21 / 63
• MuViNo from the University of Klagenfurt which is a Component of ViTooKi - The
Video ToolKit (see http://vitooki.sourceforge.net/components/muvino/code/in-
dex.html - last accessed: 10.11.2006). The tool supports temporal labelling, supports
Linux, Windows and is a C++ open source project.
• Anvil from the DFKI GmbH is a video annotation research tool in Java.
http://www2.dfki.de/~kipp/anvil/ - last accessed: 10.11.2006
Moreover, special tools for audio and image annotations exist as well, e.g.:
• The Transcriber (http://trans.sourceforge.net/en/presentation.php - last accessed:
10.11.2006) is a open-source audio annotation tool written in * Tcl/Tk, which runs
under Linux und Windows. The tool allows to generate a transcription of the audio
track and to enter and modify tracks of metadata streams which may encode arbitrary
classifications (speaker name, genre, silence, music, jingle, etc.). Metadata are ex-
ported as an XML-document.
• The M-OntoMat-Annotizer from the Project acemedia
(http://annotation.semanticweb.org/ontomat/index.html - last accessed: 10.11.2006) is
a spatial annotation tool for images using ontologies. It is used in the IST-BOEMIE
project. It uses a predefined ontology and enables the user to label the metadata ac-
cordingly to the controlled vocabulary.
Besides these rather academic annotation tools, there is a variety of special purpose software
products which help customer to annotate multimedia data. Let us mention the three important
approaches in the area of consumer annotation (social tagging):
• PICASA (http://picasa.google.com - last accessed: 22.11.2006) is a free digital photo
management tool. It organizes digital photos and offers algorithms for efficient search
and retrieval. It supports free text annotation as well as metadata according to the
IPTC standard. It helps the user to produce slide shows, photo books and web pages
by offering additional tools for image manipulation, image enhancement and layout as
well as upload functionality to photo finishers and online communities in order to pub-
lish the results. Similar tools exist from photo finishers, such as Fujifilm, in order to
ensure simple and efficient production of photo albums or printed third-party products
such as personalized calendars, mugs, t-shirts, etc.
• Flickr (http://flickr.com - last accessed: 22.11.2006) is a portal for sharing digital pho-
tos on the internet. Flickr supports an internet community by providing infrastructure
in combination with web-based software technology for image search and retrieval.
Besides free textual annotation, it supports also spatial annotation of images: the user
is able to associate arbitrary image regions with free text annotations.
• YouTube (http://www.youtube.com - last accessed 23.11.2006) is an Internet commu-
nity portal for sharing self-made videos. Categories such as “Arts and Animation”,
Sports, etc. must be manually entered before the upload of audiovisual content. The
portal provides search and retrieval on a clip-basis, in which user-preferences, e.g. fre-
quency of viewing, are directly used to generate recommendations.
The reader shall be referred to a recent workshop report containing a comparison of multimo-
dal annotation tools ([Roh2006]). Moreover, the upcoming deliverable [LIVE D5.2] “Report
on LIVE Human annotation” will contain more information on annotation tools and details of
the manual annotation tool which is developed and used in LIVE’s workpackage 5.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 22 / 63
3.3.2 Automatic Annotation of Content Metadata
The aim of automatic and semiautomatic tools for generating content metadata is to extract as
much useful information from the essence (AV-content) as possible. Various algorithms from
computer vision, pattern recognition, speech recognition, spoken document retrieval and sig-
nal processing are applied to solve this difficult task. Despite significant progress over the last
decades, only a few of the classification tags users are interested in can be defined automati-
cally, e.g. whether a scene is currently “funny” or “ironic” is far too complicated. On the other
hand, hard cuts generated by editing or sapping can be detected automatically. This is usually
called “semantic gap” since computers cannot “understand” essences in the way humans do.
Algorithms with acceptable performance already exist for the following tasks: temporal video
segmentation, transition classification into cut and dissolves, face detection, camera estima-
tion, tracking of objects, audio segmentation, speech-nonspeech detection, melody detection
and audiovisual fingerprinting. For instance, temporal video segmentation and face detection
has found already its way into the acquisition hardware: Camera vendors use face detection in
photo cameras for illumination enhancement and temporal video segmentation is available ei-
ther on the hardware level in video cameras or encoder cards or provided as software (based
either on different SMPTC time stamps of the signal or simple image processing algorithms).
For instance, the video annotation tools “VideoAnnEx” and “MuViNo” mentioned above are
both able to perform hard cut detection on their own.
The state-of-the-art in video indexing and retrieval is assessed on a regular basis by the TREC
video retrieval evaluation (TRECVID2). However, any system for multimedia analysis and
recognition becomes relevant if it is able to produce useful information which describes the
essence. Hence, this field is still a very active field of research.
Web sites such as Flickr, Google (http://images.google.com - last accessed: 10.11.2006), Riya
(http://www.riya.com/ - last accessed: 10.11.2006) or Like (http://www.like.com/ - last ac-
cessed: 10.11.2006) have recently begun to apply algorithms for automatic extraction of con-
tent metadata (e.g. shapes, colour or texture features according to MPEG-7). They combine
similarity-based search and retrieval technology based on low-level features with high-level
concepts, in which the latter disambiguates the search by limiting it to smaller and well-
defined domains. Moreover, some sites already use high level pattern recognition technology
such as face detection and identification algorithms to help users to train and to identify hu-
man faces on digital pictures (e.g. Riya).
Besides these rather community-driven solutions, there are some commercial products which
are able to perform automatic analysis of audiovisual content, e.g. “Media Archive” from
Blue Order, “VideoLogger” from Virage, the “Media Mining Indexer” from Saillabs,
“Video/AudioIngest” from Visual Century or “Stream Sage”, to name only a few of them.
The upcoming deliverables [LIVE D5.3] “Description of low- and mid-level offline metadata
Extraction from sports video” and [LIVE D5.4] “Description of low-level compressed domain
online metadata extraction from sports video” will cover in detail which algorithms are used
in the LIVE project for real-time and offline metadata generation.
2 TRECVID: http://www-nlpir.nist.gov/projects/trecvid/ - last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 23 / 63
3.4 Video Conducting Systems
Within the context of LIVE the term “Video Conducting System” comprises a software sys-
tem providing the functions needed by professional users to stage live events. A special focus
is set on so called “staging tools”: These tools provide the user interface for the different tasks
and responsibilities of the video conducting team according to a staging concept. The staging
concepts and associated content models are developed in LIVE’s workpackage 4 and are de-
scribed in the corresponding deliverables: [LIVE D4.1] describes the artistic research road-
map for the development of content models and staging concepts. [LIVE D4.2] describes the
practical workflow and contains first suggestions for staging tools and [LIVE D4.3] presents
the scientific reasoning of our content research work. Since detailed information on staging
concepts and staging tools is provided in these deliverables, we restrict this document to a
general overview.
Software OSC FFT audio/
analysis
Formats Built-in
Effects
Free-
Frame
Special Layers SDK Platform
FlowMotion N Y
(advanced)
QT, AVI,
SWF,
Director
>100 Y fixed layout, scratch, Mask layers,
scratch pad can be controlled by MIDI
or linked to any parameter
5 Y PC, Mac
Livid Union N Y (beat) QT >100 N fixed layout, clip sequencer, scratch,
3D effects
4 Y PC, Mac
Modul8 Y Y all QT,
SWF
basic N modular interface, 3D objects,
advanced pre-visualization panel, loop
mode type: normal, loop or ping-pong
10 Python Mac
VVVV Y Y AVI,
.MPEG
>100 Y patch-based interface (MAX/MSP, PD-
like), 3D positioning
unlimited N PC
ArKaos VJ N Y AVI,
DIVX,
MPEG,
ASF,
WMV
>60 Y fixed layout, with drag & drop and key-
punch functionality at all levels, multi-
screen, DMX
unlimited N PC, Mac
MXWendler
2.0
Y Y AVI,QT,
SWF
>100 N completely hardware based realtime
compositing suite, based on OpenGL
and exclusively uses pixelshader
technology for transformations,
layering, wiping, keystoning,
softedges, HD DV, feedback loop
8 N PC
EyesWeb Y Y AVI,
MPEG
>10 Y patch interface, not specifically VJ
software, advanced connectivity
unlimited Y PC
Pilgrim N Y QT,AVI,
MPEG,
SWF,ASF
>100 N 3D effects, editor, Time slider: total
control of time. Go back in time or go
back to the future. For video, text and
3D Layers
8 N PC
Isadora Y Y QT >40 Y/Mac
only
patch interface, 3D effects, serial out 6 C, C++ Mac, PC
(beta)
Resolume N 18 band AVI,
MPEG
>60 Y fixed layout, easy interface, skratch,
stream video over network, multi-
screen
3 N PC
VJamm N Y MPEG,
AVI
basic Y fixed layout, individual clip control
(size, shape, soft edge luma key,
opacity, position, direction
and scratch)
16 N PC
GridPro N Mac only QT, SWF >70 N fixed grid layout (1 layer interface),
HD DV
unlimited N PC, Mac
Table 1: Summary of features for prominent VJ tools [LIVE D4.3]
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 24 / 63
Digital Content Research
[LIVE D4.1] provides a state of the art report on digital content research and starts with a
presentation of “Video Jockeys (VJs)”: The work of VJs is in so far closely related to the role
of a video conductor as they create live visual performances by composing different video
clips to a coherent show. The deliverable gives information about design principles for video
composing, such as the “proactive moderator driven navigation” or possible “temptations for
reactive navigation” which is driven by audiovisual patterns (picture in picture, inserts, etc.)
Tools and Techniques for Live Staging
[LIVE D4.3] (“Methods, design guidelines, workflows for online staging”) describes several
techniques a video conductor can use to cope with the challenges of live staging (e.g. the use
of loops, “AV Ballets” or scratching to close small gaps between two clips and/or streams).
Moreover, the deliverable presents a comparison of different video editing tools typically
used by video jockeys. The Table 1 from [LIVE D4.3] contains a survey of VJ tools, which is
cited for a quick reference below:
3.5 Interface Technologies
The focus of this section are commonly used technologies allowing separate software compo-
nents to communicate via a programmatic mechanism called “interfaces”: Interfaces define
the communication boundary between two software components. They generally constitute an
abstraction that a component provides of itself to the outside, thereby separating the methods
of external communication from internal operation. This allows a component to be internally
modified without affecting the way other components interact with the modified component.
Moreover, interfaces can provide multiple abstractions of one component. Interfaces allow to
cross program language boundaries and potentially allow interacting components to be im-
plemented in different program languages. One of the drawbacks of the use of interface tech-
nologies is the additional overhead, which is depending on the higher abstraction level,
(remote) method invocation and service lookup of the interfaces.
This section focuses on those technologies which support remote communication and are
likely to be applicable in the context of the LIVE project. Further Information is available in
[Tan2001] and [New2004a].
3.5.1 Remote Procedure Invocation
The Remote Procedure Invocation is usually associated to the client-server model of distrib-
uted computing. An interface is initiated by the caller (client) sending a request message to a
remote system (the server) to execute a certain procedure using arguments supplied. A result
message is returned to the caller. Remote procedure invocation technologies usually require
both peers be on-line with network connectivity during the invocation.
In this section we will introduce most relevant technologies that involve remote procedure in-
vocation.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 25 / 63
RPC
Remote Procedure Calls (RPC) have been used since 70s. RPC is based on extending the no-
tion of conventional, or local procedure calling, so that the called procedure need not exist in
the same address space as the calling procedure. The two processes may be on the same sys-
tem, or they may be on different systems with a network connecting them.
There are several RPC solutions for different platforms, some of them incompatibles.
The most common are:
• SUN-RPC (also called ONC-RPC), developed by SUN Microsystems and available in
most of UNIX OSs
• DCE-RPC developed by the Open Software Foundation
• MSRPC, developed by Microsoft, based on DCE-RPC. Used by Microsoft DCOM
RPC solutions are being falling into disuse in favor of other technologies that permit a greater
abstraction level
CORBA
CORBA stands for “Common Object Request Broker Architecture” and is an open architec-
ture and infrastructure industry standard created and supported by the OMG (Object Man-
agement Group). CORBA is only the specification, a CORBA implementations are known as
an ORB (or Object Request Broker).
CORBA defines APIs, communication protocol, and object/service information models to en-
able applications to interoperate. An important characteristic of this interface solution is its
independency from hardware/software platform and programming language, being available
in most of them. CORBA is intended for Object Oriented software following the Distributed
Object Model, providing platform and location transparency for sharing well-defined objects
across a distributed computing platform.
CORBA defines GIOP, which is an abstract and network independent protocol, for communi-
cation between peers. The implementation of GIOP protocol for the TCP/IP is denominated
IIOP.
Java RMI (Java Remote Method Invocation)
Java RMI, is the Java API for performing remote procedure invocation. Initially was oriented
towards intercommunication between Java Virtual Machines, interoperability with CORBA
was added later. Java RMI usually is used directly over TCP connection, but can be also used
over other protocols such as HTTP.
There are two different protocols:
• Java Remote Method Protocol (JRMP), is a Java native protocol, using this protocol
communication could only occur between Java Virtual Machines.
• RMI over IIOP (RMI-IIOP) , which permits interoperability with CORBA
Web-Services
The term “Web service” refers to any service provided machine to machine using web proto-
cols, we will restrict here to the W3C definition which restrict to SOAP/WSDL binding: “A
Web service is a software system designed to support interoperable machine-to-machine in-
teraction over a network. It has an interface described in a machine-processable format (spe-
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 26 / 63
cifically WSDL). Other systems interact with the Web service in a manner prescribed by its
description using SOAP messages, typically conveyed using HTTP with an XML serialization
in conjunction with other Web-related standards” [Haas2004].
With this definition Web-Services allow the invocation of remote methods (procedures) using
two different XML technologies: SOAP and WSDL.
WSDL describes Web services interface by defining the messages that are exchanged be-
tween the requester and provider agents. The messages themselves are described abstractly
and then bound to a concrete network protocol and message format, usually HTTP, although
other protocols are also supported as SMTP, JMS, etc. SOAP is used to package and ex-
change the XML messages with the requests and responses.
SOAP/WSDL themselves don’t provide solutions for session management, security, authenti-
cation, transactions and other models of communication (as publish/subscribe, asynchronous
communication…), however several standards have appeared that provide them. To cite some
of them:
• WS-Security, from OASIS, provides enhancements to SOAP messages for integrity,
confidentiality and authentication purposes. Additionally WS-I Basic Security
Profile provides guidance on the use of WS-Security and the User Name and X. 509
security token formats.
• WS-Business Activity, WS-Atomic Transaction and WS-Coordination from BEA-
Systems and IBM specify different levels of transaction and coordination,
• WS-Notification, from OASIS, defines an approach to notification using a topic-
based publish/subscribe pattern.
• WS-Addressing, from W3C, provides transport-neutral mechanisms to address Web
services and messages allowing secure end-to-end endpoint identification in messages.
• WS-Eventing, from BEA Systems, Computer Associates, IBM, Microsoft, Sun
Microsystems and TIBCO Software defines a baseline set of operations that allow
Web services to provide asynchronous notifications to interested parties.
Microsoft .NET Remoting
.NET Remoting is the solution for communication within Microsoft .NET framework, being a
proprietary solution. It is the successor of Microsoft Distributed COM (DCOM).
It is a generic system for different .NET applications to communicate. .NET is based on ob-
jects, that are exposed to remote processes to allow interprocess communication.
The most important characteristics are:
• Requires the clients to be built using .NET, which allows the developers to use
different programming languages (any available for .NET), but it is restricted to
Microsoft Windows platforms.
• .NET Remoting eliminates the difficulties of DCOM by supporting different
transport protocol formats and communication protocols. This allows .NET
Remoting to be adaptable to the network environment in which it is being used.
• It supports state management options.
• Applications can use binary encoding where performance is critical, or XML
encoding where interoperability with other remoting frameworks is essential.
• All XML encoding uses the SOAP protocol in transporting messages
• Allows objects to be passed by value or by reference.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 27 / 63
.NET Remoting will be superseded by Windows Communication Foundation (WCF), which
is expected to be released in 2007. WCF will allow, among other things, interoperability with
Web-services.
3.5.2 Message Oriented Solutions
Opposed to a request/response metaphor commonly provided by all the previously explained
solutions, message oriented solutions generally rely on asynchronous message-passing, allow-
ing other communication scenarios as publish and subscribe, multicast, broadcast, asynchro-
nous and synchronous messaging.
Message oriented solutions often rely on the existence of a message oriented middleware.
Most message-oriented middleware (MOM) systems depend on a message queue system, al-
though some implementations rely on broadcast or on multicast messaging systems. Charac-
teristics usually available in MOMs are the provision of guaranteed message delivery,
security, and priority-based messaging.
The most widely used technologies associated to message oriented solutions are Microsoft
Message Queuing and Java Message Service.
Microsoft Message Queue
The Microsoft Message Queuing (MSMQ) is both a technology and an implementation, re-
stricted to Microsoft Windows platform.
Some of the key characteristics of the MSMQ are:
• It allows sending and receiving messages over http protocol.
• Send messages to multiple destinations
• It integrates with the security features of the Windows operating system through the
use of access control, auditing, encryption, and authentication, and using security
features such as the Kerberos security protocol. It can also use SSL for authentication
over HTTP/HTTPS messaging.
JMS Java Message Service
The Java Message Service (JMS) API is a Java Message Oriented Middleware (MOM) API
for sending messages between two or more clients. JMS is a specification developed under
the Java Community Process as JSR 914. It allows application components based on the Java
2 Platform, Enterprise Edition (J2EE) to create, send, receive, and read messages. It enables
distributed communication that is loosely coupled, reliable, and asynchronous.
JMS supports two models:
• point to point
• publish and subscribe
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 28 / 63
4 Related Standards
The Intelligent Media Framework in its role as an integrating middleware for different knowl-
edge sources within the LIVE system is highly dependent on standardization. This applies to
both, interfaces with external systems and to internal communication with the metadata gen-
eration system, the video conducting system and the recommender system.
This section provides an overview of standards relevant for the Intelligent Media Framework
and investigated the following categories of standards:
• Media and Broadcasting Domain: This domain is of importance because it includes
state of the art content models and interface definitions to deep archives, digital asset
management- and playout systems.
• Semantic Web and Ontology Engineering Domain: This domain defines base tech-
nologies and knowledge models for the definition of the Intelligent Media Assets
(IMA).
• The journalism and news domain do already heavily use www infrastructure in there
daily work. Therefore there are widely used standards available for this domain. These
standards are very important for the IMF and the LIVE System in general because
dealing with live events conditionally means dealing news systems.
4.1 Standards Related to the Broadcasting Domain and
Media
This section gives an overview about standards to describe assets in the broadcasting domain.
In addition some of the standards also define some workflows and/or interfaces to software
components like playout server or video servers.
As pointed out in section 3.1.1 (“Media Asset Management (MAM)”), the usage of media as-
sets within the broadcasting domain includes a lot of different user scenarios leading to differ-
ent requirements based on the way content is stored (long term archive, production archive,
etc.) and the way content is used (preproduction, playout, content exchange between broad-
casting companies, etc.). Therefore it is no surprise that there are a lot of different standards
used to describe content within this domain.
The different standards of the Moving Picture Expert Group (MPEG) are well known but also
other organisations like the European Telecommunications Standards Institute (ETSI), the
Society of Motion Picture and Television Engineers (SMPTE), the European Broadcast Union
(EBU) and the International Standardization Organisation (ISO) do provide metadata schemas
and controlled vocabulary used within this domain.
Furthermore, vendors and their products as well as running systems used in the broadcasting
domain often use proprietary descriptions and encodings for describing media assets. The fol-
lowing section does not claim provide a detailed description to the standards, it rather pro-
vides references to relevant background material. A good summary of the standards is
available in [Jon2003].
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 29 / 63
Dublin Core
The Dublin Core Metadata Initiative3 (DCMI) develops and entertains a universal set of de-
scriptors, the DCMI Metadata terms. It consists of elements, element refinements, encoding
schemes, and vocabulary terms. The basic metadata exchange scheme is a list of 15 metadata
fields (Title, Subject, Description, Creator, Publisher, Contributor, Date, Type, Format, Iden-
tifier, Source, Language, Relation, Coverage, Rights). The element refinements may detail the
basic elements by adding specific qualifiers (e.g. Created, Valid, Available refine the element
Date).
The DCMI Metadata Terms [DCMI2006] are mainly used to exchange (descriptive) metadata
on the web. It may fit into the audiovisual production environment if combined with standards
for broadcast production and distribution. The DCMI Metadata Terms are a recommendation
of the DCMI. Legacy documents covering subsets of the definition are in the public domain,
including the IETF RFC 2413, the CEN Workshop Agreement CWA 13874, NISO Standard
Z39.85-2001, ISO Standard 15836-2003, and a variety of national standards that include some
or all of the Dublin Core Metadata Set.
MPEG-7
MPEG-7 is a metadata standard developed by the Motion Pictures Expert Group. Unlike other
MPEG standards, MPEG-7 does not say anything about the coding of essences (audiovisual
data) but deals only with the description of the essence. For each entity to be described, a de-
scriptor is defined. Descriptors are combined to description schemes which themselves can be
part of more complex description schemes [Mar2002]. Those are formed by the Description
Definition Language (DDL) which is based on XML Schema. The MPEG-7 description
schemes are divided into a visual part, an audio part and the semantically rich multimedia de-
scription scheme (MDS). MPEG-7 descriptions can be coded using XML or in binary format
called BiM (Binary format for MPEG-7). The latter may be integrated in the transport streams
of other MPEG formats (i.e. MPEG-2, MPEG-4).
The MPEG-7 standard consists of 11 parts. In 2002, part 1 to 6 where published and since
then several amendments to these parts were made. The newest parts are 9: Profiles, 10:
Schema Definitions, and 11: Profile Schemas which were published in 2005.
Information on current work can be found at the according page of the homepage of the Mov-
ing Picture Experts Group4. The MPEG-7 standard documents are published by the Interna-
tional Standardization Organisation (ISO).5
MPEG-21
MPEG-21 [Bor2002] aims at defining a framework for multimedia delivery and consumption
which supports a variety of businesses engaged in the trading of digital objects. The frame-
work offers to the users transparent and interoperable consumption and delivery of rich mul-
timedia content. In standardising the set of technologies that comprise the MPEG-21 standard,
3 DCMI Homepage: http://www.dublincore.org - last accessed: 10.11.2006
4 MPEG7 section of the official homepage of the Moving Picture Experts Group:
http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-7 - last accessed:
10.11.2006
5 International Standardization Organisation (ISO): http://www.iso.org - last accessed:
10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 30 / 63
MPEG has built on its previous coding and metadata standards (MPEG-1, -2, -4 and -7) such
that it links these together to produce a protectable universal package for collecting, relating,
referencing and structuring multimedia content for the consumption by users.
The standard defines the following key areas which are the main components for building up
an enhanced multimedia framework:
• Digital Item Declaration (DID): DID describes a set of abstract terms and concepts to
form a useful model for defining Digital Items. The DID Model defines digital items,
containers, fragments or complete resources, assertions, statements & annotations on
digital items.
• Digital Item Identification and Description (DII): The DII deals with unique identifi-
cation of complete or partial Digital Items by encapsulating Uniform Resource Identi-
fiers into the Identification DS. It also enables the identification of Digital Items via a
Registry Authority.
• Intellectual Property Management and Protection (IPMP): It deals with management
and protection of intellectual property within MPEG-21.
• Rights Expression Language (REL): REL helps declare rights and permissions using
the terms as defined in the Rights Data Dictionary. An MPEG REL grant consists of:
the principal to whom the grant is issued; the right that the grant specifies; the re-
source to which the right in the grant applies and the condition that must be met before
the right can be exercised.
• Rights Data Dictionary (RDD): The Rights Data Dictionary (RDD) comprises a set of
uniquely identified Terms to support the REL. RDD is designed to support mapping
and transformation of metadata from the terminology of one namespace into that of
another namespace.
• Digital Item Adaptation (DIA): It enables adaptation of digital content to preserve
quality of user experience taking care of user, terminal or network characteristics.
• Reference Software: It deals with the architecture for processing Digital Items.
• File Format: MPEG-21 Digital Item consists of the content of multiple formats, both
textual (XML) and binary (still images).
• Digital Item Processing (DIP): DIP was motivated by the need to make DIDs active as
these are just declarative and provide no usage instructions. DIP should improve the
processing of ‘static’ Digital Items by providing tools that allow users to add function-
ality to a DID. DIP specifies Digital Item Methods (DIM) using the Digital Item
Method Language (DIML). One possible use of DIP is to provide a Digital Item with a
set of methods that can are visible to users and could be performed on the DI
TV-Anytime
The TV-Anytime Forum6 develops specifications [TVA2002] for audiovisual services based
on digital storage in consumer electronic platforms. Digital content shall be searched, se-
lected, and located and acquired location (i.e. television, internet) and time (i.e. scheduled or
on demand) independent. TV Anytime enables content descriptions with a number of attrib-
utes including programme genre, title, synopsis, actors, production country, etc. In addition
the TV Anytime’s content description schemes enable descriptions of the type of mood for
which some TV programme is suitable. The TV-Anytime metadata schema is based on
MPEG-7. It uses XML to encode the metadata and the MPEG-7 Description Definition Lan-
6 TV-Anytime Forum: http://www.tv-anytime.org/ - last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 31 / 63
guage to describe the metadata structure. In addition, TV-Anytime uses several MPEG-7 data
types and MPEG-7 Classification Schemes.
The application domain of this standard is restricted to professional broadcast delivery in the
context of interactive television (iTV). The current version of the standard was published by
the European Telecommunications Standards Institute (ETSI)7 in January 2006 [ETSI2006]
[ETSI2006a].
P/Meta
P/Meta [Ric2002] is a working group of the European Broadcast Union (EBU)8. It developed
a schema for the exchange of information on program material between organisations (B2B).
It defines attributes, attribute types, sets of attributes and sets of sets. One focus of the attrib-
utes’ development was machine readability to enable to exchange of information between sys-
tems (S2S). The P/Meta standard is finished and published by the EBU.
SMPTE Metadata Dictionary
The SMPTE9 Metadata Dictionary is developed by the Society of Motion Pictures and Televi-
sion Engineers. It is a dynamic collection of metadata items and their definition. It should
serve as a public reference for all media descriptors that are used in the broadcast production
chain. Its aim is to overcome interoperability constrains between network devices for digital
broadcast production caused by incompatible formats for audio and video. All items are
uniquely identifiable by their so-called Universal Labels (UL) which is a 16 byte string, de-
fined in SMPTE 298M. Some items are nodes that combine the leafs (i.e. the single metadata
entities) of the dictionary to classes and sub-classes.
The SMPTE Metadata Dictionary is designed to be used in a professional broadcast produc-
tion environment. The current version of the standard can be downloaded in form of a Micro-
soft Excel Sheet at http://www.smpte-ra.org/mdd/index.html or browsed online at
http://www.smpte-ra.org/diffuser/mg-diffuser.php. The SMPTE Metadata Dictionary is de-
signed as a dynamic list that will constantly be enhanced.
MXF
The Material Exchange Format [Bru2002] was developed by a group of specialists that are
part of the Professional-MPEG Forum10 and is standardised by the SMPTE. MXF is not a
metadata standard but a file format specification that defines a wrapper for both essence and
time synchronized metadata within one file. Its aim is to improve file-based interoperability
between servers, workstations and other content creation devices. The built-in metadata ele-
ments are fully taken from the SMPTE Metadata Dictionary. Additional descriptive metadata
can be added to MXF files. Part of the standard is the Descriptive Metadata Scheme Part 1
(DMS-1) that defines a set of common descriptive metadata items derived from typical sce-
narios within the broadcast production chain. Other descriptive metadata schemas can be used
within MXF. The coding of MXF files is based on the Key Length Value format (KLV), de-
7 European Telecommunications Standards Institute (ETSI) http://www.etsi.org
8 European Broadcast Union (EBU) http://www.ebu.ch/
9 Socienty of Motion Picture and Television Engineers (SMPTE) http://www.smpte.org and
the SMPTE Registration Authority; http://www.smpte-ra.org - last accessed: 10.11.2006
10 The Professional MPEG Forum http://www.pro-mpeg.org - last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 32 / 63
fined in the Data Encoding Protocol by the SMPTE (SMPTE 336M). Unique Material Identi-
fier (UMID) is used to uniquely label the essence within an MXF file.
The core standard document SMPTE 377M has been published in 2004 in is currently under
revision. Current states of the standard documents are listed and frequently updated by the
IRT at http://www.irt.de/IRT/mxf/information/specification/index.html - last accessed:
10.11.2006
UMID (Unique Material Identifier)
The Unique Material Identifier is a standardized way to identify audio-visual material on dif-
ferent levels of granularity (i.e. single video frame to whole program item). The standard is
defined by the SMPTE. The Basic-UMID consists of 32 bytes which contains a Universal La-
bel (UL) of 16 byte, identifying this byte string as an UMID. The remaining 16 bytes are used
to form a globally unique number. The Extended-UMID allows to add information about
where, when and by whom the material was created. UMID is used to uniquely identify mate-
rial within media asset management systems and to associate any essence item with its exter-
nal metadata. UMID might be transported using SMPTE 291M Ancillary Data packets of the
Serial Digital Interface (SDI). UMID is also part of the MXF specification where it is a built-
in part of the structural metadata.
Video Disc Control Protocol (VDCP)
Video Disk Control Protocol (VDCP)11 is a communications protocol used to control hard
disk video servers as commonly used in the broadcasting domain. VDCP is an industry stan-
dard originally developed by Louth Automation. VDCP is supported by most of the providers
of video and playout servers in the Broadcasting domain and therefore highly relevant for in-
tegrating LIVE Components with production environments of today.
The VDCP uses a tightly coupled master-slave methodology. The controlling device takes the
initiative in communications between the controlling device (e.g. the LIVE Staging System)
and the controlled device (video disk, playout system). VDCP conforms to the Open Systems
Interconnection (OSI) reference model. VDCP is a serial communications protocol and con-
firms to EIA RS-422A. It is therefore designed based on an full dublex communication chan-
nel with a transfer rate of 38,4 kBit/s. The different commands defined by the protocol are
realised as bit sequences. Note that the VDCP is only an automation and control protocol and
do not transfer multi media data.
The Advanced Media Protocol (AMP) is a successor of the VDCP and adds two important
new features First AMP adds commands to navigate through a folder structure on the video
disk server. With VDCP all files must be located in a single location. Second AMP can be
used via Ethernet and RS-422 serial port. Ethernet definition is provided via DCOM and TCP.
Audiovisual Documentation Models
[Bau2005] contains an analysis of audiovisual documentation models adopted by the major
organisations participating in the Prestospace project (http://www.prestospace.org - last ac-
cessed: 10.11.2006) plus a set of standard-models. This includes the RAI documentation
11 Documentation: http://support.omneon.com/Updates/Omneon/Documentation/vdcp.pdf -
last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 33 / 63
model, the INA documentation model, FARAO (ORF), IMMIX /B&G) and the DR metadata
standard. Furthermore it includes a description of the most popular metadata schemes and in-
cludes a mapping between the different models.
4.2 Knowledge Representation Standards
Knowledge Representation Standards are needed to formalize the knowledge used by the
LIVE system. The section is split up in three subsections:
1. Knowledge representation languages define the formal language which can be used to
express computer understandable facts. Section 4.2.1 introduces the most popular
knowledge representation languages.
2. Knowledge representation models define actual models by using one ore several
knowledge representation languages. These models can be used as controlled vocabu-
lary (to annotate existing items) or to create new knowledge items (instances). Section
4.2.2 provides a general introduction to ontologies and to their classification.
3. Foundational ontologies are a special types of knowledge representation models,
which define rather general knowledge structures. These knowledge structures can be
used to derive more specific ones or to align existing models (even not semantic en-
abled one) to a knowledge framework.
As we are focussing on the relevance of knowledge representation standards with respect to
LIVE and the Intelligent Media Framework, the three subsections do not try to exhaustively
cover the state of the art in this area. They rather are supposed to create awareness for the
conceptual background of content integration by means of knowledge technologies. Detailed
information is provided in the referenced sources.
4.2.1 Knowledge Representation Languages
RDF(S) [Bri2004] and the OWL family of languages (see: http://www.w3.org/TR/owl-
features/ - last accessed: 10.11.2006) are the most popular ontology languages that are avail-
able and used today. These two languages (or language families) undergo W3C standardiza-
tion activities.
Before the development of these languages other knowledge representation languages were
used: Early standards of representation languages include for example CycL [Cyc2002],
Frame Logic (FLogic) [Kif1989] or Description logics [Baa2003]. The next era of ontology
languages included OIL [Fen2001], DAML, and the unification of the two called
DAML+OIL [Hor2001].
RDF [Man2004] is a data model that has been augmented by the ontology language RDF-S.
RDF-S offers a minimum of expressiveness and supports little reasoning, but has gained some
importance due to the availability of scalable repositories and reasoners.
Besides the W3C efforts, the activities of the WSMO Working group have yielded proposals
of new ontology languages, namely WSML [Bru2006], OWL- (“OWL minus”) [Bru2005a]
and OWL Flight [Bru2005b]. OWL- for example is a well-founded reduction of OWL that
combines efficient reasoning with a high degree of expressiveness.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 34 / 63
Surveys of the most popular ontology languages can be found in [Ant2004]. Survey of the
older ontology languages can be found here [Gom2002]. An overview of the WSML language
family is available in [Bru2006].
4.2.2 Knowledge Representation Models (Ontologies)
Up to now different knowledge representation languages were introduced. This section pro-
vides some information on which types of models one can build by using these languages.
The term “ontology” has become important in the recent years in the field of computer and in-
formation science. The first use of “ontology” in computer science occurs already in 1967, in
a work on the foundations of data modelling by Mealy (referenced in [Smi2002]). He is con-
cerned with questions which are recognizably ontological in the philosophical sense. Al-
though the concept of ontologies is very similar to the concepts used in the database
community [Smi2002] the first implementations of information science ontologies were done
in artificial intelligence by McCarthy [Car1980] and Hayes (referenced in [Smi2002]). Since
then researchers have found various new application fields for ontologies, e.g. data engineer-
ing, knowledge engineering, etc. In the mid 90s ontologies attracted attention of web engi-
neering developers (starting with the SHOE12 ontology) and have now become an
increasingly active research area.
The use of ontologies is manifold. At first very simple ontologies like glossaries und taxono-
mies can be used to define a controlled vocabulary for a community or a sector. A similar but
a little bit more demanding usage is the building of a lingua franca for information systems:
resolving the terminological and conceptual incompatibilities between databases of different
sorts and of different provenance. In this context, “An ontology is a dictionary of terms for-
mulated in a canonical syntax and with commonly accepted definitions …” [Smi2002]. More
expressive ontologies can also be used as data models and last but not least such kind of
knowledge models also support reasoning of new facts based on available information.
This wide range of different usages for ontologies demand knowledge models with very diva-
gate properties. Therefore a more precise classification of ontologies is needed. In the follow-
ing subsections different classification schemes for knowledge representation models are
presented. This classification schemes are later used to classify different available founda-
tional ontologies.
Classification Based on Purpose
Various authors describe the purposes of knowledge models. The purpose of a knowledge
representation models allows a first understanding of requirements for the ontology.
The most general approach described in [Uschold1996] divides the space of uses of ontolo-
gies into the following three categories:
1) Communication between Humans
2) Inter-Operability between computer systems
3) Systems-engineering: specification, reliability and reusability
12 Simple HTML Ontology Extensions, http://www.cs.umd.edu/projects/plus/SHOE/ – last
accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 35 / 63
Relating to the same classification schema but with the knowledge management related ap-
proach of [Mika2004] there are three layers of functionality of ontologies in applications:
Figure 4: Mika’s pyramid of increasing levels of formality [Mika2004]
Supporting communication between humans for understanding requires less formality than in-
tegration (relating) of applications. The most complex, and therefore most demanding level of
formality would be reasoning (automatically deriving new conclusions from existing knowl-
edge).
Classification Based on Generality and Scope
Guarino [Guar1998] identifies four different types of ontologies based on their level of gener-
ality (see Figure 5):
Figure 5: Types of ontologies [Guar1998]
• Top-level ontologies describe very general concepts like space, time, matter, object,
event, action, etc., which are independent of a particular problem or domain: it seems
therefore reasonable, at least in theory, to have unified top-level ontologies for large
communities of users.
• Domain ontologies and task ontologies describe, respectively, the vocabulary related
to a generic domain (like medicine, or automobiles) or a generic task or activity (like
diagnosing or selling), by specializing the terms introduced in the top-level ontology.
• Application ontologies describe concepts depending both on a particular domain and
task, which are often specializations of both the related ontologies. These concepts of-
ten correspond to roles played by domain entities while performing a certain activity,
like replaceable unit or spare component.”
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 36 / 63
Classification Based on Expressiveness
What is an Ontology?
Catalog/
ID
General
Logical
constraints
Terms/
glossary
Thesauri
“narrower
term”
relation
Formal
is-a
Frames
(properties)
Informal
is-a
Formal
instance Value Restrs.
Disjointness,
Inverse, part-
of…
Figure 6: Ontology Expressiveness Spectrum [Guin2001]
Ontologies can also be classified according to their expressiveness (see [Guin2001]):
• Controlled vocabulary: this is the simplest form and basically a list of terms
• Thesaurus: A thesaurus can be seen as a controlled vocabulary, where the terms are
connected through relations; a hierarchy can be modelled.
• Informal taxonomy: An informal taxonomy models an explicit hierarchy between con-
cepts (generalization and specialization are supported); there is no strict inheritance,
thus an instance of a subclass is not necessarily an instance of the superclass.
• Formal taxonomy: In a formal taxonomy strict inheritance is supported.
• Frames: this form is similar to classes in object oriented programming: Frames contain
a number of properties and there can be instances of frames. These properties are in-
herited by subclasses and instances (see also section 3.2.1)
• Value restrictions: In this form values of properties are or can be restricted
• General logic constraints: If a greater expressiveness is needed values can be restricted
or defined by logical constructs using values from other properties
• First-order logic constraints: This form is allowed by very expressive ontology lan-
guages. These allow more detailed relationships between terms such as disjointness,
inverse properties/relationships or part-whole relationship expressed first order logic
constraints.
4.2.3 Foundational Ontologies
As the classification based on generality and scope showed, there are top level ontologies
which define the foundation for more concrete domain and application ontologies.
In this section different foundational ontologies are presented and classified based on their
purpose and there expressiveness.
SOWA
John F. Sowa introduced the Knowledge Representation Ontology in his book Knowledge
Representation [Sow2000]. The basic categories and distinctions of this ontology have been
derived from various scientific research areas like logic, linguistics, philosophy, and artificial
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 37 / 63
intelligence. For the ontology to remain extensible Sowa does not define a fixed set of catego-
ries. Instead it provides a set of basic distinctions which ontology designers can extend by ap-
plication dependent distinctions.
A distinction is a certain characteristic of a category in Sowa’s ontology, e.g. a category may
be independent or physical. Sowa defines nine primitive distinctions; each of them describes a
certain category at the top-level of his ontology. The top level categories related by subsump-
tion are shown in the following figure.
Figure 7: Sowa’s top level lattice [Sow2000]
Sowa distinguishes between top level categories which are characterised by one property
(primitives) and those which are described by a conjunction of several properties (non-
primitives).
Sowa’s ontological categorization scheme relies on foundations established by a number of
philosophers including Plato, Aristotle, Kant, Hegel, Husserl, Whitehead, Heidegger, and (es-
pecially) Peirce. The twelve top level categories (without the primitives) for classification can
be seen as the result of 2x2x3 factoring as shown in the following table.
Physical Abstract
Continuant Occurrent Continuant Occurrent
Independent Object Process Schema Script
Relative Juncture Participation Description History
Mediating Structure Situation Reason Purpose
Table 2: Matrix style of Sowa's primitives [Sow2000]
To represent the ontology Sowa used the conceptual graph notation. This formalism is com-
prised of a graphically based language of nodes and labelled arcs, which serve as a readable,
but formal design and specification language. Sowa’s Ontology corresponds to a high level of
expressivness.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 38 / 63
Cyc
Cyc is a very large multi-contextual knowledge base and inference engine. The development
of Cyc was started at the Microelectronics and Computer Technology Corporation (MCC)
during the early 1980s and continued at Cycorp Inc.
Cyc builds a knowledge infrastructure containing assertions about everyday objects and ac-
tions in certain contexts. Commonsense substrates like “One can see people’s noses but not
their hearts” make up its foundation. These manually crafted axioms, cover a significant por-
tion of the knowledge part of Cyc. The Purpose of Cyc is serve as a bases for AI systems and
enable further knowledge collection through natural language processing and machine learn-
ing (Level 3 based on the classification scheme on purpose). The Cyc ontology roughly has
105 general concepts spanning time, space, substance, intension, contradiction, uncertainty,
belief, emotions, planning and so on [Len1995].
Cyc puts each of its assertions into one or more explicit contexts called microtheoriesm.
Every microtheory instance (context) is relatively small and solid and may have connections
with other microtheories (contexts) allowing importing assertions from them.
The Cyc knowledge infrastructure includes a knowledge base, an inference engine and natural
language processing tools.
Cyc defines its own representation language based on the first-order predicate calculus plus
ZF set theory, meta-level assertions, contexts, and modal operators [Cyc2002].
SUMO
SUMO is an upper level ontology that is freely available under the GNU Public License13.
SUMO is a summarization of different publicly available ontologies which combines all best
practice concepts to build a single ontology from them. SUMO was created at Teknowledge14
with an extensive input from the Standard Upper level Ontology working group (SUO)15. This
ontology provides definitions for general-purpose terms and acts as a foundation for more
specific domain ontologies [Nil2001]. The purpose of SUMO is to promote data interopera-
bility, information search and retrieval, automated inference, and natural language processing.
(Level 2 based on the purpose classification)
SUMO is formalized in a derived form of the Knowledge Interchange Format (KIF)16 called
SUO-KIF and periodically translated into DAML, LOOM, XML OWL, and Protégé formats.
When it is translated in a format with less expressiveness like Protégé or OWL most/some of
the axioms are lost.
SUMO contains around 4000 assertions with over 800 rules to define its 1000 concepts.
SUMO terms are mapped to all 100000 WordNet synonym sets [Nil2002]. The development
of SUMO is completed and it is being maintained.
DOLCE
DOLCE is a foundational ontology developed in the WonderWeb EU-Project and described at
[Mas2003]. The acronym DOLCE stands for Descriptive Ontology for Linguistic and Cogni-
tive Engineering. The original version of DOLCE is axiomatized full first-order logic wit mo-
13 The source file for the SUMO, http://ontology.teknowledge.com/cgi-
bin/cvsweb.cgi/SUO/Merge.txt?rev=1.60&content-type=text/x-cvsweb-markup
The License is available http://ontology.teknowledge.com/IEEE_license.htm
14 http://ontology.teknowledge.com
15 http://suo.ieee.org/
16 Knowledge Interchange Format KIF; http://logic.stanford.edu/kif/dpans.html
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 39 / 63
dality and formalized in KIF, but there is also an OWL-DL version available. Based on
DOLCE there some extensions available. DnS (Description and Situations) [Mas2004] de-
scribing Roles, Tasks, Palns and Situations or Information Objects [Mik2004].
The following figure shows the core Concepts of DOLCE in a hierarchical fashion.
Figure 8: Taxonomy of DOLCE basic categories [Mas2003]
DOLCE is based on a fundamental distinction between enduring and perduring entities, i.e.,
between what philosophers usually call continuants and occurrents. The main assumption be-
hind that is, that that objects (endurants like persons) cannot be parts of occurrences (per-
durants like events, actions …), but rather they participate in them. In addition to these two
main categories DOLCE defines Quality and Abstract. Qualities are defined as basic entities,
which can be measured like colour, size, temperature… Abstracts do not have any own quali-
ties but define regions and sets in time, space or other (abstract) dimensions.
DOLCE provides a very high level of precision. Its purpose includes reasoning (Level 3) and
the level of expressiveness is on the level of first-order logic. In comparison to other founda-
tional Ontologies DOLCE is rather small. The core ontology contains 37 Concepts 2nd 70 Re-
lations. By including all extensions and modules (DOLCE Lite Plus) includes 210 Concepts
and 325 Relations. The development of DOLCE is completed and it is being maintained.
Some of the extensions are still in development.
PROTON
The PROTON Ontology is a rather young Ontology developed in the European IST integrated
project SEKT. The Ontology contains about 300 concepts and 100 properties and structured
in four modules. PROTON is designed as light-weight ontology and defines firstly a hierarchy
on concepts with secondly relations between them. PROTON was developed on the base of
the KIMO-Ontology and is formalized in OWL-Lite. The four modules of PROTON are the
system-module, defining some concepts used by knowledge management systems for man-
agement purpose, the top module containing the top-level concepts and relations, the upper
module containing more domain related concepts and the knowledge management module,
containing concepts typical used by knowledge extraction systems. The last module is
strongly related to the working domain of the SEKT project.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 40 / 63
The top level concepts pf Proton are Abstract, Happening and Object. The foundation of these
concepts is defined by referring to the according concepts of the DOLCE Ontology. Object
stands for endurant; Happenings are perdurants and abstract stands for abstract as defined in
DOLCE. The following image shows the concept hierarchy as defined in the Top Module
Figure 9: Taxonomy of PROTON
The purpose of PROTON is to enable data interoperability, information search and retrieval
and natural language processing. (level 2 based on the purpose classification). The expres-
siveness of PROTON is on the border between light and heavy weight ontologies.
4.3 News and Journalism Domain Standards
The International Press Telecommunications Council17 (IPTC) is the main standardization
body for the content description in the news and journalism domain. The IPTC is an interna-
tional consortium of news agencies, editors and newspapers distributors. IPTC has developed
standards like the Information Interchange Model (IIM), NewsCodes (formerly the Subject
Reference System), the News Industry Text Format (NITF), NewsML or SportML. Almost all
of them are defined by using XML technologies. There objective is to represent and manage
news along their whole lifecycle, including their creation, exchange and consumption.
Most of these Standards are highly relevant in the context of LIVE. To exemplify,
NewsCodes may be useable as a controlled vocabulary to classify LIVE content. NewsML
and SportML are very interesting for the definition of standard interfaces to external Informa-
tion systems as well as LIVE internal interfaces between components.
The following subsections provide an overview about the different standards.
IPTC News Codes
The IPTC creates and maintains lists of values for certain metadata values. This universe of
NewsCodes is currently split into 28 individual sets. These sets are technical terms controlled
17 IPTC: http://www.iptc.org/ - last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 41 / 63
vocabularies (CV). Some of these sets are structured in a hierarchical fashion similar to tax-
onomies.
The set which defines the subject codes is the most important one of the IPTC news code
standard. The subject codes are designed to classify the subject of content itmes (mostly news
items). They are defined in a hierarchy with three levels. The first level defines subjects,
which provide a high level classification of content items. Subjects include categories like
“arts, culture and entertainment”, “crime, law and justice”, “disaster and accident”, “economy,
business and finance“… The second level is build by subject matters providing a description
at a more precise level. The final and most precise level is defined by subject details. Both
subject matters and subject details are assigned to a specific more general category by using
an 8-digit numbering system where the first two digits represent the subject the next three the
subject matter and the last three the subject detail.
Some examples of other CV sets part of the NewsCodes are audiocodecs, confidence, genre,
media type and MIME types, urgency, … All these sets of CV are used as values for metadata
fields defined in other IPTC related standards like NewsML or SportML.
There are some initiatives to make these vocabularies accessible to semantic web technolo-
gies. The most important issue within these efforts is the translation of the XSL/XML re-
sources to OWL/RDFS/RDS.
Within the Neptuno Project18 a RDFS schema for the IPTC NewsCodes was defined. Based
on this Schema all Terms of the CV are defined as instances.
Further Information about the IPTC News Codes is available at [Ste2005].
NewsML
NewsML is branded by IPTC as “ .. the versatile News Markup Language for global news ex-
change.”. This sentenced describes very well the aim of NewsML. Technically NewsML is
defined using XML technologies. It defines various envelopes, by which the actual news con-
tent is described.
The first envelope is called content layer and provides a uniform interface to content irre-
spective of the media type. Format, MIME-type, encoding… are examples of metadata items
defined within this envelope. The base XML-complex type for this layer is called Content-
Item.
The second envelope provides metadata about the internal structure of the news item and is
called the structure layer. This layer uses the NewsComponent as base XML-complex types.
NewsComponents define metadata and the internal structure. A NewsComponent is therefore a
container which can hold one or more sub items (NewsComponents, ContetnItems, News
Items or links to NewsItems). The metadata of NewsComponents is separated in administrative
metadata (provider, creator, source, …), descriptive metadata (language, genere, subject code,
date line date, …), Rights Metadata (copyright, usageRight) and Miscellaneous Metadata.
18 Neptuno aims to apply Semantic Web technologies to improve current state of the art for
digital news archive management and exploitation http://nets.ii.uam.es/neptuno/ - last ac-
cessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 42 / 63
The third envelope defines a management infrastructure above the other two envelopes.
This layer defines NewsItem as an identified and publishable piece of news. This is the object
that a provider will create, store, manage, reuse, link to and from other NewsItems; this is the
unit of interchange in a news environment.
Each NewsItems is identified by an UID (globally unique identifier) including the Provider-
ID, DateID, ItemID, RevisionID and a PublicIdentifier (an URN combining all the other IDs).
In addition a NewsItem provides management properties like information on the first creation
date, status, urgency, revision history, … Based on these management metadata the standard
defines some usage cases (so called management strategies) for NewsItems.
News ML uses the CV defined by the NewsCodes to restrict values of the different defined
metadata fields. This design ensures interoperability between different systems and organiza-
tions. More information about NewsML is available in the NewsML guidelines [Meu2004].
SportsML
SportsML is a markup language19, created in a project launched by IPTC – International Press
Telecommunications Council. SportsML is a cross-sport cross-language XML standard for
the interchange of sports data and statistics. SportsML supports identification and description
of a large number of sports characteristics. Highlights include:
• Scores: who won, how many goals were scored, how did the score change
• Schedules: which players/team are playing and what are his/theis opponents, when is
the match and where
• Standings: who is leading the championship or a specific tournament, who’s the clos-
est to qualifying for the championship
• Statistics: How do the players and/or teams measure up against one another in various
categories
• News: How do we combine editorial coverage of sports with all these data feeds? How
do we package metadata- and multimedia-filled articles together with sports data?
SportsML consists of a core Schema/DTD that contains a great amount of properties that de-
scribe a wide range of sports coverage. Much useful sports reporting can be done through the
core Schema/DTD. In addition, SportsML contains several "plug-in" specific-sport Sche-
mas/DTDs, which are only necessary when the publisher needs to go in-depth for a specific
sport. There are only seven sports covered in SportsML's initial release, but users can also de-
fine additional plug-ins. The core Schema/DTD provides good starting point for many sports,
and the development of many of them is already in process.
19 SportsML: http://www.sportsml.org/ - last accessed: 10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 43 / 63
5 Requirements for the Intelligent Media Framework
This section gives an overview about different aspects which have an impact on the Intelligent
Media Framework (IMF) and which are derived from the state of the art analysis. These re-
quirements act as additional - mostly technical - constraints on the choice of technology
which will be needed for the IMF.
5.1 Architectural Requirements
The Intelligent Media Framework acts as a middleware for the overall LIVE system. Figure
10 shows the basic system components which were introduced in the system architecture in
[LIVE D9.3].
The Intelligent Media Framework will integrate several components of the LIVE system:
1) The Intelligent Media Asset Information System (IMAIS) providing access to ser-
vices for the storage of media, knowledge models and metadata relevant for the live
staging process and providing services for the creation and management and delivery
of intelligent media assets. This will be the central component of the Intelligent Media
Framework.
2) The Recommender System, giving content recommendations to the user based on the
user's personal profile and on previous user feedback.
3) The Metadata Generation System, dealing with the detection, extraction and annota-
tion of knowledge from audiovisual material
4) The Video Conducting System, dealing with the real-time staging of a live event.
Figure 10: LIVE System Overview (from [LIVE D9.3])
The Intelligent Media Framework will provide services to the components of the LIVE sys-
tem. Its main role is to act as a middleware for these subsystems, passing information between
them and integrating the information coming from external information systems. With respect
to that, the IMF is intended to be an open framework, thus it needs to implement and offer
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 44 / 63
standardized interfaces and it must comply with metadata standards for the annotation tools,
the production/playout server and the recommender system.
5.2 Combined Requirements of Knowledge-, Metadata- and
Asset Management
In order to assess knowledge and content models with respect to the state of the art we need a
comparison scheme for content models. Such a comparison was done in 2002 [Schel2002],
but the focus then was more on advanced multimedia capabilities and less on knowledge rep-
resentation and meta-data. The "-" symbol means no support, "o" means partial support and
"+" means full support.
Table 3: Comparison of the respective strengths and weaknesses of web-based multimedia
document modelling approaches (from [Schel2002])
The requirements for advanced media systems have grown significantly since 2002. The IMF
needs to deal with information which is normally held in different systems and it needs to
combine these disparate information objects into a unified view. To serve this purpose, LIVE
postulated the need for Intelligent Media Assets (IMA): IMAs act as homogeneously struc-
tured entities that are exchanged in the system between the different components. They are
capable of storing the integrated information separated into contextual facets and they hold
actionable knowledge about that information.
There are many possible relations between knowledge and content: Classical media asset
management systems rely heavily on metadata about the media assets. Within LIVE both,
content-derived annotations and knowledge descriptions will be used to enrich media objects.
Content descriptions can come from low level features extracted from the raw media material,
information available from long term archives and structured descriptions given by humans
by using the human annotation tool. Knowledge descriptions are provided by the video con-
ductor and his team and by consumers via the feedback channels.
Based on current applications and projects we have derived the following requirements for the
knowledge and content model of the IMF:
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 45 / 63
1. Defined Data Structures - Knowledge must be encoded using a formal language:
Knowledge content must be interpretable by machines, to enable the implementation
of systems supporting knowledge workers in handling content objects. Therefore, it is
a requirement to encode the knowledge in a formal language. The use of statistical
methods only (e.g. as used for page ranking) will not be sufficient in closed, corporate
environments.
2. Cross domain knowledge and metadata alignment: Some aspects of knowledge
about content are independent from the application domain. Media coding information
and rights management are the most prominent examples. Established standards of
such domains must be respected and aligned to the concept of knowledge content.
3. Make knowledge independent of content - it is a many-to-many relationship: In
traditional media asset management system knowledge is typically dependent on the
content object. As shown, this is not a valid assumption for most of the interrelations
between knowledge and content. Particularly planning activities of knowledge work-
ers first create a description of the topic of interest and later search for suitable content
objects which support the conceived story. Put simply, the relationship between
knowledge and content is many-to-many (i.e. one item of knowledge can be supported
by different media assets and one media asset can be associated with different items of
knowledge).
4. Different meta data aspects of content depending on the perspective: In a broad-
casting situation, content has a technical perspective (e.g. the compression format
used), a genre-perspective (e.g. a sports-related news item) and a knowledge perspec-
tive (e.g. a report on a new test for banned substances in sports). These elements of in-
formation may be managed by different systems and need to be re-integrated at the
level of the overall information asset.
5. Distributed, heterogeneous storage of virtually homogeneous objects: We cannot
assume that all content models take a unified view on digital content. On the contrary,
it is more likely that digital content is split into individual media assets which are then
managed individually in media asset management systems. The IMF will need to offer
the capability of assembling and managing complex, merged content structures and it
needs to manage the semantics of how the elements of the structures relate to each
other.
6. Compatibility with semantic web technology: It will be necessary for the IMF to
combine the strengths of media production and management with up-to-date knowl-
edge technologies. The standardisation efforts of the W3C cannot be ignored and
therefore, existing and emerging standards such as RDF, OWL and Semantic Web
Services will have to be taken into consideration when developing the IMF.
7. Contextualisation on the basis of user profiles: There will be a specific need to
foresee the representation of user interests and preferences in order to customise IMAs
to the end user.
8. Real-time delivery of knowledge and content, driven by the data stream: Last but
not least, fast media delivery will have to be matched by equally fast knowledge ser-
vices that ensure that up to date, contextualised information is available at the user's
home cinema at the same time as the live picture is.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 46 / 63
6 Assessment of Models for Intelligent Content
A content model is a structural definition for a “type” of object (e.g. a learning object, image,
book). It defines what components may be comprised in an object (document) or a certain part
of an object (document section). It may also contain a set of constraints on digital objects and
a set of rules for creating these objects. In a narrower sense, a content model mainly describes
the kind of content that a specific type of document may contain. The term intelligent con-
tent is a notation for semantically annotated and structured data or in other words for content
containing information with explicit semantic descriptions of its properties. In this chapter,
the term knowledge content will be used as a synonym for "intelligent content".
Intelligent Content Models, as assessed for example in [Bür2006], can be seen as a metadata
integration architecture allowing to combine knowledge coming from diverse sources and
covering different properties of specific content. An intelligent content model for the IMF is
thought to enable the integration of information coming from the participating systems like
the Metadata Generation system, the Video Conducting system or other external information
systems. The term Intelligent Media Asset (IMA) will be used to refer to content objects
which comply with the specific intelligent content model of LIVE.
In the following sections we first provide an overview of current scenarios for the use and ap-
plication of intelligent content models. Section 6.2 then outlines past and current approaches
to intelligent content models, section 6.3 presents Knowledge Content Objects (KCO) in more
detail to illustrate the potential of sophisticated, knowledge based content models and section
6.4 compares the relative strengths and weaknesses of the major knowledge and content mod-
els.
6.1 Metadata Annotation in the Web
Today, the strongest driver for determining the semantics of content is its usage in the WWW.
The typical approach to semantics of multimedia is to simply add annotations to the meta data
set. Some of them are generated manually and some of them (semi-)automatically. The source
of such annotations and their appearance and complexity is thereby manifold: Some systems
try to derive annotations just from the low-level features detected in the raw media data, other
systems try to analyze the different modalities of a video, analyse the usage context of the
media or rely on human annotation/interpretation. Recent research efforts however try to
combine low level features with background knowledge available via the Web [Dow2005] or
other knowledge sources. Some approaches allow to attach several interpretations to the con-
tent (like LastFM), others allow to attach descriptions with concepts using ontologies
[Sta2006]: An application called Rich News [Dow2005] deals with automatic annotation and
extraction of semantics from news videos: In a first step, the system extracts text from speech
and then it tries to extract the most important topics from that. With these extracted topics, the
system starts a Google search to find news stories on the web that cover the same topic(s).
Google did exactly the same in a research project [Hen2003] to enhance American TV news
with background information from the Internet: They extract important topics from the subti-
tle channel that is broadcasted along with every news show and give consumers the possibility
to access background information about these news items by displaying links to related web
pages.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 47 / 63
MediaMill20, a system that was developed at the University of Amsterdam, uses all modalities
of a video to derive the semantics of it, which is especially important when the visual content
is not reflected in the associated text like close captions or speech-to-text transcripts. There
are also different examples for attaching knowledge to content showing up on the Web: Two
of the most popular content-based services are Flickr and LastFM, both of which get their us-
ers to classify (tag) images or music files within these systems. These systems do not classify
the content according to predefined categories, but instead, taxonomies evolve from the users'
tags (folksonomies). These tags can be regarded as user-based knowledge about certain items
that on these platforms, is used to filter or recommend content to other users.
The examples above show that annotations attached to content can have a different appear-
ance (from keyword lists to ontological structures), can be at different ontological levels (de-
scribing low level properties or high level semantics) and can come from different sources
which clearly should be marked. Sometimes it is not allowed to access content but only the
generated metadata, this also has to be ensured by the content itself.
Within the LIVE system the IMF has the responsibility to combine content based knowledge,
with social based knowledge of the different users of the system. Content knowledge is pro-
vided by archives, by automatic extraction of low level features and by human annotations.
User knowledge is split up in two parts: Firstly, the consumer knowledge, which is derived
from the consumers’ feedback by the personalisation and feedback components of the LIVE
system. Secondly, knowledge which is gathered by professional users such as the video con-
ductor, within the event preparation phase and during the live production phase by interaction
with the video conducting tools. In this setting, the Intelligent Media Framework has the re-
sponsibility to combine all the different sources of knowledge based on a unified knowledge
content model which will form the basis for the Intelligent Media Assets (IMA). Such a com-
prehensive set of tasks is currently neither envisaged nor understood as a possible requirement
in the Web. However, as current activities show, there is a growing sense at least in the re-
search and standardisation communities (see e.g.[Tro2006]), of a future need for more com-
prehensive and ontologically sound multimedia models.
6.2 Approaches to Intelligent Content Models
The field of content modelling attracted interest in the hypermedia and multimedia communi-
ties in the 1990s when information systems began to handle web sites and when office and
consumer hardware became powerful enough to handle multimedia in real time.
6.2.1 Early Models - from Office Automation to Media Presentation
Autonomous Knowledge Objects
Tsichritzis et al. proposed the notion of an autonomous knowledge object [Tsi1987]: They in-
troduced an object-oriented environment targeting the area of office automation and giving
the managed objects the ability to migrate to new environments and to acquire new operations
dynamically, which means that they were intended to learn and to adapt themselves to new
circumstances and environments. While this introduced the idea of "intelligent objects" not
20 MediaMill - Semantic Video Search Engine: http://www.mediamill.nl/ - last accessed:
10.11.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 48 / 63
much attention was given at the time, to multimedia capabilities or to global networking as
the proposal was made several years before the invention of the WWW.
Multimedia Information Presentation System - MIPS
One of the early attempts at integrating multimedia, distributed systems and knowledge based
annotation was the Multimedia Information Presentation System (MIPS, 1992 - 1995),
[Mac1995] which was developed in an EU project in the third Framework Programme. MIPS
developed a distributed, knowledge based media annotation and presentation system three
years before Tim Berners-Lee published his first ideas about the semantic web.
MIPS distinguished between a presentation layer, a knowledge based middleware and a dis-
tributed database layer. The schemas of the participating databases were represented in a
knowledge based system which acted as the "global schema" and wrapper for the heterogene-
ous databases. The user interface had a query tool which enabled the user to access the het-
erogeneous distributed databases by posing queries to the knowledge base which in turn
translated the query into the proprietary languages of the databases.
At the presentation layer, the query results from the databases were cast into presentation
templates defined in HyTime. Again, it was the task of the knowledge based system to deter-
mine which of the predefined presentation templates suited the query results best.
Overall, MIPS was an example of a visionary project attacking too many unsolved problems
at the same time: it developed a semantic modelling language three years before the early
ideas of the semantic web evolved; it created an open database access layer shortly before
ODBC became common place, it used an untried research prototype of an object request bro-
ker in the absence of any web services infrastructure and it developed one of the first inter-
preters of a subset of the HyTime standard which was the precursor of XML (1995) and - to
some extent - of MPEG-7 (1996). Last but not least, it used C++ in the absence of Java and
required a dedicated C++ to Prolog interface in order communicate between the knowledge
base and the surrounding components of the system.
6.2.2 The MPEG Family of Content Models
Since 1996, the MPEG family of standards has developed three content models starting with
the metadata standard MPEG-7, the definition of digital items through MPEG-21 [Bor2002]
and currently working on MPEG-A [Cha2004] which is the attempt to bundle the growing
universe of meta data requirements so that they become manageable in actual applications.
MPEG-7 for example has standardized tools for describing different aspects of multimedia at
different levels of abstraction. These amongst others include tools to describe the semantics
(e.g. objects and events) of multimedia [Ben2002]. These tools mainly serve the purpose of
annotating what can be seen on the image or video but do not offer means for adding highly
structured background information. Furthermore, the possibilities of MPEG-7 are not formal
enough to allow reasoning in order to deduce new knowledge from known facts. This lack of
semantic formality should not come as a surprise: the model was conceived of in the mid-90s
as a metadata scheme for broadcasting. Early on, its proponents realised the need to extend
the modelling capabilities and this was done by allowing the definition of add-on descriptor
schemes. However, any descriptor scheme is only as useful as the operational semantics de-
fined for it. This makes it difficult for new Descriptor Schemes to become accepted by the
MPEG user communities. However, the MPEG standards do offer a platform for convergence
of knowledge based and meta data based content annotation.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 49 / 63
6.2.3 Semantic Web and Multimedia Standards
For a long time, the combination of knowledge and content did not play much of a role in ei-
ther of the research communities: On the one hand there were metadata models for content
like Dublin Core or MPEG-7, and on the other hand there were domain ontologies developed
by the Semantic Web community (see sections 4.1 and 4.2). However, in the past few years
much work has been done on the specification of ontologies that aim to combine traditional
multimedia description models, thus trying to develop models that allow reasoning over the
structure and semantics of multimedia data [Hun2001, Blo2005].
ZYX, Enhanced Multimedia Meta Objects, Smart Content, Information Objects
Between 1999 and 2002, researchers became increasingly aware of the need to combine
document models, content standards and the emerging Semantic Web Technologies. The
ZYX Model [Bol1999] was one of the first presentation models which accounted formally for
temporal and spatial views and which also distinguished presentation from interaction.
EMMOs (developed in the IST project CULTOS21) were a continuation of the work on ZYX
and were used to represent relations between content objects in order to build cultural units of
learning. A similar concept of "Information Object" was developed at the same time, in the
context of the INKASS project which dealt with the creation of a market place for digital
products [Abe2003].
Figure 11: A smart content package [Beh2003]
The definition of EMMOS was already similar to the vision of smart content as introduced in
the EP2010 study, which was a strategic European study on the future of electronic publishing
towards 2010 [Beh2003]. In the EP2010 study, the buzz word smart content was first intro-
duced and agreed as a term to describe a unifying vision for knowledge management tech-
nologies and any activities pertaining to digital content value chains. The smartness of these
objects refers to the ease with which the content can be manipulated and re-purposed in dif-
21 CULTOS: http://www.cultos.org – last accessed: 10.06.2006
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 50 / 63
ferent environments [New2004]. A smart content - package – as visualized in Figure 11 - con-
sists of two main parts:
• The structural component (the smart content) described by the content and the knowl-
edge about the content. Knowledge is separated in knowledge about the properties of
the media objects, knowledge about how the content is interpreted in the actual do-
main and knowledge about how the content is typically used.
• The functional component, described by several interfaces to some ambient environ-
ment in which external actors are designed to understand smart content objects. These
contact points can be viewed as service interfaces that can be used to query for knowl-
edge or act on data [Beh2003].
Inspired by the smart content - vision and based on EMMOS and INKASS Information Ob-
jects, the so-called Knowledge Content Objects (KCO) were defined: KCOs are based on the
DOLCE foundational ontology (see section 4.2.3) and have so-called semantic facets that
form modular entities to describe the properties of KCOs, including the raw content object or
media file, metadata and knowledge specific to the content object and knowledge about the
topics of the content (its meaning). Through these facets all the information is modelled, that
is necessary to deal with KCOs in different situations [Beh2005]. An overview of the state of
the art in intelligent content models and a reference model for intelligent content can be found
in [Bür2006a].
6.2.4 Active Media Objects for Mobile Communications
The aceMedia integrated project is defining an infrastructure for the use of active media ob-
jects that can be used in mobile applications. ACE stands for "Autonomous Content Entity"
and an ACE-enabled device consists of four components:
• the aceManager, coordinates communication between the components within the ace-
Framework and also with external systems..
• the aceRepository provides functionality for storage and retrieval of ACEs. This is
implemented as a system-independent wrapper middleware to handle multiple aceMe-
dia platforms such as mobile phones, set-top boxes as well as high-end servers at a
content provider.
• The aceExEnv is the ACE execution environment. It is a sand-boxed environment to
run the intelligence layer of an ACE using a virtual machine approach as known from
Java applets. The execution of ACEs is initiated and controlled by the aceManager.
• The aceNetworkLayer handles communication with any third party device and appli-
cation.
An ACE [Kom2004] consists of three layers:
• the digital content itself,
• its associated metadata in the metadata layer to store manual and automatically ex-
tracted semantic annotations and
• a content management "intelligence" layer, consisting of distributed functions that
enable the content to instantiate itself according to its context including its network
environment, the user terminal, and recorded user preferences.
.
The ACEMEDIA project has also developed an ontology structure to include low-level
audiovisual features, descriptors and behavioural models in order to support knowledge-aware
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 51 / 63
multimedia analysis, media-aware semantic inferencing and high-level semantic reasoning,
which will provide the means for automatic content annotation and generation of the ACE
scalable metadata layer. This ontology structure is based on the DOLCE foundational ontol-
ogy and its spatio-temporal extensions, a Visual Descriptor Ontology that contains the repre-
sentation of MPEG-7 visual descriptors, the Multimedia Structure Ontology which models
basic multimedia entities from the MPEG-7 Multimedia Description Scheme and some do-
main ontologies [Blo2005].
6.3 Knowledge Content Objects (KCOs)
Knowledge Content Objects, as proposed by Behrendt et al. [Beh2005], were designed to ad-
dress environments where actors have knowledge and receive information. Furthermore, the
actors have access to a large content space from which they can draw further information. The
actors make use of their existing knowledge space, weave the new information into their ex-
isting knowledge and eventually interpret their growing knowledge space with respect to their
current or future action space. If the interpretation is done with reference to a future action
space then it is called planning, with respect to the current action space it is called doing. Ac-
tors communicate with each other by exchanging meaningful statements: Human beings are
capable of using natural language for this task. When machines are involved, surrogates must
be found for natural language and for the notion of meaningful statements. KCOs can be un-
derstood as such a surrogate which can be exchanged between humans and humans, humans
and machines, as well as machines and machines.
Figure 12: The structure of a KCO [Bür2006b]
The described setting -- where actors can draw further information from content in the re-
source space and later have to weave it into their knowledge space -- is represented by the
structure of the KCO in three different levels:
1. Resource Level: This level refers to the actual content object (File, Stream, image,
text, structured text like XML or MS Office documents). Content objects consist of
the actual content and some additional information facilitating its consumption. To at-
tach any knowledge to content objects it is required to uniquely identify such object by
defining it as a resource. Web based systems usually use URIs as unique identifiers.
2. Meta Level: This level refers to knowledge describing features of the content object,
e.g. frame rate, compression type or colour coding scheme. These features are more
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 52 / 63
intuitive types of metadata. Publication rights, access restrictions or presentation de-
scriptions are metadata, too, because they are specific to the content object.
3. Subject Matter Level: This level comprises knowledge about the topic (subject) of
the content as interpreted by an actor. The content object realises this interpretation.
Because this level represents interpretations of actors about topics, such knowledge is
not dependent on the content object. This independence can be illustrated by the fact
that one can define knowledge objects without owning content that realizes this se-
mantic descriptions (e.g. in order to search for content that realizes these descriptions).
Figure 12 visualizes the core structure of a KCO: The numbers 1, 2 and 3 in the figure refer to
the three levels of knowledge described above: The content object and a number of knowl-
edge objects are the main building blocks of a KCO. However there are two types of knowl-
edge objects. The first type represents knowledge objects which are about the content object.
Therefore they belong to the meta level (2). The meta level consists of knowledge objects
containing descriptions about different aspects of the content object like encoding, presenta-
tion, security, trust and business related information. The second type are knowledge objects
representing an interpretation of the content object. Such knowledge objects are about the
topic of the content and belong to the subject matter level (3). Note that a content object can
realise diverse interpretations conceived by different actors. This reflects the fact that actors
can refer to different aspects of the content or may even disagree about the correct interpreta-
tion of a content object.
The foundation for KCOs is an ontological design pattern specified within an extension of the
DOLCE foundational ontology [Mas2003]; Details about DOLCE and this ontological design
pattern can be found in [Bür2006] or [Beh2005].
Table 4: The facets of a KCO [Bür2006b]
In addition to the knowledge structure the KCO also defines a structure based on the different
domains of the knowledge objects. This structure is divided into six so-called facets of which
some facets are further separated into facet elements. Each facet and facet element defines a
knowledge structure which is optimized for a specific usage (see Table 4).
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 53 / 63
Most of the facets and their elements belong to the meta level (2), because they define knowl-
edge structures to hold information about features specific to the content object. The proposi-
tional description and the content classification belong to the subject matter level (3) because
they specify the subject matter. The first knowledge level (the resource level ) is represented
by the content object itself. Further information about the Knowledge Content Object model
can be found in [Beh2005], [Beh2006], [Bür2006], [Bür2006a] and [Bür2006b].
It is important to note that the KCO model seems to offer a good starting point for Intelligent
Media Assets because it addresses several of the core requirements set out earlier:
• Many-to-many assignment of knowledge items and content items
• Being rooted in a foundational ontology
• A defined conceptual data structure which supports multiple views
• Alignment with semantic web technologies
• Support for multiple content standards
However, IMAs will have to add the following capabilities to the KCO model:
• Real time capability
• Ability to contextualise content depending on user preferences
• More comprehensive knowledge content storage model
With the above basis, IMAs may also be made compatible with emerging semantic web tech-
nologies as well as with competing or complementary models such as the aceMedia Content
Model.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 54 / 63
6.4 Comparison of Selected Knowledge & Content Models
The following table summarizes how IMA will extend the current State of the Art with re-
spect to the requirements set out for intelligent content models.
MIPS
1995
MPEG-7
1996
ongoing
ZYX &
EMMO
2002
KCO
2005
ACE
2005
IMA
2008
Many-to-many
assignment of
knowledge to
media assets
Yes No Partial Yes unclear Yes
Knowledge-
level support
for metadata
alignment
No No No Yes Yes Yes
Defined data
structure at con-
ceptual level,
asset exchange
and storage lev-
els
No Partial Partial Yes Yes Yes
Real-time capa-
bility
No Partial No No Partial Yes
Ability to con-
textualise the
object accord-
ing to user pref-
erences
Partial
(tem-
plates)
Partial (user
model in
MPEG-21)
Partial
(tem-
plates)
Partial
(tem-
plates)
Partial Yes
Alignment with
semantic web
technology
No Yes
Partial Yes Yes Yes
Support for (in-
tegrated) meta-
data manage-
ment according
to multiple con-
tent standards
No No No Yes unclear Yes
Defined storage
model
Yes unclear Yes Partial
(RDF)
Partial Yes
Table 5: Comparison of intelligent content models according to major selection criteria
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 55 / 63
7 Conclusions for the Design of the Intelligent Me-
dia Framework
We have given an overview of asset management systems and of standards related to media,
news, and broadcasting. This was put into the context the LIVE system which aims at provid-
ing novel ways of enriching digital video content upon delivery, by adding knowledge based
information to media objects. We compared recent approaches to the modelling of intelligent
content and identified research gaps that the LIVE project should be able to close.
The Intelligent Media Framework will mediate between LIVE's Intelligent Media Asset In-
formation System, the Recommender System, the Metadata Generation System and the Video
Conducting System. The IMF must be seen conceptually as an actor that operates on data
structures which we call Intelligent Media Assets (IMAs). In other words, IMAs are not per se
living objects that act and re-act in the LIVE environment like for example intelligent agents.
Compared to raw media objects IMAs get their “intelligence” from their rich annota-tions
added by either humans, manually or by software applications (semi-)automatically. The ho-
mogeneous data structure of IMAs enable them to get annotated by concepts coming from on-
tologies and this is the basis for reasoning and infering new information from existing facts.
In order to enable IMAs to find their way to the interested users the recommender system
plays a crucial role, as it can match IMAs against each other and match IMAs against user
profiles. The interaction of consumers with IMAs is essentially modelled in the staging tem-
plates which are part of the Video Conducting System.
We have seen that research into multimedia standards, content models and knowledge based
technologies points clearly at the convergence of knowledge and content technologies. While
current metadata standards such as MPEG-7 are gaining acceptance in the user communities,
knowledge based technologies are still not understood well enough to have found their way
into commercial products used by media producers. LIVE will demonstrate that knowledge
based content production systems can deliver superior quality of information and a better
match between information needs and information provision, while not compromising on the
real-time aspects (video and knowledge data arrive synchronised on the user's screen). In or-
der to make this scenario a reality, the Intelligent Media Framework must offer answers to
these questions:
• Is it possible to associate different sources of additional information to the objects
which participate in a media event? Associating different knowledge sources means
we must be able to attach multiple statements to one content stream (even to a single
items in that stream).
• Is it possible for meta data schemes - which may come at different levels of sophisti-
cation - to be aligned with the overall model? This means that the semantics offered
by IMAs can be expanded by aligning a new metadata scheme to the overall ontology
on which the IMA is founded.
• Is it possible to not only conceptualise an IMA, but does it also have a logical and
physical design, in the form of a machine-readable data structure which can be stan-
dardised and used as a data exchange format?
• Is it possible to generate synchronised streams of content and associated knowledge
items, and what information needs to be modelled in the IMF in order to provide this
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 56 / 63
synchronised stream? Alternatively, what scope is there for supplying good quality of
information service even if the provision of knowledge-based items is not concurrent
with the content streams?
• Is it possible to characterise user preferences (profiles) in such a way that the IMA can
provide data to match between the preferences and the features exhibited by content
objects? This may require careful consideration how much user-related information
will be associated with a content object, either by directly putting the information into
the object or by storing the relationships between the content and its users in some
other place (e.g. in a trusted repository).
• Is it possible to integrate the technology developed in LIVE, into other open frame-
works and can the solutions be made accessible via semantic web technology? This
means it must be possible to extend applications of the IMF by other components pro-
vided they are also using open standards such as RDF and OWL.
• Is it possible to separate media related metadata from subject domain related metadata
and from business related metadata? For example, an image of a sailing boat, the pur-
chase of the rights for displaying the image on a web site, and information about
which boat it is and what history is associated with it, may well be kept in different
systems, but should be accessible through the multiple views of the IMF.
• Is it possible to operate on Intelligent Media Assets in ways that are similar to data-
bases? In other words, does the IMA offer a well-defined data model or schema? This
may require a trade-off between the richness of an expressive knowledge model and
the practical needs of end user organisations.
In the design stage of the LIVE project, this presentation of the state of the art in intelligent
content models will act as the baseline for the design. The notion of Intelligent Media Assets
is an extension to a line of research which started approximately in 1990 and which has since
led to a convergence of multimedia content standards, metadata schemes and semantic models
including foundational ontologies.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 57 / 63
8 References
[Abe2003] Abecker, A., Apostolou, D., Maass, W., Mentzas, G., Reuschling, C. and
Tabor, S., Towards an Information Ontology for Knowledge Asset Trad-
ing. in ICE 2003 - 9th International Conference of Concurrent Enterpris-
ing, (Espoo, Finland, 2003).
[Ada2003] Adams, WH and Iyengar, G. and Lin, C.Y. and Naphade, M.R. and Neti,
C. and Nock, H.J. and Smith, J.R. "Semantic indexing of multimedia
content using visual, audio and text cues" In: EURASIP Journal on Ap-
plied Signal Processing Vol.3 No. 2, pp. 170--185, 2003.
[Ant2004] Antoniou, G. and van Harmelen, F. A Semantic Web Primer MIT, 2004
[Baa2003] F. Baader, D. Calvanese, D. L. McGuiness, D. Nardi, P. F. Patel-
Schneider (2003): The Description Logic Handbook: Theory, Implemen-
tation, Applications. Cambridge University Press, Cambridge, UK, 2003.
[Bar2001] Barnard, K. and Forsyth, D. “Learning the semantics of words and pic-
tures,” in Proc. International Conf. on Computer Vision, vol. 2, pp. 408–
415, IEEE, Vancouver, Canada, July 2001.
[Bau2005] Christoph Bauer, Florian Rosensprung, Stefan Lajtos, Laurent Boch,
Philippe Poncin, and Conny Herben-Leffring Analysis of current audio-
visual documentation models, Mapping of current standards, Prestospace
Deliverable D15.1MDS1, 2005. Available at:
http://prestospace.org/project/deliverables/D15-
1_Analysis_AV_documentation_models.pdf - last accessed: 27.10.2006)
[Beh2003] Behrendt, W., Geser, G., Mulrenin, A.: EP2010 - the Future of Electronic
Publishing Towards 2010, 2003.
[Beh2005] Behrendt, W., Gangemi, A., Maass, W., Westenthaler, R.: Towards an
Ontology- Based Distributed Architecture for Paid Content In: A. Go-
mez-Perez and J. Euzenat (Eds.): ESWC 2005, LNCS 3532, pp. 257-271,
2005.
[Beh2006] Behrendt, W., Arora, N., Bürger, T., and Westenthaler, R.: "A Manage-
ment System for Distributed Knowledge and Content Objects" Proceed-
ings of the 2nd International Conference on Automated Production of
Cross Media Content for Multi-channel Distribution, December 12-15,
2006, Leeds, UK.
[Ben2001] Benitez, A.B. and Change, S.F. and Smith, J.R. "IMKA: a multimedia
organization system combining perceptual and semantic knowledge" In:
Proceedings of the ninth ACM international conference on Multimedia,
2001
[Ben2002] Benitez, A. B. et al.: Semantics of Multimedia in MPEG-7. In: Proceed-
ings of the IEEE International Conference on Image Processing (ICIP
2002), 2002.
[Ber2001] Berners-Lee, T., Hendler, J., & Lassila, O. 2001. The Semantic Web.
Scientific American, 284(5), 34--43.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 58 / 63
[Bez2002] Bezzera, B.; Carvalho, F.; Ramalho, G.; Zucker, J.-D.: Speeding up re-
commender systems with meta-prototypes. Lecture Notes in Computer
Science, Advances in Artificial Intelligence, 2002. Springer-Verlag
GmbH, 227–236.
[Blo2005] Bloehdorn, S. et al.: Semantic Annotation of Images and Videos for Mul-
timedia Analysis In: Proceedings of the 2nd European Semantic Web
Conference, ESWC 2005, Heraklion, Greece, May 2005.
[Bol1999] Boll, S. and W. Klas: 1999, `Z Y X - A Semantic Model for Multimedia
Documents and Presentations'. In: Proc. of the 8th IFIP Conference on
Data Semantics (DS-8): "Semantic Issues in Multimedia Systems". Roto-
rua, New Zealand: Kluwer Academic Publishers.
http://citeseer.ist.psu.edu/boll99semantic.html
[Bor2002] Bormans, J. and Hill, K.: Mpeg-21 overview v.5.
http://www.chiariglione.org/ mpeg/standards/mpeg-21/mpeg-21.htm,
2002. ISO/IEC JTC1/SC29/WG11. (last accessed: 06.04.2006)
[Bre1998] Breese, J. S.; Heckerman, D.; Kadie, C.: Empirical analysis of predicitve
algorithms for collaborative filtering. Proceedings of the Fourteenth Con-
ference on Uncertainty in Artificial Intelligence, Madison,USA, 1998.
Morgan Kaufmann Publisher.
[Bri2004] Brickley D, Guha RV (2004) RDF Vocabulary Description Language
1.0: RDF Schema. W3C Recommendation, http://www.w3.org/TR/rdf-
schema/ (last accessed: 20.10.2006)
[Bru2002] Bruce Devlin of Snell & Wilcox for the EBU Technical Review, July
2002. http://www.ebu.ch/en/technical/trev/trev_291-devlin.pdf
[Bru2005a] de Bruijn J, Polleres A (eds) (2005): OWL-.WSML Deliverable D20.1
v0.2. WSML Working Draft May, 15 2005.
http://www.wsmo.org/TR/d20/d20.1/v0.2/20050515/ (last accessed:
27.10.2006)
[Bru2005a] de Bruijn J, Polleres A (eds) (2005): OWL-.WSML Deliverable D20.1
v0.2. WSML Working Draft May, 15 2005.
http://www.wsmo.org/TR/d20/d20.1/v0.2/20050515/ (last accessed:
27.10.2006)
[Bru2005b] de Bruijn J, Polleres A, Lara R, and Fensel D (2005), OWL DL vs. OWL
Flight: Conceptual Modeling and Reasoning for the Semantic Web, Pro-
ceedings of the 14th International World Wide Web Conference
(WWW2005), Chiba, Japan.
[Bru2006] Jos de Bruijn, Holger Lausen, Axel Polleres, Dieter Fensel: The Web
Service Modeling Language WSML: An Overview. In: Proceedings of
the 3rd European Semantic Web Conference (ESWC 2006). Springer,
2006.
[Buc2002] Buczak, A; Zimmerman, J.; et al.: Personalization: improving ease-of-
use, trust and accuracy of a tv show recommender. In proceedings of the
AH2002 Workhop on personalization in future TV, Malaga, Spain 2002
[Bur2002] Burke, R.: Hybrid recommender systems: Survey and experiments. User
Modeling and User-Adapted Interaction 12 (2002), 331–370.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 59 / 63
[Bür2005] Tobias Bürger, Erich Gams, Georg Güntner: "Smart Content Factory -
Assisting Search for Digital Objects by Generic Linking Concepts to
Multimedia Content", in Proceedings of the Sixteenth ACM Conference
on Hypertext and Hypermedia (HT '05), 2005.
[Bür2006] Bürger, Tobias and Westenthaler, Rupert Mind the gap – Requirements
for the combination of content and knowledge. In Proceedings of the
SAMT Conference, 2006.
[Bür2006a] Tobias Bürger, Ioan Toma, Omair Shafiq, Daniel Doegl "State of the Art
in SWS, GRID and 'Intelligent' Content Objects - Can they meet?" GRIS-
INO Deliverable D1.1, available online at http://www.grisino.at (last ac-
cessed: 17.10.2006)
[Bür2006b] Bürger, Tobias and Westenthaler Rupert. „Why the combination of con-
tent and knowledge matters“, Salzburg Research Technical Report, 2006.
[Car1980] John McCarthy, Circumscription – A Form of Non-Monotonic Reason-
ing, Artificial Intelligence, Volume 13, pp. 27-39, 1980
[Cha2004] Chang, W.: MPEG-A multimedia application format overview and re-
quirements v.2. http://www.chiariglione.org/mpeg/standards/mpeg-
a/mpeg-a.htm, 2004. ISO/IEC JTC 1/SC 29/WG 11 - last accessed:
06.04.2006.
[Chan1998] S. F. Chang, W. Chen, and H. Sundaram, “Semantic visual templates—
linking features to semantics,” in Proc. IEEE International Conference on
Image Processing, vol. 3, pp. 531–535, IEEE, Chicago, Ill, USA, October
1998.
[Cyc2002] CycCorp (2002): Ontological Engineers Handbook,
http://www.cyc.com/doc/handbook/oe/oe-handbook-toc-opencyc.html -
last accessed: 27.10.2006
[DCMI2006] Dublin Core Metadata Initiative (2006) DCMI Metadata Terms
http://dublincore.org/documents/dcmi-terms/ - last accessed: 15.10.2006
[Dif2002] Difino, A.; Negro, B.; et al.: A multi agent system for a personalized
electronic program guide. In proceedings of the AH2002 Workhop on
personalization in future TV, Malaga, Spain 2002
[Din2004] Li Ding, Tim Finin, Anupam Joshi, Rong Pan, R. Scott Cost, Joel Sachs,
Vishal Doshi, Pavan Reddivari, and Yun Peng, Swoogle: A Search and
Metadata Engine for the Semantic Web, Thirteenth ACM Conference on
Information and Knowledge Management (CIKM'04), Washington DC,
November 2004
[Dow2005] Dowman, M., Tablin, V., Ursu, C., Cunningham, H. and Popov, B.: Se-
mantically enhanced television news through web and video integration
In Proceedings of the Workshop on Multimedia and the Semantic Web at
the European Semantic Web Conference (ESWC 2005), 2005.
[ETSI2006] ETSI TS 102 822-3-1 V1.3.1 (2006-01): Broadcast and On-line Services:
Search, select, and rightful use of content on personal storage systems
("TV-Anytime"); Part 3: Metadata; Sub-part 1: Phase 1 - Metadata sche-
mas
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 60 / 63
[ETSI2006a] ETSI TS 102 822-3-3 V1.1.1 (2006-01): Broadcast and On-line Services:
Search, select, and rightful use of content on personal storage systems
("TV-Anytime"); Part 3: Metadata; Sub-part 3: Phase 2 - Extended Meta-
data Schema
[Fen2001] Fensel D, van Harmelen F, Horrocks I, McGuinness DL, Patel-Schneider
PF (2001) OIL: An ontology infrastructure for the Semantic Web. IEEE
Intelligent Systems & their applications 16(2): pp. 38-44
[Fer2004] Fergus, R. , Perona, P. and Zisserman, A.: A Visual Category Filter for
Google Images Proceedings of the 8th European Conference on Com-
puter Vision, Prague, Czech Republic, 2004.
[Gom2002] Gomez-Perez, A and O. Corcho, Ontology Languages for the Semantic
Web, IEEE Intelligent Systems 17(1) (2002) 54-59.
[Guar1998] N Guarino (1998) “Formal Ontology and Information Systems”. In: N.
Guarino (ed.), Formal Ontology in Information Systems, 1998. Proceed-
ings of FOIS’98, Trento, Italy, 6-8 June 1998. Amsterdam, IOS Press,
pp. 3-15.
[Guin2001] Deborah L. McGuinness (2001) “Ontologies Come of Age”. In: Dieter
Fensel, Jim Hendler, Henry Lieberman, and Wolfgang Wahlster, editors.
The Semantic Web: Why, What, and How, MIT Press, 2001.
[Haas2004] Hugo Haas, Allen Bown: “Web Services Glossary”
http://www.w3.org/TR/ws-gloss/ - last accessed: 22.10.2006
[Han2001] Hand, D.; Smyth, H. M. P.: Principles of data mining. Cambridge: The
MIT Press, 2001.
[Hen2003] Henzinger, M. Chang, B.-W., Milch, B. and Brin, S.: Query-Free News
Search. In Proceedings of the 12th World Wide Web Conference, pp. 1-
10, 2003.
[Hor2001] Horrocks I, van Harmelen F (eds) (2001) Reference Description of the
DAML+OIL (March 2001) Ontology Markup Language. Technical re-
port. http://www.daml.org/2001/03/reference (last accessed: 27.10.2006)
[Hun2001] Hunter,J.: Adding Multimedia to the Semantic Web - Building an
MPEG-7 On- tology, Proc. of the 1st International Semantic Web Work-
ing Symposium (SWWS 2001), Stanford, USA, 2001.
[Jon2004] Annemieke de Jong: Metadata in the audiovisual production environ-
ment, published by Nederlands Instituut voor Beeld en Geluid, Hilver-
sum/Netherlands, October 2003
[Kha2003] Kang, H. and Shneiderman, B.: "MediaFinder: an interface for dynamic
personal media management with semantic regions" In: Proceedingsof
the Conference on Human Factors in Computing Systems, 2003
[Kif1989] Kifer, M. and Lausen, G. (1989). F-logic: a higher-order language for
reasoning about objects, inheritance, and scheme. In Proceedings of the
1989 ACM SIGMOD international Conference on Management of Data.
J. Clifford, B. Lindsay, and D. Maier, Eds. SIGMOD '89. ACM Press,
New York, NY, pp. 134-146
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 61 / 63
[Kob2001] Kobsa, A.: Generic user modeling systems. User Modeling and User-
Adapted Interaction 11 (2001), 49–63.
[Kom2004] Kompatsiaris, I. et al.: Integrating knowledge, semantics and content for
user-centred intelligent media services: the acemedia project. In: Proc. of
WIAMIS 2004, Lisboa, Portugal, 23 April 2004.
[Kur2001] Kurapati, K.; Gutta, S.; et al.: A multi agent recommender. Proceedings
of the 8th International Conference on user modeling UM2001,
Sonthofen, Germany, 2001, Springer.
[Kur2002] Kurapati, K.; Gutta, S.: Tv personalization through stereotypes. Proceed-
ings of the AHAˆ ’2002 Workshop on Personalization in Future TV,
2002.
[Len1995] Douglas Lenat. Cyc: A Large-Scale Investment in Knowledge Infrastruc-
ture. Communications of the ACM, 38(11). p33-38, 1995. ACM Press,
New York.
[Leo2002] Riccardo Leonardi, Pierangelo Migliorati, "Semantic Indexing of Multi-
media Documents," IEEE MultiMedia ,vol. 09, no. 2, pp. 44-51, April-
June, 2002.
[Mac1995] Macnee, C A; Behrendt, W; Wilson,M D; Jeffery, K G; Kalmus, J R;
Hutchinson, E K: 'Presenting Dynamically Expandable Hypermedia' In-
formation and Software Technology 37 (7) pp 339-350, 1995
http://citeseer.ist.psu.edu/macnee95presenting.html
[Man2004] Manola F and Miller E (2004) RDF Primer. W3C Recommendation.
http://www.w3.org/TR/REC-rdf-syntax/ (last accessed: 27.10.2006)
[Mar2002] Martinez-Sanchez, J. M., Koenen, R. and Pereira, F.: MPEG-7: The Ge-
neric Multimedia Content Description Standard, Part 1. IEEE MultiMe-
dia 9(2): 78-87 (2002)
[Mas2003] Claudio Masolo, Stefano Borgo, Aldo Gangemi, Nicola Guarino, Ales-
sandro Oltramari. “WonderWeb Deliverable D18 Ontology Library, De-
cember 31, 2003.
http://wonderweb.semanticweb.org/deliverables/documents/D18.pdf –
last accessed: 10.11.2006
[Mas2004] Masolo, C., Vieu, L., Bottazzi, E., Catenacci, C., Ferrario, R., Gangemi,
A., Guarino, N.: Social roles and their descriptions. In: Procs. of KR’04.
(2004)
[May2003] James Mayfield and Tim Finin, Information retrieval on the Semantic
Web: Integrating inference and retrieval, SIGIR Workshop on the Se-
mantic Web, Toronto, 1 August 2003.
[Meu2004] L. Le Meur, Jo Rabin: NewsML 1.2 Guidelines V 1.00.
http://www.newsml.org/IPTC/NewsML/1.2/documentation/NewsML_1.
2-doc-Guidelines_1.00.pdf, 2004 - last accessed: 3.11.2006
[Mik2004] P. Mika, D. Oberle, A. Gangemi, and M. Sabou. Foundations for service
ontologies: Aligning owl-s to dolce. In Proceedings of the 13th Interna-
tional World Wide Web Conference. ACM, 2004.
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 62 / 63
[Mika2004] Mika, P. Akkermans, H. (2004) Towards a new synthesis of ontology
technology and Knowledge Management. In: The Knowledge Engineer-
ing Review, Volume 19, Issue 04, 01 Dec2004, pp 317-345.
[Nap2004] Milind R. Naphade, John R. Smith "On the detection of semantic con-
cepts at TRECVID" In Proceedings of ACM Multimedia 2004: 660-667
[New2004] Newcomb, S. and Blunden, B.:Smart Content, Opportunity Discovery,
Knowledge Trading and Alliance Making (slides), presented at XML
Tsichritzis et al: KNOs - Knowledge acquisition, dissemination, and
manipulation objects In: ACM Transactions on O±ce Information Sys-
tems No. 5, Vol. 2, April 1987.
[New2004a] Newcomer, E. and Lomow, G. "Understanding SOA with Web Services
(Independent Technology Guides)", Addison-Wesley, 2004.
[Nil2001] Niles, I., and Pease, A. 2001. “Towards a Standard Upper Ontology.” In:
Proceedings of the 2nd International Conference on Formal Ontology in
Information Systems (FOIS-2001), Chris Welty and Barry Smith, eds.,
Ogunquit, Maine, October 17-19, 2001.
http://portal.acm.org/citation.cfm?id=505170 - last accessed: 10.11.2006
[Nil2002] Ian Niles, Mapping Word Net to the SUMO Ontology,
http://ontology.teknowledge.com/nilesWordNet.pdf - last accessed:
10.11.2006
[Par2003] Paralic, J. and Kostial, I.: Ontology-based Information Retrieval. Proc. of
the 14th International Conference on Information and Intelligent systems,
IIS 2003, Varazdin, Croatia, September 2003, pp. 23-28.
[Pog2004] Pogacnik, M.; Tasic, J.; Tomazic, S.: Personalized content retrieval. Pro-
ceedings of the International Symposium on Telecommunications
VITEL, Ljubljana: Electrotechnical Society of Slovenia, 2004.
[Ric2002] Richard Hopper, 2002, P/Meta Metadata Exchange Scheme, V1.0
http://www.ebu.ch/en/technical/trev/trev_290-hopper.pdf - last accessed:
10.11.2006
[Roh2006] Katharina Rohlfing, e.a.: Comparison of multimodal annotation tools -
workshop report. In: Gesprächsforschung – Online-Zeitschrift zur verba-
len Interaktion (ISSN 1617-1837), Ausgabe 7 (2006), p. 99-123
(www.gespraechsforschung-ozs.de); 2006.
[Sal2006] Salter, J.; Antonopoulos, N.: Cinemascreen recommender agent: Com-
bining collaborative and content-based filtering. IEEE Intelligent Sys-
tems 21 (2006), 35–41.
[Schel2002] Schellner K, Zillner S, Klas W, 2002, "Document Models and Enhanced
Multimedia Meta Objects", CULTOS Internal Deliverable 3.2.1.
[Smi2002] Barry Smith, Ontology Information Systems,
http://ontology.buffalo.edu/ontology(PIC).pdf – last accessed:
10.11.2006
[Sno2006] Cees G.M. Snoek, Marcel Worring, Jan C. van Gemert, Jan-Mark Geu-
sebroek, and Arnold W.M. Smeulders. “The Challenge Problem for
Automated Detection of 101 Semantic Concepts in Multimedia” In Pro-
State of the Art Report IMF LIVE - D7.1
© LIVE Consortium, 2006 Page 63 / 63
ceedings of ACM Multimedia, pp. 421-430, Santa Barbara, USA, Octo-
ber 2006.
[Sow2000] John Sowa. Knowledge Representation: Logical, Philosophical, and
Computational Foundations. Brooks/Cole, 2000.
[Sta2006] Steffen Staab, Dionysios Anastasopoulos, Norman Timmermann, Carsten
Saathoff, Kosmas Petridis, Yiannis Kompatsiaris : "M-OntoMat-
Annotizer: Linking Ontologies with Multimedia Low Level Features for
Automatic Image Annotation", Poster and Demo Proceedings of ESCW
'06, 2006.
[Ste2005] Michael Steidl, John Minting: NewsCodes notation in NewsML 1 Top-
icSets, http://www.iptc.org/download/public/NCD0507.2-
NewsCodesAsTopicsets.pdf, 2005 - last accessed: 03.11.2006
[Tan2001] Tanenbaum, A.S. and Van Steen, M. "Distributed Systems: Principles
and Paradigms", Prentice Hall, 2001.
[Tro2006] Raphael Troncy, Jacco van Ossenbruggen, Christian Halaschek-Wiener,
Giorgos Stamou, Vassilis Tzouvaras, 2006, "Multi-Perspective Require-
ments for a Common Multimedia Ontology Framework", o.A., last ac-
cessed 26/11/2006,
http://www.acemedia.org/aceMedia/files/multimedia_ontology/cfr/MMT
F-Multimedia_Ontology.pdf
[TVA2002] TVAnytime: Specification series: S-3 metadata, SP0003 v1.3, part A,
2002 ftp://tva:tva@ftp.bbc.co.uk/pub/Specifications/SP003v13.zip - last
accessed: 10.11.2006
[Uch2002] Uchyigit, G.; Clark, K.: An agent based electronic program guide. Pro-
ceedings of the AHA 2002 Workshop on Personalization in Future TV,
2002.
[Uschold1996] Uschold, M., (1996) Building Ontologies: Towards a unified methodol-
ogy, AIAI Technical Report September 1996 (AIAI-TR-197), Edinburgh,
1996.
[Val2005] Vallet, D., Fernandez, M., and Castells, P. "An Ontology-Based Informa-
tion Retrieval Model" In: Proceedings of ESWC '05, 2005.
[Wes2005] Westermann G.-U., Zillner, S., Schellner K. and Klas W.: EMMOs –
Tradeable Units of Knowledge-Enriched Multimedia Content In: U.
Srinivasan and S. Nepal (Eds.): Managing Multimedia Semantics, Idea
Group, 2005.
[Zim2004] Zimmerman, J.; Kurapati, K.; Buczak, A. L.; Schaffer, D.; Martino, J.;
Gutta, S.: Tv personalization system: Design of a tv show recommender
engine and interface. Kluwer Academic Publishers, 2004.
|
![[OpenLink Software]](statics/openlinklogo_wstrap_2008.png)
