X-Smiles 0.6 Technical Specifications

-= X-Smiles Posse =-

26.4.2002, Kajaani


Table of Contents

The purpose of this document is to give a detailed description of the technical aspects of the X-Smiles browser and its related components. This document describes the different components, their roles and relations to other components. It also defines the environment the browser is implemented in.

Also described are some of the design limitations, requirements and suggestions concerning adding additional functionality to this browser.

This document does not attempt to describe the product to the highest possible detail at all levels. For more details, please consult the API documentation.

Understanding of this document requires that the reader has a fair knowledge of how markup-languages work, has a sufficient knowledge of UML to read the diagrams, knows the elements of object-oriented design, and has some understanding how the Internet and applications such as a browser could work. A knowledge of Java is preferred, but not necessary. If you encounter unknown acronyms, or unknown terminology, please refer to the glossary at the end of this document.

The root Java package is fi.hut.tml.xsmiles The X-Smiles files are stored in a single jar file called xsmiles.jar. To start the browser you can use a command such as: java -jar xsmiles.jar. Or if you have all required libraries and classes in the CLASSPATH: java fi.hut.tml.Xsmiles. For further detail, see the installation manual.

When setting up a development environment, be sure to include all jars in ./bin/lib directory to the classpath. Some libraries have overlapping classes, and might cause problems, when ordered wrong.

If you are using windows, and the only intension is to run the browser, the easiest solution is to use the handy installer, which comes with the release. It automagically copies the needed files to your system, and creates a shortcut on the desktop.

See our website for info on how to compile the browser.

This chapter describes the technical details of the X-Smiles browser architecture. The structure of the browser is shown and the relations between the components are displayed in Figure 3.1. The browser is composed of two layers: Browser Core and Browser modules. The Browser Core ties together different parts of the browser, by providing a means of communication, within different parts. The GUI provides a means of cross-platform, device independent GUIs. This includes different device specific GUIs, and a component factory, for instantiating device specific GUI widgets for other parts of the browser. The MLFC module includes different MLFCs, which the browser's functionality mainly consists of. The XML Processing module is somewhat linked to the MLFC Management, but is mainly provided by third party libraries, such as the Xerces XML parser, and Xalan XSLT engine.

The following sections describe the Browser Core, and XML Processing. Further on, in Chapter 5, we discuss the MLFCs in detail. We will also discuss the GUIs, in detail in Chapter 6

The browser core functionality is implemented by classes (Figure 3.2): Browser, BrowserLogic, MLFCManager, MLFCLoader, XMLParser, GUIManager, XMLBroker, and EventBroker. It is also possible to create multiple instances of the Browser, so you will result with multiple browserwindows.

The Browser class ties all components together. It has the classes shown in Figure 3.2 in it's vincity. The Browser implements the BrowserWindow interface, and this interface is used all classes refering to the Browser.

The main functionality includes initialization of a browserwindow, and controlling different parts of the browser.

Hybrid XML documents are documents, which contain several XML languages, separated by a namespace.A host language usually has the default namespace in the document and it also defines the layout for the document. Typical host languages are XHTML, SMIL, and SVG. A language, which is embedded inside a host language, is called a parasite language. XML Events and XForms are examples of parasite languages.

X-Smiles has a framework for handling hybrind XML documents. We've separated the MLFC registration and general element and attribute creation to an independent module, called XML Broker. XML Broker has three main roles:

MLFC Registering.  All MLFCs in the system register themselves to XMLBroker either by a namespace (e.g., 'http://www.w3.org/2000/12/xforms') or by the root element's unqualified name (e.g., 'smil').

Dispatching element and attribute creation.  XMLBroker checks the namespace of each element and attribute to be created, and if it matches any of the registered MLFCs, it forwards the request to the corresponding MLFC.

MLFC Instantiation.  XML Broker instantiates MLFCs on-demand, and keeps track of the MLFCs that have been instantiated for a document.

M, o, o, n. That spells, Markup Language Functional Component MLFC. They are bean-like, fairly independent components, which are used for rendering XML documents. See Chapter 4 for details on how to create your own MLFC.

Package.  fi.hut.tml.xsmiles.mlfc.smil, org.w3c.dom.smil20

Dependencies.  Java Media Framework 2.1.1 (JMF), jmf.jar Steady State CSS, ss_css2.jar

Namespace.  http://www.w3.org/2001/SMIL20/Language

This MLFC handles the interpretation and presentation of SMIL documents.

SMILMLFC consists of several packages. They are described here. Two packages, smil.basic and smil.extension are the core logic for SMIL. Smil.viewer is a interface for GUI specific implementation. Under smil.viewer.smilmlfc there is an interface called media, which is implemented by different media, such as ImageMedia, TextMedia etc. See Figure 5.1 for more detailed description for the packages.

fi.hut.tml.xsmiles.mlfc.smil .  This package has SMILMLFC, which is the Viewer for SMIL. It controls the SMIL core logic and passes required information for it, such as the screen size and other system attributes.There is also TimeChildList, which is an Iterator to find all children, which have timing information.

fi.hut.tml.xsmiles.mlfc.smil.basic .  This package contains the core logic for SMIL. It has all SMIL 2.0 basic elements and functionality. These are completely GUI and browser independent classes. The elements in this package make up the SMIL-DOM.

fi.hut.tml.xsmiles.mlfc.smil.extension .  This package has all non-Basic modules for SMIL 2.0. These are also GUI and browser independent classes.

fi.hut.tml.xsmiles.mlfc.smil.viewer .  This package contains the interface for a Viewer. A Viewer is a GUI specific implementation. The classes implementing this interface will have the GUI for the SMIL player. There are several implementations, defining different kind of GUIs.

fi.hut.tml.xsmiles.mlfc.smil.viewer.awt .  This implementation of the Viewer is written using AWT. It is a standalone SMIL player using only AWT. This package is not required by the X-Smiles, but exists to show how the player can work as a standalone implementation.

fi.hut.tml.xsmiles.mlfc.smil.viewer.smilmlfc .  This implementation of the Viewer is written for X-Smiles. It uses Swing to render the media and has references to the browser to retrieve browser related information, such as the system attributes. This also passes external links to the browser.

fi.hut.tml.xsmiles.mlfc.smil.viewer.smilmlfc.media .  This package is no longer used. All media is now rendered by the X-Smiles ContentHandlers. Note that these include a native Window player, which requires jawt.dll and NativeMedia.dll in the xsmiles/bin folder. These files must be copied to the xsmiles/bin folder manually to get the native players to work.

fi.hut.tml.xsmiles.mlfc.smil.viewer.swing .  This implementation of the Viewer is written using AWT. It is a standalone SMIL player using only Swing. The X-Smiles implementation viewer.smilmlfc is inherited from these classes.

org.w3c.dom.smil20 .  This is a new package to define the interface for SMIL 2.0. It has been partly written from scratch, because W3C hasn't defined it yet.

Rendering a SMIL presentation from the SMIL (XML) source involves the following two steps:

  1. Parsing the XML document and creating a SMIL-DOM. When the browser retrieves the XML document, it parses the source document. A DOM tree will be created. To create the DOM tree, the browser uses Xerces, which will call SMILDocument.createElementNS(). This SMIL specific method will create SMIL specific elements, instead of creating standard DOM elements. The SMIL elements are defined in the fi.hut.tml.xsmiles.mlfc.smil.basic package. The result will be a standard DOM compatible SMIL specific SMIL-DOM tree, which will implement the SMIL-DOM interface (org.w3c.dom.smil20) defined by W3C for scripting. The SMIL-DOM tree is also the presentation tree, handling all timing and logic for SMIL.

  2. Playing the SMIL-DOM.  The SMIL-DOM is ready to be played as soon as it has been completely created. The following steps must be taken to play the presentation:

    1. Call SMILDocument.initialize(). This will initialise the presentation. The SMILDocument class will call the SMIL body element's init() method, which will call its children's init() method and so on. This way the whole SMIL document will be initialised.

    2. Call SMILDocument.prefetch(). This step is optional. This can be called to prefetch all media in the presentation to local memory to speed up the playing of the presentation. Again, the SMILDocument will call SMIL body element's init() method, which will pass the call to its children.

    3. Call SMILDocument.start(). This will actually start playing the presentation. The body element's startup() will be called, which will call startup() method for its children, in a sequence (since body has the Seq semantics for timing). The children of body will call their children's startup() method in the way the timing is defined. (seq calls then in a sequence, par at the same time).

      After startup() has been called, it will create a Timer to wait until the start of simple duration. When the simple duration starts, activate() is called. It will display the media and start a new Timer to wait until the end of the simple duration.

      After the simple duration has ended, deactivate() will be called. This method will remove or freeze the media.

      When the parent time container ends, closedown() will be called for the child elements. It will remove the media (if it was only frozen) and free all timers.

    4. Call SMILDocument.stop(). This will stop the presentation, by calling closedown() for all elements. (first body, which will call its children etc.)

    5. Call SMILDocument.freeResources(). This will freeze all resources and memory used by the presentation. Again, body.destroy() will be called, which will call the children's destroy(). Every element is responsible of releasing its memory and resources.

Package.  fi.hut.tml.xsmiles.mlfc.xforms

Dependencies.  Xalan XPath,Xerced DOM implementation,Xerces Schema validations, DOM parser, javax.swing

Responsibilities.  Implements most of the XForms specification. For the table of implemented features, see the features documentation.

Constraints and the calculation engine are implemented in the package defined in the section called “MLFC Overview”. The overview of the implementation is presented in Figure 5.3. It includes three distinct types of data structures:

The presentation DOM. The memory representation of the host XML document, e.g. the SMIL document that hosts the XForms elements. The implementation of this markup language is a specialized tree extended from a generic DOM implementation. It contains host DOM nodes implemented in the host MLFC and XForms DOM nodes which implement, for example, a specific form control.

The instance DOM. It is an XForms internal DOM implementation, also derived from the generic DOM, that holds the current state of the instance, namely instance data item values and their constraint states, such as 'relevant'.

Dependency graphs. They are used by the calculation engine. There are two dependency graphs per instance: The Master Dependency Graph, which is created at initialization time, and The Pertinent Dependency Subgraph, which is generated on the fly from the main graph whenever the calculation engine is run. The dependency graphs are composed of Vertices and Edges

An XForms presentation DOM node is linked two-ways to the corresponding instance DOM node. The only relevance of the presentation DOM to the calculation engine, are the instance item value updates, which are reflected to the presentation DOM and vice versa. The dependency graph's vertices are linked one-way to the instance DOM nodes. There may be many vertices linked to the same instance node, as long as they are of different type (e.g. calculate and relevant vertices), as shown in Figure 5.3.

A computable constraint, such as calculate in a bind element, can express a computation for multiple instance data items. Each computation can refer to multiple data items. For example, a bind expression can attach a calculation of price * units to every item total.This leads to an important prequisite for building the Main Dependency Graph; the ability to obtain the referents of each XPath expression, e.g. how to find out which instance data items the calculate expression references. X-Smiles uses XPath processor in Xalan, which had to be extended to support this feature. We execute an XPath expression once in its correct context (namespace and XPath context nodes), and collect references to every node that the expression's location paths reference. We've extended some of Xalan's XPath implementations classes and use those in the package org.apache.xpath to obtain the referents.

Figure 5.4 depicts the collaboration between different objects when a value in the instance changes. In the diagram, the user changes the value in a 'selectOne' form control. The sequence is initiated when the JList component used by the 'selectOne' element dispatches the valueChanged event, which the selectOne listens to. After recording the specific form control, the base class calls setValue() for its bound InstanceItem. The instance item then calls recalculate() with a change vector containing itself. The recalculate is applied to the XForm object, which creates a new SubDependencyGraph and calls calculate() on it. SubDependencyGraph then runs the recalculation algorithm on itself, and changes the values and properties on the recalculated InstanceItems, which in turn are reflected to the user interface. After the recalculation, the revalidate() function is called on the Model object to determine whether any schema constraints have been violated or corrected.

Package.  fi.hut.tml.xsmiles.mlfc.xslfo.*

Dependencies.  fixme

fi.hut.tml.xsmiles.mlfc.xslfo.*

Dependencies.  fixme

The XSL FO MLFC uses the Apache XML FOP Project (http://xml.apache.org/fop/) [FOP]. However, FOP cannot be used in the browser as such, as it provides a way to preview a XSL FO document, but allows no means of user interaction - a very basic feature in any browser. Thus, we have extended the FOP to add user interaction using:

Simple links are defined in the XSL FO specification (i.e., the fo:simple-link element), while forms are not within the scope of the XSL FO. Forms are introduced in the XForms specification, which is still at an early stage. Thus, we use XHTML form elements (such as input fields, buttons, etc.) with the XForms data model for rendering and input, and a subset of the XForms data model for defining the data model of the form.

XForms extension affects FOP in many levels:

Detecting XForm -declaration from the DOM tree. The class mlfc.xslfo.apps.Driver2 and DocumentReader2 (taken to X-Smiles project from FOP) iterate the DOM tree. This process has been modified so that if an ?xform? element is encountered, it is recognized as Xform Data Model and sent to the mlfc.general.XformsHandler class (c.f. the section called “XForms”). This element and its children are not passed to FOTreeBulder class.

Current DOM node kept in the memory.  Our extension mechanism (XForms) have been build on top of DOM, and therefore we need the DOM element for each extension element (inside foreignObjects). Driver2 keeps the current DOM node in memory, so that Xforms extensions can access it.

New FO Tree elements.  The form UI components can only be inside the fo:instream-foreign-object. The UI components are in the xform namespace and are declared in fi.hut.tml.xsmiles.mlfc.xslfo.form.FormElementMapping.java. New FO tree elements were added to FOP for form elements. The following classes in package mlfc.xslfo.form map the DOM elements and attributes to FO Tree elements and properties:

New AreaTree elements. The form UI components can only be inside the fo:instream-foreign-object. The UI components are in the xform namespace and are declared in fi.hut.tml.xsmiles.mlfc.xslfo.form.FormElementMapping.java.

New AreaTree elements.  Mlfc.xslfo.form.FormWidgetArea Areas are created for the form widgets

Rendering.  AWTRenderer has been taken into X-Smiles as AWTRenderer2. (there were some private members which have been modified as protected.) The class .mlfc.xslfo.render.XSmilesAWTRenderer extends AWTRenderer2 and adds the form and link rendering capabilities (using CommonExtensions). The package fi.hut.tml.xsmiles.mlfc.xforms handles the form input and button clicks and maps the form widgets to corresponding xform instances (c.f. the section called “XForms”).

Package.  fi.hut.tml.xsmiles.mlfc.svg, org.csiro.svg...

Dependencies.  fixme

SVG MLFC can render vector graphic documents. It uses the CSIRO SVG Toolkit (http://sis.cmis.csiro.au/svg/), which supports most of the SVG features. There are still several things missing or only partially implemented, since the toolkit is not yet complete. See the CSIRO web site for details.

Some slight modifications has been done in order to integrate the viewer into the browser. These modification include slight user interface adjustments. These modifications affected two classes: org.csiro.svg.viewer.Viewer and org.csiro.svg.viewer.Canvas. These are taken into the XSmiles project and named Viewer2 and Canvas2.

As X-Smiles is intended for a variety of different target devices, we have also focused on device independence. First of all, the browser can be interfaced through different user interfaces, which can also be added, by implementing the GUI interface.

CC/PP

Composite Capabilities / Preference Profiles is a W3C proposal for relaying device, and software capabilities of a device to the server side. Basically it can be used to relay other context-related information also. For building context-aware applications, this is a promising technology to use to relay information. CC/PP is not yet a recommendation, but hopefully will become one.

DOM

Document Object Model. An application programming interface (API) for HTML and XML documents, which defines the logical structure of documents and the way a document is accessed and manipulated.

DTD

Document Type Definition. XML markup declaration, that provides a grammar for a class of documents (e.g. SMIL DTD).

ECMAScript

A standard for a general purpose, cross-platform scripting language, reminiscent of JavaScript.

JFC

The Java Foundation Classes extend the original Abstract Windowing Toolkit(AWT) by adding a comprehensive set of graphical user interface class libraries. These include "Swing" GUI Components and Drag & drop from other (even non-java) app.

JMF

The JavaTM Media Framework (JMF) is the application programming interface (API), that allows developers to incorporate media data types such as audio and video into their Java applications.

HTML

Hypertext Markup Language. The most common publishing language used on the World Wide Web.

HTTP

Hypertext Transfer Protocol. A protocol for transferring files (text, graphic images, sound, video, and other multimedia files) on the World Wide Web.

ML

A markup language defines a way to describe information in a document. There are several standards, such as HTML 4.0, which define certain markup languages, but new languages can be defined for different applications and domain areas with the help of XML.

Scripting Language

In the scope of this browser, a language used to add interaction and dynamics to XML documents.

SGML

See Standard Generalized Markup Language.

Standard Generalized Markup Language

Some reasonable definition here.

See Also XML.

SMIL

(Synchronized Multimedia Integration Languag. A markup language defined with XML. SMIL gives end-users the capability to integrate different media objects into one single synchronized multimedia presentation. SMIL is specifically designed for the WWW environment and is defined by the World Wide Web Consortium

UML

Unified Modelling Language. A general-purpose notational language for specifying and visualizing complex software, especially object-oriented products It is being developed under the auspices of the Open Management Group (OMG).

URI

Unifrom Resource Identifier. URI is a way to identify points of content in the Internet (e.g text, video, audio). An URI typically contains the mechanism used to access the source, the specified computer the resource is housed in and the specific name of that content on that computer.

W3C

World Wide Web Consortium. W3C is a prestigious international organization, which produces standards of the world wide web.

XLink

XML Linking Language. The Xlink specification defines the constructs that may be inserted into XML resources to describe links between objects.

XML

Extensible Markup Language. A data format for structured document interchange on the WWW and it is defined by the World Wide Web Consortium. XML itself is not a markup language but a metalanguage for markup language design.

XPointer

XML Pointer Language. The XPointer specification defines the constructs that support addressing into the internal structures of XML documents.

XSL

Extensible Style Language. A language for expressing stylesheets in XML documents.

DOCTYPE

DTD and MLName is assigned with this parameter in an XML document.

Document History

Document history is a queue, which contains the URIs the users has visited during his/hers current visit to the Internet. The URIs are placed into the queue in chronological order.

Document Source

Document source is the source code of the document, that the user is currently viewing.

Xerces DOM

The third-party DOM implementation from Apache Foundation used by the browser.

Extended DOM Implementation

A method, where a markup language is implemented by adding functionality to the DOM elements themselves. This method saves memory, and makes it easier to dynamically reflect DOM tree changes, with e.g., scripts, to the screen and the user. The SMIL, XForms, and SVG are implemented in this way. This means that after creation, each DOM element knows how to display itself.

MLFC

Markup Language Functional Component. A bean-like proprietary software component for X-Smiles, which handles documents of a specific markup language. It basically is a component, which is capable of displaying a document of a certain ML in a container.

This term and its acronym was defined by the project group and is limited to the scope of this browser.

Primary MLFC

A MLFC functions as a primary MLFC, when it is the root document, which is displayed in the browser window. If a document is embedded in another document, e.g., as image, then a corresponding MLFC will be requested to function as a Secondary MLFC, to render content in the sized container assigned to it.

Secondary MLFC

If a document is embedded in another document, e.g., as image, then a corresponding MLFC will be requested to function as a Secondary MLFC, to render content in the sized container assigned to it.

See Also Primary MLFC .

Host MLFC

The MLFC created for the host language in a hybrid XML document.

Parasite MLFC

The MLFCs created for the parasite languages in a hybrid XML document.

Source MLFC

An MLFC, which is responsible for showing the source code of the XML document

Tree MLFC

An MLFC, which is responsible for showing the tree view of the document.

X-Cries is a Java Servlet, which is capable of extracting CC/PP information, and dealing pages based on this information. On the server side this is done by a site description XML file, which configures a site. This configuration file specifies which stylesheet, or file in general is sent back. The testing is achieved by setting different tests, such as "less than", "greater than", or "equals". This way we can send a page written in inuite, if the user's CC/PP profile says that her language is inuite.

The X-Cries servlet can currently be found in the the cvs. We also have it running at xdemo.tml.hut.fi. We have links to some demonstrations located on that machine on our demo page of X-Smiles.

Currently we are working with integrating the CC/PP functionality for Cocoon.

This document was created with DocBook XML v. 1.44[DocBook]. The newest version of this file is located at /doc/TechSpec/specifications.xml. All related images and such should be located under the subdirectory TechSpec_files. To generate a XHTML version of this file, use the Ant: /src/build.sh docbook. To view a XSL-FO version of this document, open it in X-Smiles, the processing instruction is pointed to the correct stylesheet, and X-Smiles will tranform it automatically.

The stylesheets, are located under http://www.xsmiles.org/docbook/. The transformation is a bit slow, so be patient. The intention is to distribute transformed versions, in XHTML, XSL-FO, and PDF.