A Realist's SMIL Manifesto, Part II

July 17, 2002

Fabio Arciniegas A.

In the first part of this article series, I mentioned two big problems -- and addressed the first -- obstructing widespread adoption of SMIL: 

• Confusion about terminology, versioning, and structure
• Lack of business and artistic orientation from current literature

The second problem has been key in making SMIL a tough sale because -- just like Flash and SVG -- it is a creative-oriented technology. It lives in the middle of the programmer-designer spectrum, where technocrat literature fails to attract many people from either side.

On one hand, web designers seeing a bouncing ball on the screen tend to react with a simple "I can easily do that in Flash", which is true. On the other hand, programmers, who appreciate the tech-appeal of the way in which the ball is made to bounce, are not being educated about the possibilities SMIL offers for expression. As a result, people on both ends tend to dismiss the whole technology as a nice toy.

The goal of this article is to show SMIL's potential as a technology in service of narrative strategies, adding something extra to the media rich Web.

Structure and Nature

In what follows I will explore three narrative strategies and how to implement them using three important features of SMIL 2.0. The features explained are transitions, declarative animation, and SMIL 2 events. The narrative strategies are condensation, synecdoche, and spatial montage.

The examples are based on the XHTML+SMIL profile, a particular application of the modular nature of SMIL 2.0. It allows the mixture of the SMIL synchronization and timing modules with the well-known XHTML language. The only available (although partial) implementation of XHTML+SMIL 2.0 at the time of this writing is bundled in Internet Explorer 5.5 and above, so you will need Internet Explorer 5.5+ to see the examples. Microsoft refers to its implementation as HTML+TIME 2.0.

This article is not intended to provide a reference, which is nicely provided by the SMIL 2.0 specification itself and the Microsoft HTML+TIME documentation. Instead I've attempted to create a set of examples which both encourage new users into the technology by the strength of the use-cases they cover and explain the principal mechanisms of the language.

HTML+TIME 2.0

Microsoft has implemented a sizable portion of the XHTML+SMIL 2.0 profile in what it calls the HTML+TIME 2.0 behavior (from now on all references to SMIL or HTML+TIME refer to the 2.0 version of each unless otherwise stated.) In Internet Explorer, a behavior is a way of extending HTML by associating foreign tags with a particular implementation, available to the browser either as a default component or an add-in. 

Succinctly put, the HTML+TIME behavior does two things:

• Allows the use in HTML of a number of SMIL elements (video, animation, audio, par, seq,excl,img,media,ref), some of which were explored in Part 1 of this article.
• Introduces new attributes, scriptable properties, events, and objects, that implement the notion of a timeline and can be associated with existing HTML elements. This allows traditional content like paragraphs, divs, and img elements to be treated as media to synchronize with SMIL, either by declarative or scripting means.

There are three steps to including the HTML+TIME behavior in an HTML page, as shown below:

After the steps above -- which I've provided in this reusable template -- you are ready to start working and fill the body of the page with synchronized elements.

In what follows we will see declarative and scripting means of putting this partial XHTML+SMIL implementation to use. Applications on top of other SMIL 2.0 players that implement the Language or Basic Profiles (RealONE, GrinS, and SMIL Player by InterObject) can benefit from some of the techniques discussed here, even if they don't support SMIL-enabled HTML (the XHTML+SMIL profile).. 

Condensation using SMIL 2.0 Transitions

<t:par dur="5">
   <t:img src="comeplay.gif"/>
   <t:transitionfilter 
       begin="0" 
       dur="3" 
       type="starWipe" 
       subtype="fivePoint" 
       mode="in" />
</t:par>

Listing 1. Applying a star wipe effect to an image

The complete file is available (contains the template with listing 1 in the body). 

As shown in Listing 1, using basic transitions is a straightforward matter of filling in the values of self-descriptive attributes. Simply include the t:transition element, set  the begin, dur or end attributes, the type of effect you want, a subtype if desired, and whether you want the effect mode to be in or out (from less visible to more visible).  The complete Microsoft reference to HTML+TIME transitions can be found on Microsoft's site.

The transition applies to the containing SMIL element (like in Listing 1) or to the element with the id specified in the attribute targetelement. The available types and subtypes of transition are summarized in Table 1. Try playing with Listing 1 to see the different effects.

Condensation Project

Star effects and spiral wipes can be nice eye candy, but the most useful transition is fading, especially when you want transitions as a way to implement condensation.

The goal is to reduce the playtime of an embedded video by condensing long similar parts. To achieve a smooth effect both visually and mentally we will use fade effects.  The video we will cut is a 2-minute chase segment from the movie "Monkey Business" by the Marx Brothers. The idea is to condense most of the content and present a 30 second version of the clip, while still conveying the notion of an ongoing chase. While reducing this clip hurts my Marxist Brothers sensibility, the example is a realistic project for sites dealing with video: for example, DVD stores showing clips of different durations depending on the user's preferences.

The Media

In order to insert the media, we use the t:video element as shown in Listing 2. 

<body background="groucho.gif">

   <div style="position:absolute; top:10px; left:13px;
           width:350px; height:200px;
           font-family:'Haettenschweiler','Verdana';
           font-size:16pt;" class="time">

          <b>The 4 Marx Brothers in MONKEY BUSINESS</b><br/>
          ©1931 Paramount pictures ©1958 EMKA LTD.<br/>

          <t:video id="monkey" src="monkeybusiness.avi"/>
    </div>
</body>

Listing 2. Inserting the media

More interesting than the t:video element itself is the fact that it is inside a div container, which is SMIL-enabled by virtue of including a class attribute with the value time (declared as seen in the purple section of HTML+TIME 2.0 above). The t:video element would have inserted the video even if the div container was not SMIL enabled; but it was  important to illustrate the enabling technique: we will be able to play not only with transitions in the t:video element but also on the whole div area.

The Transitions

The fist transition filter we will use is a fade out right after the chase begins, that is, about 9 seconds into the action.

<t:transitionfilter begin="9" id="firstfade"
                    dur="3"  
                    type="fade" mode="out"/>

The second transition filter completes the condensation by fading in on a particular frame, about 2 minutes after the fade out. This is done by using the seekToFrame method of the t:video object as soon as the transition begins. Figure 2 shows snapshots of the result so far.

<t:transitionfilter begin="firstfade.end" id="secondfade" 
                    dur="6"
                    type="fade" mode="in"
                    onbegin="monkey.seektoframe(425)" />

Figure 2. Snapshots of Condensation using two fade transitions

Finally, to end the clip elegantly and leave the notion of continuity (an eternal Marx Bros. chase!), we end with another fade out:

<t:transitionfilter begin="secondfade.end+23" dur="10" 
     type="fade" mode="out">

Putting it all together, our resulting document looks like listing 3. The effect is not only  successful but even somewhat impressive if you consider the alternative: editing the video itself.

<html xmlns:t ="urn:schemas-microsoft-com:time">
  <head>
      <style>
         .time {behavior: url(#default#time2);}
      </style>
   <?import namespace="t" implementation="#default#time2">
 </head>
 <body background="groucho.gif">
   <b style="font-family:'Haettenschweiler','Verdana';font-size:16pt;">
   The 4 Marx Brothers in MONKEY BUSINESS</b><br/>
   ©1931 Paramount pictures ©1958 EMKA LTD.<br/>

  <div style="position:absolute; top:70px; left:200px; 
              width:350px; height:200px;"       
      class="time">
     <t:video id="monkey" src="monkeybusiness.avi"/>
     <t:transitionfilter begin="9" id="firstfade" dur="3" 
                         type="fade" mode="out"/>
     <t:transitionfilter begin="firstfade.end" id="secondfade" dur="6" 
                         type="fade" mode="in"
                         onbegin="monkey.seektoframe(425)" />
     <t:transitionfilter begin="secondfade.end+23" dur="10" 
                         type="fade" mode="out">
  </div>
 </body>
</html>

Listing 3. Complete Code for Monkey Business Condensation

Let's look at of the most notable advantages of using SMIL for this task :

• You don't need to change your video. You don't even need control over the video to condense it. It can be fetched from another site and it still can be tailored to your needs. It is all done at presentation time.

• You don't need video editing software for simple coordination tasks  like this, a simple text processor would do.

• Automating the production of HTML files like Listing 3 is considerably easier than automating the production of different videos for different user preferences. 

• The results are scriptable at runtime, everything from the source movie to the speed of the transition can be changed dependent on bandwidth demands or preferences. Furthermore, the scripting technologies involved are ordinary JavaScript over the DOM, something most web developers are somewhat familiar with. You can leverage existing knowledge.

• The code is simple and highly reusable. The code above is basically a general solution for simple post-production  web video condensation.

The main problem in this and all other HTML+TIME examples is the fact that they currently work only on Internet Explorer. Another important issue is the scarcity of tools (editing the video to do the same fade in is a matter of four seconds in an editing tool like Adobe Premiere).

Focusing attention using animate

The synecdoche is a narrative tool wherein you substitute a part for the whole. As a form of speech you can recognize it in phrases like "you should see my new wheels", where the wheels (a part) is used to refer to a car (the whole). As a visual narrative strategy you can recognize it in images that isolate a particular detail to signify a whole event, like a hand shooting a gun in a Hitchcock movie encapsulating the whole act of a murder.

Zooming into details, in general, serves as a way of concentrating the attention of the user and disambiguating the object of attention. Think for example of an online gallery that provides analysis of the exhibited works (see Figure 3). Wouldn't it be nice if, instead of trying to describe in the text the position of a detail in a painting, users could simply zoom in an out of details as the explanation progresses? 

Figure 3. Static Images with verbose textual directions vs. Progressive Zoom

In Figure 3 a progressive zoom into the details, instead of verbal directions, focuses the reader's attention, disambiguates, and clarifies the object of discussion. It also enables the text to focus only on the explanation and analysis.  In this section you will see how to create such zoom effects over existing media, such as a JPEG image, in a declarative way using SMIL. 

Implementing a zoom

The goal of this section is to generalize the tool discussed in the previous section and create a reusable declarative solution to create progressive zooms of images in HTML pages (without GIF89 animation, Flash, or even SVG). The guiding example we will construct is a series of hypothetical online lessons that explain different parts of a photograph (see Figure 4).

Figure 4. Snapshots of the final result at 1.0 sec and 6.0 sec

Strategy and Preliminaries

The idea is to maximize the versatility of a generic XHTML+SMIL zoom effect, so the strategy on this project will be to play with the margins and dimensions of a picture using the t:animate element. Although the syntax of the t:animate element and its partners t:animateColor and t:animateValue is intuitive, the values in the zooming application may not be the simplest one to start with, so before jumping into the main example, we will get some hands-on experience with the syntax by colorizing some arbitrary div elements. Figure 5 shows the start and end snapshots of the result of running Listing 4 (complete listing) in Internet Explorer 6. 

<body>
<DIV id="om1" 
     style="width:224; height:184; top: 0; left:74; 
     border-right-width:0px; border-top-width: 0px;">
</DIV>
<DIV id="om2" 
     style="width:215; height: 96; top: 140; left: 74;">
</DIV>
<!-- other divs, one for each rectangle go here -->

<t:animateColor targetElement="om1" 
                attributeName="backgroundColor" 
                from="black" to="#292152" 
                begin="0" dur="3" fill="hold"/>

<t:animateColor targetElement="om2" 
                attributeName="backgroundColor" 
                from="white" to="#ADA5B5" 
                begin="0" dur="3" fill="hold"/>

<!-- other t:animateColor go here -->
</body>

Listing 4 Colorizing divs with animateColor (abridged)

The Media

Back to our main project, the idea is to explain the significance of different parts of an unusual image and in the process create a reusable method to expand explanations with images that are zoomed to show particular details. Our explanation of the image is divided in three lessons, all sharing the same basic layout coded in Listing 4 (and shown in Figure 4), namely a title, a left div with the explanation text, and a right div with the image being analyzed.

<h1>Gordon Coles' Code 101</h1>
<!-- A tribute to David Lynch's Fire Walk With Me -->
<h3>Lesson 2: The Fist</h3>

<!-- undecorated left div for the explanation -->
<div style="width:180; height:180; top: 70; left:10; position: absolute">
 The authorities will be belligerent, that is why Lil's hand becomes a fist.
</div>

<!-- Decorated right div for the image. dashed gray border -->
<div style="width:240; height:180; overflow-x:hidden; overflow-y:hidden;
           top: 60; left:200; position: absolute; border-style: dashed; 
           border-color: gray; z-index:1;" >

<t:img src="lili.jpg" id="oSour" style="width:240; height:180"/>

</div>

Listing 5. Static Explanation

Listing 5 doesn't show some aesthetic details of the complete file  (like the background and font specifications), but it makes clear the simple layout we adopted. One important detail of the code is in the second div, in the value of the overflow-x and overflow-y styles. These styles allow the div to behave as a SMIL region with the fit attribute set to "slice", that is to discard the image information that exceeds the dimensions of the div. This will prove key for our strategy as we animate the size of the image.

The Animation

Now that we have the static explanation in place we will add t:animate elements to pass from the initial image to the zoomed image shown in Figure 4. The t:animate element is similar to t:animateColor which we saw in Listing 4, but unlike it or its partner animateMotion, t:animate has no special semantics associated with any specific attribute (like color and motion) and can be used to animate any scalar property. We will use t:animate to animate the width and height properties of the image, as shown in Listing 6:

<t:animate begin="1" dur="5" fill="hold" targetElement="oSour" 
           attributeName="width" from="240" to="1000"/>

<t:animate begin="1" dur="5" fill="hold" targetElement="oSour" 
           attributeName="height" from="180" to="750"/>

Listing 6 Animating width and weight with t:animate

The animate elements of Listing 6 begin acting one second after their container's timeline has started (i.e. since the div is displayed), they act for 5 seconds, after which the final state is preserved (fill="hold"). Both t:animate elements act simultaneously over the same element, namely the t:img with id="oSour". Finally, by default, the t:animate element iterates the given attribute between the specified values linearly, so the width changes at a steady rate from 240 to 1000 pixels during the 5 seconds the animation lasts. 

In general, any property being animated must be initialized by providing a value in the element's style. That is why our t:img element in Listing 6 specifies the width and height and the complete code of Listing 4 initializes the backgroundColor property for all divs.

So far we have created the size portion of the effect, we have an image of 1000x750 pixels. Now we need to position this image so that the area we are interested in is visible in the div, as shown in figure 6.

Figure 6 Positioning the oversized image

As you can see in Figure 6, the trick is to deal with negative values in the marginLeft and marginTop properties, so we can position the desired portion (in this case the blue rose) in the visible area of the div. The code needed to animate these two properties is as simple as before, and is shown in Listing 7:

<t:animate begin="2" dur="5" fill="hold" targetElement="oSour" 
           attributeName="marginLeft" from="0" to "-770" />

<t:animate begin="4" dur="5" fill="hold" targetElement="oSour" 
           attributeName="marginTop" from="0" to="-504"/>

Listing 7. Animating marginTop and marginLeft with t:animate

Note how the duration and begin times of these elements are similar but not quite the same as those used for the width and height animations. For example, the marginLeft animation starts a second after the image started growing, so we will first see the image "zooming" and then it will start sliding to the right. Since all four animate elements work in parallel, the effects that can be achieved by playing with the start and duration values of each animation are interesting. The three lessons spawned by the buttons below illustrate some of the possibilities, feel free to play with the code and see other options.

Finally, I would like to point out two important things about this implementation:

• This is not the only way to achieve a reasonable zoom effect using HTML+TIME. In particular we could have used the zoom property of the image, but it would have only saved us one element (animating only zoom instead of width and height). We chose the shown implementation because zoom only allows proportional magnification of the image, which is more restricting than our resizing mechanism, and the final effect requires the positioning mechanism of Figure 6 anyway.
• The way start and end values are specified in the t:animate elements in the sample files is simple and straightforward but is not the only way, splines can also be applied to this purpose. 

Spatial Montage and Choreography with Events

The goal of this section is to work further on the idea of events and show how the declarative nature of HTML+TIME 2.0 simplifies event-related tasks. We use two mechanisms: spatial montage and simple sound choreography.

Spatial montage -- the division of the screen area to allow several images to coexist in parallel -- is used in films like "Run, Lola Run" and even 1920's "Napoleon", but its application is most evident in multimedia pieces and the Web. In the following example we will use it to created a "directed reading" where pieces of a picture and a text are enhanced to give "close-ups" to different sections without loosing the overall context. Given the bandwidth considerations of an online article, the media chosen are a black and white picture, text, and small sounds, but the principles can be applied in SMIL 2.0 to heavier media like video.

Choreography of sound and page events has been traditionally associated with JavaScript and proprietary mechanisms, in this example we show a declarative and simple alternative using SMIL 2.

The Problem 

The page we want to create is a brief explanation of the lives of the people in a group photo. The idea is to be able to focus the reader's attention on synchronized details in the text and in the picture, without loosing the overall context that the complete page provides. Figure 7 shows a snapshot of the final result. Whenever a person clicks on a link, a sound (or a combination of sounds) is played. Whenever a person clicks on a person on the picture, all the relevant links are highlighted in the page. You can play around with the final page before jumping to the following sections which deal with the creation of the piece.

Figure 7. Snapshot of the Final Page at Two Stages

The Base SVG Image

Figure 8. Basic Image

The first step is to create the base SVG image, as shown in Figure 8. The file velvet.svg was created in Adobe Illustrator, using one layer for each colored silhouette, the result is not only a reasonably organized SVG file (a named group with a path inside for each layer and CSS for each color effect) but also much smaller than the alternative of creating multiple (seven) flat image files with each character highlighted. We modify the CSS in the source as shown in Listing 8 so all the color layers have an opacity of 0 (invisible) so we start with just the black and white picture.

<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 20000303 Stylable//EN"
   "http://www.w3.org/TR/2000/03/WD-SVG-20000303/
         DTD/svg-20000303-stylable.dtd" [
    <!ENTITY st0 "fill:#C0C311;stroke:none;">
    <!ENTITY st1 "fill:#00D759;stroke:none;">
    <!ENTITY st2 "opacity:0;">
    <!ENTITY st3 "opacity:0">
    <!ENTITY st4 "opacity:0;">
    <!ENTITY st5 "opacity:0;">
    <!ENTITY st6 "opacity:0;">
    <!ENTITY st7 "opacity:0;fill:#000EC0;stroke:none;">
    <!ENTITY st8 "opacity:0;fill:#E990C9;stroke:none;"> ...
]>
<svg  width="599.368pt" height="425.031pt" viewBox="0 0 
     599.368 425.031" xml:space="preserve"> ...

Listing 8. Modified CSS properties for the base SVG Image

Showing/Hiding Pieces - The Declarative way vs The Procedural Way

Before we show how to create the colored effect with SMIL 2, lets see the alternative with procedural JavaScript.

The idea is to show the colored paths as the user places the mouse over each area, so we associate a custom function showWithOpacity to the onmouseover event of each group (that is each g element in the SVG) as shown below:

<g id="Andy" style="&st13;" onmouseover="showWithOpacity(evt,'
     Andy',0.3)">
      <path style="&st14;"
            d="M206.328,397c22-7-1-33,12-48c13-13,21-3,22-29c0-
25-5-47-1-72c2-17,5-28,12-43c14-31,21-54,55-67c9-4,
              ..."/>
</g>

The body of the function is shown in Listing 9.

<script language="Javascript">
 var lastobj = document.getElementById ("nico");

function showWithOpacity(evt,targetId,opacity)
{

var target = evt.getTarget();
var svgdoc = target.getOwnerDocument();
var svgobj = svgdoc.getElementById (targetId);
if(lastobj)
{
  var oldstyle = lastobj.getStyle();
  if (oldstyle)
    oldstyle.setProperty ('opacity', 0);
}

if (svgobj)
{ 
  var parnode = svgobj.getParentNode();
  parnode.removeChild(svgobj); 
  parnode.appendChild(svgobj); 
  var svgstyle = svgobj.getStyle();
  if (svgstyle)
    svgstyle.setProperty ('opacity', opacity);
  lastobj = svgobj;
} 
} 
</script>

Listing 9. Procedural way to vary opacity

The function above not only requires a bit of proficiency with JavaScript but is limited to vary the opacity of the group (in one step, not gradually). If you wanted to add another effect, such as changing the size of the path or rotate it as it gets positioned, you would have to re-write the wheel and write the appropriate loops. This can get tedious and expensive to implement, so for the onclick effect we will change the strategy and rely instead on the animate support of SMIL 2.

With animate, the way we tie events to actions is reversed from the traditional javascript call: instead of saying in the element "execute this every time there is a click", we create the effect and specify we want to "get this executed every time the objectID.click event is fired". The following example (Listing 10) clarifies the point.

<g id="Andy" style="&st13;" onmouseover="showWithOpacity(evt,
     'Andy',0.3)">
  <animate attributeName="opacity" values=".3;1;.4"
    begin="Andy.click" keyTimes="0;0.7;1" dur="1" />
  <path style="&st14;"
    d="M206.328,397c22-7-1-33,12-48c13-13,21-
       3,22-29c0-25-5-47-1-72c2-17,5-28,12-43c14-31,
            21-54,55-67c9-4, ..."/>
</g>

Listing 10. Declarative way to vary opacity

There are at least three main advantages on this approach:

• You can leverage all the power of SMIL 2 transformations, including gradual and non-linear variations of a parameter (as shown) 
• You don't need to code or debug JavaScript
• There can be as many animate elements which begin when Andy.click is fired as you want, they are all coded independently, and the g element needs to know nothing about them.

For CSS connoisseurs who might be wondering why we did not use the visible property instead of opacity in Listing 9, the reason is an element with visibility equal to false does not emit mouse over or mouse click events in the SVG Viewer. 

Highlighting The document 

The highlighting of the document is done through classic CSS. Each link is marked with a class, as shown below, 

<a id="maureen1" href="#" class="maureen">Maureen 
Tucker's delicate drumming</a> 

and the stylesheet is provided in an external document like the following:

.maureen
    { 
     background-color: #855FB9;
     color: white;
        border-style: dashed;
        border-width: thin; 
        border-color: black;
    border-bottom-width= 0;
    }

Declarative Sound Triggers

Finally, there is the nice simplicity of declarative sound triggers in HTML+TIME 2.0. Just as the animations are associated with events in the SVG image, the sounds (namely the t:audio elements) specify their beginning to be tied to the firing of an event as shown in Listing 11. Because of our requirements we associate it with simply clicks, but this mechanism can be replicated for any SMIL 2 event, like the end of a seq element.

<t:audio src="thesedays.wav" begin="nico1.onclick; nico2.onclick;"/>
<t:audio src="hallelujah.wav" begin="cale1.onclick"/>
<t:audio src="andyIntroduces.wav" volume="10"
begin="andy1.onclick; andy2.onclick; andy3.onclick"/>
<t:audio src="wildside.wav" volume="8"
              begin="andy2.onclick; andy3.onclick; lou1.onclick; 
                     lou2.onclick; lou3.onclick; lou4.onclick"/>

Listing 11. Declarative Sound Triggers

The advantages of this mechanism versus, say, embedding proprietary sound plug-ins, include not only the simplicity of the code, but the fact that sounds and events can be combined at will in a simple way, allowing for an event to fire many sounds, or many events to be associated with the same sound. In the example above, the file wildside.wav is played in parallel with andyIntroduces.wav when the user clicks on the andy2 link, which makes for a nice effect of music in the background mixed with dialog in the foreground. In our example we have used only src, volume, and begin; see the documentation for 

You can click on the following button to see the final result.

Summary

This article discussed some of the techniques and mechanisms available to the SMIL 2 developer working on the HTML+TIME 2.0 implementation. The discussion was directed by narrative strategies common to traditional and new media and how to implement them, instead of dry API descriptions. Three main strategies were discussed through the use of three SMIL 2 mechanisms, namely transitions, declarative animation, and SMIL 2 events.  

References

[1] HTML+TIME documentation, Microsoft 2002
[2] SMIL 2.0 Specification  W3C 2002
[3] Envisioning Information Edward R. Tufte; Graphics Press; 10/1990
[4] Understanding Animation Paul Wells, Routledge 1998
[5] The Language of New Media Lev Manovich, MIT Press 2000
[6] The Film Form Sergei Eisenstein, Harcourt 1972