Hacking the XML in Your TiVo

February 15, 2006

When I was a kid, our television set picked up just a few broadcast channels over the wire aerial mounted on our roof. This offered us a limited choice of what to watch, so we saw a lot of I Love Lucy reruns. Now that we have hundreds of channels over the satellite dish, libraries of DVDs and videotapes, Netflix, a great local video store, increasing access to video over the Web, and especially a TiVo, what do we save up on the TiVo to watch together as a family besides The Simpsons and Monty Python's Flying Circus? Old I Love Lucy episodes.

We don't watch every recorded episode. The TiVo offers various options for finding out the episode titles and plot summaries of what it recorded, so that we can find Simpsons episodes we've never seen and alert our kids to particularly classic Monty Python and I Love Lucy episodes. The simplest way to see what you have is to use your TiVo remote to bring up the Now Playing menu. Things get more interesting when your TiVo—which is ultimately a Linux box with a big hard disk and a timer-triggered channel changer—has a wireless network adapter that connects to your home network. An HTTP server built into the TiVo lets you browse web page versions of TiVo metadata from a browser on the same network, and a RESTful API lets you pull XML versions of the same information. With some short stylesheets and a little help from the free wget utility, I wrote a simple application that puts onto my weblog a "TiVoRoll" of what shows our TiVo's been recording, and another app that gives me an Atom feed showing which episodes have been recorded lately.

Before I describe how to do this, I'm making two assumptions:

Your TiVo is on your home network. As the TiVo documentation explains, you just plug a network adapter into the USB port on the TiVo's back panel and run a setup routine. Make sure that you get your network adapter from someplace where you can return it if necessary, because the TiVo can be very picky about these, sometimes accepting and rejecting different adapters of the same make and model due to slight differences in the version number. I found the website tivocards.com to be very helpful.
You're using one of the Series 2 TiVos that have been available for over four years now. For information on great hacks that you can do with both Series 2 and Series 1 TiVos, see the O'Reilly book TiVo Hacks.

To retrieve data from the HTTP server in your TiVo, you first need to find out its IP address. From the TiVo main menu, pick Messages & Settings, Settings, and then Phone & Network to display the Network Settings screen, which will have your IP address. This number was dynamically assigned and may change if you reboot some part of your home network, so don't hardwire this IP address into your code when creating an application around it. The same TiVo screen also shows your MAC ID (formerly the MAK, for Media Access Key), which will be your password to gain access to the TiVo's metadata, so note that as well.

Browsing the TiVo HTTP Server

Let's say that your TiVo's IP address is 192.168.2.103. To see its web server's home page, point a browser on one of your home network's machines to https://192.168.2.103. (Remember to make it https and not http, which will give you a screen that tells you nothing about what you have on your TiVo.) You may get a dialog box saying "Web Site Certified by an Unknown Authority" and one saying "Domain Name Mismatch," and then you'll be prompted for a username and password. Your username is "tivo", and your password is the MAC ID that you copied from your TV screen when you got the TiVo's IP address. The default web page redirects your browser to https://192.168.2.103/nowplaying/index.html, which summarizes the shows you have recorded:

TiVo web server home page

Clicking "folder" on the right lists individual episodes of a show that you have recorded, and clicking "classic" at the bottom lists all the individual episodes on the TiVo hard disk.

The third method for finding out about recorded episodes, besides the web page interface and using the remote to display the Now Playing list on your television, is the most interesting: with a URL full of parameters that tell the TiVo's HTTP server to give you an XML version of the information. Pointing your browser to the URL

https://192.168.2.103/TiVoConnect?Command=QueryContainer&Container=%2FNowPlaying&Recurse=Yes

(with your own TiVo's IP address substituted if necessary) displays something like this:

Browser displaying XML of TiVo "Now Playing" information

The parameters in this URL are documented in the TiVo Home Media Option Music and Photos Server Protocol Specification, an Acrobat file. The free Home Media Option lets your TiVo display pictures and play music stored on another computer on your home network. Its API documentation mostly describes how to retrieve XML about your picture and music files (as well as the files themselves, as we'll see), but by specifying the appropriate Container parameter, as shown in the URL above, you get Now Playing metadata. The protocol specification describes other commands besides QueryContainer, but none looked particularly interesting to me. For example, QueryServer tells you the version of the TiVo server and nothing else.

Command-line Retrieval of TiVo XML

The XML retrieved with this interface doesn't do us much good in a browser. To do something interesting with it, we want to pull the XML down from the server and pass it to an XSLT stylesheet that transforms and stores the result in a file that can be passed to the next step in our application. Remember, though, about the security dialog boxes: if you used the browser to retrieve the XML version without trying the HTML example first, you had to go through the same security dialog boxes, and a non-browser tool for retrieving the XML must negotiate the same issues.

The GNU wget tool, which is available for Windows and included with most Linux distributions, makes this easy. In the words of its home page, it's a "free software package for retrieving files using HTTP, HTTPS and FTP, the most widely used Internet protocols. It is a non-interactive command-line tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc." (Fans of curl will find that it can pull XML out of their TiVos as well.)

Passing a URL as a single parameter to wget pulls down the named file to your hard disk. For example, entering the following code downloads Google's current index.html file to your current directory:

wget http://www.google.com

Dozens of optional command-line switches let you customize wget's behavior for more complex situations such as negotiating the security checks that your TiVo puts in the way of a straight HTTP request. The --http-user and --http-password parameters are self-explanatory. We'll supply the word "tivo" and the MAC ID for these values, as we did when we pointed a browser at the TiVo box and it asked for a username and password. To get past the "Web Site Certified by an Unknown Authority" warning, we'll add the --no-check-certificate switch. The -O switch is handy for setting a specific output filename, because while wget had an easy enough time figuring out that Google's homepage was in a file named index.html, the filename that it infers from a REST URL full of ampersands and percent signs can get messy. The following wget command line works identically under Windows and Linux:

wget --no-check-certificate --http-user=tivo --http-password=your-MAK-here
 -O nowplaying.xml "https://192.168.2.103/TiVoConnect?
Command=QueryContainer&Container=%2FNowPlaying&Recurse=Yes"

(This single command is split into three lines to fit here. When rejoining the pieces, make sure to put a space after your http-password value but none after the question mark after TiVoConnect.) This command downloads the list of recorded episodes and stores them in the nowplaying.xml file. The quotes around the URL may not be necessary, but they make things a little less confusing for the Linux shell, where ampersands indicate that a job should run in the background.

The Recurse=Yes parameter in the URL tells the TiVoConnect application to go through the entire tree of information in the NowPlaying container, which is why it gets information about all the individual recorded episodes. With Recurse set to No, it lists only those shows whose episodes the TiVo records, as shown in the first illustration above. This is what we want for our TiVoRoll:

wget --no-check-certificate --http-user=tivo --http-password=your-MAK-here
 -O tivoroll.xml "https://192.168.2.103/TiVoConnect?
Command=QueryContainer&Container=%2FNowPlaying&Recurse=No"

Once wget has pulled down an XML version of the information we want, we can use XSLT to convert it to the format we want. For the TiVoRoll added to my weblog, the stylesheet is pretty short and simple:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:t="http://www.tivo.com/developer/calypso-protocol-1.5/"
                xmlns="http://www.w3.org/1999/xhtml"
                exclude-result-prefixes="t"
                version="1.0">

  <xsl:output omit-xml-declaration="yes"/>

  <xsl:template match="t:TiVoContainer">
    <div class="tivoroll">
      <p>
        <xsl:apply-templates/>
      </p>
    </div>
  </xsl:template>

  <xsl:template match="t:Item/t:Details/t:Title">
    <xsl:value-of select="."/><br/>
  </xsl:template>

  <xsl:template match="text()"/>

</xsl:stylesheet>

All text nodes are suppressed because the second template rule uses an xsl:value-of instruction to get the text that it needs: the name of each show being recorded.

Note that everything in the TiVo source XML file is in the http://www.tivo.com/developer/calypso-protocol-1.5/ namespace declared at the beginning of that file, so the stylesheet must declare that before referencing specific elements in the source document.

Using the XML that I pulled off of my TiVo just now as a source document, the stylesheet above created the following result:

<div xmlns="http://www.w3.org/1999/xhtml" class="tivoroll">
  <p>Monty Python's Flying Circus
  <br/>I Love Lucy
  <br/>Daria
  <br/>The Colbert Report
  <br/>The Daily Show With Jon Stewart
  <br/>The Simpsons
  <br/>NHL Hockey
  <br/>
  </p>
</div>

Once I FTP this to the appropriate directory on my weblog's server, a Movable Type template's $MTInclude instruction incorporates it into the right-hand column of my weblog's index page.

Let's review the steps by looking at the Linux shell script that drives it. (To run it yourself, first substitute your appropriate values for your-TiVoIP-here and your-MAK-here in the shell script. This zip file stores all the scripts and stylesheets used to make this happen. It includes Windows batch file equivalents of the tivoroll.sh and nowplaying.sh files.)

#! /bin/sh

export TiVoIP=your-TiVoIP-here
export MAK=your-MAK-here

# pull the TiVoRoll XML down from the TiVo
wget --no-check-certificate --http-user=tivo --http-password=$MAK \
-O /tmp/tivoroll.xml "https://$TiVoIP/TiVoConnect?Command=QueryContainer\
&Container=%2FNowPlaying&Recurse=No"

# Convert it to HTML markup
xsltproc -o /tmp/tivoroll.html tivoroll2p.xsl /tmp/tivoroll.xml

# FTP the result to a public web server
ftp -n snee.com < tivoroll.l.ftp

The wget utility pulls down the XML we want, the libxslt XSLT processor xsltproc creates HTML from the XML, and ftp uses a script to automate the process of putting the HTML into the right directory on the snee.com server where my weblog resides.

My application that creates an Atom feed of the Now Playing list of saved episodes is almost identical to the one that creates a TiVoRoll for my weblog. The driver shell script passes a Recurse=Yes version of the URL to wget to get the details about the stored episodes, then calls xsltproc with an XSLT stylesheet that creates an Atom 1.0 file, and it finishes by using an FTP script that puts the result of the XSLT pass into a different directory on the web server (http://www.snee.com/rss/nowplaying.atom, in case you're interested in seeing what's on our TiVo). A cron job on my home Linux box runs both scripts before I get up each morning.

Querying the TiVo Desktop

The TiVo Desktop is a Windows program that, when run on your home network, lets you use your TiVo to display pictures and play music on your TV that it retrieves from the computer running the TiVo Desktop. Computers on your network can talk to the TiVo Desktop using URLs like those shown above to find out what pictures and music are available.

Because you'll be sending your queries to the Windows machine and not to the TiVo, you must first find out its IP address. The ipconfig command-line utility that comes with Windows shows you this and related information. For mine, it was 192.168.2.102, so the following URL retrieved XML that listed the high-level containers:

http://192.168.2.102:8080/TiVoConnect?Command=QueryContainer

(Note that the URL scheme prefix is http and not https and that a port number of 8080 is included in the URL.) Here is the XML that it retrieved:

<?xml version="1.0" encoding="ISO-8859-1" ?>
<TiVoContainer>
  <Details>
    <Title>BLACKDELL</Title>
    <ContentType>x-container/tivo-server</ContentType>
    <SourceFormat>x-container/folder</SourceFormat>
    <TotalItems>2</TotalItems>
  </Details>

  <Item>
    <Details>
      <Title>Bob's Music on BLACKDELL</Title>
      <ContentType>x-container/tivo-music</ContentType>
      <SourceFormat>x-container/folder</SourceFormat>
      <LastChangeDate>0x43BB4121</LastChangeDate>

    </Details>
    <Links>
      <Content>
        <Url>/TiVoConnect?Command=QueryContainer&Container=%2FTivoMusic</Url>
        <ContentType>x-container/tivo-music</ContentType>
      </Content>
    </Links>

  </Item>
  <Item>
    <Details>
      <Title>Bob's Photos on BLACKDELL</Title>
      <ContentType>x-container/tivo-photos</ContentType>
      <SourceFormat>x-container/folder</SourceFormat>
      <LastChangeDate>0x43D40340</LastChangeDate>

    </Details>
    <Links>
      <Content>
        <Url>/TiVoConnect?Command=QueryContainer&Container=%2FTivoPhotos</Url>
        <ContentType>x-container/tivo-photos</ContentType>
      </Content>
    </Links>

  </Item>
  <ItemStart>0</ItemStart>
  <ItemCount>2</ItemCount>
</TiVoContainer>
<!-- Copyright (c) 2003-2005 TiVo Inc. All rights reserved.-->

I had given that computer the name of blackdell to distinguish it from our older, off-white Dell that is now running Ubuntu Linux, which is the machine I used to test these queries. Because I had shared the My Music and My Photos folders from my "Bob" account on the blackdell computer, the two containers listed by the URL above had titles of "Bob's Music on BLACKDELL" and "Bob's Photos on BLACKDELL." Below these titles you can see relative URLs that point to the contents of these containers. The following URL (split into two lines here) showed that the TivoPhotos container had containers named My Photos.lnk and My Pictures.lnk:

http://192.168.2.102:8080/TiVoConnect?Command=QueryContainer
&Container=%2FTivoPhotos

Those have their own URLs too, and using those and the same technique you can drill down to your actual listings of photos. These image files will have URLs that let you retrieve them; for example, the following URL retrieved I102_0248.JPG and displayed it on the Ubuntu machine's screen:

http://192.168.2.102:8080/TiVoConnect/TivoPhotos/IMy%20Pictures.lnk
/F326/F466/I102_0248.JPG

The API documention lists additional parameters to control the display of the image. For example, this URL displays the I102_0248.JPG image rotated 90 degrees:

http://192.168.2.102:8080/TiVoConnect/TivoPhotos/IMy%20Pictures.lnk
/F326/F466/I102_0248.JPG?Rotation=90

The ability to see directory listings and to retrieve JPEG and MP3 files from one home network machine to another is not in itself exciting. It's ironic that the documentation explaining how do to this with the TiVo Home Media Option API holds the key to querying the TiVo box itself, which is much more interesting. The fact that we can do this with a REST API makes it simple to play with, and makes it that much easier to script the integration of your personal TiVo data into the other metadata of your life.