ITunesDB/iTunesDB File

From wikiPodLinux

iTunesDB file

The iTunesDB file consists of a sort of tree structure arranged into a flat file. Each "object" contains a header followed by some other objects. The header contains a couple of size fields that are used to define where the header ends and other objects begin. Here's the basic structure of it:

<mhbd> - This is a database
  <mhsd> - This is a list holder, which holds either a mhla
    <mhla> - This holds a list of albums
      <mhia> - This describes a particular Album Item
        <mhod> - These hold strings associated with an album title
        <mhod> - These hold strings associated with an artist name/title
      <mhia> - This is another album. And so on.
        <mhod>...
        <mhod>...
      ...
  <mhsd> - This is a list holder, which holds either a mhlt or an mhlp
    <mhlt> - This holds a list of all the songs on the iPod
      <mhit> - This describes a particular song
        <mhod>... - These hold strings associated with a song
        <mhod>... - Things like Artist, Song Title, Album, etc.
      <mhit> - This is another song. And so on.
        <mhod>...
        <mhod>...
      ...
  <mhsd> - Here's the list holder again.. This time, it's holding an mhlp
    <mhlp> - This holds a bunch of playlists. In fact, all the playlists.
      <mhyp> - This is a playlist.
        <mhod>... - These mhods hold info about the playlists like the name of the list.
        <mhip>... - This mhip holds a reference to a particular song on the iPod.
        ...
      <mhyp> - This is another playlist. And so on.
        <mhod>... - Note that the mhods also hold other things for smart playlists
        <mhip>...
        ...
      ...

And so on. What follows is a detailed description of these various headers.

Chunk Encoding

Each chunk of the file is encoded in the follow form:

The Chunk Type Identifier is intuitive: it's a 4-byte string letting you know what you're working with.

The End of Type-specific Header tells you where, relative to offset 0, the header section for this chunk ends. The header starts at offset 12 and runs through to the end of the type-specific header.

The End of Chunk or Number of Children is slightly confusing. Usually, it's an "End of Chunk" offset: what the last offset in the current chunk is. That is, it tells you how long this chunk and all its children are. There are two big exceptions to this: "mhlt" and "mhlp" chunks. In both of these, this number is how many top-level children the mhl[pt] chunk has.

Database Object

The Database Object has two or three children, which are Data Sets.

DataSet

1 = Track list - contains an MHLT
2 = Playlist List - contains an MHLP
3 = Podcast List - optional, probably. Contains an MHLP also.  
This MHLP is basically the same as the full playlist section, 
except it contains the podcasts in a slightly different way. 
See the Playlists section.
4 = Album List, first seen with iTunes 7.1.
5 = New Playlist List with Smart Playlists, first seen with iTunes 7.3.

Depending on the type of Data Set, it will contain either a Track List child or a Playlist List child. Order is not guaranteed. Example files have contained the type 3 MHSD before the type 2 MHSD. In order for the iPod to list podcasts the type 3 Data Set MUST come between the type 1 and type 2 Data Sets.

TrackList

The Track List has Track Items as its children. The number of Track Items is the same as the number of songs.

Track Item

The MHIT is followed by several Data Objects which have string types. At minimum, it must have a Location type MHOD, in order to tell the iPod where the file is located on the iPod itself. It always has a FileType MHOD as well, although it's not totally necessary.

Playlist List

The Playlist List has all the playlists as its children. The very first playlist *must* be the Library playlist. This is a normal playlist, but it has the special "hidden" bit set in it, and it contains all the songs on the iPod (in no particular order).

Playlist

The structure of the Playlists are different than most others. Each Playlist looks like this, conceptually:

   <mhyp>
      <mhod type=1>Playlist Name</mhod>
      <mhod type=50>Smart Playlist Info</mhod> (optional)
      <mhod type=51>Smart Playlist Rules</mhod> (optional)
      ...
      <mhip>Playlist Item</mhip>
             <mhod type=100>Position Indicator</mhod>
      <mhip>Playlist Item</mhip>
             <mhod type=100>Position Indicator</mhod>
      ...
   </mhyp>

The point being that these are all considered in the MHYP for the size calculations. However, in the "Data Object" child count, ONLY those MHODs that come before the first MHIP are counted. The "position indicators" are children of the MHIPs (older firmwares had a bug in this respect).

List Sort Order

1 - playlist order (manual sort order)
2 - ???
3 - songtitle
4 - album
5 - artist
6 - bitrate
7 - genre
8 - kind
9 - date modified
10 - track number
11 - size
12 - time
13 - year
14 - sample rate
15 - comment
16 - date added
17 - equalizer
18 - composer
19 - ???
20 - play count
21 - last played
22 - disc number
23 - my rating
24 - release date (I guess, it's the value for the "Podcasts" list)
25 - BPM
26 - grouping
27 - category
28 - description
29 - show
30 - season
31 - episode number

Podcasts

The podcasts playlist is organized slightly differently, in the Type 3 MHSD. For example:

mhyp (MHODs: 2, MHIPs: 5, hidden: 0, list sort order: 0x18)
  mhod (type: 1, string: 'Podcasts')
  mhod (type: 100)
  mhip (MHODs: 1, groupflag: 256, groupid: 8232, trackid: 0, timestamp: 0, groupref: 0)
    mhod (type: 1, string: 'Example podcast')
  mhip (MHODs: 1, groupflag: 0, groupid: 8233, trackid: 8230, timestamp: 3206828281, groupref: 8232)
    mhod (type: 100)
  mhip (MHODs: 1, groupflag: 0, groupid: 8234, trackid: 8226, timestamp: 3206828379, groupref: 8232)
    mhod (type: 100)
  mhip (MHODs: 1, groupflag: 0, groupid: 8235, trackid: 8228, timestamp: 3206828327, groupref: 8232)
    mhod (type: 100)
  mhip (MHODs: 1, groupflag: 0, groupid: 8236, trackid: 8224, timestamp: 3206828394, groupref: 8232)
    mhod (type: 100)

The first mhip (probably identified by groupflag==256) contains the name of a podcast which will then appear as a submenu of the Podcasts menu on the iPod. The other mhips (which reference the episodes in that podcast) seem to use the groupref field to link to their 'parent' mhip (using the groupid field). At the same time the groupids of the episodes are unique as well, but don't seem to be used anywhere else.

Playlist Item

For purposes of size calculations, Playlist Items have no children. However, every Playlist Item is invariably followed by a Data Object of type 100, which contains nothing but a number that is used to order/sort the playlist. See the Playlist description above for more information.

Please note that starting with iTunes 4.9 (mhbd file version number 0x0d) the Type 100 MHOD following the Playlist Item is considered a child of the Playlist Item and is included into the size calculation. The old behaviour was probably a bug in iTunes.

Data Object

The Data Object is a complex beast. It's used in many places in the iTunesDB file, and there are many forms of it. We'll cover the forms one by one.

String Types

The simplest form of MHOD. These are any MHOD with a "type" that is less than 15.

Podcast URLs (types 15-16)

Introduced in db version 0x0d, MHOD's with type 15 and 16 hold the Enclosure and RSS URL for the Podcast. The string is probably UTF-8, but only Unicode symbols U+0000 through U+007F (a.k.a ASCII) have been observed.

Chapter Data (type 17)

The chapter data is a style of data that's very, very out of place in this file, but is here nonetheless. It defines where the chapter stops are in the track, as well as what info should be displayed for each section of the track. It seems that the format of this mhod changed significantly over time. The following is analysed from an iTunesDB version 0x13 (iTunes 7.0).

Chapter Data mhod details

Chapter Data Atom Layout

The atoms appear to be arranged in the following layout:

sean (exactly one atom of this type is present,
        childcount is number of chapters + 1 for the hedr atom)
   chap (chapter indicator, childcount is always 1)
      name (contains a UTF16 string with name of the chapter or Artist name, 
            childcount is always 0)
   chap (next chapter)
      name (and so on)...
   ...
   hedr (signals the end, childcount is always 0)

There are multiple "chap" entries, one for each chapter.

Older DBs seem to have a different, more complex layout, like this:

sean (?)
 chap (chapter indicator ?)
  name (contains a UTF16 string with name of the chapter or Artist name)
  ploc (position/location?)
   trak (specifies track number, perhaps? This should be obvious after looking at some examples)
  urlt (contains a UTF16 string with the name of the song in the chapter or some kind of subname)
  url  (contains a UTF8 string with a URL)
 chap (next chapter)
  name (and so on)...
 ...
hedr (signals the end)

Chapter Data Atoms

Each atom consists of the following:

Caution: This is different to all other structures in the file, where the
         name always comes first. Here, the length is the first word of the atom!

Chapter Data String Atoms (UTF16)

UTF16 String entries in these atoms (like 'name') fit the following mold:

Chapter Data String Atoms (UTF8)

UTF8 entries fit the following mold:

Chapter Data hedr Atom

Smart Playlist Data (type 50)

A slightly more complex MHOD. These are any MHOD with a "type" that is 50.

This MHOD defines the stuff in the Smart playlist that is not the "rules". Basically all the checkboxes and such. It's pretty straightforward.

When the Reverse Limit Sort flag is set, the sort will be reversed. So most recently added becomes least recently added, and highest rating becomes lowest rating, and so on. It's just reversing the sorted list before applying the limit to it.

Smart Playlist Rules (type 51)

The most complex (and annoying) MHOD. These are any MHOD with a "type" that is 51.

This MHOD defines all the rules in the Smart Playlist. It is not straightforward at all. In fact, I have to break it down into subtypes itself.

If you've used iTunes, you know what a rule looks like. "Rating is less than 3 stars", for example. The rule consists of three parts.

The first part is the "field". In our example, the field is "Rating". It's the very first thing in the rule and the first pull down box in iTunes. Simple enough.

The second part is the "action". In our example, the action is "is less than". In iTunes, it's the second pull down box. Also pretty simple.

The final part is the "value". In our example, it'd be "3 stars". Since stars are always multiplied by 20 for whatever reason, it'd be "60" for our example, in the iTunesDB file.

We'll cover these one at a time.

Important Note about endian-ness

Smart Playlist Rules MHODs are NOT wholly little-endian. Everything after the "SLst" to the end of the MHOD is big-endian. This is important to remember, especially when dealing with the Action types.

Smart Playlist Rule Fields

Fields are easy. Much more so than Actions. The "expected comparison" column will make more sense when you see the Values explained below.

Special Note about Grouping and Playlist fields - They don't work with Live Updating on 3rd gen iPods, yet. This might get fixed in a future firmware release. Maybe.

Smart Playlist Rule Actions

The Action type is a 4 byte field. It is a bitmapped value, meaning that each bit of these four bytes has a different meaning.

This is complicated, obviously, and iTunes abstracts this all by defining various actions for various fields. Here's the complete possible list of actions, including those that iTunes does not have but which still work on the iPod:

Smart Playlist Rule Values

Values are generally pretty straightforward. For String rules, it's a string. For Integer and Timestamp rules, it's a bit more complicated. Furthermore, the "In the Last" type action requires a Units value as well.

So there are two major rule formats, the String Rule and the Non-String Rule.

For integer type rules, the from and to values are the ones that you care about, the date = 0, and the units = 1.

Example: BPM is less than 150
field = 0x23 (BPM)
action = 0x00000040 (is less than)
from and to value = 150
from and to date = 0
from and to units = 1

Example: BPM is in the range 70 to 150
field = 0x23 (BPM)
action = 0x00000100 (is in the range)
from value = 70
to value = 150
from and to date = 0
from and to units = 1

For binary and type rules, the from and to values are the ones that you care about, the date = 0, and the units = 1.

Example: Video Kind is TV-Show
field = 0x3c (Video Kind)
action = 0x00000400 (Is / Is Not)
from and to value = 0x0040
from and to date = 0
from and to units = 1

Example: Video Kind is not TV-Show
field = 0x3c (Video Kind / mediatype)
action = 0x00000400 (Is / Is Not)
from and to value = 0x0e22
from and to date = 0
from and to units = 1

For the latter one would expect a value of 0x0022 (either Movie or Music Video). The additional 0x0e00 in the mask hints for further video types to come.

Timestamp type rules use the same format as integer type rules, only the from and to values are Mac timestamps. This is the number of seconds since 1/1/1904.

Example: Date Added is in the range 6/19/2004 to 6/20/2004
field = 0x10 (Date Added)
action = 0x00000100 (is in the range)
from value = bcfa83ff (6/19/2004) 
to value = 0xbcfbd57f (6/20/2004)
from and to date = 0
from and to units = 1

For "in the last" type rules, the from and to values are set to a constant of 0x2dae2dae2dae2dae. The from dates become the value, and the from units becomes the unit you're measuring in. The way to think of this is that it's saying "Today (2dae) plus this number of units".

Example: Last Played Time is in the last 2 weeks.
field = 0x17 (Last Played Time)
action = 0x00000200 (is in the last)
from value = 0x2dae2dae2dae2dae
from date = -2
from units = 604800 (number of seconds in a week)
to value = 0x2dae2dae2dae2dae
to date = 0
to units = 1

That rule is saying "Today minus 2 times this number of seconds" which is 2 weeks before "now", whatever now happens to be.

If you're creating your own rules in an iTunesDB, you may find it more convienent to leave the units set to "1" all the time and just put the number of seconds into the date field. This is perfectly acceptable and the iPod can handle it just fine. If you're not sure how to make a particular rule, create it or a similar one in iTunes and put it on the iPod, then examine the iTunesDB file to see how it did it.

But if you're programming the iPod side of things, you need to be able to correctly understand the units field and deal with it accordingly. The best way to do this is to always compare the contents of the field in question with (value+date*unit), and replacing the "value" with the current timestamp ("now") when it is equal to 3,291,618,603,768,360,366 (0x2dae2dae2dae2dae). This will work for *all* integer and timestamp comparisons if done correctly. It's also exactly what the Apple iPod firmware does. It's also why you have to set the time/date on an iPod for these smart playlists to work correctly (3rd gen and up).

Also, remember that all timestamps should be dealt with in Apple format, which is the number of seconds since 1/1/1904.

Putting It All Together

So, now you know how to create a given rule. All that's left is to put all the rules together and shove them into a type 51 MHOD.

Library Playlist Index (type 52)

The type 52 MHOD is only found as a child of the Main Library Playlist. It is an index of the mhit's ordered by the major categories in the Browse menu. The purpose of these mhod's is to speed up the operation of the Browse menu itself. This is how it displays the information so quickly when selecting one of the major categories, it's all presorted for the iPod in these MHOD's.

Note that this MHOD is not mandatory, however the iPod menu system will operate much slower without it (on large libraries), as it will have to build the information provided here on the fly. Therefore it is recommended to build this MHOD for anything more than trivial numbers of songs.

Essentially, every MHIT is numbered from 0 to the total number of songs-1. The type 52 MHOD contains a list of these index numbers using one of the strings contained in these MHIT's, ordered alphabetically.

To build one of these, take all your songs, order them alphabetically by one of these fields, then simply insert the index numbers of the ordered songs into the type 52 mhod.

(Note that the above list roughly matches the limit sort list.. I think that these lists are actually the same in some way.)

Type 53

The type 53 MHOD is only found as a child of the Main Library Playlist.

I don't know how these are associated with the corresponding MHOD52s, I would guess by the sort order field, but I'm not sure because they always turn up immediately after their MHOD52. Could someone please have a look at this and the other yet unknown things above?

Type 100

There are a couple forms of Type 100 MHOD's (that we know of).

The first form is at the beginning of every playlist (before any MHIP entries). Only iTunes puts it there, and only iTunes uses it. It contains information on what columns to display, and what size to display them as, when displaying the playlist in iTunes when the iPod is in a manual mode. This is absolutely optional. The iPod itself doesn't appear to use it in any way.

The second form is after every MHIP in a playlist, there's a small type 100 MHOD. It gives an order number to the song immediately before it. It is not optional, but it's also not particularly useful.

Playlist Column Definitions Header

Column Definition

Each column definition only consists of an ID for the playlist and the sort direction for the column. The order they appear in this MHOD are the order they appear in iTunes, from left to right. The first two columns are always song position and title, in that order.

Playlist Order Entry

Album List

Album List, first seen with iTunes 7.1. It was seen on iPod nano 2nd gen and Shuffle 2nd gen restored with iTunes 7.1.1, but not on iPod Video 30GB(restored with the same version of iTunes).

The Album List has Album Items as its children. The number of Album Items is the same all albums on iPod.

Album Item

Usually mhia has two child strings: album title and artist name.