Composite Manifest Support for Rough Cut Editing scenarios in SSME

Posted: Jan 22, 2010  2 comments  

Average Rating

Tags
Media
Smooth Streaming Media Element
Smooth Streaming Player Development Kit
SSME

Share this Post

We released the Beta 2 for the IIS Smooth Streaming Player Development Kit (SSPDK) that contains the SmoothStreamingMediaElement (SSME) interface. With this release we added a fun new feature. We refer to this as Composite Manifest Support for Rough Cut Editing. The idea is simple:
  • I have many clips but want to create a composite clip taking relevant portions from these clips.
  • I have a large clip from an NBA game, I want to create a highlight for this game.

Now, let’s expand this simple idea. Not only do I want the above, I also want the ability to present this as a single stream to the user. This means the user can fast forward, rewind across the entire highlights and seek to any points in the highlight. In addition, I can add markers and present useful information to the user watching the stream. Well, this is exactly what we enabled with the Beta 2 release of SSPDK.

Composite Manifest

At the heart of this feature are small updates to the existing IIS Smooth Streaming Client manifest structure and we this new manifest a composite stream manifest (.csm). If you have used Smooth Streaming, you might already be familiar with the client manifest structure. Here is what it looks like before this feature:

Code Snippet
  1. <?xml version="1.0" encoding="utf-16" ?>
  2. <!-- Created with Expression Encoder version 3.0.1332.0-->
  3. <SmoothStreamingMedia MajorVersion="2" MinorVersion="0" Duration="300000000">
  4.   <StreamIndex Type="video" Chunks="15" QualityLevels="8" MaxWidth="640" MaxHeight="480" DisplayWidth="640" DisplayHeight="480" Url="QualityLevels({bitrate})/Fragments(video={start time})">
  5.     <QualityLevel Index="0" Bitrate="1644000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBF213F0EF8A13F83BE80C9081B22B6457400000010E5A67F840" />
  6.     <QualityLevel Index="1" Bitrate="1241000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBE613F0EF8A13F83BE80C9081A5DECBBE400000010E5A67F840" />
  7.     <QualityLevel Index="2" Bitrate="937000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBDC13F0EF8A13F83BE80C90811C97F260C00000010E5A67F840" />
  8.     <QualityLevel Index="3" Bitrate="708000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBD60D509F8A0D5827E80C9081159AD66CC00000010E5A67F840" />
  9.     <QualityLevel Index="4" Bitrate="534000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBD00D509F8A0D5827E80C9081104B412F400000010E5A67F840" />
  10.     <QualityLevel Index="5" Bitrate="403000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBCC0D509F8A0D5827E80C90808C4BE263400000010E5A67F840" />
  11.     <QualityLevel Index="6" Bitrate="305000" FourCC="WVC1" MaxWidth="364" MaxHeight="272" CodecPrivateData="250000010FC3C80B50878A0B5821E80C9080894E4A76400000010E5A67F840" />
  12.     <QualityLevel Index="7" Bitrate="230000" FourCC="WVC1" MaxWidth="364" MaxHeight="272" CodecPrivateData="250000010FC3C60B50878A0B5821E80C90800704704DC00000010E5A67F840" />
  13.     <c t="0" />
  14.     <c t="22350000" />
  15.     <c t="42370000" />
  16.     <c t="62390000" />
  17.     <c t="82410000" />
  18.     <c t="102430000" />
  19.     <c t="122450000" />
  20.     <c t="142470000" />
  21.     <c t="162490000" />
  22.     <c t="182510000" />
  23.     <c t="202530000" />
  24.     <c t="222550000" />
  25.     <c t="242570000" />
  26.     <c t="262590000" />
  27.     <c t="282610000" d="17350001" />
  28.   </StreamIndex>
  29.   <StreamIndex Type="audio" Index="0" FourCC="WMAP" Chunks="15" QualityLevels="1" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
  30.     <QualityLevel Bitrate="192000" SamplingRate="44100" Channels="2" BitsPerSample="16" PacketSize="8917" AudioTag="354" CodecPrivateData="1000030000000000000000000000E0000000" />
  31.     <c t="0" />
  32.     <c t="22291156" />
  33.     <c t="40867120" />
  34.     <c t="60371882" />
  35.     <c t="84056235" />
  36.     <c t="100774603" />
  37.     <c t="121208163" />
  38.     <c t="143034920" />
  39.     <c t="160682086" />
  40.     <c t="181580045" />
  41.     <c t="202013605" />
  42.     <c t="221518367" />
  43.     <c t="242880725" />
  44.     <c t="260789115" />
  45.     <c t="282354648" d="17993650" />
  46.   </StreamIndex>
  47. </SmoothStreamingMedia>

To explain the structure in simpler terms, we have the following relationship: SmoothStreamingMedia contains StreamIndex that contains QualityLevel and ‘c’ elements.

Clip Element

Let’s say, I only want the first 10 seconds of the clip above as one of the clips in the RCE manifest. In the new RCE structure I would represent this as below:

Note: <<source to original manifest>> actually points to the URL for the manifest from which we need to play first 10 seconds.

Code Snippet
  1. <Clip Url="<<source to original manifest>>" ClipBegin="0" ClipEnd="100000000">
  2.   <StreamIndex Type="video" Chunks="5" QualityLevels="8" MaxWidth="640" MaxHeight="480" DisplayWidth="640" DisplayHeight="480" Url="QualityLevels({bitrate})/Fragments(video={start time})">
  3.     <QualityLevel Index="0" Bitrate="1644000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBF213F0EF8A13F83BE80C9081B22B6457400000010E5A67F840" />
  4.     <QualityLevel Index="1" Bitrate="1241000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBE613F0EF8A13F83BE80C9081A5DECBBE400000010E5A67F840" />
  5.     <QualityLevel Index="2" Bitrate="937000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBDC13F0EF8A13F83BE80C90811C97F260C00000010E5A67F840" />
  6.     <QualityLevel Index="3" Bitrate="708000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBD60D509F8A0D5827E80C9081159AD66CC00000010E5A67F840" />
  7.     <QualityLevel Index="4" Bitrate="534000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBD00D509F8A0D5827E80C9081104B412F400000010E5A67F840" />
  8.     <QualityLevel Index="5" Bitrate="403000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBCC0D509F8A0D5827E80C90808C4BE263400000010E5A67F840" />
  9.     <QualityLevel Index="6" Bitrate="305000" FourCC="WVC1" MaxWidth="364" MaxHeight="272" CodecPrivateData="250000010FC3C80B50878A0B5821E80C9080894E4A76400000010E5A67F840" />
  10.     <QualityLevel Index="7" Bitrate="230000" FourCC="WVC1" MaxWidth="364" MaxHeight="272" CodecPrivateData="250000010FC3C60B50878A0B5821E80C90800704704DC00000010E5A67F840" />
  11.     <c t="0" />
  12.     <c t="22350000" />
  13.     <c t="42370000" />
  14.     <c t="62390000" />
  15.     <c t="82410000" d="20020000" />
  16.   </StreamIndex>
  17.   <StreamIndex Type="audio" Index="0" FourCC="WMAP" Chunks="5" QualityLevels="1" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
  18.     <QualityLevel Bitrate="192000" SamplingRate="44100" Channels="2" BitsPerSample="16" PacketSize="8917" AudioTag="354" CodecPrivateData="1000030000000000000000000000E0000000" />
  19.     <c t="0" />
  20.     <c t="22291156" />
  21.     <c t="40867120" />
  22.     <c t="60371882" />
  23.     <c t="84056235" d="16718368" />
  24.   </StreamIndex>
  25. </Clip>

 

 

If you notice carefully, all we have done is changed the relationship above to: SmoothStreamingMedia contains Clip and Clip contains StreamIndex. Finally StreamIndex contains QualityLevel and ‘c’ elements.

While doing this we defined a new element called Clip which has attributes called Url, ClipBegin and ClipEnd.

  • Url – this specifies the Url to the original source manifest from which these clips were cut. The value is exactly as you would set on the SmoothStreamingSource property in SSME (e.g.,  http://abcxyz.com/sample.ism/Manifest).
  • ClipBegin – This specifies the time in nanoseconds where to begin the playback for the clip.
  • ClipEnd – This specifies time in nanoseconds where to end to end the playback for the clip.

You will also notice that ‘c’ elements are still present in this composite manifest. This is done so that this manifest is self sufficient and you don’t need to download the source manifest. ‘c’ elements are essentially chunk timestamps for the FMP4 chunks the client will download.

Note: Not all ‘c’ elements are included in this manifest. The idea is to include only the ones needed to represent the section of the clip. In this case times are close to the ClipBegin and ClipEnd. They may not exactly match in timestamps as chunks are 2 seconds apart typically while the Clip is cut at finer granularity.

Multiple Clip Elements

In the section above, we saw how to compose a single clip. However, composite manifest promises to have multiple such clips stitched together. All you need to do is to have multiple such Clip elements and that takes care of that. Here is how an example of composite manifest with two clips will will look like:

Code Snippet
  1. <?xml version="1.0" encoding="utf-16"?>
  2. <SmoothStreamingMedia MajorVersion="2" MinorVersion="0" Duration="200000000">
  3.   <Clip Url="http://abcxyz.com/sample.ism/Manifest" ClipBegin="0" ClipEnd="100000000">
  4.     <StreamIndex Type="video" Chunks="5" QualityLevels="8" MaxWidth="640" MaxHeight="480" DisplayWidth="640" DisplayHeight="480" Url="QualityLevels({bitrate})/Fragments(video={start time})">
  5.       <QualityLevel Index="0" Bitrate="1644000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBF213F0EF8A13F83BE80C9081B22B6457400000010E5A67F840" />
  6.       <QualityLevel Index="1" Bitrate="1241000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBE613F0EF8A13F83BE80C9081A5DECBBE400000010E5A67F840" />
  7.       <QualityLevel Index="2" Bitrate="937000" FourCC="WVC1" MaxWidth="640" MaxHeight="480" CodecPrivateData="250000010FCBDC13F0EF8A13F83BE80C90811C97F260C00000010E5A67F840" />
  8.       <QualityLevel Index="3" Bitrate="708000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBD60D509F8A0D5827E80C9081159AD66CC00000010E5A67F840" />
  9.       <QualityLevel Index="4" Bitrate="534000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBD00D509F8A0D5827E80C9081104B412F400000010E5A67F840" />
  10.       <QualityLevel Index="5" Bitrate="403000" FourCC="WVC1" MaxWidth="428" MaxHeight="320" CodecPrivateData="250000010FCBCC0D509F8A0D5827E80C90808C4BE263400000010E5A67F840" />
  11.       <QualityLevel Index="6" Bitrate="305000" FourCC="WVC1" MaxWidth="364" MaxHeight="272" CodecPrivateData="250000010FC3C80B50878A0B5821E80C9080894E4A76400000010E5A67F840" />
  12.       <QualityLevel Index="7" Bitrate="230000" FourCC="WVC1" MaxWidth="364" MaxHeight="272" CodecPrivateData="250000010FC3C60B50878A0B5821E80C90800704704DC00000010E5A67F840" />
  13.       <c t="0" />
  14.       <c t="22350000" />
  15.       <c t="42370000" />
  16.       <c t="62390000" />
  17.       <c t="82410000" d="20020000" />
  18.     </StreamIndex>
  19.     <StreamIndex Type="audio" Index="0" FourCC="WMAP" Chunks="5" QualityLevels="1" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
  20.       <QualityLevel Bitrate="192000" SamplingRate="44100" Channels="2" BitsPerSample="16" PacketSize="8917" AudioTag="354" CodecPrivateData="1000030000000000000000000000E0000000" />
  21.       <c t="0" />
  22.       <c t="22291156" />
  23.       <c t="40867120" />
  24.       <c t="60371882" />
  25.       <c t="84056235" d="16718368" />
  26.     </StreamIndex>
  27.   </Clip>
  28.   <Clip Url="http://abcxyz.com/sample2.ism/Manifest" ClipBegin="60000000" ClipEnd="160000000">
  29.     <StreamIndex Type="video" Chunks="5" QualityLevels="8" MaxWidth="848" MaxHeight="476" DisplayWidth="848" DisplayHeight="476" Url="QualityLevels({bitrate})/Fragments(video={start time})">
  30.       <QualityLevel Index="0" Bitrate="1644000" FourCC="WVC1" MaxWidth="848" MaxHeight="476" CodecPrivateData="250000010FCBB21A70ED8A1A783B68045081B22B6457400000010E5A67F840" />
  31.       <QualityLevel Index="1" Bitrate="1241000" FourCC="WVC1" MaxWidth="848" MaxHeight="476" CodecPrivateData="250000010FCBA61A70ED8A1A783B68045081A5DECBBE400000010E5A67F840" />
  32.       <QualityLevel Index="2" Bitrate="937000" FourCC="WVC1" MaxWidth="848" MaxHeight="476" CodecPrivateData="250000010FCB9C1A70ED8A1A783B680450811C97F260C00000010E5A67F840" />
  33.       <QualityLevel Index="3" Bitrate="708000" FourCC="WVC1" MaxWidth="568" MaxHeight="320" CodecPrivateData="250000010FCB9611B09F8A11B827E8045081159AD66CC00000010E5A67F840" />
  34.       <QualityLevel Index="4" Bitrate="534000" FourCC="WVC1" MaxWidth="568" MaxHeight="320" CodecPrivateData="250000010FCB9011B09F8A11B827E8045081104B412F400000010E5A67F840" />
  35.       <QualityLevel Index="5" Bitrate="403000" FourCC="WVC1" MaxWidth="568" MaxHeight="320" CodecPrivateData="250000010FCB8C11B09F8A11B827E80450808C4BE263400000010E5A67F840" />
  36.       <QualityLevel Index="6" Bitrate="305000" FourCC="WVC1" MaxWidth="480" MaxHeight="272" CodecPrivateData="250000010FCB880EF0878A0EF821E8045080894E4A76400000010E5A67F840" />
  37.       <QualityLevel Index="7" Bitrate="230000" FourCC="WVC1" MaxWidth="480" MaxHeight="272" CodecPrivateData="250000010FCB860EF0878A0EF821E80450800704704DC00000010E5A67F840" />
  38.       <c t="60000000" />
  39.       <c t="80000000" />
  40.       <c t="100000000" />
  41.       <c t="120000000" />
  42.       <c t="140000000" d="20000000"/>
  43.     </StreamIndex>
  44.     <StreamIndex Type="audio" Index="0" FourCC="WMAP" Chunks="6" QualityLevels="1" Url="QualityLevels({bitrate})/Fragments(audio={start time})">
  45.       <QualityLevel Bitrate="192000" SamplingRate="44100" Channels="2" BitsPerSample="16" PacketSize="8917" AudioTag="354" CodecPrivateData="1000030000000000000000000000E0000000" />
  46.       <c t="42724716" />
  47.       <c t="61082992" />
  48.       <c t="80341043" />
  49.       <c t="103096598" />
  50.       <c t="120279365" />
  51.       <c t="142570521" d="21362358" />
  52.     </StreamIndex>
  53.   </Clip>
  54. </SmoothStreamingMedia>

With this, you can now upload this manifest along with the original manifests and content used by them to a HTTP accessible location. Next, set the SmoothStreamingSource property to the URL to this composite manifest and SSME should take care of the rest. Here is how it will look like in XAML:

Code Snippet
  1. <UserControl x:Class="SilverlightApplication6.MainPage"
  2.     xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  3.     xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  4.              xmlns:SSME="clr-namespace:Microsoft.Web.Media.SmoothStreaming;assembly=Microsoft.Web.Media.SmoothStreaming"
  5.     xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
  6.     mc:Ignorable="d" d:DesignWidth="640" d:DesignHeight="480">
  7.     <Grid x:Name="LayoutRoot">
  8.         <SSME:SmoothStreamingMediaElement SmoothStreamingSource="http://abcxyz.com/SampleRCEManifest.csm" x:Name="SmoothPlayer" />
  9.     </Grid>
  10. </UserControl>

We will have more detailed documentation up on iis.net very soon.In the meantime, let me know if you have any questions.

Comments

Using a composite manifest, I received an error message saying 50 consecutive chunks failed to download.

I used Fiddler to inspect HTTP traffic and it was requesting /QualityLevels({bitrate})/Fragments(video={start time}).

Obviously, it should be requesting /path/to/files/Filename.ism/QualityLevels({bitrate})/Fragments(video={start time}).

For anyone who encounters the same problem, I resolved the issue by specifying the complete URL in the Url-attribute of every StreamIndex, for example: <StreamIndex Url="somedomain.com/.../QualityLevels({bitrate})/Fragments(video={start time})">

Not sure if this is how they are intended to work. Looking at the provided sample in the article, I would expect to get similar errors as stated above.

Feb 10 2010 by Espen Hovlandsdal

Hello,

I was just wondering if DRM is supported with composite manifests. That would be most useful. I've tried creating a CSM containing a protection header under the "Clip" tag, but with no success (I copied the asset's ismc protection header).

Thank you.

Aug 09 2011 by joaonogueira

Submit a Comment

  • Plain text is accepted.
  • URLs starting with http:// are converted to links.