How to build a Smooth Streaming Windows 8 JavaScript application with advanced features?

Friday, August 10, 2012

The Smooth Streaming Client SDK for Windows 8 enables playback of Smooth Streaming content in conjunction with the Windows 8 HTML5< video> element and XAML MediaElement by using the default APIs such as Play and Pause. Besides basic playback, the Smooth Streaming SDK has rich features like the ability to restrict the quality levels, do Live DVR, switch audio streams, and listen for status updates (such as quality level changes) and error events, and so on.

In this post, I'll cover how to enable these rich features of Smooth Streaming SDK for Windows 8. For rich applications powered by these features and general player development, I strongly encourage you to use the Microsoft Media Platform Player Framework. You can get more information about the Player Framework here.

Note: This post was tested using the Smooth Streaming Client SDK Beta 2 with the Windows 8 RP and Visual Studio 2012 RC releases. Unexpected results might occur if you run the SDK on different versions of this software.

Prerequisites

· Windows 8 Release Preview (RP) (32-bit or 64-bit) OS. You can get it from here.

· Either Visual Studio 2012 RC or Visual Studio Express 2012 RC for Windows 8 installed on Windows 8. You can get more information on this here.

· Install the "Microsoft IIS Smooth Streaming Client SDK Beta 2 for Windows 8" from here.

· Basic knowledge of JavaScript.

Note: This post is based on pre-release (Beta) builds of software and components; therefore, some features might not be available or supported for some devices and subject to change without notification.

Create the JavaScript Windows Store Project

The very first thing you need to do is create a JavaScript Windows Store project.

1. Open Visual Studio Express 2012 or Visual Studio 2012.

2. Go to File –> New Project.

3. Chose JavaScript as the project type, and then select Windows Store. Choose Blank Application and name it SS_Rich_App.

4. In your project, go to Project > Add Reference > Windows> Extensions, and add "Microsoft Smooth Streaming Client SDK Beta 2 for Windows 8", "Microsoft Visual C++ Runtime Package" and "Windows Library for JavaScript 1.0"to the references.

5. Go to Build > Configuration Manager and change your project platform to x86 or x64. Otherwise, you will get warnings in reference manager and in the build process. (Please see the release notes.)

6. If you want to support local source URLs, (for example, http://mycompany/video1.ism/manifest, you will need to add "Private Networks (Client & Server)" capabilities to your application

To do this, double-click the "package.appxmanifest" file in Solution Explorer, and then choose Capabilities. Mark the check box Private Networks (Client & Server).

Note: If you want to support enterprise authentication protected sources, you should consider "Enterprise Authentication" to capabilities as well. More information can be found here.

Now, you are ready to begin coding.

Adding EventHandlers and ProxyStub to the package.appxmanifest file

To access SDK APIs and listen for EventHandlers, you need to add InterfaceIds to the package.appxmanifest file. To do this, right-click the package.appxmanifest file in Solution Explorer and select View Code. Add the following lines.

</Capabilities>

<Path>Microsoft.Media.AdaptiveStreaming.dll</Path>

</ProxyStub>

</Extension>

</Extensions>

</Package>

Enabling Smooth Streaming support with SDK APIs for the <video> element

The Windows 8 Store apps <video> element doesn’t support Smooth Streaming content out-of box. To enable Smooth Streaming support, you need to register the Smooth Streaming SDK and tie the Smooth Streaming extension and mime-types to the SDK. Prior to SDK Beta 2 release, this could only be done by using “registerByteStreamHandler”. With the SDK Beta 2 release, “registerSchemeHandler” is also introduced. I will not go into the details and the differences between “registerByteStreamHandler” and “registerSchemeHandler” in this post. For detailed information, see Scheme Handlers and Byte-Stream Handlers. Before registering the Byte-Stream and Scheme handler, you need to create a property set to interact with the SDK over WinRT APIs. (If you don’t want to use SDK APIs and just want to use the <video> element’s default controls, please refer to “How to build your first HTML5 Windows 8 Smooth Streaming Application”.) To create a property set and register a URL or MIME-type combination that will be resolved to the Smooth Streaming extension:

In the Solution Explorer js folder, open the default.js file, and add the code marked below by the "Smooth Streaming Registration" comments:

var app = WinJS.Application;

var activation = Windows.ApplicationModel.Activation;

// Smooth Streaming Registration begins

var plugins = new Windows.Media.MediaExtensionManager();

//Creates the Media Extension Manager

var property = new Windows.Foundation.Collections.PropertySet();

//Creates propertySet.

var mgr = Microsoft.Media.AdaptiveStreaming.AdaptiveSourceManager.getDefault();

//Gets the default instance of AdaptiveSourceManager which manages Smooth Streaming sources.

mgr.addEventListener("adaptivesourceopenedevent", mgrOpenedEventHandler, false);

// Subscribes to AdaptiveSourceOpenedevent

mgr.addEventListener("adaptivesourceclosedevent", mgrClosedEventHandler, false);

// Subscribes to AdaptiveSourceClosedevent

mgr.addEventListener("manifestreadyevent", mgrManifestReadyHandler, false);

// Subscribes to ManifestreadyEvent for manifest operations such as restrict/select tracks.

property["{A5CE1DE8-1D00-427B-ACEF-FB9A3C93DE2D}"] = mgr;

// Sets property key value to AdaptiveSourceManager default instance.{A5CE1DE8-1D00-427B-ACEF-FB9A3C93DE2D}" needs to be hardcoded.

plugins.registerByteStreamHandler("Microsoft.Media.AdaptiveStreaming.SmoothByteStreamHandler", ".ism", "text/xml", property);

plugins.registerByteStreamHandler("Microsoft.Media.AdaptiveStreaming.SmoothByteStreamHandler", ".ism", "application/vnd.ms-ss", property);

// Register Smooth Streaming Byte-Stream Handler for “.ism” extension and, "text/xml" and "application/vnd.ms-ss" mime-types and pass the propertyset. http://*.ism/manifest URI resources will be resolved by Byte-stream handler.

//plugins.registerSchemeHandler("Microsoft.Media.AdaptiveStreaming.SmoothSchemeHandler", "ms-sstr:", property);

// This is optional for advanced scenarios like offline playback via Downloader plugin interface. Register Smooth Streaming Scheme handler and pass the propertyset. ms-sstr://*.ism/manifest URI resources will be resolved by Scheme-handler.

// Smooth Streaming Registration ends

app.onactivated = function (args) {

Add the HTML5 Video Element

Next, you need to add the HTML5 video element.

To do this, open the default.html file. Inside the <body> tag, add a <video> tag, list box <select> and <text area> tags for logging:

<body>

<p>

</p>

<p>

<strong> AudioStreams: </strong>

</select>

</p>

<tr>

</tr>

<tr>

<td>

</tr>

</table>

var txtLog = document.getElementById("txtLog");

</script>

</body>

The above example has a video element with default controls (Play/Pause/Seek/Volume Control) that consumes a Smooth Streaming source. This is all you need for a basic layout.

Adding eventHandler functions and logging

In the “Enabling Smooth Streaming support” section we subscribed to the adaptivesourceopenedevent, adaptivesourceclosedevent and manifestreadyevent events. To take action based on these events, you need to create some functions. The code below demonstrates catching basic Opened, Closed and ManifestReadyEvent events and printing the DisplayURI property of the AdaptiveSource in the LogText section.

To add the functions, open the default.js file, and paste the following code into the end of the file.

var audioStreamsAvailable = new Array();

var selectedAudioStreamIndex;

function mgrOpenedEventHandler(e) {

try {

var DisplayedUri = e.adaptiveSource.uri.displayUri.toString();

log(' AdaptiveSource Opened: ' + DisplayedUri);

}

catch (error) {

log('error' + error.number.toString());

}

function mgrClosedEventHandler(e) {

try {

log(' AdaptiveSource Closed: ');

}

catch (error) {

log('error' + error.number.toString());

}

function log(line) {

var now = new Date();

txtLog.value += now.toDateString() + " " + now.toTimeString() + " | " + line + "\n";

txtLog.scrollTop = txtLog.scrollHeight;

}

function mgrManifestReadyHandler(e) {

//RestrictTracks can only be called during ManifestReadyEvent and not allowed during playback.

try {

log("We have the manifestready event");

saveManifest = e.adaptiveSource.manifest;

addStreamstoListBox();

}

catch (error) {

log('error' + error.number.toString());

}

Getting available streams from the Smooth Streaming file

Smooth Streaming content can include multiple streams and these streams are defined in the Smooth Streaming manifest files. By using the manifestreadyevent, you can get the available streams from the manifest data. As an example, you can get the available audio streams and display them on your application for audio switching. Our sample (available at http://ecn.channel9.msdn.com/o9/content/smf/smoothcontent/elephantsdream/Elephants_Dream_1024-h264-st-aac.ism/manifest) includes three audio streams; English, Spanish, and audio commentary. To get the list from the manifest and set the values to the html list box (id="SelectAudioStreams") you need to use the availableStreams and selectedStreams properties of the Manifest object. In the above sections, we subscribed to manifestreadyevent event. This event calls mgrManifestReadyHandler function. Inside the mgrManifestReadyHandler, we are calling addStreamstoListBox(); which gets the availableStreams and selectedStreams properties and sets the values to the id="SelectAudioStreams" list box.

To add the (addStreamstoListBox) functions, open the default.js file and paste the code below to the end of the file.

function addStreamstoListBox() {

var audioStreamIndex = -1;

try {

//clearing the items from previous set source

document.getElementById("SelectAudioStreams").innerHTML = "";

availableStreams = saveManifest.availableStreams;

selectedStreams = saveManifest.selectedStreams;

var size_availableStreams = availableStreams.size;

var size_selectedStreams = selectedStreams.size;

audioIndex = 0;

audioStreamsAvailable.splice(0, audioStreamsAvailable.length);

//initialize the listbox with the available audio streams and the streamselection array.

for (var i = size_availableStreams - 1; i > -1 ; i--) {

if (availableStreams[i].type == Microsoft.Media.AdaptiveStreaming.MediaStreamType.audio) {

var newStreamOption = document.createElement("OPTION");

newStreamOption.value = audioIndex;

newStreamOption.text = availableStreams[i].name;

document.getElementById("SelectAudioStreams").options.add(newStreamOption);

audioStreamsAvailable.push(availableStreams[i]);

audioIndex++;

}

for (j = 0; j < size_selectedStreams; j++) {

if (selectedStreams[j].type == Microsoft.Media.AdaptiveStreaming.MediaStreamType.audio) {

selectedAudioStreamIndex = j;

}

catch (error) {

log(error.message.toString());

}

The above function sets the available audio stream values to the HTML list box. To change the audio stream you need to call the selectStreamsAsync() method. The code below changes the audio stream based on the list box changes. (id="SelectAudioStreams") has an event definition for onchange.(onchange="playSelectedStream()") When the list box changes, the playSelectedStream() function is called, and this function sets the audio stream to the stream selected by the list box .

To add the (playSelectedStream) functions, open the default.js file and paste the code below to the end of the file.

function playSelectedStream() {

try {

var currentSelectedStreams = new Array();

if (document.getElementById("SelectAudioStreams").value != null) {

var SelectedAudioStreamIndexfromlist = document.getElementById("SelectAudioStreams").value;

for (i = 0; i < selectedStreams.size; i++) {

if (i == selectedAudioStreamIndex) {

currentSelectedStreams.push(audioStreamsAvailable[SelectedAudioStreamIndexfromlist]);

}

else {

currentSelectedStreams.push(selectedStreams[i]);

}

saveManifest.selectStreamsAsync(currentSelectedStreams).then(function (result) {

log(audioStreamsAvailable[document.getElementById("SelectAudioStreams").value].name + " Selected");

}, function (error) {

log("playSelectedStreams" + error.message.toString());

})

}

catch (error) {

log("playSelectedStreams" + error.message.toString());

}

Status Updates

The Smooth Streaming Client SDK includes the Source level status update event. Using this feature, you can get extended status information such as bitrate change, Live Smooth Streaming start and end times, and rebuffing update events. In this section, the sample code shows how to subscribe to these events and get the data.

To subscribe to AdaptiveSourceUpdateEvents you need add the code below to the // Smooth Streaming Registration begins section in the default.js file before calling property["{A5CE1DE8-1D00-427B-ACEF-FB9A3C93DE2D}"] = mgr;

mgr.addEventListener("adaptivesourcestatusupdatedevent", mgrStatusUpdateHandler, false);

// Subscribes to AdaptiveSourceStatusUpdatedEvent

This code block enables AdaptiveSourceUpdateEvents to call the mgrStatusUpdateHandler function.

To display Update events and data in the log field, open the default.js file and paste the code below to the end of the file.

function mgrStatusUpdateHandler(e) {

try {

var state;

UpdateData = e.updateType.toString();

switch (parseInt(UpdateData)) {

case 4:

state = "BitrateChanged";

break;

case 5:

state = "ChunkConnectHttpInvalid";

break;

case 8:

state = "ChunkHdrError";

break;

case 7:

state = "ChunkHdrHttpInvalid";

break;

case 9:

state = "EndOfLive";

break;

case 6:

state = "NextChunkHttpInvalid";

break;

case 10:

state = "OutsideWindowEdge";

break;

case 2:

state = "Rebuffer";

break;

case 3:

state = "StartEndTime";

break;

case 1:

state = "Underrun";

break;

case 1:

state = "Unknown";

break;

default:

state = " Unknown";

}

log(state + ": " + e.additionalInfo.toString() + " Startime: " + e.startTime.toString() + " Endtime: " + e.endTime.toString());

}

catch (error) {

log('error' + error.number.toString());

}

Build your Project

Change your platform type to x64, x86 or ARM depending on which platform you're targeting, and then build your project.

Note: If you leave the Any CPU option selected, Visual Studio will display the following warning in the References section, and might display compilation errors and warnings like the following:

· Error 1 The processor architecture of the project being built "Any CPU" is not supported by the referenced SDK "Microsoft.Media.AdaptiveStreamingClient, Version=1.0". Please consider changing the targeted processor architecture of your project (in visual studio this can be done through the Configuration Manager) to one of the architectures supported by the SDK: "x86, x64, ARM".

· Error 2 The processor architecture of the project being built "Any CPU" is not supported by the referenced SDK "Microsoft.VCLibs, Version=11.0". Please consider changing the targeted processor architecture of your project (in visual studio this can be done through the Configuration Manager) to one of the architectures supported by the SDK: "x86, x64, ARM".

The SDK depends on the Microsoft Visual C++ Runtime Package. If you fail to add this dependency to the project references, Visual Studio will generate compilation errors and warnings like the following:

· Warning 1 The SDK "Microsoft.Media.AdaptiveStreamingClient, Version=1.0" depends on the following SDK(s) "Microsoft.VCLibs, Version=11.0", which have not been added to the project or were not found. Please ensure that you add these dependencies to your project or you may experience runtime issues.

Run the Application

The "Elephants Dream" sample video should playback successfully:

Let me know if you have issues or need further help.

Summary

This post showed how to use the Smooth Streaming SDK APIs with the default controls for the HTML5 <video> element to play back Smooth Streaming content in Windows 8.

Note: These are samples meant for educational purposes with no guarantee conferred on code quality. All the samples are based on pre-release (Beta) builds of software and components; therefore, some features might not be available or supported for some devices or subject to change without notification.

I've followed the directions here and in the more basic tutorial to the t yet always receive the "Error: unsupported video type or invalid file path" whenever I point to any manifest file. I know the files are valid since I can get them to play in silverlight. I've poured over every comment I could find in msdn about how to get things kickstarted: I tried adding a dependency to the visualc++ library in the package.appxmanifest file (as someone suggested on another page); I tried every permutation of build configuration. I've been trying this for hours and it just seems like the video tag doesn't talk to the adaptive streaming plugin.

Is there some sort of secret sauce here? Or does this just not work? The last comment I noticed was around June, and a lot has changed since then.

I'm on a Windows8 VM (VMware Fusion), using Microsoft Visual Studio Professional 2012 Version 11.0.50727.1 RTMREL I downloaded the Microsoft Smooth Streaming Client SDK Beta 2 for Windows 8 Version 1.0

dtniland - Wednesday, October 24, 2012 10:55:44 PM

This is wonderful, thank you! I followed your instructions and got it working with my own streams without any problems. I am excited to apply the selectStreamsAsync for future multi-lingual broadcasts.

One question: My streams are being DVR'd by IIS, but for some reason this player won't let me seek while the stream is still live. I can pause and play the stream, but I cannot seek. It is not until the the stream has completed (publishing point changed to "stopped"), and the application reloaded, that I can seek within the timeline properly. Is this something I need to pragmatically extend?

PaulMartel - Wednesday, May 28, 2014 7:02:11 PM

2 Comments