Silverlight 3 Beta - What's New for Media

So, Silverlight 3 was released for public beta today, during Scott Guthrie’s keynote at MIX. There will be an on-demand version of it later on at Visitmix.com. There’s tons of info flowing around today, so I’m just going to focus on the media-related news that came out today. I’ll be posting a lot this week; check back for more details as they’re revealed. And feel free to ask questions in the comments area below.

A developer release, not a consumer release

Note that this is a developer release. Silverlight 2 was unique in scope, being the first release with managed code, so we did some consumer events using Beta 2, most notably the NBC Olympics coverage. We’re not planning on doing anything customer-facing with Silverlight 3 until the full release. So this is really a change for developers, designers, and media folks to get a head start on using the new features, and giving us feedback on any issues. End users are welcome to play around with it (I’ve got it installed on my home machines without any issues), but don’t expect to see much stuff on the web taking advantage of it until after it ships.

What’s Available

First off, here’s the Silverlight-specific downloads released today. The official Get Started page is at Silverlight.net.  And Tim Heuer has a great overview of everything.

MPEG-4 (H.264 and AAC-LC) support

First up, as we discussed back at IBC, we’re adding support for MPEG-4 content in Silverlight. Specifically, we’re supporting

  • Self-contained .mp4 (including .f4v and .m4a) and .mov file formats (no reference movies or anything fancy like that).
  • H.264 video in Simple, Main, and High 4:2:0 profiles (progressive scan only)
  • AAC-LC audio mono or stereo (HE AAC will play back with lower fidelity, as in QuickTime)
  • Local files or http progressive download.

Or, sliced another way, Silverlight 3 will be able to play pretty much all MPEG-4 files that would play back well in both QuickTime and Flash.

The H.264 and AAC-LC decoders are exposed via the MediaStreamSource used by Smooth Streaming, which means extending support to other file formats and protocols will work there as well. And of course, Smooth Streaming will support H.264 and AAC-LC as well.

Raw AV Pipeline

MediaStreamSource started with the ability to pass off VC-1 and WMA bitstreams to the decoder from managed code, enabling us to add protocol and format support within Silverlight. Most of Smooth Streaming on the client is implemented in C# as MediaStreamSource code. And we’re extending that to H.264 and AAC-LC. But there’s a huge number of other codecs out there that different markets are using and have asked to have in Silverlight. In fact, far more than we can reasonably build and test, and far more than we’d want to have to make everyone download in the installer.

So, instead we added three very simple but very powerful new outputs to MediaStreamSource: YV12, RGBA, and PCM.

Yep, managed code can now output uncompressed video frames and uncompressed audio straight to the media pipeline.

Most video codecs are going to use YV12, which is a flavor of the  a standard Y’CbCr 4:2:0 format used in most video codecs . We also wanted RGBA to enable video with alpha channels, and also any media formats that are natively RGB. And of course PCM is classic uncompressed audio.

Using a managed decoder is transparent to the user; it’s just another managed code .dll included in the .xap file. Since it’s running inside the Silverlight sandbox, there’s no download or other action required by the user to activate it; acts just like any other codec.

So, what’s codec performance like in managed code? It’s actually going to be fine for a variety of tasks. In our internal testing and prototyping, managed code offers about half the speed of a native compiled version. So for older codecs like MPEG-2, a managed decoder should be fine on a wide variety of hardware.

And remember, managed code doesn’t require C#. Existing decoders in C or C++ can be used as the basis for a managed decoder. The main restrictions are that native MMS/SSE and pointers can’t be used.

I can’t wait to see what other kinds of formats and codecs people wind up delivering with the Raw AV Pipeline. We’ve already seen some very neat demos.

GPU scaling and compositing

Silverlight 3 adds GPU support for scaling and compositing. Other products may call this something like “GPU accelerated video playback” which can be a little confusing – this isn’t GPU video decoding ala DXVA. What it does is take bitmap elements, including video, into the pipeline at their native size and then the GPU takes care of scaling each object to its final size, and then doing the final blending and compositing of the whole application. For media players, this essentially gives every player Fast Path performance. So while it does accelerate video playback, it’s not actually accelerating video decode per se. But on single-core machines, anything we can do to save CPU from other tasks to give more to the video decoder, the better. Again, the net effect is lower system requirements for any content that gets scaled; which should often improve the bitrate lower-end machines can play in Smooth Streaming.

There’s a few new facets to tuning a player for GPU playback, so it’ll need to be turned on explicitly in the application. But the performance gains for any video that’s scales are tremendous, so we’re going to help making sure everyone knows how to get it on and reap the benefits.

Full-screen GPU scaling/compositing will be supported on all Macs and all Windows machines with DirectX 9 or higher (which is XP SP1 and higher – that’s most machines and GPUs out in the wild today). Due to platform limitations, Silverlight on Mac won’t support GPU compositing inside the browser window, but will in full-screen mode. For performance-critical players (particularly in HD) a good compromise there is to set Stretch=”None” for the browser version and scale in full-screen mode, so that the windowed version is Fast Path and the full-screen is GPU composited for equivalent performance.

Improved logging

Robust and detailed logging has long been a critical feature for Windows Media to enable advertising and billing.

We’re extending that support to Silverlight 3, for both Windows Media and Smooth Streaming. I’ll provide a link to the full details when they’re posted.

Media pipeline improvements

We’ve done some under-the-hood work in Silverlight to optimize for silky-smooth video playback with as few dropped frames as possible. For the beta, we’ve focused mainly on single and dual-core systems, and you should see some significant improvements with the same player and content in SIlverlight 2 (and of course using the GPU when scaling will offer further performance gains).

Out of Browser Silverlight applications

Silverlight 3 will allow applications to be dragged out of the browser and onto the local system as a standalone, double-clickable app. This is useful for all kinds of applications, of course, but for media players it’s great to not have to launch the browser when all you want is an applet media player. This includes offline support, so a Silverlight media player could be used on an airplane or anywhere else without a network connection. And since it’s a real Silverlight app, it still uses the secure sandbox and so Admin rights on the machine aren’t required to install or run it. We’ve got nice integration for both Mac and Windows with Start Menu and Dock compatibility.  Ashish’s blog has lots of details.

No Comments