Sam Zhang's Blog : media

Live Smooth Streaming – Design Thoughts

samzhang — Fri, 27 Mar 2009 07:29:33 GMT

Last week we released the first Beta of IIS Live Smooth Streaming. If you had a chance to watch Scott Guthrie’s demo in the MIX keynote, or our program manager John Bocharov’s MIX session, you have probably seen the slick player UI and gotten a feel for how the technology works in general. I hope you like the experience and the new functionalities. Today I’d like to dig a little bit deeper and talk about some interesting design thoughts we put in when developing this product.

Before we get started, I’d like to call out the crucial distinction between “traditional streaming” –using stateful protocols such as RTSP, MMS, or Adobe’s RTMP – and “IIS Live Smooth Streaming” – which uses HTTP, a stateless protocol. Let’s take a look at the differences between these approaches in detail…

1. Push vs. Pull

First let’s take a look at how traditional live streaming works. A client first establishes a control channel to the server to do some initial setup. Then the server uses the data channel (which could be the same channel as the control channel for protocols such as HTTP) to deliver a long running live stream to the client. While streaming, the client can use the control channel to send control messages to the server to re-set-up streams, pause, seek (if supported), stop or shutdown. Although the initial request and the commands are always coming from the client, if we just focus on the data delivery part, it is actually a push model. The server keeps pushing the latest data packets to the client and the client just passively receives everything.

On the other hand, IIS Live Smooth Streaming is actually a pull model. The client initiates a manifest request to set up the streaming. Then the client issues fragment requests, one for each fragment, to build up its sample buffer for streaming. In fact, all these requests look just like normal Web requests. The server needs to either reply with data immediately or fail the request. The fragment responses can be cached by any web cache/proxy and later be returned directly from the cache server to other clients. This pull model is how standard Web requests work, and that is why IIS Live Smooth Streaming is able to take advantage of the massive existing Web infrastructure to scale it out for you.

So other than being more web-friendly, is there anything else that has changed as the result of converting from push to pull? Read on if you’re interested.

2. State Control

In traditional live streaming, the client state is managed both by the client and the server. The server keeps a record of each client for things such as playback state, streaming position, selected bit rate (if multiple bit rates are supported), etc. While this gives the streaming server more control, it also adds overhead to the server. What is more important is that each client has to maintain the server affinity throughout the streaming session, limiting scalability and creating a single point of failure. If somehow a client request is rerouted by a load balancer to another server in the middle of a streaming session, there is a good chance that the request will fail. This limitation creates big challenges in server scalability and management for CDNs and server farms.

With the pull model in Live Smooth Streaming, the client is solely responsible for maintaining its own state. In turn, the server is now stateless. Any client request (fragment or manifest) can be satisfied by any server that is configured for the same live event. The network topology can freely reroute the client requests to any server that is best for the client. From the server’s perspective, all client requests are equal. It doesn’t matter whether they are from the same client or multiple clients, whether they are in live mode or DVR mode, which bit rate they’re trying to play, whether they’re trying to do bit rate switching, etc. They’re all just fragment requests to the server, and the server's job is to manage and deliver the fragments in the most efficient way. Unlike some other implementations, the Live Smooth Streaming server’s job is once again to keep all the content readily available to empower the client’s decisions, and to make sure it presents the client with a semantically consistent picture. This has two benefits: (1) the feedback loop is much smaller as the client makes all the decisions, resulting in a much faster response (e.g. bit rate switching), and (2) it makes the server very lean and fast. The IIS Live Smooth Streaming beta server module is a mere 230KB DLL binary (32bit version)!

You can see that the division of the responsibilities between the server and the client has changed in the pull model. The server is focusing on delivering and managing fragments with the best possible performance and scalability, while the client is all about ensuring the smooth streaming/playback experience, which, in my view, is a much better solution for large-scale online video.

3. Latencies

We usually talk about two kinds of latencies in live streaming. One is startup latency, which is how fast a client can start playback after an initial open or seek. The other is end-to-end latency, which is the time delay between a real world live event and its appearance in video playback on the client.

For startup latency, the basic approach is still the same, in that client needs to fill up its buffer as fast as possible. What’s different with IIS Live Smooth Streaming is that the client can issue multiple requests simultaneously for the initial data chunks. Those requests could be served by nearby HTTP cache nodes instead of streaming servers which could potentially be much further away. Similarly, in a cache-miss scenario, multiple streaming servers could be used for more throughput than a single server can provide. The IIS Live Smooth Streaming client also tries to start with fragments from a lower bit rate to further expedite the startup process.

End-to-end latency is essentially the sum of all the delays that media samples will encounter when they travel through the entire media processing delivery pipeline. Other than encoder delay and network delay, the biggest factors affecting end-to-end delay are the local buffers on both the server and the client. Media data has to go through these FIFO buffers before it can be decoded and rendered. Smaller end-to-end latency is always good, especially for time-critical events, but it also comes at a cost. In order to lower end-to-end latency, the server and client buffers need to be tuned down to smaller sizes, which can result in a longer startup latency (there is less data in the server FIFO buffer to blast down to fill the client buffer) and more vulnerability to network jitter.

With IIS Live Smooth Streaming, the server automatically archives all the fragments that it has received since the beginning of the event. When the client first connects, the server publishes all the information about the archived fragments in the stream manifest. Based on the stream manifest, the client logic can determine where to start the playback, striking the best balance between end-to-end latency and other considerations. The startup position is no longer mandated by the server. A position closer to the “live edge” means smaller end-to-end delay with the corresponding tradeoffs, and vice versa.

4. Timing Control

Timing control in traditional live streaming is a given. The client passively accepts what the server pushes out, so it always knows how the live stream is progressing. Now with the pull model, it has become more interesting. The client is the one who initiates all the requests and it needs to know the right timing information in order to do the right scheduling. Given that the server is stateless in this pull model and the client could talk to any server for the same streaming session, it has become more challenging. The solution is to always rely on the encoder’s clock for computing timing information and design a timing protocol that’s stateless and cacheable.

5. Clock Drift

One common issue with live streaming, especially in 24x7 scenarios, is clock drift. What happens is that the encoder, server, and client are all running their own clocks for scheduling and in most cases these clocks are not synchronized. Even slight discrepancies between these clocks can result in a non-trivial clock drift over a period of time, causing buffer overflow or underflow. Let’s take a look at them in more detail below:

5.1 Buffer Overflow

This can happen if the client’s clock is running slower than the encoder’s clock, meaning that client is not consuming samples as fast as the encoder is producing them. As samples get pushed to the client, more and more get buffered, and the buffer size keeps growing. Over time this can cause the client machine to slow down and eventually run out of memory. However, with the pull model we have for IIS Live Smooth Streaming, this problem is automatically solved. The client is driving all the requests and it will only request the chunks that it needs and can handle. In other words, the client’s buffer is always synchronized to the client’s clock and never gets out of control. The only side effect of this type of clock drift would be that the client could slowly fall behind, transitioning from a “live” client to a DVR client (playing something in the past).

5.2 Buffer Underflow

Buffer underflow can be the result of a client’s clock running faster than the encoder’s clock. The reason is obvious: the client is consuming samples a little bit too fast. In this case, the client has to either keep re-buffering or tune down its renderer clock. To detect this case, the client needs to distinguish this condition from others that could also cause buffer underflow, e.g. network congestion. This determination is often difficult to implement in a valid and authoritative manner. The client would need to run statistics over an extended period of time to detect a pattern that’s most likely caused by clock drift rather than something else. Even with that, false positives can still happen.

In IIS Live Smooth Streaming’s pull model, the server has more opportunities to communicate with the client about the timing status. In the case that the client has a faster clock, it might end up requesting a chunk that’s not available on the server yet. When the server sees this request, it knows that it’s for a future chunk that will be available shortly, and can therefore return a special error code or status to indicate a “temporarily” not-found condition. This is different than requesting a chunk that’s missing on the server which is a “permanent error”. When the client gets this “temporary error” message, it knows immediately that it is running too far ahead. The server can also directly tell the client how much it was ahead to further help the client tune its clock. This seems to be an easier and more reliable way to detect and correct the clock drift problem.

To make the story complete, I need to point out that there is a more advanced mechanism in traditional streaming to help solve the clock drift problem. That is done by having the client constantly report back its buffer status via a separate feedback control channel using protocols like RTCP. Even with that, the feedback loop is quite lengthy, and the client is still at the mercy of the server to throttle the send rate up or down accordingly. Also, it usually involves UDP communication rather than TCP, which could be an issue in some server and client environments.

6. Summary

In summary, the pull model we designed for IIS Live Smooth Streaming is very different than how the traditional live streaming push model works. With this model, the server becomes a very lean stateless server with better performance, scalability, and manageability. The client has more control over different aspects of streaming/playback and is better equipped to handle all the challenges to offer a great user experience. We’d like to hear your feedback on which things you like and which things that you think we can do better.

Thank you.

Streaming == Content Protection? (Part 2)

samzhang — Fri, 23 May 2008 22:41:11 GMT

So in part one, I talked about technically how streaming and content protection works and why they're independent and not tied to each other. Today, I'd like to share with you how I view the claims behind this notion and what they're really about. The claims I'm going to quote here are from a technical paper that was published by a company that advocates this concept. If you're familiar with media streaming, you should know which company I'm talking about. :) For the purpose of this blog, let me call it "the company". (Disclaimer: this is not the same "the company" that chases Michael Scofield and his brother in "Prison Break" :-) ).

Below are some of the claims in that technical paper which is trying to prove that streaming helps with content protection.

1. Progressive-downloaded contents are always saved in browser's cache

This is true, but only with default setup. If you know HTTP protocol, you probably know that HTTP protocol defines headers that can be used to disable caching. That means your web server can actually tell browser or cache/proxy not to cache the response. It's very easy to configure and here is an example of how it can be done on IIS7:

Figure 1: Selecting "HTTP Response Headers" Icon

Figure 2: Add "no cache" headers to the response

In the example above, I created a folder called "NoCache" under my default web site. Then I select "HTTP Response Headers" icon at this folder level (by doing that I only disable caching for this folder and below). Then I go into "HTTP Response Headers" UI page and add two response headers: "Cache-Control" and "Pragma" both with value as "no-cache". "Cache-Control" header is defined in HTTP 1.1 and "Pragma: nocache" header is for legacy HTTP 1.0 clients. See HTTP protocol spec here for more detail.

Ok, that's it! If you want to verify, you can try this:

1) Before adding these two headers, use your web player to play a media file in this "NoCache" folder. Then go to your browser's temp file folder and search for the cached file.

2) Add these two headers, clean your browser's cache and then repeat (1). You should no longer be able to find the cached file.

As you can see, servers can prevent progressive downloaded files from being available in a well-behaved browsers cache.

2. SSL Encryption

This technical paper talks about their streaming server supporting SSL encryption. What makes me uncomfortable is that it in nowhere mentions the fact that you can also easily do SSL from web server (aka HTTPS). Here I'm not going to dive into the details about how to configure SSL on IIS7. I'm sure you can find a lot of references on MSDN and iis.net.

3. Proprietary Streaming Protocols

Well, I can't really argue for this one because it's a well-known truth that "the company" has decided to use their own protocols and keep them in private. But using a proprietary protocol automatically makes the content "protected" just doesn't make sense to me. It's sort of saying if I speak in another language, my words will immediately become "protected". The reality is that it's just a matter of time before people crack it with good reverse engineering effort. Technically there's nothing preventing a "full featured" streaming client from saving the streamed content once it understands the streaming protocol. All it needs to do is to re-assemble the pieces from the streaming format back into the file format. Actually today if you do a search with "%ProtocolName% download" (replace %ProtocolName% with the name of the streaming protocol of "the company"), you would find that there're already tools out there which knows how to interpret this protocol and then save the streamed content to local files. The reverse engineering effort seems to have largely succeeded.

Using proprietary streaming protocol has many other limitations outside of the scope of content protection. For example, you would be locked into one particular type of streaming server, media client and file/codec format. Your contents can not easily flow to other type of clients which support standard protocols. It would be next to impossible to add support for new media formats, etc. These are all important things to consider.

Ok, with all that, let me try to summarize the points I'm trying to make here:

1. Streaming does not automatically give you content protection because they're not tied to each other. Content protection mechanisms can also be used with progressive download from web server. Streaming is just another way to deliver media contents. Streamed content can always be reassembled back into the file format. In another word, streamed content can also be "downloaded".

2. There're still things you can do on web server to better protect your content from being saved and trivially ‘used’ by well-behaved browsers. Disabling caching and using SSL are some examples. More advanced content protection mechanism can be implemented for progressive download by implementing modules and code on web server and media clients. This is the same idea as the watermark module developed by Fabio for protecting images. Modern web server like IIS7 has become a powerful programming platform on top of which you can develop all kinds of cool solutions.

3. The concept of proprietary streaming protocol provides content protection is NOT time-withstanding. Without a real content protection mechanism, tools that understand that protocol can easily capture the contents. Using proprietary protocols has many other limitations that might cause business concerns.

To be fair, I still need to acknowledge the fact that today it is indeed a lot easier to download and save contents from web server than from streaming server. For example, some tools do not rely on browser cache, instead they directly intercept the URL to the media file and download it separately. Some tools have made it as simple as a single-click experience. As I said above, you could also find tools that can "download" and save streamed contents but those tools are not so popular today. In my view, this is mostly due the fact that most media sites today still use progressive download to deliver their contents for its low cost and ease of use. If some day a particular streaming protocol becomes very popular, I bet capturing and downloading with that streaming protocol is just going to be as easy as downloading web contents today. In that sense, if you move to streaming just for content protection, you may get some benefits for the short run. But in the long term, you're really betting that the streaming solution you chose will NOT be mainstream. Because once it is, tools will be greatly enhanced and the perceived benefit of "content protection" will disappear immediately. But if you do win the bet and get the "protection" you want, which means you ended up choosing an unpopular streaming solution, wouldn't you be losing more?

Thanks for reading.

-Sam

Streaming == Content Protection? (Part 1)

samzhang — Fri, 23 May 2008 01:38:00 GMT

Recently I got a chance to talk to a customer who does a lot of media streaming as well as progressive download. One thing he mentioned was that more and more people start to believe that if they want content protection, they need to switch to streaming instead of using progressive download from a web server. I know that this idea has been floating around for quite some time but I never imagined that content protection would be a major reason why people want to do streaming. Because technically they're just two different things.

The fundamental difference between streaming and progressive download is the protocol which defines how client controls the media as well as how media data is packaged on the wire. For progressive download from a web server, standard HTTP protocol is used and the binary format on the wire is the same as the file format. For streaming protocols, like RTSP/RTP, media specific control commands (like pause, seek, fast forward and rewind) are defined. Also when transferring the content, the original media file is broken into meaningful pieces (like headers, streams and samples) and repackaged into streaming protocol format. So in streaming case, the binary format on the wire is not the same as file format. This repackaging, however, does NOT alter the actual media data in any way. All the samples and header information stay the same. No encryption is involved. All the data is till in clear. It's sort of like converting your old word .doc file to the new .docx format. Yes, the format did change but whatever the original document has is still there in the new format.

Content protection typically happens at different layers. I usually categorize content protection into two categories: link protection and file protection. Link protection is for, as the name suggests, protecting the link. In another word, it prevents third-party application or entity that sniffs or hijacks the content transfer session. An example of link protection would be SSL. The payload is encrypted before getting transferred through the link and decrypted as soon as it reaches the destination. Link protection is usually associated with the transfer session, not the actual content or file. Because link protection is only protecting the link, it does not have any control over how the media file is used once it reaches the client machine. If you want to have control over the media file itself, you would then need a file protection mechanism. DRM is an good example of media file protection. For DRM content, the media data (usually samples) is already encrypted when the file was originally created. Web server or even streaming server can be agnostic about the media encryption. Because all they need to do is to deliver bits or samples. Whether the content is encrypted or not does not really matter. Decryption for DRM content happens when the user tries to view the content. Client machine needs to contact license server to get the license and decrypt the content.

So as you can see, content protection is not tied to streaming. Yes, you can do streaming with SSL or DRM content. But you could also do progressive download with SSL or DRM content. Streaming gives you more functionality for transferring and viewing the media content by using more specialized protocol, but from network delivery's perspective, it's just another kind of pipe. Content protection is a special sauce that can be added separately and optionally to both streaming or progressive download.

Ok, this is all about theoretical analysis. Next time in part 2, I will go into the details about the claims that some company are making about "Streaming == Content Protection". You feedback is always appreciated.

-Sam

Web Playlist and Bit-rate Throttling - Working together

samzhang — Tue, 06 May 2008 23:54:00 GMT

Hello, my name is Sam Zhang (or 章葛强 if you can read Chinese) and I'm a lead developer on IIS Media Pack team. I've always wanted to start my own blog to talk about IIS and media but couldn't do it until now. J IIS Media Pack team is a relatively new team with a vision to enable advanced media scenarios on IIS platform. This team has a group of bright and smart people who are really passionate about web media technologies. I hope you will find useful information in my blogs and at the same time provide us your valuable feedbacks.

I think you've probably known or used the two Media Pack modules we've released so far: Bit-rate throttling 1.0 and Web Playlist CTP2. There have already been some great posts about the features and functionalities in these two products. So I won't go into those details again. Today I'd like to talk about how these two modules fit into IIS pipeline and how they work with each other.

First, let's talk about Web Playlist. If you have used Web Playlist, you probably know that Web Playlist by default handles requests with ".isx" extension. In Web Playlist, ".isx" is defined for "default playlist provider" and you can add your own playlist extension by adding a playlist provider. In IIS7 pipeline, the component that handles requests and produces responses for a particular extension is called "request handler". For example, ASP.net registers a request handler with ".aspx" extension. In IIS7, request handler only gets invoked if the request comes with the extension that it registered with. In our case, default web playlist handler is only called if the request has ".isx" extension.

For Bit-rate Throttling (BRT), the design approach is different. It's less about "what to return" but more about "how to return". In another word, BRT's job is to control how fast we can send the response rather than figuring out what response needs to be sent to the client. So with that, it was naturally implemented as an IIS7 module instead of a handler. In IIS7, a module listens to one or multiple event notifications which are fired for all the requests. BRT mainly listens to RQ_SEND_RESPONSE event which is fired right before the response is going to be sent. All the magic of doing bit-rate control happens at that point.

So, if you have both Web Playlist and BRT installed and enabled, they both exist in the request processing pipeline but get invoked at different stages. At the same time, they're totally independent with each other as well. For example, it's perfectly fine to just run Web Playlist without BRT if bandwidth consumption is not a concern. Because BRT is implemented as a module in IIS7 pipeline, it can handle all type of responses, static file or dynamic responses, as long as you have the right throttling rules defined.

So how exactly is BRT working with Web Playlist? If you have used Web Playlist, you probably know that the entry URLs embedded in the ASX response are obfuscated. There's no way to tell which content is going to be returned just by looking at the URL. Web Playlist keeps track of the user session state and communicate with playlist provider to find out the exact content at runtime. Can it still be properly throttled? The answer is yes, and you get the same behavior as if the client is requesting a static media file. BRT does not rely on the request URL to determine where to look at. Instead, it directly intercepts the response entity and find out what is actually being returned. If the response contains a file handle, it knows how to trace back to the original location of the file, get the file extension and apply the corresponding throttling rules. If it's a media file, it will follow the same logic of parsing the encoded bit-rate from that file and use the proper media throttling rules (e.g. 20s/100%). So all you need to do is to set the right throttling rules for the file type you want to return with Web Playlist, and everything else will happen automatically. It works not only with entries entered with relative URIs in playlist but also with entries located by physical path which does not need to be in the public URL namespace. One thing to note here is that BRT won't work with remote URLs you set in your playlist because the actual downloading is going to happen on the remote server (well, you could still install BRT on your remote server J).

As you can see, BRT can easily work with static files as well as dynamic responses. Web Playlist is just a good example here. You can write your own request handler to dynamically return media files or any type of responses and BRT will just work.

Want to give it a try? J