Sam Zhang's Blog

Smooth Streaming Client SDK Beta for Windows 8 Released

samzhang — Mon, 12 Mar 2012 16:51:28 GMT

We just released Smooth Streaming Client SDK Beta for Windows 8. It enables developers to build Windows 8 Metro Style applications that consume On-Demand and Live Smooth Streaming content w/ PlayReady protection. If you have any questions or feedback, please contact us through this.

Smooth Streaming to XBox and Beyond

samzhang — Sat, 14 Jan 2012 02:17:54 GMT

Last month, Microsoft announced the new TV platform for XBox and a long list of content providers that will be bringing their TV and premium content onto XBox. For more detailed information, please check the press release here and here.

As someone from the IIS Media Services team where Smooth Streaming technology was born, I’m proud to announce that the majority of Xbox Live content providers are using Smooth Streaming as their video streaming technology. Also all content partners who are delivering protected premium Smooth Streaming video are using PlayReady DRM for their content protection. Our team not only provided Smooth Streaming server platform (IIS Media Services) but also delivered Smooth Streaming Client SDK to our partners to build their XBox applications upon. Combining the power of XBox platform with the wealth of TV and premium content is going to be a huge win for both consumers and content providers.

After a few years of evolution, Smooth Streaming now goes far beyond the initial scope of Silverlight online streaming. Today you can use Smooth Streaming to reach a variety of different kinds of clients/devices including browsers with Silverlight, XBox, Windows Phone, Apple iOS devices (iPhone/iPad), Windows 8 (in developer preview as of now) and TVs/STBs(Set-top Boxes). For example, Comcast’s popular Xfinity TV app for iPad/iPhone has been running on top of the Smooth Streaming iOS SDK and delivers TV content using Smooth Streaming technology (link). We also provide Smooth Streaming Client Porting Kit which can be used to easily integrate Smooth Streaming functionalities into many kinds of devices including TVs and STBs.

Smooth Streaming is right at the center of this new wave of revolution for TV and premium video delivery. The journey has just begun.

Smooth Streaming Case Studies

samzhang — Thu, 08 Dec 2011 21:52:59 GMT

Here are a few links to smooth streaming case studies based on some real world solutions that we helped build over the years. Hopefully you would find them interesting:

The World's First IIS Live Smooth Streaming Event

CTV

NBC Sunday Night Football

NBC 2010 Winter Olympics

France Télévisions

NRK

Indian Premier League

Internap

Fragmented MP4 vs. MPEG2-TS

samzhang — Fri, 02 Dec 2011 20:08:15 GMT

Today there are a few popular HTTP adaptive streaming solutions in the industry including Microsoft Smooth Streaming, Adobe HTTP Dynamic Streaming and Apple HTTP Live Streaming (HLS). One important difference among these solutions is the file format that’s being used. More specifically, both Microsoft Smooth Streaming and Adobe HTTP Dynamic Streaming use Fragmented MP4 file format while Apple HLS uses MPEG2-TS.

Very recently, a white paper was published by Tim Siglin, a well-known digital media consultant and writer with Transitions, Inc., that compares these two file formats and discusses in detail about why Fragment MP4 is a superior format than MPEG2-TS for HTTP adaptive streaming. Microsoft and Adobe also have jointly contributed to this white paper. Read from the links below if you’re interested:

File Write-Through Behavior in Live Smooth Streaming Archiving

samzhang — Thu, 17 Nov 2011 02:40:13 GMT

In IIS Media Services 4.0, we introduced the file write-through behavior with Live Smooth Streaming when doing DVR/Archiving from the publishing point. What does file write-through mean? Here is a bit of background:

On Windows, the operating system’s file cache is an important layer between the application and the physical disk. It tries to do a lot of intelligent things to cache file content in memory to improve the overall I/O performance. In Live Smooth Streaming, the server always does cache-enabled disk writes when doing archiving and it makes perfect sense: the newest incoming data that just got written to disk is most likely to be requested by clients immediately. Having those data cached in memory can greatly improve the server’s performance.

The default behavior for cache-able file-write is that when server writes a block of data using Win32 file API, the OS’s file cache first caches the data in memory instead of immediately committing it to the physical disk. Later the file cache will do bulk writes to physical disk when it accumulates enough data. In general, this approach is more efficient than committing to disk every single time.

When we were testing IIS Media Services 4.0 release on various platforms, we found an interesting behavior caused by the OS’s file cache. In some cases, when the server is running with a good number of concurrent live publishing points, due to the rate of the total incoming streams, the OS’s file cache could end up using too much physical memory before it decides to flush the cached data to disk. This could cause two problems:

1) Other OS components could starve on memory because most of the memory was used for write cache.

2) When the OS’s file cache finally decides to flush data to disk (to free up cached memory), due to the size of the data and the fact that the system is short on memory, it could cause a few seconds of system unresponsiveness which could result in things like dropped encoder connections or long response time to the client.

In our testing, this problem tends to happen on servers with low RAM (e.g. 4GB) and with Windows Server 2008. The file system team did some improvement in Windows Server 2008 R2 which helped alleviate the issue.

So in order to make our system run well in those conditions, we enabled this file write-through flag in IIS Media Services 4.0. This flag tells the file system to commit all file writes to the disk immediately while still keeping a copy of the data in memory for read cache purpose. This allows us to avoid the late-flush behavior while still getting the benefit of read cache. This is because read cache memory (the standby list) can be easily reused by other OS components so that the system’s memory usage will always stay in a stable state.

Now in IIS Media Services 4.1 release, with more adoption of Windows Server 2008 R2 and newer servers being equipped with more memory, we are making this file write-through behavior configurable so that you could get some extra performance gain if you’re running the latest OS and hardware.

Ok, hopefully you made some sense out of this long context. Anyway, here is what really matters to you:

If your server has at least 16GB of memory AND running Windows Server 2008 R2, it’s recommended that you turn OFF the file write-through behavior by changing the following IIS configuration setting:

system.webServer/media/liveStreaming/allowDiskWriteThrough

Setting this value to FALSE will disable the file write-through behavior and potentially give you better server performance. In our internal testing, the server could get up to 30% of performance improvement in live archiving which allows the server to either take on more publishing points or serve to more clients.

When you try this new setting, please keep a close eye on the available physical memory (“Memory”->”Available MBytes” performance counter) overtime and make sure the system’s memory usage is relatively stable. Please let us know if this setting helped you or you ran into some other issues.

Performance Tuning for On-Demand Smooth Streaming

samzhang — Fri, 11 Nov 2011 01:05:11 GMT

This is another blog post that has been long due. I’ve written a lot about Live Smooth Streaming in my past blog posts but so far very few on On-Demand Smooth Streaming. Well, partially because getting On-Demand Smooth Streaming to work is quite simple. You just need to copy your content to the server and then everything should just work. For low volume usage, that’s pretty much all you need to care about. But if you are doing serious production systems with a large content library and significant client traffic, you would want to tune the system to its best performance in which case this blog post should be helpful to you.

1. Overview

First let’s take a quick look at what’s the typical server logic when serving On-Demand Smooth Streaming requests:

Figure 1. On-Demand Smooth Streaming Server Data Flow

Client issues a request to get the client manifest (.ISMC file in the diagram). From the manifest file, client figures out which bitrate and timestamp it wants to request from the server.
Client issues a fragment request as shown in the diagram (first video fragment at bitrate 300kbps).
Using the server manifest file, server maps the incoming request to a particular ISMV file that contains the requested bitrate.
If it’s the first time that the server serves any fragment request from this file, server would need to first locate the “index” data at the end of the file which is based on ISO Fragmented MP4 file format.
From the index, server locates the corresponding fragment in the file and send it back to the client.

Note: This describes a typical data flow for Smooth Streaming requests. Apple Http Live Streaming (HLS) requests handling in IIS Media Services follow a similar data flow but based on different file formats (MPEG2-TS).

As you can see, the server logic is quite simple. For the most part, all it does is mapping incoming requests to sections in the media file and then serve them out. For most On-Demand Smooth Streaming deployments, the performance bottleneck is the disk I/O because the server needs to read a lot of media fragments and its logic is mostly dealing with disk read I/O with very minimum processing overhead. If we drill down into the disk I/O, in most cases, the majority of disk time is spent in seeking the disk head to locate the next fragment. This is especially true if you have a CDN which blocks most of the redundant requests which means the requests to the IIS Media server would be mostly random. As you may know, random I/O read is much slower than sequential I/O read for spindle-based disk storage.

So naturally, the performance tuning of On-Demand Smooth Streaming server is going to be mostly disk I/O related.

2. Caching, Caching, Caching

How do we improve the performance for On-Demand Smooth Streaming? – By doing lots of caching. We try to cache everything that could be cached to reduce disk I/O cost and improve performance. Here are the things that the server tries to cache:

Manifest Files – These XML based files are typically small so the server would try to cache the entire file in memory after reading it.
File Handles – Opening/closing file handles could be an expensive operation when the system is under load. The server can optionally cache all the opened file handles so that it can reuse them for future I/O reads. There are a couple of configuration settings that are related to file caching.
Critical information in ISMV Files – This includes the “Index” information at the end of each ISMV file as well as some specific information about each fragment. Note that it does NOT include the actual media data in all the fragments which is much bigger in size.
Media Content – This is the actual media data in all the fragments. Server tries to leverage Windows OS’s file cache to effectively cache those data and manage the lifetime of the caches.

So for the most part, performance tuning for On-Demand Smooth Streaming is all about tweaking these caching behavior through various configuration settings which will be discussed below.

3. On-Demand Smooth Streaming Configuration Settings

Below is a table listing the available configuration settings for On-Demand Smooth Streaming in IIS Media Services 4.0 and 4.1 (“*” denotes new in 4.1). The configuration path to this section is “system.webServer/media/smoothStreaming”.

Name	Description	Note	Default Value
enableServerCache	This setting controls whether HTTP.sys kernel cache is enabled for media responses.	If you are running Windows Server 2008 SP1 or later versions of Windows server (including Windows Server 2008 R2), you can safely turn this option on. It is disabled by default due to an issue in IIS7 in Windows Server 2008 RTM which was fixed in SP1. However due to the default size limit of HTTP.sys cache and the fact that most likely you’ve already turned on other caching mechanisms (like file cache), you may not see significant performance gain by turning on this option.	False
serverCacheTimeout	TTL for HTTP.sys kernel cache entries	Only used if enableServerCache is TRUE.	5 sec
fileHandleCache - enableFileHandleCache	Enable/disable file handle caching	Turning this option on would allow the server to cache file handles. This option should be turned on in most cases as it usually gives significant performance boost.	True
fileHandleCache - fileHandleCacheCountLimit*	Sets a limit on how many file handles the server should cache.	The default value zero means no limit in file handle caching which is the recommended setting for most deployments. This option is useful in certain scenarios where the storage device (e.g. some NAS devices) only supports a limited number of concurrent file handles/descriptors. When the cached file handle reaches the limit, server would use a LRU (Least Recently Used) algorithm to retire the least “popular” files to make space for the new ones.	0
fileHandleCache - fileHandleType*	Specifies what type of file handle the server should open: CacheOnly – One file cache enabled file handle for each file. NonCacheOnly – One file cache disabled file handle for each file. Both – Two file handles, one being file cache enabled and one being file cache disabled, for each file.	With file cache enabled file handle, the operating system’s file cache can help caching the media content in physical memory to increase the read performance. This is especially useful for frequently requested fragments. However, for “unpopular” fragments, enabling file cache actually wastes some cycle as the chance of reusing it is small. “Both” is the recommended value for most scenarios. The other two options could be useful when dealing with certain storage devices. “Both” is also the built-in behavior in IIS Media Services 4.0 release.	Both
frequentHitThreshold	Server uses a combination of frequently hit threshold and time period (see below) to determine if a particular fragment should be deemed “popular” and thus cached by file cache. For example, with the default values, fragments that were requested at least two times in the last eight seconds would be considered “frequentHit” and enabled for file cache.	This setting is only effective if the fileHandleType (see above) is set to “Both”.	2
frequentHitTimePeriod	See above	See above	8 sec
enableFileMonitoring*	When file handle is cached, this setting instructs the server whether it should monitor file changes.	When this option is turned on, server keeps monitoring any changes to the file whose handle is being cached by the server. If the file gets updated, server would trigger a refresh logic to reopen and re-parse the file. This is also useful in NAS case when the file handle becomes invalid due to either timeout or temporary network issues in which case server would try to re-open the file handle. You should only turn this option off if you don’t expect any file changes or storage device errors.	True
fileMonitoringInterval*	How often the server should check the file for potential changes as discussed above.	This is the minimum interval for file monitoring. The timing of the file checking logic is also driven by client requests. So the actual monitoring interval could be longer than the specified value if the file is not being frequently requested.	5 sec
indexCacheSizeLimit	How much memory the server should use for caching index data in ISMV files.	Even though the index data in ISMV file is pretty small, it could still occupy a significant amount of memory if you have a very large content library. If you see the worker process (w3wp.exe) is using too much memory by itself, you could set a limit on the index cache size. The default value of zero means no limit which should be ok for most cases. When the memory limit is reached, server would use the same LRU algorithm (as discussed in fileHandleCacheCountLimit) to delete index data from the least “popular” content to make room for the newer content while always keeping the total memory usage below the limit.	0
enableFragmentValidation*	Specify whether the server should perform high level data validation for each fragment.	Each fragment should comply with ISO MP4 spec by having the right MP4 box structure. With this feature turned on, server would perform some simple checks before sending it out to the client. This check does incur some additional I/O cost due to extra I/O read. So if your content is coming from trusted encoder products, you can consider turning this option off.	True
enableClientCache	Instruct the client whether it should cache the response	This affects the HTTP cache control header that server sets in the response. It has no impact on local server performance. However turning if off could mean more client requests coming back to the server because no HTTP caching can/should happen downstream.	True
cacheControlHeader	Set the TTL for cache control in HTTP response.	Only effective if enableClientCache is set to true. It has no impact on local server performance with the same caveat as above.	max-age=7200

There is one more configuration setting that’s related to On-Demand Smooth Streaming server’s performance which lives under "system.webServer/media/common". It’s called "systemFileCacheRAMPercentageLimit" which is for setting the upper limit on how much physical memory (RAM), percentage wise, the operating system should use for file cache. The default value is 50% which could be a bit conservative for high end machines with more RAM. You can try to tweak this number higher to leverage more physical memory for file cache. This is a global setting for the machine.

4. Dynamic UNC

If you compared the schema file of IIS Media Services and the table above, you might notice that I omitted a configuration element called “dynamicUNCResolution” (and its child settings) in the “system.webServer/media/smoothStreaming” section. This is a new feature we added in IIS Media Services 4.1 specifically for NAS (Network Attached Storage) scenarios where media content is accessed through UNC paths. The diagram below shows a typical configuration how this feature can be used.

Figure 2 – Dynamic UNC Configuration

The idea of Dynamic UNC is that it allows the server to simultaneously leverage multiple NAS nodes for better I/O throughput while making it completely transparent to the client and IIS core pipeline (in terms of virtual directory settings).

Here is how to set it up:

Setup a virtual directory on IIS as you normally would with a single UNC path, say \\NAS1\content .
Enable Dynamic UNC feature in IIS Media Services by setting “enabled” to True (under “dynamicUNCResolution”).
Pick a routing algorithm via the “routingAlgorithm” configuration setting. There are two options:
- CARP (Cache Array Routing Protocol) – CARP is usually used in CDNs to efficiently route the traffic to upstream cache nodes. It creates an affinity between an URL and a upstream node so that each upstream node only needs to deal with a subset of the content library. With a sizeable content library, it is also able to evenly distribute the load among the upstream nodes while maintaining the URL-to-node affinity.
- Round-robin – This is a simple routing logic that’s based on round-robin logic.
Edit the "UNCMappingList" configuration element to add entries for all the UNC nodes. You can use the built-in Configuration Editor in IIS management UI to do that:
- The “hostName” property should be the same as the virtual directory path (\\NAS1\content in this case).
- Add one entry for each of the NAS device you have. In this example, you would add four of them (NAS1 to NAS4). The “address” property for each entry can be either host names or IP addresses. If you use CARP, a list of similarly constructed host names or IP addresses would work better in terms of distributing the load more evenly, e.g. NAS1, NAS2,…. or 192.168.1.2, 192.168.1.3, … etc.

After setting it up, you need to make sure that each of the NAS node has access to the same content library (either by copying the content to each NAS machine or having them be part of the same storage cluster).

This Dynamic UNC feature can be useful in many different ways:

It enables IIS Media Services to work really well with some high-end NAS systems which expose multiple nodes from a single cluster. It allows the server to use a single “mount point” (configured as the virtual directory) to target multiple NAS nodes at the backend without the need to fiddle with multiple virtual directories and URL rewriting rules.
It allows the IIS Media server to use more SMB sessions to communicate with the NAS backend due to the usage of multiple UNC host names. This can give the system a significant boost in SMB throughput on SMB 2.0 (on Windows Server 2008) and SMB 2.1 (on Windows Server 2008 R2).
For low end scenarios, it allows you to build a high-performance storage backend with cheap machines and disks (no RAID, no cluster, etc.). It gives you an easy way to assemble more CPU power and, more importantly, disk throughput for media streaming. The built-in health monitoring feature (discussed below) completes the story by giving it a reasonably good fault tolerance capability.

The Dynamic UNC feature also has the capability to monitor the health of the NAS node and seamlessly re-route the traffic in case of node failure. Any node failure will also be reported through Windows Event Log. The “errorRetryIntervalMS” configuration setting can be used to instruct how frequently the server should check the status of each NAS node. When a node failure happens, server would first mark the failed node “unhealthy” and immediately re-route the traffic to other nodes (supported in both CARP and Round-robin mode). Server also has logic to periodically retry the failed node to see if it went back up in which case it would put the node back into the workgroup again. So everything you need for a simple failover/recovery scenario should be automatically handled.

5. Summary

As you can see, getting a On-Demand Smooth Streaming server to work is not difficult, but making it work really well with the best possible performance requires a lot more thinking and tuning. In IIS Media Services 4.1, we did a lot of work to push the server performance further with more configuration settings for fine tuning. We also put a lot of effort in better NAS integration which allows IIS Media Services to run much more efficiently in those environment. In some of the real world deployments, we saw the overall server throughput increased by multiple folds with NAS clusters.

As usual, I hope this blog is helpful to you. Please go ahead and download the latest IIS Media Services 4.1 release and let us know what you think.

IIS Media Services 4.1 just released!

samzhang — Wed, 09 Nov 2011 19:53:45 GMT

We just released IIS Media Services 4.1 that adds support for REST services API's for management of publishing points as well as performance improvements for both on demand and live scenarios. Read more from here.

IIS Smooth Streaming getting deployed for Comcast Xfinity

samzhang — Mon, 11 Apr 2011 19:17:30 GMT

Yes, the news is out and we can finally talk about it! Our IIS Smooth Streaming platform, together with Microsoft PlayReady DRM technology, is being deployed for Comcast’s Xfinity online service. We’re very excited about this opportunity and the statement below says it all:

“We’re always looking at technologies that will help us deliver high-quality, premium video to a broad range of IP-connected devices,” said Charlie Herrin, senior vice president of product development and technology for Comcast Interactive Media. “This new, integrated solution from Microsoft and thePlatform supports our ongoing efforts to deliver great entertainment experiences for our customers on all screens and devices.”

Go Smooth Streaming!

How to authenticate ONLY the encoder streams but not the clients for live smooth streaming?

samzhang — Tue, 29 Mar 2011 21:25:28 GMT

This question was asked quite a few times from our customers and on the media forum. The scenario is that I, as an site administrator, want to authenticate encoder streams that are pushing in for live smooth streaming. However, I don’t really want all the smooth streaming players having to do the same authentication. This is definitely a valid and reasonable scenario. An analogy is that I want to authenticate users who want to upload content to my web site without the need to authenticate the browsers.

To enable this, first let’s review some basics of live smooth streaming. The key thing that we will be leveraging here is the fact that encoder connections are all POST requests while the client requests all use GET verb. Given that IIS Live Smooth Streaming is built on top of IIS platform, we can enable this by using standard IIS authentication and authorization mechanisms.

So here are the steps:

1) Enable the authentication scheme of your choice (“Basic Authentication” is used here as an example) in addition to the “Anonymous Authentication”. This can be done in the “Authentication” module in IIS Manager.

2) In the “Authorization Rules” module, remove the default “Allow All User” rule.

3) In the same “Authorization Rules” module, add a new rule to allow all users with GET verb. This rule will allow anonymous GET requests coming from the smooth streaming clients.

4) In the same “Authorization Rules” module, add a new rule to restrict the users for POST requrests (in this case it’s “sam-oob\sam”). Those users are the only ones that can post encoder streams to the publishing points:

5) Now you should have the following as the authorization rules. Make sure that you check it at the level of your publishing point.

Ok, now if I use Expression Encoder to push to my publishing point, I would get the following dialog box asking for credentials:

So I enter the password for user “sam-oob\sam” as I configured in step (4). Bingo! the encoder is now able to connect to the publishing point through an authenticated connection. If I bring up my smooth streaming player, it is still able to play from this publishing point without any authentication.

Problem solved.

How to Troubleshoot Live Smooth Streaming Issues? – Part 7 (Putting It Together)

samzhang — Tue, 15 Mar 2011 06:52:00 GMT

6. Putting it all together

In previous sections, we discussed many different ways of diagnosing live smooth streaming issues. Now, let’s go through a troubleshooting example to see how to use them in real world debugging.

In this example, first we started a live smooth streaming session with three video bitrates (950k, 1500k and 1950k) and one audio bitrate (48k) as shown below:

As you can see, we have three streams with ID “8872-stream0”, “8872-stream1” and “9880-stream0” for all the video and audio tracks. Stream “8872-stream0” contains the 48k audio and 1950k video. Stream “8872-stream1” contains a single 950k video. Stream “9880-stream0” contains the 1500k video. All tracks are in “started” state with healthy “Incoming Bit Rate” and aligned fragment timestamps.

These three streams can also be validated from the source view and the worker process view:

Now let’s say I open up my smooth streaming player to play the stream and everything is great. But after some period of time, suddenly my player stopped working.

Here is how I would troubleshoot this case:

1. Use Fiddler (or any other network monitoring tool) to check if the client is getting any HTTP error responses. In this case, I found a lot of 412 error codes in the network trace.

2. Open up the IIS log on the server side to check if the server is indeed returning those error codes. This is to make sure that those error responses were not caused by some cache/proxy servers in between. In this case, yet I did find the corresponding IIS logs for those 412 errors:

1411 2011-03-15 01:27:55 fe80::745a:dc21:3afa:bad8%18 GET /test.isml QualityLevels(48000)&Fragments(audio=4867366893) 80 - fe80::745a:dc21:3afa:bad8%18 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+MS-RTC+LM+8;+.NET4.0C;+.NET4.0E) 412 0 2156396547 6059 3

As you can see, this 412 error response is for the audio stream. If you recall, in the “IIS Logs” section, we discussed a few scenarios under which 412 could be returned by the server.

3. By checking the publishing point UI, I found that somehow the audio track and two video tracks (1500k and 1950k) are showing zero incoming bitrate and their fragment timestamps are no longer updating. Also note that the corresponding streams are still in “Started” state which seems to indicate that they experienced some unexpected error.

4. To confirm the stream status, I check the “Sources” section on the publishing point UI. Indeed those two streams, namely “8872-stream0” and “8872-stream1” are now missing from the source list.

5. From the worker process view, we can see that IIS is no longer processing the POST requests for those two streams.

6. To find out exactly what happened. I switch to Windows Event Viewer and found the following two events.

Source '::1' of stream '8872-stream1' was unexpectedly disconnected from publishing point '/test.isml'. Error: The I/O operation has been aborted because of either a thread exit or an application request.
- 0X800703E3.

Source '::1' of stream '8872-stream0' was unexpectedly disconnected from publishing point '/test.isml'. Error: The I/O operation has been aborted because of either a thread exit or an application request.
- 0X800703E3.

So just as we suspected, these two streams unexpectedly disconnected from the publishing point. The event information also tells us when the disconnect happened. There was no event indicating the encoders sent EOS signal to the publishing point. Also the events confirmed that those two streams did not reconnect back to the server which probably should have happened if I had an encoder that supports retry and auto-reconnect.

Basically for this example, I was using two simulated live smooth streaming encoder instances and I killed one of them in the middle of streaming.

7. Summary

Ok, finally we’re coming to the end of this series of blogs for live smooth streaming troubleshooting. Hopefully by now you’ve got some better understanding about how live smooth streaming works and how to monitor and diagnose the servers. IIS Live Smooth Streaming is still a new technology in its early stages. We’re working hard to make it better with every release to give you the best live streaming experience with the lowest cost. Please let us know what improvements you would like to see in our Microsoft Smooth Streaming platform by voting your opinions here.

Thanks for reading thus far.

How to Troubleshoot Live Smooth Streaming Issues? – Part 6 (IIS Logs and Others)

samzhang — Fri, 11 Mar 2011 08:04:27 GMT

5. IIS Logs

Since smooth streaming client requests are all standard HTTP requests going through IIS pipeline, all these requests are being logged by IIS logs which are located in %systemdrive%\inetpub\logs\LogFiles\. An example of the log entries is shown below:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes time-taken

2010-10-26 18:05:41 2001:4898:0:fff:0:5efe:172.30.180.211 GET /test.isml QualityLevels(48000)&Fragments(audio=41795918) 80 - 2001:4898:0:fff:0:5efe:10.80.121.212 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+MS-RTC+LM+8) 200 0 0 11640 29

2010-10-26 18:05:41 2001:4898:0:fff:0:5efe:172.30.180.211 GET /test.isml QualityLevels(950000)&Fragments(video=60060000) 80 - 2001:4898:0:fff:0:5efe:10.80.121.212 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+MS-RTC+LM+8) 200 0 0 194803 334

The fields of the IIS log entries are configurable on IIS Manager UI in the “Logging” module. In general, it provides standard HTTP logging information including URIs, IPs, port, headers, status code, time taken, etc. There are many tools available for parsing and analyzing IIS logs. One tool that I often use is the “IIS Log Parser” that’s quite convenient and flexible for mining IIS log data. I have a few IIS Log Parser queries that I often use to gather statistic information for smooth streaming:

1. Status code count ordered by hit count
select sc-status, count(*) as hits from {LogFileName} group by sc-status order by hits desc

2. Client IP count ordered by hit count
select c-ip, count(*) as hits from {LogFileName} group by c-ip order by hits desc

3. Cient IPs that were hit for more than 10 times
select c-ip, count(*) as hits from {LogFileName} group by c-ip having hits>10 order by hits desc

4. Number of hits per hour during a time span
select quantize(to_timestamp(date,time), 3600) as hour, count(*) as hits from {LogFileName} where time>'22:00:00' and time<'23:59:59' group by hour

5. Get a section of the log based on a time span
select * from {LogFileName} where time>'22:00:00' and time<'23:59:59'

6. List the top 25 mostly hit URLs
select top 25 cs-uri-stem, cs-uri-query, sc-status, count(*) as hits from {LogFileName}group by cs-uri-stem, cs-uri-query, sc-status order by hits desc

7. List the top c-ip and status for a particular fragment request
select cs-uri-stem, cs-uri-query, sc-status, c-ip, count(*) as hits from {LogFileName}where cs-uri-stem='/test.isml' and cs-uri-query='QualityLevels(48000)&FragmentInfo(audio=652979953208)' group by cs-uri-stem, cs-uri-query, sc-status, c-ip order by hits desc

8. List the max/min/avg time-taken in the entire log
select max(time-taken) as max, min(time-taken) as min, avg(time-taken) as avg from {LogFileName}

9. List the top 10 verbs and Urls with the biggest time-taken
select top 10 cs-method, cs-uri-stem, time-taken from {LogFileName} order by time-taken desc

10. List the top 10 Urls for GET requests with the biggest time-taken
select top 10 cs-method, cs-uri-stem, time-taken from {LogFileName}where cs-method='GET' order by time-taken desc

11. Combining status with sub-status
select top 10 strcat(to_string(sc-status), strcat('.', to_string(sc-substatus))) as status, count(*) as hits from {LogFileName}group by status order by status asc

6. Fragment Response Code

When you look at the IIS logs for live smooth streaming, you might sometimes see some HTTP error codes. The most common error codes for live smooth streaming fragment response are 404 and 412. Here is how these error codes could happen:

HTTP 404 – This is the standard “File Not Found” error. For live smooth streaming, it could happen when the server didn’t receive a particular timestamp for certain bitrates. It’s usually caused by some data interruption for those bitrates. As we discussed in previous sections, in this case, server would still output this fragment timestamp to the manifest after some timeout. Smooth streaming client has a retry logic to automatically switch to other bitrates for this timestamp after it receives 404 error. So in most cases, a few 404 errors won’t have noticeable impact on the client’s playback experience.

HTTP 412 – This is a special error code for live smooth streaming. It means the client request is too far ahead of the encoder stream. In general due the nature of live streaming, client should not try to get ahead of the encoder (sorry but we can’t predict the future). The potential situations that could lead to this error are:

1. The client is indeed running too fast mostly because the client machine’s local clock is running faster than the encoder’s. This is possible but usually happens after a long time of playback. Smooth Streaming Client has internal logic to deal this with condition by adjusting its internal clock to better sync with the encoder’s clock. In most cases, it should have minimum impact on the playback experience.

2. The encoder stopped sending data to the server or some bitrates are terribly late. It could happen because of network issues, bandwidth issues or encoder’s internal issues. In this case, server would still send the client 412 response because the client is still running faster than the encoder stream. But in this case it’s not the fault of the client but rather due to the encoder data loss or delay. Smooth Streaming client will still do some retries trying to resume the streaming. But if all the bitrates are down for an extended period time, client would have no choice but to stop the playback and report error.

7. Others

Since IIS Live Smooth Streaming is running inside of the IIS web stack, there is quite some generic IIS information that could be useful. I’ll be just listing a few here and you can certainly find out more with your own investigation:

6.1 IIS Worker Process View

The “Worker Processes” module in IIS Manager displays all the current requests being handled by this worker process. So for live smooth streaming, you should see all your existing encoder connections here. This can help you confirm whether a particular encoder stream is still connected. From the “Time Elapsed” field, you can also tell when the connection was created and whether some connections were newly established compared to others.

6.2 HTTP.sys Log

IIS uses HTTP.sys for low level HTTP handling. HTTP.sys logs can be found under %windir%\System32\LogFiles\HTTPERR\. The log data here mostly contains similar information as IIS log but there are a few unique fields. For example, it shows the reason code for the ending of the TCP connection which can help you debug connection drop issues (e.g. was it caused by idle timeout or client reset?). Here is the link to the latest HTTP.sys log format (http://support.microsoft.com/kb/820729#2).

6.3 Failed Request Tracing Rules (aka “FREB”)

FREB is a very powerful tool to trace IIS internal request processing. It can help you identify issues that happened before Live Smooth Streaming module in the IIS pipeline (e.g. authentication failure). There are some good articles (link1 and link2) that walks through how to enable and use it.

How to Troubleshoot Live Smooth Streaming Issues? – Part 5 (Client Manifest)

samzhang — Thu, 10 Mar 2011 08:04:44 GMT

4. Client Manifest

As we discussed earlier, smooth streaming client starts streaming by first requesting the client manifest from the server using URL template: http://{serverName}/{PublishingPointPath}/{PublishingPointName}.isml/manifest . The client manifest contains information such as stream types, parameters, bitrates and fragment timestamps. By simply examining the client manifest, you could get some useful information for troubleshooting live smooth streaming issues. In IIS Media Services 4.0, we made some tweaks to the client manifest format for better efficiency. The discussion below is based on IIS Media Services 4.0 release.

In general, if your smooth streaming player is able to start the playback with both video and audio, chances are the “static information” (stream types, parameters, bitrates, codec data, etc.) is correct. Then the most important information in the client manifest for your debugging is the timestamps. Here is a short excerpt of the timestamp section from a client manifest:

……

……

</StreamIndex>

The timestamp entries are located in the <StreamIndex> element. The <StreamIndex> element includes all bitrate for a media type (e.g. video). Each bitrate is specified as a <QualityLevel> entry. As you noticed, the timestamp entries (<c> element) are not defined as child elements for a particular <QualityLevel> but rather directly under <StreamIndex> element which means that every timestamp listed here is representing and SHOULD be available for ALL bitrates. The implication is two folds:

1. Server will try to make sure that it only publishes a timestamp to the manifest after all bitrates have reported it.

2. However, if certain bitrate is terribly late or stopped working, server would still release timestamps that were only reported from a subset of the bitrates after a timeout window. This means:

2.1 It is possible that server only has some bitrates available for a particular timestamp but not others.

2.2 Client has this logic to automatically try other bitrates if a fragment request failed for a particular timestamp.

In terms of the format definition for the <c> entry, “t” attribute is the absolute timestamp for a fragment and “d” attribute is the duration value. In IIS Media Services 4.0, the logic is that “d” (duration) is a mandatory attribute while “t” is optional for each “c” entry. It might sound a little reversed but there is a good reason for doing that – compressibility. In most smooth streaming manifests, the duration values of a stream type is a fixed value (like the example above). By only listing the “d” values, the manifest becomes much easier to compress. For example if you simply enable gzip compression on the IIS server to compress the manifest response (by enabling “Dynamic content compression”), you would get a much better compression ratio with this new format (Silverlight smooth streaming player can automatically handle gzip compressed manifest). Better yet, in our 2.2 version of the manifest format, we defined a new tag called “repeat” tag which can directly compress any duplicated durations in place (see “clientManifestVersion” attribute in this blog). For example, for the three <c> entries as shown in the above example, they could be collapsed to a single line:

If you use an encoder that supports fixed duration and you picked the right encoding settings, it is possible to completely collapse the 2.2 version of the client manifest into just a few lines and the only thing keeps changing in the manifest overtime is the values of the repeat tags (“r” attribute) for each stream. So basically your live manifest is just a tiny a few hundred bytes and it almost never grows! (This feature is supported by Smooth Streaming Client SDK 1.5 on the client side)

So now you might ask, how would the client calculate the original timestamp (“t”) value then because that’s the one that really matters to the client media pipeline. Good question and it’s not hard. First of all, there will always be a “t” value in the first <c> entry to set the base (if it’s not there it’s assumed to be zero). From there, the client can calculate all following “t”s by adding the current “t” and the “d” value. Basically the logic is that you can precisely calculate the next timestamp by adding current timestamp and duration. Reversely, this becomes a requirement for the server in that server would only omit “t” attribute if the current timestamp can be precisely calculated by adding the previous “t” and “d”. Ok, then you might ask, what if they don’t add up. This is quite possible in scenarios like encoder failover leaving gaps in the stream. In that case, server would have no choice but to explicitly output a new “t” attribute. For example:

In this example, the first fragment has t=0. The next one doesn’t have “t”, so we can calculate its “t” as 0 + 20000000 = 20000000. Now the next “t” should be 20000000 + 20000000 = 40000000. But here somehow that fragment is missing and server only had t=60000000. In this case, server explicitly output a new “t” attribute for timestamp 60000000 to indicate to the client the right timestamp and also set the right base for the next calculation. So by using this knowledge, we can arrive at this conclusion:

If you see a <c> entry with “t” attribute in the client manifest, it indicates a gap in the stream. In most cases, it’s caused by ingest data interruption on the server but sometimes it could also be caused by improper encoder implementations.

Another type of problem that sometimes happens with new smooth streaming encoder is that sometimes the timestamps across all bitrates are not correctly aligned. As we discussed above, each <c> line that contains timestamp and duration information is for all the available bitrates. So it’s mandatory that fragment timestamps across all bitrates for a particular media type (e.g. video) must be precisely aligned. If they’re not, you might see strange <c> list in the client manifest like this:

In this example, even though the encoder is supposed to always output 2-second duration fragments, it sometimes outputs some odd durations (in red) that caused unaligned fragment. In this case, the client could get many HTTP 404 errors because of the misalignment and the client’s heuristic logic would also get confused. So another advice here:

Examine the duration values in the manifest and check if there are any anomalies, which can help you identify encoder problems.

These approaches saved us a lot of time when we were debugging strange streaming issues that turned out to be caused by encoder bugs. In the next post, I’ll focus more on the fragment request/response between the server and the client.

How to Troubleshoot Live Smooth Streaming Issues? – Part 4 (Performance Counters)

samzhang — Wed, 09 Mar 2011 01:21:32 GMT

3. Performance Counters

IIS Media Services also output some performance counters for tracking live smooth streaming status. When the publishing point is running, you should be able to see performance counters under “Live Smooth Streaming” category with multiple instances each of which corresponds to a track. The naming convention of the instance name is: {Publishing Point virtual path and name}-{Stream ID}-{Track ID}-{Track Type}. The description of each counters is available in the Performance Monitor UI once you click the “Show description” checkbox. In general, the information you get here is very similar to what’s exposed through the publishing point detailed view on the UI. The advantage of using performance counter is that you can easily plot the chart or log the data using Performance Monitor tool.

There are some other system performance counters that are also very useful in monitoring live smooth streaming server:

“Web Service”->”Get Request/sec” – This is a generic IIS counter for tracking GET request rate. Since that smooth streaming clients use GET request for manifest and fragment, you can use this counter to gauge the rough client load on the server. But keep in mind that if your web server is also serving other types of contents (web page, image, asp.net, etc.), those requests would also be included in this counter.

“Physical Disk”->”%Disk Time” and others – Disk I/O is often one of the bottlenecks for a live smooth streaming server because it’s constantly doing a lot of disk writes (for archiving content) and disk reads (for serving client requests). Closely monitoring this set of counters would be important.

“Memory”->”Available MBytes” and “Cache Bytes” – To minimize the physical disk reads and writes, live smooth streaming server heavily utilizes Windows’s file cache. If you know how Windows file cache works, this set of counters would be useful to determine the state of the server in terms of memory usage.

“Network Interface”->”Packets Received Errors” and others – This set of counters report the status of the network card. If there’s any hardware/firmware issues causing data corruption or connection loss, these counters might be able to help.

“Process”->”Private Bytes”, “Handle Count”, “Working Set”, etc.->”w3wp” – Standard way to monitor the performance of a particular process. In this case, w3wp.exe is the one for IIS’s worker process. Make sure you choose the right one if you have multiple worker processes running.

Given that Live smooth streaming server runs as part of IIS on Windows Server platform, there are many other system performance counters that you might find useful when monitoring live smooth streaming server. You could also check the performance tuning guideline for Windows Server 2008 and Windows Server 2008 R2 for instructions on tuning storage, network and web server workloads.

How to Troubleshoot Live Smooth Streaming Issues? – Part 3 (Event Logs )

samzhang — Tue, 08 Mar 2011 19:54:20 GMT

2. Event Logs

The next important place to check for live smooth streaming diagnostic information is the event viewer for windows event logs. In Event Viewer, go to “Windows Logs” –> “Application” and look for events from “IIS Live Smooth Streaming” source.

IIS Media Services outputs various kinds of events for live smooth streaming publishing points. These events cover publishing point state changes (start/stop/shutdown), stream information, source information and all error information. Below is a table listing all the event templates for core publishing point operations that are used in IIS Media Services 4.0. The event template in IIS Media Services 3.0 is different and has less information than 4.0 – another reason to upgrade to IIS Media Services 4.0. Events for certain specific features (e.g. Apple iOS streaming) are not listed here.

Event ID	Message Template	Notes
3001	Publishing point '%1' was started.
3002	Publishing point '%1' was stopped.
3003	Publishing point '%1' was shutdown.
3006	Failed to start publishing point '%1'. Error: %2.
3007	Failed to stop publishing point '%1'. Error: %2.
3008	Failed to shut down publishing point '%1'. Error: %2.
3009	Stream '%1' for publishing point '%2' was started.	The ‘%1’ here is the stream ID, more specifically the value that the encoder put in the /Stream() URL section.
3010	Stream '%1' for publishing point '%2' was stopped. Error: %3.	This event indicates that a particular stream has stopped. It could be caused by: 1. Manual publishing point stop command. In this case, the error string will be “The stream was stopped.” 2. Clean ending after EOS from the encoder stream with error string “The operation completed successfully. - 0X00000000. “ 3. Other errors causing stream stop. The error string will contain the actual error information.
3011	Source '%1' of stream '%2' was connected to publishing point '%3'.	A source that belongs to a stream has connected to the publishing point. Keep in mind that there could be more than more sources for a particular stream (e.g. redundant sources). %1 is the network address of the remote source.
3012	Source '%1' of stream '%2' was disconnected from publishing point '%3'. Error: %4.	This event happens every time that a source was removed from the stream processing. It could be caused by manual publishing point stop, source clean ending or any other errors. Please refer to the notes for event 3010 for error code information.
3013	Source '%1' of stream '%2' sent End-of-Stream (EOS) fragment to publishing point '%3'.	This is the event that logs EOS. Any EOS sent by a source will trigger the corresponding stream to stop.
3014	Source '%1' of stream '%2' was unexpectedly disconnected from publishing point '%3'. Error: %4.	This event means unexpected connection drop that happened to a source. When it happens, the stream will still be in “started” state (because no EOS was sent) waiting for the encoder to reconnect.
3015	Publishing point '%1' is in an error state. Stream '%2' was disconnected from the publishing point.	An encoder stream failed to connect to the publishing point because it’s in an error state.
3016	The stream event ID for Stream '%1' is not valid. The stream was disconnected from publishing point '%2'.	If the encoder is using the event ID option (a way to uniquely identify a live session) and the validation for the event ID failed (e.g. inconsistent event ID within a session), this event will get fired.
3017	Failed to create archive path '%1' for publishing point '%2'. Error: %3.	It could be caused by invalid archive path or permission issues. This one is for the archive folder.
3018	Failed to create archive file '%1' for publishing point '%2'. Error: %3.	Similar to above but for the individual archive file.
3019	Failed to write to archive file '%1' for publishing point '%2'. Error: %3.	Failed to write new data to the archive file (e.g. out of disk space).

How to Troubleshoot Live Smooth Streaming Issues? – Part 2 (Publishing Point UI and States)

samzhang — Tue, 08 Mar 2011 00:46:54 GMT

In Part one, we discussed basic data flow and protocols being used in IIS Live Smooth Streaming. In this blog, we will discuss where all the diagnostic information for live smooth streaming is available and how to make sense out of it.

1. Publishing Point UI

The publishing point UI is the first place that you should look at when troubleshooting live smooth streaming issues. This is especially true with IIS Media Services 4.0 release which has a new detailed view about streams, tracks, fragments, etc. Please refer to my other blog here for this new UI in 4.0 release. In general, we highly recommend our customers to upgrade to IIS Media Services 4.0 release for a number of improvements. The screen capture below was done with IIS Media Services 4.0.

Publishing Point State and Summary Page

Publishing Point Detail Page

One piece of important information about publishing point is its state which is available on both 3.0 and 4.0’s UI. Understanding the publishing point state is very important for any live smooth streaming troubleshooting. Below is a diagram describing key publishing point states:

Publishing Point State Diagram

Note: In the following discussion, the term “encoder” is interchangeable with “upstream server”. They both mean where the publishing point is sourcing live data from.

Idle/Shutdown – This is the default state for any publishing point that was just created or whose worker process just restarted. Shutting down a publishing point will also put it into “idle” state. In this state, the publishing point is doing nothing – it’s not receiving incoming streams nor is it responding to any client requests. You can edit any publishing point settings as you wish and they will be picked up when the publishing point next starts.

Starting – Once the publishing point is waken up, either by manual start command or automatically by incoming streams if you have the option enabled, it enters the “starting” state. In this state, the publishing point is open for accepting any encoder stream but it’s not ready to serve any client request yet (manifest or fragment). Only after the publishing point successfully received the initial information from all streams would it transit into the next state “started”.

Started – This is the state where the publishing point is fully functioning, from both the encoder and the clients’ point of view. It receives live data from the encoder, archive the content (if enabled), generating manifest and fragments to satisfy client’s requests and relaying the streams to any configured downstream publishing points. When the publishing point is in this state, normally you should be able to see constantly updating fragment information in the detailed view on the publishing point UI. However, as I described in this blog, the fact that a particular stream or the whole publishing point is in “started” state does NOT necessarily mean that it’s actively receiving data from the encoders. In Live Smooth Streaming, the state of the publishing point is decoupled from the state of each individual incoming data connections. This design allows full flexibility in redundancy and failover scenarios. The only ways to transit from “started” state to “stopped” state are manual stop or EOS (End Of Stream) signals from the encoders. If the encoders simply dropped the connections without sending EOS or reconnect with new data, the publishing point can stay in “started” state for ever.

Stopped – This is a state that often confuses some users especially who have long time experience with traditional live streaming servers. I have a full blog here just to discuss why this state is important. When a publishing point is in “stopped” state, it’s still “half functional”. It no longer receives any data from the encoder but it will still happily serve any manifest/fragment requests from the clients. Basically this is the pure “DVR” mode. To re-iterate what’s stated above, the only two ways to get into “stopped” state are: (1) Manual stop command (either from UI or automated RSCA calls) or (2) Encoder sending EOS for all the streams.

Shutdown – This is the same state as “idle” state but I just want to describe a bit more here about how a publishing point can transit from “stopped” state to this state. The most common way is to do it through manual shutdown command. Another option is to automate the shutdown and restart process (only for push publishing point) with the “restartOnEncoderReconnect” option. You can know more about this option from here and here.

There are actually some other states that I didn’t discuss above just to keep the logic simpler. Those are either transitional or error states. For example, there is an “error” state that the publishing point can get into if any unexpected error happened and the operation of the publishing point cannot continue (e.g. wrong archive path set). There are two transitional states “stopping” and “shutting down” that you should rarely see unless for some reason the stop/shutdown operation is taking a very long time.

To be continued……