How to Troubleshoot Live Smooth Streaming Issues? – Part 7 (Putting It Together)
6. Putting it all together
In previous sections, we discussed many different ways of diagnosing live smooth streaming issues. Now, let’s go through a troubleshooting example to see how to use them in real world debugging.
In this example, first we started a live smooth streaming session with three video bitrates (950k, 1500k and 1950k) and one audio bitrate (48k) as shown below:
As you can see, we have three streams with ID “8872-stream0”, “8872-stream1” and “9880-stream0” for all the video and audio tracks. Stream “8872-stream0” contains the 48k audio and 1950k video. Stream “8872-stream1” contains a single 950k video. Stream “9880-stream0” contains the 1500k video. All tracks are in “started” state with healthy “Incoming Bit Rate” and aligned fragment timestamps.
These three streams can also be validated from the source view and the worker process view:
Now let’s say I open up my smooth streaming player to play the stream and everything is great. But after some period of time, suddenly my player stopped working.
Here is how I would troubleshoot this case:
1. Use Fiddler (or any other network monitoring tool) to check if the client is getting any HTTP error responses. In this case, I found a lot of 412 error codes in the network trace.
2. Open up the IIS log on the server side to check if the server is indeed returning those error codes. This is to make sure that those error responses were not caused by some cache/proxy servers in between. In this case, yet I did find the corresponding IIS logs for those 412 errors:
1411 2011-03-15 01:27:55 fe80::745a:dc21:3afa:bad8%18 GET /test.isml QualityLevels(48000)&Fragments(audio=4867366893) 80 - fe80::745a:dc21:3afa:bad8%18 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+InfoPath.3;+MS-RTC+LM+8;+.NET4.0C;+.NET4.0E) 412 0 2156396547 6059 3
As you can see, this 412 error response is for the audio stream. If you recall, in the “IIS Logs” section, we discussed a few scenarios under which 412 could be returned by the server.
3. By checking the publishing point UI, I found that somehow the audio track and two video tracks (1500k and 1950k) are showing zero incoming bitrate and their fragment timestamps are no longer updating. Also note that the corresponding streams are still in “Started” state which seems to indicate that they experienced some unexpected error.
4. To confirm the stream status, I check the “Sources” section on the publishing point UI. Indeed those two streams, namely “8872-stream0” and “8872-stream1” are now missing from the source list.
5. From the worker process view, we can see that IIS is no longer processing the POST requests for those two streams.
6. To find out exactly what happened. I switch to Windows Event Viewer and found the following two events.
Source '::1' of stream '8872-stream1' was unexpectedly disconnected from publishing point '/test.isml'. Error: The I/O operation has been aborted because of either a thread exit or an application request.
- 0X800703E3.
Source '::1' of stream '8872-stream0' was unexpectedly disconnected from publishing point '/test.isml'. Error: The I/O operation has been aborted because of either a thread exit or an application request.
- 0X800703E3.
So just as we suspected, these two streams unexpectedly disconnected from the publishing point. The event information also tells us when the disconnect happened. There was no event indicating the encoders sent EOS signal to the publishing point. Also the events confirmed that those two streams did not reconnect back to the server which probably should have happened if I had an encoder that supports retry and auto-reconnect.
Basically for this example, I was using two simulated live smooth streaming encoder instances and I killed one of them in the middle of streaming. 
7. Summary
Ok, finally we’re coming to the end of this series of blogs for live smooth streaming troubleshooting. Hopefully by now you’ve got some better understanding about how live smooth streaming works and how to monitor and diagnose the servers. IIS Live Smooth Streaming is still a new technology in its early stages. We’re working hard to make it better with every release to give you the best live streaming experience with the lowest cost. Please let us know what improvements you would like to see in our Microsoft Smooth Streaming platform by voting your opinions here.
Thanks for reading thus far.