File Write-Through Behavior in Live Smooth Streaming Archiving
In IIS Media Services 4.0, we introduced the file write-through behavior with Live Smooth Streaming when doing DVR/Archiving from the publishing point. What does file write-through mean? Here is a bit of background:
On Windows, the operating system’s file cache is an important layer between the application and the physical disk. It tries to do a lot of intelligent things to cache file content in memory to improve the overall I/O performance. In Live Smooth Streaming, the server always does cache-enabled disk writes when doing archiving and it makes perfect sense: the newest incoming data that just got written to disk is most likely to be requested by clients immediately. Having those data cached in memory can greatly improve the server’s performance.
The default behavior for cache-able file-write is that when server writes a block of data using Win32 file API, the OS’s file cache first caches the data in memory instead of immediately committing it to the physical disk. Later the file cache will do bulk writes to physical disk when it accumulates enough data. In general, this approach is more efficient than committing to disk every single time.
When we were testing IIS Media Services 4.0 release on various platforms, we found an interesting behavior caused by the OS’s file cache. In some cases, when the server is running with a good number of concurrent live publishing points, due to the rate of the total incoming streams, the OS’s file cache could end up using too much physical memory before it decides to flush the cached data to disk. This could cause two problems:
1) Other OS components could starve on memory because most of the memory was used for write cache.
2) When the OS’s file cache finally decides to flush data to disk (to free up cached memory), due to the size of the data and the fact that the system is short on memory, it could cause a few seconds of system unresponsiveness which could result in things like dropped encoder connections or long response time to the client.
In our testing, this problem tends to happen on servers with low RAM (e.g. 4GB) and with Windows Server 2008. The file system team did some improvement in Windows Server 2008 R2 which helped alleviate the issue.
So in order to make our system run well in those conditions, we enabled this file write-through flag in IIS Media Services 4.0. This flag tells the file system to commit all file writes to the disk immediately while still keeping a copy of the data in memory for read cache purpose. This allows us to avoid the late-flush behavior while still getting the benefit of read cache. This is because read cache memory (the standby list) can be easily reused by other OS components so that the system’s memory usage will always stay in a stable state.
Now in IIS Media Services 4.1 release, with more adoption of Windows Server 2008 R2 and newer servers being equipped with more memory, we are making this file write-through behavior configurable so that you could get some extra performance gain if you’re running the latest OS and hardware.
Ok, hopefully you made some sense out of this long context. Anyway, here is what really matters to you:
If your server has at least 16GB of memory AND running Windows Server 2008 R2, it’s recommended that you turn OFF the file write-through behavior by changing the following IIS configuration setting:
system.webServer/media/liveStreaming/allowDiskWriteThrough
Setting this value to FALSE will disable the file write-through behavior and potentially give you better server performance. In our internal testing, the server could get up to 30% of performance improvement in live archiving which allows the server to either take on more publishing points or serve to more clients.
When you try this new setting, please keep a close eye on the available physical memory (“Memory”->”Available MBytes” performance counter) overtime and make sure the system’s memory usage is relatively stable. Please let us know if this setting helped you or you ran into some other issues.