Creating Recursive Directory Listing Files for FTP Clients
One of the changes that we made in FTP 7.0 and FTP 7.5 was to remove recursive directory listings, which are commonly retrieved by typing "ls -lR" from a command-line FTP client, which should send a command like "NLST -lR" over FTP to the server. There were several reasons why we decided to remove recursive directory listings, but the main reason was simply to reduce CPU usage on the server; recursive directory listing requests take a lot of resources to fulfill. With that in mind, both FTP 7.0 and FTP 7.5 will ignore the recursive switch on directory requests.
That being said - quite often it's pretty handy to have a full directory listing from an FTP server. From a client perspective you could probably write script to automate an FTP client to create a recursive listing, but that's a lot of work. Back in my younger days when I ran FTP sites on Unix servers, I would always create two types of list files on my FTP servers for FTP clients to retrieve:
- "ls-lr.txt" - I would create only one file of this type for my entire FTP server, which would go in the root of my FTP site and it would contain a full recursive listing of all files in my FTP site.
- "00index.txt" - I would create one file of this type in each folder of my FTP site, and each index file would contain a listing of files and their descriptions for that folder.
Of course, anyone that's been around the Internet since the days before we had HTTP and the world-wide-web should know that I didn't come up with this idea on my own - I learned it from other FTP site administrators. (And anyone who remembers those days should also recognize those two files with a strange sense of nostalgia. 00index.txt files of course led to index.htm files when WWW sites came along later, but that's another story.)
In any event, as I continued to host FTP sites over the years I have written various scripts to create recursive directory listings, and I thought that one of my scripts might make a good blog post. With that in mind, here is a Windows Script Host file that I created, which I named "ls-lr.vbs", and this script will create a recursive directory listing for an FTP site. I choose the Unix directory listing style for this script since that's the format that I have used for years and the broader number of FTP clients and users should recognize it.
Option Explicit On Error Resume Next ' Declare all variables. Dim objArguments Dim strBaseFolder Dim objFSO Dim objFile Dim objFolder Dim objSubFolder Dim objSubFile Dim lngFolderCount Dim lngBaseCount Set objArguments = WScript.Arguments ' Determine the number of command-line arguments. Select Case objArguments.Count Case 0: strBaseFolder = WScript.ScriptFullName strBaseFolder = Left(strBaseFolder,InStrRev(strBaseFolder,"\")) Case 1: strBaseFolder = objArguments(0) Case Else: MsgBox "This script takes a single argument for the" & vbCrLf & _ "starting directory, or specify no arguments" & vbCrLf & _ "to use the current directory.", vbInformation End Select ' Create a file system object. Set objFSO = WScript.CreateObject("Scripting.FileSystemObject") ' Test if the base folder exists. If Right(strBaseFolder,1) <> "\" Then strBaseFolder = strBaseFolder & "\" If objFSO.FolderExists(strBaseFolder) = False Then MsgBox "The specified folder does not exist.", vbCritical WScript.Quit End If ' Open the output file for the directory listing. Set objFile = objFSO.CreateTextFile(strBaseFolder & "ls-lr.txt") ' Define the initial values for the folder counters. lngFolderCount = 1 lngBaseCount = 0 ' Dimension an array to hold the folder names. ReDim strFolders(1) ' Store the root folder in the array. strFolders(lngFolderCount) = strBaseFolder ' Loop while we still have folders to process. While lngFolderCount <> lngBaseCount ' Set up a folder object to a base folder. Set objFolder = objFSO.GetFolder(strFolders(lngBaseCount+1)) ' Output the folder name to the listing file. objFile.WriteLine vbCrLf & _ Replace(Mid(strFolders(lngBaseCount+1),Len(strBaseFolder)),"\","/") & _ vbCrLf ' Loop through the collection of subfolders for the base folder. For Each objSubFolder In objFolder.SubFolders ' Increment the folder count. lngFolderCount = lngFolderCount + 1 ' Increase the array size ReDim Preserve strFolders(lngFolderCount) ' Store the folder name in the array. strFolders(lngFolderCount) = objSubFolder.Path ' Output the folder to the listing file. Call WriteEntry(objSubFolder) Next ' Loop through the collection of subfolders for the base folder. For Each objSubFile In objFolder.Files ' Output the file to the listing file. Call WriteEntry(objSubFile) Next ' Increment the base folder counter. lngBaseCount = lngBaseCount + 1 Wend Sub WriteEntry(tmpObject) Dim tmpAttributes Dim tmpSize ' Test for a symbolic link. If (tmpObject.Attributes And 1024) Then tmpAttributes = "lrwxrwxrwx" tmpSize = 0 ' Test for a directory. ElseIf (tmpObject.Attributes And 16) Then tmpAttributes = "drwxrwxrwx" tmpSize = 0 ' Otherwise - it's a file. Else tmpAttributes = "-rwxrwxrwx" tmpSize = tmpObject.Size End If ' Test for a read-only object. If (tmpObject.Attributes And 1) Then tmpAttributes = Replace(tmpAttributes,"w","-") End If ' Write the list entry to the output file. objFile.WriteLine tmpAttributes & _ " 1 owner group " & _ Right(String(15,Chr(32)) & CStr(tmpSize),15) & _ " " & FormatDate(tmpObject.DateLastModified) & _ " " & tmpObject.Name End Sub Function FormatDate(tmpDate) FormatDate = CStr(Year(tmpDate)) & _ "-" & Right("00" & CStr(Month(tmpDate)),2) & _ "-" & Right("00" & CStr(Day(tmpDate)),2) End Function
To use the script, copy the code into Windows Notepad and save it to your computer as "ls-lr.vbs." If you double-click the script it will use the current folder to create a recursive folder listing, and if you run this script from a command-line it can take a single argument of a folder path, or you can pass no arguments to the script in order to use the current folder. In either case it will create a file named "ls-lr.txt" in the root of the destination folder that contains the recursive directory listing in Unix format.
For example, the following listing was created from a folder in my music collection on my desktop computer:
/ drwxrwxrwx 1 owner group 0 2009-07-30 Against the Silence dr-xr-xr-x 1 owner group 0 2009-07-30 Collective drwxrwxrwx 1 owner group 0 2009-07-30 Speakeasy -rwxrwxrwx 1 owner group 2741 2009-09-05 ls-lr.txt /Against the Silence -rwxrwxrwx 1 owner group 9386309 2009-07-30 01-Against the Silence.wma -rwxrwxrwx 1 owner group 3974684 2009-07-30 02-Side-Stage Syndrome.wma -rwxrwxrwx 1 owner group 7539014 2009-07-30 03-The Dash on my Headstone.wma -rwxrwxrwx 1 owner group 7244819 2009-07-30 04-Teeth Like Knives.wma -rwxrwxrwx 1 owner group 9910687 2009-07-30 05-The Band Played on.wma /Collective -r-xr-xr-x 1 owner group 2767821 2009-03-05 At the Moment.wma -r-xr-xr-x 1 owner group 5259473 2009-03-05 Colt . 45.wma -r-xr-xr-x 1 owner group 2572687 2009-03-05 El Mariachi.wma -r-xr-xr-x 1 owner group 2395577 2009-03-05 Gold and Silver.wma -r-xr-xr-x 1 owner group 2269487 2009-03-05 Keep Waiting.wma -r-xr-xr-x 1 owner group 2050335 2009-03-05 Nighttown.wma -r-xr-xr-x 1 owner group 1458931 2009-03-05 Rise.wma -r-xr-xr-x 1 owner group 3140077 2009-03-05 Rivers Underneath.wma -r-xr-xr-x 1 owner group 2278489 2009-03-05 Sad Parade.wma -r-xr-xr-x 1 owner group 1909249 2009-03-05 The Hungry Wolf.wma -r-xr-xr-x 1 owner group 2467613 2009-03-05 Threshold.wma -r-xr-xr-x 1 owner group 795501 2009-03-05 Tranewreck.wma -r-xr-xr-x 1 owner group 417239 2009-03-05 Zzyzx.wma /Speakeasy -rwxrwxrwx 1 owner group 4004604 2009-03-05 01-Minuteman.wma -rwxrwxrwx 1 owner group 6309752 2009-03-05 02-Sundown Motel.wma -rwxrwxrwx 1 owner group 5504122 2009-03-05 03-Keep Waiting.wma -rwxrwxrwx 1 owner group 2766262 2009-03-05 04-You Know How It Is.wma -rwxrwxrwx 1 owner group 7495952 2009-03-05 05-Rivers Underneath.wma -rwxrwxrwx 1 owner group 6294888 2009-03-05 06-Gold and Silver.wma -rwxrwxrwx 1 owner group 8062882 2009-03-05 07-Freefall.wma -rwxrwxrwx 1 owner group 4437286 2009-03-05 08-[Untitled].wma -rwxrwxrwx 1 owner group 3355592 2009-03-05 09-St. Eriksplan.wma -rwxrwxrwx 1 owner group 4966942 2009-03-05 10-Disquiet.wma -rwxrwxrwx 1 owner group 4788302 2009-03-05 11-Fascination Street.wma -rwxrwxrwx 1 owner group 7950944 2009-03-05 12-This Love.wma
(Note/Disclaimer/etc.: It may or may not be obvious that this listing is for music from the band Stavesacre, but just to be clear and avoid any RIAA entanglements - these files aren't actually hosted on any my FTP sites; I used the script on my desktop computer to create a listing as an example for this blog.)
Customizing the Script Output
There are several customizations that you can do with this script, each of which has it's own benefits and drawbacks.
Adding Directory Sizes
It is trivial from a coding perspective to have the code calculate directory sizes since the Folder object that I use has as Size property, but it slows down the script exponentially to calculate that. That being said, if you're willing to take the performance hit, you can modify the highlighted section of the WriteEntry() function as follows:
' Test for a symbolic link. If (tmpObject.Attributes And 1024) Then tmpAttributes = "lrwxrwxrwx" tmpSize = 0 ' Test for a directory. ElseIf (tmpObject.Attributes And 16) Then tmpAttributes = "drwxrwxrwx" tmpSize = tmpObject.Size ' Otherwise - it's a file. Else tmpAttributes = "-rwxrwxrwx" tmpSize = tmpObject.Size End If
This will insert the folder size into the output, but once again it will make the script much slower and take up considerably more CPU time to compute.
Uppercase Folder Names and Lowercase File Names
Since Windows is a case-insensitive operating system, you can easily choose to display all of your folder names in all uppercase characters and your file names in all lowercase characters without causing any client confusion. This can be accomplished by adding the first highlighted section and modifying the second highlighted section of the WriteEntry() function as follows:
Dim tmpName ' Test for a directory. If (tmpObject.Attributes And 16) Then tmpName = UCase(tmpObject.Name) Else tmpName = LCase(tmpObject.Name) End If' Write the list entry to the output file. objFile.WriteLine tmpAttributes & _ " 1 owner group " & _ Right(String(15,Chr(32)) & CStr(tmpSize),15) & _ " " & FormatDate(tmpObject.DateLastModified) & _ " " & tmpName
Parting Thoughts
There are other customizations that you can easily make, such as creating a string array to sort the files and folders for each folder listing as a single list rather than listing folders first and files second as the currently script does. But that slows down the script way too much, and I prefer to see folders listed before files anyway. (Which is why I always use SET DIRCMD=/OGN for my command prompt sessions as well.)
Another easy customization would be to change the FormatDate() function to change the date format for the output file, which is why I used a function to do my date formatting. For example, you could easily use the FormatDate() function as a wrapper for VBScript's built-in FormatDateTime() function, and then use any of the vbGeneralDate, vbLongDate, vbShortDate, etc. options to specify the format. You can also use your own customized logic to return the date string, so you don't need to feel limited by my examples.
Another useful customization would be to compute the actual size for the resulting "ls-lr.txt" file and modify the output file to contain the correct file size. Currently the script in this blog adds an entry to the listing for the "ls-lr.txt" file, but that contains the temporary size of the output file as the script is running so it will seldom be accurate. (I usually run my script and update the "ls-lr.txt" file manually, but in some versions of this script I have had it ignore the "ls-lr.txt" file and remove it from the output listings.)
In closing, this script may be doing more than it might actually need to do by way of checking for symbolic links and read-only attributes, which our FTP service doesn’t actually do, but it was very easy to add that code and it runs just as fast either way.