Creating Recursive Directory Listing Files for FTP Clients

One of the changes that we made in FTP 7.0 and FTP 7.5 was to remove recursive directory listings, which are commonly retrieved by typing "ls -lR" from a command-line FTP client, which should send a command like "NLST -lR" over FTP to the server. There were several reasons why we decided to remove recursive directory listings, but the main reason was simply to reduce CPU usage on the server; recursive directory listing requests take a lot of resources to fulfill. With that in mind, both FTP 7.0 and FTP 7.5 will ignore the recursive switch on directory requests.

That being said - quite often it's pretty handy to have a full directory listing from an FTP server. From a client perspective you could probably write script to automate an FTP client to create a recursive listing, but that's a lot of work. Back in my younger days when I ran FTP sites on Unix servers, I would always create two types of list files on my FTP servers for FTP clients to retrieve:

  • "ls-lr.txt" - I would create only one file of this type for my entire FTP server, which would go in the root of my FTP site and it would contain a full recursive listing of all files in my FTP site.
  • "00index.txt" - I would create one file of this type in each folder of my FTP site, and each index file would contain a listing of files and their descriptions for that folder.

Of course, anyone that's been around the Internet since the days before we had HTTP and the world-wide-web should know that I didn't come up with this idea on my own - I learned it from other FTP site administrators. (And anyone who remembers those days should also recognize those two files with a strange sense of nostalgia. 00index.txt files of course led to index.htm files when WWW sites came along later, but that's another story.)

In any event, as I continued to host FTP sites over the years I have written various scripts to create recursive directory listings, and I thought that one of my scripts might make a good blog post. With that in mind, here is a Windows Script Host file that I created, which I named "ls-lr.vbs", and this script will create a recursive directory listing for an FTP site. I choose the Unix directory listing style for this script since that's the format that I have used for years and the broader number of FTP clients and users should recognize it.

Option Explicit
On Error Resume Next

' Declare all variables.
Dim objArguments
Dim strBaseFolder
Dim objFSO
Dim objFile
Dim objFolder
Dim objSubFolder
Dim objSubFile
Dim lngFolderCount
Dim lngBaseCount
Set objArguments = WScript.Arguments

' Determine the number of command-line arguments.
Select Case objArguments.Count
  Case 0:
    strBaseFolder = WScript.ScriptFullName
    strBaseFolder = Left(strBaseFolder,InStrRev(strBaseFolder,"\"))
  Case 1:
    strBaseFolder = objArguments(0)
  Case Else:
    MsgBox "This script takes a single argument for the" & vbCrLf & _
      "starting directory, or specify no arguments" & vbCrLf & _
      "to use the current directory.", vbInformation
End Select

' Create a file system object.
Set objFSO = WScript.CreateObject("Scripting.FileSystemObject")

' Test if the base folder exists.
If Right(strBaseFolder,1) <> "\" Then strBaseFolder = strBaseFolder & "\"
If objFSO.FolderExists(strBaseFolder) = False Then
    MsgBox "The specified folder does not exist.", vbCritical
End If

' Open the output file for the directory listing.
Set objFile = objFSO.CreateTextFile(strBaseFolder & "ls-lr.txt")
' Define the initial values for the folder counters.
lngFolderCount = 1
lngBaseCount = 0
' Dimension an array to hold the folder names.
ReDim strFolders(1)
' Store the root folder in the array.
strFolders(lngFolderCount) = strBaseFolder
' Loop while we still have folders to process.
While lngFolderCount <> lngBaseCount
  ' Set up a folder object to a base folder.
  Set objFolder = objFSO.GetFolder(strFolders(lngBaseCount+1))
  ' Output the folder name to the listing file.
  objFile.WriteLine vbCrLf & _
    Replace(Mid(strFolders(lngBaseCount+1),Len(strBaseFolder)),"\","/") & _
  ' Loop through the collection of subfolders for the base folder.
  For Each objSubFolder In objFolder.SubFolders
    ' Increment the folder count.
    lngFolderCount = lngFolderCount + 1
    ' Increase the array size
    ReDim Preserve strFolders(lngFolderCount)
    ' Store the folder name in the array.
    strFolders(lngFolderCount) = objSubFolder.Path
    ' Output the folder to the listing file.
    Call WriteEntry(objSubFolder)
  ' Loop through the collection of subfolders for the base folder.
  For Each objSubFile In objFolder.Files
    ' Output the file to the listing file.
    Call WriteEntry(objSubFile)
  ' Increment the base folder counter.
  lngBaseCount = lngBaseCount + 1

Sub WriteEntry(tmpObject)
  Dim tmpAttributes
  Dim tmpSize
  ' Test for a symbolic link.
  If (tmpObject.Attributes And 1024) Then
    tmpAttributes = "lrwxrwxrwx"
    tmpSize = 0
  ' Test for a directory.
  ElseIf (tmpObject.Attributes And 16) Then
    tmpAttributes = "drwxrwxrwx"
    tmpSize = 0
  ' Otherwise - it's a file.
    tmpAttributes = "-rwxrwxrwx"
    tmpSize = tmpObject.Size
  End If
  ' Test for a read-only object.
  If (tmpObject.Attributes And 1) Then
    tmpAttributes = Replace(tmpAttributes,"w","-")
  End If
  ' Write the list entry to the output file.
  objFile.WriteLine tmpAttributes & _
    "   1 owner    group " & _
    Right(String(15,Chr(32)) & CStr(tmpSize),15) & _
    " " & FormatDate(tmpObject.DateLastModified) & _
    " " & tmpObject.Name
End Sub

Function FormatDate(tmpDate)
  FormatDate = CStr(Year(tmpDate)) & _
    "-" & Right("00" & CStr(Month(tmpDate)),2) & _
    "-" & Right("00" & CStr(Day(tmpDate)),2)
End Function

To use the script, copy the code into Windows Notepad and save it to your computer as "ls-lr.vbs." If you double-click the script it will use the current folder to create a recursive folder listing, and if you run this script from a command-line it can take a single argument of a folder path, or you can pass no arguments to the script in order to use the current folder. In either case it will create a file named "ls-lr.txt" in the root of the destination folder that contains the recursive directory listing in Unix format.

For example, the following listing was created from a folder in my music collection on my desktop computer:


drwxrwxrwx   1 owner    group               0 2009-07-30 Against the Silence
dr-xr-xr-x   1 owner    group               0 2009-07-30 Collective
drwxrwxrwx   1 owner    group               0 2009-07-30 Speakeasy
-rwxrwxrwx   1 owner    group            2741 2009-09-05 ls-lr.txt

/Against the Silence

-rwxrwxrwx   1 owner    group         9386309 2009-07-30 01-Against the Silence.wma
-rwxrwxrwx   1 owner    group         3974684 2009-07-30 02-Side-Stage Syndrome.wma
-rwxrwxrwx   1 owner    group         7539014 2009-07-30 03-The Dash on my Headstone.wma
-rwxrwxrwx   1 owner    group         7244819 2009-07-30 04-Teeth Like Knives.wma
-rwxrwxrwx   1 owner    group         9910687 2009-07-30 05-The Band Played on.wma


-r-xr-xr-x   1 owner    group         2767821 2009-03-05 At the Moment.wma
-r-xr-xr-x   1 owner    group         5259473 2009-03-05 Colt . 45.wma
-r-xr-xr-x   1 owner    group         2572687 2009-03-05 El Mariachi.wma
-r-xr-xr-x   1 owner    group         2395577 2009-03-05 Gold and Silver.wma
-r-xr-xr-x   1 owner    group         2269487 2009-03-05 Keep Waiting.wma
-r-xr-xr-x   1 owner    group         2050335 2009-03-05 Nighttown.wma
-r-xr-xr-x   1 owner    group         1458931 2009-03-05 Rise.wma
-r-xr-xr-x   1 owner    group         3140077 2009-03-05 Rivers Underneath.wma
-r-xr-xr-x   1 owner    group         2278489 2009-03-05 Sad Parade.wma
-r-xr-xr-x   1 owner    group         1909249 2009-03-05 The Hungry Wolf.wma
-r-xr-xr-x   1 owner    group         2467613 2009-03-05 Threshold.wma
-r-xr-xr-x   1 owner    group          795501 2009-03-05 Tranewreck.wma
-r-xr-xr-x   1 owner    group          417239 2009-03-05 Zzyzx.wma


-rwxrwxrwx   1 owner    group         4004604 2009-03-05 01-Minuteman.wma
-rwxrwxrwx   1 owner    group         6309752 2009-03-05 02-Sundown Motel.wma
-rwxrwxrwx   1 owner    group         5504122 2009-03-05 03-Keep Waiting.wma
-rwxrwxrwx   1 owner    group         2766262 2009-03-05 04-You Know How It Is.wma
-rwxrwxrwx   1 owner    group         7495952 2009-03-05 05-Rivers Underneath.wma
-rwxrwxrwx   1 owner    group         6294888 2009-03-05 06-Gold and Silver.wma
-rwxrwxrwx   1 owner    group         8062882 2009-03-05 07-Freefall.wma
-rwxrwxrwx   1 owner    group         4437286 2009-03-05 08-[Untitled].wma
-rwxrwxrwx   1 owner    group         3355592 2009-03-05 09-St. Eriksplan.wma
-rwxrwxrwx   1 owner    group         4966942 2009-03-05 10-Disquiet.wma
-rwxrwxrwx   1 owner    group         4788302 2009-03-05 11-Fascination Street.wma
-rwxrwxrwx   1 owner    group         7950944 2009-03-05 12-This Love.wma

(Note/Disclaimer/etc.: It may or may not be obvious that this listing is for music from the band Stavesacre, but just to be clear and avoid any RIAA entanglements - these files aren't actually hosted on any my FTP sites; I used the script on my desktop computer to create a listing as an example for this blog.)

Customizing the Script Output

There are several customizations that you can do with this script, each of which has it's own benefits and drawbacks.

Adding Directory Sizes

It is trivial from a coding perspective to have the code calculate directory sizes since the Folder object that I use has as Size property, but it slows down the script exponentially to calculate that. That being said, if you're willing to take the performance hit, you can modify the highlighted section of the WriteEntry() function as follows:

  ' Test for a symbolic link.
  If (tmpObject.Attributes And 1024) Then
    tmpAttributes = "lrwxrwxrwx"
    tmpSize = 0
  ' Test for a directory.
  ElseIf (tmpObject.Attributes And 16) Then
    tmpAttributes = "drwxrwxrwx"
    tmpSize = tmpObject.Size
  ' Otherwise - it's a file.
    tmpAttributes = "-rwxrwxrwx"
    tmpSize = tmpObject.Size
  End If

This will insert the folder size into the output, but once again it will make the script much slower and take up considerably more CPU time to compute.

Uppercase Folder Names  and Lowercase File Names

Since Windows is a case-insensitive operating system, you can easily choose to display all of your folder names in all uppercase characters and your file names in all lowercase characters without causing any client confusion. This can be accomplished by adding the first highlighted section and modifying the second highlighted section of the WriteEntry() function as follows:

  Dim tmpName
  ' Test for a directory.
  If (tmpObject.Attributes And 16) Then
    tmpName = UCase(tmpObject.Name)
    tmpName = LCase(tmpObject.Name)
  End If
  ' Write the list entry to the output file.
  objFile.WriteLine tmpAttributes & _
    "   1 owner    group " & _
    Right(String(15,Chr(32)) & CStr(tmpSize),15) & _
    " " & FormatDate(tmpObject.DateLastModified) & _
    " " & tmpName

Parting Thoughts

There are other customizations that you can easily make, such as creating a string array to sort the files and folders for each folder listing as a single list rather than listing folders first and files second as the currently script does. But that slows down the script way too much, and I prefer to see folders listed before files anyway. (Which is why I always use SET DIRCMD=/OGN for my command prompt sessions as well.)

Another easy customization would be to change the FormatDate() function to change the date format for the output file, which is why I used a function to do my date formatting. For example, you could easily use the FormatDate() function as a wrapper for VBScript's built-in FormatDateTime() function, and then use any of the vbGeneralDate, vbLongDate, vbShortDate, etc. options to specify the format. You can also use your own customized logic to return the date string, so you don't need to feel limited by my examples.

Another useful customization would be to compute the actual size for the resulting "ls-lr.txt" file and modify the output file to contain the correct file size. Currently the script in this blog adds an entry to the listing for the "ls-lr.txt" file, but that contains the temporary size of the output file as the script is running so it will seldom be accurate. (I usually run my script and update the "ls-lr.txt" file manually, but in some versions of this script I have had it ignore the "ls-lr.txt" file and remove it from the output listings.)

In closing, this script may be doing more than it might actually need to do by way of checking for symbolic links and read-only attributes, which our FTP service doesn’t actually do, but it was very easy to add that code and it runs just as fast either way.

No Comments