Chris Fulstow

There are a few different ways in .NET to count the number of blank lines in a text file, but which is the most efficient?

To test some different approaches, I’ve written this function to create a huge text file with some empty lines.  To verify that my counting functions give correct results, I need this function to create a file with a specific number of blank lines.

private static void BuildFileWithBlankLines(string path, uint blanks)
{
    var random = new Random();
    var counter = 0;

    using (var sw = File.CreateText(path))
    {
        while (counter < blanks)
        {
            var isBlank = (random.Next(100) == 0);  // ~1% of lines blank
            sw.WriteLine(isBlank ? "" : "NOT BLANK");
            if (isBlank) { counter++; }
        }
    }
}

For my test, I’ve used 250,000 empty lines, which outputs a file just over 250 MB.  This should be big enough to expose the differences between counting methods.

1. Regular expression matching in memory

var content = File.ReadAllText(path);
var re = new Regex(@"^\r?\n", RegexOptions.Multiline);
var count = re.Matches(content).Count;

This technique loads the whole text file into a string variable using File.ReadAllText, which is really just a wrapper for StreamReader.ReadToEnd. It loads the character bytes from a file stream into a StringBuilder.  The load is relatively quick, taking about 8% of the total processing time.  Once the file is loaded into memory, a regular expression finds and counts all the blank lines. This is, not surprisingly, glacially slow.  Total duration was 16,531 ms.

2. Counting line by line through a string array

var count = File.ReadAllLines(path).Count(s => s.Length == 0);

I like this approach because it’s simple and readable.  The first step is similar to the above, but this time uses a StreamReader (via File.ReadAllLines) to load the file into a string array, rather than into a single string. IEnumerable.Count then loops through the array and tries to match each line against a predicate that checks for empty strings.  This takes less than half the time of the regex approach, only 7,006 ms.

3. Streaming through the file

uint count = 0;
using (var sr = File.OpenText(path))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {
        if (line.Length == 0) { count++; }
    }
}

Unlike the first two, this last approach doesn’t load the entire file into memory.  It uses File.OpenText to create a new StreamReader, then reads each line from the FileStream counting blank lines as it goes along.  This is blazing fast compared to the other two, taking just 2,108 ms.

Conclusion

There’s almost an order of magnitude difference between the best and worst methods.  So if you’re working with large files, try not to load them into memory first, but stream directly from disk instead.

This applies especially to parsing large data files before loading them into a database.  In some cases, it’s not just slow to load into memory first, but impossible.

For example, a 500 GB XML file wouldn’t load easily into an XDocument, but could be streamed with an XmlReader and parsed with XNode.ReadFrom. Similarly, a huge CSV file couldn’t be loaded easily into a string then String.Split all in one go.  Instead, you’d probably parse it line by line with a StreamReader, or use a streaming CsvReader library.

The DebuggerDisplay attribute lets you change how an object is displayed in the Visual Studio debugger. Take these two simple classes representing a Company and an Employee.

public class Company
{
    public string Name { get; set; }
    public string StockSymbol { get; set; }
    public IEnumerable<Employee> Employees { get; set; }
    public Company(string name) { Name = name; }
}

public class Employee
{
    public string Name { get; set; }
    public Employee(string name) { Name = name; }
}

Create and initialise a new Company object with some data.

var company = new Company("Microsoft")
{
    StockSymbol = "MSFT",
    Employees = new List<Employee>
    {
        new Employee("Anders Hejlsberg"),
        new Employee("Steve Ballmer"),
        new Employee("Scott Guthrie")
    }
};

Now, if you break into the code and start a debug session, you can hover over the company object to reveal its debugger DataTip.  By expanding the nodes you can drill down into the object’s data.

The default information shown in the DataTip, although useful, can be improved by decorating classes with the DebuggerDisplay attribute.  For companies, it might be useful to to display their Name and StockSymbol instead of the object type {Company}.

[DebuggerDisplay("{Name} - {StockSymbol}")]
public class Company

The same goes for the Employee object.  We can show the employee name instead of the type {Employee}.

[DebuggerDisplay("{Name}")]
public class Employee

The result is a more useful and descriptive view of your Company objects.

For more info about customising Visual Studio’s debugger DataTips, see this article from MSDN Magazine:

• DataTips, Visualizers and Viewers Make Debugging .NET Code a Breeze