To test some different approaches, I’ve written this function to create a huge text file with some empty lines.  To verify that my counting functions give correct results, I need this function to create a file with a specific number of blank lines.

private static void BuildFileWithBlankLines(string path, uint blanks)
{
    var random = new Random();
    var counter = 0;

    using (var sw = File.CreateText(path))
    {
        while (counter < blanks)
        {
            var isBlank = (random.Next(100) == 0);  // ~1% of lines blank
            sw.WriteLine(isBlank ? "" : "NOT BLANK");
            if (isBlank) { counter++; }
        }
    }
}

For my test, I’ve used 250,000 empty lines, which outputs a file just over 250 MB.  This should be big enough to expose the differences between counting methods.

1. Regular expression matching in memory

var content = File.ReadAllText(path);
var re = new Regex(@"^\r?\n", RegexOptions.Multiline);
var count = re.Matches(content).Count;

This technique loads the whole text file into a string variable using File.ReadAllText, which is really just a wrapper for StreamReader.ReadToEnd. It loads the character bytes from a file stream into a StringBuilder.  The load is relatively quick, taking about 8% of the total processing time.  Once the file is loaded into memory, a regular expression finds and counts all the blank lines. This is, not surprisingly, glacially slow.  Total duration was 16,531 ms.

2. Counting line by line through a string array

var count = File.ReadAllLines(path).Count(s => s.Length == 0);

I like this approach because it’s simple and readable.  The first step is similar to the above, but this time uses a StreamReader (via File.ReadAllLines) to load the file into a string array, rather than into a single string. IEnumerable.Count then loops through the array and tries to match each line against a predicate that checks for empty strings.  This takes less than half the time of the regex approach, only 7,006 ms.

3. Streaming through the file

uint count = 0;
using (var sr = File.OpenText(path))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {
        if (line.Length == 0) { count++; }
    }
}

Unlike the first two, this last approach doesn’t load the entire file into memory.  It uses File.OpenText to create a new StreamReader, then reads each line from the FileStream counting blank lines as it goes along.  This is blazing fast compared to the other two, taking just 2,108 ms.

Conclusion

There’s almost an order of magnitude difference between the best and worst methods.  So if you’re working with large files, try not to load them into memory first, but stream directly from disk instead.

This applies especially to parsing large data files before loading them into a database.  In some cases, it’s not just slow to load into memory first, but impossible.

For example, a 500 GB XML file wouldn’t load easily into an XDocument, but could be streamed with an XmlReader and parsed with XNode.ReadFrom. Similarly, a huge CSV file couldn’t be loaded easily into a string then String.Split all in one go.  Instead, you’d probably parse it line by line with a StreamReader, or use a streaming CsvReader library.

public class Company
{
    public string Name { get; set; }
    public string StockSymbol { get; set; }
    public IEnumerable<Employee> Employees { get; set; }
    public Company(string name) { Name = name; }
}

public class Employee
{
    public string Name { get; set; }
    public Employee(string name) { Name = name; }
}

Create and initialise a new Company object with some data.

var company = new Company("Microsoft")
{
    StockSymbol = "MSFT",
    Employees = new List<Employee>
    {
        new Employee("Anders Hejlsberg"),
        new Employee("Steve Ballmer"),
        new Employee("Scott Guthrie")
    }
};

Now, if you break into the code and start a debug session, you can hover over the company object to reveal its debugger DataTip.  By expanding the nodes you can drill down into the object’s data.

The default information shown in the DataTip, although useful, can be improved by decorating classes with the DebuggerDisplay attribute.  For companies, it might be useful to to display their Name and StockSymbol instead of the object type {Company}.

[DebuggerDisplay("{Name} - {StockSymbol}")]
public class Company

The same goes for the Employee object.  We can show the employee name instead of the type {Employee}.

[DebuggerDisplay("{Name}")]
public class Employee

The result is a more useful and descriptive view of your Company objects.

For more info about customising Visual Studio’s debugger DataTips, see this article from MSDN Magazine:

• DataTips, Visualizers and Viewers Make Debugging .NET Code a Breeze

For example, a web page that uses a theme setting to control look-and-feel.  If the user has selected their own theme then use that; otherwise use the site’s default theme; or, if that’s not defined, use the global theme; or finally if no global theme is defined then fall back to a system default.

The long, readable version

Using good, old fashioned, conditional logic.

string theme;
if (userTheme != null)
{
    theme = userTheme;
}
else if (siteTheme != null)
{
    theme = siteTheme;
}
else if (globalTheme != null)
{
    theme = globalTheme;
}
else
{
    theme = defaultTheme;
}

The shorter, slightly befuddling version

By nesting the ternary conditional operator in a most disagreeable way.

string theme =
    userTheme != null ? userTheme
    : siteTheme != null ? siteTheme
    : globalTheme != null ? globalTheme
    : defaultTheme;

The deluxe C# 3.0 version

With the overkill of an implicitly-typed array, an IEnumerable extension method, and a lambda predicate.

string theme = new [] {
    userTheme, siteTheme, globalTheme, defaultTheme
}.First(t => t != null);

The elegant coalescence version

With C# 2.0’s null-coalescing operator, doing what it does best.

string theme = userTheme ?? siteTheme ?? globalTheme ?? defaultTheme;

Got any more? Let me know in the comments.

O’Reilly

Web Design for Developers – A Programmer’s Guide to Design Tools and Techniques

Web Design for Developers will show you how to make your web-based application look professionally designed. We’ll help you learn how to pick the right colors and fonts, avoid costly interface and accessibility mistakes–your application will really come alive. We’ll also walk you through some common Photoshop and CSS techniques and work through a web site redesign, taking a new design from concept all the way to implementation.

Effective UI – The Art of Building Great User Experience in Software

Effective UI provides a complete roadmap to building groundbreaking software centered on user experience (UX) quality, how to get management support, employing product management strategies proven to deliver greater success, and how to manage the design, engineering, staffing, and business considerations that must be centered on the user’s needs and working effectively in tandem all throughout the project. Effective UI is a guide for business and product, software professionals, and anyone else struggling to advance the cause of better UX and working to deliver on the promise of higher quality software.

Search Patterns
Search is among the most disruptive innovations of our time. It influences what we buy and where we go. It shapes how we learn and what we believe. This provocative and inspiring book explores design patterns that apply across the categories of web, ecommerce, enterprise, desktop, mobile, social, and realtime search and discovery. Using colorful illustrations and examples, the authors bring modern information retrieval to life, covering such diverse topics as relevance ranking, faceted navigation, multi-touch, and augmented reality. Search Patterns challenges us to invent the future of discovery while serving as a practical guide to help us make search better today.

97 Things Every Programmer Should Know – Collective Wisdom from the Experts
Get 97 short and extremely useful tips from some of the most experienced and respected practitioners in the industry, including Uncle Bob Martin, Scott Meyers, Dan North, Linda Rising, Udi Dahan, Neal Ford, and many more. They encourage you to stretch yourself by learning new languages, looking at problems in new ways, following specific practices, taking responsibility for your work, and becoming as good at the entire craft of programming as you possibly can.

High Performance JavaScript
Do you want to fix performance bottlenecks in your JavaScript code to help your website function better? With this book, you’ll examine the typical code problems that cause JavaScript to run slow. High Performance JavaScript teaches you how to identify inefficient structures and patterns, and replace them with better-performing alternatives.

Programming Microsoft® ASP.NET MVC
Author Dino Esposito leads you through the features, principles, and pillars of the ASP.NET MVC framework, demonstrating how and when to use this model to gain full control of HTML, simplify testing, and design better Web sites and experiences.

APress

Ultra-Fast ASP.NET: Building Ultra-Fast and Ultra-Scalable Websites Using ASP.NET and SQL Server
Ultra-Fast ASP.NET presents a practical approach to building fast and scalable web sites using ASP.NET and SQL Server. In addition to a wealth of tips, tricks and secrets, you’ll find advice and code examples for all tiers of your application, including the client, caching, IIS 7, ASP.NET, threads, session state, SQL Server, Analysis Services, infrastructure and operations. By applying the ultra-fast approach to your projects, you’ll squeeze every last ounce of performance out of your code and infrastructure—giving your site unrivaled speed.

Introducing .NET 4.0: with Visual Studio 2010
Microsoft is introducing a large number of changes to the way that the .NET Framework operates. Familiar technologies are being altered, best practices replaced, and developer methodologies adjusted. Many developers find it hard to keep up with the pace of change across .NET’s ever-widening array of technologies. You may know what’s happening in C#, but how about the Azure cloud? How is that going to affect your work? What are the limitations of the new pLINQ syntax? What you need is a roadmap. A guide to help you see the innovations that matter and to give you a head start on the opportunities available in the new framework.

Pro HTML5 Programming: Powerful APIs for Richer Internet Application Development
HTML5 is here, and with it, rich Internet applications (RIAs) take on a power, ease, scalability, and responsiveness that neither Ajax nor Comet could ever achieve. With this book, developers will learn how to use the latest cutting-edge web technology—available in the most recent versions of modern browsers—to build real-time web applications with unparalleled functionality, speed, and responsiveness.

Pro ASP.NET MVC V2 Framework
Author Steven Sanderson has seen the ASP.NET MVC Framework mature from the start, so his experience, combined with comprehensive coverage of all the new features, including those in the official MVC development toolkit, offers the clearest understanding of how this exciting new framework can improve your coding efficiency. With this book, you’ll gain invaluable up-to-date knowledge of security, deployment, and interoperability challenges.

iPhone and Mac OS X Development for Windows Programmers: C#, C++ and C Edition
When coming to the iPhone and Mac OS X from Windows, you need to learn a new set of coding techniques and tools. Taken blindly, this can come as a shock—a shock that this book will help you avoid. It does this by teaching you to use your existing Windows programming knowledge to ease you into programming for Apple technologies.

Manning

jQuery in Action, Second Edition
A really good web development framework anticipates your needs. jQuery does more—it practically reads your mind. Developers fall in love with this JavaScript library the moment they see 20 lines of code reduced to three. jQuery is concise and readable. Its unique “chaining” model lets you perform multiple operations on a page element in succession. And with version 1.4, there’s even more to love about jQuery, including new effects and events, usability improvements, and more testing options.

Brownfield Application Development in .NET

Brownfield Application Development in .Net shows you how to approach legacy applications with the state-of-the-art concepts, patterns, and tools you’ve learned to apply to new projects. Using an existing application as an example, this book guides you in applying the techniques and best practices you need to make it more maintainable and receptive to change.

Secrets of the JavaScript Ninja
In Secrets of the JavaScript Ninja, JavaScript expert John Resig reveals the inside know-how of the elite JavaScript programmers. Written to be accessible to JavaScript developers with intermediate-level skills, this book will give you the knowledge you need to create a cross-browser JavaScript library from the ground up.

Dependency Injection in .NET
Dependency Injection in .NET is a comprehensive guide than introduces DI and provides an in-depth look at applying DI practices to .NET apps. In it, you will also learn to integrate DI together with such technologies as Windows Communication Foundation, ASP.NET MVC, Windows Presentation Foundation and other core .NET components.

C# in Depth, Second Edition

C# in Depth, Second Edition concentrates on the high-value features that make C# such a powerful and flexible development tool. Rather than re-hashing the core of C# that’s essentially unchanged since it hit the scene nearly a decade ago, this book brings you up to speed with the features and practices that have changed with C# from version 2.0 onwards.

MS Press

Microsoft ASP.NET Internals
A comprehensive reference outlining the underpinnings of Microsoft’s Web application environment
Learn how to fully exploit the critical components and rich capabilities of the ASP.NET framework, without scouring blogs for information or enduring hours of trial and error. This comprehensive book offers developers the insights they need—including established architecture patterns—to create better, stronger, faster Web sites and Web-based applications Web services. The book offers readers the guidance they need to take advantage of ASP.NET right away, and outlines the evolution and future of the technology.

Addison Wesley

LINQ to Objects Using C# 4.0: Using and Extending LINQ to Objects and Parallel LINQ (PLINQ)
Using LINQ to Objects, .NET developers can write queries over object collections with the same deep functionality that was once available only with SQL and relational databases. Now, for the first time, developers have a comprehensive and authoritative guide to applying LINQ to Objects in real-world software. Microsoft MVP Troy Magennis introduces state-of-the-art techniques for working with objects more elegantly and efficiently–and writing code that is exceptionally powerful, robust, and flexible.

Updates from comments:

.NET Performance Testing and Optimization (Part 1)

Paul Glavich and Chris Farrell’s upcoming book on .NET performance testing and optimization will be out in a few weeks, and part one is already as a free e-book.  They explain why performance testing is a good idea, and walk you through everything you need to know about setting up a test environment. An essential handbook for programmers looking to set up a .NET testing environment.

As it’s getting progressively harder to keep up with .NET’s continutally expanding scope, this poster is a handy reminder of what’s included.  It shows which classes were added in .NET 3.0, and which in .NET 3.5.

There’s a broad cross-section across all areas of the .NET Framework:

• ASP.NET
• WinForms and WPF
• Communications and Workflow
• Data, XML and LINQ
• Fundamentals

Printing

The hi-res version is Microsoft XPS format, so if you’re not using Vista or Office 2007 then you might want the Microsoft XPS Viewer. Also, for ‘easy printing’, there’s a 16 page 4×4 version, but remember: ’some assembly is required if you choose this print method’, so remember to ask an adult for help with the scissors.

My local print shop printed and laminated the PDF version onto A1, which is easily hi-res enough and looks great.

Other .NET Reference Posters

There are a few other reference posters on MSDN, in particular I like the Silverlight Developer Reference, and keyboard shortcuts for Visual Studio 2008, available for both C# and Visual Basic.

(Thanks to Chris Bowen for the tip off.)

The last upgrade, SQL Server 2000 to 2005, was a huge leap forward for developers and added significant advances like CLR integration, SQL Server Management Objects (SMO), Integration Services (SSIS) and a native XML data type; as well as T-SQL enhancements like Common Table Expressions (CTEs), structured error handling with try/catch, pivot, apply, top(n) and row_number.

I was surprised to see how much new stuff is packed into the latest SQL Server 2008 CTP release, even, finally, IntelliSense for Management Studio, which was much anticipated but conspicuously absent from SQL 2005:

Also notice the new collapsible code regions, just like you get in Visual Studio. Editing T-SQL has never been such fun! Although, you have to feel a bit sorry for RedGate, whose SQL Prompt plug-in has been filling the auto-completion gap for the last few years.

My main study guide was the official Microsoft Press book, MCPD Self-Paced Training Kit (Exam 70-547). It covered the syllabus thoroughly enough to get me through the exam, but like most similar books wasn’t much fun to read. It’s over 700 pages, and after the first few hundred the content starts to get fairly monotonous. It’s also difficult to skim because you might miss something essential for the exam. If you’re an experienced developer then you might not learn much, but if you’re new to coding then you should find plenty of useful information about broader areas of software development, such as architecture, data modelling and unit testing.

A lot of the testing chapters read like an advert for Visual Studio Team System (VSTS) and aggressively promote its unit testing and web load testing capabilities. There’s nothing wrong with that, it’s a Microsoft exam after all, but you will need Visual Studio Team Test if you want to work through the practical exercises (an evaluation version is available). The unit testing in VSTS is impressive, but doesn’t seem to offer much over the freely available NUnit. The web load testing is very good, and a good alternative to JMeter.

My best advice for the book is to read it in short frequent bursts. Try to read a little bit every day and you’ll easily work through it, so don’t be intimidated. If you find yourself slacking, just register for the exam and you’ll soon become motivated. And remember to use the 15% exam discount voucher that comes with the book.

My other study resource, which I also used for the MCTS exams, was the practice test from MeasureUp. The questions are very relevant to the actual exam, and helped highlight areas to focus my study. The most useful feature is that every answer has an explanation, so if you get one wrong you can see why and hopefully fill a gap in your knowledge.

The exam isn’t easy, but many questions appeal to common sense as well as technical expertise, and there aren’t any trick questions. There’s more than enough time to work through the 40 multiple-choice questions, so be careful to read each one closely and don’t panic. Some questions ask you to pick more than one answer so follow the instructions carefully. The passing score is 700 out of 1000.

According to Microsoft’s statistics there are only 2,147 MCPD web developers worldwide, so if you want to improve your skills, stand out in the job market, or negotiate a promotion then certification is a great career move.

My next goal is MCTS certification for SQL Server and exam 70–431. Then it shouldn’t be too long before the next generation of .NET 3.5 MCTS are available.

Over time, new features have been introduced to the site, and old ones removed. Consequently the SQL Server database now contains many redundant tables that aren’t used. So, before cascading out the current schema to the other countries, it’s time for a clean up.

I managed to identify about 60 tables that aren’t used by the application and can safely can be dropped or archived. However, I’m now left with hundreds of stored procedures (SPs) and user-defined functions (UDFs) that were associated with these tables, which can also be removed.

The problem was how to find these orphaned objects. My first approach was a small .NET console application which uses SQL Server Management Objects (SMO). It loops through all SPs and UDFs and finds any that have no dependencies.

public List<string> FindOrphans()
{
   Server server = new Server(".");
   Database db = server.Databases["MyDatabase"];
   List<string> orphans = new List<string>();

   // get list of SPs
   UrnCollection urns = new UrnCollection();
   foreach (StoredProcedure sp in db.StoredProcedures)
   {
      // exclude these objects
      if (sp.IsSystemObject) continue;
      if (sp.Name.StartsWith("aspnet_")) continue;
      urns.Add(sp.Urn);
   }

   // get dependencies
   DependencyWalker dw = new DependencyWalker(server);
   DependencyTree tree = dw.DiscoverDependencies(urns, true);

   // find all objects without any dependencies
   DependencyTreeNode node = tree.FirstChild;
   do {
      if (!node.HasChildNodes)
      {
         string name = new Urn(node.Urn).GetAttribute("Name");
         orphans.Add(name);
      }
      node = node.NextSibling;
   } while (node != null);

   return orphans;
}

This works fine, and helped satisfy my current obsession with SMO. But it’s a bit awkward, and not easily portable or modifiable, to have this pure database operation wrapped up in an executable. So I looked into doing the same thing with just a TSQL query.

-- Find all SPs and UDFs have no dependencies
select
    object_name(obj.[object_id]) as [orphaned_object_name],
    obj.type_desc as [object_type],
    'DROP ' +
    case obj.type_desc
        when 'SQL_STORED_PROCEDURE' then 'PROCEDURE'
        else 'FUNCTION'
    end
    + ' [' + object_name(obj.[object_id]) + ']'
from
    sys.objects obj
    left join (select distinct [object_id] from sys.sql_dependencies) dep
        on obj.object_id = dep.object_id
where
    type_desc in
        ('SQL_STORED_PROCEDURE','SQL_SCALAR_FUNCTION','SQL_TABLE_VALUED_FUNCTION')
    and object_name(obj.[object_id]) not like 'aspnet_%'
    and dep.object_id is null
order by
    obj.type_desc, object_name(obj.[object_id])

The query works by checking for dependencies in the catalog view sys.sql_dependencies. This, I think, is a neater solution. I also included an auto-generated column that writes the SQL drop the SP or UDF, which I copied and executed.

Now, if only I could find a quick way to check for dependencies between my application’s data access layer and the database…

Scott Hanselman’s article Reading to Be a Better Developer got me wondering why we don’t do this more with .NET code, and the problem for me seems to be finding good code examples. Scott recommends looking at the Coding4Fun Developer Kit, but I wanted something more specific to web development.

So here are a few places I found ASP.NET source code that’s worth studying and learning from.

Microsoft Enterprise Library

A great place to start is the application blocks in Microsoft’s Enterprise Library. These are application service components designed to follow Microsoft best practices and include modules for caching, cryptography, data access, exception handling, logging, policy injection, security and validation.

Website Starter Kits

Another good place to look is the ASP.NET Starter Kit Websites, a collection of working ASP.NET demos that can be examined or built on. They cover DotNetNuke, e-commerce with PayPal, blogging, project time management, media library and plenty more.

Codeplex

Lastly Codeplex, Microsoft’s open source project hosting site. There’s so much goodness here it’s hard know where to start, so try browsing the most popular or active projects to start. Here are the top ten that caught my eye:

• BlogEngine.NET
  Full featured blog engine targeted at .NET developers. It is light weight and very simple to modify and extend.
• Umbraco
  Simple, flexible and friendly ASP.NET CMS
• DinnerNow
  Sample marketplace application designed to demonstrate how you can develop a connected application using IIS7, ASP.NET Ajax Extensions, Linq, WCF, WF, WPF, Powershell, and the .NET Compact Framework.
• Community Kit for SharePoint
  Set of best practices, templates, Web Parts, tools, and source code for creating a community website based on SharePoint.
  
• Facebook Developer Toolkit and Facebook.NET
  .NET wrappers and libraries for the Facebook API.
• DbEntry.Net
  Lightweight, high performance Object Relational Mapping (ORM) database access compnent for .NET 2.0.
• PublicDomain
  .NET packages for time zone support, logging, dynamic code evaluation, GAC API, unzipping, RSS, Atom, OPML, screen scraping, and utilities for strings, arrays and cryptography.
• ASP.NET RSS Toolkit
  Gives ASP.NET applications the ability to consume and publish to RSS feeds.
• NGenerics
  Class library providing generic data structures and algorithms not implemented in the standard .NET framework
• Html Agility Pack
  Agile HTML parser that builds a read/write DOM and supports plain XPath or XSLT. The parser is very tolerant with “real world” malformed HTML. The object model is very similar to System.Xml, but for HTML documents.

If you know any other places to find good quality .NET source code then please leave a comment.

// TODO: fix catastrophic memory leak

These notes will automatically appear in the Visual Studio Task List, which you can open with the shortcut Ctrl+W, T or by selecting View – Task List from the main menu bar.

Visual Studio also supports two other comment tokens, HACK and UNDONE. You can even add your own custom comment tokens in:

Options – Environment – Task List.