Introduction

In MSBuild 4 we introduced several performance improvements, particular for large interdependent builds.  By and large they are automatic and you receive their benefit without making any changes to the way your build process in authored.  However, there are still some cases where we are unable to make the best decision.  One such case is when there is a particular external tool which is invoked as part of the build but which takes a significant amount of time.  An example of such a tool would be cl.exe, the C++ compiler.  This article discusses how to use the new yield mechanism for external tools to improve the performance of your builds.

Tool Tasks

There are a few ways MSBuild can be made to execute external, command-line tools:

1. Write a task which derives from ToolTask.
2. Use the Exec task to call your command.
3. Use the XamlTaskFactory.

All of these methods ultimately use the ToolTask class in Microsoft.Build.Utilities.v4.0.dll to handle executing a command-line task and deal with the output in the MSBuild way.  Like all tasks, however, they block any other work from happening in MSBuild while they are executing.  In cases where the task is very short, such as touching a log file or copying a file from one place to another this is perfectly acceptable.  But in the original example of invoking the C++ compiler, the amount of time MSBuild itself sits idle can be lengthy and in some cases it may be a significant impediment to good parallelization of your build.

The problem has to do with the way MSBuild utilizes its worker nodes.  Whenever a project is scheduled to be built, it is assigned to one of the worker nodes.  This node will then execute that project from start to finish, and will not accept more work until the project is either finished or the project makes an MSBuild call (for instance to satisfy a project-to-project reference.)  This is in large part because a node can only execute one task at a time, as tasks must be guaranteed their environment and current directory will not be modified during execution.

However, command-line tools do not execute in-process, and therefore their environment cannot be polluted by the running of additional tasks in parallel on the same node.  We can take advantage of this behavior to let the MSBuild node execute tasks in other projects while our long-running tool completes its work.  This is done using the YieldDuringToolExecution parameter.

YieldDuringToolExecution

In order to allow MSBuild to continue building other projects while a command-line tool in one project is running is simple.  Just set the YieldDuringToolExecution parameter to ‘True’ on your long running command-line tool.  This is a boolean parameter, so any valid MSBuild expression which resolves to a boolean value will work.  Here’s an example:

<PropertyGroup><YieldDuringToolExecution>true</YieldDuringToolExecution>
</PropertyGroup>
<Exec CommandLine=”Sleep 10000” YieldDuringToolExecution=”$(YieldDuringToolExecution)”/>

When the Exec task executes, normally it would sleep for 10000 seconds during which no other work on the node can proceed.  However, with yielding enabled, the Sleep command will still run but the MSBuild node will be free to do other work.  Once the Sleep command is finished, the node will resume building the project which launched it as soon as the node is free to do so.

Whether or not you should enable yielding for your ToolTasks depends on what they do.  Generally speaking if the task runs for less than one second, it’s probably not worth it to enable this since there is a small cost to give up the MSBuild node.  However, for longer tools you may see some wins, and the wins will likely be larger the more complex your build is and the more long running tasks you have in it.  Again, large interdependent C++ builds are a great example of this and they benefit tremendously from yielding being applied to the compiler.  You can investigate your build’s performance using the Detailed Summary feature of MSBuild 4.

Yielding interacts well with the /m switch in MSBuild as well.  For instance, if you have specified /m:4 to enable parallelization, MSBuild will ensure that no more than four parallel things are going on at once, whether they be regularly building projects or yielding tools.  So enabling yielding will not cause your machine to become more overloaded.  Instead your builds are likely to improve their parallelization and make better use of available CPU and I/O cycles that they would otherwise. 

We have already enabled yield semantic for several tool tasks.  These include:

• CL, the C++ compiler
• MIDL, the IDL compiler
• Link, the native linker – Only when the LinkTimeCodeGeneration metadata is set to UseLinkTimeCodeGeneration

It could also be enabled for the Vbc and Csc tasks since they are ToolTasks as well, but this support is not in the Microsoft.CSharp.targets and Microsoft.VisualBasic.targets shipped with .Net 4.0.  You could easily add them yourself if you wished.  More generally, if you include Microsoft.Common.targets the YieldDuringToolExecution property will be set to true unless it is overridden with the parameter /p:YieldDuringToolExecution=false being passed to MSBuild.  We will continue to use this property as the basis for selecting the tool parameter value of the same name.

Why isn’t it automatic?

Unfortunately for MSBuild 4 we didn’t get the opportunity to make this system as automatic as we would like.  In future versions we would like to automatically yield when ToolTasks are executing if they look like they will last longer than a certain threshold.  This will also work together with additional automatic improvements in build analysis and scheduling we have planned.

Cliff Hudson - MSBuild Developer

Back when we started 4.0 development, I polled readers of the MSBuild blog to find out what features were most important to them. Debugging was #1 which was very surprising to us. Thinking about it more, it makes sense. In our team we've become so proficient ourselves at reading the XML and making sense of logs that it's easy to forget how difficult it is – especially for someone new. John Robbins, debugging guru, also requested a Visual-Studio-integrated debugger.

Fast forward to the 4.0 release earlier this year, and we addressed 7 out of 16 of the requests by my count. We had to balance the requests with what Visual Studio itself needed from MSBuild. There were two major requirements it had on MSBuild: to enable VC++ to move onto MSBuild (#5 request), and to help enable more powerful and fine grained multi-targeting.

It turned out that these two in turn required many other features, most of which were happily also popular requests on that blog poll. We added the ability to define a task with inline code (#7 – see powershell example) a new, comprehensive object model (#14; in three parts, one, two, three), improved performance and scalability in many cases (#8 -- and here), property and item functions (#9 – albeit not currently extensible) , and accurate automatic dependency checking by performing file system interception (#11), plus some small syntax additions (label, import group, import by wildcard) and a more configurable build engine (eg see here, here, and here), plus easier build extensibility and some performance diagnostics.

We didn't have time, unfortunately, to address converting the solution file to MSBuild (#3) – which we would dearly love to do – nor to add a Visual Studio integrated debugger (#1).

At least, not a supported one!

Mike Stall approached us to demonstrate an ingenious reflection-emit idea which made it considerably more feasible to create an MSBuild specific debugger with many of the features of the real managed code debugger. While on leave I wrote and checked-in the code to do it. Unfortunately we couldn't complete it in time to make the 4.0 schedule.

For that reason, it's in the product, but disabled by default. It does work, it's just not supported or documented, and has a few limitations and bugs: it may be slow, it's not always pretty, and in at least one case, it's a little inaccurate. This blog post is "unofficial" documentation of how to use it in the hope it will be useful. Although it's not supported we will welcome Connect feedback, but it will likely will be moved to our backlog rather than fixed immediately. It would also be a great idea to add any bug reports and feedback to the comments on this blog post.

Debugging Walkthrough

I'm going to walk through each debugging scenario in turn.

Before you start, open Visual Studio briefly and make sure that "Just My Code" is enabled. It's essential for this to work properly:

There's a lot of screenshots here, but this blog is rather narrow, so some of them are distorted – you can click on them to see the full size version.

Scenario 1 – Command Line only

First, enable the undocumented "/debug" switch on MSBuild.exe by setting the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSBuild\4.0 key to have debuggerenabled=true, as I've done here with reg.exe in an elevated Visual Studio prompt:

You should now have these keys, assuming C: is your system drive.

Run MSBuild /? and you'll see the new switch has appeared.

We are now ready to debug.

Normally you'd be debugging some build process you've customized or authored, but for illustrative purposes I'm going to debug a brand new C# Windows Forms project. I'm going to build it with the /debug switch, and it will immediately stop:

In my case I get a prompt to elevate, and hit Yes:

Then I get the standard JIT debugging prompt. Make sure you check "Manually choose the debugging engines".

that causes a dialog to appear to choose the debugging engine: you want Managed only. (Mixed will work, is more clunky.)

And you are now debugging!

The first thing to notice is that we are right at the top of the first project file, the very first line MSBuild is evaluating. You are breaking in automatically at the very start, as if you started debugging a regular application with "F11".  Well, almost the very start: MSBuild already read in the environment and its other initial settings:

Now hit F10 and you will step line by line:

As you step over properties, you'll see the locals window is updating:

As you probably know, MSBuild evaluates in passes. The first pass evaluates just properties, pulling in any imports as they're encountered. Try to set a breakpoint (F9) on an item tag right now – you can't! MSBuild is unaware of them at this point.

Set a breakpoint on the <Import> tag at the bottom and run to it (F5):

Now step in (F11). You'll enter the file that's being imported, which in this case is Microsoft.CSharp.targets.

The Callstack window shows that jump like a function call, including the location in the file:

Of course, <Import> does not have the semantics of a function call at all. Like an #include in C++, it simply logically inserts the content of another file. But I chose to make it work this way so that you can see the chain of imports in the Callstack window and figure out your context.

By setting some more breakpoints on Imports and doing step-into, I can go deeper to illustrate:

To get past the property pass, given that I can't set a breakpoint on items yet, I'll use a trick. I'll Step Out repeatedly (Shift-F11) until we get to the project file again, then step to get to the next pass, which is Item Definitions. The C++ build process uses Item Definitions a great deal, but they're not very interesting for C#, in fact there's only one:

Use the same trick to get to the Item pass, and we'll get to the first item. I've then set a breakpoint to illustrate that I can do that now.

Conditional Breakpoints work too, by the way as I believe do Trace Points.

Stepping a bit further, I can see items in the locals window, and also their metadata. A small bug here -- ignore the red message, and go into "Non-public members" to see the names and values:

Sometimes you'll want to figure what a condition evaluates to at the current moment. To do that, in the immediate window, pass the condition to the function EvaluateCondition:

It's much the same if you want to evaluate (expand) an expression, but the function is named EvaluateExpression:

This is also a convenient way to see what a property value is, or what's in an item list, without navigating through the locals window. Be sure to escape any slashes, as I've done here.

The Autos window doesn't work, but Watch does:

In the Immediate window you can change almost any project state during the build, using the new object model. For example, I'll modify this property while I'm stopped here:

You can do a lot through the new object model, so it's very useful to be able to call it here.

My Watch window updated to match:

That's the end of what I'm going to show for debugging MSBuild evaluation.

How it works

What's happening at the high level (you can find out more from Mike's blog) is that MSBuild is pretending the script is actually VB or C#. It's doing this by emitting IL on the fly that's semantically equivalent to what it's really doing as it goes through the XML. The code of MSBuild itself is of course optimized, so Just My Code hides it, but conveniently the IL isn't optimized, so it shows up. Inside the IL MSBuild emits line directives that point to the right place in the project file, completing the trick. As for the "locals", they're actually parameters passed to functions in the IL so that they appear. EvaluateCondition and EvaluateExpression are just delegates passed the same way.

As such, a large part of the basic features you get with the regular VB/C# debugger just work. Some that don't: hovering over an expression doesn't give you the result; you can't just use the "?" syntax in the immediate window; Threads and Processes windows don't make sense; I doubt Intellitrace works. Plus, there's some of our internals leaking out in the windows here and there. But by using this trick, it was vastly less work to get the basics of an integrated debugger. I believe I spent a day or two tidying up Mike's sample code, and another three days wiring it straightforwardly into MSBuild. Creating a real debugger engine would be much more costly; and something comparable with what you get for C# would be fantastically costly, so I expect that long term, this will be the MSBuild debugging story. I hope you'll agree it's a lot better than staring at XML and logs or adding <Message> tags.

In my next post I'm going to cover

• Debugging during the build – ie., debugging what happens inside targets, and project references;
• Debugging a multiprocessor build;
• Debugging the build of projects loaded into Visual Studio

See you then!

Dan

Visual Studio Project & Build Dev Lead

In my previous post, I showed how to enable the hidden Visual Studio debugger for MSBuild script, and demonstrated it by stepping through the evaluation of a C# project. In this post, I'll keep debugging into the actual build of that project.

Note that this blog is rather narrow, so some of the screenshots may be hard to see – you can click on them to see the full size version.

Starting from where we left off last post, I'll set a breakpoint in Microsoft.CSharp.targets on the <Csc> task tag. That's where the compiler will run. Then hit F5 to run to it.

Ideally, I'd be able to set a breakpoint on the enclosing <Target> tag, but unfortunately there's a bug: you can't. As a workaround you could inspect the values of a target's attributes when you get to the first tag in the body of that target. If the target's condition is false, or if its inputs and outputs are up to date so that it skips, it's not so simple. You'd have to work around this by stepping up to the target before.

I'd like to be able to evaluate the condition on the task there, but because it's batchable it could have multiple results: the EvaluateCondition delegate I used before won't accept it. If this was a parameter on the task, I'd probably step into the task's code itself to see the value. Since it's a condition, I'd probably look through the metadata on the item list directly, or query the object model in some way.

Something like the value of Sources is easy to evaluate here, though:

Now I want to step into the task implementation.

You might think that at this point, you can simply Step In (F11). However, you can't – it happens that MSBuild will run this task on a different thread. To know to jump threads properly here, the debugger has to support what they call "causality" for this kind of debugging, and since it doesn't know anything about MSBuild, it doesn't.

It's easy to get the job done though – set a breakpoint in the task and run until it's hit.

I have the source code for the Csc task, so I set a breakpoint here on the setter of the WarningLevel property, and did Continue (F5). I can see the task is getting "4" as the value of that property here. I can debug this code just like any other C# code, stepping through methods and so forth.

To get out to the XML, I'll set a breakpoint in it and run – the same trick I used to get into the C#, but in reverse. Here I'm at the next tag after the task:

I used Csc as an example here, but you'll generally be debugging a custom task (or possibly, logger). Just make sure the assembly is not optimized: since you have Just-My-Code on, it won't be debuggable otherwise. If you only have access to an optimized one, you can switch off Just-My-Code temporarily.

There's a CallTarget tag here: you can step into those, and like imports they'll look like a new frame on the callstack – although unlike imports, that's correct for their semantics.

Probably the biggest limitation (bug) with the debugger right now is that you can't see item and property changes made inside a target. For example, at this point @(_CoreCompileResourceInputs) should be empty because of the line above, but the immediate window tells me it isn't:

When you get past that target, you can see the changes.

Scenario 2: Debugging a build with multiple projects

Typically there's more than one project in a build. I've added a project reference from this project to another. I've put a breakpoint at the top of that project, and run to it:

The bottom of the callstack is the point in Microsoft.Common.targets where, early in the build of WindowsFormsApplication1, it invoked WindowsFormsApplication2.

In my next posts I'm going to cover

• Debugging a multiprocessor build
• Debugging the build of projects loaded into Visual Studio

See you then!

Dan

Visual Studio Project & Build Dev Lead

In my last two posts (here and here) I showed how to enable the unsupported MSBuild debugger to debug a build started on the command line with MSBuild.exe. In this final post, I'll mention some other variations.

Note that this blog is rather narrow, so some of the screenshots may be hard to see – you can click on them to see the full size version.

Scenario 3: Debug a Solution file on the command line

In the previous example, I launched msbuild.exe against a project file. You well might want to start with a solution file instead. If you do, you may get an error that looks like this:

Microsoft.Build.Shared.InternalErrorException: MSB0001: Internal MSBuild Error: Mismatched leave was C:\Users\danmose\Do
cuments\visual studio 2010\Projects\WindowsFormsApplication1\WindowsFormsApplication1.sln.metaproj expected C:\Users\dan
mose\Documents\visual studio 2010\Projects\WindowsFormsApplication1\WindowsFormsApplication1.csproj (2,1)

The workaround is as follows. First, set an environment variable MSBUILDEMITSOLUTION=1. Then do a build of the solution in the regular way: if you want, you can cancel it after it starts building projects. Next to the solution file, you should see a file with the extension ".sln.metaproj". This is essentially the solution file translated into MSBuild-format. Now do the usual debugging procedure, but this time launch msbuild.exe against this sln.metaproj file instead of the original solution file.

Scenario 4: Multiprocessor build

When you're debugging your build process or tasks, it's much easier to follow if only one project is building at a time, and you'll probably do it that way whenever you can. What's more, debugging slows down the build so much that it may not help you diagnose a timing problem anyway.

If however for some reason you do need to debug a multiprocessor build, it's possible.

As you probably know, MSBuild launches child processes to build more than one project at once, and they persist for a while to be ready for another build. The /debug switch doesn't propagate to them. To get this to work, first terminate any msbuild.exe processes that are still alive. Then in a command window set an environment variable MSBUILDDEBUGGING=1. That environment variable is equivalent to the /debug switch, but unlike the switch it will get propagated to any child processes.

From this command window now start msbuild.exe with the /debug switch as usual. Everything will work much the same as before, but you'll get a new JIT prompt as each child process starts to do its work, and you'll have to use a new instance of Visual Studio for each one of them.

For example, I'm building a solution here, using the .metaproj workaround I mentioned above. Attaching at the first JIT prompt, I can see I am starting in the .metaproj itself:

If I hit Continue (F5) I'll get to the top of the next project, and stop there as usual. In the callstack window, I can see that the solution has invoked that project, and I can double click on the lower frame in the callstack to see where it did that – it's an MSBuild task, of course:

While I'm stopped here, I get a JIT prompt again. This time it's for the other project in my solution, which has already started to load in parallel in another msbuild.exe process. I'll go through the JIT prompts as I did before, and select to start a new instance of Visual Studio. That will in turn break in at the top of that other project.

Here's what I see now:

Visual Studio supports debugging more than one process at the same time, which would be more convenient, but I don't think the JIT launcher will let you do that. If you have a lot of child processes, it could get rather cumbersome. Remember that the "/m" switch defaults to the same number of processes as you have CPU's, so you may want to cut it down with "/m:2".

Scenario 5: Debugging projects while they are loaded in Visual Studio

When I wrote the prototype, it was also possible to debug the evaluation and building of projects loaded in Visual Studio.

Unfortunately in walking through this scenario to write this blog post, I was sorry to find that it's somewhat broken in the release version of Visual Studio 2010. In my experiments debugging works through the first evaluation, as the projects are loaded, but then it hits a bug and terminates. Since it's an untested feature there's always the chance something can break without detection and that's apparently what happened here.

Given that, you'll most likely have to stick with msbuild.exe. However, I'll go ahead and explain a little about how one would do this in Visual Studio, in case you have better luck than I do.

Start off by setting the registry key as described at the start of my first post and kill any lingering instances of MSBuild.exe as you did in the last scenario.

Make sure the environment variable MSBUILDDEBUGGING is not set and open your first Visual Studio to act as your debugger. You can do this from the start menu. As before make sure Just-My-Code is switched on in this instance.

Open a command prompt and set the environment variable MSBUILDDEBUGGING=1 again. Then launch Visual Studio from that command prompt -- most likely you'd start it with a command like "%ProgramFiles%\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe". This second instance is your debuggee, which you will load your projects into.

Go back to the debugger process, and attach to that debuggee, choosing the "Managed (v4.0)" engine as always. In this debugger process, open up the project or targets file you want to start debugging through. Make sure you open it with "Open File" into the XML editor, rather than loading it as an actual project.

Debugging may not start automatically on the first line this time, so set a breakpoint where you want to begin. I opened the "WindowsFormsApplication1.csproj" file I used before into the XML editor, and set a breakpoint on the first line to get hit when the project is first loaded.

Open up your project or projects in the normal way in the debuggee now and you should see your breakpoint is hit. Step a little further, and for me, Visual Studio terminates.

End of the Walkthrough

Now I've shown you all the features of the MSBuild debugger, I hope you'll try it out.

If you have feedback, do post it here. In particular, how important is it to be able to debug projects loaded inside Visual Studio? Also, as you've seen, MSBuild must be in "debugging mode" from launch -- it's not possible to walk up to a regular build that's in progress and attach to it. Is it important to be able to do that?

Most of all, how useful is this to you, and how problematic are the bugs and limitations you run into?

Thanks for reading

Dan

Visual Studio Project & Build Dev Lead

Since the release of Visual Studio 2010 we have received a few reports of crashing behavior which can be traced back to issues with MSBuild.  We’ve analyzed all of these and there are several particular cases where a crash can occur.  We’ve also added a notification to Windows Error Reporting to help guide those who hit these errors.  You can determine you your error is one of these either by matching the problem description below, or looking in the Event Viewer as follows:

1. Open the Event Viewer
2. Search for Information events with ID = 1001 and Source = Windows Error Reporting.  Look for those with the time that approximately matches when you saw the crash.
3. At the top of the details pane for the event is text that would look like the below.  If the bucket number is not 1055654512, then this post may not apply to you.

Fault bucket 1055654512, type 1

Event Name: APPCRASH

Response: xxxxxxx

Cab Id: 0

Crash when debugging using F5

Problem: This can occur if the build process is missing a required target.  This is normally due to an improperly customized build process.  If you are using the .NET MicroFramework 4, which is not supported in Visual Studio 2010, you may also see this issue.

Solution: Provide the missing target.  Try building the project/solution on the command-line.  If MSBuild logs an error that a target is missing, that could be the problem. 

Crash when registering COM component

Problem: COM registration requires the user have permission to certain registry keys, and lacking that permission the RegASM task crashes.

Solution: Ensure the registry keys needed to perform the registration are accessible to the account doing the registration.  Look under HKCR\Record for the GUID matching the type (or types) associated with the COM components you are registering.

Invalid Project Exception when building with .NET MicroFramework 4

Problem: When building projects with Visual Studio 2010 and the .NET MicroFramework 4, the build would fail with an InvalidProjectFileException.  The .NET MicroFramework 4 is not compatible with Visual Studio 2010.

Workaround: There is unfortunately no workaround for this.  According to the .NET MicroFramework 4 team, the next version of the MicroFramework will support VS2010.  Check out http://www.microsoft.com/netmf/default.mspx for updated information.

MSBuild throws an error that it cannot find msbuild.exe

Problem: Building on the command-line or from Visual Studio displays an error that MSBuild could not find MSBuild.exe during a C++ or multiproc build.  If your username is exactly 20 character long excluding your domain (the maximum allowed under Windows), there is a bug in the .NET Framework which will prevent us from authenticating the other MSBuild nodes.

Workaround: Build under an account with a name that is less than 20 characters.  The bug should be fixed in the next version of the .NET Framework.

MSBuild throws an OutOfMemory exception

Problem: Your build in Visual Studio aborts with an OutOfMemory exception.  It may or may not also do this when built from the command-line.  This occurs typically when there are a very large number of projects with a lot of interdependencies, or when projects have an extremely large number of source files.

Workaround: Build your solution on the command-line, so that your process does not have Visual Studio’s initial memory foot print; or build on a 64-bit machine under a 64-bit command window so we can take advantage of the additional virtual memory spacel or split your solution into smaller chunks which can be built individually; or create a solution configuration in which only a subset of your projects build.

Visual Studio crashes when building a solution containing C++ and WiX projects

Problem: When building a solution which contains C++ and WiX projects, Visual Studio may crash.

Solution: This problem has been traced to a bug in the WiX project system.  Please contact the WiX project for a newer version with the correct bits.

Other issues

For some crashing issues we have set up Windows Error Reporting so that it will automatically request for you to send us additional information and contact us directly.  If you see such a request, please consider providing us with the requested additional information so we can either determine that your issue is already known or address it in the next version of the product.

We've had a few reports of cases where Visual Studio, and previous versions of MSBuild, will build the projects in the solution in the correct order, but the 4.0 version of MSBuild.exe gets the order wrong.

Here's a full description of what's going on, why it began in 4.0, and the fix we recommend to your projects to solve the problem. If you're not interested in the "why", skip ahead to the workaround.

Archetypical case exhibiting the problem

This diagram shows a solution file containing three projects, A, B, and C. Let's say they are C# projects.

A has a regular project reference to B, so it will invoke B, then when B comes back done, it will build itself. At the same time in the solution file, there is a manually specified dependency: B depends on C.

I've shown regular project references with solid lines, and the information in the solution with dotted lines.

This manually specified dependency was set up in the solution by right clicking on the solution and choosing Project Dependencies…, then checking a box. Below I've shown the context menu and what you see in the dialog when you have this setup.

The build ordering you expect here is C, then B, then A, and Visual Studio shows that correctly in the Build Order tab, as you see below.

Of course a real case would have more projects in it, but it would boil down to this case.

Why does this happen (skip ahead if you just want the "fix")

Essentially the problem is that MSBuild doesn't know anything about the project files until it starts to build them.

Solution files, as you know, are not in MSBuild format (yet). On the command line, MSBuild.exe is on its own, so it parses them and generates one or more in-memory MSBuild format files that are essentially a translation. If you want to see these ugly files, set an environment variable MSBUILDEMITSOLUTION=1 then build the solution. You'll see a .sln.metaproj file emitted next to your solution file, and possibly one or more files with an extension like .csproj.metaproj next to some of your projects.

The .metaproj generated for B.csproj is how MSBuild makes sure that the solution dependency is respected -- at least, it's created so that the solution itself does not invoke B until C is built. It does this by invoking the B metaproj instead of B directly, and in the B metaproj, it builds C before B. This is exactly equivalent to someone going into B and adding a project reference to C, instead of a solution dependency, but it means that we don't have to edit the B project directly.

Here's what it looks like after this translation:

Why doesn't that work in this case? In short, the problem is the project reference from A to B. Here's what happens: the solution invokes A.csproj, B.metaproj, and C.csproj concurrently (or at least, in undefined order), which would normally be fine. B.metaproj invokes C.csproj, and waits, then invokes B.csproj. However in the meantime, A.csproj was invoked, and because it has a project reference to B.csproj, it invokes B.csproj -- it "goes around the back". C.csproj hasn't necessarily built yet, so the build breaks.

Why did this work in previous versions of MSBuild?

In previous versions, we loaded and scanned every project file listed in the solution, and any they referenced, in order to draw a complete graph. Then we used the graph to create the MSBuild format equivalent of the solution file. The reason we did all this scanning was not actually to address this problem, it was to make interop with old-style non-MSBuild VC projects (".vcproj") work correctly. It was also slow, especially for large solutions.

In VS2010, VC projects converted to MSBuild, so in 4.0 we took out this complex interop code. After making .metaproj's to express any dependencies stored in the solution file, we could now simply invoke all the projects in the solution and the build would order itself. That was potentially much faster, because we didn't need to load any projects (potentially hundreds) to scan them before building anything. (Of course, when MSBuild 4.0 is fed a VS2005 or VS2008 solution file, it still calls into the old code in the old assembly to do it the old way, since they may contain .vcproj's. So those guys don't have this problem.) The oversight was this case -- where a project reference "goes behind the back" of a solution-expressed dependency.

To fix this we would have to revert to loading and scanning, which slows things down -- the correct approach is to use project references instead of solution dependencies, as I explain below.

How to fix this

Follow this principle: do not use dependencies expressed in the solution file at all! Better to express dependencies in the file that has the dependency: put a project reference in the project, instead. In our example, that would be a project reference from B to C.

You may not have done that before because you didn't want to reference the target of the project reference, but merely order the build. However, in 4.0 you can create a project reference that only orders the build without adding a reference. It would look like this - note the metadata element, and all this is inside an <ItemGroup> tag of course:

<ProjectReference Include="... foo.csproj"> 
    <ReferenceOutputAssembly>false</ReferenceOutputAssembly> 
</ProjectReference>

Note that you have to add the child element with a text editor -- Visual Studio can add a project reference, but doesn't expose UI for this metadata.

I can tidy up by removing the dependency in the solution file as well - removing now-unnecessary lines like this -- your GUID will be different, but use the VS dialog and it will do the job.

    ProjectSection(ProjectDependencies) = postProject 
        {B79CE0B0-565B-4BC5-8D28-8463A05F0EDC} = {B79CE0B0-565B-4BC5-8D28-8463A05F0EDC}

    EndProjectSection

If you're using C++ projects, you are less likely to have this problem, because in the upgrade process that converts .vcproj's to .vcxproj's, it moves any solution dependencies relating to them to project references for you. However, if you do, there's a similar fix. For C++/CLI project references to other managed projects, use project references like the one above. For the equivalent situation with a project reference to a static lib, where you want a project reference without linking in the referenced lib, the metadata you want is

Summary

Although it's tiresome to have to edit your projects in this way to make the bug go away, it's a best practice to use project references instead and consider the solution file merely a "view", and you'll end up with projects that if you want can be built without a solution file.

Post Script

I know of one other, more obscure and completely different case, where MSBuild 4.0 does not order correctly but Visual Studio does. This can happen if you have web application projects, AND you build with 64 bit MSBuild (which is the default, in Team Build 2010). I won't go into the tedious details but the fix is to do one of these things: (1) set a property or environment variable named MSBuildExtensionsPath to C:\program files (x86)\msbuild or (2) Build with 32 bit MSBuild, which I recommend in general for other reasons or (3) copy the web application targets files under the 64 bit program files MSBuild folder to the equivalent location in the 32 bit program files MSBuild folder.

If you find any other case where the solution is not ordering correctly yet this workaround does not work, that's interesting. Please make a minimal repro like the one in this bug, and send it to Dan atmsbuild@microsoft.com.

Dan

Update (12/25)

Luc C points out below that sometimes removing a project reference in favor of a new solution dependency reference is an alternative solution. That assumes you're only using the project reference for ordering , or you replace it with a file reference. Still, I do recommend project references in general, on the principle of "express the dependency in the place it applies" – you can look at the project and see that the dependency is correct, and you can include the projects in more than one solution easily.

Luc also points out the case of a managed project depending on a non-CLR native project (presumably for PInvoke). In my experiments, VS will let you add a project reference, albeit with an ugly bang, and it will do the ordering correctly, as will msbuild.exe.

Not many books that are reviewed like this on Amazon:

Now the heavily augmented second edition has just come out, written by several people at Microsoft and reviewed by the product team.

• New extensive coverage of building C++ with MSBuild
• Complete rewrite of the Team Build sections to cover the new Windows Workflow Foundation build orchestration
• Detail on new MSBuild 4.0 features like inline tasks, and item and property functions.

If you need a reference to Team Build or MSBuild, you should get it.

-- Dan, VS Solution/Project/Build dev lead

Update: Here's a link to the book on Amazon

We have an infrastructure in developer division for performance measurement. Before a feature crew integrates into their product unit branch, or a product unit does one of their periodic integrations of changes into a main branch, they must pass RPS ("regression prevention system"). This runs of the order of 100 performance scenarios in a clean, isolated environment to get very precise numbers and a whole bunch of performance counters. If certain counters regress - elapsed time, disk I/O, or memory allocations I think - the run is rejected and the performance regression must be fixed before integration can occur. Towards Orcas RTM MSBuild was still missing a performance goal for certain command line solution build scenarios and we had to find a quick way to speed them up.

When you launch MSBuild.exe on a solution file -- either directly, or using Team Build - some code has to run to create an in-memory equivalent MSBuild format project file that MSBuild can actually build. Ideally this ought to be trivial. We would essentially construct a project file that simply ran the MSBuild task on all projects listed in the solution. In reality not all projects in the solution are in MSBuild format, most especially VC projects. It turns out that we have to load and evaluate every project in the solution to get some information we need to order the solution file correctly with respect to those VC projects, and in 3.5 we also have to figure out possible parallelism between projects in the solution. After this is done, we build the result, which of course involves loading and evaluating those projects again to actually build them.

You may have set MSBuildEmitSolution=1 in the past to see the resulting file. Editing this isn't supported, because the format will change from version to version, and it needs to be reconstructed if the solution is changed. However, it can be useful.

The solution cache file is a simple serialization of the in-memory solution file to disk. Much like MSBuildEmitSolution, but we read it too. When asked to build a solution file, MSBuild will use the solution cache if it's present and is valid for the configuration, tools version, and MSBuild version requested, and is up to date with respect to the solution file itself. This saves the time spent constructing the in memory solution project. It can go right ahead and build it.

This isn't very pretty, because it's exposing ugly implementation details in a plain text file. Also, there's no intermediate directory for a solution file like there is for a project, so we have to write it in the same directory, which is impolite, and not desirable if you wish to keep your source tree clean (for example, you have redirected your intermediate and output directories elsewhere). There is one other issue. You can't safely build the solution concurrently with two different configurations, because the configuration isn't part of the filename, and the content of the file is configuration specific. Team Build needs to do this.

The workaround for both issues is to set an environment variable to disable creating this file. Set MSBuildUseNoSolutionCache=1.

When solutions and VC projects are MSBuild format this kind of problem will go away.

Dan

[edit: fixed typo spotted by Cullen Waters]

Six months ago, the developers on MSBuild began to use an internal tool then called "StyleCop" in order to clean up our source code going forward. Think of it as a little like "FXCop" for your C# style. It flags as build warnings all kinds of deviations from a set of fairly strict style rules. We excluded all our existing source code files from inspection, since they generated thousands of warnings, and used it for all new source files, and for heavily rewritten old code, and required every check-in to be clean.

For a few days, it was frustrating, mainly because you have to accept its rules over your own ingrained conventions. I don't like some of them. Frankly however many of these conventions are important only for consistency, and not intrinsically better or worse than any others.

After that first week, we were surprised to find that once we got used to the rules, it was little or no effort to avoid the warnings, and the readability and consistency of our source code has improved considerably. I thought I would have to push the team to use it, but they picked it up enthusiastically, and one of the developers even enabled it on all our unit test code as well. There's no more "interesting" use of tabs, or personalized curly placement, and code reviews can focus entirely on the content of the change. As others joined our team, they picked it up easily too.

Well, StyleCop has just been released as "Microsoft Source Analysis". It's a free download from here. I highly recommend it.

Dan

I am very happy to pass on news from Mike Fourie that has released v1 of a remarkably extensive MSBuild Extensions Pack on codeplex. In his words: 

The MSBuild Extension Pack is the successor to the FreeToDev MSBuild Tasks Suite and provides a collection of over 170 MSBuild tasks designed for the .net 3.5 Framework. A high level summary of what the tasks currently cover includes the following:

• System Items: Certificates, COM+, Console, Date and Time, Drives, Environment Variables, Event Logs, Files and Folders, GAC, Network, Performance Counters, Registry, Services, Sound
• Code: Assemblies, CAB Files, Code Signing, File Detokenisation, GUID’s, Mathematics, Strings, Threads, Zip
• Applications: BizTalk 2006, Email, IIS7, MSBuild, SourceSafe, StyleCop, Team Foundation Server, Visual Basic 6, WMI

Each task is documented and provided with an example in the help file. Where applicable, tasks are remote enabled, simply specify a MachineName and the task will target the remote machine.

Additional tasks and improvements to the documentation will be released frequently. I would encourage users to make use of the Issues and Discussions features on CodePlex and to contribute code to help drive this project forward.

Mike's goal is to make this the standard MSBuild task pack. With the energy he is putting into it, he has a good chance of that. If you're interested in contributing drop him a line at mailto:feedback@msbuildextensionpack.com

-- Dan

Well the CTP is now out, and I hope all are enjoying a preview of the new features being released in Visual Studio 2010.

Of particular interest is the walkthrough "How to Create an Inline Task".

• In this walkthrough, "How to Create an Inline Task", we are highlighting our new feature "Inline Tasks" that allows you to write tasks in C# right inside your MSBuild (Project) file. This means you no longer have to build a separate assembly. It helps make custom build steps easy. Our sample walkthrough highlights how you might write a custom build rule to send an email when your build is complete.

However, with the quick pace to get the CTP bits out the door, we realized that some critical files for the Inline Tasks walkthrough did not make it onto the VPC.

In order to provide you with this really cool walkthrough, we have updated the documentation and have provided the missing bits in a zip file.

Please check out the post in our forums for more information.

   http://social.msdn.microsoft.com/Forums/en-US/vs2010ctpdeploy/thread/f30edddb-4183-499b-942b-105d58a78a86/

Thanks,

Chuck England
Visual Studio Platform
Program Manager - Project and Build

We aren’t going away -- we’re just moving to a better home. To give you even more news, articles, and walkthroughs on Visual Studio, the MSBuild Team blog is moving to become part of The Visual Studio Blog.  As part of The Visual Studio Blog, we’ll be joining the rest of the Visual Studio Platform Team in blogging not only about MSBuild, but other topics such as the core IDE and UI, the editor, the project system, VS extensibility, and more. 

So please make sure to update your RSS readers to our new home. 

See you all at the new blog location – http://blogs.msdn.com/visualstudio!

One of the cool new features of MSBuild 4.0 is the extensible task factory.  Task factories allow you to include scripts directly in your project file (or an imported .targets file) and have those scripts parsed and executed by your favorite interpreter.  Those scripts might even be C# or VB.NET code snippets that get compiled into assemblies and executed on-the-fly during the build.  This significantly lowers the bar to doing specialized work in your build for which there is no built-in MSBuild Task, since you don’t have to check into your source control repository a precompiled .dll with your custom task any more.  You may still want to do that for performance reasons or for the design-time experience of writing MSBuild tasks in C# and get all the C# Intellisense that you only get in a C# project, but for short tasks, “inline tasks” that are built and executed on-the-fly may be just the ticket you need.

MSBuild 4.0 ships with C#, VB.NET and XAML task factories built-in, so you can define a custom task in C# or VB.NET inline today.  But writing your own task factory will allow you to write inline tasks in Perl, Python, or in this case… Windows Powershell script.

Consider, for the sake of example, that there is no MSBuild task that sends an email at the conclusion of a build.  Fixing that with a custom MSBuild task, compiled into a .dll, seems a bit heavyweight.  But defining that task right there in your project file is easy.  Here’s how you might write a Powershell inline task using the Windows Powershell task factory.  Notice below how we first define the task in terms of the code it executes and its input and output parameters.  Then in a standard MSBuild Target we invoke that inline task as we would any other MSBuild task. 

<?xmlversion="1.0"encoding="utf-8"?>

<ProjectToolsVersion="4.0"DefaultTargets="Build"xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <UsingTask

      TaskFactory="WindowsPowershellTaskFactory"

      TaskName="SendMail"

      AssemblyFile="$(TaskFactoryPath)WindowsPowershellTaskFactory.dll">

        <ParameterGroup>

            <FromRequired="true"ParameterType="System.String" />

            <RecipientsRequired="true"ParameterType="System.String" />

            <SubjectRequired="true"ParameterType="System.String" />

            <BodyRequired="true"ParameterType="System.String" />

            <RecipientCountOutput="true" />

        </ParameterGroup>

        <Task>

            <![CDATA[

        $smtp = New-Object System.Net.Mail.SmtpClient

        $smtp.Host = "mail.microsoft.com"

        $smtp.Credentials = [System.Net.CredentialCache]::DefaultCredentials

        $smtp.Send($From, $Recipients, $Subject, $Body)

        $RecipientCount = $Recipients.Split(';').Length

        $log.LogMessage([Microsoft.Build.Framework.MessageImportance]"High", "Send mail to {0} recipients.", $recipientCount)

      ]]>

        </Task>

    </UsingTask>

    <PropertyGroup>

        <BuildLabEmail>buildlab@yourcompany.com</BuildLabEmail>

        <BuildRecipients>interested@party.com;boss@guy.com</BuildRecipients>

    </PropertyGroup>

    <TargetName="Build">

        <SendMailFrom="$(BuildLabEmail)"Recipients="$(BuildRecipients)"Subject="Build status"Body="Build completed">

            <OutputTaskParameter="RecipientCount"PropertyName="RecipientCount" />

            </Add>

        </Target>

</Project>

This project would build, and send a built completion email to recipients determined by the project itself.  The only assembly you need is the Windows Powershell task factory itself, which can be reused for all your projects that include inline Windows Powershell scripts.

Note that the inline task above will not run without the task factory .dll that it points to, which in this case is not shipped with VS2010.  The Windows Powershell task factory is available as a sample task factory on the MSDN Code Gallery.  Feel free to check it out, and give feedback.  It’s just a sample, so all the “no guarantees for fitness to any particular task” limitations apply.  But you get the source code, so you can tweak it or file bugs if you wish.

Andrew Arnott