Setup 100% non-windows build-deploy-test flow from Tfs 2015

Tfs 2015 comes with many new build management features. One of my favorites is the new cross-platform build agent that they introduced and its open sourced at github. Feels like this capability unlocks huge opportunities for those who have invested in VS ALM stack. Already VS ALM offers best suite tools for Windows/Microsoft world of things. With this new cross platform build capability and revamped build authoring from web browser experience makes the job much easier.

Lets assume, I’ve a web app and want to deploy that to Nginx on an Ubuntu server. In order to coordinate Build-Deploy-Test flow, I’m going to use the vso-agent. This is 100% non-Microsoft stuff and lets see how it works together.

Setup a brand new Ubuntu box

I use Vagrant to spin up new base Ubuntu from Hashicorp.

I dont have NodeJs, Nginx or Npm installed.

Screen Shot 2015-07-29 at 7.39.54 AM

Setup Ansible playbook for app deployment

Assuming you are familiar with Ansible. I’m not an expert in this cool technology but it was too hard to pick up lately, very interesting one for orchestrating DevOps flow from environment provisioning thru monitoring. I got a simple playbook that will install NodeJs, Nginx then copy my static content over to the Nginx box, run post deployment xml/json transformation.

Below is the snippet, pretty simple self explanatory .yml file..

  • highlighted ones are the modules I use to install NodeJs and other prerequisites
  • task to copy the index.html
  • task to transform the config file using exising node modules

Screen Shot 2015-07-29 at 8.33.39 PM

Prepare_env module, just installs nginx and set of node modules required for my app

Screen Shot 2015-07-29 at 8.38.15 PM

Another simple run.sh to kick off the playbook (I’ve setup .ssh keys behind the scene and sudo password for remote installation is encrypted using ansible vault module and vault password is given for execution via –vault-password-file)

Screen Shot 2015-07-29 at 8.41.57 PM

Setup Tfs cross-platform build agents

Just follow the instructions here and from this walkthru video, you should be all set. Just run the agent interactively.

I’ve installed the agent on my mac, its up and running

Screen Shot 2015-07-29 at 8.49.46 PM

Setup Ansible on the build Agent

In order to run the deployments on Ubuntu box, I’m going to use Ansible. vso-agent will kick off the playbook and hence we need to install Ansible on the agent machine. In my case its the Mac book. Run “brew install ansible” to install Ansible.

Connect the dots

Now that we got all fundamentals covered, idea is to create a build that will pull my source code from tfs git repo, run Mocha tests (unit and integration tests) if successful deploy thru Ansible.

I use built-in tasks provided by the new build system.

Npm Install – to install dependencies

Screen Shot 2015-07-29 at 8.57.01 PM

Shell script that kicks off Mocha tests (really it’s one line and that is “npm test”. I dont know if there is a better way to kick off npm tests)

Screen Shot 2015-07-29 at 8.57.13 PM

Shell script that kicks off Ansible playbook that I mentioned earlier (run.sh above)

Screen Shot 2015-07-29 at 8.57.33 PM

Queue the build ..

Make sure to select the correct Agent, in this case not Hosted, its Default because I added my Mac vso-agent part of Default pool

Screen Shot 2015-07-29 at 9.04.53 PM

Running the build in my local vso-agent

Screen Shot 2015-07-29 at 9.11.10 PM

Seeing the Build status in VSO .. nice rolling log

Screen Shot 2015-07-29 at 9.05.15 PM

Screen Shot 2015-07-29 at 9.05.22 PM

Below output shows NodeJs, Nginx installation, copying files, transforming the sample .xml/.json config and finally starting Nginx..

Screen Shot 2015-07-29 at 9.46.03 PM

Screen Shot 2015-07-29 at 9.46.17 PM

So far, cross-platform build agent is very impressive capability, we might see some shortcomings interms of existing OOB tasks, like it comes with Gulp and if we want to run Grunt which is still widely being used, we need to wrap the grunt execution in some shell script and run that shell script. However, step in right direction to enable everyone to use VS ALM stack.

Classic Build Maturity path

Often there are some questions from the management with respect to build management space, especially when code promotions/release take more time, late night fire fighting over a period or build promotion flunked and QA team raises concerns

1. What is my build maturity?

2. Where does it stand against industry best practices?

3. My build engineer says we have the best process but still it fails often, takes longer duration and I feel some gap….

I tried putting together a classic build maturity path which could help us to comprehend the current state and best state too…I certainly think the maturity could vary based on the application/product architecture and there could be genuine reasons towards some of the current facts … however, I trust this could be a starting point when we stand clueless

I categorize them from Stage 0 thru Stage 4 as shown below

If you are following Agile methodology, I swear anything lesser than Stage 4 is affecting your delivery, quality & productivity. There are plenty of tools in the market which could help you to uplift from wherever you are today and reach Stage 4 fairly in about 6 months time.

Hope this post is useful.

Thanks!!

Find and replace a content in Web.Config using MSBuild….

If you wonder how to read & update a specific node in web.config(or any other xml file) using MsBuild, here is a sample…

This is very handy for automatic, unattended deployments…

You need to have Tigris MSBuild Community tasks installed ….. this script uses XMLRead & XMLUpdate tasks from that MSBuild extension….

You need to have .NET FWK 3.5 installed too…

——————————————————————————————————————————————————————

<Project

xmlns=”http://schemas.microsoft.com/developer/msbuild/2003&#8243; ToolsVersion =”3.5″ InitialTargets=”FindReplace”>
<!– Required Import to use MSBuild Community Tasks –>
<UsingTask AssemblyFile=”$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.dll”
TaskName=”MSBuild.Community.Tasks.XmlRead”/>
<UsingTask AssemblyFile=”$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.dll”
TaskName=”MSBuild.Community.Tasks.XmlUpdate”/>
<!– *********************  Target to read the xml file ********************************–>
<Target Name=”Read”>
<!– Read Test Service end point –>
<XmlRead
XPath=”//configuration/appSettings/add[@key=’TestService’]/@value”
XmlFileName=”Web.config”>
<Output TaskParameter=”Value” PropertyName=”TestServiceEndPoint” />
</XmlRead>
<Message Text=”$(TestServiceEndPoint)”/>
<!– Read ConnectionString –>
<XmlRead
XPath=”//configuration/connectionStrings/add[@name=’DevConnection’]/@connectionString”
XmlFileName=”Web.config”>
<Output TaskParameter=”Value” PropertyName=”ConnectionString” />
</XmlRead>
<Message Text=”(ConnectionString)”/>
</Target>
<!– *********************  Target to find and replace a value within the xml file *************–>
<Target Name =”FindReplace”>
<XmlUpdate
XmlFileName=”Web.config”
XPath=”//configuration/connectionStrings/add[@name=’DevConnection’]/@connectionString”
Value=”CrapCrapCrap123456″ />
</Target>
</Project>

Add build steps in TFS 2005

<BuildStep Msbuild task is not supported in TFS 2005. If you wonder how to update the build step with your custom messages, here is a Power Shell script… pretty easy …

# load the required dll
[void][System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.TeamFoundation.Client”);
[void][System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.Build.Framework”);
[void][System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.TeamFoundation.Build.Common”);
[void][System.Reflection.Assembly]::LoadWithPartialName(“Microsoft.TeamFoundation.Build.Proxy”);

# create networkcredential object instance with (“username”,”pass”,”domain”)
$netcredential=new-object system.net.networkcredential (“BuildSvcID”,”*****”,”Self”);
$tfs=new-object Microsoft.TeamFoundation.Client.teamfoundationServer(“
http://Demo-TFS:8080″,$netcredential);
Write-Host $tfs;
$Buildstore=$tfs.getService([Microsoft.TeamFoundation.Build.Proxy.BuildStore]);
#construct the build uri – provide team project name and build number as parameters
[string]$builduri=$buildstore.getbuilduri($teamProject,$buildNumber);
Write-Host $builduri;
#add build step….
$buildstore.addbuildstep($builduri,”custom build step-TFS 2005″,”custom build step message”);
#to update the build step status
$now=get-date;
$buildstore.updatebuildstep($builduri, “custom build step-TFS2005”, $now, “failed”);

TEAM FOUNDATION BUILD: BRIEF INTRO

Team Foundation Build is a build orchestrator, part of Microsoft Application Lifecycle Management suite. Tightly integrated with other Visual Studio Team System components such as version control, work-item tracking, testing and reporting.

Team Build is built on top of the Microsoft Build Engine (MSBuild). Team Build consists of the Team Build Service layered on top of the MSBuild build system. MSBuild is responsible for the build itself, while the Team Build Service is responsible for communicating with the TFS application-tier.

Features highlights:

  • Facilitates out of box build process for most the of .NET applications
  • 5 steps out of box end-end build process
    • Retrieve the code from Team Foundation Source Control
    • Compile, Run tests, static code analysis against compiled code
    • Release the builds onto a file server
    • Create/update work items appropriately
    • Publish the build reports
  • Supports executing build on multiple build machines without any script changes
  • Efficient retention policy to delete old builds
  • Locking mechanism to keep the build forever
  • Provides the capability for public and private or desktop builds
  • Provides APIs for extension
  • Ability to build projects/solutions in parallel fashion
  • Beyond build process, other parts of software lifecycle like packaging, deploy could be integrated with Team Build

Refer the high level Architecture here

Team Foundation Build logical flow:

Build types, build creation, execution and viewing results are largely wizard driven and straight forward – let’s move on to the actual Build Process flow

List of customizable Team Build Targets – If you wish to extend, tailor Team Foundation Build according to your project needs

For full information on this specific topic – have a look at this

BeforeEndToEndIteration BeforeCompileConfiguration
AfterEndToEndIteration BeforeCompileSolution
BeforeInitializeWorkspace AfterCompileSolution
AfterInitializeWorkspace AfterCompileConfiguration
BuildNumberOverrideTarget AfterCompile
BeforeClean BeforeGetChangesetsAndUpdateWorkItems
AfterClean AfterGetChangesetsAndUpdateWorkItems
BeforeGet BeforeTest
AfterGet BeforeTestConfiguration
BeforeLabel AfterTestConfiguration
AfterLabel AfterTest
BeforeCompile BeforeDropBuild
AfterDropBuild BeforeOnBuildBreak
BeforeCreateWorkItem AfterOnBuildBreak
AfterCreateWorkItem GenerateDocumentation

List of customizable Team Build Properties

Full information

CustomPropertiesForClean LabelName
CustomPropertiesForBuild OutDir
SkipClean UpdateAssociatedWorkItemsOnBuildBreak
IncrementalGet StopOnFirstFailure
IncrementalBuild SkipGet
CleanCompilationOutputOnly SkipLabel
SkipGetChangesetsUpdateWorkItems SkipInitializeWorkspace
SolutionRoot SkipInvalidConfigurations
BinariesRoot SkipPostBuild
LabelComment SkipDropBuild
CustomizablePublishDir SkipWorkItemCreation
CustomizableOutDir