Setup 100% non-windows build-deploy-test flow from Tfs 2015

Tfs 2015 comes with many new build management features. One of my favorites is the new cross-platform build agent that they introduced and its open sourced at github. Feels like this capability unlocks huge opportunities for those who have invested in VS ALM stack. Already VS ALM offers best suite tools for Windows/Microsoft world of things. With this new cross platform build capability and revamped build authoring from web browser experience makes the job much easier.

Lets assume, I’ve a web app and want to deploy that to Nginx on an Ubuntu server. In order to coordinate Build-Deploy-Test flow, I’m going to use the vso-agent. This is 100% non-Microsoft stuff and lets see how it works together.

Setup a brand new Ubuntu box

I use Vagrant to spin up new base Ubuntu from Hashicorp.

I dont have NodeJs, Nginx or Npm installed.

Screen Shot 2015-07-29 at 7.39.54 AM

Setup Ansible playbook for app deployment

Assuming you are familiar with Ansible. I’m not an expert in this cool technology but it was too hard to pick up lately, very interesting one for orchestrating DevOps flow from environment provisioning thru monitoring. I got a simple playbook that will install NodeJs, Nginx then copy my static content over to the Nginx box, run post deployment xml/json transformation.

Below is the snippet, pretty simple self explanatory .yml file..

  • highlighted ones are the modules I use to install NodeJs and other prerequisites
  • task to copy the index.html
  • task to transform the config file using exising node modules

Screen Shot 2015-07-29 at 8.33.39 PM

Prepare_env module, just installs nginx and set of node modules required for my app

Screen Shot 2015-07-29 at 8.38.15 PM

Another simple run.sh to kick off the playbook (I’ve setup .ssh keys behind the scene and sudo password for remote installation is encrypted using ansible vault module and vault password is given for execution via –vault-password-file)

Screen Shot 2015-07-29 at 8.41.57 PM

Setup Tfs cross-platform build agents

Just follow the instructions here and from this walkthru video, you should be all set. Just run the agent interactively.

I’ve installed the agent on my mac, its up and running

Screen Shot 2015-07-29 at 8.49.46 PM

Setup Ansible on the build Agent

In order to run the deployments on Ubuntu box, I’m going to use Ansible. vso-agent will kick off the playbook and hence we need to install Ansible on the agent machine. In my case its the Mac book. Run “brew install ansible” to install Ansible.

Connect the dots

Now that we got all fundamentals covered, idea is to create a build that will pull my source code from tfs git repo, run Mocha tests (unit and integration tests) if successful deploy thru Ansible.

I use built-in tasks provided by the new build system.

Npm Install – to install dependencies

Screen Shot 2015-07-29 at 8.57.01 PM

Shell script that kicks off Mocha tests (really it’s one line and that is “npm test”. I dont know if there is a better way to kick off npm tests)

Screen Shot 2015-07-29 at 8.57.13 PM

Shell script that kicks off Ansible playbook that I mentioned earlier (run.sh above)

Screen Shot 2015-07-29 at 8.57.33 PM

Queue the build ..

Make sure to select the correct Agent, in this case not Hosted, its Default because I added my Mac vso-agent part of Default pool

Screen Shot 2015-07-29 at 9.04.53 PM

Running the build in my local vso-agent

Screen Shot 2015-07-29 at 9.11.10 PM

Seeing the Build status in VSO .. nice rolling log

Screen Shot 2015-07-29 at 9.05.15 PM

Screen Shot 2015-07-29 at 9.05.22 PM

Below output shows NodeJs, Nginx installation, copying files, transforming the sample .xml/.json config and finally starting Nginx..

Screen Shot 2015-07-29 at 9.46.03 PM

Screen Shot 2015-07-29 at 9.46.17 PM

So far, cross-platform build agent is very impressive capability, we might see some shortcomings interms of existing OOB tasks, like it comes with Gulp and if we want to run Grunt which is still widely being used, we need to wrap the grunt execution in some shell script and run that shell script. However, step in right direction to enable everyone to use VS ALM stack.

What to focus in API testing effort

Test Automation pyramid and related discussions offers enough education as to why GUI tests could become a beast of burden over time. It could risk one’s neck if GUI tests were written without considering the user intentions. Heard situations where regression tests were passing but the system isn’t fit for use because it doesn’t fulfil a meaningful workflow.. tests were barely checking whether UI elements were populated and populated with values. Isn’t the job of UI unit tests? There are plenty of guidance around BDD, designing and writing meaningful GUI tests.. and in addition to that, what I think will add more value and minify the burden of brittle GUI tests is more emphasis on API/service tests. These days, most of the products are built around service oriented architecture, offers RESTful or some sort of web services. Most of the UI is modern, developed for the web, decoupled from the business logic and talks to the web service to perform business logic. In these kind of situations, API tests are absolutely necessary. Even though, web services are private, known to my GUI only , it would add a lot of value testing thru web services, will reduce considerable overhead and heavy lifting that the GUI tests carry.

While there are plenty of tools around to help with API testing, I’m very impressed with Visual Studio Web Testing. Exploring other tools like Frisby Js and Yadda. More tooling specific write up will be out sooner, but for now, staying away from tooling and stressing more around the intentions. Tests with good intentions will yield better results, so, what needs be tested at the API layer that can offer more confidence?

Assume for every button click or tab out or equivalent action from GUI, your application calls out an API/web service to perform some business logic. GUI is updated based on web service call response. So, GUI needs to be tested for user interactions and GUI representation …data driven verification, thorough business logic verification can be offloaded to API testing. Also, API tests can be written while writing API code and need not wait for GUI to be ready., short list in my opinion, of course, there are more..

  • How do we send data – Headers, Query string parameters, payloads?
  • What headers are mandatory?
  • What parameters are mandatory? What is optional?
  • What if expected header information is not sent?
  • How payload is validated?
  • What is the content type for payload?
  • Is payload validated for types, null, or correct values?

One of my recent experience reminds the last point. One of the steps from the API test workflow made a call to an endpoint with appropriate headers and payload. In the business logic that call serializes the data and calls out to rules engine. We observed Rules engine crash and after 2 hours of debugging with the developer, we found that data serialization returned null and a call to rules engine with null caused all the trouble. Root cause was the API call did carry payload but not content-type for the payload.. although this may not happen as our GUI always calls out with appropriate headers, this leaves an easy whole to crash our system for intruders. Also, surfaces real code quality issues. It was evident that the entire class/controller implementation had no null, type check what so ever.. forgotten hero “Defensive coding” ..

Pluralsight course
“Defensive Coding in C#”.

Especially, API testing will help elevating confidence around Validation and Exception handling

  • Does input validation work?
  • Do we evaluate input data schema?
  • How do we handle missing parameters?
  • How do we handle wrong input?
  • Is API giving right error codes?
  • Does it issue appropriate error if wrong content type is requested?
  • Are these errors logged?
  • What if whole system or some part of the system is unavailable, how would it affect user?
  • What if system crash during transaction, how would it recover? What error would it give?’

Invariably, some of these checks will happen during some transaction from GUI and we would end up spending hours in analyzing and finding API issues.

Almost the time spent in proper API testing vs bug fix may be the same, however,

It’s about bug prevention vs bug fix,

It’s about Developer mindset, pressure while developing vs bug fixing under the gun,

It’s about more stable API tests vs brittle GUI tests,

It’s about test within the sprint and gain confidence vs defer GUI tests for subsequent sprints and building Tech Debt

API testing is certainly valuable and worthy investment

Extend Visual Studio Web Test framework to extract JSon value

Previous post hinted about how Visual Studio Web Test framework can be used to perform Service/API level tests. These type of tests will come handy to gain more coverage, hit those end points that your GUI hits but without depending on the GUI, drive end-end or system tests with data mix combinations etc

Visual Studio Web Performance Test capability has been there for a while, mainly used in performance and load testing. However, with some tweaks, we may use the same framework to test our APIs.

First requirement could be, VS web tests handle HTML responses from the call. APIs calls typically return xml or Json responses. How to handle this? Pretty simple, VS web test framework is extensible, here is the API documentation.

Below is a sample gist how I created a custom extraction rule to extract JSon node value for my API test. Using Newtonsoft Json library, JToken.parse to parse the Json response..nothing much to explain..

 

using System.ComponentModel;

using Microsoft.VisualStudio.TestTools.WebTesting;

using Newtonsoft.Json.Linq;

 

namespace My.TestingComponents.WebTestExtensions

{

[DisplayName(“Extract Json node value”)]

[Description(“Extracts the value of a Json node from the response”)]

public class CustomExtractJsonNodeValue : ExtractionRule

{

 

[Description(“Json path to extract value with”)]

public string JsonNodeName { get; set; }

 

// The Extract method. The parameter e contains the web performance test context.

public override void Extract(object sender, ExtractionEventArgs e)

{


var o = JToken.Parse(e.Response.BodyString);

string propertyValue = (string)o.SelectToken(JsonNodeName);

if (propertyValue != null)

{

e.WebTest.Context.Add(ContextParameterName, propertyValue);

e.Message = “Successfully added ” + propertyValue + ” to the context parameter ” + ContextParameterName;

e.Success = true;

}

else

{

e.Message = “Couldn’t add ” + JsonNodeName + ” to the context parameter ” + ContextParameterName;

e.Success = false;

 

}

}

}

}

Test Automation Pyramid and TFS

In last post, there was a little bit of grousing about Test Automation, need for approaching it strategically and side effects if Test Automation == GUI automation.

Also, there was another post talking about Quality practices for Agile teams..

My intend here to connect them and share how I would approach the Test Automation strategy for a .NET project using Visual Studio stack of tools

If I approach the entire Quality engineering with different perspectives that Janet Gregory & Lisa Crispin explained in their book, below diagram will be my goal

 

 

In that, Test Automation plays significant role in Quad 2 and hypothesis being a .NET team, VS ALM will help them to be up and running in no time. Beyond coverage on Test Automation, tight integration with other ALM capabilities are a great plus.

 

 

Additionally, on the Quad 4 tests such as Performance and Load testing, Visual Studio offers native support for local execution. Any of the tests written already can be reused. Also, Visual Studio Online offers support to execute load tests from the cloud.

Test Automation Pyramid in reality

Test Automation Pyramid?

Book Succeeding with Agile: Software Development Using Scrum coined the importance of approaching the Test Automation in a systematic way, do’s and don’s..

In essence, the idea here is approach Test automation in a meaningful way that offers ROI.

Sadly, in many places, Tools steal attention when it comes to automation. Precisely, those GUI automation tools steal much attention as those places got Testers who prefer to sit on top of the application and just automate to test the functionality. Those who hate record and replay tools, get obsessed with other set of tools which programmatically talk to the web browsers and other GUI technologies and help automating GUI tests.

While these approaches offer great value compared to manual testing, the real road to success could be in potential risk and raise the need to look at Test Automation strategically.

For example, awareness towards Unit testing which tests the lowest level unit of the product does exist, many frameworks exist and teams invest time in writing unit tests. While the quality is relative, there are other tools like Sonar could help injecting thoughts around continuous inspection and emphasis on continuous quality inspection vs one time quality audit. In reality, knowingly/unknowingly many teams extend their unit tests to be functional tests and suffer from side effects.

On top of Unit tests, many teams just right into GUI automations. There are zillion tools out there to help with record and replay, efficient techniques to spy the GUI, automatically spin up different browsers and run tests.

Pyramid becomes a cone..adding the manual test effort on top, it’ll look like an ice-cream cone

So, what do we do, how to get back to basics and align with the Pyramid.. First of all, Is the Pyramid important, meaningful, does one size fit for all? The book definitely offers answers for most of your questions. In my opinion, aligning to Pyramid will increase the ROI, reliability, trust and help hitting go to market timeline. Approaching test automation strategically is mission critical to gain the best of out of automation which could increase the possibility of delivering value to our customers.

Too heavy GUI automation might work initially. Over time, when product grows, functionalities gets changed for a business goal, GUI automation will require lot of attention, rework to make sure test failures make sense. Because these automations depend on the GUI, these tend to break when a simple change occurs on the GUI and needs careful coordination between team members to deliver value. industry standard GUI automation tools don’t offer the capability of automated detention and correction when something changes in GUI. GUI automation needs completed GUI and hence they always lag behind, sometimes, n-2 or 3 sprints behind the code. When automating functionalities lag behind, late feedback to the development team is a biggest challenge. Another critical side effect could be that end-end test is at great risk, below picture says the rest.

 

 

Quality practices for Agile projects, a different perspective

Almost everything has evolved, reshaped and been taken to a different level in software development. It’s highly likely that we are no more approaching the product development/software development the same way that we would have been 10 years back. A lot of enterprises embraced the Agile movement that guided and emphasized the left over right.

At a high level, we are developing in short cycles, adding value in continuously, demonstrating value addition to the stakeholders, being receptive to their candid feedback.

That being the drive at high level, internal engineering practices went thru vivid changes notably

Silo functional team members vs cross functional team members

Managers planning vs whole team planning games

Weekly status reports vs daily standups

PUSH vs PUSH work units

Large big bang release vs shorter incremental releases and supporting practices

UAT testing at the end vs continuous usability testing

One Performance at the end vs continuous performance testing

and so on..

Plenty of changes have been forced at the engineering level, these are not mere changes in terminologies and ceremonies but require significant shift in mindset.

While there are plenty of motivation to transform the teams and help adopt Agile practices, most of them tend to be focusing on developer practices such as Whole team planning, Test Driven Development, Simple design, Pair programming, Continuous Integration, Continuous delivery etc, where do the Quality engineering practices fit in? no more specialist testers, testers learn to develop code, do developers learn those needed testing skills? What do we do to prevent bugs, detect bugs early in the cycle and how do we transform those traditional quality engineering practices into Agile, what would be the role of a tester in an Agile team..many more questions..

If you have questions somewhere along the lines, please read Agile Testing by Janet Gregory. They talk about a lot of good practices can be adopted by Agile teams to build quality in.

Amongst many great thoughts, one of the most interesting factor is that Quality != Testing

They remind the Underlined goal that ultimately we need to make sure our implementation meets our intention which meets the business need on a continuous basis.

Q

This is a wonderful goal, how do we accomplish that..looks like we need 360 degree check on everything we do? How can we do this with every sprint?

Further reading on that book will guide us towards achieving this goal. However, some highlights that I like

Quality is about making sure that the team shares common understanding about the problem space. Team collaboration, ask right questions to the PO and understands before committing. This is something not new, what quality has to do here? Correct question but the emphasis here is to think beyond a story during planning, think like an eco-system and how this little story could affect the eco system, think like this little story is part of the bigger puzzle board and the importance of making it right, think about chain reactions that this little change could trigger and try to identify those end-user workflows that this story contributes or affects and hence derive those testing scenarios. So at the end of the planning meeting, the entire team would have talked about the story, especially, beyond design and coding, understood how an end-user’s action could cross this functionality, what an end-user would do before and after and essentially help the whole team to understand the problem space. When time comes, it’s highly likely that someone will write code and someone else might write tests but towards the same goal. Potentially, tests can be designed, written even before the code is ready. Upfront Test design and test case development could potentially help the developer and prevent bugs. Chances of finishing the story with desired quality within the sprint is high rather than deferring the test automation to the subsequent sprint. Of course, Automation plays a role here but Automation with appropriate intend plays a greater role.

These are partly my takes from that book from Quadrant 2. There are other Quadrants

Quad 3 – talks more about the importance of applying quality practices from Customer/end-user perspective and asking hard questions to your product

Quad 4 – talks about how Tools and technologies helps to apply quality practices from customer/end-user perspective

We started practicing Quadrant 2 in some of our projects and seeing great value..

Why TFS Backlog Management is an exciting tool for Product Owners

Product Owner is one of the key stakeholder in Scrum. Part of their responsibility is to have and demonstrate clear vision of what they want to build and why they want to build. That is the part where the Product Owner demonstrates the value of connecting the “Need” with “Intention” and which will be “Implemented” by the technical team. Other side of the Product Owner could be that they are the elite user of the product who wears multiple hats like

someone from marketing

someone from product management

someone with a solid understanding of users

someone with solid understanding of the market place and competition

someone with clear vision of future trends in that domain

and the list goes on..

So when they write requirements/features/user stories, many of the above factors play significant role. Not only in writing the story, they have the birds eye view, know what market wants, know what will give an edge in competitive market then order/prioritization features and stories accordingly. This part is critical. Typically, a product or product suite has many applications and features. Several Product Owners work in various parts of the product(s), however, there are high chances that they are in some silo focusing on their part with limited exposure to the larger eco system. Often end up developing parts in silos and late integrations that has high chances of impacting quality, goto market and integrations don’t work the way they were intended…inturn the implementation doesn’t align with the need…and you guess the rest…

What am I saying here – a tool can fix all these? Hmm, not really. However, would it be nice to have a tool that offers solution to some of these? TFS 2013 helps here and atleast 10 good things that come to mind

  1. Birds eye view of objects
    1. Goals à Features à Stories à Tasks/Issues/Bugs etcc
  2. Ability to pick and choose stories that meet minimum viable product
    1. Goal – Release x meeting minimum viable product with
      1. Feature 10
        1. Story 90
        2. Story 100
        3. Story 45
      2. Feature 22
        1. Story 78
        2. Story 33
      3. Feature 4
        1. Bug 100
        2. Bug 13
  3. Capabilities supporting prioritization and ordering at any level
  4. Capability to engage multiple teams to get the work done
    1. Teams have their own Product backlog (subset of Product backlog that contains Backlog Items pulled by the team)
    2. Teams have their own Sprint backlog (subset of Team’s product backlog that they committed on a given sprint)
  5. Ability to track the progress at any level
    1. Automatic aggregation of data from teams upto Product – example, Product X showing aggregate burn down from Team Blue and Team Red belongs to that product
  6. Capability to organize POs work such as grooming etc without troubling the teams
    1. Kanban board at each level to organize POs work – example, at product backlog level for Product X, PO can have New à Grooming à Groomed à Active à Done whereas these won’t bother Team’s workflow..
  7. Sprint management capabilities – although POs don’t mind how team’s work internally, it’d be nice take stab at it once in a while..
    1. Team level resource planning, capacity planning for a given sprint
    2. Task breakdown of sprint backlog items
    3. Task boards to manage daily standups
    4. Lightweight charts – create charts as needed easily in clicks and pin them for the team in home page
  8. Inbuilt Feedback mechanism – POs can explore the product and provide actionable feedback using the feedback tool. The same tool may be used to gain feedback from real end users as well and those feedbacks can be rated and shall converted to a backlog item in the Product Backlog or shall be converted to be a bug
  9. Consumable stories with storyboards
  10. Commitment and roadmap – A product with commitment from the makers and clear roadmap towards SAFe adoption

Mostly all are lightweight browser based interfaces without having to install anything – that means PO can be in any platform, as long as the OS offers a browser, everything should work

 

Further readings

Gartner Magic Quadrant for ADLM

http://blogs.msdn.com/b/dannawi/archive/2013/11/26/microsoft-leading-position-in-new-gartner-magic-quadrant-for-adlm.aspx

Manage backlogs across multiple teams

http://msdn.microsoft.com/en-us/library/dn306083.aspx

Get and Manage Actionable feedback

http://msdn.microsoft.com/en-us/library/hh301769.aspx

PowerPoint Storyboarding

http://msdn.microsoft.com/en-us/library/hh409276.aspx

http://www.incyclesoftware.com/2012/10/easy-storyboarding-visual-studio-2012/

Lightweight Charts

https://tfsrocks.wordpress.com/2014/02/15/what-can-we-expect-in-tfs-2013-update-2/

 

How to export Test Artifacts from TFS

TFS 2013 Update 2 (CTP 1) brings in another cool update. Till now, we have been using Test Scribe to export Testing related artifacts from MTM. With the latest update, some of those artifact export features are part of the product. From the web interface, we should be able to export the whole Test plan and Test suites. Artifacts can be exported as email or printed and configured to print as PDF or to OneNote or whatever you prefer…

Below is my sample Test plan and suites

Export at the Test Plan level

Export at the Suite level

You will see another dialog asking for some inputs to export appropriate content

 

  1. What to export from “Test Plan”
  2. What to export from “Test Suite”
  3. Where to export – to “Email”
  4. Where to export – to “Printer”

Though these are pretty simple options, output offers moderately attractive data

Information about the whole plan.. suite hierarchy (the ID next to the plan is not work item id..)

 

Test case and associated Requirement and Parameters

 

Although, we don’t get the Test runs, execution summary, results and other things that Test Scribe offers, those might be on the way in next update..

What can we expect in TFS 2013 Update 2

Earlier this month, MS released VS/TFS 2013 Update 2 CTP 1. Although, there was a VS 2013 Update 1, TFS – the ALM platform didn’t get any update. However, this time, 2013.2 carries pretty good list of interesting items. This update can be downloaded from http://go.microsoft.com/fwlink/?LinkId=390521

Some of the very useful features are being release with this update.

Tags

Tagging is one of the cool and widely used features. With current versions of VS/TFS, user can apply Tags only from the web interface.
And system offers the capability to filter work items based on Tags. These are good and bare minimum features. If a user wants to perform next level of tasks like add Tags part of another complex query, that is not possible. 

However, with 2013.2, Tags can be a part of any type of work item query.
Below, I’m editing a “Tree of work items” query and adding Tags clause

Tags can be applied from both Visual Studio and Web UI

Additionally, if we want to apply some security around Tags, that is possible too. This will become very handy, allows the team to manage their Tags cloud by controlling who can apply the Tags.. Tags stay reasonable and standard. This can be controlled at Team Project level or at individual Team level as well.

 

Lightweight Charts

 

Although Charting based on query was introduced with VS/TFS 2013, there are couple of cool features got added. Now, we can manage colors within the charts and pin the charts back to the team home page. So, your home page need not be decorated with just numbers anymore. The gotcha here is Charts can be created only for Flat list query and not for any other type of query. This doesn’t change in this update.

Below I’m creating a quick chart from a query and pinning that to my home page..

 

I can change colors within the chart as appropriate

 

 

Make sure your query is part of Shared Queries

 

 

Once its in Shared Queries, you should see an option to “Pin to homepage”

The same option shall be set from the chart as well

 

 

And I’m here on the Home page and seeing Charts …

 

There are some more enhancements on the Testing and Git version control side as well..