Tuesday, July 29, 2008

Browser-agnostic color verification using WebAii & Automation Design Canvas

Color style verification in WebAii and Automation Design Canvas is one of those nice-to-have features that saves testers some cycles especially when trying to make an automated test run cross browser (specifically in IE and Firefox). If the application you are testing uses style sheets - which most of today's rich web applications do – there is a good chance you might have to verify a style on a specific tag or that a specific tag has changed color based on its cascaded style sheet. WebAii and Automation Design Canvas offer specific functionality to help address the color formatting for computed styles.

Let's first take at look at why computed styles for color verification causes problems for testers:

The issue is related to how IE and Firefox return colors when it comes to computed styles:

In IE, computed styles accessed using (element.currentStyle) return color (i.e. backgroundColor, foreColor) using the following format : #xxyyzz

FireFox on the other hand always returns the color in the following format: rgb(x,y,x)

Typically , when writing automated tests for such a scenario (that you wish to run on both IE and Firefox), you end-up having to fork your code based on each browser type and do the validation using the specified format.

Well not anymore…., let's take a look at what WebAii 1.1 offers in terms of support in this scenario:

WebAii 1.1 Framework

There are several built-in features in WebAii framework that address this scenario:

  1. HtmlStyle object:

    If you notice, our HtmlControl.GetComputedStyle() returns an HtmlStyle object, not a string value. The GetComputedStyle() function retrieves the style from the browser DOM and places it in the HtmlStyle object for further processing. If you wish to get the value immediately, you always call the HtmlControl.GetComputedStyleValue() which gets the actual style value as a string.

    The HtmlStyle object offers the following facilities to make style validation simple and more straigh-forward for colors:

  • HtmlStyle.IsColor(): This call checks to see if the style returned is a color or not. If for any reason you have code that generically processes styles, you can use this to distinguish between color styles and other styles
  • HtmlStyle.ToColor(): This method is very powerful and is where we abstract out the formatting between IE and Firefox. This method simple parses the value out for you and returns a System.Drawing.Color object that you can then use to validate against another System.Drawing.Color you might want to check against.

For example:

The following line of code can be used to validate that the background computed style color for a div is red. This line of code works for both IE and Firefox:

Assert.AreEqual<Color>(Color.Red, myDiv.GetComputedStyle("backgroundColor").ToColor());

  1. AssertStyle() extensions.

    In WebAii 1.1, we introduced verification extensions to help facilitate HTML specific validation. The AssertStyle() class takes advantage of the above functionality to abstract out the browser formatting specifics and make color verification completely seamless.

    For example:

    The following validation will assert that the backgroundColor style on the Div tag is red regardless of the browser used.

    myDiv.AssertStyle().ColorAndBackground(HtmlStyleColorAndBackground.BackgroundColor, "Red");

Automation Design Canvas 1.0

In Automation Design Canvas 1.0 (ADC), we consume the above functionality inside the style verification sentences and we make it completely straight forward to craft and execute color verification. In ADC, we internally persist colors in #XXYYXX format and always covert (named/rgb/#xxyyzz) colors to the #XXYYZZ format. Using that format comparisons are done. Here is how a color style verification looks like in ADC 1.0:

For example, if you invoke verification to validate the forecolor of the following text on msn.com home page:

The verification sentence builder will automatically resolve the color to its #XXYYZZ format as shown below.

Definietly a craft and forget color validation. J.

Tuesday, July 22, 2008

Automation Design Canvas 1.0 : Test Failure Resolution

One of the key features that we are proud of in Automation Design Canvas 1.0, is the designer's ability to help testers maintain and fix their automated tests. Automated testing is not only about how fast you can record a test but also how fast you can analyze and resolve a test failure. Some of the key features included in Automation Design Canvas 1.0 are:

  1. Visual State Capturing: As you record each step, the designer captures the browser state at that point in time with the target element highlighted on the surface. Each visual step is then added to the story board so that you can visually go back to it. Each visual step is also tied to the actual step, so as you re-order your steps and move them around or delete them, your visual story board of your test is always up to date with your latest changes. The goal of this feature is to help you understand where elements were at the time of recording and what the entire page looks like at that moment in time. In some cases, these visual captures can help you pin-point a failure by simply looking at the captured images.
  2. DOM State Capturing: One thing we realized too is that capturing the visual state is not enough. In many cases, the failures are due to issues like ID changes or attribute value changes on HTML elements and such failures rarely exhibit any visual change. You actually need the DOM state at that moment in time to help you understand what changed in that DOM.
  3. DOM Tree Diff View: When loading a failure in the Automation Design Canvas, the DOM Tree tool window enables a "DOM Diff" view that you can be use to help understand the differences between Recorded & Executed DOMs. Such a side-by-side view is an enormous help in allowing you to inspect property or DOM structure changes between when the step was recorded vs its execution.
  4. DOM Element Index View: Each element in the DOM Tree is prefixed with its TagIndex. Combined with the Diff View discussed about, this gives you a fast path to detected and match two DOM nodes between the recorded and executed DOM trees.
  5. Live Validation: For both Element Identification & Verification Rules. As you are investigating your failures, you can easily edit the failing rule for identification or verification and validate it against the executed DOM.

To help us demonstrate these features, lets take a simple example of a common failure for web automation: Element Id Changes. Let's start by taking this HTML Page:

As you can see, the input button has a "btn1" id associated with it. Let's load this page in Automation Design Canvas 1.0 and record the button click. The Automation Design Canvas will determine that the best method to identify this element on the page is using its id (btn1) given that the ID is currenly set on the page and is unique. For more information about how Automation Design Canvas builds identifications for elements including how to customize them, check out our help topic here.

Here is what the FindParam editor looks like:

If you run the test, obviously everything should pass…

Let's now say a few weeks later, the developer of this application decides to change the button's ID from "Btn1" to "Btn2". Let's go ahead and make that change in the page and run the test again:

Now that we are at this stage, let's see how Automation Design Canvas, can help us resolve this issue. To debug a failure in Automation Design Canvas, we will simply double click on the red x right next to the failing step. In design canvas, you get debug capability at the step level. When clicking on the red x, the Automation Design Canvas will open up the Debug UI for you. Let's take a close look at what that UI looks like and all its pieces:

If you open up the DOM Explorer at this point, you will notice that it also has a drop-down similar to the one listed above in the "Actions" section:

The first thing that comes to mind when debugging such a failure is what was the state of the DOM when I recorded my test.

Why can't my test find the "btn1" id. [As a tester, you might not be aware that this ID has changed]. Let's switch the Target DOM to "Recorded" and then click on the "Validate" button to see if I can still find the element in the DOM when I recorded it.

As you can see, the designer was able to find this element against the DOM that was captured at the time of recording. If you actually pull up the DOM Explorer and switch it to "Recorded", you will see that element highlighted in the tree:

We now atleast know that the FindParam used to identify this element is correct and that the failure must be due to a change in the page, not a problem in how we are identifying it. The next question to answer then, what changed and what is the new value for that change. To help identify that, let's switch the "View" of the "Recorded" dom to "TagName" so that we get a flat list of each tag with its index that looks like this:

Next we are going to compare this element with the actual "Execution" dom that the test failed again. To do so, let's simply switch the "Displayed DOM" to (both). And switch also the "Execution" "View" to "TagName". The DOM Explorer now displays both recorded and executed DOM side by side:

Now that we have both DOMs side by side, we can easily navigate the Execution DOM to the INPUT tag with index "1".


Now we see that this element's id is btn2 not btn1. At this point there are two ways to proceed:

  1. This is a bug in the product and should be reported.
  2. This is an acceptable change and we simply need to update our automated test.

To update our automated test, simply go to the editor UI shown above and change the btn1 to btn2. Once you make that change, you might want to validate it against the execution DOM. Simply select "Execution" in the Target DOM drop down in the editor and click on the "Validate" button. The element should validate correctly and you are good to go at this point.

Click OK on the debug UI and you should be able to run the test again and it should pass.

Note: As soon as you click OK, the old recorded DOM is discarded for the Execution DOM since this is the DOM that the element has validated successfully against.

Thursday, July 10, 2008

Powerful DOM Search & Validation using WebAii 1.1 & LINQ

One of the most powerful new (and probably least discovered) features of WebAii 1.1, is its expanded search support using LINQ. LINQ is Microsoft's latest technology introduced in .NET 3.5 that offers a highly rich and integrated query language inside .NET languages like C# or VB.NET.

It was relatively easy for us to realize how we can leverage the power of LINQ in WebAii. LINQ's most common use is to enable a rich syntax to query structured information and we thought it would serve a great purpose in also querying the DOM. Given that WebAii offers a 100% managed HTML DOM to the testcase, users can harness the full power of LINQ with their DOM Searches.

We first started by adding support to do LINQ queries on the Element object. So we introduced the Find.AllElements() that allows you to query the entire DOM tree using a LINQ expression.

For example:

Such powerful searchs makes it a lot easier and concise to do rich searches. Although this was a great enhancement to our search routine, we felt there is still more to be done. Most users are actually interested in querying a certain type of Html elements and not all of them. For example, if they want to query all tables, they will need to add another clause in the Where() clause that says element.tag == "table". In addition that, we realized that some of these searches might also want to leverage the strongly-typed HtmlControls suite that we have in WebAii. The HtmlControls suite offers a lot of rich context specific information regarding a specific Html tag. (i.e. Table's row count or even cell count). So if you are, for example, trying to use the Find.AllELements() to validate that all tables on a page have alteast 5 columns, you might have to write some pretty complex 'where' clause that defeats the purpose of making search easier using LINQ.

With that in mind, we decided that we definietly need to enable html filtering as part of our search API. We also thought that it would be great if we can make such filter based on a strongly typed HtmlControl type. For that purpose, we introduced the Find.AllControls<T>() function. This function is very powerful. I will use the rest of this blog post to go over some samples that use it.

Let's start with something simple. Let's say you simply want to get all the Images on a certain web page. With this API, you can simply do:

As you can see the Find.AllControls<T> already has all the information you need to get you the right set of images on the page. You can then loop through those images to do any validation or actions on them.

Let's take another example that leverages the rich API on our HtmlControl. Let's say you want to validate that all images on the www.gettyimages.com home page are actually loaded properly. In WebAii 1.1, we introduced the HtmlImage.IsLoaded() function that queries the DOM to validate that the image has actually loaded in the browser and is not a broken image. You can perform this validation in one line of code like this:

Very Nice, eh? Let's have some more fun with this API. Given that we are in elections season, let's say we want to go to Google's Elections News website and highlight all links that contain "Obama" with blue and "McCain" with Red. [Sounds like a CNN application to see who is getting more press J]. Here is the code that does this for us using WebAii 1.1:

If you have WebAii running, place the above code in a TestMethod and run it, looks pretty cool.

Ok, last, let's take a bit more complex validation scenario. Let's say we have a testcase that wants to validate that an ASP.NET DataGrid when put in edit mode, changes all the cells in Column 1 from labels to actual TextBox. How can we validate that?

We basically need to validate that:

  1. All column 1 cells have one TextBox.
  2. All column1 cells have only that textbox in that cell.

Here is the code using LINQ and Find.AllControls<> that does this for us:

That's it for now. We would love to hear your feedback about this feature or any other features in WebAii. We are in the process of revamping our online documentation to be more up to date with the WebAii 1.1 release. We are also starting an new effort to start blogging more about WebAii features and our Automation Design Canvas!!