Automation Design Canvas 1.0 : Test Failure Resolution

One of the key features that we are proud of in Automation Design Canvas 1.0, is the designer's ability to help testers maintain and fix their automated tests. Automated testing is not only about how fast you can record a test but also how fast you can analyze and resolve a test failure. Some of the key features included in Automation Design Canvas 1.0 are:

1. Visual State Capturing: As you record each step, the designer captures the browser state at that point in time with the target element highlighted on the surface. Each visual step is then added to the story board so that you can visually go back to it. Each visual step is also tied to the actual step, so as you re-order your steps and move them around or delete them, your visual story board of your test is always up to date with your latest changes. The goal of this feature is to help you understand where elements were at the time of recording and what the entire page looks like at that moment in time. In some cases, these visual captures can help you pin-point a failure by simply looking at the captured images.
   
2. DOM State Capturing: One thing we realized too is that capturing the visual state is not enough. In many cases, the failures are due to issues like ID changes or attribute value changes on HTML elements and such failures rarely exhibit any visual change. You actually need the DOM state at that moment in time to help you understand what changed in that DOM.
   
3. DOM Tree Diff View: When loading a failure in the Automation Design Canvas, the DOM Tree tool window enables a "DOM Diff" view that you can be use to help understand the differences between Recorded & Executed DOMs. Such a side-by-side view is an enormous help in allowing you to inspect property or DOM structure changes between when the step was recorded vs its execution.
   
4. DOM Element Index View: Each element in the DOM Tree is prefixed with its TagIndex. Combined with the Diff View discussed about, this gives you a fast path to detected and match two DOM nodes between the recorded and executed DOM trees.
   
5. Live Validation: For both Element Identification & Verification Rules. As you are investigating your failures, you can easily edit the failing rule for identification or verification and validate it against the executed DOM.
   

To help us demonstrate these features, lets take a simple example of a common failure for web automation: Element Id Changes. Let's start by taking this HTML Page:

As you can see, the input button has a "btn1" id associated with it. Let's load this page in Automation Design Canvas 1.0 and record the button click. The Automation Design Canvas will determine that the best method to identify this element on the page is using its id (btn1) given that the ID is currenly set on the page and is unique. For more information about how Automation Design Canvas builds identifications for elements including how to customize them, check out our help topic here.

Here is what the FindParam editor looks like:

If you run the test, obviously everything should pass…

Let's now say a few weeks later, the developer of this application decides to change the button's ID from "Btn1" to "Btn2". Let's go ahead and make that change in the page and run the test again:

Now that we are at this stage, let's see how Automation Design Canvas, can help us resolve this issue. To debug a failure in Automation Design Canvas, we will simply double click on the red x right next to the failing step. In design canvas, you get debug capability at the step level. When clicking on the red x, the Automation Design Canvas will open up the Debug UI for you. Let's take a close look at what that UI looks like and all its pieces:

If you open up the DOM Explorer at this point, you will notice that it also has a drop-down similar to the one listed above in the "Actions" section:

The first thing that comes to mind when debugging such a failure is what was the state of the DOM when I recorded my test.

Why can't my test find the "btn1" id. [As a tester, you might not be aware that this ID has changed]. Let's switch the Target DOM to "Recorded" and then click on the "Validate" button to see if I can still find the element in the DOM when I recorded it.

As you can see, the designer was able to find this element against the DOM that was captured at the time of recording. If you actually pull up the DOM Explorer and switch it to "Recorded", you will see that element highlighted in the tree:

We now atleast know that the FindParam used to identify this element is correct and that the failure must be due to a change in the page, not a problem in how we are identifying it. The next question to answer then, what changed and what is the new value for that change. To help identify that, let's switch the "View" of the "Recorded" dom to "TagName" so that we get a flat list of each tag with its index that looks like this:

Next we are going to compare this element with the actual "Execution" dom that the test failed again. To do so, let's simply switch the "Displayed DOM" to (both). And switch also the "Execution" "View" to "TagName". The DOM Explorer now displays both recorded and executed DOM side by side:

Now that we have both DOMs side by side, we can easily navigate the Execution DOM to the INPUT tag with index "1".

Ah!

Now we see that this element's id is btn2 not btn1. At this point there are two ways to proceed:

1. This is a bug in the product and should be reported.
   
2. This is an acceptable change and we simply need to update our automated test.
   

To update our automated test, simply go to the editor UI shown above and change the btn1 to btn2. Once you make that change, you might want to validate it against the execution DOM. Simply select "Execution" in the Target DOM drop down in the editor and click on the "Validate" button. The element should validate correctly and you are good to go at this point.

Click OK on the debug UI and you should be able to run the test again and it should pass.

Note: As soon as you click OK, the old recorded DOM is discarded for the Execution DOM since this is the DOM that the element has validated successfully against.