It's been over a year now since we outlined our vision for supporting Silverlight test automation. We have looked into Silverlight automation from every angle and considered all approaches including UIAutomation. Our automation support for Silverlight doesn't really rely on UIAutomation but at the same time doesn't exclude it 100%. We will probably dedicate a full blog post about this topic and discuss how we are integrating UIAutomation and our reasoning for not relying on it 100%. But for now, let's just dig into our first preview release that will include Silverlight Test Automation support. Officially we are calling this release : WebAii's Silverlight Extension (CTP1) and the release will be available early next week. We are hoping this will be the first extension in a series of WebAii extensions that customers can pick and choose from that cater to specialized technology areas. (i.e. extensions for WebServices, WCF, Data …etc).
Getting started with Silverlight Automation in WebAii
To get started, let's first use a simple application and demo some basic automation scenarios like button click and set text. We will then expand into more advanced topics and scenarios.
Let's take the following simple Silverlight application that contains a button and a text box. When you click the button, it simply displays hello in a textblock on the canvas. The application looks like this:
To automate the application above to set the text, click the button and then verify the hello text in WebAii, we can simply write the following code using WebAii's Silverlight Extension (which currently resides under ArtOfTest.WebAii.Silverlight). Eventually we will be moving this into its own dll/extension:
Let's take a closer look at the code above line by line:
Line 319-323
This is basic WebAii code that launches the browser and navigates to a specific page.
Line 326
As soon as you include the Silverlight extension namespace in your test, (ArtOfTest.WebAii.Silverlight), the 'SilverlightApps()' extension method is added to the ActiveBrowser. The extension method offers you a list of all Silverlight applications found on that page. The extension method returns a list that has two accessors. An index accessor that takes an int (as shown above) and an string accessor that takes a hostid [The id of the html element that contains the <object> tag for the plug-in]. You can pick and choose what works for you. In this sample, given that we only have one Silverlight plugin on the page, we simply accessed the first and only one.
Line 329
As soon as you have an instance of a SilverlightApp, you now have full access to the application including the entire VisualTree. The app offers a short-cut method 'FindName', that can be used to search the application using an element name. In the above sample, given that the TextBox has a name associated with it, we can use FindName to locate it. The SilverlightApp object offers two versions of 'FindName': The first is a generic method that can return a strongly-typed object of the element (As shown above , i.e. a Button, a TextBox, a Grid…etc). The second is simpler and returns the base FrameworkElement object. We will discuss the strongly-typed object model offered in WebAii Silverlight Extension later in this post.
To go back to the line above, we are simply accessing the TextBox that has a name 'myName' and then setting its Text property to "ArtOfTest". Note that given we are accessing a property on the element, this is analogous to SetText on a DOM element in web automation. You are not really typing the text in the textbox but rather setting the Text property inside the application to that text. Our extension enables you to access and set properties directly against any FrameworkElement. If we wanted to simulate real typing of the text we would have written code like this:
app.FindName<TextBox>("myName").User.TypeText("ArtOfTest", 50);
Line 332
Same as above in terms of accessing the button. The difference here is that we are using the 'User' object to click the button. The User object is presented off all elements that inherit from FrameworkElement and offers real user interactions with Silverlight elements. So a Click off 'User' will actually move the mouse over that button an click it. That is exactly what this line does.
Line 335
Now that the button is clicked, we need to verify that the message is correct. The difference here is that we can't really use FindName to locate the TextBlock because the TextBlock doesn't really have a name. The SilverlightApp object offers extensive search support within the VisualTree beyond FindName() by utilizing similar approach to the Find object off the ActiveBrowser (that is used for HTML DOM searches) but adapted to specific XAML search scenarios.
In our code above, we are searching for a TextBlock (All Find.xxText() routines return TextBlocks) that partially contains 'Hello', hence 'p:Hello' (Similar to Find.ByContent in HTML search). Once we find the TextBlock, we then verify that the Text of that label is actually 'Hello ArtOfTest' using the Assert.IsTrue.
Now that we understand this sample, let's dig into some of the key objects available in WebAii's Silverlight Extension.
The SilverlightApp object
The SilverlightApp object is the main object you interact with when automating a silverlight plugin and is the entry point into the silverlight plugin. Below are some of the key properties/methods offered by the SilverlightApp object:
SilverlightApp.Bounds | Returns a Rectangle object that is the actual screen coordinates of this Silverlight-plugin. |
SilverlightApp.Content | Exposes the properties found on plugin.content. (i.e. ActualHeight/ActualWidth/FullScreen)
Note: All properties are currently read-only. |
SilverlightApp.Settings | Exposes the properties found on plugin.settings. (i.e. Background/EnableHtmlAccess/EnableFramerateCounter…etc)
Note: All properties are currently read-only. |
SilverlightApp.VisualTree | The entire VisualTree of this application. The VisualTree can be navigated up and down and the root element can be accessed using VisualTree.Root. The elements accessed from the VisualTree are FrameworkElement objects. |
SilverlightApp.Plugin | Returns an HtmlControl object that represents the actual HTML object tag of the plugin. |
SilverlightApp.Host | Returns an HtmlControl object that represents the actual html tag that the plugin object tag is contained in. Typically this is a Div tag if you are using Visual Studio's Silverlight designer. |
SilverlightApp.Find | Exposes a Find object used to search the VisualTree for FrameworkElements. This property is a shortcut to the VisualTree.Find object. |
SilverlightApp.FindName/FindName<T> | A shortcut to the Find.ByName & Find.ByName<> methods used to find an element using its xaml set name. |
The VisualTree
The VisualTree object exposed by the SilverlightApp is a hierarchal representation of the actual VisualTree of the Silverlight application at a specific point in time. If your Silverlight application changes, you can always refresh this tree by calling VisualTree.Refresh(). The VisualTree also exposes the VisualTree.Root property which is a FrameworkElement that represents the root element of the VisualTree of the app.
Note: To help eliminate any synchronization issues between the VisualTree available to the test client and the actual app running in the browser, every time the Find object attempts to find an element in the tree, it will always refresh before starting the search to ensure that the tree is up-to-date with the application.
Locating elements
For complex applications, locating elements is probably going to be one of the more time consuming areas where testers need to spend time understanding how to reliably find an element. Real Silverlight applications that we studied that rely heavily on control templates and databinding produce quite complex visual trees that contain elements that are not easily searchable by FindName() given the different namescopes created and the duplication of names within these templates. For example, check out the following application built by Telerik (http://www.telerik.com/demos/silverlight ) , the VisualTree for this application is quite complex.
The SilverlightApp.Find (Same object as VisualTree.Find) offers a rich set of search methods that enable you to not only find elements using their names but also by their text or type. Combined with type filtering, LINQ, FrameworkElement navigation (discussed below) and the Find.Allxxx routines, you get a pretty rich set of routines that enables you to find any framework element in the tree. That being said, this area is probably the area where we would really like lots of feedback in terms of additional search routines customers would like to see and scenarios that we need to support better or make easier. Here is a quick listing of the search routines available in this release:
Find.AllByName()/Find.AllByName<T> | Find all elements that have a specific name. Allows filtering on a specific control type. |
Find.AllByText() | Find all TextBlocks that contain a specific text. Use p:text to search for partial text |
Find.AllByType()/Find.AllByType<T> | Find all elements of certain type. i.e Button, Grid..etc. Filtering on type is inherit here. |
Find.ByName()/Find.ByName<T> | Same as FindName(). Search using a specific name. |
Find.ByText() | Returns the first TextBlock that matches the text provided. |
Note: All the generic find methods also have a none generic companion for cases where you are working with control types that are not available in the ArtOfTest.WebAii.Silverlight.UI namespace.
Find.Strategy
Unlike the ActiveBrowser.Find, the SilverlightApp.Find allows you to configure the FindStrategy that you wish to use when locating elements. By default we use the AlwaysWaitForElementsVisible which provides the most reliable approach to locating elements and eliminates any timing issues between the application and the test. So by default when we locate an element we will try to find it, if we don't find it, we keep trying until the element is available up to Find.WaitOnElementsTimeout. You can change this strategy at any point in time. Your options are:
FrameworkElement object
The FrameworkElement available in WebAii's Silverlight Extension is analogous to the FrameworkElement in Silverlight and is the base object that all visual elements and controls inherit from in the ArtOfTest.WebAii.Silverlight.UI namespace (discussed later below). For customers interested in our extensibility model, WebAii's Silverlight Extension does not expose a UIElement and therefore, our FrameworkElement can be viewed as the combination of Silverlight's UIElement + FrameworkElement.
The FrameworkElement and all objects that inherit from it get the following infrastructure methods and services:
VisualTree navigation: All FrameworkElements enable you to navigate the visual tree up or down. The object exposes methods and properties like Parent, Children, NextSibling, PreviousSibling, AnySibling<T>. The navigation also supports control sensitive navigation. For example, if you would like to find the parent 'Grid' that a certain text is contained in you do the following:
Grid containerGrid = app.Find.ByText("SomeText").Parent<Grid>();
OR
If you are working with a custom control that doesn't have a strongly-typed object under ArtOfTest.WebAii.Silverlight.UI, the navigation methods all offer a non-generic version that can be used to search for a certain type. For example, let's say you are trying to find the custom control "Bar" that contains some text, then you can do the following:
FrameworkElement barElement = app.Find.ByText("SomeText").Parent("Bar");
Parent, NextSibling, PreviousSibling & AnySibling all offer a non-generic versions in addition to the generic one.
User Interaction object:
The FrameworkElement object exposes a user interaction object. The object is exposed as a property named 'User'. User then exposes all the real automation methods like moving the mouse to click an element, type a text, mouseenter, mouseleave…etc. Here is a listing of all the methods exposed in this preview release
Note: One of the reasons why we moved these methods into their own object under 'User.xxx' is to help users differentiate between automation that is done by setting/getting properties vs real automation that does use the mouse or keyboard to invoke an action. That is one feedback we got on our design for the HtmlControls suite where customers didn't know which method actually used the real mouse vs direct DOM interaction.
Element coordinates: All FrameworkElements offer the ability to get both the relative coordinates of the element within the silverlight application and the actual coordinates in screen coordinates. The two methods are GetRectangle()
and GetScreenRectangle().
Note: Given that WPF/Silverlight applications allow for rich transforms for visual elements (i.e rotate, zoom….etc) and some elements like ellipses don't really conform to a rectangle per-say, our coordinate calculations will always return the largest rectangle that contains the actual element with its center right at the center of the element. For example, a GetRectangle on an ellipse will return this:
[Note the highlighting above is using the FrameworkElement.Highlight() method]
Scoped search within an element visual children.
Every single FrameworkElement also has a Find object associated with it that searches only this element's child visual elements. This object will prove very useful when getting into more complex application automation scenarios.
Example: Find the first StackPanel in the application that contains a specific text:
StackPanel panel = app.Find.AllByType<StackPanel>().Where(sp => sp.Find.AllByText("SomeText").Count > 0).First();
Visual Synchronization:
Silverlight applications and the richness they offer to developers in terms of scenario building, accentuate
the need for rich synchronization support within an automated testing framework. Any testing framework that wants to be considered as a serious and dependable test automation solution within the enterprise for Silverlight needs to offer rich support for synchronization between the application under test and the test being executed. For that reason, all FrameworkElement objects also expose a Wait object that is used to syncronize that element against specific check points. Currently our support is limited to Visible, VisibleNot. Future support is planned for event waiting, animation synchronization and much more. Again feedback on this feature from customers would be of great help.
Xml Representation:
Every FrameworkElement has a ToXml() method on it that returns the VisualTree of that element as an XML formatted string. This is very useful when trying to understand the visual tree at a certain state. [Hint: You can get the entire visual tree of an app by calling app.VisualTree.Root.ToXml()]
ArtOfTest.WebAii.Silverlight.UI namespace
The UI namespace includes all the strongly-typed controls like Button, TextBox…etc including their properties/methods and enums. The namespace also includes types used by these properties like Brush, SolidBrush…etc. In this preview we are only claiming support for simple primitive property setting/getting and enums. Later on, we will be supporting the ability to retrieve and set complex types like setting a SolidBrush on a Background or retrieving a specific Transform properties from an object. Although setting/getting these types of properties might seem outside the scope of test automation but these properties are going to be much needed to help with test case synchronization with that application and will make it much easier to craft verification points within an automated test. [i.e. Enabling scenarios where a test will need to wait until a specific storyboard has finished a specific animation or waiting/verifying that a specific trigger has been triggered or a certain DependencyProperty has changed…etc.]
Current limitations with this release:
- On Vista, if you are automating an external website, then make sure to add the url to your Trusted WebSites list in IE.
- Our automation targets Silverlight 2.0 (Beta2) and doesn't support any previous versions.
- The automation supports only .XAP deployment method.
- EnableHtmlAccess needs to be set to 'True' on your application.
- ArtOfTest.WebAii.Silverlight.UI does not contain controls in the System.Windows.Controls.Extended namespace. This will be available in future releases.
- There are known issues with Silverlight applications that attempt to perform webservice calls within the application. This issue will be resolved in future versions.
- This release does not support side-by-side installation with any previous versions of WebAii or Design Canvas.
- The application needs to be running off http:// not C:\...
- VS Debugging. Currently the experience of inspecting objects while broken in the debugger is a bit rough. You get lots of timeouts. This will be resolve in future versions.
What's next?
This is our first CTP preview for Silverlight automation. One of the goals of this release is to gather feedback from customers on our design and get specific feature requests that customers want to see implemented. That being said, we still have a set of features that will be implemented in future previews and beta releases of this extension to WebAii, specifically:
- More synchronization support within the Wait object for animation and events.
- Ability to trigger specific events on an element.
- Extended support for setting/getting complex types.
- Rich support for interacting with custom Scriptable objects.
- Debugging and Tracing support.
That's pretty much it for now. We are very excited about this release and look forward to your feedback. Feel free to send feedback directly to us: http://www.artoftest.com/contact.aspx
You can download CTP1 build from here.
