Wednesday, May 30, 2007

Automated Testing of Silverlight Applications – PART I


Update (11/10/09): Record/Playback tool for Silverlight automation release.

One of the latest hot web development technologies on the market is Microsoft’s Silverlight framework. When Microsoft introduced WPF/e (known now as Silverlight Beta), the focus was on building rich media applications - similar to what you can do today with Flash - using a subset of their latest client development technology known as the Windows Presentation Foundation (aka WPF). Earlier this month, Microsoft introduced Silverlight Alpha which expanded the scope of Silverlight by introducing a light-weight CLR that allows developers to write managed code applications and host these applications inside the browser. To allow Silverlight applications to integrate well with current HTML/Ajax applications, Silverlight also enabled easy access to the HTML document DOM hosting these applications. Pretty cool!

As many in the development community ponder the feature set of this technology and how it can be leveraged in future development projects, us and
others in the software testing community are investigating how automated testing would work in Silverlight world! There are quite a bit of challenges to overcome especially with it only being an Alpha! But again, it never hurts to get a head start on looking at this technology from a testing prospective and providing Microsoft with feedback on how they can make our lives a bit easier when it comes to automated testing :).

When thinking about Silverlight applications and automated testing, there are a couple of different approaches that come to mind:

1. Isolated API/Unit Testing: This can be efficiently executed by running test cases inside Silverlight in a light host similar to what Jamie did with TestDriven.NET or what Visual Studio Team Test might provide in future versions. [Think of Visual Studio Team Test's ASP.NET testing support today where they allow tests to have access to the ASP.NET HttpContext. They would simply enable a Silverlight configuration where they provide a light-weight Silverlight host that tests can run in.]

This approach works well when:

  • UnitTesting the business logic of your Silverlight application.
  • When you have a pure windowless Silverlight application that standsalone and has very limited interaction with the HTML on the page.

This approach might not be suiteable for:

  • Heavy UI applications that perform heavy interactions with other HTML elements on the page. Especially when HTML elements trigger events in Silverlight Apps or when using Ajax on your pages.
  • Automating usage scenarios or end-to-end scenarios that require a browser host.
  • Testing your application in different browsers (IE/Firefox/Safari)

2. Browser Testing: Basically automating the browser by accessing the HTML DOM of the page including the XAML DOM of the hosted Silverlight app(s) and being able to execute actions and query states of these two DOM trees in a consistent and interchangeable manner. This approach is suitable for the scenarios described above that are not fit for Isolated API/UnitTesting.

Here at ArtOfTest, we’ve spent some time prototyping a few solutions that we think can help solve some of the challenges introduced by Silverlight. We are currently focused on approach #2 above since we think the progression of leveraging Silverlight is going to be through integration of small Silverlight apps within existing HTML/Ajax applications.

So, how are we going about supporting Silverlight?

Well, our web automation infrastructure WebAii currently provides a rich infrastructure for automating HTML based web applications including rich support for Ajax. One of our goals is to extend WebAii’s rich support to Silverlight. We want to extend our current API to be used interchangeably between HTML elements and XAML elements. This API should allow both action invokation and data retrieval for both element types (XAML & HTML). If we can accomplish that we can offer customers:

  • One consistent API for automating the HTML DOM and the XAML DOM.
  • Eliminate the need to learn a new automation framework for Silverlight and the need to integrate yet another new library into testing environments.
  • Enable a single test case to automate and interact with both the HTML & XAML elements on the same page to allow testing of the interaction points that mix HTML/XAML applications might have.
  • No breaking changes to existing testcases that are currently using WebAii. You can simply upgrade the library and then slowly update existing tests to include some Silverlight scenarios or start writing new automated tests for Silverlight within your existing testbed.

In my next post I’ll be digging into how WebAii’s Silverlight support will look like and discuss some of the current automation limitations with Silverlight and what Microsoft can do to help improve its testability.

Stay tuned!


Tuesday, May 22, 2007

Light-Weight ASP.NET Unit Testing - Go Browser-less, Server-less!

ASP.NET comes with great set of features especially for extensibility. Web Hosting is one of those features that is extremely poweful but not widely used since it's not really common for web developers to need to host their own ASP.NET processing server.

When it comes to testing, there is great power and agility gained with having a host processing requests within the test process. It allows us to eliminate the need for a browser and a webserver. It allows for an extemely quick and light-weight execution of tests that is suited for scenarios such as:

  • Running Build Verification Tests (aka BVTs): These tests commonly run as part of a continuous integration server process and they are very basic tests that ensure basic functionality and cover a broad set of features but do not go deep into each feature. BVTs can also be used during the dev check-ins process to ensure certain code quality at check-in. Having a quick turn around on these tests is important so they don't become a bottle neck for product building and development check-in.
  • UnitTesting business logic behind UI elements: If you want to write unit tests for your business logic that is tightly coupled with certain UI elements or UI logic.
  • Any other testing that does not require client javascript execution or a browser.

The Plasma project on codeplex attempts to accomplish that but is still at a very early stage in development. With WebAii Beta2 refresh, we updated WebAii to include a similar Asp.Net host and integrated it with our DOM parser, element identification engine and Actions interface. We enabled the scenario of executing ASP.NET Unit Tests using a light weight process without a need for a browser or a webserver.

The Actions object implements all the common actions like Click(), SetText(), SelectDropDown()..etc. which is the same interface you would use to automate IE or Firefox; so there is no new interface to learn or setups to perform. Simply flip a switch in your settings and you are automating without a browser or a server.

So what do you need to do to enable automation with WebAii using an in-process host? Simply set the defaultBrowser config setting to AspNetHost or set it in your code if you are not using a config:







Or set it in your test code:

settings.DefaultBrowser = BrowserType.AspNetHost;

And Voila! That is about it. You can automate your tests using the same Find.Byxx or Actions..xx and with no browser/server needed.

What about debugging?

With browser-less automation, you get the quick, snappy execution but you lose the visual rendering of your pages. But that is no longer the case with WebAii. If you want to visually view the responses coming back from the in-process server, simply set the setting:
enableUILessRequestViewing to true and you will get an IE instance that simply renders the response visually. You can set break-points in your code if the execution is going too fast to notice specific pages.

If you want to learn more about this feature, check out our documentation on the in-process host here. You can also download our complete samples here which include a sample using AspNetHost.