Introduction
Microsoft UI Automation is the new accessibility framework for Microsoft Windows. It addresses the needs of assistive technology products and automated test frameworks by providing programmatic access to information about the user interface (UI). In addition, UI Automation enables control and application developers to make their products accessible.
UI Automation provides programmatic access to most user interface (UI) elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI.
UI Automation provides full functionality in Windows Vista, Microsoft Windows XP, and Windows Server 2003.
Advertisement
Public Folder Watcher (PFW) is an add-on for Microsoft Outlook enabling support of notifications about changes in Exchange Public Folders and providing a status of unread email messages.
Who will use UI Automation?
The latest research study, commissioned by Microsoft and conducted by Forrester Research, shows that 57 percent (74.2 million) of working-age computer users (aged 18-64) in the U.S. are likely to benefit from using accessible technology. The research also confirms that the number of people who could benefit from accessible technology is increasing rapidly as the global population grows older and more people experience age-related difficulties and impairments related to vision, hearing and dexterity.
UI Automation is divided into two branches: UI Automation Providers and UI Automation Clients. UI Automation providers are applications such as MS Word, Excel, or any other program which can be used by users who need accessible technology support. UI Automation Clients are assistive technology applications, such as screen readers, screen enlargers,alternative input or others. Automated test scripts can use UI Automation for automated testing and are also considered clients in the UI Automation model.
How will this help assistive technology developers?
With current technology, assistive technology developers are required to use many different approaches to obtain and present information about the UI to computer users who rely on assistive technology products. These developers therefore spend an inordinate amount of time and resources gathering the UI information required for their application instead of spending this time innovating in their feature set. UI Automation reduces the cost for these developers by delivering the most complete accessibility model in the industry. It will become the primary source of UI information for the next generation of assistive technology products.
UI Automation Model
UI Automation exposes every piece of the UI to client applications as an AutomationElement. Elements are contained in a tree structure, with the desktop as the root element. Clients can filter the raw view of the tree as a control view or a content view. (These standard views of the structure can be easily seen using the UI Spy application). Applications can also create custom views.
AutomationElement objects expose common properties of the UI elements they represent. One of these properties is the control type, which defines its basic appearance and functionality as a single recognizable entity: for example, a button or check box. There are special helper classes and methods, that allow to obtain any UI element as AutomationElement object. So you can find any element you need by name, type, id, etc. walking through the tree.
In addition, elements expose control patterns that provide properties specific to their control types. Control patterns also expose methods that enable clients to get further information about the element and to provide input.
There is not one-to-one correspondence between control types and control patterns. A control pattern may be supported by multiple control types, and a control may support multiple control patterns, each of which exposes different aspects of its behavior. For example, a combo box has at least two control patterns: one that represents its ability to expand and collapse, and another that represents the selection mechanism.
UI Automation also provides information to client applications through events. Unlike WinEvents, UI Automation events are not based on a broadcast mechanism. UI Automation clients register for specific event notifications and can request that specific UI Automation properties and control pattern information be passed into their event handlers. In addition, a UI Automation event contains a reference to the element that raised it. Providers can improve performance by raising events selectively, depending on whether any clients are listening.
How to use this all?
All standard WPF controls support UI Automation automatically, custom controls must provide support for UI Automation through classes and interfaces. But you should bear in mind, that the implementation for Windows Presentation Foundation (WPF) elements and non-WPF elements (such as those designed for Windows Forms) is fundamentally different.
For deeper consideration of what we can do with UI Automation, there are two projects, which make it more obvious.
- UITree - Finds an application by title or by process ID, starting from a desktop as a root and builds its UI tree like UI Spy utility. Different types of view are available:
- Raw View. The raw view of the UI Automation tree is the full tree of AutomationElement objects for which the desktop is the root. The raw view closely follows the native programmatic structure of an application and therefore is the most detailed view available. It is also the base on which the other views of the tree are built. Because this view depends on the underlying UI framework, the raw view of a WPF button will have a different raw view than a Win32 button.
- Control View. The control view of the UI Automation tree simplifies the assistive technology product's task of describing the UI to the end user and helping that end user interact with the application because it closely maps to the UI structure perceived by an end user.
The control view is a subset of the raw view. It includes all UI items from the raw view that an end user would understand as interactive or contributing to the logical structure of the control in the UI. Examples of UI items that contribute to the logical structure of the UI, but are not interactive themselves, are item containers such as list view headers, toolbars, menus, and the status bar. Non-interactive items used simply for layout or decorative purposes will not be seen in the control view. An example is a panel that was used only to lay out the controls in a dialog but does not itself contain any information. Non-interactive items that will be seen in the control view are graphics with information and static text in a dialog. Non-interactive items that are included in the control view cannot receive keyboard focus.
- Content View. The content view of the UI Automation tree is a subset of the control view. It contains UI items that convey the true information in a user interface, including UI items that can receive keyboard focus and some text that is not a label on a UI item. For example, the values in a drop-down combo box will appear in the content view because they represent the information being used by an end user. In the content view, a combo box and list box are both represented as a collection of UI items where one, or perhaps more than one, item can be selected. The fact that one is always open and one can expand and collapse is irrelevant in the content view because it is designed to show the data, or content, that is being presented to the user.
Note: If you want to explore your own application user interfase, you must make all UI Automation calls on a separate thread.
- WPFStressTest - This is a nice sample that uses the UI Automation API's that come with WPF to stress test your application. Just run Stress.exe -pid [processId] or Stress.exe -title [windowTitle]. And see if your app can survive being taken through its paces over a 24 hour period.
UITree.zip (11,09 kb)
WPFStressTest.zip (17,26 kb)
Be the first to rate this post
- Currently 0/5 Stars.
- 1
- 2
- 3
- 4
- 5