Testing GUIs with Abbot and CostelloElliotte Rusty HaroldThursday, March 16, 2006elharo@metalab.unc.eduhttp://www.cafeaulait.org/ |
![]() |
Faster development; faster time to market
More robust, error free code
YAGNI: Writing tests first avoids writing unneeded code.
Easier to find and fix bugs
Easier to add features or change behavior; less worry about unintentionally introducing bugs
Makes refactoring/optimization possible: any change that doesn't break a test suite is de facto acceptable.
Often a developer will want to make some optimizations to a piece of code on which a lot of user-interface behavior depends. Sure, the new table sorter will be ten times faster and more memory efficient, but are you sure the changes won't affect the report generator? Run the tests and find out. A developer is much more likely to run a test that has encapsulated how to set up and test a report than he is to launch the application by hand and try to track down who knows how to do that particular test.
Run automatically without any human participation beyond launching the test suite
And maybe not even that: Cruise Control, Continuum, etc.
Results should be pass/fail
Test failures need to be blindingly obvious
Test must be reproducible and consistent
GUIs are designed to be driven by user; not a program.
The public interface of a GUI app is the GUI.
Test code often outruns the GUI it's supposed to be testing.
Glass box: testing the nonpublic API
White box: testing the public API
Grey box: testing internal components (Abbot; java.awt.Robot)
Black box: testing the GUI itself (Costello)
public class PlayerFrameTest extends TestCase {
private JFrame frame;
private JMenuBar menubar;
protected void setUp() throws QTException {
frame = new PlayerFrame();
menubar = frame.getJMenuBar();
}
public void testTitleOfUntitledFrameIsAmateurPlayer()
throws QTException {
assertEquals("Amateur Player", frame.getTitle());
}
public void testInitialFrameIsNotTooSmall() throws QTException {
Dimension d = frame.getSize();
assertTrue(d.width > 300);
assertTrue(d.height > 300);
}
}
public void testMinimizeCommandKeyEquivalent() {
Component[] fileitems = menubar.getMenu(3).getMenuComponents();
for (int i = 0; i < fileitems.length; i++) {
if (fileitems[i] instanceof JMenuItem) {
JMenuItem item = (JMenuItem) fileitems[i];
if (item.getText().equals("Minimize")) {
assertEquals('M', item.getAccelerator().getKeyCode());
}
}
}
}
List windows on screen
List dialogs onscreen
Find frontmost window
Find component by name
Find component by name and type (menu item, button, etc.)
Find component by label (including internationalized label).
Find menu item by label
Find menu item by name
Delay until window/menu/dialog is shown
Is there the seed of a library here?
Frame[] allFrames = Frame.getFrames();
For open windows test each one with isVisible()
For front window test with isFocused()
Get all frames
Get each frame's owned windows with getOwnedWindows()
public void testPresentMovieDialog() {
JMenuItem presentMovie = findJMenuItem(menubar, "Present Movie");
presentMovie.doClick();
Frame[] allFrames = Frame.getFrames();
for (int i = 0; i < allFrames.length; i++) {
Window[] owned = allFrames[i].getOwnedWindows();
for (int j = 0; j < allFrames.length; j++) {
Dialog dialog = (Dialog) owned[j];
if ("Present Movie".equals(dialog.getTitle())) {
return;
}
}
}
fail("No present dialog");
}
You can use (potentially recursively)
the getComponents()
method to find the
component you want to test:
private static void listComponents(Container container) {
Component[] components = container.getComponents();
for (int i = 0; i < components.length; i++) {
System.out.println(components[i]);
if (components[i] instanceof Container) {
listComponents((Container) components[i]);
}
}
}
Use getMenu()
, getMenuComponents()
, and getMenuCount()
to walk the menu bar:
private JMenuItem findJMenuItem(String name) {
for (int menu = 0; menu < menubar.getMenuCount(); menu++) {
Component[] menuitems = menubar.getMenu(menu).getMenuComponents();
for (int i = 0; i < menuitems.length; i++) {
if (menuitems[i] instanceof JMenuItem) {
JMenuItem item = (JMenuItem) menuitems[i];
if (item.getText().equals(name)) {
return item;
}
}
}
}
return null;
}
Menu items can be activated with the doClick()
method:
public void testNewMenuItem() {
int oldNumberOfWindows = Frame.getFrames().length;
JMenuItem newMenuItem = findJMenuItem("New Player");
newMenuItem.doClick();
int newNumberOfWindows = Frame.getFrames().length;
assertEquals(oldNumberOfWindows+1, newNumberOfWindows);
}
Other pieces can be tested by posting events into the event queue using EventQueue.postEvent()
:
public void postEvent(AWTEvent theEvent)
Or by dispatching straight to the component using Component.dispatchEvent()
public final void dispatchEvent(AWTEvent e)
Still can be hard to read the results
Control mouse and keyboard programmatically
Generates native system input events
Will actually move the mouse cursor instead of just generating mouse move events.
May need special privileges or extensions. If disallowed, the Robot
constructor will throw an AWTException
.
package java.awt;
public class Robot {
public Robot() throws AWTException
public Robot(GraphicsDevice screen) throws AWTException
public void mouseMove(int x, int y);
public void mousePress(int buttons);
public void mouseRelease(int buttons);
public void mouseWheel(int scrollAmount)
public void keyPress(int keycode);
public void keyRelease(int keycode);
public Color getPixelColor(int x, int y);
public BufferedImage createScreenCapture(Rectangle screenRect);
public boolean isAutoWaitForIdle();
public void setAutoWaitForIdle(boolean isOn);
public int getAutoDelay();
public void setAutoDelay(int ms);
public void delay(int ms);
public void waitForIdle();
public String toString();
}
Test that clicking the minimize button iconifies the window.
Automatic in Swing but not necessarily in the AWT.
public void testMinimizeButton() throws AWTException {
frame.setVisible(true);
Point p = frame.getLocationOnScreen();
Robot r = new Robot();
r.setAutoWaitForIdle(true);
r.mouseMove(p.x+35, p.y+10);
r.mousePress(InputEvent.BUTTON1_MASK);
r.mouseRelease(InputEvent.BUTTON1_MASK);
r.delay(1000); // needs a couple of seconds for the state to update
int state = frame.getExtendedState();
assertEquals(Frame.ICONIFIED, state);
}
Test individual pixels
This opens a file handcrafted in Photoshop to contain nothing but pure blue, and tests that this is indeed what is seen:
public void testOpenPNGFile() throws QTException, AWTException {
QTFile file = new QTFile("data/pureblue.png");
FileOpener.openFile(file);
Robot r = new Robot();
r.delay(5000);
Color expected = new Color(0, 0, 255);
Color actual = r.getPixelColor(50, 100);
assertEquals(expected, actual);
}
Improve robustness by getting coordinates to grab from the component you're testing with getLocationOnScreen()
:
public Point getLocationOnScreen()
Sometimes you can't really test pixel by pixel; or carefully contrive the pixels.
Less robust. Requires pixel perfection.
Platform, locale, and look and feel differences mean this can only really test pictures; not GUI components
Very dependent on exact position of components
Changes to GUI can break the test code
Different platforms can move things around enough to break the tests
Different localizations can move things around enough to break the tests
Mix and match: find the component; get its position, then move the mouse there
The events the Robot provides are like assembly language for GUI testing. To facilitate and encourage testing, you need a higher-level representation.
--Timothy Wall, Abbot Developer
A better Robot class to drive GUIs
A ComponentTester to drive components (at a higher, more semantic level than the Robot) and make assertions about them
Testers for individual components: JTable, Window, JComboBox, JMenuItem, JSpinner, etc. optimized to drive each such component
A finder package to locate components including a matcher interface (like FileFilter)
A component that allows the user to type in a non-negative integer (and only a non-negative integer)
A subclass of JTextField
Extra methods:
getNumericValue()
and setNumericValue()
0
1
2
3
4
5
6
7
8
9
10
100
1234
123456789
2^32-1
2^64-1
2^64+1
getNumberValue()
before anything is typed
-1
-2
-10
-123456789
-2^32-1
-2^64-1
-2^64+1
Extend ComponentFixture
Get a tester for the component we're testing.
Show the component in a frame.
"Type" text in the component with actionKeyString
package com.elharo.swing.test;
import com.elharo.swing.WholeNumberField;
import abbot.tester.ComponentTester;
import junit.extensions.abbot.ComponentTestFixture;
public class WholeNumberFieldTestCase extends ComponentTestFixture {
private ComponentTester tester;
private WholeNumberField field;
protected void setUp() {
tester = ComponentTester.getTester(WholeNumberField.class);
field = new WholeNumberField();
showFrame(field);
}
public void testOne() {
tester.actionKeyString(field, "1");
int result = field.getNumberValue();
assertEquals(1, result);
}
}
public void testText() {
tester.actionKeyString(field, "12ab34");
int result = field.getNumberValue();
assertEquals(1234, result);
}
public void actionClick(Component c, ComponentLocation loc, int buttons, int count)
public void actionClick(Component c, ComponentLocation loc, int buttons)
public void actionClick(Component c, ComponentLocation loc)
public void actionClick(Component comp, int x, int y, int buttons, int count)
public void actionClick(Component comp, int x, int y, int buttons)
public void actionClick(Component comp, int x, int y)
public void actionClick(Component comp)
public void actionDrag(Component dragSource, ComponentLocation loc, int modifiers)
public void actionDrag(Component dragSource, ComponentLocation loc, String modifiers)
public void actionDrag(Component dragSource, ComponentLocation loc)
public void actionDrag(Component dragSource, int sx, int sy, String modifiers)
public void actionDrag(Component dragSource, int sx, int sy)
public void actionDrag(Component dragSource)
public void actionDragOver(Component target, ComponentLocation where)
public void actionDrop(Component dropTarget, ComponentLocation loc)
public void actionDrop(Component dropTarget, int x, int y)
public void actionDrop(Component dropTarget)
public void actionFocus(Component comp)
public void actionKeyPress(Component comp, int keyCode)
public void actionKeyPress(int keyCode)
public void actionKeyRelease(Component comp, int keyCode)
public void actionKeyRelease(int keyCode)
public void actionKeyString(Component c, String string)
public void actionKeyString(String string)
public void actionKeyStroke(Component c, int keyCode, int modifiers)
public void actionKeyStroke(Component c, int keyCode)
public void actionKeyStroke(int keyCode, int modifiers)
public void actionKeyStroke(int keyCode)
public void actionSelectAWTMenuItem(Frame menuFrame, String path)
public void actionSelectAWTMenuItemByLabel(Frame menuFrame, String path)
public void actionSelectAWTPopupMenuItem(Component invoker, String path)
public void actionSelectAWTPopupMenuItemByLabel(Component invoker, String path)
public void actionSelectMenuItem(Component item)
public void actionSelectPopupMenuItem(Component invoker, ComponentLocation loc, String path)
public void actionSelectPopupMenuItem(Component invoker, int x, int y, String path)
public void actionSelectPopupMenuItem(Component invoker, String path)
public void actionShowPopupMenu(Component invoker, ComponentLocation loc)
public void actionShowPopupMenu(Component invoker, int x, int y)
public void actionShowPopupMenu(Component invoker)
A location relative to a component, to the center of the component unless specified otherwise
Subclasses provide locations for specific component types; for instance a JListLocation
can convert a row index into a location
public Point getPoint(Component c) throws LocationUnavailableException
public Rectangle getBounds(Component c) throws LocationUnavailableException
JLabel
JButton
JMenuItem
JPopupMenu
There are subclasses for each component (JLabelTester
, JButtonTester
, etc.) but you can ignore these.
package abbot.tester;
public class WindowTester extends ContainerTester {
public String deriveTag(Component comp);
public void actionClose(Component c);
public void actionMove(Component w, int screenx, int screeny);
public void actionMoveBy(Component w, int dx, int dy);
public void actionResize(Component w, int width, int height);
public void actionResizeBy(Component w, int dx, int dy);
public void actionActivate(Window w);
}
import java.awt.Dimension;
import quicktime.QTException;
import abbot.tester.WindowTester;
import com.elharo.quicktime.*;
public class RatioTest extends junit.extensions.abbot.ComponentTestFixture {
private PlayerFrame frame;
private WindowTester tester = new WindowTester();
static {
try {
QuicktimeInit.setup();
}
catch (QTException ex) {
throw new RuntimeException(ex);
}
}
protected void setUp() throws QTException {
frame = new PlayerFrame();
frame.show();
}
protected void tearDown() {
frame.hide();
frame.dispose();
}
public void testRatioIsConstant() {
Dimension d = frame.getSize();
double ratio = d.getHeight() / (double) d.getWidth();
tester.actionResize(frame, 2*d.width, d.height);
d = frame.getSize();
assertEquals(ratio, d.getHeight() / (double) d.getWidth(), 0.01);
}
}
package abbot.tester;
public class ButtonTester extends ComponentTester {
public void click(Component c, int x, int y,
int mask, int count)
}
package abbot.tester;
public class CheckboxTester extends ComponentTester {
public void click(Component c, int x, int y,
int mask, int count)
}
package abbot.tester;
public class ChoiceTester extends ComponentTester {
public void actionSelectIndex(Component c, int index)
public void actionSelectItem(Component c, String item)
}
package abbot.tester
public class JListTester extends JComponentTester {
public void actionSelectIndex(Component c, int index)
public void actionSelectItem(Component c, String item)
public void actionSelectValue(Component c, String item)
public void actionSelectRow(Component c, JListLocation location)
public ComponentLocation getLocation(Component c, Point index)
public int getSize(JList list)
public String getContents(JList list)
public Object getElementAt(JList list, int index)
}
package abbot.tester;
public class JFileChooserTester extends JComponentTester {
public void actionSetDirectory(Component c, String path)
public void actionCancel(Component c)
public void actionSetFilename(Component c, String filename)
public void actionApprove(Component c)
}
package abbot.tester;
public class JComboBoxTester extends JComponentTester {
public void actionSelectIndex(Component c, int index)
public void actionSelectItem(Component c, String item)
public JList findComboList(JComboBox X)
public String getValueAsString(JComboBox c, JList list, Object item, int index)
public String getContents(JComboBox c)
}
package abbot.tester;
public class JSliderTester extends JComponentTester {
public void actionIncrement(Component c)
public void actionDecrement(Component c)
public void actionSlide(Component c, int value)
public void actionSlideMaximum(Component c)
public void actionSlideMinimum(Component c)
}
package abbot.tester;
public class JSpinnerTester extends JComponentTester {
public void actionIncrement(Component c)
public void actionDecrement(Component c)
public void actionSetValue(Component c, String value)
}
package abbot.tester;
public class JSplitPaneTester extends JComponentTester {
public void actionMoveDivider(Component c, double X)
public void actionMoveDividerAbsolute(Component c, int index)
}
package abbot.tester;
public class JTableTester extends JComponentTester {
public void actionSelectCell(Component c, JTableLocation X)
public void actionSelectCell(Component c, int index, int index)
public ComponentLocation getLocation(Component c, Point index)
}
package abbot.tester;
public class JTextComponentTester extends JComponentTester {
public void actionClick(Component c, int index)
public void actionEnterText(Component c, String text)
public void actionSetCaretPosition(Component c, int index)
public void actionStartSelection(Component c, int index)
public void actionEndSelection(Component c, int index)
public void actionSelect(Component c, int start, int end)
public void actionSelectText(Component c, int start, int end)
}
package abbot.tester;
public class JTreeTester extends JComponentTester {
public void actionClick(Component c, ComponentLocation location)
public ComponentLocation parseLocation(String location)
public void actionSelectRow(Component c, ComponentLocation location)
public void actionSelectRow(Component c, int index)
public TreePath pathToStringPath(JTree tree, TreePath path)
public void actionClickRow(Component c, int index, String modifiers)
public void actionClickRow(Component c, int index, String modifiers, int index)
public void actionClickRow(Component c, int index)
public void actionMakeVisible(Component c, TreePath path)
public void actionSelectPath(Component c, TreePath path)
public void actionToggleRow(Component c, ComponentLocation location)
public void actionToggleRow(Component c, int index)
public boolean assertPathExists(Component c, TreePath path)
public ComponentLocation getLocation(Component c, Point index)
public boolean isLocationInExpandControl(JTree tree, int index, int index)
}
abbot.robot.auto_delay
abbot.robot.mode
abbot.robot.event_post_delay
abbot.robot.default_delay
abbot.robot.popup_delay
abbot.robot.component_delay
package abbot.finder;
public class BasicFinder implements ComponentFinder {
public BasicFinder();
public BasicFinder(Hierarchy h);
public static ComponentFinder getDefault();
public Component find(Container root, Matcher m);
throws ComponentNotFoundException, MultipleComponentsFoundException
public Component find(Matcher m)
throws ComponentNotFoundException, MultipleComponentsFoundException;
}
package abbot.finder;
public interface Matcher {
public boolean matches(Component c);
}
public NewPlayerMatcher implements Matcher {
public boolean matches(Component c) {
return c instanceof JMenuItem
&& "New Player".equals(((JMenuItem) c).getText());
}
}
Matcher matcher = new NewPlayerMatcher();
BasicFinder finder = new BasicFinder();
JMenuItem item = (JMenuItem) finder.find(container, matcher);
TestCase subclass your own test cases should extend
Sets up and cleans up the GUI.
Methods to place GUI component within a frame
Methods to show/hide windows and dialogs
Converts exceptions thrown from the event dispatch thread into test failures.
Script Recorder
Example: java -cp lib/abbot.jar junit.extensions.abbot.ScriptFixture doc/editor-tutorial-1.xml
Launch the script editor with java -jar abbot.jar
File/New Script
Test/Launch the application you're testing
Select Capture/All Actions (or shift+F2)
Use the GUI
Press Shift-F2 when the GUI reaches a state you want to test.
Subclass ScriptFixture
Write a suite()
method that returns a ScriptTestSuite
object specifying the saved scrripts to run
Scripts are loaded from the file system and filtered from directories.
Abbot Home Page: http://abbot.sourceforge.net/
This presentation: http://www.cafeaulait.org/slides/sd2006west/guitesting/
Robot class: http://java.sun.com/j2se/1.4.2/docs/api/java/awt/Robot.html
JFCUnit: http://jfcunit.sourceforge.net/
Jemmy: http://jemmy.netbeans.org
SwingUnit: https://swingunit.dev.java.net/
Abbott and Costello radio shows: http://www.radiolovers.com/pages/abbottcostello.html