I’m pleased to announce TestMerge.app, a tool that we’ve developed as part of the Core Plot project to aid managing the output of the Google Toolbox for Mac (GTM) framework.
Download
You can download TestMerge.app from the Core Plot downloads section or build from source by pulling from the Core Plot repository (hg clone https://core-plot.googlecode.com/hg/ core-plot).
Using TestMerge
GTM’s image and state comparison macros generate named rendered output of NS/UIViews, CALayers, NSWindows or the encoded state of NSObjects respectively. If an existing reference file of the same already exists in the test bundle’s Resources/ folder, the new output is compared against the reference with the test failing in the case that there is a mismatch (pixel-by-pixel or text) between the files. If no reference file exists, a new output file is generated. The GTM test macros make unit testing UI’s almost too easy. In Core Plot, we generate tens or hundereds of outputs from these macros on each test run.
When tests fail because the generated output doesn’t match the reference output, users of GTM’s test macros must manually compare the reference and output files, decide which to keep, then move files appropriately if needed. Since the failed output has a ‘_Failed’ suffix appended to the image/encoded state file’s name, the developer must rename the file if moving it to overwrite the existing reference file. Clearly, this can become a little cumbersome and will eventually lead to bit rot of the reference files.
Enter TestMerge.app. Taking the path to the folder of reference images and the path to the folder of output images in $TM_REFERENCE_PATH and $TM_OUTPUT_PATH environment variables respectively, TestMerge.app presents a UI for browsing and selecting from the reference/output comparisons. Only new output files (those with no corresponding reference) or failed output files (those that do not match their reference) are shown by TestMerge. Selecting a name on the left panel will show the reference and output for the image or state file with that name.
- Screenshot of TestMerge image comparison
In the screenhot shown at the right (of TestMerge showing the output of its own unit tests), a named image pair is selected in the left pane. The referenc image is then shown on the left and the new output is shown on the right with the pixel differences highlighted in red. Encoded state output is shown similarly (though no fancy highlighting is done on the text diff yet).
Selecting “Select Reference” from the “Merge” menu selects the existing reference as the correct image or state (indicating a true test failure). Conversely, selecting “Select Output” from the “Merge” menu marks the new output as the correct new reference image/state (indicating that the programs out has changed and the test needs to be updated to match this new, correct, output).
A blue outline indicates the selected file (reference or output) in the right pane.
When you quit TestMerge, you’ll be promted to commit the choices you’ve made. If you selected the reference file for a pair, the new output will be deleted. If you selected the new output for a pair, the old reference file will be replaced (after appropriately renaming the output file). If you made no selection for a pair, both files will be left untouched. Obviously choosing to not commit the selections leaves all files untouched.
You’ll still have to manually add new reference files to your Xcode project unit test target, and you’ll have to add new files to your SCM, but TestMerge can’t do it all on the first version or there’d be nothing left to work on.
That’s it. A simple tool for a simple job.
The Core Plot (and TestMerge) projects are already configured to run TestMerge automatically after each unit test run. To do the same in your Xcode project, add a Run Script build phase after the Run Script build phase that calls your test runner. In the new phases’ script use
# Run TestMerge to merge GTM output, if necessary
export TM_REFERENCE_PATH=${PROJECT_DIR}/TestResources
export TM_OUTPUT_PATH=~/Desktop
TEST_MERGE_PATH/TestMerge.app/Contents/MacOS/TestMerge
You’ll have to modify the export lines to give the appropriate paths for your project. GTM’s output goes to ~/Desktop by default. You’ll also have to replace the TEST_MERGE_PATH with the appropriate path on your system.
Please file issues on the Core Plot issues tracker. Happy testing!
Tagged cocoa, CorePlot, oss, TestMerge