I have been working with eggplant over the past few months, after being put in charge of building a smoke test suite for our application.
Although Eggplant is fabulous and is incredibly powerful if used correctly, I run into some problems that I am at a loss on how to fix.
The Test suites I have built work. Functionally on a succesful run they work great.
However say 1 run in 10 or sometimes even 1 in 5 run fail due to some previously identified image not being recognized (usually resulting in my having to replace the picture).
Now I use the tolerant option during all my image captures and make sure that each picture can be identified. When something fails, I capture the screen and find to my dismay that the picture that could not be found was present on the screen, (I am not talking about pictures that change… i.e, different icons, text, font), but could not be found by the running script.
Although Eggplant is supposed to be resolution-independent, I have frozen my SUT to ensure that there are no external factors that could result in images not being found.
I have messed around with the default time taken for imagefound and other times, with varying success.
Is a 100% success rate impossible to achieve? What are some of the strategies you guys use to test eggplant to make sure that the script fails due to a genuine bug and not because the program didn’t run correctly?