Retrying failed tests / Statement.evaluate() never seems to throw an exception

classic Classic list List threaded Threaded
5 messages Options
ps
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Retrying failed tests / Statement.evaluate() never seems to throw an exception

ps
Hello everybody,

I tried to use a custom TestRule to get spock to retry failed tests. The implementation is copied from http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately and works with plain junit but not with spock. Is there a way to get this working? Is there another way to retry failed tests?

Thanks a lot,
Pascal

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/spockframework?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
ps
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Retrying failed tests / Statement.evaluate() never seems to throw an exception

ps
Using a deprecated custom MethodRule as shown in https://gist.github.com/abyx/897229 works, but the setup and cleanup closures are skipped. I guess the best way to solve this is a custom spock extension? Sadly I could not figure out how to rerun features with a spock extension from the only explanation I found (http://ldaley.com/post/971946675/annotation-driven-extensions-with-spock). Any hints how to achieve this?

Thanks,
Pascal

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/spockframework?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Retrying failed tests / Statement.evaluate() never seems to throw an exception

Peter Niederwieser
Administrator
Pascal,

this topic has been discussed before, both here and on Stack Overflow.

Spock's rule execution is fully compatible with JUnit's, except for one difference: In Spock, an exception occurring inside a test method is caught and handled before control is passed back to a TestRule, whereas in JUnit it isn't. This isn't something that's easy to change. MethodRule doesn't have this problem, and isn't deprecated anymore in latest JUnit. But as you said, it differs from TestRule in that it only wraps around the test method, but not around setup and cleanup.

In general, retrying with a TestRule doesn't seem ideal. If nothing else, the same test instance will be used each time (also in JUnit). If you can live with these kinds of limitations, you could implement a MethodRule and use setup/cleanup blocks instead of setup/cleanup methods. I can think of a few other potential (Spock-specific) solutions, but they aren't straightforward to explain or implement. Eventually, I'd like to enhance the extension SPI in a way that makes it easy to express repetition/retry.

Hope this helps. What's your motivation for retrying?

Cheers,
Peter

On May 30, 2013, at 10:34 PM, PascalSchumacher <[hidden email]> wrote:

Using a deprecated custom MethodRule as shown in https://gist.github.com/abyx/897229 works, but the setup and cleanup closures are skipped. I guess the best way to solve this is a custom spock extension? Sadly I could not figure out how to rerun features with a spock extension from the only explanation I found (http://ldaley.com/post/971946675/annotation-driven-extensions-with-spock). Any hints how to achieve this?

Thanks,
Pascal

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/spockframework?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/spockframework?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
ps
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Retrying failed tests / Statement.evaluate() never seems to throw an exception

ps
Hello Peter,

Am Donnerstag, 30. Mai 2013 23:59:51 UTC+2 schrieb Peter Niederwieser:
this topic has been discussed before, both here and on Stack Overflow.

 sorry for the duplication. I did not find any discussion other that a post on stackoverflow which suggests use of the method rule.


Spock's rule execution is fully compatible with JUnit's, except for one difference: In Spock, an exception occurring inside a test method is caught and handled before control is passed back to a TestRule, whereas in JUnit it isn't. This isn't something that's easy to change. MethodRule doesn't have this problem, and isn't deprecated anymore in latest JUnit. But as you said, it differs from TestRule in that it only wraps around the test method, but not around setup and cleanup.

In general, retrying with a TestRule doesn't seem ideal. If nothing else, the same test instance will be used each time (also in JUnit). If you can live with these kinds of limitations, you could implement a MethodRule and use setup/cleanup blocks instead of setup/cleanup methods. I can think of a few other potential (Spock-specific) solutions, but they aren't straightforward to explain or implement. Eventually, I'd like to enhance the extension SPI in a way that makes it easy to express repetition/retry.

O.k. thanks for the information. :)


Hope this helps. What's your motivation for retrying?
 
We use spock for end-to-end testing gray-box testing of our system. Most tests check the database or call webservices of external systems. Sometimes the response is a bit slow so that one or two out of over a hundred  tests fail. As rerunning all tests takes a lot of time I just want to repeat the failed ones, if I can a way to do this easily. In the long run I will try to adjust the scaffolding to make the test less brittle, but this takes time I do not have at the moment.

Thanks again for the input,
Pascal

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/spockframework?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Retrying failed tests / Statement.evaluate() never seems to throw an exception

Levon Saldamli
In reply to this post by Peter Niederwieser
Hello Peter,

I'm trying to implement retry functionality that also can catch errors in setup methods, but I can't find any extension approach that would work. I'm working on spock-retry which adds an interceptor to the feature method, catches exceptions, and reruns proceed() a number of times. But this only works on feature methods, not on e.g. iteration interceptors or feature interceptors. Is there nowadays a way to implement a retry functionality which can run setup/feature/cleanup and catch errors and do retries?

/Levon

Den torsdag 30 maj 2013 kl. 23:59:51 UTC+2 skrev Peter Niederwieser:
Pascal,

this topic has been discussed before, both here and on Stack Overflow.

Spock's rule execution is fully compatible with JUnit's, except for one difference: In Spock, an exception occurring inside a test method is caught and handled before control is passed back to a TestRule, whereas in JUnit it isn't. This isn't something that's easy to change. MethodRule doesn't have this problem, and isn't deprecated anymore in latest JUnit. But as you said, it differs from TestRule in that it only wraps around the test method, but not around setup and cleanup.

In general, retrying with a TestRule doesn't seem ideal. If nothing else, the same test instance will be used each time (also in JUnit). If you can live with these kinds of limitations, you could implement a MethodRule and use setup/cleanup blocks instead of setup/cleanup methods. I can think of a few other potential (Spock-specific) solutions, but they aren't straightforward to explain or implement. Eventually, I'd like to enhance the extension SPI in a way that makes it easy to express repetition/retry.

Hope this helps. What's your motivation for retrying?

Cheers,
Peter

On May 30, 2013, at 10:34 PM, PascalSchumacher <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="0SIvxoLhX7UJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">pascalsc...@...> wrote:

Using a deprecated custom MethodRule as shown in <a href="https://gist.github.com/abyx/897229" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgist.github.com%2Fabyx%2F897229\46sa\75D\46sntz\0751\46usg\75AFQjCNEvgB9kXgUcJYZ_Act9HdD8gaKYXA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgist.github.com%2Fabyx%2F897229\46sa\75D\46sntz\0751\46usg\75AFQjCNEvgB9kXgUcJYZ_Act9HdD8gaKYXA&#39;;return true;">https://gist.github.com/abyx/897229 works, but the setup and cleanup closures are skipped. I guess the best way to solve this is a custom spock extension? Sadly I could not figure out how to rerun features with a spock extension from the only explanation I found (<a href="http://ldaley.com/post/971946675/annotation-driven-extensions-with-spock" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fldaley.com%2Fpost%2F971946675%2Fannotation-driven-extensions-with-spock\46sa\75D\46sntz\0751\46usg\75AFQjCNHQI5K9twRk2HQGJWfwdwxCdOKZHw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fldaley.com%2Fpost%2F971946675%2Fannotation-driven-extensions-with-spock\46sa\75D\46sntz\0751\46usg\75AFQjCNHQI5K9twRk2HQGJWfwdwxCdOKZHw&#39;;return true;">http://ldaley.com/post/971946675/annotation-driven-extensions-with-spock). Any hints how to achieve this?

Thanks,
Pascal

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0SIvxoLhX7UJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">spockframewor...@googlegroups.com.
To post to this group, send email to <a href="javascript:" target="_blank" gdf-obfuscated-mailto="0SIvxoLhX7UJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">spockfr...@googlegroups.com.
Visit this group at <a href="http://groups.google.com/group/spockframework?hl=en" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://groups.google.com/group/spockframework?hl\75en&#39;;return true;" onclick="this.href=&#39;http://groups.google.com/group/spockframework?hl\75en&#39;;return true;">http://groups.google.com/group/spockframework?hl=en.
For more options, visit <a href="https://groups.google.com/groups/opt_out" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://groups.google.com/groups/opt_out&#39;;return true;" onclick="this.href=&#39;https://groups.google.com/groups/opt_out&#39;;return true;">https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "Spock Framework - User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at http://groups.google.com/group/spockframework.
For more options, visit https://groups.google.com/d/optout.
Loading...