Suggestion for improved feedback on failing tests

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Suggestion for improved feedback on failing tests

iracooke
Hi All, 

My guess is that one of the most common ways in which tools will fail tests on the build-bot is when a dependency fails to install properly.  I think this is what is happening to my tool xtandem 


One potential improvement that I think could make it easier to debug this situation would be to show test status for repositories even when they contain no tools (the test would simply attempt an install and show the installation log under the test details).   This would be particularly useful for repositories that exist purely to install a dependency eg


Naturally a tool with proper functional tests would fail if its dependency installations fail .. but it would be a huge help in narrowing down the issue to be able to see where those failures occurred.  Another point is that this is pretty much the only way of getting feedback on why a tool might fail installation on the build-bot without having a perfect clone of the build-bot locally.

Cheers
Ira


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: [galaxy-iuc] Suggestion for improved feedback on failing tests

Peter Cock
On Tue, May 14, 2013 at 3:32 AM, Ira Cooke <[hidden email]> wrote:
> Hi All,
>
> My guess is that one of the most common ways in which tools will fail tests
> on the build-bot is when a dependency fails to install properly.  I think
> this is what is happening to my tool xtandem
>
> http://testtoolshed.g2.bx.psu.edu/view/iracooke/xtandem

You may be suffering from the same "missing test results" problem
as me, see the long thread "Missing test results on (Test) Tool Shed":
http://dev.list.galaxyproject.org/Missing-test-results-on-Test-Tool-Shed-td4659531.html

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: [galaxy-iuc] Suggestion for improved feedback on failing tests

iracooke
Hi Peter,

Yes that's most likely part of it ... thanks for the link to that thread.  I think I am suffering from the same issue.

 ... but I'm also assuming when my test results come back I'll still need to figure out where my repository dependencies failed.  Last test result I saw it looked like a failure compiling ruby ... but it's hard to know what's missing to fix it.

Cheers
Ira

On 14/05/2013, at 6:54 PM, Peter Cock <[hidden email]> wrote:

> On Tue, May 14, 2013 at 3:32 AM, Ira Cooke <[hidden email]> wrote:
>> Hi All,
>>
>> My guess is that one of the most common ways in which tools will fail tests
>> on the build-bot is when a dependency fails to install properly.  I think
>> this is what is happening to my tool xtandem
>>
>> http://testtoolshed.g2.bx.psu.edu/view/iracooke/xtandem
>
> You may be suffering from the same "missing test results" problem
> as me, see the long thread "Missing test results on (Test) Tool Shed":
> http://dev.list.galaxyproject.org/Missing-test-results-on-Test-Tool-Shed-td4659531.html
>
> Peter


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: [galaxy-iuc] Suggestion for improved feedback on failing tests

Peter Cock
On Tue, May 14, 2013 at 10:39 AM, Ira Cooke <[hidden email]> wrote:

> Hi Peter,
>
> Yes that's most likely part of it ... thanks for the link to that thread.
> I think I am suffering from the same issue.
>
>  ... but I'm also assuming when my test results come back I'll still
> need to figure out where my repository dependencies failed.  Last
> test result I saw it looked like a failure compiling ruby ... but it's
> hard to know what's missing to fix it.
>
> Cheers
> Ira

Me too - I've been struggling with tests failing due to partial installs
(where only the beginning of the tool_dependencies.xml is processed
but no error from the installation process is show on the Tool Shed):
http://dev.list.galaxyproject.org/Handling-of-tool-dependencies-xml-errors-in-Tool-Shed-testing-tt4659720.html

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion for improved feedback on failing tests

Greg Von Kuster
In reply to this post by iracooke
We're fairly close to having a new container in the "Tool test results" container.  The new container will be something like "Installation errors" and for each tool it will list the tool dependencies that have installation errors.  Any installation errors will result in the tool not being tested.  This new feature should hopefully be available today.

Greg Von Kuster


On May 13, 2013, at 10:32 PM, Ira Cooke wrote:

Hi All, 

My guess is that one of the most common ways in which tools will fail tests on the build-bot is when a dependency fails to install properly.  I think this is what is happening to my tool xtandem 


One potential improvement that I think could make it easier to debug this situation would be to show test status for repositories even when they contain no tools (the test would simply attempt an install and show the installation log under the test details).   This would be particularly useful for repositories that exist purely to install a dependency eg


Naturally a tool with proper functional tests would fail if its dependency installations fail .. but it would be a huge help in narrowing down the issue to be able to see where those failures occurred.  Another point is that this is pretty much the only way of getting feedback on why a tool might fail installation on the build-bot without having a perfect clone of the build-bot locally.

Cheers
Ira

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/