Tests not being run on toolsheds?

classic Classic list List threaded Threaded
10 messages Options
| Threaded
Open this post in threaded view
|

Tests not being run on toolsheds?

Peter Briggs
Hello

Apologies as this seems to be quite a frequent question (and I may have
missed something) however: are the tool tests being executed on the main
or test toolsheds at the moment?

The most recent tests results that I can find for any of the tools I've
uploaded is the 29th January. I've also looked at a couple of tools from
other users and the situation appears to be the same (at least in those
cases).

Thanks for your help!

Best wishes

Peter

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Cock
On Tue, Mar 17, 2015 at 1:20 PM, Peter Briggs
<[hidden email]> wrote:

> Hello
>
> Apologies as this seems to be quite a frequent question (and I may have
> missed something) however: are the tool tests being executed on the main or
> test toolsheds at the moment?
>
> The most recent tests results that I can find for any of the tools I've
> uploaded is the 29th January. I've also looked at a couple of tools from
> other users and the situation appears to be the same (at least in those
> cases).
>
> Thanks for your help!
>
> Best wishes
>
> Peter

Hi Peter B.,

You're probably seeing some of my emails on this topic - there
used to be a nightly test run, then dropped to every second night,
but that Galaxy team have been having trouble with the system.

On a positive note, at least some tools are getting tested. This is
one I updated earlier this month on 2015-03-06, but didn't include
all the named test files - so the test failed:

https://toolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
Automated test environment
Time tested: 2015-03-16 15:15:33
...
One or more test files are missing for tool sample_seqs:
MID4_GLZRM4E04_rnd30.sff

I updated this on the test tool shed on 2015-03-16 but it has
yet to be tested there:

https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs

For now I would urge you to include as much local testing as
you can, and explore setting up another testing service like
TravisCI or Jenkins for your tool development. e.g.

http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html

Note that John's plameno tool looks set to become extremely
useful for both running the tests, and other sanity checking
with its "lint" command (and I plan to use this for my TravisCI
test setup one day):

https://github.com/galaxyproject/planemo

Peter C.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Briggs
Hello Peter C.

Thanks for your comments and advice Ironically after I sent that email
the tests for the next tool I looked at in the toolshed had been run in
the past couple of days.

I'm interested because I've got a couple of tools that seem to work okay
when I run the tests locally, but are reported as failing on the
toolshed, and I wanted to see if I'd addressed the issue for one of them
by updating the tool dependencies.

(As an aside I'd also assumed that passing tests is one of the
conditions for the tool to be marked as "verified", but maybe that's not
the case?)

However as you say, more testing locally seems to be a good way to go -
thanks for your suggestions. I looked at planemo briefly a while ago and
it looked good. The other issue I've had with testing is actually
testing tool installation (i.e. tool_dependencies.xml) - I recall that
planemo didn't deal with that side of things, so I had to set up the
environment for the tests manually.

Thanks again for your advice and the links, I will investigate trying to
set up a local solution (including some CI testing!).

Best wishes

Peter

On 17/03/15 13:40, Peter Cock wrote:

> On Tue, Mar 17, 2015 at 1:20 PM, Peter Briggs
> <[hidden email]> wrote:
>> Hello
>>
>> Apologies as this seems to be quite a frequent question (and I may have
>> missed something) however: are the tool tests being executed on the main or
>> test toolsheds at the moment?
>>
>> The most recent tests results that I can find for any of the tools I've
>> uploaded is the 29th January. I've also looked at a couple of tools from
>> other users and the situation appears to be the same (at least in those
>> cases).
>>
>> Thanks for your help!
>>
>> Best wishes
>>
>> Peter
>
> Hi Peter B.,
>
> You're probably seeing some of my emails on this topic - there
> used to be a nightly test run, then dropped to every second night,
> but that Galaxy team have been having trouble with the system.
>
> On a positive note, at least some tools are getting tested. This is
> one I updated earlier this month on 2015-03-06, but didn't include
> all the named test files - so the test failed:
>
> https://toolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
> Automated test environment
> Time tested: 2015-03-16 15:15:33
> ...
> One or more test files are missing for tool sample_seqs:
> MID4_GLZRM4E04_rnd30.sff
>
> I updated this on the test tool shed on 2015-03-16 but it has
> yet to be tested there:
>
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
>
> For now I would urge you to include as much local testing as
> you can, and explore setting up another testing service like
> TravisCI or Jenkins for your tool development. e.g.
>
> http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html
>
> Note that John's plameno tool looks set to become extremely
> useful for both running the tests, and other sanity checking
> with its "lint" command (and I plan to use this for my TravisCI
> test setup one day):
>
> https://github.com/galaxyproject/planemo
>
> Peter C.
>

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Cock
On Wed, Mar 18, 2015 at 10:13 AM, Peter Briggs
<[hidden email]> wrote:
> Hello Peter C.
>
> Thanks for your comments and advice Ironically after I sent that email the
> tests for the next tool I looked at in the toolshed had been run in the past
> couple of days.

That's good. I can also confirm that the Test Tool Shed example I
gave was tested overnight, although it looks like a novel failure:
https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs

> I'm interested because I've got a couple of tools that seem to work okay
> when I run the tests locally, but are reported as failing on the toolshed,
> and I wanted to see if I'd addressed the issue for one of them by updating
> the tool dependencies.

I suggest asking on this mailing list, posting the shareable Tool Shed URL
and a copy of the error message. Hopefully it will be a simple issue
(forgetting a test file is my most common mistake), or it may be
that another tool author has seen the same error (which can happen
due to problems in Galaxy itself).

> (As an aside I'd also assumed that passing tests is one of the conditions
> for the tool to be marked as "verified", but maybe that's not the case?)

I think there is still a human involved, but passing tests is expected
for a "verified" tool.

> However as you say, more testing locally seems to be a good way to go -
> thanks for your suggestions. I looked at planemo briefly a while ago and it
> looked good. The other issue I've had with testing is actually testing tool
> installation (i.e. tool_dependencies.xml) - I recall that planemo didn't
> deal with that side of things, so I had to set up the environment for the
> tests manually.

There is some work on tool_dependencies.xml ongoing within planemo,
John Chilton would be the person to ask (CC'd directly).

> Thanks again for your advice and the links, I will investigate trying to set
> up a local solution (including some CI testing!).

This is something where using planemo ought to be very useful - for
now I still do my local testing with a test Galaxy instance via:

$ ./run_tests.sh -id my_tool_id

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Dave Bouvier
Gentlemen,

The issue with the nightly testing was due to a stalled test run
blocking subsequent tests. I've cleared out that blockage and a manual
test run appears to have completed successfully, as should future
automated test runs. As always, feel free to let us know if you
encounter any additional inexplicable behavior.

    --Dave B.

On 03/18/2015 07:00 AM, Peter Cock wrote:

> On Wed, Mar 18, 2015 at 10:13 AM, Peter Briggs
> <[hidden email]> wrote:
>> Hello Peter C.
>>
>> Thanks for your comments and advice Ironically after I sent that email the
>> tests for the next tool I looked at in the toolshed had been run in the past
>> couple of days.
>
> That's good. I can also confirm that the Test Tool Shed example I
> gave was tested overnight, although it looks like a novel failure:
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
>
>> I'm interested because I've got a couple of tools that seem to work okay
>> when I run the tests locally, but are reported as failing on the toolshed,
>> and I wanted to see if I'd addressed the issue for one of them by updating
>> the tool dependencies.
>
> I suggest asking on this mailing list, posting the shareable Tool Shed URL
> and a copy of the error message. Hopefully it will be a simple issue
> (forgetting a test file is my most common mistake), or it may be
> that another tool author has seen the same error (which can happen
> due to problems in Galaxy itself).
>
>> (As an aside I'd also assumed that passing tests is one of the conditions
>> for the tool to be marked as "verified", but maybe that's not the case?)
>
> I think there is still a human involved, but passing tests is expected
> for a "verified" tool.
>
>> However as you say, more testing locally seems to be a good way to go -
>> thanks for your suggestions. I looked at planemo briefly a while ago and it
>> looked good. The other issue I've had with testing is actually testing tool
>> installation (i.e. tool_dependencies.xml) - I recall that planemo didn't
>> deal with that side of things, so I had to set up the environment for the
>> tests manually.
>
> There is some work on tool_dependencies.xml ongoing within planemo,
> John Chilton would be the person to ask (CC'd directly).
>
>> Thanks again for your advice and the links, I will investigate trying to set
>> up a local solution (including some CI testing!).
>
> This is something where using planemo ought to be very useful - for
> now I still do my local testing with a test Galaxy instance via:
>
> $ ./run_tests.sh -id my_tool_id
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Briggs
Hello Dave

Thanks for addressing the specific issue of the tests failing to run.

Re my initial problem vis-a-vis a tool with tests that worked locally
but failed on the toolsheds, this appears to be a result of the two
environments using differing versions of perl. This caused the ordering
of data in the output files to differ from the reference output in the
tests on the toolshed.

As the tool wraps a third-party program, I've addressed this by
explicitly specifying the version of perl that it uses (at least it
seems to have worked for my latest version on the test toolshed - I will
update the main toolshed next).

The repositories in question are:
https://toolshed.g2.bx.psu.edu/view/pjbriggs/pal_finder
https://testtoolshed.g2.bx.psu.edu/view/pjbriggs/pal_finder

I see a separate problem with a different tool which I'll investigate
next and report if I can't understand the toolshed test failure.

Thanks again for your help,

Best wishes

Peter

On 18/03/15 13:46, Dave Bouvier wrote:

> Gentlemen,
>
> The issue with the nightly testing was due to a stalled test run
> blocking subsequent tests. I've cleared out that blockage and a manual
> test run appears to have completed successfully, as should future
> automated test runs. As always, feel free to let us know if you
> encounter any additional inexplicable behavior.
>
>     --Dave B.
>
> On 03/18/2015 07:00 AM, Peter Cock wrote:
>> On Wed, Mar 18, 2015 at 10:13 AM, Peter Briggs
>> <[hidden email]> wrote:
>>> Hello Peter C.
>>>
>>> Thanks for your comments and advice Ironically after I sent that
>>> email the
>>> tests for the next tool I looked at in the toolshed had been run in
>>> the past
>>> couple of days.
>>
>> That's good. I can also confirm that the Test Tool Shed example I
>> gave was tested overnight, although it looks like a novel failure:
>> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
>>
>>> I'm interested because I've got a couple of tools that seem to work okay
>>> when I run the tests locally, but are reported as failing on the
>>> toolshed,
>>> and I wanted to see if I'd addressed the issue for one of them by
>>> updating
>>> the tool dependencies.
>>
>> I suggest asking on this mailing list, posting the shareable Tool Shed
>> URL
>> and a copy of the error message. Hopefully it will be a simple issue
>> (forgetting a test file is my most common mistake), or it may be
>> that another tool author has seen the same error (which can happen
>> due to problems in Galaxy itself).
>>
>>> (As an aside I'd also assumed that passing tests is one of the
>>> conditions
>>> for the tool to be marked as "verified", but maybe that's not the case?)
>>
>> I think there is still a human involved, but passing tests is expected
>> for a "verified" tool.
>>
>>> However as you say, more testing locally seems to be a good way to go -
>>> thanks for your suggestions. I looked at planemo briefly a while ago
>>> and it
>>> looked good. The other issue I've had with testing is actually
>>> testing tool
>>> installation (i.e. tool_dependencies.xml) - I recall that planemo didn't
>>> deal with that side of things, so I had to set up the environment for
>>> the
>>> tests manually.
>>
>> There is some work on tool_dependencies.xml ongoing within planemo,
>> John Chilton would be the person to ask (CC'd directly).
>>
>>> Thanks again for your advice and the links, I will investigate trying
>>> to set
>>> up a local solution (including some CI testing!).
>>
>> This is something where using planemo ought to be very useful - for
>> now I still do my local testing with a test Galaxy instance via:
>>
>> $ ./run_tests.sh -id my_tool_id
>>
>> Peter
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>>

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Briggs
In reply to this post by Peter Cock
Thanks Peter. I've reported the details of the specific tool that I was
having problems with in my message to Dave B.

planemo already looks really good, if it could handle the tool
dependencies as well then that would be the icing on the cake.

Thanks for your help, best wishes

Peter

On 18/03/15 11:00, Peter Cock wrote:

> On Wed, Mar 18, 2015 at 10:13 AM, Peter Briggs
> <[hidden email]> wrote:
>> Hello Peter C.
>>
>> Thanks for your comments and advice Ironically after I sent that email the
>> tests for the next tool I looked at in the toolshed had been run in the past
>> couple of days.
>
> That's good. I can also confirm that the Test Tool Shed example I
> gave was tested overnight, although it looks like a novel failure:
> https://testtoolshed.g2.bx.psu.edu/view/peterjc/sample_seqs
>
>> I'm interested because I've got a couple of tools that seem to work okay
>> when I run the tests locally, but are reported as failing on the toolshed,
>> and I wanted to see if I'd addressed the issue for one of them by updating
>> the tool dependencies.
>
> I suggest asking on this mailing list, posting the shareable Tool Shed URL
> and a copy of the error message. Hopefully it will be a simple issue
> (forgetting a test file is my most common mistake), or it may be
> that another tool author has seen the same error (which can happen
> due to problems in Galaxy itself).
>
>> (As an aside I'd also assumed that passing tests is one of the conditions
>> for the tool to be marked as "verified", but maybe that's not the case?)
>
> I think there is still a human involved, but passing tests is expected
> for a "verified" tool.
>
>> However as you say, more testing locally seems to be a good way to go -
>> thanks for your suggestions. I looked at planemo briefly a while ago and it
>> looked good. The other issue I've had with testing is actually testing tool
>> installation (i.e. tool_dependencies.xml) - I recall that planemo didn't
>> deal with that side of things, so I had to set up the environment for the
>> tests manually.
>
> There is some work on tool_dependencies.xml ongoing within planemo,
> John Chilton would be the person to ask (CC'd directly).
>
>> Thanks again for your advice and the links, I will investigate trying to set
>> up a local solution (including some CI testing!).
>
> This is something where using planemo ought to be very useful - for
> now I still do my local testing with a test Galaxy instance via:
>
> $ ./run_tests.sh -id my_tool_id
>
> Peter
>

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Cock
In reply to this post by Dave Bouvier
Hi Dave,

Can you check for stalls on the following repositories please?
They've not been tested for nearly two months:

https://toolshed.g2.bx.psu.edu/view/peterjc/clinod
Last tested 2015-01-29, Exception: Job in error state.

https://toolshed.g2.bx.psu.edu/view/peterjc/blast2go
Last tested 2015-01-29, Exception: Job in error state.

https://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3
Last tested 2015-01-29, Exception: Job in error state.

https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad
Last tested 2015-01-29, Converting local (test-data) bam to sam failed

The cause of the last failure is different (and is also failing on the Test
Tool Shed), raised separately here:

https://lists.galaxyproject.org/pipermail/galaxy-dev/2015-March/021650.html
http://dev.list.galaxyproject.org/test-base-twilltestcase-py-Converting-local-test-data-bam-to-sam-failed-tc4666509.html

--

Could you tweak the "Latest revision: failing tool tests" output
to include the date last tested in the table of results?

Over on the test tool shed there are plenty of recent failures
which I will email about separately - I'll report back here if I
spot any more stall entries where there tests look backlogged.

Thanks,

Peter

On Wed, Mar 18, 2015 at 1:46 PM, Dave Bouvier <[hidden email]> wrote:
> Gentlemen,
>
> The issue with the nightly testing was due to a stalled test run blocking
> subsequent tests. I've cleared out that blockage and a manual test run
> appears to have completed successfully, as should future automated test
> runs. As always, feel free to let us know if you encounter any additional
> inexplicable behavior.
>
>    --Dave B.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Cock
On Mon, Mar 23, 2015 at 9:54 AM, Peter Cock <[hidden email]> wrote:

> Hi Dave,
>
> Can you check for stalls on the following repositories please?
> They've not been tested for nearly two months:
>
> https://toolshed.g2.bx.psu.edu/view/peterjc/clinod
> Last tested 2015-01-29, Exception: Job in error state.
>
> https://toolshed.g2.bx.psu.edu/view/peterjc/blast2go
> Last tested 2015-01-29, Exception: Job in error state.
>
> https://toolshed.g2.bx.psu.edu/view/peterjc/effectivet3
> Last tested 2015-01-29, Exception: Job in error state.
>
> https://toolshed.g2.bx.psu.edu/view/peterjc/samtools_depad
> Last tested 2015-01-29, Converting local (test-data) bam to sam failed
>
> The cause of the last failure is different (and is also failing on the Test
> Tool Shed), raised separately here:
>
> https://lists.galaxyproject.org/pipermail/galaxy-dev/2015-March/021650.html
> http://dev.list.galaxyproject.org/test-base-twilltestcase-py-Converting-local-test-data-bam-to-sam-failed-tc4666509.html
>
> --
>
> Could you tweak the "Latest revision: failing tool tests" output
> to include the date last tested in the table of results?

Issue filed for this enhancement suggestion:

https://trello.com/c/eBXs2LxL/2571-show-date-last-tested-in-toolshed-searches-like-latest-revision-failing-tool-tests

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Tests not being run on toolsheds?

Peter Cock
On Tue, Mar 24, 2015 at 4:33 PM, Peter Cock <[hidden email]> wrote:

> On Mon, Mar 23, 2015 at 9:54 AM, Peter Cock <[hidden email]> wrote:
>> Hi Dave,
>>
>> ...
>>
>> Could you tweak the "Latest revision: failing tool tests" output
>> to include the date last tested in the table of results?
>
> Issue filed for this enhancement suggestion:
>
> https://trello.com/c/eBXs2LxL/2571-show-date-last-tested-in-toolshed-searches-like-latest-revision-failing-tool-tests
>
> Peter

Hi again Dave,

As far as I can tell from my limited random sampling, the last
tests on the Test Tool Shed were back on 2015-03-18 (about
three weeks ago).

I take it there are some technical problems again? :(

I'd particularly like to see the test results for this repository
(which I expect to pass, in which case I will update the main
Tool Shed entry):

https://testtoolshed.g2.bx.psu.edu/view/devteam/ncbi_blast_plus/

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/