Handling of tool_dependencies.xml errors in Tool Shed testing

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
Hi all,

For defining the 'install script' for some of my tool wrappers, I have
mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures
and looking at one or two examples.

I've got some of my tools to work, but not all. For instance, this one
appears not to be downloading or moving JAR files correctly:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965

<?xml version="1.0"?>
<tool_dependency>
    <package name="effectiveT3" version="1.0.1">
        <install version="1.0">
            <actions>
                <!-- Set environment variable so Python script knows
where to look -->
                <action type="set_environment">
                    <environment_variable name="EFFECTIVET3"
action="set_to">$INSTALL_DIR</environment_variable>
                </action>
                <!-- Main JAR file -->
                <action
type="download_by_url">http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action>
                <!-- Three model JAR files -->
                <action type="make_directory">$INSTALL_DIR/module</action>
                <action
type="download_by_url">http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                <action
type="download_by_url">http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                <action
type="download_by_url">http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
            </actions>
        </install>
        <readme>
Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/
        </readme>
    </package>
</tool_dependency>

Here my Python wrapper script has evidently read the $EFFECTIVET3
environment variable (defined in the tool_dependencies.xml to point
to the tool's $INSTALL_DIR), but the JAR file isn't there:

Effective T3 JAR file not found:
/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar


My previous revision set the $EFFECTIVET3 environment variable
at the end of the tool_dependencies.xml file, and from the test failure
it appeared not to be getting set. My hunch is something breaks
part way through the installation but the Tool Shed isn't reporting
this - or am I just not looking in the right place? It would make
sense to show any installation issues under the "Tool Dependencies"
entry on the main tool page.

Related to this, I'd like to suggest another couple of assertion action
types, used for checking if directories or files exist, which would
abort the installation with a clear error if the check fails:

<action type="assert_dir_exists">$INSTALL_DIR/module/</action>
<action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action>

An obvious application of these would be immediately after an
<action type="download_by_url"> line to sanity test the download
(and decompression) worked as expected - and didn't for example
just download an HTML error page for instance.

(In this case I am worried that perhaps the download_by_url
action has been overly enthusiastic and unzipped the JAR file,
perhaps based on the mime-type?).

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Björn Grüning
Hi Peter,

> Hi all,
>
> For defining the 'install script' for some of my tool wrappers, I have
> mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures
> and looking at one or two examples.
>
> I've got some of my tools to work, but not all. For instance, this one
> appears not to be downloading or moving JAR files correctly:
> http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965
>
> <?xml version="1.0"?>
> <tool_dependency>
>     <package name="effectiveT3" version="1.0.1">
>         <install version="1.0">
>             <actions>
>                 <!-- Set environment variable so Python script knows
> where to look -->
>                 <action type="set_environment">
>                     <environment_variable name="EFFECTIVET3"
> action="set_to">$INSTALL_DIR</environment_variable>
>                 </action>
>                 <!-- Main JAR file -->
>                 <action
> type="download_by_url">http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action>
>                 <action
> type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action>
>                 <!-- Three model JAR files -->
>                 <action type="make_directory">$INSTALL_DIR/module</action>
>                 <action
> type="download_by_url">http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action>
>                 <action
> type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
>                 <action
> type="download_by_url">http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action>
>                 <action
> type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
>                 <action
> type="download_by_url">http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action>
>                 <action
> type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
>             </actions>
>         </install>
>         <readme>
> Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/
>         </readme>
>     </package>
> </tool_dependency>

Afaik "download_by_url" is only supported as first action. To work
around that limitation, until its fixed, I used wget to fetch any
additional file.

Ciao,
Björn

> Here my Python wrapper script has evidently read the $EFFECTIVET3
> environment variable (defined in the tool_dependencies.xml to point
> to the tool's $INSTALL_DIR), but the JAR file isn't there:
>
> Effective T3 JAR file not found:
> /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar
>
>
> My previous revision set the $EFFECTIVET3 environment variable
> at the end of the tool_dependencies.xml file, and from the test failure
> it appeared not to be getting set. My hunch is something breaks
> part way through the installation but the Tool Shed isn't reporting
> this - or am I just not looking in the right place? It would make
> sense to show any installation issues under the "Tool Dependencies"
> entry on the main tool page.
>
> Related to this, I'd like to suggest another couple of assertion action
> types, used for checking if directories or files exist, which would
> abort the installation with a clear error if the check fails:
>
> <action type="assert_dir_exists">$INSTALL_DIR/module/</action>
> <action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action>
>
> An obvious application of these would be immediately after an
> <action type="download_by_url"> line to sanity test the download
> (and decompression) worked as expected - and didn't for example
> just download an HTML error page for instance.
>
> (In this case I am worried that perhaps the download_by_url
> action has been overly enthusiastic and unzipped the JAR file,
> perhaps based on the mime-type?).
>
> Thanks,
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
On Thu, May 9, 2013 at 4:57 PM, Björn Grüning
<[hidden email]> wrote:
> Hi Peter,
>
> Afaik "download_by_url" is only supported as first action. To work
> around that limitation, until its fixed, I used wget to fetch any
> additional file.
>
> Ciao,
> Björn

Interesting tip - I don't see that on the wiki but it sounds believable.
This would resolve by concerns about how this altered the current
working directory since (assuming I understand correctly) after
decompressing a tar ball, it changes into that new directory.

I'll try the wget alternative - thanks!

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Greg Von Kuster
Currently I feel that a tool dependency definition should allow for only a singe download of some item, and if that item requires other items. they should be defined as separate dependencies that are installed using the prior_installation_required=True attribute.

Is there a case where this is not possible?  If so, I'm willing to enable downloading more than 1 item per dependency package definition, but I think we have to be careful in how we go about this.

In any case, wget should not be used yet as it requires wget to be on the installation server, and we have not yet decided what 3rd party utilities we can assume exist in that environment.

If there is justification for allowing download_by_url multiple times in the same package dependency definition, then I would rather enhance the tool shed framework to support that.

Greg Von Kuster


On May 9, 2013, at 12:15 PM, Peter Cock wrote:

> On Thu, May 9, 2013 at 4:57 PM, Björn Grüning
> <[hidden email]> wrote:
>> Hi Peter,
>>
>> Afaik "download_by_url" is only supported as first action. To work
>> around that limitation, until its fixed, I used wget to fetch any
>> additional file.
>>
>> Ciao,
>> Björn
>
> Interesting tip - I don't see that on the wiki but it sounds believable.
> This would resolve by concerns about how this altered the current
> working directory since (assuming I understand correctly) after
> decompressing a tar ball, it changes into that new directory.
>
> I'll try the wget alternative - thanks!
>
> Peter
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Björn Grüning
Hi Greg and Peter,

two examples from my side.

One of my tools (osra) has 7 dependencies. A few of them, GraphicsMagick
for example, makes sense to include as separate repo. But I feel that
the other are too special.

- openbabel (is an extra repo)
- GraphicsMagick
- potrace
- gocr
- tclap
- ocrad
- cuneiform-linux

Osra, on its own, needs a patch to be applied to install with my
parameters. I used wget to fetch these patch. Maybe patch-action can be
created to download and patch a package. But other corner cases will
arise ;)  

Cheers,
Bjoern


> Currently I feel that a tool dependency definition should allow for only a singe download of some item, and if that item requires other items. they should be defined as separate dependencies that are installed using the prior_installation_required=True attribute.
>
> Is there a case where this is not possible?  If so, I'm willing to enable downloading more than 1 item per dependency package definition, but I think we have to be careful in how we go about this.
>
> In any case, wget should not be used yet as it requires wget to be on the installation server, and we have not yet decided what 3rd party utilities we can assume exist in that environment.
>
> If there is justification for allowing download_by_url multiple times in the same package dependency definition, then I would rather enhance the tool shed framework to support that.
>
> Greg Von Kuster
>
>
> On May 9, 2013, at 12:15 PM, Peter Cock wrote:
>
> > On Thu, May 9, 2013 at 4:57 PM, Björn Grüning
> > <[hidden email]> wrote:
> >> Hi Peter,
> >>
> >> Afaik "download_by_url" is only supported as first action. To work
> >> around that limitation, until its fixed, I used wget to fetch any
> >> additional file.
> >>
> >> Ciao,
> >> Björn
> >
> > Interesting tip - I don't see that on the wiki but it sounds believable.
> > This would resolve by concerns about how this altered the current
> > working directory since (assuming I understand correctly) after
> > decompressing a tar ball, it changes into that new directory.
> >
> > I'll try the wget alternative - thanks!
> >
> > Peter
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >  http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >  http://galaxyproject.org/search/mailinglists/
> >
>



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
In reply to this post by Greg Von Kuster
On Thu, May 9, 2013 at 6:05 PM, Greg Von Kuster <[hidden email]> wrote:
> Currently I feel that a tool dependency definition should allow for only a singe
> download of some item, and if that item requires other items. they should be
> defined as separate dependencies that are installed using the
> prior_installation_required=True attribute.

That seems too idealised: Not every tool is bundled up nicely for a
clean install, often there are bits and pieces which must be fetched
individually. In some cases this is a judgement call about when this
should be a separate dependency.

> Is there a case where this is not possible?  If so, I'm willing to enable
> downloading more than 1 item per dependency package definition, but I
> think we have to be careful in how we go about this.

Yes, effectiveT3 comes as four JAR files - the core tool and three models:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3

This is an example of a fairly common pattern where there is a core tool,
plus one or more essential data files to be downloaded individually.

> In any case, wget should not be used yet as it requires wget to be on the
> installation server, and we have not yet decided what 3rd party utilities we
> can assume exist in that environment.
>
> If there is justification for allowing download_by_url multiple times in the
> same package dependency definition, then I would rather enhance the
> tool shed framework to support that.

If you won't want us to use wget, then we will likely also need a
download_by_url variant which DOES NOT decompress anything.
I'm thinking of compressed data files which should stay compressed.

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
In reply to this post by Björn Grüning
On Thu, May 9, 2013 at 6:21 PM, Björn Grüning
<[hidden email]> wrote:
> Hi Greg and Peter,
>
> two examples from my side.
>
> One of my tools (osra) has 7 dependencies. A few of them, GraphicsMagick
> for example, makes sense to include as separate repo. But I feel that
> the other are too special.

I made the same judgement call with clinod (command line NoD), which
needs a single binary tool 'batchman' (either on the path or next to the
clinod install) from a 3rd party package, the Stuttgart Neural Network
Simulator (SNNS).

http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/

I could have tried to work out how to download and install the whole of
the SNNS package, but that in turn would trigger more dependencies.
Instead I am just installing batchman by putting it in the clinod
$INSTALL_DIR - this is far simpler and less likely to break.

(If it turns out in six months time that some other Galaxy Tools also
use bits of SNNS, then I'd reconsider this choice.)

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
In reply to this post by Peter Cock
On Thu, May 9, 2013 at 4:39 PM, Peter Cock <[hidden email]> wrote:

> Hi all,
>
> For defining the 'install script' for some of my tool wrappers, I have
> mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures
> and looking at one or two examples.
>
> I've got some of my tools to work, but not all. For instance, this one
> appears not to be downloading or moving JAR files correctly:
> http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965
>
> ...
>
> Here my Python wrapper script has evidently read the $EFFECTIVET3
> environment variable (defined in the tool_dependencies.xml to point
> to the tool's $INSTALL_DIR), but the JAR file isn't there:
>
> Effective T3 JAR file not found:
> /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar
>
> My previous revision set the $EFFECTIVET3 environment variable
> at the end of the tool_dependencies.xml file, and from the test failure
> it appeared not to be getting set. My hunch is something breaks
> part way through the installation but the Tool Shed isn't reporting
> this - or am I just not looking in the right place? It would make
> sense to show any installation issues under the "Tool Dependencies"
> entry on the main tool page.
>
> Related to this, I'd like to suggest another couple of assertion action
> types, used for checking if directories or files exist, which would
> abort the installation with a clear error if the check fails:
>
> <action type="assert_dir_exists">$INSTALL_DIR/module/</action>
> <action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action>
>
> An obvious application of these would be immediately after an
> <action type="download_by_url"> line to sanity test the download
> (and decompression) worked as expected - and didn't for example
> just download an HTML error page for instance.
>
> (In this case I am worried that perhaps the download_by_url
> action has been overly enthusiastic and unzipped the JAR file,
> perhaps based on the mime-type?).
>
> Thanks,
>
> Peter

Thanks for the input Björn and Greg about not being able to make
multiple calls to download_by_url - I've tried tweaking both the
effectivet3 and clinod tool_dependencies.xml files to use wget
but they are still failing:

http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5
http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63

I believe that either I have a simple error in both install descriptions,
leading to the install to 'work' but not put things where I expect them
(which is my problem), or the install is really failing yet there is no
feedback about this (which is a Galaxy Tool Shed problem).

The idea of my new <action> types for checking if files and folders
exist is to catch some types of install failure as early as possible -
in this case verify the install put the relevant files where it was
intended to.

Could you direct me at the source code which handles running
the tool_dependencies.xm install scripts? I'd like to look at what
kind of error handling it does (e.g. I'd hope non-zero return codes
from a shell_command <action> would be treated as an error),
and how easy it would be to add the new actions myself.

Thanks,

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Björn Grüning
Hi Peter,

> > For defining the 'install script' for some of my tool wrappers, I have
> > mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures
> > and looking at one or two examples.
> >
> > I've got some of my tools to work, but not all. For instance, this one
> > appears not to be downloading or moving JAR files correctly:
> > http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965
> >
> > ...
> >
> > Here my Python wrapper script has evidently read the $EFFECTIVET3
> > environment variable (defined in the tool_dependencies.xml to point
> > to the tool's $INSTALL_DIR), but the JAR file isn't there:
> >
> > Effective T3 JAR file not found:
> > /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar
> >
> > My previous revision set the $EFFECTIVET3 environment variable
> > at the end of the tool_dependencies.xml file, and from the test failure
> > it appeared not to be getting set. My hunch is something breaks
> > part way through the installation but the Tool Shed isn't reporting
> > this - or am I just not looking in the right place? It would make
> > sense to show any installation issues under the "Tool Dependencies"
> > entry on the main tool page.
> >
> > Related to this, I'd like to suggest another couple of assertion action
> > types, used for checking if directories or files exist, which would
> > abort the installation with a clear error if the check fails:
> >
> > <action type="assert_dir_exists">$INSTALL_DIR/module/</action>
> > <action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action>
> >
> > An obvious application of these would be immediately after an
> > <action type="download_by_url"> line to sanity test the download
> > (and decompression) worked as expected - and didn't for example
> > just download an HTML error page for instance.
> >
> > (In this case I am worried that perhaps the download_by_url
> > action has been overly enthusiastic and unzipped the JAR file,
> > perhaps based on the mime-type?).
> >
> > Thanks,
> >
> > Peter
>
> Thanks for the input Björn and Greg about not being able to make
> multiple calls to download_by_url - I've tried tweaking both the
> effectivet3 and clinod tool_dependencies.xml files to use wget
> but they are still failing:
>
> http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5
> http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63

I'm not sure, but can you try to fetch the first jar file with
download_by_url. The last time I was struggling with that the
download_by_url was required, if I remember correctly.

> I believe that either I have a simple error in both install descriptions,
> leading to the install to 'work' but not put things where I expect them
> (which is my problem), or the install is really failing yet there is no
> feedback about this (which is a Galaxy Tool Shed problem).

> The idea of my new <action> types for checking if files and folders
> exist is to catch some types of install failure as early as possible -
> in this case verify the install put the relevant files where it was
> intended to.
>
> Could you direct me at the source code which handles running
> the tool_dependencies.xm install scripts? I'd like to look at what
> kind of error handling it does (e.g. I'd hope non-zero return codes
> from a shell_command <action> would be treated as an error),
> and how easy it would be to add the new actions myself.

Have a look at:

lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py

Hope that helps!
Björn

> Thanks,
>
> Peter
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Dave Bouvier
Björn, Peter,

I've created a trello card (https://trello.com/c/QY0Z7L7p) for this
requirement, and will be working on enhancing that feature as my next
priority.

    --Dave B.

On 5/10/13 09:59:37.000, Björn Grüning wrote:

> Hi Peter,
>
>>> For defining the 'install script' for some of my tool wrappers, I have
>>> mainly following http://wiki.galaxyproject.org/ToolShedToolFeatures
>>> and looking at one or two examples.
>>>
>>> I've got some of my tools to work, but not all. For instance, this one
>>> appears not to be downloading or moving JAR files correctly:
>>> http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/5644c28cf965
>>>
>>> ...
>>>
>>> Here my Python wrapper script has evidently read the $EFFECTIVET3
>>> environment variable (defined in the tool_dependencies.xml to point
>>> to the tool's $INSTALL_DIR), but the JAR file isn't there:
>>>
>>> Effective T3 JAR file not found:
>>> /var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpYR9FXD/effectiveT3/1.0.1/peterjc/effectivet3/5644c28cf965/TTSS_GUI-1.0.1.jar
>>>
>>> My previous revision set the $EFFECTIVET3 environment variable
>>> at the end of the tool_dependencies.xml file, and from the test failure
>>> it appeared not to be getting set. My hunch is something breaks
>>> part way through the installation but the Tool Shed isn't reporting
>>> this - or am I just not looking in the right place? It would make
>>> sense to show any installation issues under the "Tool Dependencies"
>>> entry on the main tool page.
>>>
>>> Related to this, I'd like to suggest another couple of assertion action
>>> types, used for checking if directories or files exist, which would
>>> abort the installation with a clear error if the check fails:
>>>
>>> <action type="assert_dir_exists">$INSTALL_DIR/module/</action>
>>> <action type="assert_file_exists">$INSTALL_DIR/module/TTSS_STD-1.0.1.jar</action>
>>>
>>> An obvious application of these would be immediately after an
>>> <action type="download_by_url"> line to sanity test the download
>>> (and decompression) worked as expected - and didn't for example
>>> just download an HTML error page for instance.
>>>
>>> (In this case I am worried that perhaps the download_by_url
>>> action has been overly enthusiastic and unzipped the JAR file,
>>> perhaps based on the mime-type?).
>>>
>>> Thanks,
>>>
>>> Peter
>>
>> Thanks for the input Björn and Greg about not being able to make
>> multiple calls to download_by_url - I've tried tweaking both the
>> effectivet3 and clinod tool_dependencies.xml files to use wget
>> but they are still failing:
>>
>> http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/c70df461aae5
>> http://testtoolshed.g2.bx.psu.edu/view/peterjc/clinod/20e44218fa63
>
> I'm not sure, but can you try to fetch the first jar file with
> download_by_url. The last time I was struggling with that the
> download_by_url was required, if I remember correctly.
>
>> I believe that either I have a simple error in both install descriptions,
>> leading to the install to 'work' but not put things where I expect them
>> (which is my problem), or the install is really failing yet there is no
>> feedback about this (which is a Galaxy Tool Shed problem).
>
>> The idea of my new <action> types for checking if files and folders
>> exist is to catch some types of install failure as early as possible -
>> in this case verify the install put the relevant files where it was
>> intended to.
>>
>> Could you direct me at the source code which handles running
>> the tool_dependencies.xm install scripts? I'd like to look at what
>> kind of error handling it does (e.g. I'd hope non-zero return codes
>> from a shell_command <action> would be treated as an error),
>> and how easy it would be to add the new actions myself.
>
> Have a look at:
>
> lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
>
> Hope that helps!
> Björn
>
>> Thanks,
>>
>> Peter
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
In reply to this post by Björn Grüning
On Fri, May 10, 2013 at 2:59 PM, Björn Grüning
<[hidden email]> wrote:

> Hi Peter,
>
>> I believe that either I have a simple error in both install descriptions,
>> leading to the install to 'work' but not put things where I expect them
>> (which is my problem), or the install is really failing yet there is no
>> feedback about this (which is a Galaxy Tool Shed problem).
>
>> The idea of my new <action> types for checking if files and folders
>> exist is to catch some types of install failure as early as possible -
>> in this case verify the install put the relevant files where it was
>> intended to.
>>
>> Could you direct me at the source code which handles running
>> the tool_dependencies.xm install scripts? I'd like to look at what
>> kind of error handling it does (e.g. I'd hope non-zero return codes
>> from a shell_command <action> would be treated as an error),
>> and how easy it would be to add the new actions myself.
>
> Have a look at:
>
> lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
>
> Hope that helps!
> Björn

That is educational, thanks Björn :)

The good news is that the shell_command handling code does
check the return code and record an error if it failed as part of
the tool_dependency object (status and error_message).

The bad news is that in general install_and_build_package
doesn't seem to catch any other errors - for instance a
failing download, corrupt zip file, invalid path name, etc.

Further bad news, it looks like by guess about the JAR
files wrongly being unzipped was accurate - digging
down the imports, the common_util.iszip function seems
to just be doing this:

>>> import zipfile
>>> zipfile.is_zipfile("/opt/clinod/clinod-1.3.jar")
True

Therefore (right now) as I feared, we cannot use the
download_by_url action with JAR files because it will
unzip them. As suggested earlier, one fix would be an
option for controlling any decompression (defaulting to
automatic as now, but allowing it to be explicitly set to
true or false?).

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Dave Bouvier
Peter,

Yes, jar files are just a directory of .class (and other) files packaged
with the zip algorithm, which has its benefits and drawbacks, as you've
discovered. In the new action type I'm working on, I do intend to
include the option to extract a downloaded file, or leave it as-is,
which will allow the download of jar files needed at build time or run
time, as well as any other compressed file that should not be
automatically extracted.

    --Dave B.

On 5/10/13 11:13:36.000, Peter Cock wrote:

> On Fri, May 10, 2013 at 2:59 PM, Björn Grüning
> <[hidden email]> wrote:
>> Hi Peter,
>>
>>> I believe that either I have a simple error in both install descriptions,
>>> leading to the install to 'work' but not put things where I expect them
>>> (which is my problem), or the install is really failing yet there is no
>>> feedback about this (which is a Galaxy Tool Shed problem).
>>
>>> The idea of my new <action> types for checking if files and folders
>>> exist is to catch some types of install failure as early as possible -
>>> in this case verify the install put the relevant files where it was
>>> intended to.
>>>
>>> Could you direct me at the source code which handles running
>>> the tool_dependencies.xm install scripts? I'd like to look at what
>>> kind of error handling it does (e.g. I'd hope non-zero return codes
>>> from a shell_command <action> would be treated as an error),
>>> and how easy it would be to add the new actions myself.
>>
>> Have a look at:
>>
>> lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
>>
>> Hope that helps!
>> Björn
>
> That is educational, thanks Björn :)
>
> The good news is that the shell_command handling code does
> check the return code and record an error if it failed as part of
> the tool_dependency object (status and error_message).
>
> The bad news is that in general install_and_build_package
> doesn't seem to catch any other errors - for instance a
> failing download, corrupt zip file, invalid path name, etc.
>
> Further bad news, it looks like by guess about the JAR
> files wrongly being unzipped was accurate - digging
> down the imports, the common_util.iszip function seems
> to just be doing this:
>
>>>> import zipfile
>>>> zipfile.is_zipfile("/opt/clinod/clinod-1.3.jar")
> True
>
> Therefore (right now) as I feared, we cannot use the
> download_by_url action with JAR files because it will
> unzip them. As suggested earlier, one fix would be an
> option for controlling any decompression (defaulting to
> automatic as now, but allowing it to be explicitly set to
> true or false?).
>
> Peter
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Dave Bouvier
In reply to this post by Peter Cock
Peter,

As of 9696:b4733e42a2c9, Galaxy now supports an additional tool
dependency installation action called download_file, which downloads a
file from the specified URL and saves it to the current working
directory, without extracting it. The format is:

<action type="download_file">[url]</action>

For your effectivet3 repository, the recommended tool_dependencies.xml
now looks like:

<?xml version="1.0"?>
<tool_dependency>
     <package name="effectiveT3" version="1.0.1">
         <install version="1.0">
             <actions>
                 <!-- Download jar files required by this tool. -->
                 <action
type="download_file">http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action>
                 <action
type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action>
                 <action type="make_directory">$INSTALL_DIR/module</action>
                 <action
type="download_file">http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action>
                 <action
type="download_file">http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action>
                 <action
type="download_file">http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action>
                 <action
type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                 <action
type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                 <action
type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                 <!-- Set environment variable so Python script knows
where to look -->
                 <action type="set_environment">
                     <environment_variable name="EFFECTIVET3"
action="set_to">$INSTALL_DIR</environment_variable>
                 </action>
             </actions>
         </install>
         <readme>
Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/
         </readme>
     </package>
</tool_dependency>

An enhancement I intend to make on Monday is the option to specify a
target path and filename, which will eliminate the need for the
move_file steps you see in the above XML.

As this is a new feature, if you discover any bugs or anomalous
behavior, please feel free to let me know, and I'll get that resolved as
soon as I can.

    --Dave B.

On 5/10/13 11:13:36.000, Peter Cock wrote:

> On Fri, May 10, 2013 at 2:59 PM, Björn Grüning
> <[hidden email]> wrote:
>> Hi Peter,
>>
>>> I believe that either I have a simple error in both install descriptions,
>>> leading to the install to 'work' but not put things where I expect them
>>> (which is my problem), or the install is really failing yet there is no
>>> feedback about this (which is a Galaxy Tool Shed problem).
>>
>>> The idea of my new <action> types for checking if files and folders
>>> exist is to catch some types of install failure as early as possible -
>>> in this case verify the install put the relevant files where it was
>>> intended to.
>>>
>>> Could you direct me at the source code which handles running
>>> the tool_dependencies.xm install scripts? I'd like to look at what
>>> kind of error handling it does (e.g. I'd hope non-zero return codes
>>> from a shell_command <action> would be treated as an error),
>>> and how easy it would be to add the new actions myself.
>>
>> Have a look at:
>>
>> lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
>>
>> Hope that helps!
>> Björn
>
> That is educational, thanks Björn :)
>
> The good news is that the shell_command handling code does
> check the return code and record an error if it failed as part of
> the tool_dependency object (status and error_message).
>
> The bad news is that in general install_and_build_package
> doesn't seem to catch any other errors - for instance a
> failing download, corrupt zip file, invalid path name, etc.
>
> Further bad news, it looks like by guess about the JAR
> files wrongly being unzipped was accurate - digging
> down the imports, the common_util.iszip function seems
> to just be doing this:
>
>>>> import zipfile
>>>> zipfile.is_zipfile("/opt/clinod/clinod-1.3.jar")
> True
>
> Therefore (right now) as I feared, we cannot use the
> download_by_url action with JAR files because it will
> unzip them. As suggested earlier, one fix would be an
> option for controlling any decompression (defaulting to
> automatic as now, but allowing it to be explicitly set to
> true or false?).
>
> Peter
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Björn Grüning
Hi Dave,

that was fast.

Thanks a lot!
Björn


> <?xml version="1.0"?>
> <tool_dependency>
>      <package name="effectiveT3" version="1.0.1">
>          <install version="1.0">
>              <actions>
>                  <!-- Download jar files required by this tool. -->
>                  <action
> type="download_file">http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action>
>                  <action
> type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action>
>                  <action type="make_directory">$INSTALL_DIR/module</action>
>                  <action
> type="download_file">http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action>
>                  <action
> type="download_file">http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action>
>                  <action
> type="download_file">http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action>
>                  <action
> type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
>                  <action
> type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
>                  <action
> type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
>                  <!-- Set environment variable so Python script knows
> where to look -->
>                  <action type="set_environment">
>                      <environment_variable name="EFFECTIVET3"
> action="set_to">$INSTALL_DIR</environment_variable>
>                  </action>
>              </actions>
>          </install>
>          <readme>
> Downloads effectiveT3 v1.0.1 and the three models from http://effectors.org/
>          </readme>
>      </package>
> </tool_dependency>
>
> An enhancement I intend to make on Monday is the option to specify a
> target path and filename, which will eliminate the need for the
> move_file steps you see in the above XML.
>
> As this is a new feature, if you discover any bugs or anomalous
> behavior, please feel free to let me know, and I'll get that resolved as
> soon as I can.
>
>     --Dave B.
>
> On 5/10/13 11:13:36.000, Peter Cock wrote:
> > On Fri, May 10, 2013 at 2:59 PM, Björn Grüning
> > <[hidden email]> wrote:
> >> Hi Peter,
> >>
> >>> I believe that either I have a simple error in both install descriptions,
> >>> leading to the install to 'work' but not put things where I expect them
> >>> (which is my problem), or the install is really failing yet there is no
> >>> feedback about this (which is a Galaxy Tool Shed problem).
> >>
> >>> The idea of my new <action> types for checking if files and folders
> >>> exist is to catch some types of install failure as early as possible -
> >>> in this case verify the install put the relevant files where it was
> >>> intended to.
> >>>
> >>> Could you direct me at the source code which handles running
> >>> the tool_dependencies.xm install scripts? I'd like to look at what
> >>> kind of error handling it does (e.g. I'd hope non-zero return codes
> >>> from a shell_command <action> would be treated as an error),
> >>> and how easy it would be to add the new actions myself.
> >>
> >> Have a look at:
> >>
> >> lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
> >>
> >> Hope that helps!
> >> Björn
> >
> > That is educational, thanks Björn :)
> >
> > The good news is that the shell_command handling code does
> > check the return code and record an error if it failed as part of
> > the tool_dependency object (status and error_message).
> >
> > The bad news is that in general install_and_build_package
> > doesn't seem to catch any other errors - for instance a
> > failing download, corrupt zip file, invalid path name, etc.
> >
> > Further bad news, it looks like by guess about the JAR
> > files wrongly being unzipped was accurate - digging
> > down the imports, the common_util.iszip function seems
> > to just be doing this:
> >
> >>>> import zipfile
> >>>> zipfile.is_zipfile("/opt/clinod/clinod-1.3.jar")
> > True
> >
> > Therefore (right now) as I feared, we cannot use the
> > download_by_url action with JAR files because it will
> > unzip them. As suggested earlier, one fix would be an
> > option for controlling any decompression (defaulting to
> > automatic as now, but allowing it to be explicitly set to
> > true or false?).
> >
> > Peter
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >    http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >    http://galaxyproject.org/search/mailinglists/
> >
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
On Fri, May 10, 2013 at 8:11 PM, Björn Grüning
<[hidden email]> wrote:
> Hi Dave,
>
> that was fast.
>
> Thanks a lot!
> Björn

Thanks Dave,

I see this also should fix handling of JAR files in download_by_url :)
https://bitbucket.org/galaxy/galaxy-central/commits/b4733e42a2c98c42401bca5e0643fc4d4dcf2db6

Could I suggest an enhancement to abort on undefined action types,
rough patch below (untested)? This will help give a clear error if a new
command is used but the Galaxy host is too old to implement it (which
is exactly the situation we'd see right now on the main Tool Shed if
the new download_file action is used).

Regards,

Peter


$ hg diff
diff -r 65a81aead95e
lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
--- a/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Sun
May 12 11:52:55 2013 -0400
+++ b/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py Mon
May 13 10:29:17 2013 +0100
@@ -152,6 +152,12 @@
                             url = action_dict[ 'url' ]
                             filename = url.split( '/' )[ -1 ]
                             common_util.url_download( current_dir,
filename, url )
+                        else:
+                            # This will happen if the
tool_dependencies.xml is using a very new action:
+                            tool_dependency.status =
app.model.ToolDependency.installation_status.ERROR
+                            tool_dependency.error_message =
"Undefined action type %r" % action_type
+                            return
+

 def log_results( command, fabric_AttributeString, file_path ):
     """

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
On Mon, May 13, 2013 at 10:34 AM, Peter Cock <[hidden email]> wrote:

> On Fri, May 10, 2013 at 8:11 PM, Björn Grüning
> <[hidden email]> wrote:
>> Hi Dave,
>>
>> that was fast.
>>
>> Thanks a lot!
>> Björn
>
> Thanks Dave,
>
> I see this also should fix handling of JAR files in download_by_url :)
> https://bitbucket.org/galaxy/galaxy-central/commits/b4733e42a2c98c42401bca5e0643fc4d4dcf2db6
>
> Could I suggest an enhancement to abort on undefined action types,
> rough patch below (untested)? This will help give a clear error if a new
> command is used but the Galaxy host is too old to implement it (which
> is exactly the situation we'd see right now on the main Tool Shed if
> the new download_file action is used).
>
> Regards,
>
> Peter

Another data point, with my old wget based tool_wrapper.xml file:
http://testtoolshed.g2.bx.psu.edu/view/peterjc/effectivet3/b863c4bec840

>From the test failure of the previous commit I suspected this, but this
revision checks specifically for the 'module' subdirectory and confirms
it is not being created:

Effective T3 module folder not found:
'/var/opt/buildslaves/buildslave-ec2-1/buildbot-install-test-test-tool-shed-py27/build/test/install_and_test_tool_shed_repositories/tmp/tmpAzkY3r/effectiveT3/1.0.1/peterjc/effectivet3/b863c4bec840/module'

I infer the 'install script' is failing silently after setting the environment
variable but before or when creating the sub-directory $INSTALL_DIR/module,

           <actions>
                <!-- Set environment variable so Python script knows
where to look -->
                <action type="set_environment">
                    <environment_variable name="EFFECTIVET3"
action="set_to">$INSTALL_DIR</environment_variable>
                </action>
                <!-- Main JAR file -->
                <action type="shell_command">wget
http://effectors.org/download/version/TTSS_GUI-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_GUI-1.0.1.jar</source><destination>$INSTALL_DIR/</destination></action>
                <!-- Three model JAR files -->
                <action type="make_directory">$INSTALL_DIR/module</action>
                <action type="shell_command">wget
http://effectors.org/download/module/TTSS_ANIMAL-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_ANIMAL-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                <action type="shell_command">wget
http://effectors.org/download/module/TTSS_PLANT-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_PLANT-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
                <action type="shell_command">wget
http://effectors.org/download/module/TTSS_STD-1.0.1.jar</action>
                <action
type="move_file"><source>TTSS_STD-1.0.1.jar</source><destination>$INSTALL_DIR/module/</destination></action>
            </actions>

The first possible cause of failure is the set_environment action,
but from the limited information from the failing test that worked.

The second possible causes of failure is the wget call, perhaps that
command line tool isn't even installed? However, from the limited
information reported via the failing test I know the JAR file was
downloaded (and moved into place).

The third possible failure is the move_file action, and again I know
from the limited information reported via the failing test that
the TTSS_GUI-1.0.1.jar file was moved into place.

My suspicion therefore falls on the make_directory action - my
debug output from the failing test indicates the expected directory
was not created.

However, any exception within fabric_util.install_and_build_package
should be caught by install_util.install_and_build_package_via_fabric
and recorded in the tool_dependency object.

Tracing up the call tree, I found this puzzling bit in the
common_install_util.py file:

https://bitbucket.org/galaxy/galaxy-central/src/65a81aead95e147f709ee3969d49766f35d6a2e2/lib/tool_shed/util/common_install_util.py?at=default

def handle_tool_dependencies( app, tool_shed_repository,
tool_dependencies_config, tool_dependencies ):
    ...
                if tool_dependency.can_install:
                    try:
                        tool_dependency = install_package( app, elem,
tool_shed_repository, tool_dependencies=tool_dependencies )
                    except Exception, e:
                        error_message = "Error installing tool
dependency %s version %s: %s" % ( str( package_name ), str(
package_version ), str( e ) )
                        log.debug( error_message )
                        if tool_dependency:
                            tool_dependency.status =
app.model.ToolDependency.installation_status.ERROR
                            tool_dependency.error_message = error_message
                            sa_session.add( tool_dependency )
                            sa_session.flush()
                    if tool_dependency and tool_dependency.status in [
app.model.ToolDependency.installation_status.INSTALLED,

app.model.ToolDependency.installation_status.ERROR ]:
                        installed_tool_dependencies.append( tool_dependency )

It seems even if install_package fails, the tool_dependency is still
considered to be installed? If so, why?

This would explain the behaviour I am seeing with effectivet3 and
other tools on the Tool Shed - despite the tool_dependencies.xml
seeming to fail part way though, the tests are run anyway.

(I would instead expect the Tool Shed to tell me the install failed,
and not bother trying to run the tests, which are probably going to
fail anyway).

Regards,

Peter

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
On Mon, May 13, 2013 at 11:42 AM, Peter Cock <[hidden email]> wrote:

>
> ...
>
> This would explain the behaviour I am seeing with effectivet3 and
> other tools on the Tool Shed - despite the tool_dependencies.xml
> seeming to fail part way though, the tests are run anyway.
>
> (I would instead expect the Tool Shed to tell me the install failed,
> and not bother trying to run the tests, which are probably going to
> fail anyway).

It looks like this is coming soon - including displaying something
about the install failure reasons if I'm reading this correctly :)
https://bitbucket.org/galaxy/galaxy-central/commits/ef2b705c7de3

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
On Mon, May 13, 2013 at 8:35 PM, Peter Cock <[hidden email]> wrote:

> On Mon, May 13, 2013 at 11:42 AM, Peter Cock <[hidden email]> wrote:
>>
>> ...
>>
>> This would explain the behaviour I am seeing with effectivet3 and
>> other tools on the Tool Shed - despite the tool_dependencies.xml
>> seeming to fail part way though, the tests are run anyway.
>>
>> (I would instead expect the Tool Shed to tell me the install failed,
>> and not bother trying to run the tests, which are probably going to
>> fail anyway).
>
> It looks like this is coming soon - including displaying something
> about the install failure reasons if I'm reading this correctly :)
> https://bitbucket.org/galaxy/galaxy-central/commits/ef2b705c7de3

I'm seeing the new install failures section on some tools now :)
(I've just updated the tool concerned, so hopefully that install
error will go away).

Thanks,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Greg Von Kuster
Hi Peter,

I'm going to be enhancing this new feature further today, creating a new list "Latest revision: installation errors", so this category will be extracted from the current "failing tests" category.  At the same time I'll be adding the new feature allowing the owner of a repository to set the "skip test" flag.

Greg Von Kuster

On May 15, 2013, at 9:08 AM, Peter Cock wrote:

> On Mon, May 13, 2013 at 8:35 PM, Peter Cock <[hidden email]> wrote:
>> On Mon, May 13, 2013 at 11:42 AM, Peter Cock <[hidden email]> wrote:
>>>
>>> ...
>>>
>>> This would explain the behaviour I am seeing with effectivet3 and
>>> other tools on the Tool Shed - despite the tool_dependencies.xml
>>> seeming to fail part way though, the tests are run anyway.
>>>
>>> (I would instead expect the Tool Shed to tell me the install failed,
>>> and not bother trying to run the tests, which are probably going to
>>> fail anyway).
>>
>> It looks like this is coming soon - including displaying something
>> about the install failure reasons if I'm reading this correctly :)
>> https://bitbucket.org/galaxy/galaxy-central/commits/ef2b705c7de3
>
> I'm seeing the new install failures section on some tools now :)
> (I've just updated the tool concerned, so hopefully that install
> error will go away).
>
> Thanks,
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Handling of tool_dependencies.xml errors in Tool Shed testing

Peter Cock
In reply to this post by Peter Cock
Hi again,

I've realised that my current Blast2GO install script would be a perfect
example of where this tweak to fabric_util.py would help - when I wrote
the tool_dependencies.xml I didn't appreciate that the download_by_url
action was only allowed as the first action - currently it is ignored if listed
later in the script (just as any unknown actions are currently ignored).

http://testtoolshed.g2.bx.psu.edu/view/peterjc/blast2go/6e7694a0ae00

(I'm about to try and fix this tool_dependencies.xml file)

Peter

On Mon, May 13, 2013 at 10:34 AM, Peter Cock <[hidden email]> wrote:

>
>
> Could I suggest an enhancement to abort on undefined action types,
> rough patch below (untested)? This will help give a clear error if a new
> command is used but the Galaxy host is too old to implement it (which
> is exactly the situation we'd see right now on the main Tool Shed if
> the new download_file action is used).
>
> Regards,
>
> Peter
>
>
> $ hg diff
> diff -r 65a81aead95e
> lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py
> --- a/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py     Sun
> May 12 11:52:55 2013 -0400
> +++ b/lib/tool_shed/galaxy_install/tool_dependencies/fabric_util.py     Mon
> May 13 10:29:17 2013 +0100
> @@ -152,6 +152,12 @@
>                              url = action_dict[ 'url' ]
>                              filename = url.split( '/' )[ -1 ]
>                              common_util.url_download( current_dir,
> filename, url )
> +                        else:
> +                            # This will happen if the
> tool_dependencies.xml is using a very new action:
> +                            tool_dependency.status =
> app.model.ToolDependency.installation_status.ERROR
> +                            tool_dependency.error_message =
> "Undefined action type %r" % action_type
> +                            return
> +
>
>  def log_results( command, fabric_AttributeString, file_path ):
>      """
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/