Incorrect revision getting referenced in a complex tool dependency - why?

classic Classic list List threaded Threaded
4 messages Options
| Threaded
Open this post in threaded view
|

Incorrect revision getting referenced in a complex tool dependency - why?

Melissa Cline
Hi Folks,

Okay, I really need debugging ideas on this one.

I have three repositories I'm developing under the test toolshed.  They're named start_xena, xena_import and xena_find_datasets (they're all under visualization).  start_xena contains a simple tool dependency to a package named installXena.  xena_import and xena_find_datasets contain complex dependencies to installXena.  I've installed these tools on two computers.  On one, everything works great - amazingly well, in fact!  Kudos, Development Team!  On the other computer, the env.sh files associated with the complex dependencies keep referencing the wrong version of start_xena/installXena.

Here are the gory details.  

The start_xena tool is installed in testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena.  I'm a bit confused about the revision part of the path, because the latest revision is different (75c7d80df9c1), and I have uninstalled and re-installed the tool since its last update.  Its package, installXena, is installed under tool_dependencies at tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/.  That directory contains all the files I'd expect to be there, and its env.sh file contains all the environment variables I'd intended to set.   So far, so good.

xena_import is installed on my computer at testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import.  Its tool_dependencies.xml reference installXena under start_xena as:

<?xml version="1.0"?>

<tool_dependency>

  <package name="installXena" version="1.0">

    <repository toolshed="http://testtoolshed.g2.bx.psu.edu" name="start_xena" owner="melissacline" changeset_revision="82755b0ee5a5"/>

  </package>

</tool_dependency>

(Aside: is it necessary to have the changeset_revision in there?  I'm a bit worried about the maintenance task of updating it for all of my dependent tools every time I update start_xena, but as I'll explain, it's not clear that this revision info is getting used).

The tool dependency dir for xena_import is at tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/.  Its env.sh reads:

if [ -f /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh ] ; then . /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh ; fi

Notice that the wrong revision number is in there.  The 06 revision was an older one.  Needless to say, the files and environment variables I need aren't in the 06 directory.

Where does this incorrect revision number come from?  I've deleted all the 06 directories, repeatedly.  I've deleted and reinstalled the tools, and verified that after deletion, the directories in question were in fact gone.  I've tried resetting the metadata for these tools under the Galaxy Admin menu.  Now I need new ideas.  As I mentioned, all of this works beautifully on a second computer.  It's as if the first Galaxy installation, on the first computer, got stuck with some outdated information, and I haven't figured out how to clear it out.

Thanks!

Melissa



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Incorrect revision getting referenced in a complex tool dependency - why?

Björn Grüning-3
Hi Melissa,

a few remarks to your wrappers, hopefully it will fix a few issues.

* If you put your python files and your java files next to your wrapper
you do not need to install them. I think you can remove the
tool_dependency file and it should work.
* if you stick to this approach you should rename JAVA_JAR_PATH to
XENA_JAR_PATH or something like this
* do you really need the dependencies in the other repositories?
Dependencies are used to access binaries or libaries from other
installations, as far as I could see you are accessing a local port or?
So no dependency
* Question: do you need the server running all the time, or can you that
it during the tool execution (find, import)? We need a proper mechanism
to support such things like servers (on some ports). You will here have
the issue that multiple users can't start a server at once. We need to
get devteam help here. Would you be so kind and create a trello card for it?


> Okay, I really need debugging ideas on this one.
>
> I have three repositories I'm developing under the test toolshed.  They're
> named start_xena, xena_import and xena_find_datasets (they're all under
> visualization).  start_xena contains a simple tool dependency to a package
> named installXena.  xena_import and xena_find_datasets contain complex
> dependencies to installXena.  I've installed these tools on two computers.
>   On one, everything works great - amazingly well, in fact!  Kudos,
> Development Team!  On the other computer, the env.sh files associated with
> the complex dependencies keep referencing the wrong version of
> start_xena/installXena.
>
> Here are the gory details.
>
> The start_xena tool is installed in
> testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena.
>   I'm a bit confused about the revision part of the path, because the latest
> revision is different (75c7d80df9c1), and I have uninstalled and
> re-installed the tool since its last update.  Its package, installXena, is
> installed under tool_dependencies at
> tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/.
>   That directory contains all the files I'd expect to be there, and its
> env.sh file contains all the environment variables I'd intended to set.
> So far, so good.
>
> xena_import is installed on my computer at
> testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import.
>   Its tool_dependencies.xml reference installXena under start_xena as:
>
> <?xml version="1.0"?>
>
> <tool_dependency>
>
>    <package name="installXena" version="1.0">
>
>      <repository toolshed="http://testtoolshed.g2.bx.psu.edu"
> name="start_xena" owner="melissacline" changeset_revision="82755b0ee5a5"/>
>
>    </package>
>
> </tool_dependency>
>
> (Aside: is it necessary to have the changeset_revision in there?  I'm a bit
> worried about the maintenance task of updating it for all of my dependent
> tools every time I update start_xena, but as I'll explain, it's not clear
> that this revision info is getting used).
>
> The tool dependency dir for xena_import is at
> tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/.
>   Its env.sh reads:
>
> if [ -f
> /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
> ] ; then .
> /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
> ; fi
>
> Notice that the wrong revision number is in there.  The 06 revision was an
> older one.  Needless to say, the files and environment variables I need
> aren't in the 06 directory.
>
> Where does this incorrect revision number come from?  I've deleted all the
> 06 directories, repeatedly.  I've deleted and reinstalled the tools, and
> verified that after deletion, the directories in question were in fact
> gone.  I've tried resetting the metadata for these tools under the Galaxy
> Admin menu.  Now I need new ideas.  As I mentioned, all of this works
> beautifully on a second computer.  It's as if the first Galaxy
> installation, on the first computer, got stuck with some outdated
> information, and I haven't figured out how to clear it out.

I don't know but you should remove the toolshed= and the revision= from
your XML files. These fields will be automatically filled by the
toolshed (inserting the latest tip version of your dependencies and the
current toolshed url).

Hope this helps a little bit!
Bjoern

> Thanks!
>
> Melissa
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Incorrect revision getting referenced in a complex tool dependency - why?

Dave Bouvier
In reply to this post by Melissa Cline
Melissa,

The revision recorded in the filesystem path can seem odd at first
glance. My inline responses should hopefully clarify that.

    --Dave B.

On 09/10/2014 05:49 PM, Melissa Cline wrote:

> Hi Folks,
>
> Okay, I really need debugging ideas on this one.
>
> I have three repositories I'm developing under the test toolshed.
>   They're named start_xena, xena_import and xena_find_datasets (they're
> all under visualization).  start_xena contains a simple tool dependency
> to a package named installXena.  xena_import and xena_find_datasets
> contain complex dependencies to installXena.  I've installed these tools
> on two computers.  On one, everything works great - amazingly well, in
> fact!  Kudos, Development Team!  On the other computer, the env.sh files
> associated with the complex dependencies keep referencing the wrong
> version of start_xena/installXena.
>
> Here are the gory details.
>
> The start_xena tool is installed in
> testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena
> <http://testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena>.
>   I'm a bit confused about the revision part of the path, because the
> latest revision is different (75c7d80df9c1), and I have uninstalled and
> re-installed the tool since its last update.  Its package, installXena,
> is installed under tool_dependencies at
> tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/.
>   That directory contains all the files I'd expect to be there, and its
> env.sh file contains all the environment variables I'd intended to set.
>    So far, so good.
>
> xena_import is installed on my computer at
> testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import
> <http://testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import>.
>   Its tool_dependencies.xml reference installXena under start_xena as:
>
> <?xml version="1.0"?>
>
> <tool_dependency>
>
>    <package name="installXena" version="1.0">
>
>      <repository toolshed="http://testtoolshed.g2.bx.psu.edu"
> name="start_xena" owner="melissacline" changeset_revision="82755b0ee5a5"/>
>
>    </package>
>
> </tool_dependency>
>
> (Aside: is it necessary to have the changeset_revision in there?  I'm a
> bit worried about the maintenance task of updating it for all of my
> dependent tools every time I update start_xena, but as I'll explain,
> it's not clear that this revision info is getting used).

It is not necessary to populate the changeset_revision or toolshed
attributes. If you omit them, they will be automatically filled in based
on the most recent matching repository.

>
> The tool dependency dir for xena_import is at
> tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/.  Its
> env.sh reads:
>
> if [ -f
> /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
> ] ; then .
> /Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
> ; fi
>
> Notice that the wrong revision number is in there.  The 06 revision was
> an older one.  Needless to say, the files and environment variables I
> need aren't in the 06 directory.
>
> Where does this incorrect revision number come from?  I've deleted all
> the 06 directories, repeatedly.  I've deleted and reinstalled the tools,
> and verified that after deletion, the directories in question were in
> fact gone.  I've tried resetting the metadata for these tools under the
> Galaxy Admin menu.  Now I need new ideas.  As I mentioned, all of this
> works beautifully on a second computer.  It's as if the first Galaxy
> installation, on the first computer, got stuck with some outdated
> information, and I haven't figured out how to clear it out.

The 0676 revision number is based on the revision that was first
installed, and should never change. When you update the repository, the
database will record the new revision in a separate field, but the
original revision id needs to be recorded in order to find the env.sh
file on the filesystem.

>
> Thanks!
>
> Melissa
>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Incorrect revision getting referenced in a complex tool dependency - why?

Melissa Cline
Thanks, folks!  Everything you've said has clarified many things, and I'm working on putting them into practice.  But in the meantime, I've got a new error condition, with the same repositories (and complex repository dependencies), and could use some guidance.

I started to see this error condition when I removed the toolshed and changeset_revision tags in the tool_dependencies.xml for the dependent repository xena_install.  It now reads:

<?xml version="1.0"?>

<tool_dependency>

  <package name="installXena" version="1.0">

    <repository name="start_xena" owner="melissacline"/>

  </package>

</tool_dependency>

Short and simple.  But lately, when I try to install this repository from the test toolshed, I get a pop-up window from Chrome (my web browser) that says "The page at 127.0.0.1:8080 says:", "Initializing repository installation failed", with an OK button.  There is no error message visible in paster.log.  Here is what I do see:

127.0.0.1 - - [12/Sep/2014:11:45:57 -0700] "GET /admin_toolshed/prepare_for_ins\

tall?tool_shed_url=<a href="https://testtoolshed.g2.bx.psu.edu/&amp;repository_ids=4ab1d3705\">https://testtoolshed.g2.bx.psu.edu/&repository_ids=4ab1d3705\

ad0de2d&changeset_revisions=fe95cf1ae864 HTTP/1.1" 200 - "-" "Mozilla/5.0 (Maci\

ntosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.\

0.2062.94 Safari/537.36"

tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager \

DEBUG 2014-09-12 11:46:05,937 Creating repository dependency objects...

tool_shed.util.shed_util_common DEBUG 2014-09-12 11:46:07,214 Updating an exist\

ing row for repository 'xena_import' in the tool_shed_repository table, status \

set to 'New'.

tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager \

DEBUG 2014-09-12 11:46:07,241 Skipping installation of revision 75c7d80df9c1 of\

 repository 'start_xena' because it was installed with the (possibly updated) r\

evision 75c7d80df9c1 and its current installation status is 'Installed'.

tool_shed.galaxy_install.repository_dependencies.repository_dependency_manager \

DEBUG 2014-09-12 11:46:07,241 Building repository dependency relationships...

127.0.0.1 - - [12/Sep/2014:11:46:05 -0700] "POST /admin_toolshed/prepare_for_in\

stall HTTP/1.1" 200 - "<a href="http://127.0.0.1:8080/admin_toolshed/prepare_for_install\">http://127.0.0.1:8080/admin_toolshed/prepare_for_install\

?tool_shed_url=<a href="https://testtoolshed.g2.bx.psu.edu/&amp;repository_ids=4ab1d3705ad0d\">https://testtoolshed.g2.bx.psu.edu/&repository_ids=4ab1d3705ad0d\

e2d&changeset_revisions=fe95cf1ae864" "Mozilla/5.0 (Macintosh; Intel Mac OS X 1\

0_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36\

"

warning: testtoolshed.g2.bx.psu.edu certificate with fingerprint 79:94:82:1f:74\

:e6:f8:5c:1c:b8:3e:8d:89:f0:0e:88:9a:ac:2b:61 not verified (check hostfingerpri\

nts or web.cacerts config setting)

127.0.0.1 - - [12/Sep/2014:11:46:07 -0700] "POST /admin_toolshed/manage_reposit\

ories HTTP/1.1" 500 - "<a href="http://127.0.0.1:8080/admin_toolshed/prepare_for_install\">http://127.0.0.1:8080/admin_toolshed/prepare_for_install\

" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, li\

ke Gecko) Chrome/37.0.2062.94 Safari/537.36"

Debug at: http://127.0.0.1:8080/_debug/view/1410547505


Then when I click OK, the status of the tool changes to "Cloning", and stays in that state indefinitely while paster.log keeps logging a repeat of this message:

127.0.0.1 - - [12/Sep/2014:11:55:28 -0700] "POST /admin_toolshed/repository_ins\

tallation_status_updates HTTP/1.1" 200 - "<a href="http://127.0.0.1:8080/admin_toolshed/\">http://127.0.0.1:8080/admin_toolshed/\

prepare_for_install" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKi\

t/537.36 (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36"

Just in case this is from some cruft from a previous installation, I tried starting over with a brand new galaxy-dist directory.  It doesn't look like that changed anything.


Any ideas?  Many thanks in advance!


Melissa





On Fri, Sep 12, 2014 at 10:30 AM, Dave Bouvier <[hidden email]> wrote:
Melissa,

The revision recorded in the filesystem path can seem odd at first glance. My inline responses should hopefully clarify that.

   --Dave B.

On 09/10/2014 05:49 PM, Melissa Cline wrote:
Hi Folks,

Okay, I really need debugging ideas on this one.

I have three repositories I'm developing under the test toolshed.
  They're named start_xena, xena_import and xena_find_datasets (they're
all under visualization).  start_xena contains a simple tool dependency
to a package named installXena.  xena_import and xena_find_datasets
contain complex dependencies to installXena.  I've installed these tools
on two computers.  On one, everything works great - amazingly well, in
fact!  Kudos, Development Team!  On the other computer, the env.sh files
associated with the complex dependencies keep referencing the wrong
version of start_xena/installXena.

Here are the gory details.

The start_xena tool is installed in
testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena
<http://testtoolshed.g2.bx.psu.edu/repos/melissacline/start_xena/82755b0ee5a5/start_xena>.
  I'm a bit confused about the revision part of the path, because the
latest revision is different (75c7d80df9c1), and I have uninstalled and
re-installed the tool since its last update.  Its package, installXena,
is installed under tool_dependencies at
tool_dependencies/installXena/1.0/melissacline/start_xena/82755b0ee5a5/.
  That directory contains all the files I'd expect to be there, and its
env.sh file contains all the environment variables I'd intended to set.
   So far, so good.

xena_import is installed on my computer at
testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import
<http://testtoolshed.g2.bx.psu.edu/repos/melissacline/xena_import/dc42b6bbc22b/xena_import>.
  Its tool_dependencies.xml reference installXena under start_xena as:

<?xml version="1.0"?>

<tool_dependency>

   <package name="installXena" version="1.0">

     <repository toolshed="http://testtoolshed.g2.bx.psu.edu"
name="start_xena" owner="melissacline" changeset_revision="82755b0ee5a5"/>

   </package>

</tool_dependency>

(Aside: is it necessary to have the changeset_revision in there?  I'm a
bit worried about the maintenance task of updating it for all of my
dependent tools every time I update start_xena, but as I'll explain,
it's not clear that this revision info is getting used).

It is not necessary to populate the changeset_revision or toolshed attributes. If you omit them, they will be automatically filled in based on the most recent matching repository.


The tool dependency dir for xena_import is at
tool_dependencies/installXena/1.0/melissacline/xena_import/dc42b6bbc22b/.  Its
env.sh reads:

if [ -f
/Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
] ; then .
/Users/melissacline/src/galaxy-dist/tool_dependencies/installXena/1.0/melissacline/start_xena/0676a227dbc6/env.sh
; fi

Notice that the wrong revision number is in there.  The 06 revision was
an older one.  Needless to say, the files and environment variables I
need aren't in the 06 directory.

Where does this incorrect revision number come from?  I've deleted all
the 06 directories, repeatedly.  I've deleted and reinstalled the tools,
and verified that after deletion, the directories in question were in
fact gone.  I've tried resetting the metadata for these tools under the
Galaxy Admin menu.  Now I need new ideas.  As I mentioned, all of this
works beautifully on a second computer.  It's as if the first Galaxy
installation, on the first computer, got stuck with some outdated
information, and I haven't figured out how to clear it out.

The 0676 revision number is based on the revision that was first installed, and should never change. When you update the repository, the database will record the new revision in a separate field, but the original revision id needs to be recorded in order to find the env.sh file on the filesystem.



Thanks!

Melissa




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/