tool shed repository update bug

classic Classic list List threaded Threaded
4 messages Options
| Threaded
Open this post in threaded view
|

tool shed repository update bug

Björn Grüning
Hi Greg,

during the addition of the next dependency (libeigen) I found an other
bug related to repository updates. Please see revision 6c8f72ee4a51 of
confab in the testtoolshed. I installed that and want to update to
revision aac0c82ac354 with new dependency. I saw that a new metadata is
associated, but I'm not able to update:

'The installed repository named 'confab' is current, there are no
updates available.'

Cheers,
Bjoern



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: tool shed repository update bug

Björn Grüning
Hi Greg,

thats actually not a bug ... sorry, but wouldn't it be better to display
a message liek "a new version is installable, do you want to install"?

Cheers,
Bjoern

> Hi Greg,
>
> during the addition of the next dependency (libeigen) I found an other
> bug related to repository updates. Please see revision 6c8f72ee4a51 of
> confab in the testtoolshed. I installed that and want to update to
> revision aac0c82ac354 with new dependency. I saw that a new metadata is
> associated, but I'm not able to update:
>
> 'The installed repository named 'confab' is current, there are no
> updates available.'
>
> Cheers,
> Bjoern
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: tool shed repository update bug

Greg Von Kuster
Hello Björn,

I'll take a close look at this.  It seems that the contents of revision 6c8f72ee4a51 are a subset of the contents of revision aac0c82ac354 as the only changes are additional dependency definitions.  Am I seeing this correctly?  If so, then an additional metadata record should not have been generated, but rather  aac0c82ac354 should now be the only metadata revision between the 2 - if this is the case, then this is a bug.

Thanks!

Greg Von Kuster

On May 14, 2013, at 4:35 AM, Björn Grüning wrote:

Hi Greg,

thats actually not a bug ... sorry, but wouldn't it be better to display
a message liek "a new version is installable, do you want to install"?

Cheers,
Bjoern

Hi Greg,

during the addition of the next dependency (libeigen) I found an other
bug related to repository updates. Please see revision 6c8f72ee4a51 of
confab in the testtoolshed. I installed that and want to update to
revision aac0c82ac354 with new dependency. I saw that a new metadata is
associated, but I'm not able to update:

'The installed repository named 'confab' is current, there are no
updates available.'

Cheers,
Bjoern



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: tool shed repository update bug

Greg Von Kuster
In reply to this post by Björn Grüning
Hello Björn,

I've analyzed this and discovered some very important points to make sure everyone understands, so I hope my information below is very clear.  Please don't hesitate to ask any questions for clarification.  The items fall into 2 categories.

First item:
========
The first item for discussion is a known weakness of the tool shed upload process (or, to be more correct, the process of adding a new changeset revision to a repository).  When you commit a changeset to a repository that has previous revisions, the tool shed repository metadata process is executed and the contents of the new changeset are analyzed to determine if a new metadata record should be created for the changeset or if the previous metadata record should be moved up in the repository changelog chain (this is a very complex analysis).  

Until about 1 year ago, I reset all metadata on the entire repository chjangelog when a new changeset was committed.  This process worked very well, and metadata was cleanly set for the entire changelog.  However, this process began to take significant periods of time as soon as repositories became larger and the changelogs became longer (a good example is http://testtoolshed.g2.bx.psu.edu/view/mzytnicki/s_mart).

To improve performance when adding new changesets to repositories, the metadata analysis process was enhanced to inspect only those changesets that go back to the previous revision with associated metadata.  This process works very well in most cases.  However, there are unusual cases where the process breaks, and your confab repository fell into this category.  The issue arises when the contents of a specific change set do not properly move the previous metadata revision forward in the changelog, but instead create a new, additional metadata revision.  

When I initially inspected your confab repository had metadata associated with the following revisions:


I inspected the main contents (tools and dependencies) of these revisions and noticed that some did not seem to have any differences (contents of the containers had the same labels between revisions - I did not inspect the changelog itself though).  Because of this I decided to reset all metadata on the repository (available in the Repository Actions menu), and now metadata is associated with only the following revisions:


This is a very complex issue to handle, as there are many things to consider.  Obviously automatically inspecting the entire changelog when a new changeset is added will not work.  I have added the ability for a repository owner to reset all metadata on the repository, but there is nothing to alert them to initiate this process (and I'm not sure how / when to do so).  Also, resetting all metadata on a repository eliminates the tool test results associated with each of the original metadata revisions.

Second item:
==========

It seems there is some confusion about the changeset revision that is defined in dependency definitions.  This is a fairly complex subject, and it seems I may have muddied the waters a bit yesterday.  If so, I apologize, but I think Peter's last response to my example clarifies the issue.

Looking at the changelog of your confab repository, I see the following.  You have unnecessarily changed the changeset_revision setting in your tool dependency definitions, and this has resulted in a new additional metadata record being associated with changeset 9:dd3ee8e742dc of your confab repository.  This changeset revision setting should not have been changed - see my discussion below.

Repository 'confab'
===============
Changeset 9e38b8bd4cdb
modified:
tool_dependencies.xml
tool_dependencies.xml
--- a/tool_dependencies.xmlTue May 14 04:12:06 2013 -0400
+++ b/tool_dependencies.xmlTue May 14 04:17:08 2013 -0400
@@ -1,13 +1,13 @@
 <tool_dependency>
     <package name="eigen" version="2.0.17">
-        <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="294a30630e0b" prior_installation_required="True" />
+        <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="09eb05087cd0" prior_installation_required="True" />
     </package>
     <package name="confab" version="1.0.1">
         <install version="1.0">
             <actions>
                 <!-- populate the environment variables from the dependend repos -->
                 <action type="set_environment_for_install">
-                    <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="294a30630e0b">
+                    <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="09eb05087cd0">
                         <package name="eigen" version="2.0.17" />
                     </repository>
                 </action>


Looking at the changelog of your package_eigen_2_0 repository, I see the following:

Repository 'package_eigen_2_0'
==========================
Changesets
Repository metadata is associated with this change set.
Uploaded
2:09eb05087cd0
1:294a30630e0b
bgruening
~ 11 hours ago

Uploaded
1:294a30630e0b
0:5c44ce7e3f7b
bgruening
~ 11 hours ago

Uploaded
0:5c44ce7e3f7b
None
bgruening
~ 11 hours ago

Notice that metadata is associated with only changeset revision 2:09eb05087cd0.  What this implies is that any existing dependency definition that defines the changeset_revision value as one of 5c44ce7e3f7b294a30630e0b or 09eb05087cd0 will all get the same changeset revision installed, and that installed revision will be 09eb05087cd0.  This is a very important point, so please let me know if the reasons for this are not clear.  

Yesterday we had a discussion about the "minimum" setting for a changeset revision.  Here is Peter's feedback at the end of that discussion.

====================

Perhaps this is clearer?:

changset revision       Installable revision
-----------------------------------------------------------------------------
0: sjekvub        not any more, superseded by r1
1: jjtofvp           not any more, superseded by r2
2: htocegy        yes, final revision with this metadata
-----------------------------------------------------------------------------
3: jswofpt          not any more, superseded by r4
4: jaqvkrc          yes, final revision with this metadata

====================

Do you understand why you should not have changed the value of the changeset_revision setting in the tool dependency definition your confab repository?  It is because this concept of "minimum" setting comes into play here.  When the value of the changeset_revision setting is changed, it will always generate a new metadata revision, which is not what you wanted in this case.

-        <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="294a30630e0b" prior_installation_required="True" />
+        <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="09eb05087cd0" prior_installation_required="True" />


-                    <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="294a30630e0b">
+                    <repository toolshed="http://testtoolshed.g2.bx.psu.edu/" name="package_eigen_2_0" owner="bgruening" changeset_revision="09eb05087cd0">


Greg Von Kuster



On May 14, 2013, at 3:59 AM, Björn Grüning wrote:

Hi Greg,

during the addition of the next dependency (libeigen) I found an other
bug related to repository updates. Please see revision 6c8f72ee4a51 of
confab in the testtoolshed. I installed that and want to update to
revision aac0c82ac354 with new dependency. I saw that a new metadata is
associated, but I'm not able to update:

'The installed repository named 'confab' is current, there are no
updates available.'

Cheers,
Bjoern



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/