Keeping Galaxy up to date

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Keeping Galaxy up to date

Thomas Bellembois
Hello,

I try to keep our Galaxy instance up to date.
Very often, some tools disappear and other do not work anymore. This
breaks some users workflows.

I may miss something. Is there a "best update pratice" to keep a Galaxy
instance and the tool-sheds fully functionnal ?

Regards,

Thomas

--
Thomas Bellembois, Network and System Administrator, IGFL (France)
http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
(IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Keeping Galaxy up to date

Sebastian Schaaf
Hi Thomas,

We had this topic at the GCC this year, especially within the Galaxy
Admins BoF. The truth is (please correct me anyone if this is not the case
anymore!) that you are touching an unresolved point. Keeping track of
tools, settings etc. across the versions is not given. People have their
'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in
spare in order to invest several hours up to some days to test (with or
without participation of the users) applicability to local
constraints/histories/workflow/tools. The more testing is automated and
the less users (or non-default pieces) the faster the update procedure can
be. But there are also instances which did not receive updates for months
or even years.

On our side we will be facing reality within the next two weeks as we
really need the update. Although preconditions are pretty good (few users,
not that deep modifications, high grade of automation) we expect some
issues. happy to have a virtualized environment...

To wrap up my answer in a nutshell: no, there is no best practice guide
and no migration tool, but every single contribution is warmly welcome :).

It would be *very* interesting how updates are handled at Galaxy Main...?

Cheers,

Sebastian
(very interested in further feedback)


> Hello,
>
> I try to keep our Galaxy instance up to date.
> Very often, some tools disappear and other do not work anymore. This
> breaks some users workflows.
>
> I may miss something. Is there a "best update pratice" to keep a Galaxy
> instance and the tool-sheds fully functionnal ?
>
> Regards,
>
> Thomas
>
> --
> Thomas Bellembois, Network and System Administrator, IGFL (France)
> http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>


--
Sebastian Schaaf, M.Sc. Bioinformatics
Faculty Coordinator NGS Infrastructure
Chair of Biometry and Bioinformatics
Department of Medical Informatics,
 Biometry and Epidemiology (IBE)
University of Munich
Marchioninistr. 15, K U1 (postal)
Marchioninistr. 17, U 006 (office)
D-81377 Munich (Germany)
Tel: +49 89 2180-78178

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Keeping Galaxy up to date

Thomas Bellembois
Hi Sebastian,

Thanks a lot for this detailed answer.
I will think about a custom recipe and try to contribute to this topic.

Thomas


--
Thomas Bellembois, Network and System Administrator, IGFL (France)
http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
(IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)

On 09/09/2014 12:55, Sebastian Schaaf wrote:

> Hi Thomas,
>
> We had this topic at the GCC this year, especially within the Galaxy
> Admins BoF. The truth is (please correct me anyone if this is not the case
> anymore!) that you are touching an unresolved point. Keeping track of
> tools, settings etc. across the versions is not given. People have their
> 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in
> spare in order to invest several hours up to some days to test (with or
> without participation of the users) applicability to local
> constraints/histories/workflow/tools. The more testing is automated and
> the less users (or non-default pieces) the faster the update procedure can
> be. But there are also instances which did not receive updates for months
> or even years.
>
> On our side we will be facing reality within the next two weeks as we
> really need the update. Although preconditions are pretty good (few users,
> not that deep modifications, high grade of automation) we expect some
> issues. happy to have a virtualized environment...
>
> To wrap up my answer in a nutshell: no, there is no best practice guide
> and no migration tool, but every single contribution is warmly welcome :).
>
> It would be *very* interesting how updates are handled at Galaxy Main...?
>
> Cheers,
>
> Sebastian
> (very interested in further feedback)
>
>
>> Hello,
>>
>> I try to keep our Galaxy instance up to date.
>> Very often, some tools disappear and other do not work anymore. This
>> breaks some users workflows.
>>
>> I may miss something. Is there a "best update pratice" to keep a Galaxy
>> instance and the tool-sheds fully functionnal ?
>>
>> Regards,
>>
>> Thomas
>>
>> --
>> Thomas Bellembois, Network and System Administrator, IGFL (France)
>> http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
>> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Keeping Galaxy up to date

John Chilton-4
In reply to this post by Thomas Bellembois
Hello Thomas,

  After public debate on Trello, there have been a couple tools that
have indeed disappeared completely but nearly all tools that have gone
away have been migrated to the tool shed and can be recovered with the
migration scripts that are included with each new release
(https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution).
Are you running those scripts and still having those problems or were
you not aware of these scripts?

  My understanding is that if you are running these scripts workflows
and tool re-runs should continue to work. (Am I wrong about this?).

  If particular tools don't disappear but no longer work - this sounds
like a bug and you should let the list or Trello know so that the
devteam can attempt to fix the problem or at least be aware it exists.

  Hopefully at some point soon nearly all of the tools will be
migrated out the core release and the devteam will provide some
mechanism to quickly deploy complete say NGS packages all at once to
new releases. Once everything is in the tool shed then hopefully these
problems will go away. (There will be new problems of course - but
hopefully updates will easier).

-John


On Tue, Sep 9, 2014 at 5:49 AM, Thomas Bellembois
<[hidden email]> wrote:

> Hello,
>
> I try to keep our Galaxy instance up to date.
> Very often, some tools disappear and other do not work anymore. This
> breaks some users workflows.
>
> I may miss something. Is there a "best update pratice" to keep a Galaxy
> instance and the tool-sheds fully functionnal ?
>
> Regards,
>
> Thomas
>
> --
> Thomas Bellembois, Network and System Administrator, IGFL (France)
> http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Keeping Galaxy up to date

Thomas Bellembois
Hello John,

I had forgotten that these migration scripts exist.
It should work better now. :)

Regards,

Thomas

--
Thomas Bellembois, Network and System Administrator, IGFL (France)
http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
(IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)

On 09/09/2014 15:05, John Chilton wrote:

> Hello Thomas,
>
>   After public debate on Trello, there have been a couple tools that
> have indeed disappeared completely but nearly all tools that have gone
> away have been migrated to the tool shed and can be recovered with the
> migration scripts that are included with each new release
> (https://wiki.galaxyproject.org/MigratingToolsFromGalaxyDistribution).
> Are you running those scripts and still having those problems or were
> you not aware of these scripts?
>
>   My understanding is that if you are running these scripts workflows
> and tool re-runs should continue to work. (Am I wrong about this?).
>
>   If particular tools don't disappear but no longer work - this sounds
> like a bug and you should let the list or Trello know so that the
> devteam can attempt to fix the problem or at least be aware it exists.
>
>   Hopefully at some point soon nearly all of the tools will be
> migrated out the core release and the devteam will provide some
> mechanism to quickly deploy complete say NGS packages all at once to
> new releases. Once everything is in the tool shed then hopefully these
> problems will go away. (There will be new problems of course - but
> hopefully updates will easier).
>
> -John
>
>
> On Tue, Sep 9, 2014 at 5:49 AM, Thomas Bellembois
> <[hidden email]> wrote:
>> Hello,
>>
>> I try to keep our Galaxy instance up to date.
>> Very often, some tools disappear and other do not work anymore. This
>> breaks some users workflows.
>>
>> I may miss something. Is there a "best update pratice" to keep a Galaxy
>> instance and the tool-sheds fully functionnal ?
>>
>> Regards,
>>
>> Thomas
>>
>> --
>> Thomas Bellembois, Network and System Administrator, IGFL (France)
>> http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
>> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Keeping Galaxy up to date

Nate Coraor (nate@bx.psu.edu)
In reply to this post by Sebastian Schaaf
Hi Sebastian,

As for Galaxy Main (usegalaxy.org), our process is automated with Ansible, the playbook for which can be found here:


A bit of documentation on what you should look for (new versions of universe_wsgi.ini.sample, datatypes_conf.xml.sample, etc.) would certainly be helpful, so I'll expand the "keeping up to date" page on getgalaxy.org into a new page and add more details.

On Tue, Sep 9, 2014 at 6:55 AM, Sebastian Schaaf <[hidden email]> wrote:
Hi Thomas,

We had this topic at the GCC this year, especially within the Galaxy
Admins BoF. The truth is (please correct me anyone if this is not the case
anymore!) that you are touching an unresolved point. Keeping track of
tools, settings etc. across the versions is not given. People have their
'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine in
spare in order to invest several hours up to some days to test (with or
without participation of the users) applicability to local
constraints/histories/workflow/tools. The more testing is automated and
the less users (or non-default pieces) the faster the update procedure can
be. But there are also instances which did not receive updates for months
or even years.

On our side we will be facing reality within the next two weeks as we
really need the update. Although preconditions are pretty good (few users,
not that deep modifications, high grade of automation) we expect some
issues. happy to have a virtualized environment...

To wrap up my answer in a nutshell: no, there is no best practice guide
and no migration tool, but every single contribution is warmly welcome :).

It would be *very* interesting how updates are handled at Galaxy Main...?

Cheers,

Sebastian
(very interested in further feedback)


> Hello,
>
> I try to keep our Galaxy instance up to date.
> Very often, some tools disappear and other do not work anymore. This
> breaks some users workflows.
>
> I may miss something. Is there a "best update pratice" to keep a Galaxy
> instance and the tool-sheds fully functionnal ?
>
> Regards,
>
> Thomas
>
> --
> Thomas Bellembois, Network and System Administrator, IGFL (France)
> http://perso.ens-lyon.fr/thomas.bellembois - <a href="tel:%2B33%204%2026%2073%2013%2067" value="+33426731367">+33 4 26 73 13 67
> (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>


--
Sebastian Schaaf, M.Sc. Bioinformatics
Faculty Coordinator NGS Infrastructure
Chair of Biometry and Bioinformatics
Department of Medical Informatics,
 Biometry and Epidemiology (IBE)
University of Munich
Marchioninistr. 15, K U1 (postal)
Marchioninistr. 17, U 006 (office)
D-81377 Munich (Germany)
Tel: <a href="tel:%2B49%2089%202180-78178" value="+4989218078178">+49 89 2180-78178

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Keeping Galaxy up to date

Sebastian Schaaf
Hi Nate,

Thank you very much for enlightening me - I tried to get an impression on
Ansible playbooks for the last two minutes and I am really exited about
it. Principle and syntax are close to what we are doing in plain BASH plus
config files, but the respective scripts are for sure more complex,
because we have to write explicitly the tasks which are 'ready to use'
modules in Ansible (and I really like YAML btw :) ).

I am really looking forward to your updated article on the wiki - in the
meantime I will forward the verve to my colleagues ;). Maybe we manage to
switch on the playbook principle before the planned update.

Cheers & thanks again,

Sebastian



> Hi Sebastian,
>
> As for Galaxy Main (usegalaxy.org), our process is automated with Ansible,
> the playbook for which can be found here:
>
>     https://github.com/galaxyproject/usegalaxy-playbook
>
> A bit of documentation on what you should look for (new versions of
> universe_wsgi.ini.sample, datatypes_conf.xml.sample, etc.) would certainly
> be helpful, so I'll expand the "keeping up to date" page on getgalaxy.org
> into a new page and add more details.
>
> On Tue, Sep 9, 2014 at 6:55 AM, Sebastian Schaaf <
> [hidden email]> wrote:
>
>> Hi Thomas,
>>
>> We had this topic at the GCC this year, especially within the Galaxy
>> Admins BoF. The truth is (please correct me anyone if this is not the
>> case
>> anymore!) that you are touching an unresolved point. Keeping track of
>> tools, settings etc. across the versions is not given. People have their
>> 'home-brew' solutions for this, e.g. keeping a (maybe virtual) machine
>> in
>> spare in order to invest several hours up to some days to test (with or
>> without participation of the users) applicability to local
>> constraints/histories/workflow/tools. The more testing is automated and
>> the less users (or non-default pieces) the faster the update procedure
>> can
>> be. But there are also instances which did not receive updates for
>> months
>> or even years.
>>
>> On our side we will be facing reality within the next two weeks as we
>> really need the update. Although preconditions are pretty good (few
>> users,
>> not that deep modifications, high grade of automation) we expect some
>> issues. happy to have a virtualized environment...
>>
>> To wrap up my answer in a nutshell: no, there is no best practice guide
>> and no migration tool, but every single contribution is warmly welcome
>> :).
>>
>> It would be *very* interesting how updates are handled at Galaxy
>> Main...?
>>
>> Cheers,
>>
>> Sebastian
>> (very interested in further feedback)
>>
>>
>> > Hello,
>> >
>> > I try to keep our Galaxy instance up to date.
>> > Very often, some tools disappear and other do not work anymore. This
>> > breaks some users workflows.
>> >
>> > I may miss something. Is there a "best update pratice" to keep a
>> Galaxy
>> > instance and the tool-sheds fully functionnal ?
>> >
>> > Regards,
>> >
>> > Thomas
>> >
>> > --
>> > Thomas Bellembois, Network and System Administrator, IGFL (France)
>> > http://perso.ens-lyon.fr/thomas.bellembois - +33 4 26 73 13 67
>> > (IGFL internal IT doc: http://itdoc.igfl.ens-lyon.fr/itdoc)
>> > ___________________________________________________________
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >   http://lists.bx.psu.edu/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >   http://galaxyproject.org/search/mailinglists/
>> >
>>
>>
>> --
>> Sebastian Schaaf, M.Sc. Bioinformatics
>> Faculty Coordinator NGS Infrastructure
>> Chair of Biometry and Bioinformatics
>> Department of Medical Informatics,
>>  Biometry and Epidemiology (IBE)
>> University of Munich
>> Marchioninistr. 15, K U1 (postal)
>> Marchioninistr. 17, U 006 (office)
>> D-81377 Munich (Germany)
>> Tel: +49 89 2180-78178
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/