Moving Galaxy Install From One Server To Another

classic Classic list List threaded Threaded
6 messages Options
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Moving Galaxy Install From One Server To Another

evan clark
Is there a good method for moving a fully functional galaxy install from
one machine to another in a different directory?
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Moving Galaxy Install From One Server To Another

Hans-Rudolf Hotz
Hi Evan

Well, in theory, this is a simple task, since Galaxy mostly works with
relative paths. Of course, it all depends on how sophisticated your
installation is and how different the old and the new 'machine' is (eg
same OS?)

So my advice is: just do it or rather try it.

  - make a copy of your installation on the new hardware (at first it is
sufficient to copy only the latest sub directories of database/files/)

  - make a copy of your PostgreSQL database

  - connect your copied installation to the PostgreSQL database copy and
test, and test, and test

  - pay special attention to tools you have installed via the toolshed.
You might need to change some absolute paths in the 'env.sh' files
located in the tool_dependencies/ directory.


Once your happy with all the tests, do an rsync on the database/
directory and connect to the original PostgreSQL database.


Regards, Hans-Rudolf




On 11/22/2016 07:52 PM, evan clark wrote:

> Is there a good method for moving a fully functional galaxy install from
> one machine to another in a different directory?
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Moving Galaxy Install From One Server To Another

Peter Briggs
Hello Evan, Hans-Rudolf

I'm just in the middle of doing a similar migration for our local
production server, and Hans-Rudolf's advice seems sound to me.
Definitely moving the core Galaxy server has been relatively
straightforward.

However: for the installed tools, I'm not sure that changing the paths
in the env.sh files is sufficient - in our installation the absolute
paths seemed to be baked into a lot of other files under the
'tool_dependencies' directory - including things like compiled files
(e.g. static and shared libraries). So for many of the tools I wouldn't
feel confident that they would still work after the move.

My plan has been to reinstall each of the tools from the toolshed (i.e.
uninstall via the admin interface then reinstall the same tool
repository revision(s) using the API), but I don't feel able to
recommend this approach either as in my testing this has also had
problems - ranging from some tool revisions no longer being available,
through to more serious issues (such as tool dependencies which used to
work but since become broken). I figured I'd just have to knuckle down
and work through each problem as I encountered it.

If anyone else has experiences with these kinds of migrations then I'm
also very interested to know what worked (and what didn't)!

Best wishes

Peter

On 23/11/16 08:20, Hans-Rudolf Hotz wrote:

> Hi Evan
>
> Well, in theory, this is a simple task, since Galaxy mostly works with
> relative paths. Of course, it all depends on how sophisticated your
> installation is and how different the old and the new 'machine' is (eg
> same OS?)
>
> So my advice is: just do it or rather try it.
>
>  - make a copy of your installation on the new hardware (at first it is
> sufficient to copy only the latest sub directories of database/files/)
>
>  - make a copy of your PostgreSQL database
>
>  - connect your copied installation to the PostgreSQL database copy and
> test, and test, and test
>
>  - pay special attention to tools you have installed via the toolshed.
> You might need to change some absolute paths in the 'env.sh' files
> located in the tool_dependencies/ directory.
>
>
> Once your happy with all the tests, do an rsync on the database/
> directory and connect to the original PostgreSQL database.
>
>
> Regards, Hans-Rudolf
>
>
>
>
> On 11/22/2016 07:52 PM, evan clark wrote:
>> Is there a good method for moving a fully functional galaxy install from
>> one machine to another in a different directory?
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Moving Galaxy Install From One Server To Another

Peter Cock
On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs
<[hidden email]> wrote:

> Hello Evan, Hans-Rudolf
>
> I'm just in the middle of doing a similar migration for our local production
> server, and Hans-Rudolf's advice seems sound to me. Definitely moving the
> core Galaxy server has been relatively straightforward.
>
> However: for the installed tools, I'm not sure that changing the paths in
> the env.sh files is sufficient - in our installation the absolute paths
> seemed to be baked into a lot of other files under the 'tool_dependencies'
> directory - including things like compiled files (e.g. static and shared
> libraries). So for many of the tools I wouldn't feel confident that they
> would still work after the move.

Yes, I've seen that with various third party tool installations (out side of
Galaxy), so sadly you cannot in general move the files to a different path :(

I don't know if we can do anything about this directly in Galaxy, or even
in BioConda?

> My plan has been to reinstall each of the tools from the toolshed (i.e.
> uninstall via the admin interface then reinstall the same tool repository
> revision(s) using the API), but I don't feel able to recommend this approach
> either as in my testing this has also had problems - ranging from some tool
> revisions no longer being available, through to more serious issues (such as
> tool dependencies which used to work but since become broken). I figured I'd
> just have to knuckle down and work through each problem as I encountered it.

One simple reason for this is stale URLs, where Galaxy's cache can help
but is another step for tool wrapper authors to do when setting up a new
Galaxy dependency: https://github.com/galaxyproject/cargo-port

However, even well intentioned Tool Shed updates could also break things :(

> If anyone else has experiences with these kinds of migrations then I'm also
> very interested to know what worked (and what didn't)!

Sorry - the closest I've been is setting up a new Galaxy server in
parallel with our old server, and manually installing "missing" tools
via the Tool Shed.

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Moving Galaxy Install From One Server To Another

Peter Briggs
Hi Peter

Thanks for your comments - at least it doesn't sound like I'm missing
the "perfect" way to migrate the installed tools. In the longer term
moving towards bioconda dependency resolution will probably be more stable.

Also re breaking tool repos: even repos that aren't updated on the
toolshed can break over time, for example if the URL for an executable
moves, or if an implicit Python dependency is updated and breaks an
install (these are both things that I'm seeing in my tests).

Thanks again

Best wishes

Peter

On 23/11/16 11:40, Peter Cock wrote:

> On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs
> <[hidden email]> wrote:
>> Hello Evan, Hans-Rudolf
>>
>> I'm just in the middle of doing a similar migration for our local production
>> server, and Hans-Rudolf's advice seems sound to me. Definitely moving the
>> core Galaxy server has been relatively straightforward.
>>
>> However: for the installed tools, I'm not sure that changing the paths in
>> the env.sh files is sufficient - in our installation the absolute paths
>> seemed to be baked into a lot of other files under the 'tool_dependencies'
>> directory - including things like compiled files (e.g. static and shared
>> libraries). So for many of the tools I wouldn't feel confident that they
>> would still work after the move.
>
> Yes, I've seen that with various third party tool installations (out side of
> Galaxy), so sadly you cannot in general move the files to a different path :(
>
> I don't know if we can do anything about this directly in Galaxy, or even
> in BioConda?
>
>> My plan has been to reinstall each of the tools from the toolshed (i.e.
>> uninstall via the admin interface then reinstall the same tool repository
>> revision(s) using the API), but I don't feel able to recommend this approach
>> either as in my testing this has also had problems - ranging from some tool
>> revisions no longer being available, through to more serious issues (such as
>> tool dependencies which used to work but since become broken). I figured I'd
>> just have to knuckle down and work through each problem as I encountered it.
>
> One simple reason for this is stale URLs, where Galaxy's cache can help
> but is another step for tool wrapper authors to do when setting up a new
> Galaxy dependency: https://github.com/galaxyproject/cargo-port
>
> However, even well intentioned Tool Shed updates could also break things :(
>
>> If anyone else has experiences with these kinds of migrations then I'm also
>> very interested to know what worked (and what didn't)!
>
> Sorry - the closest I've been is setting up a new Galaxy server in
> parallel with our old server, and manually installing "missing" tools
> via the Tool Shed.
>
> Peter
>

--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Moving Galaxy Install From One Server To Another

Brian Claywell-2
I wrote something in Python a while back (which I've attached) to parse a shed_tool_conf.xml file and generate a shell script to install those tools on a different server. The shell script uses Galaxy's scripts/api/install_tool_shed_repositories.py to install the tools and ends up looking something like this:


#!/bin/sh
set -eux

python ./scripts/api/install_tool_shed_repositories.py --local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps --url https://toolshed.g2.bx.psu.edu --owner devteam --name fasta_to_tabular --revision 9d189d08f2ad --panel-section-id convert
python ./scripts/api/install_tool_shed_repositories.py --local $GALAXY_INSTALL_URL --api $GALAXY_INSTALL_KEY --tool-deps --repository-deps --url https://toolshed.g2.bx.psu.edu --owner devteam --name tabular_to_fasta --revision 0b4e36026794 --panel-section-id convert
... etc


All you have to do is set those two environment variables appropriately and run the shell script from your Galaxy root directory. I did end up having to modify our nginx config to increase the proxy read timeout for the /api location to a much larger value (I used 3600s) to give the tool dependencies enough time to build, though.

Hope you find it useful.


Cheers,

Brian


On Wed, Nov 23, 2016 at 7:32 AM, Peter Briggs <[hidden email]> wrote:
Hi Peter

Thanks for your comments - at least it doesn't sound like I'm missing the "perfect" way to migrate the installed tools. In the longer term moving towards bioconda dependency resolution will probably be more stable.

Also re breaking tool repos: even repos that aren't updated on the toolshed can break over time, for example if the URL for an executable moves, or if an implicit Python dependency is updated and breaks an install (these are both things that I'm seeing in my tests).

Thanks again

Best wishes

Peter


On 23/11/16 11:40, Peter Cock wrote:
On Wed, Nov 23, 2016 at 10:49 AM, Peter Briggs
<[hidden email]> wrote:
Hello Evan, Hans-Rudolf

I'm just in the middle of doing a similar migration for our local production
server, and Hans-Rudolf's advice seems sound to me. Definitely moving the
core Galaxy server has been relatively straightforward.

However: for the installed tools, I'm not sure that changing the paths in
the env.sh files is sufficient - in our installation the absolute paths
seemed to be baked into a lot of other files under the 'tool_dependencies'
directory - including things like compiled files (e.g. static and shared
libraries). So for many of the tools I wouldn't feel confident that they
would still work after the move.

Yes, I've seen that with various third party tool installations (out side of
Galaxy), so sadly you cannot in general move the files to a different path :(

I don't know if we can do anything about this directly in Galaxy, or even
in BioConda?

My plan has been to reinstall each of the tools from the toolshed (i.e.
uninstall via the admin interface then reinstall the same tool repository
revision(s) using the API), but I don't feel able to recommend this approach
either as in my testing this has also had problems - ranging from some tool
revisions no longer being available, through to more serious issues (such as
tool dependencies which used to work but since become broken). I figured I'd
just have to knuckle down and work through each problem as I encountered it.

One simple reason for this is stale URLs, where Galaxy's cache can help
but is another step for tool wrapper authors to do when setting up a new
Galaxy dependency: https://github.com/galaxyproject/cargo-port

However, even well intentioned Tool Shed updates could also break things :(

If anyone else has experiences with these kinds of migrations then I'm also
very interested to know what worked (and what didn't)!

Sorry - the closest I've been is setting up a new Galaxy server in
parallel with our old server, and manually installing "missing" tools
via the Tool Shed.

Peter


--
Peter Briggs [hidden email]
Bioinformatics Core Facility University of Manchester
B.1083 Michael Smith Bldg Tel: (0161) 2751482
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



--
Brian Claywell | programmer/analyst
Matsen Group   | matsen.fredhutch.org
Fred Hutchinson Cancer Research Center

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

generate_install_script.py (4K) Download Attachment
Loading...