Safe / possible to sync toolshed tools and deps from another machine?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Safe / possible to sync toolshed tools and deps from another machine?

David Trudgian

Hi,

 

I was wondering whether it might be possible/safe to install tools and deps from the toolshed on a VM galaxy installation, and then rsync the shed_tools and  tool dependencies directories to a production galaxy server?

 

Unfortunately, our institution has web access only through a proxy and blocks ‘personal web browsing’ from servers. This works on a white list – so everything is blocked unless there was considered to be a valid need when the list was originally setup, or an exception request is made and approved. We have an exception in place for the main toolshed, but various dependency downloads will fail, blocked by the proxy. Getting exceptions for each URL we find not to work is painful and slow.

 

The suggestion I’ve been given is to download things on a workstation (not subject to the blocking) and transfer them manually to the server, hence the question about whether tool directories (and XML conf files) can be synced between installs? Is there anything in the SQL DB that would prevent this working, or being safe?

 

Otherwise I’m installing a lot of deps manually L

 

Thanks,


Dave Trudgian

 

 

 

 



UT Southwestern

Medical Center

The future of medicine, today.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Safe / possible to sync toolshed tools and deps from another machine?

Dave Bouvier
Dave,

There are a number of things to consider when doing this, but it should
be possible. Things to keep in mind are, in no particular order:

The tool dependency path on the workstation should match what would be
the path on the server. A substantial number of packages link their
libraries with absolute paths, so any change in paths would of course
break the package at runtime.

The operating system, hardware, and installed libraries on the server
should at worst be a superset of those on the workstation, to avoid
compiling packages on the workstation that have dependencies not present
on the server.

I would recommend setting the install_database_connection option in your
config/galaxy.ini file, so that you can export the database of installed
repositories and tool dependencies from the workstation, and import this
to the server without overwriting users, jobs, and so on.

If those hurdles are cleared, it's likely that installing and copying
dependencies from one system to another might work.

Another solution, more complicated to set up but easier to use, would be
to run a single install database that both systems access, and install
the dependencies on the workstation, into a network filesystem that is
mounted in the same path on both systems. You would still need to sync
the shed_tool_conf.xml across machines, but a restart of the server
would then show all the tools and their dependencies with whatever
status they should have.

    --Dave B.

On 03/27/2015 11:36 AM, David Trudgian wrote:

> Hi,
>
> I was wondering whether it might be possible/safe to install tools and
> deps from the toolshed on a VM galaxy installation, and then rsync the
> shed_tools and  tool dependencies directories to a production galaxy server?
>
> Unfortunately, our institution has web access only through a proxy and
> blocks ‘personal web browsing’ from servers. This works on a white list
> – so everything is blocked unless there was considered to be a valid
> need when the list was originally setup, or an exception request is made
> and approved. We have an exception in place for the main toolshed, but
> various dependency downloads will fail, blocked by the proxy. Getting
> exceptions for each URL we find not to work is painful and slow.
>
> The suggestion I’ve been given is to download things on a workstation
> (not subject to the blocking) and transfer them manually to the server,
> hence the question about whether tool directories (and XML conf files)
> can be synced between installs? Is there anything in the SQL DB that
> would prevent this working, or being safe?
>
> Otherwise I’m installing a lot of deps manually L
>
> Thanks,
>
>
> Dave Trudgian
>
>
> ------------------------------------------------------------------------
>
> UTSouthwestern
>
> Medical Center
>
> The future of medicine, today.
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Safe / possible to sync toolshed tools and deps from another machine?

David Trudgian
Dave,

Many thanks for your detailed reply. I like the sound of the second solution which should be no problem to setup on our systems.

Still hoping it'll be possible to get a proxy exception for our system and avoid this altogether longer term, but good to know it should be possible for now.

Thanks again,

DT

-----Original Message-----
From: Dave Bouvier [mailto:[hidden email]]
Sent: Monday, March 30, 2015 8:03 AM
To: David Trudgian; [hidden email]
Subject: Re: [galaxy-dev] Safe / possible to sync toolshed tools and deps from another machine?

Dave,

There are a number of things to consider when doing this, but it should be possible. Things to keep in mind are, in no particular order:

The tool dependency path on the workstation should match what would be the path on the server. A substantial number of packages link their libraries with absolute paths, so any change in paths would of course break the package at runtime.

The operating system, hardware, and installed libraries on the server should at worst be a superset of those on the workstation, to avoid compiling packages on the workstation that have dependencies not present on the server.

I would recommend setting the install_database_connection option in your config/galaxy.ini file, so that you can export the database of installed repositories and tool dependencies from the workstation, and import this to the server without overwriting users, jobs, and so on.

If those hurdles are cleared, it's likely that installing and copying dependencies from one system to another might work.

Another solution, more complicated to set up but easier to use, would be to run a single install database that both systems access, and install the dependencies on the workstation, into a network filesystem that is mounted in the same path on both systems. You would still need to sync the shed_tool_conf.xml across machines, but a restart of the server would then show all the tools and their dependencies with whatever status they should have.

    --Dave B.

On 03/27/2015 11:36 AM, David Trudgian wrote:

> Hi,
>
> I was wondering whether it might be possible/safe to install tools and
> deps from the toolshed on a VM galaxy installation, and then rsync the
> shed_tools and  tool dependencies directories to a production galaxy server?
>
> Unfortunately, our institution has web access only through a proxy and
> blocks ‘personal web browsing’ from servers. This works on a white
> list – so everything is blocked unless there was considered to be a
> valid need when the list was originally setup, or an exception request
> is made and approved. We have an exception in place for the main
> toolshed, but various dependency downloads will fail, blocked by the
> proxy. Getting exceptions for each URL we find not to work is painful and slow.
>
> The suggestion I’ve been given is to download things on a workstation
> (not subject to the blocking) and transfer them manually to the
> server, hence the question about whether tool directories (and XML
> conf files) can be synced between installs? Is there anything in the
> SQL DB that would prevent this working, or being safe?
>
> Otherwise I’m installing a lot of deps manually L
>
> Thanks,
>
>
> Dave Trudgian
>
>
> ----------------------------------------------------------------------
> --
>
> UTSouthwestern
>
> Medical Center
>
> The future of medicine, today.
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this and other
> Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/