Galaxy packages for Bio-Linux - an update + Apache proxying

classic Classic list List threaded Threaded
8 messages Options
| Threaded
Open this post in threaded view
|

Galaxy packages for Bio-Linux - an update + Apache proxying

Tim Booth
Hi All,

As previously mentioned, I'm back working on packaging the Galaxy server
as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately
pushing towards something that could be Debian compliant.  There's a way
to go in that regard, but I do now have an updated package for Bio-Linux
in final testing and it also has a new trick: doing  "apt-get install
galaxy-server-apache-proxy" will set up just that with no further
configuration needed.  The galaxy server appears at
http://localhost/galaxy and users log in with their regular system
username and password.  Uploads are enabled via regular SFTP so no
special FTP server configuration is needed.

It's a little hacky in parts but I'm generally pleased with the result.
If anyone want to take a look I'd welcome comments.  It's not in the
main BL repo yet but can be found here:

https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra

Cheers,

TIM

--
Tim Booth <[hidden email]>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
+44 1491 69 2705

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

John Chilton
This is quite awesome! Thanks for all the hard work Tim.

In response to the eggs comment in the changelog - there are upsides
and downsides to the way Galaxy handles dependency fetching but I
think there should at least be an off switch. I thought that would be
easy, but it sounds like not:

>From Twitter this morning:

@jmchilton "@natefoo its great it "just works" for development and
testing and desktop efforts. It just needs an off switch :)."

@natefoo "@jmchilton Agreed, although versions would still be a problem then."

Nonetheless, I have created a Trello card for this.

https://trello.com/c/lthRjVZq

-John

On Fri, Dec 13, 2013 at 6:03 AM, Tim Booth <[hidden email]> wrote:

> Hi All,
>
> As previously mentioned, I'm back working on packaging the Galaxy server
> as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately
> pushing towards something that could be Debian compliant.  There's a way
> to go in that regard, but I do now have an updated package for Bio-Linux
> in final testing and it also has a new trick: doing  "apt-get install
> galaxy-server-apache-proxy" will set up just that with no further
> configuration needed.  The galaxy server appears at
> http://localhost/galaxy and users log in with their regular system
> username and password.  Uploads are enabled via regular SFTP so no
> special FTP server configuration is needed.
>
> It's a little hacky in parts but I'm generally pleased with the result.
> If anyone want to take a look I'd welcome comments.  It's not in the
> main BL repo yet but can be found here:
>
> https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
>
> Cheers,
>
> TIM
>
> --
> Tim Booth <[hidden email]>
> NERC Environmental Bioinformatics Centre
>
> Centre for Ecology and Hydrology
> Maclean Bldg, Benson Lane
> Crowmarsh Gifford
> Wallingford, England
> OX10 8BB
>
> http://nebc.nerc.ac.uk
> +44 1491 69 2705
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

cjav
In reply to this post by Tim Booth
Hi Tim,

This sounds great. I'll be happy to help testing and hopefully find
some time to help packaging once it gets into Debian Med(are you
submitting all your packages there?).

One question, for apache/nginx configuration why not use something ala
phpMyAdmin which ask you if you want to preconfigure the package with
a webserver in particular. The name of the DEB packaging technology to
ask these kind of questions is evading me now. I think using something
like that could open many possibilities in the future, like database
backend to use, home URL, admin user/password, etc...

Thanks for your work on this,
Carlos


On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth <[hidden email]> wrote:

> Hi All,
>
> As previously mentioned, I'm back working on packaging the Galaxy server
> as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately
> pushing towards something that could be Debian compliant.  There's a way
> to go in that regard, but I do now have an updated package for Bio-Linux
> in final testing and it also has a new trick: doing  "apt-get install
> galaxy-server-apache-proxy" will set up just that with no further
> configuration needed.  The galaxy server appears at
> http://localhost/galaxy and users log in with their regular system
> username and password.  Uploads are enabled via regular SFTP so no
> special FTP server configuration is needed.
>
> It's a little hacky in parts but I'm generally pleased with the result.
> If anyone want to take a look I'd welcome comments.  It's not in the
> main BL repo yet but can be found here:
>
> https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
>
> Cheers,
>
> TIM
>
> --
> Tim Booth <[hidden email]>
> NERC Environmental Bioinformatics Centre
>
> Centre for Ecology and Hydrology
> Maclean Bldg, Benson Lane
> Crowmarsh Gifford
> Wallingford, England
> OX10 8BB
>
> http://nebc.nerc.ac.uk
> +44 1491 69 2705
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

Tim Booth
Hi John and Carlos,

Thanks for your replies regarding my DEB packaging work.  Some quick
answers to questions you raised.
>
> In response to the eggs comment in the changelog - there are upsides
> and downsides to the way Galaxy handles dependency fetching but I
> think there should at least be an off switch. I thought that would be
> easy, but it sounds like not:
> ...
>  I have created a Trello card for this.
> https://trello.com/c/lthRjVZq
>
Of course the ultimate goal for Debian compliance is that Galaxy uses
100% system Python libs and DPKG looks after all the interdependencies.
I'm not sure how realistic that is, but I'm working on it.  I'll need
good regression tests for sure.

On Mon, 2013-12-16 at 16:27 +0000, Carlos Borroto wrote:
> Hi Tim,
>
> This sounds great. I'll be happy to help testing and hopefully find
> some time to help packaging once it gets into Debian Med(are you
> submitting all your packages there?).

Yes, it will all go in the Debian-Med SVN but for it to actually work on
Debian I'll need a non-Upstart init script and probably some other
changes.  Maybe you can help me with that?  I'll ping you in the New
Year.

> One question, for apache/nginx configuration why not use something ala
> phpMyAdmin which ask you if you want to preconfigure the package with
> a webserver in particular. The name of the DEB packaging technology to
> ask these kind of questions is evading me now. I think using something
> like that could open many possibilities in the future, like database
> backend to use, home URL, admin user/password, etc...

I think you mean Debconf, which I'll probably use for a couple of things
like the initial list of admin users but which I try to limit.  If I put
the packages out on the live DVD/USB or the Cloud image then I have to
configure them with default settings in any case, and then it's easier
for the user to just edit the config files than to remember a command
like:

% sudo dpkg-reconfigure -p medium galaxy-server-apache-proxy

As a case in point, the apache2 package itself no longer uses debconf at
all.

I'm not currently planning to add Nginx to this packaging, as the
priority was to enable uploads via SFTP, and this required use of system
accounts in Galaxy, which called for authentication via PAM, which
needed the relevant Apache2 modules.

Cheers,

TIM

> Thanks for your work on this,
> Carlos
>
>
> On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth <[hidden email]> wrote:
> > Hi All,
> >
> > As previously mentioned, I'm back working on packaging the Galaxy server
> > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately
> > pushing towards something that could be Debian compliant.  There's a way
> > to go in that regard, but I do now have an updated package for Bio-Linux
> > in final testing and it also has a new trick: doing  "apt-get install
> > galaxy-server-apache-proxy" will set up just that with no further
> > configuration needed.  The galaxy server appears at
> > http://localhost/galaxy and users log in with their regular system
> > username and password.  Uploads are enabled via regular SFTP so no
> > special FTP server configuration is needed.
> >
> > It's a little hacky in parts but I'm generally pleased with the result.
> > If anyone want to take a look I'd welcome comments.  It's not in the
> > main BL repo yet but can be found here:
> >
> > https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
> >
> > Cheers,
> >
> > TIM
> >
> > --
> > Tim Booth <[hidden email]>
> > NERC Environmental Bioinformatics Centre
> >
> > Centre for Ecology and Hydrology
> > Maclean Bldg, Benson Lane
> > Crowmarsh Gifford
> > Wallingford, England
> > OX10 8BB
> >
> > http://nebc.nerc.ac.uk
> > +44 1491 69 2705
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/

--
Tim Booth <[hidden email]>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
+44 1491 69 2705

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

Tim Booth
In reply to this post by cjav
Hi,

This is mainly a message to Carlos and Eric who volunteered to get
involved with my Debian packaging, but of course any other help will
also be appreciated.  If either of you can spare time now it is a good
moment to do so.

I've been working on my package stuff over the last couple of weeks and
have uploaded the latest to Debian-Med ready to build:

http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log

Note that you need the re-packed tarball as generated by
get-orig-source.sh and if you have a problem generating that you can
grab a copy of it here:

https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz

I've only built this for Ubuntu, and I know that to get it working on
Debian you'll at least need to replace the upstart job with
an /etc/init.d script.  After that I think you should have something
working (see my commit notes).

My latest efforts have been to try and get tool-shed installs working.
Galaxy expects to be able to write to its own shed_tools_*_conf.xml
files as well as to shed_tools and the tool-data directory.  It looks
like there is work to have a separate shed_tool-data folder but this is
not fully working so I'm seeing if I can patch it.  Either way, it's
vital for packaging that the files managed by DPKG and the files
(over)written by the Galaxy server are separated out into /var and /usr
respectively.

Cheers,

TIM

On Mon, 2013-12-16 at 16:27 +0000, Carlos Borroto wrote:

> Hi Tim,
>
> This sounds great. I'll be happy to help testing and hopefully find
> some time to help packaging once it gets into Debian Med(are you
> submitting all your packages there?).
>
> One question, for apache/nginx configuration why not use something ala
> phpMyAdmin which ask you if you want to preconfigure the package with
> a webserver in particular. The name of the DEB packaging technology to
> ask these kind of questions is evading me now. I think using something
> like that could open many possibilities in the future, like database
> backend to use, home URL, admin user/password, etc...
>
> Thanks for your work on this,
> Carlos
>
>
> On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth <[hidden email]> wrote:
> > Hi All,
> >
> > As previously mentioned, I'm back working on packaging the Galaxy server
> > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately
> > pushing towards something that could be Debian compliant.  There's a way
> > to go in that regard, but I do now have an updated package for Bio-Linux
> > in final testing and it also has a new trick: doing  "apt-get install
> > galaxy-server-apache-proxy" will set up just that with no further
> > configuration needed.  The galaxy server appears at
> > http://localhost/galaxy and users log in with their regular system
> > username and password.  Uploads are enabled via regular SFTP so no
> > special FTP server configuration is needed.
> >
> > It's a little hacky in parts but I'm generally pleased with the result.
> > If anyone want to take a look I'd welcome comments.  It's not in the
> > main BL repo yet but can be found here:
> >
> > https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
> >
> > Cheers,
> >
> > TIM
> >
> > --
> > Tim Booth <[hidden email]>
> > NERC Environmental Bioinformatics Centre
> >
> > Centre for Ecology and Hydrology
> > Maclean Bldg, Benson Lane
> > Crowmarsh Gifford
> > Wallingford, England
> > OX10 8BB
> >
> > http://nebc.nerc.ac.uk
> > +44 1491 69 2705
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/

--
Tim Booth <[hidden email]>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
+44 1491 69 2705

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

Nate Coraor (nate@bx.psu.edu)
Hi Tim,

I have recently been working on getting Galaxy Main's configs and server-modified files and directories out of the galaxy-dist directory, so our goals are aligning. Not everything can be moved without some trickery (e.g. symlinks) but most paths, including the paths to shed_*.xml are configurable in universe_wsgi.ini (which itself need not live in the galaxy-dist directory).

--nate


On Mon, Mar 24, 2014 at 1:38 PM, Tim Booth <[hidden email]> wrote:
Hi,

This is mainly a message to Carlos and Eric who volunteered to get
involved with my Debian packaging, but of course any other help will
also be appreciated.  If either of you can spare time now it is a good
moment to do so.

I've been working on my package stuff over the last couple of weeks and
have uploaded the latest to Debian-Med ready to build:

http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log

Note that you need the re-packed tarball as generated by
get-orig-source.sh and if you have a problem generating that you can
grab a copy of it here:

https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz

I've only built this for Ubuntu, and I know that to get it working on
Debian you'll at least need to replace the upstart job with
an /etc/init.d script.  After that I think you should have something
working (see my commit notes).

My latest efforts have been to try and get tool-shed installs working.
Galaxy expects to be able to write to its own shed_tools_*_conf.xml
files as well as to shed_tools and the tool-data directory.  It looks
like there is work to have a separate shed_tool-data folder but this is
not fully working so I'm seeing if I can patch it.  Either way, it's
vital for packaging that the files managed by DPKG and the files
(over)written by the Galaxy server are separated out into /var and /usr
respectively.

Cheers,

TIM

On Mon, 2013-12-16 at 16:27 +0000, Carlos Borroto wrote:
> Hi Tim,
>
> This sounds great. I'll be happy to help testing and hopefully find
> some time to help packaging once it gets into Debian Med(are you
> submitting all your packages there?).
>
> One question, for apache/nginx configuration why not use something ala
> phpMyAdmin which ask you if you want to preconfigure the package with
> a webserver in particular. The name of the DEB packaging technology to
> ask these kind of questions is evading me now. I think using something
> like that could open many possibilities in the future, like database
> backend to use, home URL, admin user/password, etc...
>
> Thanks for your work on this,
> Carlos
>
>
> On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth <[hidden email]> wrote:
> > Hi All,
> >
> > As previously mentioned, I'm back working on packaging the Galaxy server
> > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and ultimately
> > pushing towards something that could be Debian compliant.  There's a way
> > to go in that regard, but I do now have an updated package for Bio-Linux
> > in final testing and it also has a new trick: doing  "apt-get install
> > galaxy-server-apache-proxy" will set up just that with no further
> > configuration needed.  The galaxy server appears at
> > http://localhost/galaxy and users log in with their regular system
> > username and password.  Uploads are enabled via regular SFTP so no
> > special FTP server configuration is needed.
> >
> > It's a little hacky in parts but I'm generally pleased with the result.
> > If anyone want to take a look I'd welcome comments.  It's not in the
> > main BL repo yet but can be found here:
> >
> > https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
> >
> > Cheers,
> >
> > TIM
> >
> > --
> > Tim Booth <[hidden email]>
> > NERC Environmental Bioinformatics Centre
> >
> > Centre for Ecology and Hydrology
> > Maclean Bldg, Benson Lane
> > Crowmarsh Gifford
> > Wallingford, England
> > OX10 8BB
> >
> > http://nebc.nerc.ac.uk
> > <a href="tel:%2B44%201491%2069%202705" value="+441491692705">+44 1491 69 2705
> >
> > ___________________________________________________________
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   http://lists.bx.psu.edu/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/mailinglists/

--
Tim Booth <[hidden email]>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
<a href="tel:%2B44%201491%2069%202705" value="+441491692705">+44 1491 69 2705

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

Tim Booth
Hi Nate,

It's great to hear that my rearrangements for packaging Galaxy are
somewhat in line with what you are doing.  My package is a bit of a
rats-nest of symlinks right now but I'll clean these up as things become
more configurable.

One thing I'm a bit stuck on is the tool_data_table_conf.xml and
shed_tool_data_table_conf.xml files.  If the former is installed and
updated by my package and the latter is managed by the internal
tool-shed logic then this leaves nowhere for the user to make manual
edits (we can use the debian config-files system to share ownership but
it's awkward).  Also I've made a galaxy-tools-bl package that adds some
standard tool definitions in a file called bl_tool_conf.xml.  I can set
things up so this is seen by Galaxy:

[app:main]
...
tool_config_file = tool_conf.xml,shed_tool_conf.xml,bl_tool_conf.xml

So I can split out tool_conf.xml but currently I can't split out items
in tool_data_table_conf.xml (unless I make a startup script that
compiles it on the fly from XML fragments, in the style of
build_universe_config.py).  Now if it were me I'd try and eliminate this
XML file altogether, and have Galaxy just collect up all the
available .loc files on startup without the need for a central registry
file.  This would involve embedding the couple of bits of metadata from
the tool_data_table_conf.xml XML file into the .loc files themselves,
and maybe there is some reason I've not spotted why this would be a bad
idea.  Any thoughts on this?  Is it something you have considered or
would consider?

Cheers,

TIM

On Fri, 2014-03-28 at 18:49 +0000, Nate Coraor wrote:

> Hi Tim,
>
>
> I have recently been working on getting Galaxy Main's configs and
> server-modified files and directories out of the galaxy-dist
> directory, so our goals are aligning. Not everything can be moved
> without some trickery (e.g. symlinks) but most paths, including the
> paths to shed_*.xml are configurable in universe_wsgi.ini (which
> itself need not live in the galaxy-dist directory).
>
>
> --nate
>
>
> On Mon, Mar 24, 2014 at 1:38 PM, Tim Booth <[hidden email]> wrote:
>         Hi,
>        
>         This is mainly a message to Carlos and Eric who volunteered to
>         get
>         involved with my Debian packaging, but of course any other
>         help will
>         also be appreciated.  If either of you can spare time now it
>         is a good
>         moment to do so.
>        
>         I've been working on my package stuff over the last couple of
>         weeks and
>         have uploaded the latest to Debian-Med ready to build:
>        
>         http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log
>        
>         Note that you need the re-packed tarball as generated by
>         get-orig-source.sh and if you have a problem generating that
>         you can
>         grab a copy of it here:
>        
>         https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz
>        
>         I've only built this for Ubuntu, and I know that to get it
>         working on
>         Debian you'll at least need to replace the upstart job with
>         an /etc/init.d script.  After that I think you should have
>         something
>         working (see my commit notes).
>        
>         My latest efforts have been to try and get tool-shed installs
>         working.
>         Galaxy expects to be able to write to its own
>         shed_tools_*_conf.xml
>         files as well as to shed_tools and the tool-data directory.
>          It looks
>         like there is work to have a separate shed_tool-data folder
>         but this is
>         not fully working so I'm seeing if I can patch it.  Either
>         way, it's
>         vital for packaging that the files managed by DPKG and the
>         files
>         (over)written by the Galaxy server are separated out into /var
>         and /usr
>         respectively.
>        
>         Cheers,
>        
>         TIM
>        
>         On Mon, 2013-12-16 at 16:27 +0000, Carlos Borroto wrote:
>        
>         > Hi Tim,
>         >
>         > This sounds great. I'll be happy to help testing and
>         hopefully find
>         > some time to help packaging once it gets into Debian Med(are
>         you
>         > submitting all your packages there?).
>         >
>         > One question, for apache/nginx configuration why not use
>         something ala
>         > phpMyAdmin which ask you if you want to preconfigure the
>         package with
>         > a webserver in particular. The name of the DEB packaging
>         technology to
>         > ask these kind of questions is evading me now. I think using
>         something
>         > like that could open many possibilities in the future, like
>         database
>         > backend to use, home URL, admin user/password, etc...
>         >
>         > Thanks for your work on this,
>         > Carlos
>         >
>         >
>         > On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth
>         <[hidden email]> wrote:
>         > > Hi All,
>         > >
>         > > As previously mentioned, I'm back working on packaging the
>         Galaxy server
>         > > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and
>         ultimately
>         > > pushing towards something that could be Debian compliant.
>          There's a way
>         > > to go in that regard, but I do now have an updated package
>         for Bio-Linux
>         > > in final testing and it also has a new trick: doing
>          "apt-get install
>         > > galaxy-server-apache-proxy" will set up just that with no
>         further
>         > > configuration needed.  The galaxy server appears at
>         > > http://localhost/galaxy and users log in with their
>         regular system
>         > > username and password.  Uploads are enabled via regular
>         SFTP so no
>         > > special FTP server configuration is needed.
>         > >
>         > > It's a little hacky in parts but I'm generally pleased
>         with the result.
>         > > If anyone want to take a look I'd welcome comments.  It's
>         not in the
>         > > main BL repo yet but can be found here:
>         > >
>         > >
>         https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
>         > >
>         > > Cheers,
>         > >
>         > > TIM
>         > >
>         > > --
>         > > Tim Booth <[hidden email]>
>         > > NERC Environmental Bioinformatics Centre
>         > >
>         > > Centre for Ecology and Hydrology
>         > > Maclean Bldg, Benson Lane
>         > > Crowmarsh Gifford
>         > > Wallingford, England
>         > > OX10 8BB
>         > >
>         > > http://nebc.nerc.ac.uk
>         > > +44 1491 69 2705
>         > >
>         > >
>         ___________________________________________________________
>         > > Please keep all replies on the list by using "reply all"
>         > > in your mail client.  To manage your subscriptions to this
>         > > and other Galaxy lists, please use the interface at:
>         > >   http://lists.bx.psu.edu/
>         > >
>         > > To search Galaxy mailing lists use the unified search at:
>         > >   http://galaxyproject.org/search/mailinglists/
>        
>         --
>         Tim Booth <[hidden email]>
>         NERC Environmental Bioinformatics Centre
>        
>         Centre for Ecology and Hydrology
>         Maclean Bldg, Benson Lane
>         Crowmarsh Gifford
>         Wallingford, England
>         OX10 8BB
>        
>         http://nebc.nerc.ac.uk
>         +44 1491 69 2705
>        
>         ___________________________________________________________
>         Please keep all replies on the list by using "reply all"
>         in your mail client.  To manage your subscriptions to this
>         and other Galaxy lists, please use the interface at:
>           http://lists.bx.psu.edu/
>        
>         To search Galaxy mailing lists use the unified search at:
>           http://galaxyproject.org/search/mailinglists/
>        
>
>

--
Tim Booth <[hidden email]>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
+44 1491 69 2705

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Galaxy packages for Bio-Linux - an update + Apache proxying

Nate Coraor (nate@bx.psu.edu)
Hi Tim,

I agree the single tool_data_table_conf.xml is a problem and it would be nice to implement functionality for multiple. I've added a Trello card for it here: https://trello.com/c/rm2p8PpZ

As to the location file collections, I can foresee a few problems, e.g. conflicting definitions, bad formatting of the metadata in location file comments, and still needing to specify multiple paths where these location files can exist.

--nate


On Mon, Mar 31, 2014 at 8:00 AM, Tim Booth <[hidden email]> wrote:
Hi Nate,

It's great to hear that my rearrangements for packaging Galaxy are
somewhat in line with what you are doing.  My package is a bit of a
rats-nest of symlinks right now but I'll clean these up as things become
more configurable.

One thing I'm a bit stuck on is the tool_data_table_conf.xml and
shed_tool_data_table_conf.xml files.  If the former is installed and
updated by my package and the latter is managed by the internal
tool-shed logic then this leaves nowhere for the user to make manual
edits (we can use the debian config-files system to share ownership but
it's awkward).  Also I've made a galaxy-tools-bl package that adds some
standard tool definitions in a file called bl_tool_conf.xml.  I can set
things up so this is seen by Galaxy:

[app:main]
...
tool_config_file = tool_conf.xml,shed_tool_conf.xml,bl_tool_conf.xml

So I can split out tool_conf.xml but currently I can't split out items
in tool_data_table_conf.xml (unless I make a startup script that
compiles it on the fly from XML fragments, in the style of
build_universe_config.py).  Now if it were me I'd try and eliminate this
XML file altogether, and have Galaxy just collect up all the
available .loc files on startup without the need for a central registry
file.  This would involve embedding the couple of bits of metadata from
the tool_data_table_conf.xml XML file into the .loc files themselves,
and maybe there is some reason I've not spotted why this would be a bad
idea.  Any thoughts on this?  Is it something you have considered or
would consider?

Cheers,

TIM

On Fri, 2014-03-28 at 18:49 +0000, Nate Coraor wrote:
> Hi Tim,
>
>
> I have recently been working on getting Galaxy Main's configs and
> server-modified files and directories out of the galaxy-dist
> directory, so our goals are aligning. Not everything can be moved
> without some trickery (e.g. symlinks) but most paths, including the
> paths to shed_*.xml are configurable in universe_wsgi.ini (which
> itself need not live in the galaxy-dist directory).
>
>
> --nate
>
>
> On Mon, Mar 24, 2014 at 1:38 PM, Tim Booth <[hidden email]> wrote:
>         Hi,
>
>         This is mainly a message to Carlos and Eric who volunteered to
>         get
>         involved with my Debian packaging, but of course any other
>         help will
>         also be appreciated.  If either of you can spare time now it
>         is a good
>         moment to do so.
>
>         I've been working on my package stuff over the last couple of
>         weeks and
>         have uploaded the latest to Debian-Med ready to build:
>
>         http://anonscm.debian.org/viewvc/debian-med/trunk/packages/galaxy/trunk/debian/?view=log
>
>         Note that you need the re-packed tarball as generated by
>         get-orig-source.sh and if you have a problem generating that
>         you can
>         grab a copy of it here:
>
>         https://launchpad.net/~nebc/+archive/galaxy/+files/galaxy_1.bl.py27.20140210.orig.tar.xz
>
>         I've only built this for Ubuntu, and I know that to get it
>         working on
>         Debian you'll at least need to replace the upstart job with
>         an /etc/init.d script.  After that I think you should have
>         something
>         working (see my commit notes).
>
>         My latest efforts have been to try and get tool-shed installs
>         working.
>         Galaxy expects to be able to write to its own
>         shed_tools_*_conf.xml
>         files as well as to shed_tools and the tool-data directory.
>          It looks
>         like there is work to have a separate shed_tool-data folder
>         but this is
>         not fully working so I'm seeing if I can patch it.  Either
>         way, it's
>         vital for packaging that the files managed by DPKG and the
>         files
>         (over)written by the Galaxy server are separated out into /var
>         and /usr
>         respectively.
>
>         Cheers,
>
>         TIM
>
>         On Mon, 2013-12-16 at 16:27 +0000, Carlos Borroto wrote:
>
>         > Hi Tim,
>         >
>         > This sounds great. I'll be happy to help testing and
>         hopefully find
>         > some time to help packaging once it gets into Debian Med(are
>         you
>         > submitting all your packages there?).
>         >
>         > One question, for apache/nginx configuration why not use
>         something ala
>         > phpMyAdmin which ask you if you want to preconfigure the
>         package with
>         > a webserver in particular. The name of the DEB packaging
>         technology to
>         > ask these kind of questions is evading me now. I think using
>         something
>         > like that could open many possibilities in the future, like
>         database
>         > backend to use, home URL, admin user/password, etc...
>         >
>         > Thanks for your work on this,
>         > Carlos
>         >
>         >
>         > On Fri, Dec 13, 2013 at 7:03 AM, Tim Booth
>         <[hidden email]> wrote:
>         > > Hi All,
>         > >
>         > > As previously mentioned, I'm back working on packaging the
>         Galaxy server
>         > > as DEB packages for Bio-Linux (ie. Ubuntu 12.04 LTS) and
>         ultimately
>         > > pushing towards something that could be Debian compliant.
>          There's a way
>         > > to go in that regard, but I do now have an updated package
>         for Bio-Linux
>         > > in final testing and it also has a new trick: doing
>          "apt-get install
>         > > galaxy-server-apache-proxy" will set up just that with no
>         further
>         > > configuration needed.  The galaxy server appears at
>         > > http://localhost/galaxy and users log in with their
>         regular system
>         > > username and password.  Uploads are enabled via regular
>         SFTP so no
>         > > special FTP server configuration is needed.
>         > >
>         > > It's a little hacky in parts but I'm generally pleased
>         with the result.
>         > > If anyone want to take a look I'd welcome comments.  It's
>         not in the
>         > > main BL repo yet but can be found here:
>         > >
>         > >
>         https://launchpad.net/~nebc/+archive/galaxy/+sourcepub/3711751/+listing-archive-extra
>         > >
>         > > Cheers,
>         > >
>         > > TIM
>         > >
>         > > --
>         > > Tim Booth <[hidden email]>
>         > > NERC Environmental Bioinformatics Centre
>         > >
>         > > Centre for Ecology and Hydrology
>         > > Maclean Bldg, Benson Lane
>         > > Crowmarsh Gifford
>         > > Wallingford, England
>         > > OX10 8BB
>         > >
>         > > http://nebc.nerc.ac.uk
>         > > <a href="tel:%2B44%201491%2069%202705" value="+441491692705">+44 1491 69 2705
>         > >
>         > >
>         ___________________________________________________________
>         > > Please keep all replies on the list by using "reply all"
>         > > in your mail client.  To manage your subscriptions to this
>         > > and other Galaxy lists, please use the interface at:
>         > >   http://lists.bx.psu.edu/
>         > >
>         > > To search Galaxy mailing lists use the unified search at:
>         > >   http://galaxyproject.org/search/mailinglists/
>
>         --
>         Tim Booth <[hidden email]>
>         NERC Environmental Bioinformatics Centre
>
>         Centre for Ecology and Hydrology
>         Maclean Bldg, Benson Lane
>         Crowmarsh Gifford
>         Wallingford, England
>         OX10 8BB
>
>         http://nebc.nerc.ac.uk
>         <a href="tel:%2B44%201491%2069%202705" value="+441491692705">+44 1491 69 2705
>
>         ___________________________________________________________
>         Please keep all replies on the list by using "reply all"
>         in your mail client.  To manage your subscriptions to this
>         and other Galaxy lists, please use the interface at:
>           http://lists.bx.psu.edu/
>
>         To search Galaxy mailing lists use the unified search at:
>           http://galaxyproject.org/search/mailinglists/
>
>
>

--
Tim Booth <[hidden email]>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
<a href="tel:%2B44%201491%2069%202705" value="+441491692705">+44 1491 69 2705


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/