Once-run galaxy archives

classic Classic list List threaded Threaded
21 messages Options
12
| Threaded
Open this post in threaded view
|

Once-run galaxy archives

Eric Rasche
Currently the checkout options consist of hg clones, and archives that mercurial produces.
 
Having pulled or cloned galaxy a few times lately, I'm wondering if anyone would have a use for a once-run galaxy instance in an archive? I.e., I'd clone, run once to grab eggs and do the db migration, then re-tar result and store online. Might cut down on build/test times for those who are using travis or other CIs. Thoughts/opinions?

 
Cheers,
Eric
--
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M Univesity
College Station, TX 77843
Ph: <a href="http://tel:4046922048">4046922048
 
 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Aaron Petkau
Hello Eric,

I think that's a great idea.  You may be interested in checking out Björn's Galaxy Docker image https://github.com/bgruening/docker-recipes/tree/master/galaxy which is very similar to what you're describing.  A few other testing frameworks, like Galaxy bootstrap https://github.com/jmchilton/galaxy-bootstrap, use there own caching mechanism too.

Aaron


On Mon, Jul 21, 2014 at 12:51 PM, Eric Rasche <[hidden email]> wrote:
Currently the checkout options consist of hg clones, and archives that mercurial produces.
 
Having pulled or cloned galaxy a few times lately, I'm wondering if anyone would have a use for a once-run galaxy instance in an archive? I.e., I'd clone, run once to grab eggs and do the db migration, then re-tar result and store online. Might cut down on build/test times for those who are using travis or other CIs. Thoughts/opinions?

 
Cheers,
Eric
--
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M Univesity
College Station, TX 77843
Ph: <a href="http://tel:4046922048" target="_blank">4046922048
 
 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche

Hi Aaron,

Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case.

Cheers,

Eric


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Aaron Petkau
Hello Eric,

Your right about that, downloading the archive, installing all the eggs, and then updating the database takes a bit of time (especially if you're like me and like re-running tests on nearly every change you make :P).  I think it would be cool to have a pre-package Galaxy for integration testing which is quick to setup.  I once thought of downloading Björn's Docker image from Galaxy Bootstrap and using it that way, but thinking is about as far as I got with that one.  One problem I could see is it would have to be re-built on every release of Galaxy you want to test against (whereas mercurial cloning/pulling makes sure you're always up to date with the latest code).

Aaron


On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]> wrote:

Hi Aaron,

Good points, I was considering using galaxy bootstrap. This is mostly for the CI folk who want to download an archive, unpack it, and be ready to install/test their tools. The hg clone and egg/db steps seem like unnecessary overhead for this specific use case.

Cheers,

Eric



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche
Hi Aaron,

Yeah, absolutely understandable. I want my tools tested early and often.

I abuse my CI server for everything, especially for building and
packaging software. In this case I was imagining that I might have it
produce an archive on every tagged release, as well as producing a
"daily" archive. All of these would be available on some ftp/http server
somewhere with symlinks for latest archives (e.g.
galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work
for your use case as well?

Eric

On 07/21/2014 03:02 PM, Aaron Petkau wrote:

> Hello Eric,
>
> Your right about that, downloading the archive, installing all the eggs,
> and then updating the database takes a bit of time (especially if you're
> like me and like re-running tests on nearly every change you make :P).
> I think it would be cool to have a pre-package Galaxy for integration
> testing which is quick to setup.  I once thought of downloading Björn's
> Docker image from Galaxy Bootstrap and using it that way, but thinking
> is about as far as I got with that one.  One problem I could see is it
> would have to be re-built on every release of Galaxy you want to test
> against (whereas mercurial cloning/pulling makes sure you're always up
> to date with the latest code).
>
> Aaron
>
>
> On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Aaron,
>
>     Good points, I was considering using galaxy bootstrap. This is
>     mostly for the CI folk who want to download an archive, unpack it,
>     and be ready to install/test their tools. The hg clone and egg/db
>     steps seem like unnecessary overhead for this specific use case.
>
>     Cheers,
>
>     Eric
>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Aaron Petkau
Hello Eric,

That sounds like a pretty good idea.  If there was a pre-built image available for whatever release I wanted to test against I could just cache it and (hopefully) get my tests running a bit faster.  I'm not sure if anyone else is already doing this?

Also, I remember there being mentioned pre-building docker images for each release of Galaxy, which would accomplish something similar, but I'm not really sure how that's being handled.  I think Björn's Docker image is kept up to date with Galaxy stable each time it's built https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.  So, this could be handled by modifying his Dockerfile to build Galaxy at whatever tagged release you want to test against.

One downside to docker is that you need to get it installed on your CI server, which may or may not be possible (needs a very recent kernel for example).

Aaron


On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]> wrote:
Hi Aaron,

Yeah, absolutely understandable. I want my tools tested early and often.

I abuse my CI server for everything, especially for building and
packaging software. In this case I was imagining that I might have it
produce an archive on every tagged release, as well as producing a
"daily" archive. All of these would be available on some ftp/http server
somewhere with symlinks for latest archives (e.g.
galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work
for your use case as well?

Eric

On 07/21/2014 03:02 PM, Aaron Petkau wrote:
> Hello Eric,
>
> Your right about that, downloading the archive, installing all the eggs,
> and then updating the database takes a bit of time (especially if you're
> like me and like re-running tests on nearly every change you make :P).
> I think it would be cool to have a pre-package Galaxy for integration
> testing which is quick to setup.  I once thought of downloading Björn's
> Docker image from Galaxy Bootstrap and using it that way, but thinking
> is about as far as I got with that one.  One problem I could see is it
> would have to be re-built on every release of Galaxy you want to test
> against (whereas mercurial cloning/pulling makes sure you're always up
> to date with the latest code).
>
> Aaron
>
>
> On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Aaron,
>
>     Good points, I was considering using galaxy bootstrap. This is
>     mostly for the CI folk who want to download an archive, unpack it,
>     and be ready to install/test their tools. The hg clone and egg/db
>     steps seem like unnecessary overhead for this specific use case.
>
>     Cheers,
>
>     Eric
>
>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Peter Cock
In reply to this post by Eric Rasche
On Mon, Jul 21, 2014 at 6:51 PM, Eric Rasche <[hidden email]> wrote:
> Currently the checkout options consist of hg clones, and archives that
> mercurial produces.
>
> Having pulled or cloned galaxy a few times lately, I'm wondering if anyone
> would have a use for a once-run galaxy instance in an archive? I.e., I'd
> clone, run once to grab eggs and do the db migration, then re-tar result and
> store online. Might cut down on build/test times for those who are using
> travis or other CIs. Thoughts/opinions?

Hi Eric,

Given how close you can get now for minimal effort,
this seem unnecessary.

http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html

My TravisCI setup this fetches the latest Galaxy as
a tar ball (from a GitHub mirror as it was faster than
a git clone which was faster than getting the tar ball
from BitBucket, which in turn was faster than using
hg clone), and a per-migrated SQLite database
(using a bit of Galaxy functionality originally with
$GALAXY_TEST_DB_TEMPLATE added to speed
up running the functional tests).

Note this does not cache the eggs and all the other
side effects of the first run like creating config files,
so there is room for some speed up.

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Björn Grüning-3
In reply to this post by Aaron Petkau
Hi Aaron and Eric,

Am 21.07.2014 22:58, schrieb Aaron Petkau:

> Hello Eric,
>
> That sounds like a pretty good idea.  If there was a pre-built image
> available for whatever release I wanted to test against I could just cache
> it and (hopefully) get my tests running a bit faster.  I'm not sure if
> anyone else is already doing this?
>
> Also, I remember there being mentioned pre-building docker images for each
> release of Galaxy, which would accomplish something similar, but I'm not
> really sure how that's being handled.  I think Björn's Docker image is kept
> up to date with Galaxy stable each time it's built
> https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
> So, this could be handled by modifying his Dockerfile to build Galaxy at
> whatever tagged release you want to test against.

I will try hard to create with every Galaxy stable release a new Galaxy
docker image. You can create a github branch with a specific tag, that
will end up as a new tagged version of the main Galaxy Docker image.

> One downside to docker is that you need to get it installed on your CI
> server, which may or may not be possible (needs a very recent kernel for
> example).

So true. SL7 for the win! :)

Docker, Docker, Docker!
Bjoern

> Aaron
>
>
> On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]> wrote:
>
>> Hi Aaron,
>>
>> Yeah, absolutely understandable. I want my tools tested early and often.
>>
>> I abuse my CI server for everything, especially for building and
>> packaging software. In this case I was imagining that I might have it
>> produce an archive on every tagged release, as well as producing a
>> "daily" archive. All of these would be available on some ftp/http server
>> somewhere with symlinks for latest archives (e.g.
>> galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that work
>> for your use case as well?
>>
>> Eric
>>
>> On 07/21/2014 03:02 PM, Aaron Petkau wrote:
>>> Hello Eric,
>>>
>>> Your right about that, downloading the archive, installing all the eggs,
>>> and then updating the database takes a bit of time (especially if you're
>>> like me and like re-running tests on nearly every change you make :P).
>>> I think it would be cool to have a pre-package Galaxy for integration
>>> testing which is quick to setup.  I once thought of downloading Björn's
>>> Docker image from Galaxy Bootstrap and using it that way, but thinking
>>> is about as far as I got with that one.  One problem I could see is it
>>> would have to be re-built on every release of Galaxy you want to test
>>> against (whereas mercurial cloning/pulling makes sure you're always up
>>> to date with the latest code).
>>>
>>> Aaron
>>>
>>>
>>> On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
>>> <mailto:[hidden email]>> wrote:
>>>
>>>      Hi Aaron,
>>>
>>>      Good points, I was considering using galaxy bootstrap. This is
>>>      mostly for the CI folk who want to download an archive, unpack it,
>>>      and be ready to install/test their tools. The hg clone and egg/db
>>>      steps seem like unnecessary overhead for this specific use case.
>>>
>>>      Cheers,
>>>
>>>      Eric
>>>
>>>
>>
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche
Hi Björn,

On July 22, 2014 3:17:38 AM CDT, "Björn Grüning" <[hidden email]> wrote:

>Hi Aaron and Eric,
>
>Am 21.07.2014 22:58, schrieb Aaron Petkau:
>> Hello Eric,
>>
>> That sounds like a pretty good idea.  If there was a pre-built image
>> available for whatever release I wanted to test against I could just
>cache
>> it and (hopefully) get my tests running a bit faster.  I'm not sure
>if
>> anyone else is already doing this?
>>
>> Also, I remember there being mentioned pre-building docker images for
>each
>> release of Galaxy, which would accomplish something similar, but I'm
>not
>> really sure how that's being handled.  I think Björn's Docker image
>is kept
>> up to date with Galaxy stable each time it's built
>>
>https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
>> So, this could be handled by modifying his Dockerfile to build Galaxy
>at
>> whatever tagged release you want to test against.
>
>I will try hard to create with every Galaxy stable release a new Galaxy
>
>docker image. You can create a github branch with a specific tag, that
>will end up as a new tagged version of the main Galaxy Docker image.

Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!

>> One downside to docker is that you need to get it installed on your
>CI
>> server, which may or may not be possible (needs a very recent kernel
>for
>> example).
>
>So true. SL7 for the win! :)
>
>Docker, Docker, Docker!
>Bjoern
>
>> Aaron
>>
>>
>> On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]>
>wrote:
>>
>>> Hi Aaron,
>>>
>>> Yeah, absolutely understandable. I want my tools tested early and
>often.
>>>
>>> I abuse my CI server for everything, especially for building and
>>> packaging software. In this case I was imagining that I might have
>it
>>> produce an archive on every tagged release, as well as producing a
>>> "daily" archive. All of these would be available on some ftp/http
>server
>>> somewhere with symlinks for latest archives (e.g.
>>> galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that
>work
>>> for your use case as well?
>>>
>>> Eric
>>>
>>> On 07/21/2014 03:02 PM, Aaron Petkau wrote:
>>>> Hello Eric,
>>>>
>>>> Your right about that, downloading the archive, installing all the
>eggs,
>>>> and then updating the database takes a bit of time (especially if
>you're
>>>> like me and like re-running tests on nearly every change you make
>:P).
>>>> I think it would be cool to have a pre-package Galaxy for
>integration
>>>> testing which is quick to setup.  I once thought of downloading
>Björn's
>>>> Docker image from Galaxy Bootstrap and using it that way, but
>thinking
>>>> is about as far as I got with that one.  One problem I could see is
>it
>>>> would have to be re-built on every release of Galaxy you want to
>test
>>>> against (whereas mercurial cloning/pulling makes sure you're always
>up
>>>> to date with the latest code).
>>>>
>>>> Aaron
>>>>
>>>>
>>>> On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
>>>> <mailto:[hidden email]>> wrote:
>>>>
>>>>      Hi Aaron,
>>>>
>>>>      Good points, I was considering using galaxy bootstrap. This is
>>>>      mostly for the CI folk who want to download an archive, unpack
>it,
>>>>      and be ready to install/test their tools. The hg clone and
>egg/db
>>>>      steps seem like unnecessary overhead for this specific use
>case.
>>>>
>>>>      Cheers,
>>>>
>>>>      Eric
>>>>
>>>>
>>>
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/mailinglists/
>>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche
In reply to this post by Peter Cock
Hi Peter,

On July 22, 2014 3:15:41 AM CDT, Peter Cock <[hidden email]> wrote:

>On Mon, Jul 21, 2014 at 6:51 PM, Eric Rasche <[hidden email]>
>wrote:
>> Currently the checkout options consist of hg clones, and archives
>that
>> mercurial produces.
>>
>> Having pulled or cloned galaxy a few times lately, I'm wondering if
>anyone
>> would have a use for a once-run galaxy instance in an archive? I.e.,
>I'd
>> clone, run once to grab eggs and do the db migration, then re-tar
>result and
>> store online. Might cut down on build/test times for those who are
>using
>> travis or other CIs. Thoughts/opinions?
>
>Hi Eric,
>
>Given how close you can get now for minimal effort,
>this seem unnecessary.
>
>http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html
>
>My TravisCI setup this fetches the latest Galaxy as
>a tar ball (from a GitHub mirror as it was faster than
>a git clone which was faster than getting the tar ball
>from BitBucket, which in turn was faster than using

Yes, that post was at least part of the thinking behind this.

>hg clone), and a per-migrated SQLite database
>(using a bit of Galaxy functionality originally with
>$GALAXY_TEST_DB_TEMPLATE added to speed
>up running the functional tests).

I know I've seen that used but was never able to get that working in practice (then again I didn't try that hard). If that's a working/usable feature, then that is already the majority of setup time. Eggs would be nice but not the biggest thing in the world.

>Note this does not cache the eggs and all the other
>side effects of the first run like creating config files,
>so there is room for some speed up.
>
>Regards,
>
>Peter

Cheers,
Eric
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Peter Cock
On Tue, Jul 22, 2014 at 1:15 PM, Eric Rasche <[hidden email]> wrote:

> Hi Peter,
>
> On July 22, 2014 3:15:41 AM CDT, Peter Cock <[hidden email]> wrote:
>>
>>Given how close you can get now for minimal effort,
>>this seem unnecessary.
>>
>>http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html
>>
>>My TravisCI setup this fetches the latest Galaxy as
>>a tar ball (from a GitHub mirror as it was faster than
>>a git clone which was faster than getting the tar ball
>>from BitBucket, which in turn was faster than using
>>hg clone),
>
> Yes, that post was at least part of the thinking behind this.

:)

>> .., and a per-migrated SQLite database
>>(using a bit of Galaxy functionality originally with
>>$GALAXY_TEST_DB_TEMPLATE added to speed
>>up running the functional tests).

Apologies for grammatical error - I pasted in the environment
variable at the wrong point in the sentence.

> I know I've seen that used but was never able to get that
> working in practice (then again I didn't try that hard). If
> that's a working/usable feature, then that is already the
> majority of setup time.

Yes, the creation of the test-database and all the migrations
was an obvious low-hanging fruit when we were looking at
making running the tool functional tests faster - although
originally in the context of running the tests on a local
development Galaxy instance.

As to using this in practise, currently my TravisCI setup has:

export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0117.sqlite

I also added that line at the start of my local copy of script
run_functional_tests.sh to benefit from this while doing
development. That should be all there is to it (but from
memory, this is only for use with the SQLite backend).

John - could you add a current schema snapshot to
https://github.com/jmchilton/galaxy-downloads/ please?

>>Note this does not cache the eggs and all the other
>>side effects of the first run like creating config files,
>>so there is room for some speed up.
>
> Eggs would be nice but not the biggest thing in the world.

Right. I do like your idea of automatically generated
cutting-edge or each stable release Docker images
though (even if I have no personal need for them at
the moment).

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Björn Grüning-3
In reply to this post by Eric Rasche
Hi Eric,

>>> That sounds like a pretty good idea.  If there was a pre-built image
>>> available for whatever release I wanted to test against I could just
>> cache
>>> it and (hopefully) get my tests running a bit faster.  I'm not sure
>> if
>>> anyone else is already doing this?
>>>
>>> Also, I remember there being mentioned pre-building docker images for
>> each
>>> release of Galaxy, which would accomplish something similar, but I'm
>> not
>>> really sure how that's being handled.  I think Björn's Docker image
>> is kept
>>> up to date with Galaxy stable each time it's built
>>>
>> https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
>>> So, this could be handled by modifying his Dockerfile to build Galaxy
>> at
>>> whatever tagged release you want to test against.
>>
>> I will try hard to create with every Galaxy stable release a new Galaxy
>>
>> docker image. You can create a github branch with a specific tag, that
>> will end up as a new tagged version of the main Galaxy Docker image.
>
> Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!

The hardest part is to remind myself ;)
The procedure is:

1) create a new branch for the galaxy-docker github account
2) change the version string in the git-clone command in the Dockerfile
3) login into the docker-index site and re-associate a new tag to the
new branch ... click the build button

I could simplify that a little bit if the galaxy-docker image has its
own repository. The docker-index has a build-on-push feature.
But currently every image (all branches) are build again. There is no
way to only trigger one branch build. Until that is fixed in the
docker-index I will do it manually.

So you see any improvements in that setup? Let me know!
Cheers,
Bjoern

>>> One downside to docker is that you need to get it installed on your
>> CI
>>> server, which may or may not be possible (needs a very recent kernel
>> for
>>> example).
>>
>> So true. SL7 for the win! :)
>>
>> Docker, Docker, Docker!
>> Bjoern
>>
>>> Aaron
>>>
>>>
>>> On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]>
>> wrote:
>>>
>>>> Hi Aaron,
>>>>
>>>> Yeah, absolutely understandable. I want my tools tested early and
>> often.
>>>>
>>>> I abuse my CI server for everything, especially for building and
>>>> packaging software. In this case I was imagining that I might have
>> it
>>>> produce an archive on every tagged release, as well as producing a
>>>> "daily" archive. All of these would be available on some ftp/http
>> server
>>>> somewhere with symlinks for latest archives (e.g.
>>>> galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that
>> work
>>>> for your use case as well?
>>>>
>>>> Eric
>>>>
>>>> On 07/21/2014 03:02 PM, Aaron Petkau wrote:
>>>>> Hello Eric,
>>>>>
>>>>> Your right about that, downloading the archive, installing all the
>> eggs,
>>>>> and then updating the database takes a bit of time (especially if
>> you're
>>>>> like me and like re-running tests on nearly every change you make
>> :P).
>>>>> I think it would be cool to have a pre-package Galaxy for
>> integration
>>>>> testing which is quick to setup.  I once thought of downloading
>> Björn's
>>>>> Docker image from Galaxy Bootstrap and using it that way, but
>> thinking
>>>>> is about as far as I got with that one.  One problem I could see is
>> it
>>>>> would have to be re-built on every release of Galaxy you want to
>> test
>>>>> against (whereas mercurial cloning/pulling makes sure you're always
>> up
>>>>> to date with the latest code).
>>>>>
>>>>> Aaron
>>>>>
>>>>>
>>>>> On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
>>>>> <mailto:[hidden email]>> wrote:
>>>>>
>>>>>       Hi Aaron,
>>>>>
>>>>>       Good points, I was considering using galaxy bootstrap. This is
>>>>>       mostly for the CI folk who want to download an archive, unpack
>> it,
>>>>>       and be ready to install/test their tools. The hg clone and
>> egg/db
>>>>>       steps seem like unnecessary overhead for this specific use
>> case.
>>>>>
>>>>>       Cheers,
>>>>>
>>>>>       Eric
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>     http://lists.bx.psu.edu/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>     http://galaxyproject.org/search/mailinglists/
>>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche
Hi Björn,

22.07.2014, 14:26, "Björn Grüning" <[hidden email]>:

>  Hi Eric,
>>>>   That sounds like a pretty good idea.  If there was a pre-built image
>>>>   available for whatever release I wanted to test against I could just
>>>   cache
>>>>   it and (hopefully) get my tests running a bit faster.  I'm not sure
>>>   if
>>>>   anyone else is already doing this?
>>>>
>>>>   Also, I remember there being mentioned pre-building docker images for
>>>   each
>>>>   release of Galaxy, which would accomplish something similar, but I'm
>>>   not
>>>>   really sure how that's being handled.  I think Björn's Docker image
>>>   is kept
>>>>   up to date with Galaxy stable each time it's built
>>>   https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
>>>>   So, this could be handled by modifying his Dockerfile to build Galaxy
>>>   at
>>>>   whatever tagged release you want to test against.
>>>   I will try hard to create with every Galaxy stable release a new Galaxy
>>>
>>>   docker image. You can create a github branch with a specific tag, that
>>>   will end up as a new tagged version of the main Galaxy Docker image.
>>   Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!
>  The hardest part is to remind myself ;)
>  The procedure is:
>
>  1) create a new branch for the galaxy-docker github account
>  2) change the version string in the git-clone command in the Dockerfile
>  3) login into the docker-index site and re-associate a new tag to the
>  new branch ... click the build button
>
>  I could simplify that a little bit if the galaxy-docker image has its
>  own repository. The docker-index has a build-on-push feature.
>  But currently every image (all branches) are build again. There is no
>  way to only trigger one branch build. Until that is fixed in the
>  docker-index I will do it manually.
>
>  So you see any improvements in that setup? Let me know!

Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant.

Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say "hey, associate the branch/trigger a build"

>  Cheers,
>  Bjoern
>>>>   One downside to docker is that you need to get it installed on your
>>>   CI
>>>>   server, which may or may not be possible (needs a very recent kernel
>>>   for
>>>>   example).
>>>   So true. SL7 for the win! :)
>>>
>>>   Docker, Docker, Docker!
>>>   Bjoern
>>>>   Aaron
>>>>
>>>>   On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]>
>>>   wrote:
>>>>>   Hi Aaron,
>>>>>
>>>>>   Yeah, absolutely understandable. I want my tools tested early and
>>>   often.
>>>>>   I abuse my CI server for everything, especially for building and
>>>>>   packaging software. In this case I was imagining that I might have
>>>   it
>>>>>   produce an archive on every tagged release, as well as producing a
>>>>>   "daily" archive. All of these would be available on some ftp/http
>>>   server
>>>>>   somewhere with symlinks for latest archives (e.g.
>>>>>   galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that
>>>   work
>>>>>   for your use case as well?
>>>>>
>>>>>   Eric
>>>>>
>>>>>   On 07/21/2014 03:02 PM, Aaron Petkau wrote:
>>>>>>   Hello Eric,
>>>>>>
>>>>>>   Your right about that, downloading the archive, installing all the
>>>   eggs,
>>>>>>   and then updating the database takes a bit of time (especially if
>>>   you're
>>>>>>   like me and like re-running tests on nearly every change you make
>>>   :P).
>>>>>>   I think it would be cool to have a pre-package Galaxy for
>>>   integration
>>>>>>   testing which is quick to setup.  I once thought of downloading
>>>   Björn's
>>>>>>   Docker image from Galaxy Bootstrap and using it that way, but
>>>   thinking
>>>>>>   is about as far as I got with that one.  One problem I could see is
>>>   it
>>>>>>   would have to be re-built on every release of Galaxy you want to
>>>   test
>>>>>>   against (whereas mercurial cloning/pulling makes sure you're always
>>>   up
>>>>>>   to date with the latest code).
>>>>>>
>>>>>>   Aaron
>>>>>>
>>>>>>   On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
>>>>>>   <mailto:[hidden email]>> wrote:
>>>>>>
>>>>>>         Hi Aaron,
>>>>>>
>>>>>>         Good points, I was considering using galaxy bootstrap. This is
>>>>>>         mostly for the CI folk who want to download an archive, unpack
>>>   it,
>>>>>>         and be ready to install/test their tools. The hg clone and
>>>   egg/db
>>>>>>         steps seem like unnecessary overhead for this specific use
>>>   case.
>>>>>>         Cheers,
>>>>>>
>>>>>>         Eric
>>>>   ___________________________________________________________
>>>>   Please keep all replies on the list by using "reply all"
>>>>   in your mail client.  To manage your subscriptions to this
>>>>   and other Galaxy lists, please use the interface at:
>>>>       http://lists.bx.psu.edu/
>>>>
>>>>   To search Galaxy mailing lists use the unified search at:
>>>>       http://galaxyproject.org/search/mailinglists/

Cheers,
Eric

--
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M Univesity
College Station, TX 77843
Ph: 4046922048
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Björn Grüning-3
:) great I like it!
Will do it shortly!

Am 22.07.2014 15:36, schrieb Eric Rasche:

> Hi Björn,
>
> 22.07.2014, 14:26, "Björn Grüning" <[hidden email]>:
>>   Hi Eric,
>>>>>    That sounds like a pretty good idea.  If there was a pre-built image
>>>>>    available for whatever release I wanted to test against I could just
>>>>    cache
>>>>>    it and (hopefully) get my tests running a bit faster.  I'm not sure
>>>>    if
>>>>>    anyone else is already doing this?
>>>>>
>>>>>    Also, I remember there being mentioned pre-building docker images for
>>>>    each
>>>>>    release of Galaxy, which would accomplish something similar, but I'm
>>>>    not
>>>>>    really sure how that's being handled.  I think Björn's Docker image
>>>>    is kept
>>>>>    up to date with Galaxy stable each time it's built
>>>>    https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
>>>>>    So, this could be handled by modifying his Dockerfile to build Galaxy
>>>>    at
>>>>>    whatever tagged release you want to test against.
>>>>    I will try hard to create with every Galaxy stable release a new Galaxy
>>>>
>>>>    docker image. You can create a github branch with a specific tag, that
>>>>    will end up as a new tagged version of the main Galaxy Docker image.
>>>    Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!
>>   The hardest part is to remind myself ;)
>>   The procedure is:
>>
>>   1) create a new branch for the galaxy-docker github account
>>   2) change the version string in the git-clone command in the Dockerfile
>>   3) login into the docker-index site and re-associate a new tag to the
>>   new branch ... click the build button
>>
>>   I could simplify that a little bit if the galaxy-docker image has its
>>   own repository. The docker-index has a build-on-push feature.
>>   But currently every image (all branches) are build again. There is no
>>   way to only trigger one branch build. Until that is fixed in the
>>   docker-index I will do it manually.
>>
>>   So you see any improvements in that setup? Let me know!
>
> Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant.
>
> Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say "hey, associate the branch/trigger a build"
>
>>   Cheers,
>>   Bjoern
>>>>>    One downside to docker is that you need to get it installed on your
>>>>    CI
>>>>>    server, which may or may not be possible (needs a very recent kernel
>>>>    for
>>>>>    example).
>>>>    So true. SL7 for the win! :)
>>>>
>>>>    Docker, Docker, Docker!
>>>>    Bjoern
>>>>>    Aaron
>>>>>
>>>>>    On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]>
>>>>    wrote:
>>>>>>    Hi Aaron,
>>>>>>
>>>>>>    Yeah, absolutely understandable. I want my tools tested early and
>>>>    often.
>>>>>>    I abuse my CI server for everything, especially for building and
>>>>>>    packaging software. In this case I was imagining that I might have
>>>>    it
>>>>>>    produce an archive on every tagged release, as well as producing a
>>>>>>    "daily" archive. All of these would be available on some ftp/http
>>>>    server
>>>>>>    somewhere with symlinks for latest archives (e.g.
>>>>>>    galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that
>>>>    work
>>>>>>    for your use case as well?
>>>>>>
>>>>>>    Eric
>>>>>>
>>>>>>    On 07/21/2014 03:02 PM, Aaron Petkau wrote:
>>>>>>>    Hello Eric,
>>>>>>>
>>>>>>>    Your right about that, downloading the archive, installing all the
>>>>    eggs,
>>>>>>>    and then updating the database takes a bit of time (especially if
>>>>    you're
>>>>>>>    like me and like re-running tests on nearly every change you make
>>>>    :P).
>>>>>>>    I think it would be cool to have a pre-package Galaxy for
>>>>    integration
>>>>>>>    testing which is quick to setup.  I once thought of downloading
>>>>    Björn's
>>>>>>>    Docker image from Galaxy Bootstrap and using it that way, but
>>>>    thinking
>>>>>>>    is about as far as I got with that one.  One problem I could see is
>>>>    it
>>>>>>>    would have to be re-built on every release of Galaxy you want to
>>>>    test
>>>>>>>    against (whereas mercurial cloning/pulling makes sure you're always
>>>>    up
>>>>>>>    to date with the latest code).
>>>>>>>
>>>>>>>    Aaron
>>>>>>>
>>>>>>>    On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
>>>>>>>    <mailto:[hidden email]>> wrote:
>>>>>>>
>>>>>>>          Hi Aaron,
>>>>>>>
>>>>>>>          Good points, I was considering using galaxy bootstrap. This is
>>>>>>>          mostly for the CI folk who want to download an archive, unpack
>>>>    it,
>>>>>>>          and be ready to install/test their tools. The hg clone and
>>>>    egg/db
>>>>>>>          steps seem like unnecessary overhead for this specific use
>>>>    case.
>>>>>>>          Cheers,
>>>>>>>
>>>>>>>          Eric
>>>>>    ___________________________________________________________
>>>>>    Please keep all replies on the list by using "reply all"
>>>>>    in your mail client.  To manage your subscriptions to this
>>>>>    and other Galaxy lists, please use the interface at:
>>>>>        http://lists.bx.psu.edu/
>>>>>
>>>>>    To search Galaxy mailing lists use the unified search at:
>>>>>        http://galaxyproject.org/search/mailinglists/
>
> Cheers,
> Eric
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Aaron Petkau
That's awesome Björn and Eric.  And, I'll also have to go through your Travis CI document Peter.  It looks really cool.

Aaron


On Tue, Jul 22, 2014 at 9:35 AM, Björn Grüning <[hidden email]> wrote:
:) great I like it!
Will do it shortly!

Am 22.07.2014 15:36, schrieb Eric Rasche:

Hi Björn,

22.07.2014, 14:26, "Björn Grüning" <[hidden email]>:
  Hi Eric,
   That sounds like a pretty good idea.  If there was a pre-built image
   available for whatever release I wanted to test against I could just
   cache
   it and (hopefully) get my tests running a bit faster.  I'm not sure
   if
   anyone else is already doing this?

   Also, I remember there being mentioned pre-building docker images for
   each
   release of Galaxy, which would accomplish something similar, but I'm
   not
   really sure how that's being handled.  I think Björn's Docker image
   is kept
   up to date with Galaxy stable each time it's built
   https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
   So, this could be handled by modifying his Dockerfile to build Galaxy
   at
   whatever tagged release you want to test against.
   I will try hard to create with every Galaxy stable release a new Galaxy

   docker image. You can create a github branch with a specific tag, that
   will end up as a new tagged version of the main Galaxy Docker image.
   Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!
  The hardest part is to remind myself ;)
  The procedure is:

  1) create a new branch for the galaxy-docker github account
  2) change the version string in the git-clone command in the Dockerfile
  3) login into the docker-index site and re-associate a new tag to the
  new branch ... click the build button

  I could simplify that a little bit if the galaxy-docker image has its
  own repository. The docker-index has a build-on-push feature.
  But currently every image (all branches) are build again. There is no
  way to only trigger one branch build. Until that is fixed in the
  docker-index I will do it manually.

  So you see any improvements in that setup? Let me know!

Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant.

Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say "hey, associate the branch/trigger a build"

  Cheers,
  Bjoern
   One downside to docker is that you need to get it installed on your
   CI
   server, which may or may not be possible (needs a very recent kernel
   for
   example).
   So true. SL7 for the win! :)

   Docker, Docker, Docker!
   Bjoern
   Aaron

   On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]>
   wrote:
   Hi Aaron,

   Yeah, absolutely understandable. I want my tools tested early and
   often.
   I abuse my CI server for everything, especially for building and
   packaging software. In this case I was imagining that I might have
   it
   produce an archive on every tagged release, as well as producing a
   "daily" archive. All of these would be available on some ftp/http
   server
   somewhere with symlinks for latest archives (e.g.
   galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that
   work
   for your use case as well?

   Eric

   On 07/21/2014 03:02 PM, Aaron Petkau wrote:
   Hello Eric,

   Your right about that, downloading the archive, installing all the
   eggs,
   and then updating the database takes a bit of time (especially if
   you're
   like me and like re-running tests on nearly every change you make
   :P).
   I think it would be cool to have a pre-package Galaxy for
   integration
   testing which is quick to setup.  I once thought of downloading
   Björn's
   Docker image from Galaxy Bootstrap and using it that way, but
   thinking
   is about as far as I got with that one.  One problem I could see is
   it
   would have to be re-built on every release of Galaxy you want to
   test
   against (whereas mercurial cloning/pulling makes sure you're always
   up
   to date with the latest code).

   Aaron

   On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
   <mailto:[hidden email]>> wrote:

         Hi Aaron,

         Good points, I was considering using galaxy bootstrap. This is
         mostly for the CI folk who want to download an archive, unpack
   it,
         and be ready to install/test their tools. The hg clone and
   egg/db
         steps seem like unnecessary overhead for this specific use
   case.
         Cheers,

         Eric
   ___________________________________________________________
   Please keep all replies on the list by using "reply all"
   in your mail client.  To manage your subscriptions to this
   and other Galaxy lists, please use the interface at:
       http://lists.bx.psu.edu/

   To search Galaxy mailing lists use the unified search at:
       http://galaxyproject.org/search/mailinglists/

Cheers,
Eric



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche
In reply to this post by Björn Grüning-3

For my part:


Script/infra to generate branches
https://cpt.tamu.edu/gitlab/rasche.eric/docker-branch-generator/tree/master

Generated branches:
https://cpt.tamu.edu/gitlab/rasche.eric/generated-docker-branches/branches/recent

Need to patch up a couple issues, but I'm pretty much done on my end. Whenever you're ready, we can move this to github and someone can just run a cron job once a month. :)


22.07.2014, 15:35, "Björn Grüning" <[hidden email]>:

 :) great I like it!
 Will do it shortly!

 Am 22.07.2014 15:36, schrieb Eric Rasche:
  Hi Björn,

  22.07.2014, 14:26, "Björn Grüning" <[hidden email]>:
    Hi Eric,
     That sounds like a pretty good idea.  If there was a pre-built image
     available for whatever release I wanted to test against I could just
     cache
     it and (hopefully) get my tests running a bit faster.  I'm not sure
     if
     anyone else is already doing this?

     Also, I remember there being mentioned pre-building docker images for
     each
     release of Galaxy, which would accomplish something similar, but I'm
     not
     really sure how that's being handled.  I think Björn's Docker image
     is kept
     up to date with Galaxy stable each time it's built
     https://github.com/bgruening/docker-recipes/blob/master/galaxy/Dockerfile#L51.
     So, this could be handled by modifying his Dockerfile to build Galaxy
     at
     whatever tagged release you want to test against.
     I will try hard to create with every Galaxy stable release a new Galaxy

     docker image. You can create a github branch with a specific tag, that
     will end up as a new tagged version of the main Galaxy Docker image.
     Try hard to create? No no, what can we do to automatically create these? I'm not so familiar with how one might build a galaxy release specific docker image, but if you can provide a generalized process, let's stick it in a CI server/cron job somewhere and never worry about it again!
    The hardest part is to remind myself ;)
    The procedure is:

    1) create a new branch for the galaxy-docker github account
    2) change the version string in the git-clone command in the Dockerfile
    3) login into the docker-index site and re-associate a new tag to the
    new branch ... click the build button

    I could simplify that a little bit if the galaxy-docker image has its
    own repository. The docker-index has a build-on-push feature.
    But currently every image (all branches) are build again. There is no
    way to only trigger one branch build. Until that is fixed in the
    docker-index I will do it manually.

    So you see any improvements in that setup? Let me know!
  Naturally! 1 and 2 could be automated out with a script. 3 could probably be fixed with a script, but that requires parsing pages/crafting cURL queries and that's less pleasant.

  Let's have a new repository just for galaxy-docker images. I'll write up a script to check for updates and create+push branches as needed, we can place this in a cron/CI job and have it email you whenever it's run to say "hey, associate the branch/trigger a build"
    Cheers,
    Bjoern
     One downside to docker is that you need to get it installed on your
     CI
     server, which may or may not be possible (needs a very recent kernel
     for
     example).
     So true. SL7 for the win! :)

     Docker, Docker, Docker!
     Bjoern
     Aaron

     On Mon, Jul 21, 2014 at 3:12 PM, Eric Rasche <[hidden email]>
     wrote:
     Hi Aaron,

     Yeah, absolutely understandable. I want my tools tested early and
     often.
     I abuse my CI server for everything, especially for building and
     packaging software. In this case I was imagining that I might have
     it
     produce an archive on every tagged release, as well as producing a
     "daily" archive. All of these would be available on some ftp/http
     server
     somewhere with symlinks for latest archives (e.g.
     galaxy-latest-release.tgz and galaxy-latest-daily.tgz). Would that
     work
     for your use case as well?

     Eric

     On 07/21/2014 03:02 PM, Aaron Petkau wrote:
     Hello Eric,

     Your right about that, downloading the archive, installing all the
     eggs,
     and then updating the database takes a bit of time (especially if
     you're
     like me and like re-running tests on nearly every change you make
     :P).
     I think it would be cool to have a pre-package Galaxy for
     integration
     testing which is quick to setup.  I once thought of downloading
     Björn's
     Docker image from Galaxy Bootstrap and using it that way, but
     thinking
     is about as far as I got with that one.  One problem I could see is
     it
     would have to be re-built on every release of Galaxy you want to
     test
     against (whereas mercurial cloning/pulling makes sure you're always
     up
     to date with the latest code).

     Aaron

     On Mon, Jul 21, 2014 at 2:45 PM, Eric Rasche <[hidden email]
     <mailto:[hidden email]>> wrote:

           Hi Aaron,

           Good points, I was considering using galaxy bootstrap. This is
           mostly for the CI folk who want to download an archive, unpack
     it,
           and be ready to install/test their tools. The hg clone and
     egg/db
           steps seem like unnecessary overhead for this specific use
     case.
           Cheers,

           Eric
     ___________________________________________________________
     Please keep all replies on the list by using "reply all"
     in your mail client.  To manage your subscriptions to this
     and other Galaxy lists, please use the interface at:
         http://lists.bx.psu.edu/

     To search Galaxy mailing lists use the unified search at:
         http://galaxyproject.org/search/mailinglists/
  Cheers,
  Eric

--
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M Univesity
College Station, TX 77843
Ph: 4046922048


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

John Chilton-4
In reply to this post by Peter Cock
On Tue, Jul 22, 2014 at 7:51 AM, Peter Cock <[hidden email]> wrote:

> On Tue, Jul 22, 2014 at 1:15 PM, Eric Rasche <[hidden email]> wrote:
>> Hi Peter,
>>
>> On July 22, 2014 3:15:41 AM CDT, Peter Cock <[hidden email]> wrote:
>>>
>>>Given how close you can get now for minimal effort,
>>>this seem unnecessary.
>>>
>>>http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html
>>>
>>>My TravisCI setup this fetches the latest Galaxy as
>>>a tar ball (from a GitHub mirror as it was faster than
>>>a git clone which was faster than getting the tar ball
>>>from BitBucket, which in turn was faster than using
>>>hg clone),
>>
>> Yes, that post was at least part of the thinking behind this.
>
> :)
>
>>> .., and a per-migrated SQLite database
>>>(using a bit of Galaxy functionality originally with
>>>$GALAXY_TEST_DB_TEMPLATE added to speed
>>>up running the functional tests).
>
> Apologies for grammatical error - I pasted in the environment
> variable at the wrong point in the sentence.
>
>> I know I've seen that used but was never able to get that
>> working in practice (then again I didn't try that hard). If
>> that's a working/usable feature, then that is already the
>> majority of setup time.
>
> Yes, the creation of the test-database and all the migrations
> was an obvious low-hanging fruit when we were looking at
> making running the tool functional tests faster - although
> originally in the context of running the tests on a local
> development Galaxy instance.
>
> As to using this in practise, currently my TravisCI setup has:
>
> export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0117.sqlite
>
> I also added that line at the start of my local copy of script
> run_functional_tests.sh to benefit from this while doing
> development. That should be all there is to it (but from
> memory, this is only for use with the SQLite backend).
>
> John - could you add a current schema snapshot to
> https://github.com/jmchilton/galaxy-downloads/ please?

Hey All,

Love this thread and effort! Keep up the good work - would love to
replace say blend4j's automatic travisci testing to be backed by
dockerized -stable and -central instance.

At any rate, I have uploaded a more updated sqlite template:
https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0120.sqlite.
The old template still exists at the same URL so hopefully this
doesn't break anything.

-John

>
>>>Note this does not cache the eggs and all the other
>>>side effects of the first run like creating config files,
>>>so there is room for some speed up.
>>
>> Eggs would be nice but not the biggest thing in the world.
>
> Right. I do like your idea of automatically generated
> cutting-edge or each stable release Docker images
> though (even if I have no personal need for them at
> the moment).
>
> Regards,
>
> Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Eric Rasche
John,

How are those generated? Would you be amenable to scripting that portion and running it once a month? (...say in a cron job, with a passwordless ssh key so you never have to touch it again)

Cheers,
Eric

22.07.2014, 19:08, "John Chilton" <[hidden email]>:

> On Tue, Jul 22, 2014 at 7:51 AM, Peter Cock <[hidden email]> wrote:
>>  On Tue, Jul 22, 2014 at 1:15 PM, Eric Rasche <[hidden email]> wrote:
>>>  Hi Peter,
>>>
>>>  On July 22, 2014 3:15:41 AM CDT, Peter Cock <[hidden email]> wrote:
>>>> Given how close you can get now for minimal effort,
>>>> this seem unnecessary.
>>>>
>>>> http://blastedbio.blogspot.co.uk/2013/09/using-travis-ci-for-testing-galaxy-tools.html
>>>>
>>>> My TravisCI setup this fetches the latest Galaxy as
>>>> a tar ball (from a GitHub mirror as it was faster than
>>>> a git clone which was faster than getting the tar ball
>>>> from BitBucket, which in turn was faster than using
>>>> hg clone),
>>>  Yes, that post was at least part of the thinking behind this.
>>  :)
>>>>  .., and a per-migrated SQLite database
>>>> (using a bit of Galaxy functionality originally with
>>>> $GALAXY_TEST_DB_TEMPLATE added to speed
>>>> up running the functional tests).
>>  Apologies for grammatical error - I pasted in the environment
>>  variable at the wrong point in the sentence.
>>>  I know I've seen that used but was never able to get that
>>>  working in practice (then again I didn't try that hard). If
>>>  that's a working/usable feature, then that is already the
>>>  majority of setup time.
>>  Yes, the creation of the test-database and all the migrations
>>  was an obvious low-hanging fruit when we were looking at
>>  making running the tool functional tests faster - although
>>  originally in the context of running the tests on a local
>>  development Galaxy instance.
>>
>>  As to using this in practise, currently my TravisCI setup has:
>>
>>  export GALAXY_TEST_DB_TEMPLATE=https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0117.sqlite
>>
>>  I also added that line at the start of my local copy of script
>>  run_functional_tests.sh to benefit from this while doing
>>  development. That should be all there is to it (but from
>>  memory, this is only for use with the SQLite backend).
>>
>>  John - could you add a current schema snapshot to
>>  https://github.com/jmchilton/galaxy-downloads/ please?
>
> Hey All,
>
> Love this thread and effort! Keep up the good work - would love to
> replace say blend4j's automatic travisci testing to be backed by
> dockerized -stable and -central instance.
>
> At any rate, I have uploaded a more updated sqlite template:
> https://github.com/jmchilton/galaxy-downloads/raw/master/db_gx_rev_0120.sqlite.
> The old template still exists at the same URL so hopefully this
> doesn't break anything.
>
> -John
>>>> Note this does not cache the eggs and all the other
>>>> side effects of the first run like creating config files,
>>>> so there is room for some speed up.
>>>  Eggs would be nice but not the biggest thing in the world.
>>  Right. I do like your idea of automatically generated
>>  cutting-edge or each stable release Docker images
>>  though (even if I have no personal need for them at
>>  the moment).
>>
>>  Regards,
>>
>>  Peter

--
Eric Rasche
Programmer II
Center for Phage Technology
Texas A&M Univesity
College Station, TX 77843
Ph: 4046922048

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

Peter Cock
On Tue, Jul 22, 2014 at 7:41 PM, Eric Rasche <[hidden email]> wrote:
> John,
>
> How are those generated? Would you be amenable to scripting that
> portion and running it once a month? (...say in a cron job, with a
> passwordless ssh key so you never have to touch it again)
>
> Cheers,
> Eric

How to generate it was going to be my next question too ;)

I'm impressed with Eric's zeal to automate things. Having a script
for making the SQLite template would be good - under git in the
same repository?

Peter

P.S. The schema version 120 template works great, thanks!:

https://travis-ci.org/peterjc/pico_galaxy/builds/30592828
https://travis-ci.org/peterjc/galaxy_blast/builds/30592097
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Once-run galaxy archives

John Chilton-4
Problem with automation is I could create dozens of templates over the
next several years and consume less time in aggregate than it would
take me to automate this. Nonetheless, there is a documentation
component here that is important so I did enough to document - if
someone wants to automate from there feel free.

The template is just a copy of the sqlite database after a fresh
Galaxy is launched. I usually just do this against whatever
development instance of Galaxy I am working on. For completeness
though I have put together a script to automate this task against a
fresh install (I think):

https://github.com/jmchilton/galaxy-downloads/blob/master/build_sqlite_template.sh

Good luck!

-John



On Tue, Jul 22, 2014 at 4:13 PM, Peter Cock <[hidden email]> wrote:

> On Tue, Jul 22, 2014 at 7:41 PM, Eric Rasche <[hidden email]> wrote:
>> John,
>>
>> How are those generated? Would you be amenable to scripting that
>> portion and running it once a month? (...say in a cron job, with a
>> passwordless ssh key so you never have to touch it again)
>>
>> Cheers,
>> Eric
>
> How to generate it was going to be my next question too ;)
>
> I'm impressed with Eric's zeal to automate things. Having a script
> for making the SQLite template would be good - under git in the
> same repository?
>
> Peter
>
> P.S. The schema version 120 template works great, thanks!:
>
> https://travis-ci.org/peterjc/pico_galaxy/builds/30592828
> https://travis-ci.org/peterjc/galaxy_blast/builds/30592097
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
12