Galaxy Bare/Core and custom upload tool

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Galaxy Bare/Core and custom upload tool

Keith Suderman
Dear Galaxy Team,

Two questions.

1) Is there a bare-bones version of Galaxy available somewhere?  That is, Galaxy with no tools pre-installed?  We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed.  Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.

Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?

2) Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool?  Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded.  I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.  I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.

However, could I follow the approach suggested in that thread?  I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool?  How far back in time would I have to go to find an upload.py that was a standalone tool?

Cheers,
Keith

------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Stephan Oepen
dear keith,

in our work on building the Linguistic Analysis Portal (LAP) at the
University of Oslo, we have had to confront the same two issues you
mention.  our general approach has been to separate LAP-specific code
(and specifications) from the Galaxy tree as much as possible.

for example, we keep our tool descriptions, tool implementations, job
runner configurations, job resource specifications, and such in a
separately versioned tree and use the various options in ‘galaxy.ini’
to point to our own variants, e.g.

  tool_path = /home/laportal/tools
  tool_config_file = /home/laportal/tools/config.xml
  job_config_file = /home/laportal/tools/jobs.xml
  job_resource_params_file = /home/laportal/tools/resources.xml

for the upload tool, we modify the JavaScript files (‘upload-row.js’
and ‘upload-view.js’) to hide UI elements (notably the Genome
selection) that do not apply to us, and we have added custom code to
the actual tool implementation to post-process incoming data.  as for
the latter changes, our modified ‘upload.py’ and ‘upload.xml’ reside
outside the Galaxy tree (with the other tools), so our only risk is
that we need to validate compatibility (or import updates) when moving
to newer Galaxy versions.  for the JavaScript changes, we keep a
separate copy of the handful of UI files (from below ‘client/galaxy/’)
that we modify outside of Galaxy and use a modified ‘GruntFile.js’
(with different ‘dest’ targets) to make grunt(1) propagate our changes
into the Galaxy tree.

maybe some of these techniques could work for you too?  oe


On Thu, Mar 31, 2016 at 10:17 PM, Suderman Keith <[hidden email]> wrote:

> Dear Galaxy Team,
>
> Two questions.
>
> 1) Is there a bare-bones version of Galaxy available somewhere?  That is, Galaxy with no tools pre-installed?  We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed.  Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
>
> Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
>
> 2) Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool?  Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded.  I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.  I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
>
> However, could I follow the approach suggested in that thread?  I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool?  How far back in time would I have to go to find an upload.py that was a standalone tool?
>
> Cheers,
> Keith
>
> ------------------------------
> Research Associate
> Department of Computer Science
> Vassar College
> Poughkeepsie, NY
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Björn Grüning-3
In reply to this post by Keith Suderman
Hi Keith,

with this PR merged
https://github.com/bgruening/docker-galaxy-stable/pull/164

You can use the Galaxy Docker Image and pass in the -e BARE=True option.
This will give you an production ready Galaxy Instance without tools,
but with FTP Server, scheduler and activated TS.

You could even create your own flavour out of it.
Have a look at the readme for more informations:

https://github.com/bgruening/docker-galaxy-stable/blob/master/README.md#extending-the-docker-image

Hope this helps, I will merge the PR over the weekend hopefully.

Cheers,
Bjoern

Am 31.03.2016 um 22:17 schrieb Suderman Keith:

> Dear Galaxy Team,
>
> Two questions.
>
> 1) Is there a bare-bones version of Galaxy available somewhere?  That is, Galaxy with no tools pre-installed?  We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed.  Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
>
> Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
>
> 2) Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool?  Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded.  I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.  I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
>
> However, could I follow the approach suggested in that thread?  I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool?  How far back in time would I have to go to find an upload.py that was a standalone tool?
>
> Cheers,
> Keith
>
> ------------------------------
> Research Associate
> Department of Computer Science
> Vassar College
> Poughkeepsie, NY
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Keith Suderman
In reply to this post by Stephan Oepen
Hi Stephan,

Thank you for the pointers, they will definitely help.  Would you be willing to share any of the code you have for the upload tool?

Will you be at the LREC conference in Portorož?  It might be fun to arrange a meet up of all Galaxy users if we can.

Cheers,
Keith

> On Mar 31, 2016, at 5:12 PM, Stephan Oepen <[hidden email]> wrote:
>
> dear keith,
>
> in our work on building the Linguistic Analysis Portal (LAP) at the
> University of Oslo, we have had to confront the same two issues you
> mention.  our general approach has been to separate LAP-specific code
> (and specifications) from the Galaxy tree as much as possible.
>
> for example, we keep our tool descriptions, tool implementations, job
> runner configurations, job resource specifications, and such in a
> separately versioned tree and use the various options in ‘galaxy.ini’
> to point to our own variants, e.g.
>
>  tool_path = /home/laportal/tools
>  tool_config_file = /home/laportal/tools/config.xml
>  job_config_file = /home/laportal/tools/jobs.xml
>  job_resource_params_file = /home/laportal/tools/resources.xml
>
> for the upload tool, we modify the JavaScript files (‘upload-row.js’
> and ‘upload-view.js’) to hide UI elements (notably the Genome
> selection) that do not apply to us, and we have added custom code to
> the actual tool implementation to post-process incoming data.  as for
> the latter changes, our modified ‘upload.py’ and ‘upload.xml’ reside
> outside the Galaxy tree (with the other tools), so our only risk is
> that we need to validate compatibility (or import updates) when moving
> to newer Galaxy versions.  for the JavaScript changes, we keep a
> separate copy of the handful of UI files (from below ‘client/galaxy/’)
> that we modify outside of Galaxy and use a modified ‘GruntFile.js’
> (with different ‘dest’ targets) to make grunt(1) propagate our changes
> into the Galaxy tree.
>
> maybe some of these techniques could work for you too?  oe
>
>
> On Thu, Mar 31, 2016 at 10:17 PM, Suderman Keith <[hidden email]> wrote:
>> Dear Galaxy Team,
>>
>> Two questions.
>>
>> 1) Is there a bare-bones version of Galaxy available somewhere?  That is, Galaxy with no tools pre-installed?  We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed.  Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
>>
>> Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
>>
>> 2) Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool?  Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded.  I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.  I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
>>
>> However, could I follow the approach suggested in that thread?  I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool?  How far back in time would I have to go to find an upload.py that was a standalone tool?
>>
>> Cheers,
>> Keith
>>
>> ------------------------------
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Keith Suderman
In reply to this post by Björn Grüning-3
Hi Björn,

Have you pushed the new version to the Docker Hub?  When I run the galaxy-stable image with the -e BARE=true option I still get a Galaxy instance with the default tools.

Also, does this really give me a “bare” Galaxy instance, or just a Galaxy instance with an empty tool_conf.xml file?

Cheers,
Keith

> On Apr 1, 2016, at 11:21 AM, Björn Grüning <[hidden email]> wrote:
>
> Hi Keith,
>
> with this PR merged
> https://github.com/bgruening/docker-galaxy-stable/pull/164
>
> You can use the Galaxy Docker Image and pass in the -e BARE=True option.
> This will give you an production ready Galaxy Instance without tools,
> but with FTP Server, scheduler and activated TS.
>
> You could even create your own flavour out of it.
> Have a look at the readme for more informations:
>
> https://github.com/bgruening/docker-galaxy-stable/blob/master/README.md#extending-the-docker-image
>
> Hope this helps, I will merge the PR over the weekend hopefully.
>
> Cheers,
> Bjoern
>
> Am 31.03.2016 um 22:17 schrieb Suderman Keith:
>> Dear Galaxy Team,
>>
>> Two questions.
>>
>> 1) Is there a bare-bones version of Galaxy available somewhere?  That is, Galaxy with no tools pre-installed?  We are creating Ansible play books for configuring Docker/Galaxy instances with our NLP tools installed.  Currently we are using a fork of the Galaxy project with the bio tools removed, but this is less than ideal when we try to update to new versions of Galaxy.
>>
>> Alternatively, does anyone have an automated process to delete unwanted tools that we could run immediately after cloning the main Galaxy repo?
>>
>> 2) Is is possible to either a) modify the existing upload tool, or b) implement a new upload tool?  Several options in the upload tool do not make sense in our domain (e.g. selecting a genome) and we would also like to do some post-processing to files after they have been uploaded.  I've looked at the code for the existing upload tool and it seems quite tightly coupled to the Galaxy code base.  I was hoping the "Downgrade upload tool" thread from a few weeks ago was going to help, but unfortunately for me (and good for the user) they found a solution that didn't involve installing a new upload tool.
>>
>> However, could I follow the approach suggested in that thread?  I.E. go back in time and find an old upload.[py | xml] and use that as a starting point for my own custom upload tool?  How far back in time would I have to go to find an upload.py that was a standalone tool?
>>
>> Cheers,
>> Keith
>>
>> ------------------------------
>> Research Associate
>> Department of Computer Science
>> Vassar College
>> Poughkeepsie, NY
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>>
------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Björn Grüning-3
Hi Keith,


> Hi Björn,
>
> Have you pushed the new version to the Docker Hub?  When I run the
> galaxy-stable image with the -e BARE=true option I still get a Galaxy
> instance with the default tools.

Yes it is, but you need the :dev version, aka:

docker run -i -t -p 8080:80 bgruening/galaxy-stable:dev

(hope it works, I'm currently not on my PC)

> Also, does this really give me a “bare” Galaxy instance, or just a
> Galaxy instance with an empty tool_conf.xml file?

It only deactivates the tools in tool_conf.xml files. Is this is
problem? These tools do not have any dependencies other than python and
disc space should not be a problem as these are only a bunch of XML files.

The my comments inline.


>>
>> Am 31.03.2016 um 22:17 schrieb Suderman Keith:
>>> Dear Galaxy Team,
>>>
>>> Two questions.
>>>
>>> 1) Is there a bare-bones version of Galaxy available somewhere?
>>> That is, Galaxy with no tools pre-installed?  We are creating
>>> Ansible play books for configuring Docker/Galaxy instances with
>>> our NLP tools installed.  Currently we are using a fork of the
>>> Galaxy project with the bio tools removed, but this is less than
>>> ideal when we try to update to new versions of Galaxy.

Can you explain what you delete? I do really support this idea as we are
also developing cheminformatics or imaging flavours.

>>> Alternatively, does anyone have an automated process to delete
>>> unwanted tools that we could run immediately after cloning the
>>> main Galaxy repo?

I don't think so you are pioneering here! But I think the gain of this
step is currently not much, isn't it? What are your concerns?
Disc-space? As a fist shoot I would try to hide genomic-specifc stuff
from users. In a long run we can talk about this with the Galaxy team,
isn't it. I know there is already a consensus of getting tools into the
TS and have as less as possible tools in Galaxy by default.
If you look 3 years back, Galaxy was full of tools, nowadays it's more
or less text manipulation tools without dependencies. So plain XML files
with a few scripts.

>>> 2) Is is possible to either a) modify the existing upload tool,
>>> or b) implement a new upload tool?  Several options in the upload
>>> tool do not make sense in our domain (e.g. selecting a genome)
>>> and we would also like to do some post-processing to files after
>>> they have been uploaded.  I've looked at the code for the
>>> existing upload tool and it seems quite tightly coupled to the
>>> Galaxy code base.  

Yes it is, actually you need to have the upload tool, otherwise many
things do not work.
I would like to go with a)!

That is put big #if #else conditions in the JS and Galaxy code to
display genomic-specific content ... we could have a galaxy.ini config
to trigger this #if.

I do think we have a good chance to get this accepted.

Second step, and this will take for careful thoughts, is a plugin
mechanism for post-upload modifications.
The currently upload tool has some kind of these functionality, for
example the "Space-To-Tab" converter. Ideally exposing tools (even from
the TS)?


>>> I was hoping the "Downgrade upload tool"
>>> thread from a few weeks ago was going to help, but unfortunately
>>> for me (and good for the user) they found a solution that didn't
>>> involve installing a new upload tool.

Let's try to stay as upstream as possible. upload.xml is so tightly
integrated in Galaxy that it is better to get code upstream I assume.
And I know Galaxy devs like the idea of being less genomic centric and
more a general framework. It's just that no one had time for this until
know.

>>> However, could I follow the approach suggested in that thread?
>>> I.E. go back in time and find an old upload.[py | xml] and use
>>> that as a starting point for my own custom upload tool?  How far
>>> back in time would I have to go to find an upload.py that was a
>>> standalone tool?

I think this will cause more trouble ... :(
Let me know what you decide I will try to help you as much as I can.

Cheers,
Bjoern

>>> Cheers, Keith
>>>
>>> ------------------------------ Research Associate Department of
>>> Computer Science Vassar College Poughkeepsie, NY
>>>
>>>
>>>
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all" in your
>>> mail client.  To manage your subscriptions to this and other
>>> Galaxy lists, please use the interface at:
>>> https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>> http://galaxyproject.org/search/mailinglists/
>>>
>
> ------------------------------ Research Associate Department of
> Computer Science Vassar College Poughkeepsie, NY
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Suderman Keith-2
Hi Björn,

> Yes it is, but you need the :dev version,

DOH, I should have known…

> Can you explain what you delete? I do really support this idea as we are
> also developing cheminformatics or imaging flavours.

Currently I think I have just removed the datatypes.  I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.

>
>>>> Alternatively, does anyone have an automated process to delete
>>>> unwanted tools that we could run immediately after cloning the
>>>> main Galaxy repo?
>
> I don't think so you are pioneering here! But I think the gain of this
> step is currently not much, isn't it? What are your concerns?

Not really concerns, I just would like to remove as much unneeded code as possible.  I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use.  Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers.  I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.

> Yes it is, actually you need to have the upload tool, otherwise many
> things do not work.
> I would like to go with a)!

I think you are correct; modifying the current upload tool is the way to go.  A plugin mechanism for post-processing uploaded files sounds like a great idea!

Cheers,
Keith

>
> That is put big #if #else conditions in the JS and Galaxy code to
> display genomic-specific content ... we could have a galaxy.ini config
> to trigger this #if.
>
> I do think we have a good chance to get this accepted.
>
> Second step, and this will take for careful thoughts, is a plugin
> mechanism for post-upload modifications.
> The currently upload tool has some kind of these functionality, for
> example the "Space-To-Tab" converter. Ideally exposing tools (even from
> the TS)?
>
>
>>>> I was hoping the "Downgrade upload tool"
>>>> thread from a few weeks ago was going to help, but unfortunately
>>>> for me (and good for the user) they found a solution that didn't
>>>> involve installing a new upload tool.
>
> Let's try to stay as upstream as possible. upload.xml is so tightly
> integrated in Galaxy that it is better to get code upstream I assume.
> And I know Galaxy devs like the idea of being less genomic centric and
> more a general framework. It's just that no one had time for this until
> know.
>
>>>> However, could I follow the approach suggested in that thread?
>>>> I.E. go back in time and find an old upload.[py | xml] and use
>>>> that as a starting point for my own custom upload tool?  How far
>>>> back in time would I have to go to find an upload.py that was a
>>>> standalone tool?
>
> I think this will cause more trouble ... :(
> Let me know what you decide I will try to help you as much as I can.
>
> Cheers,
> Bjoern
>
>>>> Cheers, Keith
>>>>
>>>> ------------------------------ Research Associate Department of
>>>> Computer Science Vassar College Poughkeepsie, NY
>>>>
>>>>
>>>>
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all" in your
>>>> mail client.  To manage your subscriptions to this and other
>>>> Galaxy lists, please use the interface at:
>>>> https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>> http://galaxyproject.org/search/mailinglists/
>>>>
>>
>> ------------------------------ Research Associate Department of
>> Computer Science Vassar College Poughkeepsie, NY
>>
------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Suderman Keith-2
In reply to this post by Björn Grüning-3
[Resending because GMail keeps mangling my email address…]

Hi Björn,

Yes it is, but you need the :dev version,

DOH, I should have known…

Can you explain what you delete? I do really support this idea as we are
also developing cheminformatics or imaging flavours.

Currently I think I have just removed the datatypes.  I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.


Alternatively, does anyone have an automated process to delete
unwanted tools that we could run immediately after cloning the
main Galaxy repo?

I don't think so you are pioneering here! But I think the gain of this
step is currently not much, isn't it? What are your concerns?

Not really concerns, I just would like to remove as much unneeded code as possible.  I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use.  Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers.  I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.

Yes it is, actually you need to have the upload tool, otherwise many
things do not work.
I would like to go with a)!

I think you are correct; modifying the current upload tool is the way to go.  A plugin mechanism for post-processing uploaded files sounds like a great idea!

Cheers,
Keith


That is put big #if #else conditions in the JS and Galaxy code to
display genomic-specific content ... we could have a galaxy.ini config
to trigger this #if.

I do think we have a good chance to get this accepted.

Second step, and this will take for careful thoughts, is a plugin
mechanism for post-upload modifications.
The currently upload tool has some kind of these functionality, for
example the "Space-To-Tab" converter. Ideally exposing tools (even from
the TS)?


I was hoping the "Downgrade upload tool"
thread from a few weeks ago was going to help, but unfortunately
for me (and good for the user) they found a solution that didn't
involve installing a new upload tool.

Let's try to stay as upstream as possible. upload.xml is so tightly
integrated in Galaxy that it is better to get code upstream I assume.
And I know Galaxy devs like the idea of being less genomic centric and
more a general framework. It's just that no one had time for this until
know.

However, could I follow the approach suggested in that thread?
I.E. go back in time and find an old upload.[py | xml] and use
that as a starting point for my own custom upload tool?  How far
back in time would I have to go to find an upload.py that was a
standalone tool?

I think this will cause more trouble ... :(
Let me know what you decide I will try to help you as much as I can.

Cheers,
Bjoern

Cheers, Keith

------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY



___________________________________________________________
Please keep all replies on the list by using "reply all" in your
mail client.  To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY

On Apr 3, 2016, at 5:05 AM, Björn Grüning <[hidden email]> wrote:

Hi Keith,


Hi Björn,

Have you pushed the new version to the Docker Hub?  When I run the
galaxy-stable image with the -e BARE=true option I still get a Galaxy
instance with the default tools.

Yes it is, but you need the :dev version, aka:

docker run -i -t -p 8080:80 bgruening/galaxy-stable:dev

(hope it works, I'm currently not on my PC)

Also, does this really give me a “bare” Galaxy instance, or just a
Galaxy instance with an empty tool_conf.xml file?

It only deactivates the tools in tool_conf.xml files. Is this is
problem? These tools do not have any dependencies other than python and
disc space should not be a problem as these are only a bunch of XML files.

The my comments inline.



Am 31.03.2016 um 22:17 schrieb Suderman Keith:
Dear Galaxy Team,

Two questions.

1) Is there a bare-bones version of Galaxy available somewhere?
That is, Galaxy with no tools pre-installed?  We are creating
Ansible play books for configuring Docker/Galaxy instances with
our NLP tools installed.  Currently we are using a fork of the
Galaxy project with the bio tools removed, but this is less than
ideal when we try to update to new versions of Galaxy.

Can you explain what you delete? I do really support this idea as we are
also developing cheminformatics or imaging flavours.

Alternatively, does anyone have an automated process to delete
unwanted tools that we could run immediately after cloning the
main Galaxy repo?

I don't think so you are pioneering here! But I think the gain of this
step is currently not much, isn't it? What are your concerns?
Disc-space? As a fist shoot I would try to hide genomic-specifc stuff
from users. In a long run we can talk about this with the Galaxy team,
isn't it. I know there is already a consensus of getting tools into the
TS and have as less as possible tools in Galaxy by default.
If you look 3 years back, Galaxy was full of tools, nowadays it's more
or less text manipulation tools without dependencies. So plain XML files
with a few scripts.

2) Is is possible to either a) modify the existing upload tool,
or b) implement a new upload tool?  Several options in the upload
tool do not make sense in our domain (e.g. selecting a genome)
and we would also like to do some post-processing to files after
they have been uploaded.  I've looked at the code for the
existing upload tool and it seems quite tightly coupled to the
Galaxy code base.  

Yes it is, actually you need to have the upload tool, otherwise many
things do not work.
I would like to go with a)!

That is put big #if #else conditions in the JS and Galaxy code to
display genomic-specific content ... we could have a galaxy.ini config
to trigger this #if.

I do think we have a good chance to get this accepted.

Second step, and this will take for careful thoughts, is a plugin
mechanism for post-upload modifications.
The currently upload tool has some kind of these functionality, for
example the "Space-To-Tab" converter. Ideally exposing tools (even from
the TS)?


I was hoping the "Downgrade upload tool"
thread from a few weeks ago was going to help, but unfortunately
for me (and good for the user) they found a solution that didn't
involve installing a new upload tool.

Let's try to stay as upstream as possible. upload.xml is so tightly
integrated in Galaxy that it is better to get code upstream I assume.
And I know Galaxy devs like the idea of being less genomic centric and
more a general framework. It's just that no one had time for this until
know.

However, could I follow the approach suggested in that thread?
I.E. go back in time and find an old upload.[py | xml] and use
that as a starting point for my own custom upload tool?  How far
back in time would I have to go to find an upload.py that was a
standalone tool?

I think this will cause more trouble ... :(
Let me know what you decide I will try to help you as much as I can.

Cheers,
Bjoern

Cheers, Keith

------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY



___________________________________________________________
Please keep all replies on the list by using "reply all" in your
mail client.  To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY


------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Suderman Keith-2
In reply to this post by Suderman Keith-2
[Resending (again) as GMail keeps mangling my email address… sorry for triple posting...]

Hi Björn,

Yes it is, but you need the :dev version,

DOH, I should have known…

Can you explain what you delete? I do really support this idea as we are
also developing cheminformatics or imaging flavours.

Currently I think I have just removed the datatypes.  I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.


Alternatively, does anyone have an automated process to delete
unwanted tools that we could run immediately after cloning the
main Galaxy repo?

I don't think so you are pioneering here! But I think the gain of this
step is currently not much, isn't it? What are your concerns?

Not really concerns, I just would like to remove as much unneeded code as possible.  I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use.  Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers.  I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.

Yes it is, actually you need to have the upload tool, otherwise many
things do not work.
I would like to go with a)!

I think you are correct; modifying the current upload tool is the way to go.  A plugin mechanism for post-processing uploaded files sounds like a great idea!

Cheers,
Keith


That is put big #if #else conditions in the JS and Galaxy code to
display genomic-specific content ... we could have a galaxy.ini config
to trigger this #if.

I do think we have a good chance to get this accepted.

Second step, and this will take for careful thoughts, is a plugin
mechanism for post-upload modifications.
The currently upload tool has some kind of these functionality, for
example the "Space-To-Tab" converter. Ideally exposing tools (even from
the TS)?


I was hoping the "Downgrade upload tool"
thread from a few weeks ago was going to help, but unfortunately
for me (and good for the user) they found a solution that didn't
involve installing a new upload tool.

Let's try to stay as upstream as possible. upload.xml is so tightly
integrated in Galaxy that it is better to get code upstream I assume.
And I know Galaxy devs like the idea of being less genomic centric and
more a general framework. It's just that no one had time for this until
know.

However, could I follow the approach suggested in that thread?
I.E. go back in time and find an old upload.[py | xml] and use
that as a starting point for my own custom upload tool?  How far
back in time would I have to go to find an upload.py that was a
standalone tool?

I think this will cause more trouble ... :(
Let me know what you decide I will try to help you as much as I can.

Cheers,
Bjoern

Cheers, Keith

------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY



___________________________________________________________
Please keep all replies on the list by using "reply all" in your
mail client.  To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY


------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY
On Apr 4, 2016, at 7:01 PM, Suderman Keith <[hidden email]> wrote:

Hi Björn,

Yes it is, but you need the :dev version,

DOH, I should have known…

Can you explain what you delete? I do really support this idea as we are
also developing cheminformatics or imaging flavours.

Currently I think I have just removed the datatypes.  I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.


Alternatively, does anyone have an automated process to delete
unwanted tools that we could run immediately after cloning the
main Galaxy repo?

I don't think so you are pioneering here! But I think the gain of this
step is currently not much, isn't it? What are your concerns?

Not really concerns, I just would like to remove as much unneeded code as possible.  I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use.  Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers.  I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.

Yes it is, actually you need to have the upload tool, otherwise many
things do not work.
I would like to go with a)!

I think you are correct; modifying the current upload tool is the way to go.  A plugin mechanism for post-processing uploaded files sounds like a great idea!

Cheers,
Keith


That is put big #if #else conditions in the JS and Galaxy code to
display genomic-specific content ... we could have a galaxy.ini config
to trigger this #if.

I do think we have a good chance to get this accepted.

Second step, and this will take for careful thoughts, is a plugin
mechanism for post-upload modifications.
The currently upload tool has some kind of these functionality, for
example the "Space-To-Tab" converter. Ideally exposing tools (even from
the TS)?


I was hoping the "Downgrade upload tool"
thread from a few weeks ago was going to help, but unfortunately
for me (and good for the user) they found a solution that didn't
involve installing a new upload tool.

Let's try to stay as upstream as possible. upload.xml is so tightly
integrated in Galaxy that it is better to get code upstream I assume.
And I know Galaxy devs like the idea of being less genomic centric and
more a general framework. It's just that no one had time for this until
know.

However, could I follow the approach suggested in that thread?
I.E. go back in time and find an old upload.[py | xml] and use
that as a starting point for my own custom upload tool?  How far
back in time would I have to go to find an upload.py that was a
standalone tool?

I think this will cause more trouble ... :(
Let me know what you decide I will try to help you as much as I can.

Cheers,
Bjoern

Cheers, Keith

------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY



___________________________________________________________
Please keep all replies on the list by using "reply all" in your
mail client.  To manage your subscriptions to this and other
Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


------------------------------ Research Associate Department of
Computer Science Vassar College Poughkeepsie, NY


------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY

------------------------------
Research Associate
Department of Computer Science
Vassar College
Poughkeepsie, NY


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

signature.asc (507 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy Bare/Core and custom upload tool

Björn Grüning-3
In reply to this post by Suderman Keith-2
Hi Keith,

> Hi Björn,
>
>> Yes it is, but you need the :dev version,
> DOH, I should have known…

:)

>> Can you explain what you delete? I do really support this idea as we are
>> also developing cheminformatics or imaging flavours.
> Currently I think I have just removed the datatypes.  I haven’t removed anything else as I didn’t know how many, if any, were tightly coupled as the upload tool is.

We could think about removing datatypes as well from the BARE Galaxy
Docker container, similar to what we have done now for tools, if you like.

>>>>> Alternatively, does anyone have an automated process to delete
>>>>> unwanted tools that we could run immediately after cloning the
>>>>> main Galaxy repo?
>> I don't think so you are pioneering here! But I think the gain of this
>> step is currently not much, isn't it? What are your concerns?
> Not really concerns, I just would like to remove as much unneeded code as possible.  I have removed all (most) of the default datatypes shipped with Galaxy as many of the file extensions conflict with extension I want to use.

Mh, but I see a bigger problem here. If you have conflicting extensions,
could you not prefix them? I have the small dream of having a university
workbench with all tools.
So offering your tools as well as genomics and in this case I will have
the problems again.
Removing genomic datatypes makes sense in your case and I'm happy to
work on this. But it would be nice if we could also try to make these
tools, datatypes work on every Galaxy instance.

Can we prepend them with 'nltk_'?

>   Since the tools are most likely broken without the correct datatypes available my feeling is that these tools simply provide added points of failure and/or additional attack vectors for hackers.  I have been dealing with some brute-force attacks on our Wordpress installation the past few weeks that has taught me that any unused code, regardless of how innocuous it seems, simply provides avenues of attack.

True, but if you deactivate datatypes the code should not be loaded and
with every modification we do that is not tested we increase the attack
vector again.

Let me know what you decide and if you like to use the Docker container
and I will find time to get rid of datatypes in BARE mode.

Thanks,
Bjoern

>> Yes it is, actually you need to have the upload tool, otherwise many
>> things do not work.
>> I would like to go with a)!
> I think you are correct; modifying the current upload tool is the way to go.  A plugin mechanism for post-processing uploaded files sounds like a great idea!
>
> Cheers,
> Keith
>
>> That is put big #if #else conditions in the JS and Galaxy code to
>> display genomic-specific content ... we could have a galaxy.ini config
>> to trigger this #if.
>>
>> I do think we have a good chance to get this accepted.
>>
>> Second step, and this will take for careful thoughts, is a plugin
>> mechanism for post-upload modifications.
>> The currently upload tool has some kind of these functionality, for
>> example the "Space-To-Tab" converter. Ideally exposing tools (even from
>> the TS)?
>>
>>
>>>>> I was hoping the "Downgrade upload tool"
>>>>> thread from a few weeks ago was going to help, but unfortunately
>>>>> for me (and good for the user) they found a solution that didn't
>>>>> involve installing a new upload tool.
>> Let's try to stay as upstream as possible. upload.xml is so tightly
>> integrated in Galaxy that it is better to get code upstream I assume.
>> And I know Galaxy devs like the idea of being less genomic centric and
>> more a general framework. It's just that no one had time for this until
>> know.
>>
>>>>> However, could I follow the approach suggested in that thread?
>>>>> I.E. go back in time and find an old upload.[py | xml] and use
>>>>> that as a starting point for my own custom upload tool?  How far
>>>>> back in time would I have to go to find an upload.py that was a
>>>>> standalone tool?
>> I think this will cause more trouble ... :(
>> Let me know what you decide I will try to help you as much as I can.
>>
>> Cheers,
>> Bjoern
>>
>>>>> Cheers, Keith
>>>>>
>>>>> ------------------------------ Research Associate Department of
>>>>> Computer Science Vassar College Poughkeepsie, NY
>>>>>
>>>>>
>>>>>
>>>>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all" in your
>>>>> mail client.  To manage your subscriptions to this and other
>>>>> Galaxy lists, please use the interface at:
>>>>> https://lists.galaxyproject.org/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>> http://galaxyproject.org/search/mailinglists/
>>>>>
>>> ------------------------------ Research Associate Department of
>>> Computer Science Vassar College Poughkeepsie, NY
>>>
> ------------------------------
> Research Associate
> Department of Computer Science
> Vassar College
> Poughkeepsie, NY
>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/