Non-admin tool install/import

classic Classic list List threaded Threaded
6 messages Options
| Threaded
Open this post in threaded view
|

Non-admin tool install/import

karthikg
Hello,
  We have a centralized Galaxy installation for out cluster which can be used by many users (backend interfaces with HTCondor to run jobs). Is there any way for non-administrator users to import/install their own tools so that they may use it in their workflows (under a subsection called "unstable", for example? Ideally, without having to restart the Galaxy daemons whenever a change is made.
It would be fine if these tools are not visible to other users till the administrator says so.
Thanks,
Karthik

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Non-admin tool install/import

John Chilton-4
Galaxy doesn't really support this use case and it will be major
effort to get it work this way I suspect. Pieces to look at include:

The Galaxy Tool Factory (it has the ability to create reusable tools
from scripts):

http://www.ncbi.nlm.nih.gov/pubmed/23024011

You may be able to modify it in such a way that each tool is tagged
with who created and then use ToolBox filters to limit added tools to
a given user:

https://wiki.galaxyproject.org/UserDefinedToolboxFilters

I think the latest version of Galaxy has improved support for adding
tools without requiring restarts (using message queues). I don't know
if this will automatically work with the tool factory or not.

I suspect fighting Galaxy at every step on this will frustrate you and
the users - and you are exposing all of your users data to every user
you give this privilege to. Is this a shared cluster or is dedicated
to Galaxy? If it is shared - it might be better for advanced users to
just get importing and exporting data to user directories really well.
In my previous position at MSI we created a set of tools that allowed
Galaxy to SCP files as the user to our login nodes (using a SSH key
scheme) - likewise creating directories user's can upload files can
enabled power users.

Hopefully this helps,

-John

On Sat, Jun 7, 2014 at 1:12 PM, Karthik Gururaj
<[hidden email]> wrote:

> Hello,
>   We have a centralized Galaxy installation for out cluster which can be
> used by many users (backend interfaces with HTCondor to run jobs). Is there
> any way for non-administrator users to import/install their own tools so
> that they may use it in their workflows (under a subsection called
> "unstable", for example? Ideally, without having to restart the Galaxy
> daemons whenever a change is made.
> It would be fine if these tools are not visible to other users till the
> administrator says so.
> Thanks,
> Karthik
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Non-admin tool install/import

Ross-2
Hi, Karthik and John.

Some details on the tool factory for anyone interested. 

Executive summary: it may be helpful in this context but only for trusted administrators.

TL;DR:

Firstly, it will refuse to run for anyone other than a local Galaxy administrator. This is because it exposes unrestricted scripting so should only be installed if you can trust your administrative users not to run "cd /; rm -rf *". I'd advise installing ONLY on your own private instance and NEVER on a public Galaxy.

Secondly, it has two modes of operation - script running and tool generation. 

When executed without the option to generate a tool archive, it will run a pasted (perl, python, R, bash) script creating an output in the history. This history output is re-doable in the usual Galaxy way including allowing the script to be edited and rerun, so it's possible to (eg) get a script working interactively - galaxy as an IDE anyone ? :)  

Once a script runs on some test data, the tool factory will optionally generate a complete tool shed compatible gzip which can be uploaded to any tool shed as a new or updated repository. The generated tool includes the supplied test data as a proper Galaxy functional test. Once a tool is in a toolshed, it is just another Galaxy tool, ready to be installed to any Galaxy like any other tool - but will require restarting of multiple web processes as John mentions. 

If the script is safe, the tool is safe - there are no specific security risks for tool factory generated tools other than the script itself.

Finally, currently it takes only one input and generates one output which is a substantial restriction - but of course the generated tool source code is easy to edit if you need more complex I/O. It has a really neat option to create a simple but useful HTML display with links and thumbnails to arbitrary (eg) pdf or other output files from a script - the tool form includes examples in all 4 scripting languages ready to cut and paste, including one which generates 50 random images and presents them in a grid as an HTML page for the user.

On Mon, Jun 9, 2014 at 10:32 AM, John Chilton <[hidden email]> wrote:
Galaxy doesn't really support this use case and it will be major
effort to get it work this way I suspect. Pieces to look at include:

The Galaxy Tool Factory (it has the ability to create reusable tools
from scripts):

http://www.ncbi.nlm.nih.gov/pubmed/23024011

You may be able to modify it in such a way that each tool is tagged
with who created and then use ToolBox filters to limit added tools to
a given user:

https://wiki.galaxyproject.org/UserDefinedToolboxFilters

I think the latest version of Galaxy has improved support for adding
tools without requiring restarts (using message queues). I don't know
if this will automatically work with the tool factory or not.

I suspect fighting Galaxy at every step on this will frustrate you and
the users - and you are exposing all of your users data to every user
you give this privilege to. Is this a shared cluster or is dedicated
to Galaxy? If it is shared - it might be better for advanced users to
just get importing and exporting data to user directories really well.
In my previous position at MSI we created a set of tools that allowed
Galaxy to SCP files as the user to our login nodes (using a SSH key
scheme) - likewise creating directories user's can upload files can
enabled power users.

Hopefully this helps,

-John

On Sat, Jun 7, 2014 at 1:12 PM, Karthik Gururaj
<[hidden email]> wrote:
> Hello,
>   We have a centralized Galaxy installation for out cluster which can be
> used by many users (backend interfaces with HTCondor to run jobs). Is there
> any way for non-administrator users to import/install their own tools so
> that they may use it in their workflows (under a subsection called
> "unstable", for example? Ideally, without having to restart the Galaxy
> daemons whenever a change is made.
> It would be fine if these tools are not visible to other users till the
> administrator says so.
> Thanks,
> Karthik
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Non-admin tool install/import

karthikg
Thanks - will take a look at the Tool Shed and see if there is anything there we can use in safe manner. Else, the old fashioned way of asking users to test out on their tools on a standalone Galaxy system and then requesting the administrator to pull in the relevant XML files.
Thanks,
Karthik


On Sun, Jun 8, 2014 at 6:25 PM, Ross <[hidden email]> wrote:
Hi, Karthik and John.

Some details on the tool factory for anyone interested. 

Executive summary: it may be helpful in this context but only for trusted administrators.

TL;DR:

Firstly, it will refuse to run for anyone other than a local Galaxy administrator. This is because it exposes unrestricted scripting so should only be installed if you can trust your administrative users not to run "cd /; rm -rf *". I'd advise installing ONLY on your own private instance and NEVER on a public Galaxy.

Secondly, it has two modes of operation - script running and tool generation. 

When executed without the option to generate a tool archive, it will run a pasted (perl, python, R, bash) script creating an output in the history. This history output is re-doable in the usual Galaxy way including allowing the script to be edited and rerun, so it's possible to (eg) get a script working interactively - galaxy as an IDE anyone ? :)  

Once a script runs on some test data, the tool factory will optionally generate a complete tool shed compatible gzip which can be uploaded to any tool shed as a new or updated repository. The generated tool includes the supplied test data as a proper Galaxy functional test. Once a tool is in a toolshed, it is just another Galaxy tool, ready to be installed to any Galaxy like any other tool - but will require restarting of multiple web processes as John mentions. 

If the script is safe, the tool is safe - there are no specific security risks for tool factory generated tools other than the script itself.

Finally, currently it takes only one input and generates one output which is a substantial restriction - but of course the generated tool source code is easy to edit if you need more complex I/O. It has a really neat option to create a simple but useful HTML display with links and thumbnails to arbitrary (eg) pdf or other output files from a script - the tool form includes examples in all 4 scripting languages ready to cut and paste, including one which generates 50 random images and presents them in a grid as an HTML page for the user.

On Mon, Jun 9, 2014 at 10:32 AM, John Chilton <[hidden email]> wrote:
Galaxy doesn't really support this use case and it will be major
effort to get it work this way I suspect. Pieces to look at include:

The Galaxy Tool Factory (it has the ability to create reusable tools
from scripts):

http://www.ncbi.nlm.nih.gov/pubmed/23024011

You may be able to modify it in such a way that each tool is tagged
with who created and then use ToolBox filters to limit added tools to
a given user:

https://wiki.galaxyproject.org/UserDefinedToolboxFilters

I think the latest version of Galaxy has improved support for adding
tools without requiring restarts (using message queues). I don't know
if this will automatically work with the tool factory or not.

I suspect fighting Galaxy at every step on this will frustrate you and
the users - and you are exposing all of your users data to every user
you give this privilege to. Is this a shared cluster or is dedicated
to Galaxy? If it is shared - it might be better for advanced users to
just get importing and exporting data to user directories really well.
In my previous position at MSI we created a set of tools that allowed
Galaxy to SCP files as the user to our login nodes (using a SSH key
scheme) - likewise creating directories user's can upload files can
enabled power users.

Hopefully this helps,

-John

On Sat, Jun 7, 2014 at 1:12 PM, Karthik Gururaj
<[hidden email]> wrote:
> Hello,
>   We have a centralized Galaxy installation for out cluster which can be
> used by many users (backend interfaces with HTCondor to run jobs). Is there
> any way for non-administrator users to import/install their own tools so
> that they may use it in their workflows (under a subsection called
> "unstable", for example? Ideally, without having to restart the Galaxy
> daemons whenever a change is made.
> It would be fine if these tools are not visible to other users till the
> administrator says so.
> Thanks,
> Karthik
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Non-admin tool install/import

Ross-2


On Tue, Jun 10, 2014 at 7:37 AM, Karthik Gururaj <[hidden email]> wrote:
Thanks - will take a look at the Tool Shed and see if there is anything there we can use in safe manner. Else, the old fashioned way of asking users to test out on their tools on a standalone Galaxy system and then requesting the administrator to pull in the relevant XML files.

Sorry this is a bit confusing...
Just to be quite clear: The Tool Factory is available from the main Tool Shed, but Tool Factory =/= Tool Shed !!

The Tool Factory is just another Galaxy tool administrators can use to run scripts interactively in Galaxy. 
It installs automatically from the main Tool Shed and optionally generates new Galaxy tools from working scripts. 
New generated tools are in a tgz archive ready to be uploaded into Tool Shed repositories. 

The Tool Shed is a specialised web server that supports version control and management of Galaxy tool source code and automated installation into Galaxy instances.
You could run a local Tool Shed and yes, your admins could use it to install properly configured user provided tools. 

The Tool Factory allows your scripting-capable users of creating new Galaxy tools from scripts if they run it as administrators of their own laptop/development instances, or your administrators could use it to run scripts directly on your private instance and optionally generate new safe tools (by uploading the archives to your local tool shed then installing those new tools into your local Galaxy) for ordinary users to use in their workflows.
  

 
Thanks,
Karthik


On Sun, Jun 8, 2014 at 6:25 PM, Ross <[hidden email]> wrote:
Hi, Karthik and John.

Some details on the tool factory for anyone interested. 

Executive summary: it may be helpful in this context but only for trusted administrators.

TL;DR:

Firstly, it will refuse to run for anyone other than a local Galaxy administrator. This is because it exposes unrestricted scripting so should only be installed if you can trust your administrative users not to run "cd /; rm -rf *". I'd advise installing ONLY on your own private instance and NEVER on a public Galaxy.

Secondly, it has two modes of operation - script running and tool generation. 

When executed without the option to generate a tool archive, it will run a pasted (perl, python, R, bash) script creating an output in the history. This history output is re-doable in the usual Galaxy way including allowing the script to be edited and rerun, so it's possible to (eg) get a script working interactively - galaxy as an IDE anyone ? :)  

Once a script runs on some test data, the tool factory will optionally generate a complete tool shed compatible gzip which can be uploaded to any tool shed as a new or updated repository. The generated tool includes the supplied test data as a proper Galaxy functional test. Once a tool is in a toolshed, it is just another Galaxy tool, ready to be installed to any Galaxy like any other tool - but will require restarting of multiple web processes as John mentions. 

If the script is safe, the tool is safe - there are no specific security risks for tool factory generated tools other than the script itself.

Finally, currently it takes only one input and generates one output which is a substantial restriction - but of course the generated tool source code is easy to edit if you need more complex I/O. It has a really neat option to create a simple but useful HTML display with links and thumbnails to arbitrary (eg) pdf or other output files from a script - the tool form includes examples in all 4 scripting languages ready to cut and paste, including one which generates 50 random images and presents them in a grid as an HTML page for the user.

On Mon, Jun 9, 2014 at 10:32 AM, John Chilton <[hidden email]> wrote:
Galaxy doesn't really support this use case and it will be major
effort to get it work this way I suspect. Pieces to look at include:

The Galaxy Tool Factory (it has the ability to create reusable tools
from scripts):

http://www.ncbi.nlm.nih.gov/pubmed/23024011

You may be able to modify it in such a way that each tool is tagged
with who created and then use ToolBox filters to limit added tools to
a given user:

https://wiki.galaxyproject.org/UserDefinedToolboxFilters

I think the latest version of Galaxy has improved support for adding
tools without requiring restarts (using message queues). I don't know
if this will automatically work with the tool factory or not.

I suspect fighting Galaxy at every step on this will frustrate you and
the users - and you are exposing all of your users data to every user
you give this privilege to. Is this a shared cluster or is dedicated
to Galaxy? If it is shared - it might be better for advanced users to
just get importing and exporting data to user directories really well.
In my previous position at MSI we created a set of tools that allowed
Galaxy to SCP files as the user to our login nodes (using a SSH key
scheme) - likewise creating directories user's can upload files can
enabled power users.

Hopefully this helps,

-John

On Sat, Jun 7, 2014 at 1:12 PM, Karthik Gururaj
<[hidden email]> wrote:
> Hello,
>   We have a centralized Galaxy installation for out cluster which can be
> used by many users (backend interfaces with HTCondor to run jobs). Is there
> any way for non-administrator users to import/install their own tools so
> that they may use it in their workflows (under a subsection called
> "unstable", for example? Ideally, without having to restart the Galaxy
> daemons whenever a change is made.
> It would be fine if these tools are not visible to other users till the
> administrator says so.
> Thanks,
> Karthik
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Non-admin tool install/import

karthikg
Sorry, I meant Tool Factory when I wrote Tool Shed in my previous email.


On Tue, Jun 10, 2014 at 5:25 PM, Ross <[hidden email]> wrote:


On Tue, Jun 10, 2014 at 7:37 AM, Karthik Gururaj <[hidden email]> wrote:
Thanks - will take a look at the Tool Shed and see if there is anything there we can use in safe manner. Else, the old fashioned way of asking users to test out on their tools on a standalone Galaxy system and then requesting the administrator to pull in the relevant XML files.

Sorry this is a bit confusing...
Just to be quite clear: The Tool Factory is available from the main Tool Shed, but Tool Factory =/= Tool Shed !!

The Tool Factory is just another Galaxy tool administrators can use to run scripts interactively in Galaxy. 
It installs automatically from the main Tool Shed and optionally generates new Galaxy tools from working scripts. 
New generated tools are in a tgz archive ready to be uploaded into Tool Shed repositories. 

The Tool Shed is a specialised web server that supports version control and management of Galaxy tool source code and automated installation into Galaxy instances.
You could run a local Tool Shed and yes, your admins could use it to install properly configured user provided tools. 

The Tool Factory allows your scripting-capable users of creating new Galaxy tools from scripts if they run it as administrators of their own laptop/development instances, or your administrators could use it to run scripts directly on your private instance and optionally generate new safe tools (by uploading the archives to your local tool shed then installing those new tools into your local Galaxy) for ordinary users to use in their workflows.
  

 
Thanks,
Karthik


On Sun, Jun 8, 2014 at 6:25 PM, Ross <[hidden email]> wrote:
Hi, Karthik and John.

Some details on the tool factory for anyone interested. 

Executive summary: it may be helpful in this context but only for trusted administrators.

TL;DR:

Firstly, it will refuse to run for anyone other than a local Galaxy administrator. This is because it exposes unrestricted scripting so should only be installed if you can trust your administrative users not to run "cd /; rm -rf *". I'd advise installing ONLY on your own private instance and NEVER on a public Galaxy.

Secondly, it has two modes of operation - script running and tool generation. 

When executed without the option to generate a tool archive, it will run a pasted (perl, python, R, bash) script creating an output in the history. This history output is re-doable in the usual Galaxy way including allowing the script to be edited and rerun, so it's possible to (eg) get a script working interactively - galaxy as an IDE anyone ? :)  

Once a script runs on some test data, the tool factory will optionally generate a complete tool shed compatible gzip which can be uploaded to any tool shed as a new or updated repository. The generated tool includes the supplied test data as a proper Galaxy functional test. Once a tool is in a toolshed, it is just another Galaxy tool, ready to be installed to any Galaxy like any other tool - but will require restarting of multiple web processes as John mentions. 

If the script is safe, the tool is safe - there are no specific security risks for tool factory generated tools other than the script itself.

Finally, currently it takes only one input and generates one output which is a substantial restriction - but of course the generated tool source code is easy to edit if you need more complex I/O. It has a really neat option to create a simple but useful HTML display with links and thumbnails to arbitrary (eg) pdf or other output files from a script - the tool form includes examples in all 4 scripting languages ready to cut and paste, including one which generates 50 random images and presents them in a grid as an HTML page for the user.

On Mon, Jun 9, 2014 at 10:32 AM, John Chilton <[hidden email]> wrote:
Galaxy doesn't really support this use case and it will be major
effort to get it work this way I suspect. Pieces to look at include:

The Galaxy Tool Factory (it has the ability to create reusable tools
from scripts):

http://www.ncbi.nlm.nih.gov/pubmed/23024011

You may be able to modify it in such a way that each tool is tagged
with who created and then use ToolBox filters to limit added tools to
a given user:

https://wiki.galaxyproject.org/UserDefinedToolboxFilters

I think the latest version of Galaxy has improved support for adding
tools without requiring restarts (using message queues). I don't know
if this will automatically work with the tool factory or not.

I suspect fighting Galaxy at every step on this will frustrate you and
the users - and you are exposing all of your users data to every user
you give this privilege to. Is this a shared cluster or is dedicated
to Galaxy? If it is shared - it might be better for advanced users to
just get importing and exporting data to user directories really well.
In my previous position at MSI we created a set of tools that allowed
Galaxy to SCP files as the user to our login nodes (using a SSH key
scheme) - likewise creating directories user's can upload files can
enabled power users.

Hopefully this helps,

-John

On Sat, Jun 7, 2014 at 1:12 PM, Karthik Gururaj
<[hidden email]> wrote:
> Hello,
>   We have a centralized Galaxy installation for out cluster which can be
> used by many users (backend interfaces with HTCondor to run jobs). Is there
> any way for non-administrator users to import/install their own tools so
> that they may use it in their workflows (under a subsection called
> "unstable", for example? Ideally, without having to restart the Galaxy
> daemons whenever a change is made.
> It would be fine if these tools are not visible to other users till the
> administrator says so.
> Thanks,
> Karthik
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/