Run Jobs as Real User - How to Configure it for TORQUE

classic Classic list List threaded Threaded
5 messages Options
| Threaded
Open this post in threaded view
|

Run Jobs as Real User - How to Configure it for TORQUE

Ping
This post was updated on .
I have installed pbs_python module on our cluster to interface Galaxy with
TORQUE. I am able to submit and run jobs on our cluster as the Galaxy user.
I need to run Galaxy jobs as the real user. The instruction in the user
guide is for DRMAA interface. How can I configure running jobs as real user
for TORQUE?

thank,

Ping

Update:

I read the user guide more carefully. It seems I can install pbs-drmaa for TORQUE. I tried this approach but I got
segmenation fault when starting the galaxy server. Has anybody successfully integrated TORQUE with galaxy
through pbs-drmaa?


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Run Jobs as Real User - How to Configure it for TORQUE

John Chilton-4
Running jobs as the "real" user is not available with the PBS job
runner - one has to use the DRMAA interface to submit jobs as the real
user.

I have created a Trello card to add this functionality:
https://trello.com/c/OddS8bMP

Would be happy to field pull requests to add this - because I doubt
anyone on the core team will be able to get to this anytime soon.
Sorry I don't have better news.

-John

On Sun, Jul 20, 2014 at 11:26 PM, Ping Luo <[hidden email]> wrote:

> I have installed pbs_python module on our cluster to interface Galaxy with
> TORQUE. I am able to submit and run jobs on our cluster as the Galaxy user.
> I need to run Galaxy jobs as the real user. The instruction in the user
> guide is for DRMAA interface. How can I configure running jobs as real user
> for TORQUE?
>
> thank,
>
> Ping
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Run Jobs as Real User - How to Configure it for TORQUE

Nate Coraor (nate@bx.psu.edu)
On Jul 22, 2014, at 2:13 PM, John Chilton <[hidden email]> wrote:

Running jobs as the "real" user is not available with the PBS job
runner - one has to use the DRMAA interface to submit jobs as the real
user.

That said, the DRMAA runner plugin is compatible with Torque, you just need to compile and use pbs-drmaa, rather than Torque's libdrmaa:


--nate


I have created a Trello card to add this functionality:
https://trello.com/c/OddS8bMP

Would be happy to field pull requests to add this - because I doubt
anyone on the core team will be able to get to this anytime soon.
Sorry I don't have better news.

-John

On Sun, Jul 20, 2014 at 11:26 PM, Ping Luo <[hidden email]> wrote:
I have installed pbs_python module on our cluster to interface Galaxy with
TORQUE. I am able to submit and run jobs on our cluster as the Galaxy user.
I need to run Galaxy jobs as the real user. The instruction in the user
guide is for DRMAA interface. How can I configure running jobs as real user
for TORQUE?

thank,

Ping

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Run Jobs as Real User - How to Configure it for TORQUE

Ping
Nate and John,

I tried with pbs-drmaa and it crashed my galaxy server when the server was started and tried to load the drmaa library .
However, I am able to make the drmaa library that comes with TORQUE working with galaxy if sudo is not use.. With sudo,
there are still problems I am working to resolve.

Ping


On Thu, Jul 24, 2014 at 10:10 AM, Nate Coraor <[hidden email]> wrote:
On Jul 22, 2014, at 2:13 PM, John Chilton <[hidden email]> wrote:

Running jobs as the "real" user is not available with the PBS job
runner - one has to use the DRMAA interface to submit jobs as the real
user.

That said, the DRMAA runner plugin is compatible with Torque, you just need to compile and use pbs-drmaa, rather than Torque's libdrmaa:


--nate


I have created a Trello card to add this functionality:
https://trello.com/c/OddS8bMP

Would be happy to field pull requests to add this - because I doubt
anyone on the core team will be able to get to this anytime soon.
Sorry I don't have better news.

-John

On Sun, Jul 20, 2014 at 11:26 PM, Ping Luo <[hidden email]> wrote:
I have installed pbs_python module on our cluster to interface Galaxy with
TORQUE. I am able to submit and run jobs on our cluster as the Galaxy user.
I need to run Galaxy jobs as the real user. The instruction in the user
guide is for DRMAA interface. How can I configure running jobs as real user
for TORQUE?

thank,

Ping

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Run Jobs as Real User - How to Configure it for TORQUE

Ping
Nate and John,

I managed to make sudo work with galaxy and TORQUE with some fixes:

1. the configuration for external runner/killer/chown cannot be read from ether universe_wsgi.ini or job_conf.xml.
    I haven't looked into why but just hard coded them into the galaxy source code.
2. drmaa cannot recognize the key word 'arg' and 'email'. Again I did a dirty fix by deleting them in both drmaa.py
    and drmaa_external_runner.py.
3. 'sudo -E' will rename LD_LIBRARY_PATH to LIBRARY_PATH. So I have to create a wrapper for each of the
    three scripts (runner/killer/chown) to correct this.
4. The sudo file should now contain the three wrappers. Here I would like to point out an error in the user guide.
    Even for people who doesn't need the wrappers,  the chown script shouldn't not be '/bin/chown' in the sudo file,
    but the python script /path/to/external_chown_script.py.


After these steps, I am able to run galaxy jobs as the real user. Now I need to see how to fix 1 and 2 in an elegant way.


Ping



On Thu, Jul 24, 2014 at 12:05 PM, Ping Luo <[hidden email]> wrote:
Nate and John,

I tried with pbs-drmaa and it crashed my galaxy server when the server was started and tried to load the drmaa library .
However, I am able to make the drmaa library that comes with TORQUE working with galaxy if sudo is not use.. With sudo,
there are still problems I am working to resolve.

Ping


On Thu, Jul 24, 2014 at 10:10 AM, Nate Coraor <[hidden email]> wrote:
On Jul 22, 2014, at 2:13 PM, John Chilton <[hidden email]> wrote:

Running jobs as the "real" user is not available with the PBS job
runner - one has to use the DRMAA interface to submit jobs as the real
user.

That said, the DRMAA runner plugin is compatible with Torque, you just need to compile and use pbs-drmaa, rather than Torque's libdrmaa:


--nate


I have created a Trello card to add this functionality:
https://trello.com/c/OddS8bMP

Would be happy to field pull requests to add this - because I doubt
anyone on the core team will be able to get to this anytime soon.
Sorry I don't have better news.

-John

On Sun, Jul 20, 2014 at 11:26 PM, Ping Luo <[hidden email]> wrote:
I have installed pbs_python module on our cluster to interface Galaxy with
TORQUE. I am able to submit and run jobs on our cluster as the Galaxy user.
I need to run Galaxy jobs as the real user. The instruction in the user
guide is for DRMAA interface. How can I configure running jobs as real user
for TORQUE?

thank,

Ping

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/