Improved DRMAAJobRunner

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Improved DRMAAJobRunner

Matthias Bernt
Dear list,

I have implemented a new DRMAA job runner allows to detect runtime and
memory violations. This is done by using the drmaa library functions
job_info + wait or as fallback the commandline tools qstat + qacct. This
has the additional advantage that the runner works also in setups using
the external runner scripts to start jobs as real user (the original
DRMAAJobRunner can not query these jobs at all).

You may have a look here:

https://github.com/galaxyproject/galaxy/pull/4275

I have successfully tested the but settings: galaxy user/real user
submits the jobs. Also the resubmission in case of memory and time
violations seems to work.

Would be great to get some comments.

One problem that I encountered is that upload jobs do not work in the
real user setting (for the original and the new runner): The permissions
of the uploaded file in
/gpfs1/data/galaxy_server/galaxy-dev/database/tmp/ are not changed. Any
idea what needs to be changed to get this running?

Best,
Matthias

--

-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone +49 341 235 482296,
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Improved DRMAAJobRunner

Peter van Heusden
Code looks interesting. This is for Univa Grid Engine?

In terms of the upload jobs, are those not designed to be run as 'local' jobs and not with the 'real user' setting?

On Fri, 7 Jul 2017 at 17:32 Matthias Bernt <[hidden email]> wrote:
Dear list,

I have implemented a new DRMAA job runner allows to detect runtime and
memory violations. This is done by using the drmaa library functions
job_info + wait or as fallback the commandline tools qstat + qacct. This
has the additional advantage that the runner works also in setups using
the external runner scripts to start jobs as real user (the original
DRMAAJobRunner can not query these jobs at all).

You may have a look here:

https://github.com/galaxyproject/galaxy/pull/4275

I have successfully tested the but settings: galaxy user/real user
submits the jobs. Also the resubmission in case of memory and time
violations seems to work.

Would be great to get some comments.

One problem that I encountered is that upload jobs do not work in the
real user setting (for the original and the new runner): The permissions
of the uploaded file in
/gpfs1/data/galaxy_server/galaxy-dev/database/tmp/ are not changed. Any
idea what needs to be changed to get this running?

Best,
Matthias

--

-------------------------------------------
Matthias Bernt
Bioinformatics Service
Molekulare Systembiologie (MOLSYB)
Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
Helmholtz Centre for Environmental Research GmbH - UFZ
Permoserstraße 15, 04318 Leipzig, Germany
Phone <a href="tel:+49%20341%20235482296" value="+49341235482296" target="_blank">+49 341 235 482296,
[hidden email], www.ufz.de

Sitz der Gesellschaft/Registered Office: Leipzig
Registergericht/Registration Office: Amtsgericht Leipzig
Handelsregister Nr./Trade Register Nr.: B 4703
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
MinDirig Wilfried Kraus
Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
Prof. Dr. Dr. h.c. Georg Teutsch
Administrative Geschäftsführerin/ Administrative Managing Director:
Prof. Dr. Heike Graßmann
-------------------------------------------
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/