Load balancing and job configuration

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Load balancing and job configuration

Olivier Inizan
Dear galaxy-dev-list,

I am trying to use new-style job configuration whith a load-balancing on 2
web servers and 2 job hanlders.
I have configured load balancing as follows (universe_wsgi.ini):

# Configuration of the internal HTTP server.

[server:web0]
use = egg:Paste#http
port = 8083
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 7

[server:web1]
use = egg:Paste#http
port = 8082
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 7

[server:manager]
use = egg:Paste#http
port = 8079
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler0]
use = egg:Paste#http
port = 8090
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler1]
use = egg:Paste#http
port = 8091
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5


This configuration works fine (manager.log):
galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler
galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner
workers
galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner
'galaxy.jobs.runners.local:LocalJobRunner' as 'local'
galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner
workers
galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner
'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr'
galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id
'local:///', url 'local:///' converted, got params:
galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners
plugins: lwr:local
galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue
started
galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started

To use new-style job configuration I have create the following
job_conf.xml:


<?xml version="1.0"?>
<!-- A sample job config that explicitly configures job running the way it
is configured by default (if there is no explicit config). -->
<job_conf>
     <plugins>
         <plugin id="local" type="runner"
load="galaxy.jobs.runners.local:LocalJobRunner"/>
     </plugins>
     <handlers default="handlers">
         <!-- Additional job handlers - the id should match the name of a
              [server:<id>] in universe_wsgi.ini.
          -->
         <handler id="server:web0" tags="handlers"/>
         <handler id="server:web1" tags="handlers"/>
     </handlers>

     <destinations default="local">
         <destination id="local" runner="local"/>
     </destinations>
</job_conf>


When I restart the instance the manager.log outputs:

galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from
./job_conf.xml
galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler
'server:web0'
galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler
'server:web1'
galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <handlers> default set to child
with id or tag 'handlers'
galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <destinations> default set to
child with id or tag 'local'
galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration

So everything seems fine but when I try to run a tool, the corresponding
job is in permanent waiting status (gray color in history).

I think I have missed something in the configuration process,
Thanks for your help,

Olivier


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Load balancing and job configuration

Adam Brenner
Oliver,

The new job running file is also providing me a lot of headaches. The
documentation on site is correct, but some of the items are not yet
implemented, for example, DRMAA external scripts still need to be in
the universe_wsgi.ini, yet the site say to put them in job_conf.xml
file! Lot of hair pulling and waiting response from IRC to figure this
out...wasn't fun.


As for your issue, you want to specify your handlers, not your
webserver for your handler types.

<handler id="handler0" tags="handlers"/>
<handler id="handler1" tags="handlers"/>

Let me know if this helps,
-Adam

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

Research Computing Support
Office of Information Technology
http://www.oit.uci.edu/rcs/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
[hidden email]

On Thu, May 2, 2013 at 9:00 AM, Olivier Inizan
<[hidden email]> wrote:

> Dear galaxy-dev-list,
>
> I am trying to use new-style job configuration whith a load-balancing on 2
> web servers and 2 job hanlders.
> I have configured load balancing as follows (universe_wsgi.ini):
>
> # Configuration of the internal HTTP server.
>
> [server:web0]
> use = egg:Paste#http
> port = 8083
> host = 127.0.0.1
> use_threadpool = true
> threadpool_workers = 7
>
> [server:web1]
> use = egg:Paste#http
> port = 8082
> host = 127.0.0.1
> use_threadpool = true
> threadpool_workers = 7
>
> [server:manager]
> use = egg:Paste#http
> port = 8079
> host = 127.0.0.1
> use_threadpool = true
> threadpool_workers = 5
>
> [server:handler0]
> use = egg:Paste#http
> port = 8090
> host = 127.0.0.1
> use_threadpool = true
> threadpool_workers = 5
>
> [server:handler1]
> use = egg:Paste#http
> port = 8091
> host = 127.0.0.1
> use_threadpool = true
> threadpool_workers = 5
>
>
> This configuration works fine (manager.log):
> galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler
> galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner
> workers
> galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner
> 'galaxy.jobs.runners.local:LocalJobRunner' as 'local'
> galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner
> workers
> galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner
> 'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr'
> galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id
> 'local:///', url 'local:///' converted, got params:
> galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners
> plugins: lwr:local
> galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue
> started
> galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started
>
> To use new-style job configuration I have create the following job_conf.xml:
>
>
> <?xml version="1.0"?>
> <!-- A sample job config that explicitly configures job running the way it
> is configured by default (if there is no explicit config). -->
> <job_conf>
>     <plugins>
>         <plugin id="local" type="runner"
> load="galaxy.jobs.runners.local:LocalJobRunner"/>
>     </plugins>
>     <handlers default="handlers">
>         <!-- Additional job handlers - the id should match the name of a
>              [server:<id>] in universe_wsgi.ini.
>          -->
>         <handler id="server:web0" tags="handlers"/>
>         <handler id="server:web1" tags="handlers"/>
>     </handlers>
>
>     <destinations default="local">
>         <destination id="local" runner="local"/>
>     </destinations>
> </job_conf>
>
>
> When I restart the instance the manager.log outputs:
>
> galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from
> ./job_conf.xml
> galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler
> 'server:web0'
> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler
> 'server:web1'
> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <handlers> default set to child
> with id or tag 'handlers'
> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <destinations> default set to
> child with id or tag 'local'
> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration
>
> So everything seems fine but when I try to run a tool, the corresponding job
> is in permanent waiting status (gray color in history).
>
> I think I have missed something in the configuration process,
> Thanks for your help,
>
> Olivier
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Load balancing and job configuration

Olivier Inizan

Thanks Adam when I specify handlers it works fine :)

I also try to add a drmaa plugin and create a new destination:

<job_conf>
     <plugins>
         <plugin id="local" type="runner"
load="galaxy.jobs.runners.local:LocalJobRunner"/>
         <plugin id="sge" type="runner"
load="galaxy.jobs.runners.drmaa:DRMAAJobRunner"/>
     </plugins>
     <handlers default="handlers">
         <handler id="server:handler0" tags="handlers"/>
         <handler id="server:handler1" tags="handlers"/>
     </handlers>

     <destinations default="local">
         <destination id="sge_default" runner="sge"/>
         <destination id="local" runner="local"/>
     </destinations>
</job_conf>

My env var DRMAA_LIBRARY_PATH is set to /usr/lib64/libdrmaa.so.1.0, and
when I restart the instance I have the following error in handler's log
files:

galaxy.jobs.manager DEBUG 2013-05-03 17:06:50,978 Starting job handler
galaxy.jobs.runners DEBUG 2013-05-03 17:06:50,979 Starting 4 LocalRunner
workers
galaxy.jobs DEBUG 2013-05-03 17:06:50,980 Loaded job runner
'galaxy.jobs.runners.local:LocalJobRunner' as 'local'
Traceback (most recent call last):
   File "/home/galaxy/galaxy-dist/lib/galaxy/webapps/galaxy/buildapp.py",
line 37, in app_factory
     app = UniverseApplication( global_conf = global_conf, **kwargs )
   File "/home/galaxy/galaxy-dist/lib/galaxy/app.py", line 159, in __init__
     self.job_manager = manager.JobManager( self )
   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/manager.py", line 32, in
__init__
     self.job_handler = handler.JobHandler( app )
   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 29, in
__init__
     self.dispatcher = DefaultJobDispatcher( app )
   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/handler.py", line 543, in
__init__
     self.job_runners = self.app.job_config.get_job_runner_plugins()
   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/__init__.py", line 486,
in get_job_runner_plugins
     rval[id] = runner_class( self.app, runner[ 'workers' ], **runner.get(
'kwds', {} ) )
   File "/home/galaxy/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line
75, in __init__
     self.ds.initialize()
   File
"/home/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/__init__.py",
line 274, in initialize
     _w.init(contactString)
   File
"/home/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/wrappers.py",
line 59, in init
     return _lib.drmaa_init(contact, error_buffer, sizeof(error_buffer))
   File
"/home/galaxy/galaxy-dist/eggs/drmaa-0.4b3-py2.6.egg/drmaa/errors.py",
line 90, in error_check
     raise _ERRORS[code-1]("code %s: %s" % (code, error_buffer.value))
DrmCommunicationException: code 2: range_list containes no elements
Removing PID file handler0.pid


Any ideas ?
Thanks again for your help.




Olivier Inizan
Unité de Recherches en Génomique-Info (UR INRA 1164),
INRA, Centre de recherche de Versailles, bat.18
RD10, route de Saint Cyr
78026 Versailles Cedex, FRANCE

[hidden email]

Tél: +33 1 30 83 38 25
Fax: +33 1 30 83 38 99
http://urgi.versailles.inra.fr [urgi.versailles.inra.fr]
Twitter: @OlivierInizan


On Thu, 2 May 2013, Adam Brenner wrote:

> Oliver,
>
> The new job running file is also providing me a lot of headaches. The
> documentation on site is correct, but some of the items are not yet
> implemented, for example, DRMAA external scripts still need to be in
> the universe_wsgi.ini, yet the site say to put them in job_conf.xml
> file! Lot of hair pulling and waiting response from IRC to figure this
> out...wasn't fun.
>
>
> As for your issue, you want to specify your handlers, not your
> webserver for your handler types.
>
> <handler id="handler0" tags="handlers"/>
> <handler id="handler1" tags="handlers"/>
>
> Let me know if this helps,
> -Adam
>
> --
> Adam Brenner
> Computer Science, Undergraduate Student
> Donald Bren School of Information and Computer Sciences
>
> Research Computing Support
> Office of Information Technology
> http://www.oit.uci.edu/rcs/
>
> University of California, Irvine
> www.ics.uci.edu/~aebrenne/
> [hidden email]
>
> On Thu, May 2, 2013 at 9:00 AM, Olivier Inizan
> <[hidden email]> wrote:
>> Dear galaxy-dev-list,
>>
>> I am trying to use new-style job configuration whith a load-balancing on 2
>> web servers and 2 job hanlders.
>> I have configured load balancing as follows (universe_wsgi.ini):
>>
>> # Configuration of the internal HTTP server.
>>
>> [server:web0]
>> use = egg:Paste#http
>> port = 8083
>> host = 127.0.0.1
>> use_threadpool = true
>> threadpool_workers = 7
>>
>> [server:web1]
>> use = egg:Paste#http
>> port = 8082
>> host = 127.0.0.1
>> use_threadpool = true
>> threadpool_workers = 7
>>
>> [server:manager]
>> use = egg:Paste#http
>> port = 8079
>> host = 127.0.0.1
>> use_threadpool = true
>> threadpool_workers = 5
>>
>> [server:handler0]
>> use = egg:Paste#http
>> port = 8090
>> host = 127.0.0.1
>> use_threadpool = true
>> threadpool_workers = 5
>>
>> [server:handler1]
>> use = egg:Paste#http
>> port = 8091
>> host = 127.0.0.1
>> use_threadpool = true
>> threadpool_workers = 5
>>
>>
>> This configuration works fine (manager.log):
>> galaxy.jobs.manager DEBUG 2013-04-23 12:20:22,901 Starting job handler
>> galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,902 Starting 5 LocalRunner
>> workers
>> galaxy.jobs DEBUG 2013-04-23 12:20:22,904 Loaded job runner
>> 'galaxy.jobs.runners.local:LocalJobRunner' as 'local'
>> galaxy.jobs.runners DEBUG 2013-04-23 12:20:22,909 Starting 3 LWRRunner
>> workers
>> galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Loaded job runner
>> 'galaxy.jobs.runners.lwr:LwrJobRunner' as 'lwr'
>> galaxy.jobs DEBUG 2013-04-23 12:20:22,911 Legacy destination with id
>> 'local:///', url 'local:///' converted, got params:
>> galaxy.jobs.handler DEBUG 2013-04-23 12:20:22,911 Loaded job runners
>> plugins: lwr:local
>> galaxy.jobs.handler INFO 2013-04-23 12:20:22,912 job handler stop queue
>> started
>> galaxy.jobs.handler INFO 2013-04-23 12:20:22,919 job handler queue started
>>
>> To use new-style job configuration I have create the following job_conf.xml:
>>
>>
>> <?xml version="1.0"?>
>> <!-- A sample job config that explicitly configures job running the way it
>> is configured by default (if there is no explicit config). -->
>> <job_conf>
>>     <plugins>
>>         <plugin id="local" type="runner"
>> load="galaxy.jobs.runners.local:LocalJobRunner"/>
>>     </plugins>
>>     <handlers default="handlers">
>>         <!-- Additional job handlers - the id should match the name of a
>>              [server:<id>] in universe_wsgi.ini.
>>          -->
>>         <handler id="server:web0" tags="handlers"/>
>>         <handler id="server:web1" tags="handlers"/>
>>     </handlers>
>>
>>     <destinations default="local">
>>         <destination id="local" runner="local"/>
>>     </destinations>
>> </job_conf>
>>
>>
>> When I restart the instance the manager.log outputs:
>>
>> galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Loading job configuration from
>> ./job_conf.xml
>> galaxy.jobs DEBUG 2013-05-02 17:25:28,402 Read definition for handler
>> 'server:web0'
>> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Read definition for handler
>> 'server:web1'
>> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <handlers> default set to child
>> with id or tag 'handlers'
>> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 <destinations> default set to
>> child with id or tag 'local'
>> galaxy.jobs DEBUG 2013-05-02 17:25:28,403 Done loading job configuration
>>
>> So everything seems fine but when I try to run a tool, the corresponding job
>> is in permanent waiting status (gray color in history).
>>
>> I think I have missed something in the configuration process,
>> Thanks for your help,
>>
>> Olivier
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/