LSF cluster wierd behaviours!

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

LSF cluster wierd behaviours!

Hakeem Almabrazi-2

Hello everyone,

 

I have posted this earlier but I am afraid it did not go through I hope J

 

I was able to setup galaxy to work with our HPC cluster using the LSF scheduler.  So far so good except with few exceptions:

 

1)      I noticed one thing that submitting a job after a long period (for example overnight) the jobs do not get executed and more will not show up as jobs in the queue when I execute the “bjobs” command from the command line.   As if the jobs were never submitted to the LSF. However, if I submit a job from the command (i.e >bsub sleep -5), then I check the jobs in the queue using the bjobs command I see this job as well as the other jobs that were submitted and could not see them before.

Weird …. 

Has anyone seen this behavior before?  Is this related to galaxy setup?  Is there anything I should try out to get rid of such behavior?

 

2)       Also related to LSF setup.  Every time I restart galaxy it will not restart rather it will crash.  Then if I start it again it will start after that.  Here is the error I keep seeing after the first restart

“galaxy.jobs.runners.state_handler_factory DEBUG 2015-08-04 08:12:17,484 Loaded 'failure' state handler from module galaxy.jobs.runners.state_handlers.resubmit  “

Any idea to get rid of this as well?  Is this a job still in the database that I need to clean manually?  If so can you tell me what table(s) to look into to clear out.

 

3.  Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?

 

Thank you in advance for any tips or hints to resolve these issues. 

 

Best regards,

 

Hak

 

Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: LSF cluster wierd behaviours!

Nicola Soranzo-2
Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:

> Hello everyone,
>
> I was able to setup galaxy to work with our HPC cluster using the LSF
> scheduler. So far so good except with few exceptions:
>
>...
>
> 3. Finally, how do I control the resources (i.e cores for a job )
> given
> to a submitted job on Galaxy?

Hi Hakeem,
you need to specify new destinations in config/job_conf.xml, for
example if you want to submit jobs asking for 4 cores on the same
cluster node, use:

         <destination id="queue_name_4t" runner="drmaa">
             <param id="nativeSpecification">-q queue_name -n 4 -R
"span[hosts=1]"</param>
         </destination>

Then in the <tools> section of config/job_conf.xml add

         <tool id="tool_id" destination="queue_name_4t"/>

for each tool that should use 4 cores.

More info at:

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
https://wiki.galaxyproject.org/Admin/Config/Jobs

Ciao,
Nicola


Connetti gratis il mondo con la nuova indoona:  hai la chat, le
chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: LSF cluster wierd behaviours!

Hakeem Almabrazi-2
Thank you Nicola,

I will play with these options and see how it works.

Have you played with LSF and Galaxy setup?

Thank you,

-----Original Message-----
From: Nicola Soranzo [mailto:[hidden email]]
Sent: Tuesday, August 04, 2015 2:23 PM
To: Hakeem Almabrazi
Cc: [hidden email]
Subject: Re: [galaxy-dev] LSF cluster wierd behaviours!

Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
> Hello everyone,
>
> I was able to setup galaxy to work with our HPC cluster using the LSF
> scheduler. So far so good except with few exceptions:
>
>...
>
> 3. Finally, how do I control the resources (i.e cores for a job )
> given to a submitted job on Galaxy?

Hi Hakeem,
you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use:

         <destination id="queue_name_4t" runner="drmaa">
             <param id="nativeSpecification">-q queue_name -n 4 -R "span[hosts=1]"</param>
         </destination>

Then in the <tools> section of config/job_conf.xml add

         <tool id="tool_id" destination="queue_name_4t"/>

for each tool that should use 4 cores.

More info at:

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
https://wiki.galaxyproject.org/Admin/Config/Jobs

Ciao,
Nicola


Connetti gratis il mondo con la nuova indoona:  hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/


Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: LSF cluster wierd behaviours!

Nicola Soranzo-2
Yes, I'm using LSF with Galaxy at TGAC.

Unfortunately I don't have suggestions for your other problems.

Ciao,
Nicola

Il 04.08.2015 13:27 Hakeem Almabrazi ha scritto:

Thank you Nicola,

I will play with these options and see how it works.

Have you played with LSF and Galaxy setup?

Thank you,

-----Original Message-----
From: Nicola Soranzo [mailto:[hidden email]]
Sent: Tuesday, August 04, 2015 2:23 PM
To: Hakeem Almabrazi
Cc: [hidden email]Subject: Re: [galaxy-dev] LSF cluster wierd behaviours!

Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
Hello everyone, I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions: ... 3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?
Hi Hakeem,
you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use:-q queue_name -n 4 -R "span[hosts=1]"
         

Then in the  section of config/job_conf.xml add

         

for each tool that should use 4 cores.

More info at:

https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
https://wiki.galaxyproject.org/Admin/Config/Jobs

Ciao,
Nicola


Connetti gratis il mondo con la nuova indoona:  hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/


Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.



Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: LSF cluster wierd behaviours!

Hakeem Almabrazi-2

Nicola,

 

I have added your suggestion and seems to work J.  Is there a way to control these parameters based on a tool?  For example, lets say I want to require more cores for certain tools such as bwa.  How can I do that?

 

Here is my job_conf.xml.  Please let me know if you have better suggestion to it. 

 

<job_conf>

    <plugins>

        <plugin id="drmaa" type="runner" load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/>

    </plugins>

    <handlers default="handlers">

        <!--handler id="main" /-->

        <handler id="handler0" tags="handlers"/>

        <!--handler id="handler1" tags="handlers"/-->

    </handlers>

    <destinations default="lsf">

        <!--destination id="lsf" runner="drmaa"/-->

        <!-- suggested by Nicola-->

        <destination id="lsf" runner="drmaa">

             <param id="nativeSpecification">-n 32 -R "span[hosts=1]"</param>

        </destination>

    </destinations>

</job_conf>

 

Regards,

Hak

 

 

From: Nicola Soranzo [mailto:[hidden email]]
Sent: Tuesday, August 04, 2015 2:37 PM
To: Hakeem Almabrazi
Cc: [hidden email]
Subject: RE: [galaxy-dev] LSF cluster wierd behaviours!

 

Yes, I'm using LSF with Galaxy at TGAC.

Unfortunately I don't have suggestions for your other problems.

Ciao,
Nicola

Il 04.08.2015 13:27 Hakeem Almabrazi ha scritto:

Thank you Nicola,
 
I will play with these options and see how it works.
 
Have you played with LSF and Galaxy setup?
 
Thank you,
 
-----Original Message-----
From: Nicola Soranzo [mailto:[hidden email]]
Sent: Tuesday, August 04, 2015 2:23 PM
To: Hakeem Almabrazi
Cc: [hidden email]Subject: Re: [galaxy-dev] LSF cluster wierd behaviours!
 
Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:

Hello everyone, I was able to setup galaxy to work with our HPC cluster using the LSF scheduler. So far so good except with few exceptions: ... 3. Finally, how do I control the resources (i.e cores for a job ) given to a submitted job on Galaxy?

Hi Hakeem,
you need to specify new destinations in config/job_conf.xml, for example if you want to submit jobs asking for 4 cores on the same cluster node, use:-q queue_name -n 4 -R "span[hosts=1]"
         
 
Then in the  section of config/job_conf.xml add
 
         
 
for each tool that should use 4 cores.
 
More info at:
 
https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
https://wiki.galaxyproject.org/Admin/Config/Jobs
 
Ciao,
Nicola
 
 
Connetti gratis il mondo con la nuova indoona:  hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/
 
 
Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.




Connetti gratis il mondo con la nuova indoona: hai la chat, le chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/

Disclaimer: This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient, any reading, printing, storage, disclosure, copying or any other action taken in respect of this e-mail is prohibited and may be unlawful. If you are not the intended recipient, please notify the sender immediately by using the reply function and then permanently delete what you have received. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Sidra Medical and Research Center.
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: LSF cluster wierd behaviours!

Nicola Soranzo-2
Hi Hakeem,
you have to _add_ a destination for each number of cores you need, and
then specify this destination in the corresponding tools as I said in
the previous email, i.e.:

<job_conf>
     <plugins>
         <plugin id="drmaa" type="runner"
load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/>
     </plugins>
     <handlers default="handlers">
         <!--handler id="main" /-->
         <handler id="handler0" tags="handlers"/>
         <!--handler id="handler1" tags="handlers"/-->
     </handlers>
     <destinations default="lsf">
         <destination id="lsf" runner="drmaa"/>
         <destination id="lsf_4t" runner="drmaa">
              <param id="nativeSpecification">-n 4 -R
"span[hosts=1]"</param>
         </destination>
         <destination id="lsf_32t" runner="drmaa">
              <param id="nativeSpecification">-n 32 -R
"span[hosts=1]"</param>
         </destination>
     </destinations>
     <tools>
         <tool id="spades" destination="lsf_4t"/>
         <tool id="bwa" destination="lsf_32t"/>
     </tools>
</job_conf>

Cheers,
Nicola

Il 04.08.2015 14:12 Hakeem Almabrazi ha scritto:

> Nicola,
>
> I have added your suggestion and seems to work J. Is there a way to
> control these parameters based on a tool? For example, lets say I
> want to
> require more cores for certain tools such as bwa. How can I do that?
>
> Here is my job_conf.xml. Please let me know if you have better
> suggestion
> to it.
>
> <job_conf>
>     <plugins>
>         <plugin id="drmaa" type="runner"
> load="galaxy.jobs.runners.drmaa:DRMAAJobRunner" workers="4"/>
>     </plugins>
>     <handlers default="handlers">
>         <!--handler id="main" /-->
>         <handler id="handler0" tags="handlers"/>
>         <!--handler id="handler1" tags="handlers"/-->
>     </handlers>
>     <destinations default="lsf">
>         <!--destination id="lsf" runner="drmaa"/-->
>         <!-- suggested by Nicola-->
>         <destination id="lsf" runner="drmaa">
>              <param id="nativeSpecification">-n 32 -R
> "span[hosts=1]"</param>
>         </destination>
>     </destinations>
> </job_conf>
>
> Regards,
>
> Hak
>
> FROM: Nicola Soranzo [mailto:[hidden email]]
> SENT: Tuesday, August 04, 2015 2:37 PM
> TO: Hakeem Almabrazi
> CC: [hidden email]
> SUBJECT: RE: [galaxy-dev] LSF cluster wierd behaviours!
>
> Yes, I'm using LSF with Galaxy at TGAC.
>
> Unfortunately I don't have suggestions for your other problems.
>
> Ciao,
> Nicola
>
> Il 04.08.2015 13:27 Hakeem Almabrazi ha scritto:
>
>> Thank you Nicola,
>>
>> I will play with these options and see how it works.
>>
>> Have you played with LSF and Galaxy setup?
>>
>> Thank you,
>>
>> -----Original Message-----
>>
>> From: Nicola Soranzo [mailto:[hidden email] [1]]
>>
>> Sent: Tuesday, August 04, 2015 2:23 PM
>>
>> To: Hakeem Almabrazi
>>
>> Cc: [hidden email] [2]Subject: Re: [galaxy-dev]
>> LSF
>> cluster wierd behaviours!
>>
>> Il 04.08.2015 07:57 Hakeem Almabrazi ha scritto:
>>
>>> Hello everyone, I was able to setup galaxy to work with our HPC
>>> cluster using the LSF scheduler. So far so good except with few
>>> exceptions: ... 3. Finally, how do I control the resources (i.e
>>> cores
>>> for a job ) given to a submitted job on Galaxy?
>>
>> Hi Hakeem,
>>
>> you need to specify new destinations in config/job_conf.xml, for
>> example if you want to submit jobs asking for 4 cores on the same
>> cluster node, use:-q queue_name -n 4 -R "span[hosts=1]"
>>
>> Then in the section of config/job_conf.xml add
>>
>> for each tool that should use 4 cores.
>>
>> More info at:
>>
>> https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster [3]
>>
>> https://wiki.galaxyproject.org/Admin/Config/Jobs [4]
>>
>> Ciao,
>>
>> Nicola
>>
>> Connetti gratis il mondo con la nuova indoona: hai la chat, le
>> chiamate, le video chiamate e persino le chiamate di gruppo.
>>
>> E chiami gratis anche i numeri fissi e mobili nel mondo!
>>
>> Scarica subito l'app Vai su https://www.indoona.com/ [5]
>>
>> Disclaimer: This email and its attachments may be confidential and
>> are
>> intended solely for the use of the individual to whom it is
>> addressed.
>> If you are not the intended recipient, any reading, printing,
>> storage,
>> disclosure, copying or any other action taken in respect of this
>> e-mail
>> is prohibited and may be unlawful. If you are not the intended
>> recipient, please notify the sender immediately by using the reply
>> function and then permanently delete what you have received. Any
>> views
>> or opinions expressed are solely those of the author and do not
>> necessarily represent those of Sidra Medical and Research Center.
>
> Connetti gratis il mondo con la nuova indoona: hai la chat, le
> chiamate,
> le video chiamate e persino le chiamate di gruppo.
> E chiami gratis anche i numeri fissi e mobili nel mondo!
> Scarica subito l'app Vai su https://www.indoona.com/ [6] Disclaimer:
> This email and its attachments may be confidential and are intended
> solely for the use of the individual to whom it is addressed. If you
> are
> not the intended recipient, any reading, printing, storage,
> disclosure,
> copying or any other action taken in respect of this e-mail is
> prohibited
> and may be unlawful. If you are not the intended recipient, please
> notify
> the sender immediately by using the reply function and then
> permanently
> delete what you have received. Any views or opinions expressed are
> solely
> those of the author and do not necessarily represent those of Sidra
> Medical and Research Center.


Links:
------
[1] mailto:[hidden email]
[2] mailto:[hidden email]
[3] https://wiki.galaxyproject.org/Admin/Config/Performance/Cluster
[4] https://wiki.galaxyproject.org/Admin/Config/Jobs
[5] https://www.indoona.com/
[6] https://www.indoona.com/



Connetti gratis il mondo con la nuova indoona:  hai la chat, le
chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/