Restart Galaxy without interruption

classic Classic list List threaded Threaded
4 messages Options
| Threaded
Open this post in threaded view
|

Restart Galaxy without interruption

Gildas Le Corguillé
Hi,

Since Galaxy need to be restarted sometime (load datatype, add ressources in .loc, apply changes in the galaxy.ini), is anybody of you have solutions to restart it without interrupt the service?

We are now using nginx, uwsgi and supervisord.

My hope is to split the Galaxy process in two group (1 uwsgi and 2 job handlers per group). That way, we will be able to restart the group1 and then the group2.


I was planning to set the 2 uwsgi servers and delegate the load balancing to ngnix (https://www.nginx.com/resources/admin-guide/load-balancer/
But maybe, some of you already implemented that and can provide me some advices or warnings.

The job handler part seems easier to deal with 
--server-name=handler1_%(process_num)s
--server-name=handler2_%(process_num)s

Thanks by advance

Gildas

-----------------------------------------------------------------
Gildas Le Corguillé - Bioinformatician/Bioanalyste

Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
http://abims.sb-roscoff.fr

Member of the Workflow4Metabolomics project
http://workflow4metabolomics.org

Station Biologique de Roscoff - UPMC/CNRS - FR2424
Place Georges Teissier 29680 Roscoff FRANCE
tel: +33 2 98 29 23 81
------------------------------------------------------------------




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Restart Galaxy without interruption

Laure QUINTRIC

Hi Gildas,

Have a look here : https://github.com/martenson/dagobah-training/tree/master/advanced/002a-systemd-supervisor

See ya.
Laure


Le 14/11/2016 à 21:35, Gildas Le Corguillé a écrit :
Hi,

Since Galaxy need to be restarted sometime (load datatype, add ressources in .loc, apply changes in the galaxy.ini), is anybody of you have solutions to restart it without interrupt the service?

We are now using nginx, uwsgi and supervisord.

My hope is to split the Galaxy process in two group (1 uwsgi and 2 job handlers per group). That way, we will be able to restart the group1 and then the group2.


I was planning to set the 2 uwsgi servers and delegate the load balancing to ngnix (https://www.nginx.com/resources/admin-guide/load-balancer/
But maybe, some of you already implemented that and can provide me some advices or warnings.

The job handler part seems easier to deal with 
--server-name=handler1_%(process_num)s
--server-name=handler2_%(process_num)s

Thanks by advance

Gildas

-----------------------------------------------------------------
Gildas Le Corguillé - Bioinformatician/Bioanalyste

Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
http://abims.sb-roscoff.fr

Member of the Workflow4Metabolomics project
http://workflow4metabolomics.org

Station Biologique de Roscoff - UPMC/CNRS - FR2424
Place Georges Teissier 29680 Roscoff FRANCE
tel: +33 2 98 29 23 81
------------------------------------------------------------------





___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Restart Galaxy without interruption

Martin Čech-2
Hi Gildas,

the approach which Main Galaxy takes is as described by Nate in the exercise document linked by Laure. Exerpt here:

"In addition, you can gracefully restart the uWSGI Galaxy process with sudo supervisorctl signal HUP gx:galaxy. uWSGI is configured to start Galaxy in a "master" process and then fork the configured number of worker processes. Because of this, if sent a SIGHUP signal, it will kill the workers but the master process will hold its socket open, blocking client (browser) connections until new workers are forked. This prevents users from seeing a proxy error page during restarts."

Using this, you can restart both the uWSGI server and Galaxy handlers with:

$ sudo supervisorctl signal HUP gx:galaxy && sudo supervisorctl restart gx:handler0 gx:handler1

without supervisor it would be something like $ pkill -HUP -o -u <user> uwsgi

Thanks for using Galaxy,

Martin

On Wed, Nov 16, 2016 at 5:19 AM Laure QUINTRIC <[hidden email]> wrote:


Laure


Le 14/11/2016 à 21:35, Gildas Le Corguillé a écrit :
Hi,

Since Galaxy need to be restarted sometime (load datatype, add ressources in .loc, apply changes in the galaxy.ini), is anybody of you have solutions to restart it without interrupt the service?

We are now using nginx, uwsgi and supervisord.

My hope is to split the Galaxy process in two group (1 uwsgi and 2 job handlers per group). That way, we will be able to restart the group1 and then the group2.


I was planning to set the 2 uwsgi servers and delegate the load balancing to ngnix (https://www.nginx.com/resources/admin-guide/load-balancer/
But maybe, some of you already implemented that and can provide me some advices or warnings.

The job handler part seems easier to deal with 
--server-name=handler1_%(process_num)s
--server-name=handler2_%(process_num)s

Thanks by advance

Gildas

-----------------------------------------------------------------
Gildas Le Corguillé - Bioinformatician/Bioanalyste

Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
http://abims.sb-roscoff.fr

Member of the Workflow4Metabolomics project
http://workflow4metabolomics.org

Station Biologique de Roscoff - UPMC/CNRS - FR2424
Place Georges Teissier 29680 Roscoff FRANCE
tel: <a href="tel:+33%202%2098%2029%2023%2081" value="+33298292381" class="gmail_msg" target="_blank">+33 2 98 29 23 81
------------------------------------------------------------------





___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Restart Galaxy without interruption

Rémy Dernat
Hi,

Thanks Laure and Martin.

This is very interesting. However, can it handle the load balancing part as Gildas asked ? Anyway, thanks for the tip.

Can it be possible to share /etc/supervisor/conf.d/galaxy.conf without having to use ansible ? I downloaded the tarball, however, I did not find this file; is it (seems to be) galaxy.j2 in the template directory ?

Best regards,
Remy

2016-11-16 16:36 GMT+01:00 Martin Čech <[hidden email]>:
Hi Gildas,

the approach which Main Galaxy takes is as described by Nate in the exercise document linked by Laure. Exerpt here:

"In addition, you can gracefully restart the uWSGI Galaxy process with sudo supervisorctl signal HUP gx:galaxy. uWSGI is configured to start Galaxy in a "master" process and then fork the configured number of worker processes. Because of this, if sent a SIGHUP signal, it will kill the workers but the master process will hold its socket open, blocking client (browser) connections until new workers are forked. This prevents users from seeing a proxy error page during restarts."

Using this, you can restart both the uWSGI server and Galaxy handlers with:

$ sudo supervisorctl signal HUP gx:galaxy && sudo supervisorctl restart gx:handler0 gx:handler1

without supervisor it would be something like $ pkill -HUP -o -u <user> uwsgi

Thanks for using Galaxy,

Martin

On Wed, Nov 16, 2016 at 5:19 AM Laure QUINTRIC <[hidden email]> wrote:


Laure


Le 14/11/2016 à 21:35, Gildas Le Corguillé a écrit :
Hi,

Since Galaxy need to be restarted sometime (load datatype, add ressources in .loc, apply changes in the galaxy.ini), is anybody of you have solutions to restart it without interrupt the service?

We are now using nginx, uwsgi and supervisord.

My hope is to split the Galaxy process in two group (1 uwsgi and 2 job handlers per group). That way, we will be able to restart the group1 and then the group2.


I was planning to set the 2 uwsgi servers and delegate the load balancing to ngnix (https://www.nginx.com/resources/admin-guide/load-balancer/
But maybe, some of you already implemented that and can provide me some advices or warnings.

The job handler part seems easier to deal with 
--server-name=handler1_%(process_num)s
--server-name=handler2_%(process_num)s

Thanks by advance

Gildas

-----------------------------------------------------------------
Gildas Le Corguillé - Bioinformatician/Bioanalyste

Plateform ABiMS (Analyses and Bioinformatics for Marine Science)
http://abims.sb-roscoff.fr

Member of the Workflow4Metabolomics project
http://workflow4metabolomics.org

Station Biologique de Roscoff - UPMC/CNRS - FR2424
Place Georges Teissier 29680 Roscoff FRANCE
tel: <a href="tel:+33%202%2098%2029%2023%2081" value="+33298292381" class="m_-7685101076329386380gmail_msg" target="_blank">+33 2 98 29 23 81
------------------------------------------------------------------





___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/