Job launching slowed down by conda auto-install

classic Classic list List threaded Threaded
2 messages Options
| Threaded
Open this post in threaded view
|

Job launching slowed down by conda auto-install

Tiago Macedo
Hello all,

I recently realized that setting conda_auto_install to true in
galaxy.ini has a noticeable effect in slowing down the start of job
execution. On the first job run after activating conda_auto_install the
job takes a lot longer as conda is actually installing packages, which
is expected. On the following runs, even if there are no packages to be
installed jobs still take longer to start. What's more surprising is
that after deactivating conda_auto_install jobs still take longer to run.
My particular test case was a simple small (12MB) file upload. It
consistently went from 15s to over 30s after activating
conda_auto_install and remained at that even after deactivating it. We
tested running a few tools which presented a noticeable and irreversible
delay after activation of conda_auto_install, but I didn't measure these
delays. I also tested this on fresh installs of galaxy 16.07 and 16.10
with very similar results.
Is this supposed to happen? Is there a way around it besides never
activating conda_auto_install?

Thanks in advance,
Tiago


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Job launching slowed down by conda auto-install

Marius van den Beek
Hello Tiago,

conda_auto_install will keep on trying to install the dependencies for a tool,
each and every time, even if that dependency does not exist. 
Starting with release 16.07 you can see if a conda dependency is installed for a specific repository,
and you can opt to install conda dependencies during installation.
I would strongly recommend to not activate conda_auto_install on a normal production server.

Best,
Marius

On 21 December 2016 at 17:21, Tiago Macedo <[hidden email]> wrote:
Hello all,

I recently realized that setting conda_auto_install to true in galaxy.ini has a noticeable effect in slowing down the start of job execution. On the first job run after activating conda_auto_install the job takes a lot longer as conda is actually installing packages, which is expected. On the following runs, even if there are no packages to be installed jobs still take longer to start. What's more surprising is that after deactivating conda_auto_install jobs still take longer to run.
My particular test case was a simple small (12MB) file upload. It consistently went from 15s to over 30s after activating conda_auto_install and remained at that even after deactivating it. We tested running a few tools which presented a noticeable and irreversible delay after activation of conda_auto_install, but I didn't measure these delays. I also tested this on fresh installs of galaxy 16.07 and 16.10 with very similar results.
Is this supposed to happen? Is there a way around it besides never activating conda_auto_install?

Thanks in advance,
Tiago


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/