Error during tool installation: "inotify instance limit reached"

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Error during tool installation: "inotify instance limit reached"

Previti
Dear all,

I've been getting this error for a few days now, it basically prevents
anything from being installed on the first go.
Individual tools can be installed, but even then I have to try multiple
times.

Any ideas how to fix this?

Cheers,
Christopher

The details from galaxy.log are (installing an older VCFtools package):

172.22.24.119 - - [10/Jul/2019:10:44:32 +0200] "POST
/admin_toolshed/prepare_for_install HTTP/1.1" 200 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?change
set_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F"
"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
Firefox/67.0"
galaxy.tools.toolbox.base DEBUG 2019-07-10 10:44:55,592 Appending to
tool panel section: NGS:VCFtools
172.22.24.119 - - [10/Jul/2019:10:44:58 +0200] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
_for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
Gecko/20100101 Firefox/67.0"
172.22.24.119 - - [10/Jul/2019:10:45:01 +0200] "POST
/admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
_for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
Gecko/20100101 Firefox/67.0"
172.22.24.119 - - [10/Jul/2019:10:44:55 +0200] "POST
/admin_toolshed/install_repositories HTTP/1.1" 500 -
"http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install"
"Moz
illa/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
*Error - <type 'exceptions.OSError'>: inotify instance limit reached*
URL:
http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/install_repositories
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/middleware/error.py',
line 154 in __call__
  app_iter = self.application(environ, sr_checker)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py',
line 85 in __call__
  return self.application(environ, start_response)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py',
line 640 in __call__
  return self.application(environ, start_response)
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line 143 in
__call__
  return self.handle_request(environ, start_response)
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line 222 in
handle_request
  body = method(trans, **kwargs)
File '/opt/galaxy/galaxy/lib/galaxy/web/framework/decorators.py', line
101 in decorator
  return func(self, trans, *args, **kwargs)
File
'/opt/galaxy/galaxy/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
line 575 in install_repositories
  reinstalling=reinstalling,
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
line 846 in install_repositories
  tool_panel_section_mapping=tool_panel_section_mapping)
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
line 893 in install_tool_shed_repository
  tool_panel_section_mapping=tool_panel_section_mapping)
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
line 544 in __handle_repository_contents
  repository_tools_tups = irmm.get_repository_tools_tups()
File
'/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py',
line 75 in get_repository_tools_tups
  tool = self.app.toolbox.load_tool(os.path.abspath(load_relative_path),
guid=guid, use_cached=False)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line 766 in
load_tool
  self.watch_tool(tool)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line 774 in
watch_tool
  self._tool_watcher.watch_file(tool.config_file, tool.id)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py', line 198
in watch_file
  self.monitor(tool_dir)
File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py', line 190
in monitor
  self.observer.schedule(self.event_handler, dir, recursive=False)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/api.py',
line 293 in schedule
  emitter.start()
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/utils/__init__.py',
line 110 in start
  self.on_thread_start()
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify.py',
line 121 in on_thread_start
  self._inotify = InotifyBuffer(path, self.watch.is_recursive)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_buffer.py',
line 35 in __init__
  self._inotify = Inotify(path, recursive)
File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
line 188 in __init__
  Inotify._raise_error()
*File
'/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
line 415 in _raise_error**
**  raise OSError("inotify instance limit reached")**
**OSError: inotify instance limit reached**
*
CGI Variables
-------------
  CONTENT_LENGTH: '12169'
  CONTENT_TYPE: 'application/x-www-form-urlencoded; charset=UTF-8'
  HTTP_ACCEPT: 'text/html, */*; q=0.01'
  HTTP_ACCEPT_ENCODING: 'gzip,deflate'
  HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
  HTTP_CONNECTION: 'Keep-Alive'
  HTTP_COOKIE:
'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'
  HTTP_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de'
  HTTP_REFERER:
'http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install'
  HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
Gecko/20100101 Firefox/67.0'
  HTTP_X_FORWARDED_FOR: '172.22.24.119'
  HTTP_X_FORWARDED_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de'
  HTTP_X_FORWARDED_SERVER: 'fe80::216:3eff:fe80:c02b'
  HTTP_X_REQUESTED_WITH: 'XMLHttpRequest'
  ORGINAL_HTTP_HOST: 'localhost:8080'
  ORGINAL_REMOTE_ADDR: '127.0.0.1'
  PATH_INFO: '/admin_toolshed/install_repositories'
  REMOTE_ADDR: '172.22.24.119'
  REQUEST_METHOD: 'POST'
  SERVER_NAME: '127.0.0.1'
  SERVER_PORT: '8080'
  SERVER_PROTOCOL: 'HTTP/1.1'


WSGI Variables
--------------
  application: <paste.recursive.RecursiveMiddleware object at
0x7f9f48207410>
  controller_action_key: u'web.admin_toolshed.install_repositories'
  is_api_request: False
  paste.cookies: (<SimpleCookie:
galaxysession='c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'>,
'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e')
  paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>]
  paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler
object at 0x7f9f48c86fd0>
  paste.httpserver.proxy.host: 'dummy'
  paste.httpserver.proxy.scheme: 'http'
  paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at
0x7f9f486fcd90>
  paste.recursive.forward: <paste.recursive.Forwarder from />
  paste.recursive.include: <paste.recursive.Includer from />
  paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from />
  paste.recursive.script_name: ''
  paste.throw_errors: True
  request_id: 'fd8cc98ea2ee11e9a89100163e80c02b'
  webob._body_file: (<_io.BufferedReader>, <socket._fileobject object at
0x7f9f3a4b5ad0 length=12169>)
  webob._parsed_post_vars: (MultiDict([('tool_shed_repository_ids',
u"[u'ac218635d795e337', u'e3e09dfaa8fad8c8', u'1efe9eb0bff40152',
u'358d8704ef89cb6b', u'cbc5119f47881258']"), ('encoded_kwd',
u'59b03a7799b953da9827424b9cef2236c5b9f461:7b22737461747573223a2022646f6e65222c20226861735f7265706f7369746f72795f646570656e64656e63696573223a2066616c73652c2022696e636c756465735f746f6f6c735f666f725f646973706c61795f696e5f746f6f6c5f70616e656c223a20747275652c2022746f6f6c5f736865645f7265706f7369746f72795f696473223a205b2261633231383633356437393565333337222c202265336530396466616138666164386338222c202231656665396562306266663430313532222c202233353864383730346566383963623662222c202263626335313139663437383831323538225d2c2022736865645f746f6f6c5f636f6e66223a20222e2f636f6e6669672f736865645f746f6f6c5f636f6e662e786d6c222c2022696e7374616c6c5f7265706f7369746f72795f646570656e64656e63696573223a20747275652c2022746f6f6c5f70617468223a20222f6f70742f67616c6178792f736865645f746f6f6c73222c20227265706f5f696e666f5f6469637473223a2...
0x7f9f35cdfed0>)
  webob._parsed_query_vars: (GET([]), '')
  webob.is_body_seekable: True
  wsgi process: 'Multithreaded'
  wsgi.input_terminated: True




--
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

[hidden email] <http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Error during tool installation: "inotify instance limit reached"

Marius van den Beek
Hi Christopher,

you might want to turn off `watch_tools` if you are using it,
this is only needed when using the pretty-much undocumented
feature where you dump tools into a directory without registering them
in any tool_conf.xml file. All other tool reloading happens by another
means.

Similarly, if you're using 19.05 and you don't manipulate the tool data
table
entries manually you can disable `watch_tool_data_dir`.

Finally this is usually a sign of allocating too many processes and threads
for your webservers/handlers/mules,
as each one starts their own watchdog. A common misconception is that this
makes Galaxy run faster.
but it actually just increases the amount of concurrent requests Galaxy can
handle.

Best,
Marius

On Wed, 10 Jul 2019 at 11:43, Previti <
[hidden email]> wrote:

> Dear all,
>
> I've been getting this error for a few days now, it basically prevents
> anything from being installed on the first go.
> Individual tools can be installed, but even then I have to try multiple
> times.
>
> Any ideas how to fix this?
>
> Cheers,
> Christopher
>
> The details from galaxy.log are (installing an older VCFtools package):
>
> 172.22.24.119 - - [10/Jul/2019:10:44:32 +0200] "POST
> /admin_toolshed/prepare_for_install HTTP/1.1" 200 -
> "
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?change
>
> set_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
> <http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?changeset_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F>
> "
> "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
> Firefox/67.0"
> galaxy.tools.toolbox.base DEBUG 2019-07-10 10:44:55,592 Appending to
> tool panel section: NGS:VCFtools
> 172.22.24.119 - - [10/Jul/2019:10:44:58 +0200] "POST
> /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
> "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
> _for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
> Gecko/20100101 Firefox/67.0"
> 172.22.24.119 - - [10/Jul/2019:10:45:01 +0200] "POST
> /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
> "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
> _for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
> Gecko/20100101 Firefox/67.0"
> 172.22.24.119 - - [10/Jul/2019:10:44:55 +0200] "POST
> /admin_toolshed/install_repositories HTTP/1.1" 500 -
> "
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install
> "
> "Moz
> illa/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
> *Error - <type 'exceptions.OSError'>: inotify instance limit reached*
> URL:
>
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/install_repositories
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/middleware/error.py',
> line 154 in __call__
>   app_iter = self.application(environ, sr_checker)
> File
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py',
> line 85 in __call__
>   return self.application(environ, start_response)
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py',
> line 640 in __call__
>   return self.application(environ, start_response)
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line 143 in
> __call__
>   return self.handle_request(environ, start_response)
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line 222 in
> handle_request
>   body = method(trans, **kwargs)
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/decorators.py', line
> 101 in decorator
>   return func(self, trans, *args, **kwargs)
> File
>
> '/opt/galaxy/galaxy/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
> line 575 in install_repositories
>   reinstalling=reinstalling,
> File
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
> line 846 in install_repositories
>   tool_panel_section_mapping=tool_panel_section_mapping)
> File
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
> line 893 in install_tool_shed_repository
>   tool_panel_section_mapping=tool_panel_section_mapping)
> File
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
> line 544 in __handle_repository_contents
>   repository_tools_tups = irmm.get_repository_tools_tups()
> File
>
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py',
> line 75 in get_repository_tools_tups
>   tool = self.app.toolbox.load_tool(os.path.abspath(load_relative_path),
> guid=guid, use_cached=False)
> File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line 766 in
> load_tool
>   self.watch_tool(tool)
> File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line 774 in
> watch_tool
>   self._tool_watcher.watch_file(tool.config_file, tool.id)
> File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py', line 198
> in watch_file
>   self.monitor(tool_dir)
> File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py', line 190
> in monitor
>   self.observer.schedule(self.event_handler, dir, recursive=False)
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/api.py',
> line 293 in schedule
>   emitter.start()
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/utils/__init__.py',
> line 110 in start
>   self.on_thread_start()
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify.py',
> line 121 in on_thread_start
>   self._inotify = InotifyBuffer(path, self.watch.is_recursive)
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_buffer.py',
> line 35 in __init__
>   self._inotify = Inotify(path, recursive)
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
> line 188 in __init__
>   Inotify._raise_error()
> *File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
> line 415 in _raise_error**
> **  raise OSError("inotify instance limit reached")**
> **OSError: inotify instance limit reached**
> *
> CGI Variables
> -------------
>   CONTENT_LENGTH: '12169'
>   CONTENT_TYPE: 'application/x-www-form-urlencoded; charset=UTF-8'
>   HTTP_ACCEPT: 'text/html, */*; q=0.01'
>   HTTP_ACCEPT_ENCODING: 'gzip,deflate'
>   HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
>   HTTP_CONNECTION: 'Keep-Alive'
>   HTTP_COOKIE:
>
> 'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'
>   HTTP_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de'
>   HTTP_REFERER:
> '
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install
> '
>   HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
> Gecko/20100101 Firefox/67.0'
>   HTTP_X_FORWARDED_FOR: '172.22.24.119'
>   HTTP_X_FORWARDED_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de'
>   HTTP_X_FORWARDED_SERVER: 'fe80::216:3eff:fe80:c02b'
>   HTTP_X_REQUESTED_WITH: 'XMLHttpRequest'
>   ORGINAL_HTTP_HOST: 'localhost:8080'
>   ORGINAL_REMOTE_ADDR: '127.0.0.1'
>   PATH_INFO: '/admin_toolshed/install_repositories'
>   REMOTE_ADDR: '172.22.24.119'
>   REQUEST_METHOD: 'POST'
>   SERVER_NAME: '127.0.0.1'
>   SERVER_PORT: '8080'
>   SERVER_PROTOCOL: 'HTTP/1.1'
>
>
> WSGI Variables
> --------------
>   application: <paste.recursive.RecursiveMiddleware object at
> 0x7f9f48207410>
>   controller_action_key: u'web.admin_toolshed.install_repositories'
>   is_api_request: False
>   paste.cookies: (<SimpleCookie:
>
> galaxysession='c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'>,
>
> 'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e')
>   paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'>]
>   paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler
> object at 0x7f9f48c86fd0>
>   paste.httpserver.proxy.host: 'dummy'
>   paste.httpserver.proxy.scheme: 'http'
>   paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at
> 0x7f9f486fcd90>
>   paste.recursive.forward: <paste.recursive.Forwarder from />
>   paste.recursive.include: <paste.recursive.Includer from />
>   paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from
> />
>   paste.recursive.script_name: ''
>   paste.throw_errors: True
>   request_id: 'fd8cc98ea2ee11e9a89100163e80c02b'
>   webob._body_file: (<_io.BufferedReader>, <socket._fileobject object at
> 0x7f9f3a4b5ad0 length=12169>)
>   webob._parsed_post_vars: (MultiDict([('tool_shed_repository_ids',
> u"[u'ac218635d795e337', u'e3e09dfaa8fad8c8', u'1efe9eb0bff40152',
> u'358d8704ef89cb6b', u'cbc5119f47881258']"), ('encoded_kwd',
>
> u'59b03a7799b953da9827424b9cef2236c5b9f461:7b22737461747573223a2022646f6e65222c20226861735f7265706f7369746f72795f646570656e64656e63696573223a2066616c73652c2022696e636c756465735f746f6f6c735f666f725f646973706c61795f696e5f746f6f6c5f70616e656c223a20747275652c2022746f6f6c5f736865645f7265706f7369746f72795f696473223a205b2261633231383633356437393565333337222c202265336530396466616138666164386338222c202231656665396562306266663430313532222c202233353864383730346566383963623662222c202263626335313139663437383831323538225d2c2022736865645f746f6f6c5f636f6e66223a20222e2f636f6e6669672f736865645f746f6f6c5f636f6e662e786d6c222c2022696e7374616c6c5f7265706f7369746f72795f646570656e64656e63696573223a20747275652c2022746f6f6c5f70617468223a20222f6f70742f67616c6178792f736865645f746f6f6c73222c20227265706f5f696e666f5f6469637473223a2...
> 0x7f9f35cdfed0>)
>   webob._parsed_query_vars: (GET([]), '')
>   webob.is_body_seekable: True
>   wsgi process: 'Multithreaded'
>   wsgi.input_terminated: True
>
>
>
>
> --
> *Dr. Christopher Previti*
> Genomics and Proteomics Core Facility
> High Throughput Sequencing (W190)
> Bioinformatician
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> Room: B2.102 (INF580/TP3)
> Phone: +49 6221 42-4661
>
> [hidden email] <http://www.dkfz.de/>
> www.dkfz.de <http://www.dkfz.de/>
>
> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> VAT-ID No.: DE143293537
>
> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> Personen bestimmt, an die sie adressiert ist.
> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> Informationen enthalten. Sollten Sie nicht
> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
> Absender und löschen Sie die Mitteilung.
> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> untersagt.
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Error during tool installation: "inotify instance limit reached"

Previti
Thank you very much!

We have 19.01 running and the parameter that I had to put to "false" was
"watch_tools"

All the best,
Christopher


On 7/10/19 12:00 PM, Marius van den Beek wrote:

> Hi Christopher,
>
> you might want to turn off `watch_tools` if you are using it,
> this is only needed when using the pretty-much undocumented
> feature where you dump tools into a directory without registering them
> in any tool_conf.xml file. All other tool reloading happens by another
> means.
>
> Similarly, if you're using 19.05 and you don't manipulate the tool
> data table
> entries manually you can disable `watch_tool_data_dir`.
>
> Finally this is usually a sign of allocating too many processes and
> threads for your webservers/handlers/mules,
> as each one starts their own watchdog. A common misconception is that
> this makes Galaxy run faster.
> but it actually just increases the amount of concurrent requests
> Galaxy can handle.
>
> Best,
> Marius
>
> On Wed, 10 Jul 2019 at 11:43, Previti
> <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Dear all,
>
>     I've been getting this error for a few days now, it basically prevents
>     anything from being installed on the first go.
>     Individual tools can be installed, but even then I have to try
>     multiple
>     times.
>
>     Any ideas how to fix this?
>
>     Cheers,
>     Christopher
>
>     The details from galaxy.log are (installing an older VCFtools
>     package):
>
>     172.22.24.119 - - [10/Jul/2019:10:44:32 +0200] "POST
>     /admin_toolshed/prepare_for_install HTTP/1.1" 200 -
>     "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?change
>     set_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
>     <http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?changeset_revisions=34a6b690e4b5&repository_ids=2ba42187f2588c4b&tool_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F>"
>     "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
>     Firefox/67.0"
>     galaxy.tools.toolbox.base DEBUG 2019-07-10 10:44:55,592 Appending to
>     tool panel section: NGS:VCFtools
>     172.22.24.119 - - [10/Jul/2019:10:44:58 +0200] "POST
>     /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
>     "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
>     _for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
>     Gecko/20100101 Firefox/67.0"
>     172.22.24.119 - - [10/Jul/2019:10:45:01 +0200] "POST
>     /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
>     "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
>     _for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
>     Gecko/20100101 Firefox/67.0"
>     172.22.24.119 - - [10/Jul/2019:10:44:55 +0200] "POST
>     /admin_toolshed/install_repositories HTTP/1.1" 500 -
>     "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install"
>     "Moz
>     illa/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
>     Firefox/67.0"
>     *Error - <type 'exceptions.OSError'>: inotify instance limit reached*
>     URL:
>     http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/install_repositories
>     File
>     '/opt/galaxy/galaxy/lib/galaxy/web/framework/middleware/error.py',
>     line 154 in __call__
>       app_iter = self.application(environ, sr_checker)
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py',
>     line 85 in __call__
>       return self.application(environ, start_response)
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py',
>     line 640 in __call__
>       return self.application(environ, start_response)
>     File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line
>     143 in
>     __call__
>       return self.handle_request(environ, start_response)
>     File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line
>     222 in
>     handle_request
>       body = method(trans, **kwargs)
>     File '/opt/galaxy/galaxy/lib/galaxy/web/framework/decorators.py', line
>     101 in decorator
>       return func(self, trans, *args, **kwargs)
>     File
>     '/opt/galaxy/galaxy/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
>     line 575 in install_repositories
>       reinstalling=reinstalling,
>     File
>     '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
>     line 846 in install_repositories
>       tool_panel_section_mapping=tool_panel_section_mapping)
>     File
>     '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
>     line 893 in install_tool_shed_repository
>       tool_panel_section_mapping=tool_panel_section_mapping)
>     File
>     '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
>     line 544 in __handle_repository_contents
>       repository_tools_tups = irmm.get_repository_tools_tups()
>     File
>     '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py',
>     line 75 in get_repository_tools_tups
>       tool =
>     self.app.toolbox.load_tool(os.path.abspath(load_relative_path),
>     guid=guid, use_cached=False)
>     File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line
>     766 in
>     load_tool
>       self.watch_tool(tool)
>     File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line
>     774 in
>     watch_tool
>       self._tool_watcher.watch_file(tool.config_file, tool.id
>     <http://tool.id>)
>     File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py',
>     line 198
>     in watch_file
>       self.monitor(tool_dir)
>     File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py',
>     line 190
>     in monitor
>       self.observer.schedule(self.event_handler, dir, recursive=False)
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/api.py',
>     line 293 in schedule
>       emitter.start()
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/utils/__init__.py',
>     line 110 in start
>       self.on_thread_start()
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify.py',
>     line 121 in on_thread_start
>       self._inotify = InotifyBuffer(path, self.watch.is_recursive)
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_buffer.py',
>     line 35 in __init__
>       self._inotify = Inotify(path, recursive)
>     File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
>     line 188 in __init__
>       Inotify._raise_error()
>     *File
>     '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/watchdog/observers/inotify_c.py',
>     line 415 in _raise_error**
>     **  raise OSError("inotify instance limit reached")**
>     **OSError: inotify instance limit reached**
>     *
>     CGI Variables
>     -------------
>       CONTENT_LENGTH: '12169'
>       CONTENT_TYPE: 'application/x-www-form-urlencoded; charset=UTF-8'
>       HTTP_ACCEPT: 'text/html, */*; q=0.01'
>       HTTP_ACCEPT_ENCODING: 'gzip,deflate'
>       HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.5'
>       HTTP_CONNECTION: 'Keep-Alive'
>       HTTP_COOKIE:
>     'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'
>       HTTP_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de
>     <http://dkfzgalaxy.inet.dkfz-heidelberg.de>'
>       HTTP_REFERER:
>     'http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install'
>       HTTP_USER_AGENT: 'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
>     Gecko/20100101 Firefox/67.0'
>       HTTP_X_FORWARDED_FOR: '172.22.24.119'
>       HTTP_X_FORWARDED_HOST: 'dkfzgalaxy.inet.dkfz-heidelberg.de
>     <http://dkfzgalaxy.inet.dkfz-heidelberg.de>'
>       HTTP_X_FORWARDED_SERVER: 'fe80::216:3eff:fe80:c02b'
>       HTTP_X_REQUESTED_WITH: 'XMLHttpRequest'
>       ORGINAL_HTTP_HOST: 'localhost:8080'
>       ORGINAL_REMOTE_ADDR: '127.0.0.1'
>       PATH_INFO: '/admin_toolshed/install_repositories'
>       REMOTE_ADDR: '172.22.24.119'
>       REQUEST_METHOD: 'POST'
>       SERVER_NAME: '127.0.0.1'
>       SERVER_PORT: '8080'
>       SERVER_PROTOCOL: 'HTTP/1.1'
>
>
>     WSGI Variables
>     --------------
>       application: <paste.recursive.RecursiveMiddleware object at
>     0x7f9f48207410>
>       controller_action_key: u'web.admin_toolshed.install_repositories'
>       is_api_request: False
>       paste.cookies: (<SimpleCookie:
>     galaxysession='c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e'>,
>     'galaxysession=c6ca0ddb55be603a9d01bef828912a83dd6a820b8a47ad278d223920b71ba8b43726aaf1140b008e')
>       paste.expected_exceptions: [<class
>     'paste.httpexceptions.HTTPException'>]
>       paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler
>     object at 0x7f9f48c86fd0>
>       paste.httpserver.proxy.host: 'dummy'
>       paste.httpserver.proxy.scheme: 'http'
>       paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object at
>     0x7f9f486fcd90>
>       paste.recursive.forward: <paste.recursive.Forwarder from />
>       paste.recursive.include: <paste.recursive.Includer from />
>       paste.recursive.include_app_iter:
>     <paste.recursive.IncluderAppIter from />
>       paste.recursive.script_name: ''
>       paste.throw_errors: True
>       request_id: 'fd8cc98ea2ee11e9a89100163e80c02b'
>       webob._body_file: (<_io.BufferedReader>, <socket._fileobject
>     object at
>     0x7f9f3a4b5ad0 length=12169>)
>       webob._parsed_post_vars: (MultiDict([('tool_shed_repository_ids',
>     u"[u'ac218635d795e337', u'e3e09dfaa8fad8c8', u'1efe9eb0bff40152',
>     u'358d8704ef89cb6b', u'cbc5119f47881258']"), ('encoded_kwd',
>     u'59b03a7799b953da9827424b9cef2236c5b9f461:7b22737461747573223a2022646f6e65222c20226861735f7265706f7369746f72795f646570656e64656e63696573223a2066616c73652c2022696e636c756465735f746f6f6c735f666f725f646973706c61795f696e5f746f6f6c5f70616e656c223a20747275652c2022746f6f6c5f736865645f7265706f7369746f72795f696473223a205b2261633231383633356437393565333337222c202265336530396466616138666164386338222c202231656665396562306266663430313532222c202233353864383730346566383963623662222c202263626335313139663437383831323538225d2c2022736865645f746f6f6c5f636f6e66223a20222e2f636f6e6669672f736865645f746f6f6c5f636f6e662e786d6c222c2022696e7374616c6c5f7265706f7369746f72795f646570656e64656e63696573223a20747275652c2022746f6f6c5f70617468223a20222f6f70742f67616c6178792f736865645f746f6f6c73222c20227265706f5f696e666f5f6469637473223a2...
>     0x7f9f35cdfed0>)
>       webob._parsed_query_vars: (GET([]), '')
>       webob.is_body_seekable: True
>       wsgi process: 'Multithreaded'
>       wsgi.input_terminated: True
>
>
>
>
>     --
>     *Dr. Christopher Previti*
>     Genomics and Proteomics Core Facility
>     High Throughput Sequencing (W190)
>     Bioinformatician
>
>     German Cancer Research Center (DKFZ)
>     Foundation under Public Law
>     Im Neuenheimer Feld 580
>     69120 Heidelberg
>     Germany
>     Room: B2.102 (INF580/TP3)
>     Phone: +49 6221 42-4661
>
>     [hidden email] <mailto:[hidden email]>
>     <http://www.dkfz.de/>
>     www.dkfz.de <http://www.dkfz.de> <http://www.dkfz.de/>
>
>     Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
>     VAT-ID No.: DE143293537
>
>     Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
>     Personen bestimmt, an die sie adressiert ist.
>     Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
>     Informationen enthalten. Sollten Sie nicht
>     der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
>     Absender und löschen Sie die Mitteilung.
>     Jegliche unbefugte Verwendung der Informationen in dieser
>     Nachricht ist
>     untersagt.
>
>
>     ___________________________________________________________
>     Please keep all replies on the list by using "reply all"
>     in your mail client.  To manage your subscriptions to this
>     and other Galaxy lists, please use the interface at:
>       %(web_page_url)s
>
>     To search Galaxy mailing lists use the unified search at:
>       http://galaxyproject.org/search/
>

--
*Dr. Christopher Previti*
Genomics and Proteomics Core Facility
High Throughput Sequencing (W190)
Bioinformatician

German Cancer Research Center (DKFZ)
Foundation under Public Law
Im Neuenheimer Feld 580
69120 Heidelberg
Germany
Room: B2.102 (INF580/TP3)
Phone: +49 6221 42-4661

[hidden email] <http://www.dkfz.de/>
www.dkfz.de <http://www.dkfz.de/>

Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
VAT-ID No.: DE143293537

Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
Personen bestimmt, an die sie adressiert ist.
Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
Informationen enthalten. Sollten Sie nicht
der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den
Absender und löschen Sie die Mitteilung.
Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
untersagt.


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/