Galaxy moves files form job directory but attempts to look for them at original path

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Galaxy moves files form job directory but attempts to look for them at original path

Evan Clark-2
Below is an error obtained from our galaxy install. We have custom
directories sets for the files directory and the jobs directory. Galaxy
appears to move the files but then attempts to look for them at the
original path. I have tried unsetting these options but it still persists.

galaxy.jobs DEBUG 2017-11-24 17:53:19,057 finish(): Moved
/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat
to /mnt/ceph/home/galaxy/galaxy/files_directory/000/dataset_401.dat
galaxy.model.metadata DEBUG 2017-11-24 17:53:19,128 setting metadata
externally failed for HistoryDatasetAssociation 431: [Errno 2] No such
file or directory
galaxy.jobs.runners ERROR 2017-11-24 17:53:19,319 (320/179084) Job
wrapper finish method failed
Traceback (most recent call last):
   File "/var/web_services/galaxy/lib/galaxy/jobs/runners/__init__.py",
line 630, in finish_job
     job_state.job_wrapper.finish( stdout, stderr, exit_code )
   File "/var/web_services/galaxy/lib/galaxy/jobs/__init__.py", line
1266, in finish
     dataset.datatype.set_meta( dataset, overwrite=False )
   File "/var/web_services/galaxy/lib/galaxy/datatypes/binary.py", line
391, in set_meta
     exit_code = subprocess.call( args=command, stderr=open(
stderr_name, 'wb' ) )
   File "/usr/lib64/python2.7/subprocess.py", line 524, in call
     return Popen(*popenargs, **kwargs).wait()
   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
     errread, errwrite)
   File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.jobs ERROR 2017-11-24 17:53:19,337 fail(): Missing output file in
working directory: [Errno 2] No such file or directory:
'/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat'
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Reply | Threaded
Open this post in threaded view
|

Re: Galaxy moves files form job directory but attempts to look for them at original path

John Chilton-4
Well that is no good - I take it you have outputs_to_working_directory
set to True in your galaxy.ini? That option doesn't get a lot of
testing and it seems like it is interfering with
retry_metadata_internally = True. What version of Galaxy are you
running? Another work around would be to try to figure out why you
need to retry_metadata_internally to start with - it shouldn't be
needed - in this case it looks like maybe samtools wasn't available on
the cluster where this ran. Putting that on the Galaxy PATH might help
- I'm not sure.

I've created a Github issue here
(https://github.com/galaxyproject/galaxy/issues/5097). Sorry about
this.

-John

On Fri, Nov 24, 2017 at 5:57 PM, Evan Clark <[hidden email]> wrote:

> Below is an error obtained from our galaxy install. We have custom
> directories sets for the files directory and the jobs directory. Galaxy
> appears to move the files but then attempts to look for them at the original
> path. I have tried unsetting these options but it still persists.
>
> galaxy.jobs DEBUG 2017-11-24 17:53:19,057 finish(): Moved
> /var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat
> to /mnt/ceph/home/galaxy/galaxy/files_directory/000/dataset_401.dat
> galaxy.model.metadata DEBUG 2017-11-24 17:53:19,128 setting metadata
> externally failed for HistoryDatasetAssociation 431: [Errno 2] No such file
> or directory
> galaxy.jobs.runners ERROR 2017-11-24 17:53:19,319 (320/179084) Job wrapper
> finish method failed
> Traceback (most recent call last):
>   File "/var/web_services/galaxy/lib/galaxy/jobs/runners/__init__.py", line
> 630, in finish_job
>     job_state.job_wrapper.finish( stdout, stderr, exit_code )
>   File "/var/web_services/galaxy/lib/galaxy/jobs/__init__.py", line 1266, in
> finish
>     dataset.datatype.set_meta( dataset, overwrite=False )
>   File "/var/web_services/galaxy/lib/galaxy/datatypes/binary.py", line 391,
> in set_meta
>     exit_code = subprocess.call( args=command, stderr=open( stderr_name,
> 'wb' ) )
>   File "/usr/lib64/python2.7/subprocess.py", line 524, in call
>     return Popen(*popenargs, **kwargs).wait()
>   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
>     errread, errwrite)
>   File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
>     raise child_exception
> OSError: [Errno 2] No such file or directory
> galaxy.jobs ERROR 2017-11-24 17:53:19,337 fail(): Missing output file in
> working directory: [Errno 2] No such file or directory:
> '/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat'
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/