upload problems

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

upload problems

Shrum, Donald C
Hi all,

I'm working with a problem with user uploaded files.  After digging through the logs a bit and running the commands on at a time manually I think I've narrowed it to a permissions problem.  This was confirmed by just running galaxy as root and the problem went away ;)

-bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/
-bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf         6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat
Traceback (most recent call last):
  File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module>
    __main__()
  File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__
    registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] )
  File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes
    tree = galaxy.util.parse_xml( config )
  File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml
    tree = ElementTree.parse(fname)
  File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse
  File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 576, in parse
IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf'


-bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
-rw------- 1 dcshrum dcshrum 317 Jun 10 16:30 /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf


It does not look like galaxy is using sudo to run the script.  Suggestions to work this out?



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: upload problems

John Chilton-4
You didn't include this context, but I am guessing you are attempting
to run jobs as the "real" user? If not, ignore the rest of the e-mail.

I would generally not recommend running the uploads as "real" user -
it is a complex process but should go relatively quick.

Understand that may not be possible though. So that file is the
integrated datatypes configuration file I believe. There is just one
global copy that gets created with Galaxy boots up - so it cannot be
chown-ed on a per job basis. The thing is that Galaxy should be
modifying it to be world readable
(https://bitbucket.org/galaxy/galaxy-central/annotate/e2b761a9b1d6d41db71b28df8b62862c7c300eba/lib/galaxy/datatypes/registry.py?at=default#cl-811)
- something is going wrong if it is not. Can you verify the file is
644?

That leads me to believe that users don't have read access to the
global temp directory. Can you check if users can read
/panfs/storage.local/scratch/galaxy-data/tmp/? I think they will need
to to use some tools including uploads?

If you cannot make this directory accessible to users - can you change
Galaxy's new_file_path so that it is some directory globally readable?

-John

On Tue, Jun 10, 2014 at 4:10 PM, Shrum, Donald C <[hidden email]> wrote:

> Hi all,
>
> I'm working with a problem with user uploaded files.  After digging through the logs a bit and running the commands on at a time manually I think I've narrowed it to a permissions problem.  This was confirmed by just running galaxy as root and the problem went away ;)
>
> -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/
> -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf         6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat
> Traceback (most recent call last):
>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module>
>     __main__()
>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__
>     registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] )
>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes
>     tree = galaxy.util.parse_xml( config )
>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml
>     tree = ElementTree.parse(fname)
>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse
>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 576, in parse
> IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf'
>
>
> -bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
> -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30 /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>
>
> It does not look like galaxy is using sudo to run the script.  Suggestions to work this out?
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: upload problems

Shrum, Donald C
Hi Jon and thanks for the reply.    I am attempting to run jobs as the real user as jobs will go to our HPC cluster.  This will be an enterprise server.  

/panfs/storage.local/scratch/galaxy-data/ is world writable-
drwxrwxrwx  4 galaxy      galaxy       4096 May  7 09:08 galaxy-data

as is tmp
-bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/
total 160
drwxrwxrwx 2 galaxy  galaxy  20480 Jun 10 17:00 tmp

I got a little lost on the integrated datatypes configuration file... is that an XML file?  I'm not sure which file I'm looking for and I'm new to galaxy.

--Donny

-----Original Message-----
From: John Chilton [mailto:[hidden email]]
Sent: Tuesday, June 10, 2014 5:33 PM
To: Shrum, Donald C
Cc: [hidden email]
Subject: Re: [galaxy-dev] upload problems

You didn't include this context, but I am guessing you are attempting to run jobs as the "real" user? If not, ignore the rest of the e-mail.

I would generally not recommend running the uploads as "real" user - it is a complex process but should go relatively quick.

Understand that may not be possible though. So that file is the integrated datatypes configuration file I believe. There is just one global copy that gets created with Galaxy boots up - so it cannot be chown-ed on a per job basis. The thing is that Galaxy should be modifying it to be world readable
(https://bitbucket.org/galaxy/galaxy-central/annotate/e2b761a9b1d6d41db71b28df8b62862c7c300eba/lib/galaxy/datatypes/registry.py?at=default#cl-811)
- something is going wrong if it is not. Can you verify the file is 644?

That leads me to believe that users don't have read access to the global temp directory. Can you check if users can read /panfs/storage.local/scratch/galaxy-data/tmp/? I think they will need to to use some tools including uploads?

If you cannot make this directory accessible to users - can you change Galaxy's new_file_path so that it is some directory globally readable?

-John

On Tue, Jun 10, 2014 at 4:10 PM, Shrum, Donald C <[hidden email]> wrote:

> Hi all,
>
> I'm working with a problem with user uploaded files.  After digging
> through the logs a bit and running the commands on at a time manually
> I think I've narrowed it to a permissions problem.  This was confirmed
> by just running galaxy as root and the problem went away ;)
>
> -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/
> -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf         6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat
> Traceback (most recent call last):
>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module>
>     __main__()
>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__
>     registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] )
>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes
>     tree = galaxy.util.parse_xml( config )
>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml
>     tree = ElementTree.parse(fname)
>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse
>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py",
> line 576, in parse
> IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf'
>
>
> -bash-4.1$ ls -l
> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
> -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30
> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>
>
> It does not look like galaxy is using sudo to run the script.  Suggestions to work this out?
>
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this and other
> Galaxy lists, please use the interface at:
>   http://lists.bx.psu.edu/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: upload problems

John Chilton-4
So the permissions on these files seem fine right. That is odd.

Next things I would check are that each directory leading up to
/panfs/storage.local/scratch/galaxy-data/ is world executable and that
the whole thing (/panfs/storage.local/scratch/galaxy-data/) is mounted
and with the same permissions on the cluster worker nodes.

If users can indeed read that file - the error message must be
misleading some how (that would not be entirely surprising).

-John

On Tue, Jun 10, 2014 at 4:48 PM, Shrum, Donald C <[hidden email]> wrote:

> Hi Jon and thanks for the reply.    I am attempting to run jobs as the real user as jobs will go to our HPC cluster.  This will be an enterprise server.
>
> /panfs/storage.local/scratch/galaxy-data/ is world writable-
> drwxrwxrwx  4 galaxy      galaxy       4096 May  7 09:08 galaxy-data
>
> as is tmp
> -bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/
> total 160
> drwxrwxrwx 2 galaxy  galaxy  20480 Jun 10 17:00 tmp
>
> I got a little lost on the integrated datatypes configuration file... is that an XML file?  I'm not sure which file I'm looking for and I'm new to galaxy.
>
> --Donny
>
> -----Original Message-----
> From: John Chilton [mailto:[hidden email]]
> Sent: Tuesday, June 10, 2014 5:33 PM
> To: Shrum, Donald C
> Cc: [hidden email]
> Subject: Re: [galaxy-dev] upload problems
>
> You didn't include this context, but I am guessing you are attempting to run jobs as the "real" user? If not, ignore the rest of the e-mail.
>
> I would generally not recommend running the uploads as "real" user - it is a complex process but should go relatively quick.
>
> Understand that may not be possible though. So that file is the integrated datatypes configuration file I believe. There is just one global copy that gets created with Galaxy boots up - so it cannot be chown-ed on a per job basis. The thing is that Galaxy should be modifying it to be world readable
> (https://bitbucket.org/galaxy/galaxy-central/annotate/e2b761a9b1d6d41db71b28df8b62862c7c300eba/lib/galaxy/datatypes/registry.py?at=default#cl-811)
> - something is going wrong if it is not. Can you verify the file is 644?
>
> That leads me to believe that users don't have read access to the global temp directory. Can you check if users can read /panfs/storage.local/scratch/galaxy-data/tmp/? I think they will need to to use some tools including uploads?
>
> If you cannot make this directory accessible to users - can you change Galaxy's new_file_path so that it is some directory globally readable?
>
> -John
>
> On Tue, Jun 10, 2014 at 4:10 PM, Shrum, Donald C <[hidden email]> wrote:
>> Hi all,
>>
>> I'm working with a problem with user uploaded files.  After digging
>> through the logs a bit and running the commands on at a time manually
>> I think I've narrowed it to a permissions problem.  This was confirmed
>> by just running galaxy as root and the problem went away ;)
>>
>> -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/
>> -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf         6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat
>> Traceback (most recent call last):
>>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module>
>>     __main__()
>>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__
>>     registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] )
>>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes
>>     tree = galaxy.util.parse_xml( config )
>>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml
>>     tree = ElementTree.parse(fname)
>>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse
>>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py",
>> line 576, in parse
>> IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf'
>>
>>
>> -bash-4.1$ ls -l
>> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>> -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30
>> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>>
>>
>> It does not look like galaxy is using sudo to run the script.  Suggestions to work this out?
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and other
>> Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: upload problems

Shrum, Donald C
Hi John,

Here is what I've tracked down this morning.  On the off chance this seems wildly off the mark I don't know much about python or galaxy.  I am drinking fancy coffee though ;)

I did confirm all the folders are world executable.  The upload is running as a local job, not a cluster job.  

I put some debugging code into galaxy-dist/lib/galaxy/datatypes/registry.py and the chmod in that code block is not called for the uploaded file.

I made an edit to scripts/external_chown_script.py
I added the line:
os.system('chmod 0644 %s' %(path))

that made the uploaded file world readable and pushed me to the next error that reads:
galaxy.jobs ERROR 2014-06-11 09:33:31,562 fail(): Missing output file in working directory:
[Errno 13] Permission denied: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/8/galaxy_dataset_8.dat'

It looks like the problem is with the permissions on the newly created folder (8 in this case)
-bash-4.1$ ls -l database/job_working_directory/000/
total 64
drw-r--r-- 2 galaxy galaxy 4096 Jun 11 09:33 8

The folder is does not have execute permissions turned on.  Could someone point me to the block of code where the mkdir is run?  Or of course if there is some setting I've missed some place please let me know!

Thanks for the help.
Donny
FSU RCC



-----Original Message-----
From: John Chilton [mailto:[hidden email]]
Sent: Wednesday, June 11, 2014 9:30 AM
To: Shrum, Donald C
Cc: [hidden email]
Subject: Re: [galaxy-dev] upload problems

So the permissions on these files seem fine right. That is odd.

Next things I would check are that each directory leading up to /panfs/storage.local/scratch/galaxy-data/ is world executable and that the whole thing (/panfs/storage.local/scratch/galaxy-data/) is mounted and with the same permissions on the cluster worker nodes.

If users can indeed read that file - the error message must be misleading some how (that would not be entirely surprising).

-John

On Tue, Jun 10, 2014 at 4:48 PM, Shrum, Donald C <[hidden email]> wrote:

> Hi Jon and thanks for the reply.    I am attempting to run jobs as the real user as jobs will go to our HPC cluster.  This will be an enterprise server.
>
> /panfs/storage.local/scratch/galaxy-data/ is world writable-
> drwxrwxrwx  4 galaxy      galaxy       4096 May  7 09:08 galaxy-data
>
> as is tmp
> -bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/
> total 160
> drwxrwxrwx 2 galaxy  galaxy  20480 Jun 10 17:00 tmp
>
> I got a little lost on the integrated datatypes configuration file... is that an XML file?  I'm not sure which file I'm looking for and I'm new to galaxy.
>
> --Donny
>
> -----Original Message-----
> From: John Chilton [mailto:[hidden email]]
> Sent: Tuesday, June 10, 2014 5:33 PM
> To: Shrum, Donald C
> Cc: [hidden email]
> Subject: Re: [galaxy-dev] upload problems
>
> You didn't include this context, but I am guessing you are attempting to run jobs as the "real" user? If not, ignore the rest of the e-mail.
>
> I would generally not recommend running the uploads as "real" user - it is a complex process but should go relatively quick.
>
> Understand that may not be possible though. So that file is the
> integrated datatypes configuration file I believe. There is just one
> global copy that gets created with Galaxy boots up - so it cannot be
> chown-ed on a per job basis. The thing is that Galaxy should be
> modifying it to be world readable
> (https://bitbucket.org/galaxy/galaxy-central/annotate/e2b761a9b1d6d41d
> b71b28df8b62862c7c300eba/lib/galaxy/datatypes/registry.py?at=default#c
> l-811)
> - something is going wrong if it is not. Can you verify the file is 644?
>
> That leads me to believe that users don't have read access to the global temp directory. Can you check if users can read /panfs/storage.local/scratch/galaxy-data/tmp/? I think they will need to to use some tools including uploads?
>
> If you cannot make this directory accessible to users - can you change Galaxy's new_file_path so that it is some directory globally readable?
>
> -John
>
> On Tue, Jun 10, 2014 at 4:10 PM, Shrum, Donald C <[hidden email]> wrote:
>> Hi all,
>>
>> I'm working with a problem with user uploaded files.  After digging
>> through the logs a bit and running the commands on at a time manually
>> I think I've narrowed it to a permissions problem.  This was
>> confirmed by just running galaxy as root and the problem went away ;)
>>
>> -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/
>> -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf         6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat
>> Traceback (most recent call last):
>>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module>
>>     __main__()
>>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__
>>     registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] )
>>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes
>>     tree = galaxy.util.parse_xml( config )
>>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml
>>     tree = ElementTree.parse(fname)
>>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse
>>   File
>> "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py",
>> line 576, in parse
>> IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf'
>>
>>
>> -bash-4.1$ ls -l
>> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>> -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30
>> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>>
>>
>> It does not look like galaxy is using sudo to run the script.  Suggestions to work this out?
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and other
>> Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: upload problems

Shrum, Donald C
In reply to this post by John Chilton-4
Derp... that is a result of the change I just made.  I'll see if I can add a line that checks to see if the uploaded file is a folder and sets it to 0755 if that is the case.  Seems like a security risk to just do a blanket 0755 and let people upload executable files.

If (path is a file)
        os.system('chmod 0644 %s' %(path))
else
        os.system('chmod 0755 %s' %(path))

-----Original Message-----
From: Shrum, Donald C
Sent: Wednesday, June 11, 2014 9:48 AM
To: 'John Chilton'
Cc: [hidden email]
Subject: RE: [galaxy-dev] upload problems

Hi John,

Here is what I've tracked down this morning.  On the off chance this seems wildly off the mark I don't know much about python or galaxy.  I am drinking fancy coffee though ;)

I did confirm all the folders are world executable.  The upload is running as a local job, not a cluster job.  

I put some debugging code into galaxy-dist/lib/galaxy/datatypes/registry.py and the chmod in that code block is not called for the uploaded file.

I made an edit to scripts/external_chown_script.py I added the line:
os.system('chmod 0644 %s' %(path))

that made the uploaded file world readable and pushed me to the next error that reads:
galaxy.jobs ERROR 2014-06-11 09:33:31,562 fail(): Missing output file in working directory:
[Errno 13] Permission denied: '/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/8/galaxy_dataset_8.dat'

It looks like the problem is with the permissions on the newly created folder (8 in this case) -bash-4.1$ ls -l database/job_working_directory/000/
total 64
drw-r--r-- 2 galaxy galaxy 4096 Jun 11 09:33 8

The folder is does not have execute permissions turned on.  Could someone point me to the block of code where the mkdir is run?  Or of course if there is some setting I've missed some place please let me know!

Thanks for the help.
Donny
FSU RCC



-----Original Message-----
From: John Chilton [mailto:[hidden email]]
Sent: Wednesday, June 11, 2014 9:30 AM
To: Shrum, Donald C
Cc: [hidden email]
Subject: Re: [galaxy-dev] upload problems

So the permissions on these files seem fine right. That is odd.

Next things I would check are that each directory leading up to /panfs/storage.local/scratch/galaxy-data/ is world executable and that the whole thing (/panfs/storage.local/scratch/galaxy-data/) is mounted and with the same permissions on the cluster worker nodes.

If users can indeed read that file - the error message must be misleading some how (that would not be entirely surprising).

-John

On Tue, Jun 10, 2014 at 4:48 PM, Shrum, Donald C <[hidden email]> wrote:

> Hi Jon and thanks for the reply.    I am attempting to run jobs as the real user as jobs will go to our HPC cluster.  This will be an enterprise server.
>
> /panfs/storage.local/scratch/galaxy-data/ is world writable-
> drwxrwxrwx  4 galaxy      galaxy       4096 May  7 09:08 galaxy-data
>
> as is tmp
> -bash-4.1$ ls -l /panfs/storage.local/scratch/galaxy-data/
> total 160
> drwxrwxrwx 2 galaxy  galaxy  20480 Jun 10 17:00 tmp
>
> I got a little lost on the integrated datatypes configuration file... is that an XML file?  I'm not sure which file I'm looking for and I'm new to galaxy.
>
> --Donny
>
> -----Original Message-----
> From: John Chilton [mailto:[hidden email]]
> Sent: Tuesday, June 10, 2014 5:33 PM
> To: Shrum, Donald C
> Cc: [hidden email]
> Subject: Re: [galaxy-dev] upload problems
>
> You didn't include this context, but I am guessing you are attempting to run jobs as the "real" user? If not, ignore the rest of the e-mail.
>
> I would generally not recommend running the uploads as "real" user - it is a complex process but should go relatively quick.
>
> Understand that may not be possible though. So that file is the
> integrated datatypes configuration file I believe. There is just one
> global copy that gets created with Galaxy boots up - so it cannot be
> chown-ed on a per job basis. The thing is that Galaxy should be
> modifying it to be world readable
> (https://bitbucket.org/galaxy/galaxy-central/annotate/e2b761a9b1d6d41d
> b71b28df8b62862c7c300eba/lib/galaxy/datatypes/registry.py?at=default#c
> l-811)
> - something is going wrong if it is not. Can you verify the file is 644?
>
> That leads me to believe that users don't have read access to the global temp directory. Can you check if users can read /panfs/storage.local/scratch/galaxy-data/tmp/? I think they will need to to use some tools including uploads?
>
> If you cannot make this directory accessible to users - can you change Galaxy's new_file_path so that it is some directory globally readable?
>
> -John
>
> On Tue, Jun 10, 2014 at 4:10 PM, Shrum, Donald C <[hidden email]> wrote:
>> Hi all,
>>
>> I'm working with a problem with user uploaded files.  After digging
>> through the logs a bit and running the commands on at a time manually
>> I think I've narrowed it to a permissions problem.  This was
>> confirmed by just running galaxy as root and the problem went away ;)
>>
>> -bash-4.1$ PYTHONPATH=/panfs/storage.local/software/galaxy-dist/lib/
>> -bash-4.1$ python /panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py /panfs/storage.local/software/galaxy-dist//panfs/storage.local/scratch/galaxy-data/tmp/tmpSuHquR /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf         6:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/dataset_6_files:/panfs/storage.local/software/galaxy-dist/database/job_working_directory/000/6/galaxy_dataset_6.dat
>> Traceback (most recent call last):
>>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 394, in <module>
>>     __main__()
>>   File "/panfs/storage.local/software/galaxy-dist/tools/data_source/upload.py", line 369, in __main__
>>     registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] )
>>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/datatypes/registry.py", line 97, in load_datatypes
>>     tree = galaxy.util.parse_xml( config )
>>   File "/panfs/storage.local/software/galaxy-dist/lib/galaxy/util/__init__.py", line 154, in parse_xml
>>     tree = ElementTree.parse(fname)
>>   File "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py", line 859, in parse
>>   File
>> "build/bdist.linux-x86_64-ucs4/egg/elementtree/ElementTree.py",
>> line 576, in parse
>> IOError: [Errno 13] Permission denied: '/panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf'
>>
>>
>> -bash-4.1$ ls -l
>> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>> -rw------- 1 dcshrum dcshrum 317 Jun 10 16:30
>> /panfs/storage.local/scratch/galaxy-data/tmp/tmpYGRnAf
>>
>>
>> It does not look like galaxy is using sudo to run the script.  Suggestions to work this out?
>>
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this and other
>> Galaxy lists, please use the interface at:
>>   http://lists.bx.psu.edu/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/