Issue with job_name, UnicodeDecodeError in drmaa

classic Classic list List threaded Threaded
8 messages Options
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Issue with job_name, UnicodeDecodeError in drmaa

Raj Ayyampalayam
Hello,

I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
I fixed all the issues with except one. I am getting the following error when I submit a job.

galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
Traceback (most recent call last):
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
    method(arg)
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
    job_name = self._job_name(job_wrapper)
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)

I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.

I would really appreciate some help from the community.

Thanks!
-Raj

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Peter Cock
Time for more debugging, try inserting something like this inserted
the line above into
/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
(having backed up the file):

log.debug( 'Making job_name, tag was %r, plan to use %r with
underscores replacements' % (galaxy_id_tag, job_name))

And if that doesn't help, check all the characters:

for i, letter in enumerate(job_name): log.debug( 'Making job_name,
character %i in proposed job_name %r is %i' % (i, letter,
ord(letter)))

My hunch is that the locale of the Galaxy server differs from that on
the cluster, meaning DRMAA is stricter about allowed characters
because Python's string.letters would be whatever is allowed on the
Galaxy server's locale.

If I am right, a crude fix would be switching to using
string.ascii_letters, i.e.

job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
'_' ) else '_' for x in job_name )

Peter

On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:

> Hello,
>
> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
> I fixed all the issues with except one. I am getting the following error when I submit a job.
>
> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
> Traceback (most recent call last):
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>     method(arg)
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>     job_name = self._job_name(job_wrapper)
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>     job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>     job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>
> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>
> I would really appreciate some help from the community.
>
> Thanks!
> -Raj
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Raj Ayyampalayam
Peter,

The job name looks OK. Here is the output of the debug statements:

galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, tag was '15132', plan to use u'[hidden email]' with underscores replacements
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 0 in proposed job_name u'g' is 103
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 1 in proposed job_name u'1' is 49
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 2 in proposed job_name u'5' is 53
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 3 in proposed job_name u'1' is 49
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 4 in proposed job_name u'3' is 51
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 5 in proposed job_name u'2' is 50
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 6 in proposed job_name u'_' is 95
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 7 in proposed job_name u'f' is 102
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 8 in proposed job_name u'a' is 97
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 9 in proposed job_name u's' is 115
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 10 in proposed job_name u't' is 116
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 11 in proposed job_name u'q' is 113
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 12 in proposed job_name u'_' is 95
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 13 in proposed job_name u't' is 116
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 14 in proposed job_name u'o' is 111
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 15 in proposed job_name u'_' is 95
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 16 in proposed job_name u'f' is 102
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 17 in proposed job_name u'a' is 97
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 18 in proposed job_name u's' is 115
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 19 in proposed job_name u't' is 116
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 20 in proposed job_name u'a' is 97
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 21 in proposed job_name u'_' is 95
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 22 in proposed job_name u'p' is 112
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 23 in proposed job_name u'y' is 121
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 24 in proposed job_name u't' is 116
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 25 in proposed job_name u'h' is 104
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 26 in proposed job_name u'o' is 111
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 27 in proposed job_name u'n' is 110
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 28 in proposed job_name u'_' is 95
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 29 in proposed job_name u'r' is 114
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 30 in proposed job_name u'a' is 97
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 31 in proposed job_name u'j' is 106
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 32 in proposed job_name u'7' is 55
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 33 in proposed job_name u'6' is 54
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 34 in proposed job_name u'@' is 64
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 35 in proposed job_name u'u' is 117
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 36 in proposed job_name u'g' is 103
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 37 in proposed job_name u'a' is 97
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 38 in proposed job_name u'.' is 46
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 39 in proposed job_name u'e' is 101
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 40 in proposed job_name u'd' is 100
galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 41 in proposed job_name u'u' is 117
galaxy.jobs.runners ERROR 2017-05-29 10:37:21,431 (15132) Unhandled exception calling queue_job
Traceback (most recent call last):
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
    method(arg)
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
    job_name = self._job_name(job_wrapper)
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in _job_name
    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in <genexpr>
    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)


There is no sign of the 0xa6 at position 52 (the job name is only 42 chars long) in the job name.

Thanks!
-Raj


> On May 29, 2017, at 5:50 AM, Peter Cock <[hidden email]> wrote:
>
> Time for more debugging, try inserting something like this inserted
> the line above into
> /panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
> (having backed up the file):
>
> log.debug( 'Making job_name, tag was %r, plan to use %r with
> underscores replacements' % (galaxy_id_tag, job_name))
>
> And if that doesn't help, check all the characters:
>
> for i, letter in enumerate(job_name): log.debug( 'Making job_name,
> character %i in proposed job_name %r is %i' % (i, letter,
> ord(letter)))
>
> My hunch is that the locale of the Galaxy server differs from that on
> the cluster, meaning DRMAA is stricter about allowed characters
> because Python's string.letters would be whatever is allowed on the
> Galaxy server's locale.
>
> If I am right, a crude fix would be switching to using
> string.ascii_letters, i.e.
>
> job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
> '_' ) else '_' for x in job_name )
>
> Peter
>
> On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>> Hello,
>>
>> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
>> I fixed all the issues with except one. I am getting the following error when I submit a job.
>>
>> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
>> Traceback (most recent call last):
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>    method(arg)
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>    job_name = self._job_name(job_wrapper)
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>
>> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>>
>> I would really appreciate some help from the community.
>>
>> Thanks!
>> -Raj
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Peter Cock
Curious, my new guess is this is position 52 a temporary string from
this expression:

string.letters + string.digits + '_'

This would fit with it being something unusual about the locale of
your Galaxy server, and my suggested hack may work - replace
string.letters with string.ascii_letters

Separately at the Linux command line of the server running Galaxy, and
on a cluster node, what does the command "locale" give? e.g. I get:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=


Peter

On Mon, May 29, 2017 at 3:43 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:

> Peter,
>
> The job name looks OK. Here is the output of the debug statements:
>
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, tag was '15132', plan to use u'[hidden email]' with underscores replacements
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 0 in proposed job_name u'g' is 103
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 1 in proposed job_name u'1' is 49
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 2 in proposed job_name u'5' is 53
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 3 in proposed job_name u'1' is 49
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 4 in proposed job_name u'3' is 51
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 5 in proposed job_name u'2' is 50
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 6 in proposed job_name u'_' is 95
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 7 in proposed job_name u'f' is 102
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 8 in proposed job_name u'a' is 97
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 9 in proposed job_name u's' is 115
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 10 in proposed job_name u't' is 116
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 11 in proposed job_name u'q' is 113
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 12 in proposed job_name u'_' is 95
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 13 in proposed job_name u't' is 116
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 14 in proposed job_name u'o' is 111
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 15 in proposed job_name u'_' is 95
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 16 in proposed job_name u'f' is 102
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 17 in proposed job_name u'a' is 97
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 18 in proposed job_name u's' is 115
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 19 in proposed job_name u't' is 116
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 20 in proposed job_name u'a' is 97
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 21 in proposed job_name u'_' is 95
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 22 in proposed job_name u'p' is 112
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 23 in proposed job_name u'y' is 121
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 24 in proposed job_name u't' is 116
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 25 in proposed job_name u'h' is 104
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 26 in proposed job_name u'o' is 111
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 27 in proposed job_name u'n' is 110
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 28 in proposed job_name u'_' is 95
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 29 in proposed job_name u'r' is 114
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 30 in proposed job_name u'a' is 97
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 31 in proposed job_name u'j' is 106
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 32 in proposed job_name u'7' is 55
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 33 in proposed job_name u'6' is 54
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 34 in proposed job_name u'@' is 64
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 35 in proposed job_name u'u' is 117
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 36 in proposed job_name u'g' is 103
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 37 in proposed job_name u'a' is 97
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 38 in proposed job_name u'.' is 46
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 39 in proposed job_name u'e' is 101
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 40 in proposed job_name u'd' is 100
> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 41 in proposed job_name u'u' is 117
> galaxy.jobs.runners ERROR 2017-05-29 10:37:21,431 (15132) Unhandled exception calling queue_job
> Traceback (most recent call last):
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>     method(arg)
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>     job_name = self._job_name(job_wrapper)
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in _job_name
>     job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>   File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in <genexpr>
>     job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>
>
> There is no sign of the 0xa6 at position 52 (the job name is only 42 chars long) in the job name.
>
> Thanks!
> -Raj
>
>
>> On May 29, 2017, at 5:50 AM, Peter Cock <[hidden email]> wrote:
>>
>> Time for more debugging, try inserting something like this inserted
>> the line above into
>> /panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
>> (having backed up the file):
>>
>> log.debug( 'Making job_name, tag was %r, plan to use %r with
>> underscores replacements' % (galaxy_id_tag, job_name))
>>
>> And if that doesn't help, check all the characters:
>>
>> for i, letter in enumerate(job_name): log.debug( 'Making job_name,
>> character %i in proposed job_name %r is %i' % (i, letter,
>> ord(letter)))
>>
>> My hunch is that the locale of the Galaxy server differs from that on
>> the cluster, meaning DRMAA is stricter about allowed characters
>> because Python's string.letters would be whatever is allowed on the
>> Galaxy server's locale.
>>
>> If I am right, a crude fix would be switching to using
>> string.ascii_letters, i.e.
>>
>> job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
>> '_' ) else '_' for x in job_name )
>>
>> Peter
>>
>> On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>> Hello,
>>>
>>> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
>>> I fixed all the issues with except one. I am getting the following error when I submit a job.
>>>
>>> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
>>> Traceback (most recent call last):
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>    method(arg)
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>    job_name = self._job_name(job_wrapper)
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>
>>> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>>>
>>> I would really appreciate some help from the community.
>>>
>>> Thanks!
>>> -Raj
>>>
>>> ___________________________________________________________
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>  https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>  http://galaxyproject.org/search/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Raj Ayyampalayam
Peter,

You are correct. Changing string.letters to string.ascii_letters fixed it. I had to change it in two places.

My locale is :

LANG=en_US.iso885915
LC_CTYPE="en_US.iso885915"
LC_NUMERIC="en_US.iso885915"
LC_TIME="en_US.iso885915"
LC_COLLATE="en_US.iso885915"
LC_MONETARY="en_US.iso885915"
LC_MESSAGES="en_US.iso885915"
LC_PAPER="en_US.iso885915"
LC_NAME="en_US.iso885915"
LC_ADDRESS="en_US.iso885915"
LC_TELEPHONE="en_US.iso885915"
LC_MEASUREMENT="en_US.iso885915"
LC_IDENTIFICATION="en_US.iso885915"
LC_ALL=

I will talk to our sysadmin and see if I can change the locale to en_US.UTF-8.

Thank you very much.
-Raj


> On May 29, 2017, at 10:58 AM, Peter Cock <[hidden email]> wrote:
>
> Curious, my new guess is this is position 52 a temporary string from
> this expression:
>
> string.letters + string.digits + '_'
>
> This would fit with it being something unusual about the locale of
> your Galaxy server, and my suggested hack may work - replace
> string.letters with string.ascii_letters
>
> Separately at the Linux command line of the server running Galaxy, and
> on a cluster node, what does the command "locale" give? e.g. I get:
>
> $ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC="en_US.UTF-8"
> LC_TIME="en_US.UTF-8"
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY="en_US.UTF-8"
> LC_MESSAGES="en_US.UTF-8"
> LC_PAPER="en_US.UTF-8"
> LC_NAME="en_US.UTF-8"
> LC_ADDRESS="en_US.UTF-8"
> LC_TELEPHONE="en_US.UTF-8"
> LC_MEASUREMENT="en_US.UTF-8"
> LC_IDENTIFICATION="en_US.UTF-8"
> LC_ALL=
>
>
> Peter
>
> On Mon, May 29, 2017 at 3:43 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>> Peter,
>>
>> The job name looks OK. Here is the output of the debug statements:
>>
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, tag was '15132', plan to use u'[hidden email]' with underscores replacements
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 0 in proposed job_name u'g' is 103
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 1 in proposed job_name u'1' is 49
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 2 in proposed job_name u'5' is 53
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 3 in proposed job_name u'1' is 49
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 4 in proposed job_name u'3' is 51
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 5 in proposed job_name u'2' is 50
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 6 in proposed job_name u'_' is 95
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 7 in proposed job_name u'f' is 102
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 8 in proposed job_name u'a' is 97
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 9 in proposed job_name u's' is 115
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 10 in proposed job_name u't' is 116
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 11 in proposed job_name u'q' is 113
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 12 in proposed job_name u'_' is 95
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 13 in proposed job_name u't' is 116
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 14 in proposed job_name u'o' is 111
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 15 in proposed job_name u'_' is 95
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 16 in proposed job_name u'f' is 102
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 17 in proposed job_name u'a' is 97
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 18 in proposed job_name u's' is 115
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 19 in proposed job_name u't' is 116
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 20 in proposed job_name u'a' is 97
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 21 in proposed job_name u'_' is 95
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 22 in proposed job_name u'p' is 112
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 23 in proposed job_name u'y' is 121
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 24 in proposed job_name u't' is 116
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 25 in proposed job_name u'h' is 104
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 26 in proposed job_name u'o' is 111
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 27 in proposed job_name u'n' is 110
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 28 in proposed job_name u'_' is 95
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 29 in proposed job_name u'r' is 114
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 30 in proposed job_name u'a' is 97
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 31 in proposed job_name u'j' is 106
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 32 in proposed job_name u'7' is 55
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 33 in proposed job_name u'6' is 54
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 34 in proposed job_name u'@' is 64
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 35 in proposed job_name u'u' is 117
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 36 in proposed job_name u'g' is 103
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 37 in proposed job_name u'a' is 97
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 38 in proposed job_name u'.' is 46
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 39 in proposed job_name u'e' is 101
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 40 in proposed job_name u'd' is 100
>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 41 in proposed job_name u'u' is 117
>> galaxy.jobs.runners ERROR 2017-05-29 10:37:21,431 (15132) Unhandled exception calling queue_job
>> Traceback (most recent call last):
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>    method(arg)
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>    job_name = self._job_name(job_wrapper)
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in _job_name
>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in <genexpr>
>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>
>>
>> There is no sign of the 0xa6 at position 52 (the job name is only 42 chars long) in the job name.
>>
>> Thanks!
>> -Raj
>>
>>
>>> On May 29, 2017, at 5:50 AM, Peter Cock <[hidden email]> wrote:
>>>
>>> Time for more debugging, try inserting something like this inserted
>>> the line above into
>>> /panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
>>> (having backed up the file):
>>>
>>> log.debug( 'Making job_name, tag was %r, plan to use %r with
>>> underscores replacements' % (galaxy_id_tag, job_name))
>>>
>>> And if that doesn't help, check all the characters:
>>>
>>> for i, letter in enumerate(job_name): log.debug( 'Making job_name,
>>> character %i in proposed job_name %r is %i' % (i, letter,
>>> ord(letter)))
>>>
>>> My hunch is that the locale of the Galaxy server differs from that on
>>> the cluster, meaning DRMAA is stricter about allowed characters
>>> because Python's string.letters would be whatever is allowed on the
>>> Galaxy server's locale.
>>>
>>> If I am right, a crude fix would be switching to using
>>> string.ascii_letters, i.e.
>>>
>>> job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
>>> '_' ) else '_' for x in job_name )
>>>
>>> Peter
>>>
>>> On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>>> Hello,
>>>>
>>>> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
>>>> I fixed all the issues with except one. I am getting the following error when I submit a job.
>>>>
>>>> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
>>>> Traceback (most recent call last):
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>>   method(arg)
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>>   job_name = self._job_name(job_wrapper)
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>>
>>>> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>>>>
>>>> I would really appreciate some help from the community.
>>>>
>>>> Thanks!
>>>> -Raj
>>>>
>>>> ___________________________________________________________
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>> https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>> http://galaxyproject.org/search/
>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Peter Cock
Thanks Saravanaraj,

You should not need to change the locale, after all many Galaxy servers and
their users will be labelling their files in non-English with
"special" characters -
but perhaps en_US.UTF-8 would be safer in your case as it seems that few
people use Galaxy with en_US.iso885915 otherwise this would have been
reported before?

Can you tell us both places you changed string.letters to string.ascii_letters
please? I could then submit this as a pull request to get Galaxy itself updated
for future release.

Peter

On Mon, May 29, 2017 at 4:28 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:

> Peter,
>
> You are correct. Changing string.letters to string.ascii_letters fixed it. I had to change it in two places.
>
> My locale is :
>
> LANG=en_US.iso885915
> LC_CTYPE="en_US.iso885915"
> LC_NUMERIC="en_US.iso885915"
> LC_TIME="en_US.iso885915"
> LC_COLLATE="en_US.iso885915"
> LC_MONETARY="en_US.iso885915"
> LC_MESSAGES="en_US.iso885915"
> LC_PAPER="en_US.iso885915"
> LC_NAME="en_US.iso885915"
> LC_ADDRESS="en_US.iso885915"
> LC_TELEPHONE="en_US.iso885915"
> LC_MEASUREMENT="en_US.iso885915"
> LC_IDENTIFICATION="en_US.iso885915"
> LC_ALL=
>
> I will talk to our sysadmin and see if I can change the locale to en_US.UTF-8.
>
> Thank you very much.
> -Raj
>
>
>> On May 29, 2017, at 10:58 AM, Peter Cock <[hidden email]> wrote:
>>
>> Curious, my new guess is this is position 52 a temporary string from
>> this expression:
>>
>> string.letters + string.digits + '_'
>>
>> This would fit with it being something unusual about the locale of
>> your Galaxy server, and my suggested hack may work - replace
>> string.letters with string.ascii_letters
>>
>> Separately at the Linux command line of the server running Galaxy, and
>> on a cluster node, what does the command "locale" give? e.g. I get:
>>
>> $ locale
>> LANG=en_US.UTF-8
>> LC_CTYPE="en_US.UTF-8"
>> LC_NUMERIC="en_US.UTF-8"
>> LC_TIME="en_US.UTF-8"
>> LC_COLLATE="en_US.UTF-8"
>> LC_MONETARY="en_US.UTF-8"
>> LC_MESSAGES="en_US.UTF-8"
>> LC_PAPER="en_US.UTF-8"
>> LC_NAME="en_US.UTF-8"
>> LC_ADDRESS="en_US.UTF-8"
>> LC_TELEPHONE="en_US.UTF-8"
>> LC_MEASUREMENT="en_US.UTF-8"
>> LC_IDENTIFICATION="en_US.UTF-8"
>> LC_ALL=
>>
>>
>> Peter
>>
>> On Mon, May 29, 2017 at 3:43 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>> Peter,
>>>
>>> The job name looks OK. Here is the output of the debug statements:
>>>
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, tag was '15132', plan to use u'[hidden email]' with underscores replacements
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 0 in proposed job_name u'g' is 103
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 1 in proposed job_name u'1' is 49
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 2 in proposed job_name u'5' is 53
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 3 in proposed job_name u'1' is 49
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 4 in proposed job_name u'3' is 51
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 5 in proposed job_name u'2' is 50
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 6 in proposed job_name u'_' is 95
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 7 in proposed job_name u'f' is 102
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 8 in proposed job_name u'a' is 97
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 9 in proposed job_name u's' is 115
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 10 in proposed job_name u't' is 116
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 11 in proposed job_name u'q' is 113
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 12 in proposed job_name u'_' is 95
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 13 in proposed job_name u't' is 116
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 14 in proposed job_name u'o' is 111
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 15 in proposed job_name u'_' is 95
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 16 in proposed job_name u'f' is 102
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 17 in proposed job_name u'a' is 97
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 18 in proposed job_name u's' is 115
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 19 in proposed job_name u't' is 116
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 20 in proposed job_name u'a' is 97
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 21 in proposed job_name u'_' is 95
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 22 in proposed job_name u'p' is 112
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 23 in proposed job_name u'y' is 121
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 24 in proposed job_name u't' is 116
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 25 in proposed job_name u'h' is 104
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 26 in proposed job_name u'o' is 111
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 27 in proposed job_name u'n' is 110
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 28 in proposed job_name u'_' is 95
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 29 in proposed job_name u'r' is 114
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 30 in proposed job_name u'a' is 97
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 31 in proposed job_name u'j' is 106
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 32 in proposed job_name u'7' is 55
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 33 in proposed job_name u'6' is 54
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 34 in proposed job_name u'@' is 64
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 35 in proposed job_name u'u' is 117
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 36 in proposed job_name u'g' is 103
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 37 in proposed job_name u'a' is 97
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 38 in proposed job_name u'.' is 46
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 39 in proposed job_name u'e' is 101
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 40 in proposed job_name u'd' is 100
>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 41 in proposed job_name u'u' is 117
>>> galaxy.jobs.runners ERROR 2017-05-29 10:37:21,431 (15132) Unhandled exception calling queue_job
>>> Traceback (most recent call last):
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>    method(arg)
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>    job_name = self._job_name(job_wrapper)
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in _job_name
>>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>  File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in <genexpr>
>>>    job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>
>>>
>>> There is no sign of the 0xa6 at position 52 (the job name is only 42 chars long) in the job name.
>>>
>>> Thanks!
>>> -Raj
>>>
>>>
>>>> On May 29, 2017, at 5:50 AM, Peter Cock <[hidden email]> wrote:
>>>>
>>>> Time for more debugging, try inserting something like this inserted
>>>> the line above into
>>>> /panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
>>>> (having backed up the file):
>>>>
>>>> log.debug( 'Making job_name, tag was %r, plan to use %r with
>>>> underscores replacements' % (galaxy_id_tag, job_name))
>>>>
>>>> And if that doesn't help, check all the characters:
>>>>
>>>> for i, letter in enumerate(job_name): log.debug( 'Making job_name,
>>>> character %i in proposed job_name %r is %i' % (i, letter,
>>>> ord(letter)))
>>>>
>>>> My hunch is that the locale of the Galaxy server differs from that on
>>>> the cluster, meaning DRMAA is stricter about allowed characters
>>>> because Python's string.letters would be whatever is allowed on the
>>>> Galaxy server's locale.
>>>>
>>>> If I am right, a crude fix would be switching to using
>>>> string.ascii_letters, i.e.
>>>>
>>>> job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
>>>> '_' ) else '_' for x in job_name )
>>>>
>>>> Peter
>>>>
>>>> On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>>>> Hello,
>>>>>
>>>>> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
>>>>> I fixed all the issues with except one. I am getting the following error when I submit a job.
>>>>>
>>>>> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
>>>>> Traceback (most recent call last):
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>>>   method(arg)
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>>>   job_name = self._job_name(job_wrapper)
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>>>
>>>>> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>>>>>
>>>>> I would really appreciate some help from the community.
>>>>>
>>>>> Thanks!
>>>>> -Raj
>>>>>
>>>>> ___________________________________________________________
>>>>> Please keep all replies on the list by using "reply all"
>>>>> in your mail client.  To manage your subscriptions to this
>>>>> and other Galaxy lists, please use the interface at:
>>>>> https://lists.galaxyproject.org/
>>>>>
>>>>> To search Galaxy mailing lists use the unified search at:
>>>>> http://galaxyproject.org/search/
>>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Raj Ayyampalayam
Peter,

Here is the git diff output that shows the changes I made:

diff --git a/lib/galaxy/jobs/runners/__init__.py b/lib/galaxy/jobs/runners/__init__.py
index ebf4859..6ee51c8 100644
--- a/lib/galaxy/jobs/runners/__init__.py
+++ b/lib/galaxy/jobs/runners/__init__.py
@@ -419,7 +419,7 @@ class JobState( object ):
                 job_name += '_%s' % self.job_wrapper.tool.old_id
             if self.job_wrapper.user:
                 job_name += '_%s' % self.job_wrapper.user
-            self.job_name = ''.join( map( lambda x: x if x in ( string.letters + string.digits + '_' ) else '_', job_name ) )
+            self.job_name = ''.join( map( lambda x: x if x in ( string.ascii_letters + string.digits + '_' ) else '_', job_name ) )

     @staticmethod
     def default_job_file( files_dir, id_tag ):
diff --git a/lib/galaxy/jobs/runners/drmaa.py b/lib/galaxy/jobs/runners/drmaa.py
index 7c08984..5c5dc62 100644
--- a/lib/galaxy/jobs/runners/drmaa.py
+++ b/lib/galaxy/jobs/runners/drmaa.py
@@ -394,7 +394,7 @@ class DRMAAJobRunner( AsynchronousJobRunner ):
             job_name += '_%s' % job_wrapper.tool.old_id
         if external_runjob_script is None:
             job_name += '_%s' % job_wrapper.user
-        job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
+        job_name = ''.join( x if x in ( string.ascii_letters + string.digits + '_' ) else '_' for x in job_name )
         if self.restrict_job_name_length:
             job_name = job_name[:self.restrict_job_name_length]
         return job_name

-Raj

> On May 29, 2017, at 11:33 AM, Peter Cock <[hidden email]> wrote:
>
> Thanks Saravanaraj,
>
> You should not need to change the locale, after all many Galaxy servers and
> their users will be labelling their files in non-English with
> "special" characters -
> but perhaps en_US.UTF-8 would be safer in your case as it seems that few
> people use Galaxy with en_US.iso885915 otherwise this would have been
> reported before?
>
> Can you tell us both places you changed string.letters to string.ascii_letters
> please? I could then submit this as a pull request to get Galaxy itself updated
> for future release.
>
> Peter
>
> On Mon, May 29, 2017 at 4:28 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>> Peter,
>>
>> You are correct. Changing string.letters to string.ascii_letters fixed it. I had to change it in two places.
>>
>> My locale is :
>>
>> LANG=en_US.iso885915
>> LC_CTYPE="en_US.iso885915"
>> LC_NUMERIC="en_US.iso885915"
>> LC_TIME="en_US.iso885915"
>> LC_COLLATE="en_US.iso885915"
>> LC_MONETARY="en_US.iso885915"
>> LC_MESSAGES="en_US.iso885915"
>> LC_PAPER="en_US.iso885915"
>> LC_NAME="en_US.iso885915"
>> LC_ADDRESS="en_US.iso885915"
>> LC_TELEPHONE="en_US.iso885915"
>> LC_MEASUREMENT="en_US.iso885915"
>> LC_IDENTIFICATION="en_US.iso885915"
>> LC_ALL=
>>
>> I will talk to our sysadmin and see if I can change the locale to en_US.UTF-8.
>>
>> Thank you very much.
>> -Raj
>>
>>
>>> On May 29, 2017, at 10:58 AM, Peter Cock <[hidden email]> wrote:
>>>
>>> Curious, my new guess is this is position 52 a temporary string from
>>> this expression:
>>>
>>> string.letters + string.digits + '_'
>>>
>>> This would fit with it being something unusual about the locale of
>>> your Galaxy server, and my suggested hack may work - replace
>>> string.letters with string.ascii_letters
>>>
>>> Separately at the Linux command line of the server running Galaxy, and
>>> on a cluster node, what does the command "locale" give? e.g. I get:
>>>
>>> $ locale
>>> LANG=en_US.UTF-8
>>> LC_CTYPE="en_US.UTF-8"
>>> LC_NUMERIC="en_US.UTF-8"
>>> LC_TIME="en_US.UTF-8"
>>> LC_COLLATE="en_US.UTF-8"
>>> LC_MONETARY="en_US.UTF-8"
>>> LC_MESSAGES="en_US.UTF-8"
>>> LC_PAPER="en_US.UTF-8"
>>> LC_NAME="en_US.UTF-8"
>>> LC_ADDRESS="en_US.UTF-8"
>>> LC_TELEPHONE="en_US.UTF-8"
>>> LC_MEASUREMENT="en_US.UTF-8"
>>> LC_IDENTIFICATION="en_US.UTF-8"
>>> LC_ALL=
>>>
>>>
>>> Peter
>>>
>>> On Mon, May 29, 2017 at 3:43 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>>> Peter,
>>>>
>>>> The job name looks OK. Here is the output of the debug statements:
>>>>
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, tag was '15132', plan to use u'[hidden email]' with underscores replacements
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 0 in proposed job_name u'g' is 103
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 1 in proposed job_name u'1' is 49
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 2 in proposed job_name u'5' is 53
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 3 in proposed job_name u'1' is 49
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 4 in proposed job_name u'3' is 51
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 5 in proposed job_name u'2' is 50
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 6 in proposed job_name u'_' is 95
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 7 in proposed job_name u'f' is 102
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 8 in proposed job_name u'a' is 97
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 9 in proposed job_name u's' is 115
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 10 in proposed job_name u't' is 116
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 11 in proposed job_name u'q' is 113
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 12 in proposed job_name u'_' is 95
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 13 in proposed job_name u't' is 116
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 14 in proposed job_name u'o' is 111
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 15 in proposed job_name u'_' is 95
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 16 in proposed job_name u'f' is 102
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 17 in proposed job_name u'a' is 97
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 18 in proposed job_name u's' is 115
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 19 in proposed job_name u't' is 116
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 20 in proposed job_name u'a' is 97
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 21 in proposed job_name u'_' is 95
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 22 in proposed job_name u'p' is 112
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 23 in proposed job_name u'y' is 121
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 24 in proposed job_name u't' is 116
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 25 in proposed job_name u'h' is 104
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 26 in proposed job_name u'o' is 111
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 27 in proposed job_name u'n' is 110
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 28 in proposed job_name u'_' is 95
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 29 in proposed job_name u'r' is 114
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 30 in proposed job_name u'a' is 97
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 31 in proposed job_name u'j' is 106
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 32 in proposed job_name u'7' is 55
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 33 in proposed job_name u'6' is 54
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 34 in proposed job_name u'@' is 64
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 35 in proposed job_name u'u' is 117
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 36 in proposed job_name u'g' is 103
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 37 in proposed job_name u'a' is 97
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 38 in proposed job_name u'.' is 46
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 39 in proposed job_name u'e' is 101
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 40 in proposed job_name u'd' is 100
>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 41 in proposed job_name u'u' is 117
>>>> galaxy.jobs.runners ERROR 2017-05-29 10:37:21,431 (15132) Unhandled exception calling queue_job
>>>> Traceback (most recent call last):
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>>   method(arg)
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>>   job_name = self._job_name(job_wrapper)
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in _job_name
>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in <genexpr>
>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>>
>>>>
>>>> There is no sign of the 0xa6 at position 52 (the job name is only 42 chars long) in the job name.
>>>>
>>>> Thanks!
>>>> -Raj
>>>>
>>>>
>>>>> On May 29, 2017, at 5:50 AM, Peter Cock <[hidden email]> wrote:
>>>>>
>>>>> Time for more debugging, try inserting something like this inserted
>>>>> the line above into
>>>>> /panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
>>>>> (having backed up the file):
>>>>>
>>>>> log.debug( 'Making job_name, tag was %r, plan to use %r with
>>>>> underscores replacements' % (galaxy_id_tag, job_name))
>>>>>
>>>>> And if that doesn't help, check all the characters:
>>>>>
>>>>> for i, letter in enumerate(job_name): log.debug( 'Making job_name,
>>>>> character %i in proposed job_name %r is %i' % (i, letter,
>>>>> ord(letter)))
>>>>>
>>>>> My hunch is that the locale of the Galaxy server differs from that on
>>>>> the cluster, meaning DRMAA is stricter about allowed characters
>>>>> because Python's string.letters would be whatever is allowed on the
>>>>> Galaxy server's locale.
>>>>>
>>>>> If I am right, a crude fix would be switching to using
>>>>> string.ascii_letters, i.e.
>>>>>
>>>>> job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
>>>>> '_' ) else '_' for x in job_name )
>>>>>
>>>>> Peter
>>>>>
>>>>> On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
>>>>>> I fixed all the issues with except one. I am getting the following error when I submit a job.
>>>>>>
>>>>>> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
>>>>>> Traceback (most recent call last):
>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>>>>  method(arg)
>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>>>>  job_name = self._job_name(job_wrapper)
>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>>>>>>  job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>>>>>>  job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>>>>
>>>>>> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>>>>>>
>>>>>> I would really appreciate some help from the community.
>>>>>>
>>>>>> Thanks!
>>>>>> -Raj
>>>>>>
>>>>>> ___________________________________________________________
>>>>>> Please keep all replies on the list by using "reply all"
>>>>>> in your mail client.  To manage your subscriptions to this
>>>>>> and other Galaxy lists, please use the interface at:
>>>>>> https://lists.galaxyproject.org/
>>>>>>
>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>> http://galaxyproject.org/search/
>>>>
>>

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Issue with job_name, UnicodeDecodeError in drmaa

Peter Cock
Thanks, pull request submitted:

https://github.com/galaxyproject/galaxy/pull/4121

Peter


On Mon, May 29, 2017 at 5:23 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:

> Peter,
>
> Here is the git diff output that shows the changes I made:
>
> diff --git a/lib/galaxy/jobs/runners/__init__.py b/lib/galaxy/jobs/runners/__init__.py
> index ebf4859..6ee51c8 100644
> --- a/lib/galaxy/jobs/runners/__init__.py
> +++ b/lib/galaxy/jobs/runners/__init__.py
> @@ -419,7 +419,7 @@ class JobState( object ):
>                  job_name += '_%s' % self.job_wrapper.tool.old_id
>              if self.job_wrapper.user:
>                  job_name += '_%s' % self.job_wrapper.user
> -            self.job_name = ''.join( map( lambda x: x if x in ( string.letters + string.digits + '_' ) else '_', job_name ) )
> +            self.job_name = ''.join( map( lambda x: x if x in ( string.ascii_letters + string.digits + '_' ) else '_', job_name ) )
>
>      @staticmethod
>      def default_job_file( files_dir, id_tag ):
> diff --git a/lib/galaxy/jobs/runners/drmaa.py b/lib/galaxy/jobs/runners/drmaa.py
> index 7c08984..5c5dc62 100644
> --- a/lib/galaxy/jobs/runners/drmaa.py
> +++ b/lib/galaxy/jobs/runners/drmaa.py
> @@ -394,7 +394,7 @@ class DRMAAJobRunner( AsynchronousJobRunner ):
>              job_name += '_%s' % job_wrapper.tool.old_id
>          if external_runjob_script is None:
>              job_name += '_%s' % job_wrapper.user
> -        job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
> +        job_name = ''.join( x if x in ( string.ascii_letters + string.digits + '_' ) else '_' for x in job_name )
>          if self.restrict_job_name_length:
>              job_name = job_name[:self.restrict_job_name_length]
>          return job_name
>
> -Raj
>
>> On May 29, 2017, at 11:33 AM, Peter Cock <[hidden email]> wrote:
>>
>> Thanks Saravanaraj,
>>
>> You should not need to change the locale, after all many Galaxy servers and
>> their users will be labelling their files in non-English with
>> "special" characters -
>> but perhaps en_US.UTF-8 would be safer in your case as it seems that few
>> people use Galaxy with en_US.iso885915 otherwise this would have been
>> reported before?
>>
>> Can you tell us both places you changed string.letters to string.ascii_letters
>> please? I could then submit this as a pull request to get Galaxy itself updated
>> for future release.
>>
>> Peter
>>
>> On Mon, May 29, 2017 at 4:28 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>> Peter,
>>>
>>> You are correct. Changing string.letters to string.ascii_letters fixed it. I had to change it in two places.
>>>
>>> My locale is :
>>>
>>> LANG=en_US.iso885915
>>> LC_CTYPE="en_US.iso885915"
>>> LC_NUMERIC="en_US.iso885915"
>>> LC_TIME="en_US.iso885915"
>>> LC_COLLATE="en_US.iso885915"
>>> LC_MONETARY="en_US.iso885915"
>>> LC_MESSAGES="en_US.iso885915"
>>> LC_PAPER="en_US.iso885915"
>>> LC_NAME="en_US.iso885915"
>>> LC_ADDRESS="en_US.iso885915"
>>> LC_TELEPHONE="en_US.iso885915"
>>> LC_MEASUREMENT="en_US.iso885915"
>>> LC_IDENTIFICATION="en_US.iso885915"
>>> LC_ALL=
>>>
>>> I will talk to our sysadmin and see if I can change the locale to en_US.UTF-8.
>>>
>>> Thank you very much.
>>> -Raj
>>>
>>>
>>>> On May 29, 2017, at 10:58 AM, Peter Cock <[hidden email]> wrote:
>>>>
>>>> Curious, my new guess is this is position 52 a temporary string from
>>>> this expression:
>>>>
>>>> string.letters + string.digits + '_'
>>>>
>>>> This would fit with it being something unusual about the locale of
>>>> your Galaxy server, and my suggested hack may work - replace
>>>> string.letters with string.ascii_letters
>>>>
>>>> Separately at the Linux command line of the server running Galaxy, and
>>>> on a cluster node, what does the command "locale" give? e.g. I get:
>>>>
>>>> $ locale
>>>> LANG=en_US.UTF-8
>>>> LC_CTYPE="en_US.UTF-8"
>>>> LC_NUMERIC="en_US.UTF-8"
>>>> LC_TIME="en_US.UTF-8"
>>>> LC_COLLATE="en_US.UTF-8"
>>>> LC_MONETARY="en_US.UTF-8"
>>>> LC_MESSAGES="en_US.UTF-8"
>>>> LC_PAPER="en_US.UTF-8"
>>>> LC_NAME="en_US.UTF-8"
>>>> LC_ADDRESS="en_US.UTF-8"
>>>> LC_TELEPHONE="en_US.UTF-8"
>>>> LC_MEASUREMENT="en_US.UTF-8"
>>>> LC_IDENTIFICATION="en_US.UTF-8"
>>>> LC_ALL=
>>>>
>>>>
>>>> Peter
>>>>
>>>> On Mon, May 29, 2017 at 3:43 PM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>>>> Peter,
>>>>>
>>>>> The job name looks OK. Here is the output of the debug statements:
>>>>>
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, tag was '15132', plan to use u'[hidden email]' with underscores replacements
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 0 in proposed job_name u'g' is 103
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 1 in proposed job_name u'1' is 49
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,423 Making job_name, character 2 in proposed job_name u'5' is 53
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 3 in proposed job_name u'1' is 49
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 4 in proposed job_name u'3' is 51
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 5 in proposed job_name u'2' is 50
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 6 in proposed job_name u'_' is 95
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,424 Making job_name, character 7 in proposed job_name u'f' is 102
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 8 in proposed job_name u'a' is 97
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 9 in proposed job_name u's' is 115
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 10 in proposed job_name u't' is 116
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 11 in proposed job_name u'q' is 113
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,425 Making job_name, character 12 in proposed job_name u'_' is 95
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 13 in proposed job_name u't' is 116
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 14 in proposed job_name u'o' is 111
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 15 in proposed job_name u'_' is 95
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 16 in proposed job_name u'f' is 102
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 17 in proposed job_name u'a' is 97
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,426 Making job_name, character 18 in proposed job_name u's' is 115
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 19 in proposed job_name u't' is 116
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 20 in proposed job_name u'a' is 97
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 21 in proposed job_name u'_' is 95
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 22 in proposed job_name u'p' is 112
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 23 in proposed job_name u'y' is 121
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,427 Making job_name, character 24 in proposed job_name u't' is 116
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 25 in proposed job_name u'h' is 104
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 26 in proposed job_name u'o' is 111
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 27 in proposed job_name u'n' is 110
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 28 in proposed job_name u'_' is 95
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,428 Making job_name, character 29 in proposed job_name u'r' is 114
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 30 in proposed job_name u'a' is 97
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 31 in proposed job_name u'j' is 106
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 32 in proposed job_name u'7' is 55
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 33 in proposed job_name u'6' is 54
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,429 Making job_name, character 34 in proposed job_name u'@' is 64
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 35 in proposed job_name u'u' is 117
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 36 in proposed job_name u'g' is 103
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 37 in proposed job_name u'a' is 97
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 38 in proposed job_name u'.' is 46
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,430 Making job_name, character 39 in proposed job_name u'e' is 101
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 40 in proposed job_name u'd' is 100
>>>>> galaxy.jobs.runners.drmaa DEBUG 2017-05-29 10:37:21,431 Making job_name, character 41 in proposed job_name u'u' is 117
>>>>> galaxy.jobs.runners ERROR 2017-05-29 10:37:21,431 (15132) Unhandled exception calling queue_job
>>>>> Traceback (most recent call last):
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>>>   method(arg)
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>>>   job_name = self._job_name(job_wrapper)
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in _job_name
>>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 400, in <genexpr>
>>>>>   job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>>>
>>>>>
>>>>> There is no sign of the 0xa6 at position 52 (the job name is only 42 chars long) in the job name.
>>>>>
>>>>> Thanks!
>>>>> -Raj
>>>>>
>>>>>
>>>>>> On May 29, 2017, at 5:50 AM, Peter Cock <[hidden email]> wrote:
>>>>>>
>>>>>> Time for more debugging, try inserting something like this inserted
>>>>>> the line above into
>>>>>> /panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py
>>>>>> (having backed up the file):
>>>>>>
>>>>>> log.debug( 'Making job_name, tag was %r, plan to use %r with
>>>>>> underscores replacements' % (galaxy_id_tag, job_name))
>>>>>>
>>>>>> And if that doesn't help, check all the characters:
>>>>>>
>>>>>> for i, letter in enumerate(job_name): log.debug( 'Making job_name,
>>>>>> character %i in proposed job_name %r is %i' % (i, letter,
>>>>>> ord(letter)))
>>>>>>
>>>>>> My hunch is that the locale of the Galaxy server differs from that on
>>>>>> the cluster, meaning DRMAA is stricter about allowed characters
>>>>>> because Python's string.letters would be whatever is allowed on the
>>>>>> Galaxy server's locale.
>>>>>>
>>>>>> If I am right, a crude fix would be switching to using
>>>>>> string.ascii_letters, i.e.
>>>>>>
>>>>>> job_name = ''.join( x if x in ( string.ascii_letters + string.digits +
>>>>>> '_' ) else '_' for x in job_name )
>>>>>>
>>>>>> Peter
>>>>>>
>>>>>> On Mon, May 29, 2017 at 6:54 AM, Saravanaraj Ayyampalayam <[hidden email]> wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I updated to the latest version (17.05) galaxy code. I was updating from a very old version.
>>>>>>> I fixed all the issues with except one. I am getting the following error when I submit a job.
>>>>>>>
>>>>>>> galaxy.jobs.runners ERROR 2017-05-29 01:42:44,008 (15131) Unhandled exception calling queue_job
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
>>>>>>>  method(arg)
>>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 132, in queue_job
>>>>>>>  job_name = self._job_name(job_wrapper)
>>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in _job_name
>>>>>>>  job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>>>> File "/panfs/pstor.storage/home/qbcglab/galaxy_run/galaxy-dist/lib/galaxy/jobs/runners/drmaa.py", line 397, in <genexpr>
>>>>>>>  job_name = ''.join( x if x in ( string.letters + string.digits + '_' ) else '_' for x in job_name )
>>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa6 in position 52: ordinal not in range(128)
>>>>>>>
>>>>>>> I spent a day trying to debug this and didn’t get anywhere. The job_name looks OK.
>>>>>>>
>>>>>>> I would really appreciate some help from the community.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> -Raj
>>>>>>>
>>>>>>> ___________________________________________________________
>>>>>>> Please keep all replies on the list by using "reply all"
>>>>>>> in your mail client.  To manage your subscriptions to this
>>>>>>> and other Galaxy lists, please use the interface at:
>>>>>>> https://lists.galaxyproject.org/
>>>>>>>
>>>>>>> To search Galaxy mailing lists use the unified search at:
>>>>>>> http://galaxyproject.org/search/
>>>>>
>>>
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
Loading...