upload file format is data - type not recognized in galaxy

classic Classic list List threaded Threaded
6 messages Options
| Threaded
Open this post in threaded view
|

upload file format is data - type not recognized in galaxy

Pavel Fibich
Hello,

do you have any idea why my galaxy 16.07 stop recognizing uploaded files and format of uploaded files is set to the format 'data'?

After using older version of database backup, galaxy stop recognizing format of uploaded files (e.g. fasta, fastqc) and everything is set to format 'data'. Uploaded files are OK, I have tested them on other public galaxy servers, where they were recognized well. Before changing database (after the technical problems with the server), recognizing of the files were OK. Even in debugging level I do not see any errors or problems in the log (I have compared logs before changing database with new ones, and there are no differences). From the investigating of upload.py, I guess something happen with galaxy.datatypes.registry (settings and code of galaxy were not changed). Do you have any ideas? I cannot switch the database back and changing file format in Edit attributes is annoying.

Best, Pavel

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: upload file format is data - type not recognized in galaxy

Hans-Rudolf Hotz
Hi Pavel


what happens if you connect your galaxy instance to an empty PostgreSQL
database, let galaxy build all the tables and then try to upload files.


Regards, Hans-Rudolf



On 09/22/2017 10:51 AM, Pavel Fibich wrote:

> Hello,
>
> do you have any idea why my galaxy 16.07 stop recognizing uploaded files
> and format of uploaded files is set to the format 'data'?
>
> After using older version of database backup, galaxy stop recognizing
> format of uploaded files (e.g. fasta, fastqc) and everything is set to
> format 'data'. Uploaded files are OK, I have tested them on other public
> galaxy servers, where they were recognized well. Before changing
> database (after the technical problems with the server), recognizing of
> the files were OK. Even in debugging level I do not see any errors or
> problems in the log (I have compared logs before changing database with
> new ones, and there are no differences). From the investigating of
> upload.py, I guess something happen with galaxy.datatypes.registry
> (settings and code of galaxy were not changed). Do you have any ideas? I
> cannot switch the database back and changing file format in Edit
> attributes is annoying.
>
> Best, Pavel
>
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>    https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>    http://galaxyproject.org/search/
>
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: upload file format is data - type not recognized in galaxy

Peter Cock
My first thought would be to double check the Galaxy startup logs
(which normally tell you about the datatypes as they are setup).
That ought to give some clues.

Also check the XML files where the datatypes are defined are
all still fine - do you have any of these and do they look fine?

datatypes_conf.xml
config/datatypes_conf.xml
config/datatypes_conf.xml.sample

(I know you said the settings and code were not changed,
but it seems worth double checking)

Peter

On Fri, Sep 22, 2017 at 12:31 PM, Hans-Rudolf Hotz <[hidden email]> wrote:

> Hi Pavel
>
>
> what happens if you connect your galaxy instance to an empty PostgreSQL
> database, let galaxy build all the tables and then try to upload files.
>
>
> Regards, Hans-Rudolf
>
>
>
>
> On 09/22/2017 10:51 AM, Pavel Fibich wrote:
>>
>> Hello,
>>
>> do you have any idea why my galaxy 16.07 stop recognizing uploaded files
>> and format of uploaded files is set to the format 'data'?
>>
>> After using older version of database backup, galaxy stop recognizing
>> format of uploaded files (e.g. fasta, fastqc) and everything is set to
>> format 'data'. Uploaded files are OK, I have tested them on other public
>> galaxy servers, where they were recognized well. Before changing
>> database (after the technical problems with the server), recognizing of
>> the files were OK. Even in debugging level I do not see any errors or
>> problems in the log (I have compared logs before changing database with
>> new ones, and there are no differences). From the investigating of
>> upload.py, I guess something happen with galaxy.datatypes.registry
>> (settings and code of galaxy were not changed). Do you have any ideas? I
>> cannot switch the database back and changing file format in Edit
>> attributes is annoying.
>>
>> Best, Pavel
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>    https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>    http://galaxyproject.org/search/
>>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: upload file format is data - type not recognized in galaxy

Pavel Fibich
Hi Hans-Rudolf and Peter,

thank you for replies. I tried both suggestions, but no luck.

I made empty database, let galaxy run from it. Than, uploading fasta file again finished in the format 'data', not recognized again.

Also, I did not find anything interesting in the startup logs ( logs are here: https://repeatexplorer-elixir.cerit-sc.cz/galstart2.log ), there are just fine information about datatypes, eg. for fasta:

galaxy.datatypes.registry DEBUG 2017-09-22 15:11:52,810 Retrieved datatype module galaxy.datatypes.sequence:Fasta from the datatype registry.
...
galaxy.datatypes.registry DEBUG 2017-09-22 15:11:52,929 Loaded sniffer for datatype 'galaxy.datatypes.sequence:Fasta'

I have only config/datatypes_conf.xml.sample, so I expect that it is default.

If you have any other ideas, I will be glad for them.

Best, Pavel

On Fri, Sep 22, 2017 at 2:25 PM, Peter Cock <[hidden email]> wrote:
My first thought would be to double check the Galaxy startup logs
(which normally tell you about the datatypes as they are setup).
That ought to give some clues.

Also check the XML files where the datatypes are defined are
all still fine - do you have any of these and do they look fine?

datatypes_conf.xml
config/datatypes_conf.xml
config/datatypes_conf.xml.sample

(I know you said the settings and code were not changed,
but it seems worth double checking)

Peter

On Fri, Sep 22, 2017 at 12:31 PM, Hans-Rudolf Hotz <[hidden email]> wrote:
> Hi Pavel
>
>
> what happens if you connect your galaxy instance to an empty PostgreSQL
> database, let galaxy build all the tables and then try to upload files.
>
>
> Regards, Hans-Rudolf
>
>
>
>
> On 09/22/2017 10:51 AM, Pavel Fibich wrote:
>>
>> Hello,
>>
>> do you have any idea why my galaxy 16.07 stop recognizing uploaded files
>> and format of uploaded files is set to the format 'data'?
>>
>> After using older version of database backup, galaxy stop recognizing
>> format of uploaded files (e.g. fasta, fastqc) and everything is set to
>> format 'data'. Uploaded files are OK, I have tested them on other public
>> galaxy servers, where they were recognized well. Before changing
>> database (after the technical problems with the server), recognizing of
>> the files were OK. Even in debugging level I do not see any errors or
>> problems in the log (I have compared logs before changing database with
>> new ones, and there are no differences). From the investigating of
>> upload.py, I guess something happen with galaxy.datatypes.registry
>> (settings and code of galaxy were not changed). Do you have any ideas? I
>> cannot switch the database back and changing file format in Edit
>> attributes is annoying.
>>
>> Best, Pavel

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: upload file format is data - type not recognized in galaxy

Peter Cock
That's a shame - I don't see anything in the log either at
first reading - it was hoping to see problems with missing
datatypes or sniffers, which might explain this.

Sorry, I am out of ideas.

Peter

On Fri, Sep 22, 2017 at 2:41 PM, Pavel Fibich <[hidden email]> wrote:

> Hi Hans-Rudolf and Peter,
>
> thank you for replies. I tried both suggestions, but no luck.
>
> I made empty database, let galaxy run from it. Than, uploading fasta file
> again finished in the format 'data', not recognized again.
>
> Also, I did not find anything interesting in the startup logs ( logs are
> here: https://repeatexplorer-elixir.cerit-sc.cz/galstart2.log ), there are
> just fine information about datatypes, eg. for fasta:
>
> galaxy.datatypes.registry DEBUG 2017-09-22 15:11:52,810 Retrieved datatype
> module galaxy.datatypes.sequence:Fasta from the datatype registry.
> ...
> galaxy.datatypes.registry DEBUG 2017-09-22 15:11:52,929 Loaded sniffer for
> datatype 'galaxy.datatypes.sequence:Fasta'
>
> I have only config/datatypes_conf.xml.sample, so I expect that it is
> default.
>
> If you have any other ideas, I will be glad for them.
>
> Best, Pavel
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/
| Threaded
Open this post in threaded view
|

Re: upload file format is data - type not recognized in galaxy

Pavel Fibich
I got the solution by debugging upload.py. Recently, I tried to force galaxy to run jobs in SCRATCHDIR (directory in the local disks of computer), so I put cd $SCRATCHDIR at the end of the environmental settings file. Generated tool script contains line like

python .../upload.py .../registry.xml ./files..

but before that environmental settings are applied (including cd $SCRATCHDIR). upload.py is using absolute paths except json_file = open( 'galaxy.json', 'w' ) that breaks further recognition of the uploaded file (I guess due to calling dataset = loads( line ), where loads is from json library). Removing cd command from the settings solved the problem.

Thank you for all suggestions.

Best regards, Pavel

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/