simple example of composite data types

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

simple example of composite data types

Brian Haas
Greetings.

Does anyone per chance have a simple working example of using composite data types?   I've gotten as far as I can with the galaxy documentation, and I can't seem to get the additional files bundled into an extra files directory.

(note, I have my example submitted w/ files attached, but currently pending  moderator approval)

much appreciated,

~brian

--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Peter Cock
On Sun, May 17, 2015 at 5:44 PM, Brian Haas <[hidden email]> wrote:

> Greetings.
>
> Does anyone per chance have a simple working example of using composite data
> types?   I've gotten as far as I can with the galaxy documentation, and I
> can't seem to get the additional files bundled into an extra files
> directory.
>
> (note, I have my example submitted w/ files attached, but currently pending
> moderator approval)
>
> much appreciated,
>
> ~brian

Hi Brian,

I can suggest looking at the BLAST database datatypes defined here:
https://github.com/peterjc/galaxy_blast/tree/master/datatypes/blast_datatypes

Also study the built in examples like HTML (with images as extra files):
https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/data.py

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
Thanks, Peter!   I've verified that I can get your code to work - bundling the composite files.  I'll keep plugging away at my little example and see if I can figure out what's going awry.

more later.  Thx again!

~brian


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Peter Cock
On Mon, May 18, 2015 at 1:55 PM, Brian Haas <[hidden email]> wrote:
> Thanks, Peter!   I've verified that I can get your code to work - bundling
> the composite files.  I'll keep plugging away at my little example and see
> if I can figure out what's going awry.
>
> more later.  Thx again!

Good luck - compound datatypes are one of the complex ways to
extend Galaxy. If your code was online in GitHub or BitBucket, it
would be easier for us (including the Galaxy team) to comment on.

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
In the next couple of hours, if I'm not able to resolve this, I'll post my ultra-simple example on github and I'll be grateful for others to take a look.

cheers,

~brian

On Mon, May 18, 2015 at 9:23 AM, Peter Cock <[hidden email]> wrote:
On Mon, May 18, 2015 at 1:55 PM, Brian Haas <[hidden email]> wrote:
> Thanks, Peter!   I've verified that I can get your code to work - bundling
> the composite files.  I'll keep plugging away at my little example and see
> if I can figure out what's going awry.
>
> more later.  Thx again!

Good luck - compound datatypes are one of the complex ways to
extend Galaxy. If your code was online in GitHub or BitBucket, it
would be easier for us (including the Galaxy team) to comment on.

Regards,

Peter



--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Peter Cock
Hi Brian,

Your original email with attachments has made it to the list now,
http://dev.list.galaxyproject.org/difficulty-creating-composite-data-type-tp4667193.html

Title: "difficulty creating composite data type"

Peter

On Mon, May 18, 2015 at 2:25 PM, Brian Haas <[hidden email]> wrote:
> In the next couple of hours, if I'm not able to resolve this, I'll post my
> ultra-simple example on github and I'll be grateful for others to take a
> look.
>
> cheers,
>
> ~brian
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
Fantastic!

I've actually made a few changes.   I'm putting my latest code up on github now and will share it in just a moment.   It'll be much easier this way - as you suggested.

best,

~brian


On Mon, May 18, 2015 at 9:58 AM, Peter Cock <[hidden email]> wrote:
Hi Brian,

Your original email with attachments has made it to the list now,
http://dev.list.galaxyproject.org/difficulty-creating-composite-data-type-tp4667193.html

Title: "difficulty creating composite data type"

Peter

On Mon, May 18, 2015 at 2:25 PM, Brian Haas <[hidden email]> wrote:
> In the next couple of hours, if I'm not able to resolve this, I'll post my
> ultra-simple example on github and I'll be grateful for others to take a
> look.
>
> cheers,
>
> ~brian



--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
I've uploaded my simple example here:


I can't seem to get it to bundle the output files no matter what I've tried.  Help is much appreciated. :)

best,

~brian


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
Hi Peter,

I'm actually more confused now that I was when I started...  I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file.  I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area.   I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored.   I then make 'composite_type = None' and it still stored the files to my dismay.

Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.  

Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.

I'm clearly not grasping something fundamental here.  I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;) 

Any ideas on where to go to troubleshoot this further?

many thanks,

~brian


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Daniel Blankenberg
Hi Brian,

In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path.
Then you create that directory and place the files you want within that directory.



Thanks for using Galaxy,

Dan


On May 18, 2015, at 2:02 PM, Brian Haas <[hidden email]> wrote:

> Hi Peter,
>
> I'm actually more confused now that I was when I started...  I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file.  I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area.   I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored.   I then make 'composite_type = None' and it still stored the files to my dismay.
>
> Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.  
>
> Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
>
> I'm clearly not grasping something fundamental here.  I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
>
> Any ideas on where to go to troubleshoot this further?
>
> many thanks,
>
> ~brian
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
THANK YOU!!! :)

The fundamental piece that I was missing at:

was the 

<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} 
${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>

It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.

This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).

thanks again!

~brian




On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg <[hidden email]> wrote:
Hi Brian,

In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path.
Then you create that directory and place the files you want within that directory.



Thanks for using Galaxy,

Dan


On May 18, 2015, at 2:02 PM, Brian Haas <[hidden email]> wrote:

> Hi Peter,
>
> I'm actually more confused now that I was when I started...  I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file.  I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area.   I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored.   I then make 'composite_type = None' and it still stored the files to my dismay.
>
> Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
>
> Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
>
> I'm clearly not grasping something fundamental here.  I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
>
> Any ideas on where to go to troubleshoot this further?
>
> many thanks,
>
> ~brian
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Martin Čech
It is a wiki, if you miss something - please add it.

Thanks

Martin

On Mon, May 18, 2015 at 3:08 PM Brian Haas <[hidden email]> wrote:
THANK YOU!!! :)

The fundamental piece that I was missing at:

was the 

<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} 
${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>

It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.

This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).

thanks again!

~brian




On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg <[hidden email]> wrote:
Hi Brian,

In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path.
Then you create that directory and place the files you want within that directory.



Thanks for using Galaxy,

Dan


On May 18, 2015, at 2:02 PM, Brian Haas <[hidden email]> wrote:

> Hi Peter,
>
> I'm actually more confused now that I was when I started...  I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file.  I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area.   I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored.   I then make 'composite_type = None' and it still stored the files to my dismay.
>
> Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
>
> Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
>
> I'm clearly not grasping something fundamental here.  I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
>
> Any ideas on where to go to troubleshoot this further?
>
> many thanks,
>
> ~brian
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
Ah!  will do. :)

On Mon, May 18, 2015 at 3:09 PM, Martin Čech <[hidden email]> wrote:
It is a wiki, if you miss something - please add it.

Thanks

Martin


On Mon, May 18, 2015 at 3:08 PM Brian Haas <[hidden email]> wrote:
THANK YOU!!! :)

The fundamental piece that I was missing at:

was the 

<command>someTool.sh $input1 ${os.path.join( input1.extra_files_path, 'results.txt' )} 
${os.path.join( input1.extra_files_path, 'results.dat' )} $output1</command>

It would be great if the documentation could be updated a bit for clarity on the importance of the '.extra_files_path', such as 'important: your tool needs to create this directory and store the additional files that you want there'.

This is a big leap forward for me. I'll now get on to testing the other aspects of it. (still climbing the galaxy learning curve).

thanks again!

~brian




On Mon, May 18, 2015 at 2:39 PM, Daniel Blankenberg <[hidden email]> wrote:
Hi Brian,

In your example tool, you need to pass “${ test_cdt.files_path }" to your perl script, which is a not-yet-existing directory for the files you want to be part of the dataset’s extra files path.
Then you create that directory and place the files you want within that directory.



Thanks for using Galaxy,

Dan


On May 18, 2015, at 2:02 PM, Brian Haas <[hidden email]> wrote:

> Hi Peter,
>
> I'm actually more confused now that I was when I started...  I was able to get your code working, focusing on the 'makeblastdb' with a nucleotide fasta file.  I figured that experimenting with this example would lead to some key insights, particularly by dropping certain files that are stored as extra-files area.   I copied everything over as a new tool, made it a new data type, and trimmed the blast.py code down to the bare essentials for testing just the nucleotide makeblastdb functionality. However, commenting out 'self.add_composite_file()' for various files had no effect - the files were still being stored.   I then make 'composite_type = None' and it still stored the files to my dismay.
>
> Since I couldn't get my copy to work as I had expected, I went back over to your 'blast.py' and just commented out the ''self.add_composite_file()' lines and set 'composite_type = None' .... and no effect.
>
> Of course, I restarted galaxy after each modification, and could verify that it was reading the new files and had byte-compiled the py mods.
>
> I'm clearly not grasping something fundamental here.  I can't get my code to bundle files, and I can't get your code to *not* bundle files. ;)
>
> Any ideas on where to go to troubleshoot this further?
>
> many thanks,
>
> ~brian
>
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/




--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/



--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
I've updated my code, which now generates the .extra_files directory:

I'm now trying to get the 

composite_type = 'auto_primary_file' to work, but all I'm getting is an empty file.   My module code simply includes:

 class CompositeDataTypeTest( Text ):
    file_ext = 'cdt'
    composite_type = 'auto_primary_file'
 
    def generate_primary_file( self, dataset = None ):
        log.debug("\n\n######## generate_primary_file() for Test data #######\n\n")
        return('<html><body>my test html</body></html>')


and from what I can tell, the generate_primary_file() isn't being called, as I'm not detecting the log entry in the console.


any ideas?

thanks in advance!


~brian




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: simple example of composite data types

Brian Haas
From some more testing it appears that the generate_primary_file() is called when a user manually uploads a composite data type (cdt), but it's not called when the cdt is generated as a product of a run.   That's fine. I can work around that.

I am curious about how cdts are uploaded manually, though.  If I select multiple files to upload as a cdt, they're not linked but instead uploaded as individual cdts.  Is there a way to upload multiple files as a cdt using the latest dev galaxy?  (I noticed the file upload gui has changed from earlier releases, and I'm wondering if support for cdt uploads is now lacking)

much appreciated,

~brian









___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/