connecting composite data elements to a display application

classic Classic list List threaded Threaded
7 messages Options
| Threaded
Open this post in threaded view
|

connecting composite data elements to a display application

Brian Haas
Greetings.

I have composite data stored with extra files like so:

galaxy/database/files/000/dataset_150.dat

and

galaxy/database/files/000/dataset_150_files

and I'm trying to set up a display link that provides a URL that embeds links to the files stored in the /dataset_150_files/ directory.

I've studied the documentation here:

https://wiki.galaxyproject.org/Admin/Tools/External%20Display%20Applications%20Tutorial

but can't quite figure out how to reference these extra data files.  

any help would be greatly appreciated.

many thanks,

~brian


--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: connecting composite data elements to a display application

Brian Haas
Here's a quick update on my adventure in galaxy-dev.  First - a quick review.  I have a composite data type where files are stored as:
primary data set:
galaxy/database/files/000/dataset_1.dat
and extra associated files:
galaxy/database/files/000/dataset_1_files/* 

I want to link this up to a display application (web app running on the same server via apache) such that galaxy provides a URL that will enable read access to these files.  

From what I could gather, there's no straightforward way to get a stable URL from galaxy that points to the extra_files in the composite data set, such as getting links to a bunch of bam files and indexes placed there. 

Assigning each of the extra files as metadata items (ie. similar to a bam.bai file as a metadata file for a single bam file) could have been an option, but I couldn't get that to work.

For now, since my Galaxy instance and my web app are running on the same machine with access to the same file system, I'll just have my web app read the files directly from the galaxy database directory via symlinks set up by my galaxy tool during the run.  It's a clunky way of doing things but should suffice for now given my setup.

If others have better solutions, I'm definitely interested in hearing them.

many thanks,

~brian



___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: connecting composite data elements to a display application

Björn Grüning-3
Hi Brian,

sorry to be so late to this adventure ...
Can you give me a few more details about your web-application?

Have you considered to use the VIS framework from Galaxy or the interactive environment framework?
This will give you more control over your datasets, because it's build-in into Galaxy and would open up your webapp also for other servers.

Cheers,
Bjoern

Here's a quick update on my adventure in galaxy-dev.  First - a quick review.  I have a composite data type where files are stored as:
primary data set:
galaxy/database/files/000/dataset_1.dat
and extra associated files:
galaxy/database/files/000/dataset_1_files/* 

I want to link this up to a display application (web app running on the same server via apache) such that galaxy provides a URL that will enable read access to these files.  

From what I could gather, there's no straightforward way to get a stable URL from galaxy that points to the extra_files in the composite data set, such as getting links to a bunch of bam files and indexes placed there. 

Assigning each of the extra files as metadata items (ie. similar to a bam.bai file as a metadata file for a single bam file) could have been an option, but I couldn't get that to work.

For now, since my Galaxy instance and my web app are running on the same machine with access to the same file system, I'll just have my web app read the files directly from the galaxy database directory via symlinks set up by my galaxy tool during the run.  It's a clunky way of doing things but should suffice for now given my setup.

If others have better solutions, I'm definitely interested in hearing them.

many thanks,

~brian




___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: connecting composite data elements to a display application

Brian Haas
Hi Bjoern,

My galaxy/database/files/000/dataset_1_files/ directory contains a bunch of files that I want to load into a genome viewer, including the target fasta file, .fai, several bam and .bam.bai files, and some bed files among a few others.  I'm looking to wire up an IGV-web application (a pure javascript version of IGV) to access and display these files.  The IGV-web app just needs the URLs to access these files.  I have a tool built into galaxy that generates these files and stores them as the composite data type, and I'm trying to add in the visualization capability such that one can click on a hyperlink provided in the finished job panel in order to access the igv-web view.

Again, the general issue is just - how to best enable an external application to access the extra_files of a composite data set?  

Ideally, it would involve configuration of some 

    galaxy/display_applications/display.xml 

file to indicate how the URL should be constructed.  The final url could be simply:


where data_base_url just gives the base URL to where my files would exist on galaxy, and since I know ahead of time the names of the files to expect in that directory, my web app can just reconstruct the URLs to the various files (fa, bam, bed, etc.).

Alternatively, the url could provide a list of urls to all of the required resources - with the requirement that the .bam.bai files can be accessed by simply putting a .bai on the end of the bam url for each bam, and likewise for .fai for fasta files.


many thx,

~brian




On Thu, May 21, 2015 at 4:56 AM, Bjoern Gruening <[hidden email]> wrote:
Hi Brian,

sorry to be so late to this adventure ...
Can you give me a few more details about your web-application?

Have you considered to use the VIS framework from Galaxy or the interactive environment framework?
This will give you more control over your datasets, because it's build-in into Galaxy and would open up your webapp also for other servers.

Cheers,
Bjoern


--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: connecting composite data elements to a display application

Brian Haas



The following slides show how my 'hack' works:


 It's definitely suboptimal, but at the very least, I've got something that works and meets my immediate needs.

I'm still *very* interested in following up with an alternative that somehow leverages a galaxy API or well-supported mechanism for doing this.

best,

~brian


On Thu, May 21, 2015 at 7:18 AM, Brian Haas <[hidden email]> wrote:
Hi Bjoern,

My galaxy/database/files/000/dataset_1_files/ directory contains a bunch of files that I want to load into a genome viewer, including the target fasta file, .fai, several bam and .bam.bai files, and some bed files among a few others.  I'm looking to wire up an IGV-web application (a pure javascript version of IGV) to access and display these files.  The IGV-web app just needs the URLs to access these files.  I have a tool built into galaxy that generates these files and stores them as the composite data type, and I'm trying to add in the visualization capability such that one can click on a hyperlink provided in the finished job panel in order to access the igv-web view.

Again, the general issue is just - how to best enable an external application to access the extra_files of a composite data set?  

Ideally, it would involve configuration of some 

    galaxy/display_applications/display.xml 

file to indicate how the URL should be constructed.  The final url could be simply:


where data_base_url just gives the base URL to where my files would exist on galaxy, and since I know ahead of time the names of the files to expect in that directory, my web app can just reconstruct the URLs to the various files (fa, bam, bed, etc.).

Alternatively, the url could provide a list of urls to all of the required resources - with the requirement that the .bam.bai files can be accessed by simply putting a .bai on the end of the bam url for each bam, and likewise for .fai for fasta files.


many thx,

~brian




On Thu, May 21, 2015 at 4:56 AM, Bjoern Gruening <[hidden email]> wrote:
Hi Brian,

sorry to be so late to this adventure ...
Can you give me a few more details about your web-application?

Have you considered to use the VIS framework from Galaxy or the interactive environment framework?
This will give you more control over your datasets, because it's build-in into Galaxy and would open up your webapp also for other servers.

Cheers,
Bjoern


--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 



--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: connecting composite data elements to a display application

Daniel Blankenberg
In reply to this post by Brian Haas
Hi brian,

I’ve enhanced Galaxy’s external display application framework to support accessing files in the extra files path in a pull request here: https://github.com/galaxyproject/galaxy/pull/284

To enable access you need to set allow_extra_files_access="True" for the display parameter. To access the files, you append the relative path of the extra file to the end of the URL generated for the parameter.

If you want to try this feature before it is merged into the codebase, you can apply it as a patch (https://github.com/galaxyproject/galaxy/pull/284.patch).

Please let us know if you encounter any issues using this new feature.


Thanks for using Galaxy,

Dan


On May 20, 2015, at 11:10 PM, Brian Haas <[hidden email]> wrote:

Here's a quick update on my adventure in galaxy-dev.  First - a quick review.  I have a composite data type where files are stored as:
primary data set:
galaxy/database/files/000/dataset_1.dat
and extra associated files:
galaxy/database/files/000/dataset_1_files/* 

I want to link this up to a display application (web app running on the same server via apache) such that galaxy provides a URL that will enable read access to these files.  

From what I could gather, there's no straightforward way to get a stable URL from galaxy that points to the extra_files in the composite data set, such as getting links to a bunch of bam files and indexes placed there. 

Assigning each of the extra files as metadata items (ie. similar to a bam.bai file as a metadata file for a single bam file) could have been an option, but I couldn't get that to work.

For now, since my Galaxy instance and my web app are running on the same machine with access to the same file system, I'll just have my web app read the files directly from the galaxy database directory via symlinks set up by my galaxy tool during the run.  It's a clunky way of doing things but should suffice for now given my setup.

If others have better solutions, I'm definitely interested in hearing them.

many thanks,

~brian


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: connecting composite data elements to a display application

Brian Haas
Thanks, Dan!  This will be a huge help. :)

yours truly,

~brian


On Thu, May 21, 2015 at 2:41 PM, Daniel Blankenberg <[hidden email]> wrote:
Hi brian,

I’ve enhanced Galaxy’s external display application framework to support accessing files in the extra files path in a pull request here: https://github.com/galaxyproject/galaxy/pull/284

To enable access you need to set allow_extra_files_access="True" for the display parameter. To access the files, you append the relative path of the extra file to the end of the URL generated for the parameter.

If you want to try this feature before it is merged into the codebase, you can apply it as a patch (https://github.com/galaxyproject/galaxy/pull/284.patch).

Please let us know if you encounter any issues using this new feature.


Thanks for using Galaxy,

Dan


On May 20, 2015, at 11:10 PM, Brian Haas <[hidden email]> wrote:

Here's a quick update on my adventure in galaxy-dev.  First - a quick review.  I have a composite data type where files are stored as:
primary data set:
galaxy/database/files/000/dataset_1.dat
and extra associated files:
galaxy/database/files/000/dataset_1_files/* 

I want to link this up to a display application (web app running on the same server via apache) such that galaxy provides a URL that will enable read access to these files.  

From what I could gather, there's no straightforward way to get a stable URL from galaxy that points to the extra_files in the composite data set, such as getting links to a bunch of bam files and indexes placed there. 

Assigning each of the extra files as metadata items (ie. similar to a bam.bai file as a metadata file for a single bam file) could have been an option, but I couldn't get that to work.

For now, since my Galaxy instance and my web app are running on the same machine with access to the same file system, I'll just have my web app read the files directly from the galaxy database directory via symlinks set up by my galaxy tool during the run.  It's a clunky way of doing things but should suffice for now given my setup.

If others have better solutions, I'm definitely interested in hearing them.

many thanks,

~brian


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/




--
--
Brian J. Haas
The Broad Institute
http://broadinstitute.org/~bhaas

 

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/