Getting example_watch_folder.py to work...

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Getting example_watch_folder.py to work...

burdettn
Hi,
     I'm trying to get the example_watch_folder.py to run but it seems to fail, and I'm not sure why?

I run:

 ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "My API Import" f597429621d6eb2b

I got the workflow Id from :

http://barium-rbh/csiro/api/workflows

which gave me:
[
    {
        "id": "f597429621d6eb2b", 
        "name": "extract", 
        "url": "/csiro/api/workflows/f597429621d6eb2b"
    }, 
    {
        "id": "f2db41e1fa331b3e", 
        "name": "FULL CTE", 
        "url": "/csiro/api/workflows/f2db41e1fa331b3e"
    }
]

The output I get from the command line is:
{'outputs': ['ba0fa2aed4052bce'], 'history': 'ba03619785539f8c'}

The files I put into /home/galaxy/galaxy-drop/input get copied to /home/galaxy/galaxy-drop/output

But nothing else happens.

If I go to http://barium-rbh/csiro/api/histories

I can see:
{ "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "url": "/csiro/api/histories/ba03619785539f8c" },

However when I go to:
http://barium-rbh/csiro/api/histories/ba03619785539f8c

I get:
{ "contents_url": "/csiro/api/histories/ba03619785539f8c/contents", "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "state": "error", "state_details": { "discarded": 0, "empty": 0, "error": 1, "failed_metadata": 0, "new": 0, "ok": 0, "queued": 0, "running": 0, "setting_metadata": 0, "upload": 0 } }

Any ideas much appreciated

Thanks
Neil


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Getting example_watch_folder.py to work...

burdettn
Further, it seems that it doesn't manage to get hold of the file specified in the input directory as I can see from the output:

http://barium-rbh/csiro/api/histories/964b37715ec9bd22/contents/2faba7054d92b2df

{
    "data_type": "html", 
    "deleted": false, 
    "download_url": "/csiro/datasets/2faba7054d92b2df/display?to_ext=html", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_137.dat", 
    "file_size": 194, 
    "genome_build": "?", 
    "id": "2faba7054d92b2df", 
    "metadata_data_lines": null, 
    "metadata_dbkey": "?", 
    "misc_blurb": "error", 
    "misc_info": "Wed May  8 15:07:27 2013\nbashScript is: /home/galaxy/galaxy-dist/tools/visualization/extractSlice-wrapper.sh\ninput_image is: None\ncat: None: No such file or directory\nFailed reading file /tmp/tmp.9LWrz6SaLy.nii.gz\nitk::ERROR: PNGImageIO(0x1abdcf0): PNGIma", 
    "model_class": "HistoryDatasetAssociation", 
    "name": "Extract 2D slice on None", 
    "state": "error", 
    "visible": true
}

input_image is: None

Does anybody know why the file may not be getting read? It is being copied from the specified input directory to the output directory.

I have set :
allow_library_path_paste = True

and added my user to:
admin_users

Thanks for any help

Neil



From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Wednesday, May 08, 2013 2:46 PM
To: [hidden email]
Subject: Getting example_watch_folder.py to work...

Hi,
     I'm trying to get the example_watch_folder.py to run but it seems to fail, and I'm not sure why?

I run:

 ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "My API Import" f597429621d6eb2b

I got the workflow Id from :

http://barium-rbh/csiro/api/workflows

which gave me:
[
    {
        "id": "f597429621d6eb2b", 
        "name": "extract", 
        "url": "/csiro/api/workflows/f597429621d6eb2b"
    }, 
    {
        "id": "f2db41e1fa331b3e", 
        "name": "FULL CTE", 
        "url": "/csiro/api/workflows/f2db41e1fa331b3e"
    }
]

The output I get from the command line is:
{'outputs': ['ba0fa2aed4052bce'], 'history': 'ba03619785539f8c'}

The files I put into /home/galaxy/galaxy-drop/input get copied to /home/galaxy/galaxy-drop/output

But nothing else happens.

If I go to http://barium-rbh/csiro/api/histories

I can see:
{ "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "url": "/csiro/api/histories/ba03619785539f8c" },

However when I go to:
http://barium-rbh/csiro/api/histories/ba03619785539f8c

I get:
{ "contents_url": "/csiro/api/histories/ba03619785539f8c/contents", "id": "ba03619785539f8c", "name": "colin.nii.gz - extract", "state": "error", "state_details": { "discarded": 0, "empty": 0, "error": 1, "failed_metadata": 0, "new": 0, "ok": 0, "queued": 0, "running": 0, "setting_metadata": 0, "upload": 0 } }

Any ideas much appreciated

Thanks
Neil


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Getting example_watch_folder.py to work...

burdettn

It looks like the problem may have to do with the file I am using ???

 

It is a *.nii.gz file.

 

Does galaxy try to uncompress ? do something to *.gz files when using the api (as it tries to do in ~/galaxy-dist/tools/data_source/upload.py

 

Does anyone know of where in the code I can edit so my mile can be uploaded?

 

Thanks

Neil

 

From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Wednesday, 8 May 2013 3:15 PM
To: [hidden email]
Subject: RE: Getting example_watch_folder.py to work...

 

Further, it seems that it doesn't manage to get hold of the file specified in the input directory as I can see from the output:

http://barium-rbh/csiro/api/histories/964b37715ec9bd22/contents/2faba7054d92b2df

{
    "data_type": "html", 
    "deleted": false, 
    "download_url": "/csiro/datasets/2faba7054d92b2df/display?to_ext=html", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_137.dat", 
    "file_size": 194, 
    "genome_build": "?", 
    "id": "2faba7054d92b2df", 
    "metadata_data_lines": null, 
    "metadata_dbkey": "?", 
    "misc_blurb": "error", 
    "misc_info": "Wed May  8 15:07:27 2013\nbashScript is: /home/galaxy/galaxy-dist/tools/visualization/extractSlice-wrapper.sh\ninput_image is: None\ncat: None: No such file or directory\nFailed reading file /tmp/tmp.9LWrz6SaLy.nii.gz\nitk::ERROR: PNGImageIO(0x1abdcf0): PNGIma", 
    "model_class": "HistoryDatasetAssociation", 
    "name": "Extract 2D slice on None", 
    "state": "error", 
    "visible": true
}

 

input_image is: None

Does anybody know why the file may not be getting read? It is being copied from the specified input directory to the output directory.

I have set :
allow_library_path_paste = True

and added my user to:
admin_users

Thanks for any help

Neil


From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Wednesday, May 08, 2013 2:46 PM
To: [hidden email]
Subject: Getting example_watch_folder.py to work...

Hi,
     I'm trying to get the example_watch_folder.py to run but it seems to fail, and I'm not sure why?

I run:

 ./example_watch_folder.py 64f3209856a3cf4f2d034a1ad5bf851c http://barium-rbh/csiro/api/ /home/galaxy/galaxy-drop/input /home/galaxy/galaxy-drop/output "My API Import" f597429621d6eb2b

I got the workflow Id from :

http://barium-rbh/csiro/api/workflows

which gave me:

[
    {
        "id": "f597429621d6eb2b", 
        "name": "extract", 
        "url": "/csiro/api/workflows/f597429621d6eb2b"
    }, 
    {
        "id": "f2db41e1fa331b3e", 
        "name": "FULL CTE", 
        "url": "/csiro/api/workflows/f2db41e1fa331b3e"
    }
]

The output I get from the command line is:
{'outputs': ['ba0fa2aed4052bce'], 'history': 'ba03619785539f8c'}

The files I put into /home/galaxy/galaxy-drop/input get copied to /home/galaxy/galaxy-drop/output

But nothing else happens.

If I go to http://barium-rbh/csiro/api/histories

I can see:
{
        "id": "ba03619785539f8c", 
        "name": "colin.nii.gz - extract", 
        "url": "/csiro/api/histories/ba03619785539f8c"
    },

However when I go to:
http://barium-rbh/csiro/api/histories/ba03619785539f8c

I get:
{
    "contents_url": "/csiro/api/histories/ba03619785539f8c/contents", 
    "id": "ba03619785539f8c", 
    "name": "colin.nii.gz - extract", 
    "state": "error", 
    "state_details": {
        "discarded": 0, 
        "empty": 0, 
        "error": 1, 
        "failed_metadata": 0, 
        "new": 0, 
        "ok": 0, 
        "queued": 0, 
        "running": 0, 
        "setting_metadata": 0, 
        "upload": 0
    }
}

Any ideas much appreciated

Thanks
Neil

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Getting example_watch_folder.py to work...

burdettn
Hi,
       I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated.

I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file.

My xml file:
<tool id="CopyTool" name="Copy Tool">
  <description>Test</description>
  <command interpreter="perl">$__root_dir__/tools/copy/copy.pl
          --input_image "$inputImage"
          --output_html "$output"
  </command>

  <inputs>
    <param format="txt" name="inputImage" type="data" label="Input Image" />
  </inputs>
  <outputs>
    <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" />
  </outputs>
  <help>
Copy Tool

  </help>
</tool>

And copy.pl:
#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long;
my $input_image ;
my $output_html ;

# Get options from command line (i.e. galaxy)
my $result = GetOptions ( "input_image=s" => \$input_image,
                          "output_html=s" => \$output_html )
                          or die "Bad options";

print "input_image is: $input_image\n";

open FILE,"$input_image" or die $!;
my @lines = <FILE>;
close FILE;
my $numOfLines = scalar @lines;


# Create the output HTML, with links to the files and the square gif.
#
open HTML, ">", $output_html
        or die "Failed to create output HTML file '$output_html': $! ";

print HTML<<EOF;
<html>
<head>
<style>
iframe {
        border: 0px;
        background: #eeeeee ;
}
</style>
</head>
<body>
<h1>Copy Files tool</h1>
<h2>Generated $numOfLines files</h2>
EOF

# Put direct links to each output file
foreach my $sub_filename ( @lines )
{
  print HTML "Direct link to the $sub_filename file.<br/>\n";
}

print HTML "<br/><br/><br/>\n";

print HTML "</body></html>\n" ;

close HTML ;

So pretty basic.

I run:
./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e

and get the following output:
{'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'}

I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database:
/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat

and:
http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f6b131e891
shows the file has been uploaded:
{
    "data_type": "txt", 
    "date_uploaded": "2013-05-09T04:41:10.579602", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", 
    "file_size": 76, 
    "genome_build": "?", 
    "id": "1cd8e2f6b131e891", 
    "ldda_id": "1cd8e2f6b131e891", 
    "message": "", 
    "metadata_data_lines": 8, 
    "metadata_dbkey": "?", 
    "misc_blurb": "8 lines", 
    "misc_info": "uploaded txt file", 
    "model_class": "LibraryDataset", 
    "name": "sam2.txt", 
    "template_data": {}, 
    "uploaded_by": "[hidden email]", 
    "uuid": null
}

However, looking in the histories:
http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db

The output file is zero as the input file can not be found as shown below in bold ...

{ "accessible": true, "api_type": "file", "data_type": "html", "deleted": false, "display_apps": [], "display_types": [], "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", "file_ext": "html", "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", "file_size": 0, "genome_build": "?", "hid": 1, "history_id": "33b43b4e7093c91f", "id": "a799d38679e985db", "metadata_data_lines": null, "metadata_dbkey": "?", "misc_blurb": "error", "misc_info": "Thu May 9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", "model_class": "HistoryDatasetAssociation", "name": "Copy Tool on None", "peek": null, "purged": false, "state": "error", "uuid": null, "visible": true, "visualizations": [] }

Does anyone have any idea why the input_file is not passed in/obtained in the perl script?
as the script obviously copies it to the database so part of the script is working?

the file sam2.txt looks like this

Thanks again, sorry for swamping the list with this issue

Neil


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Getting example_watch_folder.py to work...

burdettn

Hi all,

          I finally resolved the issue I’ll mention it here to help anyone else who has the same issue. In “Edit” mode on the workflow you add the “Workflow control->Inputs” to use the api.

 

Final question....

Everything gets created and stored in the database, but the history column on the left of the browser doesn’t get updated. Can user see if the task has completed successfully and all files uploaded from the browser?

 

Thanks

Neil

 

From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Thursday, 9 May 2013 2:56 PM
To: [hidden email]
Subject: Getting example_watch_folder.py to work...

 

Hi,
       I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated.

I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file.

My xml file:
<tool id="CopyTool" name="Copy Tool">
  <description>Test</description>
  <command interpreter="perl">$__root_dir__/tools/copy/copy.pl
          --input_image "$inputImage"
          --output_html "$output"
  </command>

  <inputs>
    <param format="txt" name="inputImage" type="data" label="Input Image" />
  </inputs>
  <outputs>
    <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" />
  </outputs>
  <help>
Copy Tool

  </help>
</tool>

And copy.pl:
#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long;
my $input_image ;
my $output_html ;

# Get options from command line (i.e. galaxy)
my $result = GetOptions ( "input_image=s" => \$input_image,
                          "output_html=s" => \$output_html )
                          or die "Bad options";

print "input_image is: $input_image\n";

open FILE,"$input_image" or die $!;
my @lines = <FILE>;
close FILE;
my $numOfLines = scalar @lines;


# Create the output HTML, with links to the files and the square gif.
#
open HTML, ">", $output_html
        or die "Failed to create output HTML file '$output_html': $! ";

print HTML<<EOF;
<html>
<head>
<style>
iframe {
        border: 0px;
        background: #eeeeee ;
}
</style>
</head>
<body>
<h1>Copy Files tool</h1>
<h2>Generated $numOfLines files</h2>
EOF

# Put direct links to each output file
foreach my $sub_filename ( @lines )
{
  print HTML "Direct link to the $sub_filename file.<br/>\n";
}

print HTML "<br/><br/><br/>\n";

print HTML "</body></html>\n" ;

close HTML ;

So pretty basic.

I run:
./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e

and get the following output:
{'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'}

I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database:
/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat

and:
http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f6b131e891
shows the file has been uploaded:

{
    "data_type": "txt", 
    "date_uploaded": "2013-05-09T04:41:10.579602", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", 
    "file_size": 76, 
    "genome_build": "?", 
    "id": "1cd8e2f6b131e891", 
    "ldda_id": "1cd8e2f6b131e891", 
    "message": "", 
    "metadata_data_lines": 8, 
    "metadata_dbkey": "?", 
    "misc_blurb": "8 lines", 
    "misc_info": "uploaded txt file", 
    "model_class": "LibraryDataset", 
    "name": "sam2.txt", 
    "template_data": {}, 
    "uploaded_by": "[hidden email]", 
    "uuid": null
}

However, looking in the histories:
http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db

The output file is zero as the input file can not be found as shown below in bold ...

{
    "accessible": true, 
    "api_type": "file", 
    "data_type": "html", 
    "deleted": false, 
    "display_apps": [], 
    "display_types": [], 
    "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", 
    "file_ext": "html", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", 
    "file_size": 0, 
    "genome_build": "?", 
    "hid": 1, 
    "history_id": "33b43b4e7093c91f", 
    "id": "a799d38679e985db", 
    "metadata_data_lines": null, 
    "metadata_dbkey": "?", 
    "misc_blurb": "error", 
    "misc_info": "Thu May  9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", 
    "model_class": "HistoryDatasetAssociation", 
    "name": "Copy Tool on None", 
    "peek": null, 
    "purged": false, 
    "state": "error", 
    "uuid": null, 
    "visible": true, 
    "visualizations": []
}

Does anyone have any idea why the input_file is not passed in/obtained in the perl script?
as the script obviously copies it to the database so part of the script is working?

the file sam2.txt looks like this

Thanks again, sorry for swamping the list with this issue

Neil

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Getting example_watch_folder.py to work...

Dannon Baker-2
Hey Neil,

A user browsing that history wouldn't automatically get the updates (unless the history was already refreshing for some reason), but they definitely would upon refresh.  Glad you got the rest working!

-Dannon


On Thu, May 9, 2013 at 2:02 AM, <[hidden email]> wrote:

Hi all,

          I finally resolved the issue I’ll mention it here to help anyone else who has the same issue. In “Edit” mode on the workflow you add the “Workflow control->Inputs” to use the api.

 

Final question....

Everything gets created and stored in the database, but the history column on the left of the browser doesn’t get updated. Can user see if the task has completed successfully and all files uploaded from the browser?

 

Thanks

Neil

 

From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Thursday, 9 May 2013 2:56 PM


To: [hidden email]
Subject: Getting example_watch_folder.py to work...

 

Hi,


       I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated.

I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file.

My xml file:
<tool id="CopyTool" name="Copy Tool">
  <description>Test</description>
  <command interpreter="perl">$__root_dir__/tools/copy/copy.pl
          --input_image "$inputImage"
          --output_html "$output"
  </command>

  <inputs>
    <param format="txt" name="inputImage" type="data" label="Input Image" />
  </inputs>
  <outputs>
    <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" />
  </outputs>
  <help>
Copy Tool

  </help>
</tool>

And copy.pl:
#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long;
my $input_image ;
my $output_html ;

# Get options from command line (i.e. galaxy)
my $result = GetOptions ( "input_image=s" => \$input_image,
                          "output_html=s" => \$output_html )
                          or die "Bad options";

print "input_image is: $input_image\n";

open FILE,"$input_image" or die $!;
my @lines = <FILE>;
close FILE;
my $numOfLines = scalar @lines;


# Create the output HTML, with links to the files and the square gif.
#
open HTML, ">", $output_html
        or die "Failed to create output HTML file '$output_html': $! ";

print HTML<<EOF;
<html>
<head>
<style>
iframe {
        border: 0px;
        background: #eeeeee ;
}
</style>
</head>
<body>
<h1>Copy Files tool</h1>
<h2>Generated $numOfLines files</h2>
EOF

# Put direct links to each output file
foreach my $sub_filename ( @lines )
{
  print HTML "Direct link to the $sub_filename file.<br/>\n";
}

print HTML "<br/><br/><br/>\n";

print HTML "</body></html>\n" ;

close HTML ;

So pretty basic.

I run:
./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e

and get the following output:
{'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'}

I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database:
/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat

and:
http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f6b131e891
shows the file has been uploaded:

{
    "data_type": "txt", 
    "date_uploaded": "2013-05-09T04:41:10.579602", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", 
    "file_size": 76, 
    "genome_build": "?", 
    "id": "1cd8e2f6b131e891", 
    "ldda_id": "1cd8e2f6b131e891", 
    "message": "", 
    "metadata_data_lines": 8, 
    "metadata_dbkey": "?", 
    "misc_blurb": "8 lines", 
    "misc_info": "uploaded txt file", 
    "model_class": "LibraryDataset", 
    "name": "sam2.txt", 
    "template_data": {}, 
    "uploaded_by": "[hidden email]", 
    "uuid": null
}

However, looking in the histories:
http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db

The output file is zero as the input file can not be found as shown below in bold ...

{
    "accessible": true, 
    "api_type": "file", 
    "data_type": "html", 
    "deleted": false, 
    "display_apps": [], 
    "display_types": [], 
    "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", 
    "file_ext": "html", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", 
    "file_size": 0, 
    "genome_build": "?", 
    "hid": 1, 
    "history_id": "33b43b4e7093c91f", 
    "id": "a799d38679e985db", 
    "metadata_data_lines": null, 
    "metadata_dbkey": "?", 
    "misc_blurb": "error", 
    "misc_info": "Thu May  9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", 
    "model_class": "HistoryDatasetAssociation", 
    "name": "Copy Tool on None", 
    "peek": null, 
    "purged": false, 
    "state": "error", 
    "uuid": null, 
    "visible": true, 
    "visualizations": []
}

Does anyone have any idea why the input_file is not passed in/obtained in the perl script?
as the script obviously copies it to the database so part of the script is working?

the file sam2.txt looks like this

Thanks again, sorry for swamping the list with this issue

Neil

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Getting example_watch_folder.py to work...

Hakeem Almabrazi

Sorry for bringing this back but I am having an issue with the example_watch_folder.pl script.  After running in and put a fasta file, it gets copied over but it is empty.  Any idea how to fix this issue.

 

Uploaded by: galaxy@

Date uploaded: 2013-05-30

File size: 0 bytes

Data type: txt

Build: ?

Miscellaneous information: Uploaded temporary file (/tmp/API/myseq.fasta) does not exist.

empty

Database/Build: ?

Number of data lines: None

Disk file: /usr/local/galaxy/galaxy-dist/database/files/000/dataset_299.dat

 

 

Thank you for your help,

 

Regards

From: [hidden email] [mailto:[hidden email]] On Behalf Of Dannon Baker
Sent: Thursday, May 09, 2013 6:48 AM
To: [hidden email]
Cc: Galaxy Dev
Subject: Re: [galaxy-dev] Getting example_watch_folder.py to work...

 

Hey Neil,

 

A user browsing that history wouldn't automatically get the updates (unless the history was already refreshing for some reason), but they definitely would upon refresh.  Glad you got the rest working!

 

-Dannon

 

On Thu, May 9, 2013 at 2:02 AM, <[hidden email]> wrote:

Hi all,

          I finally resolved the issue I’ll mention it here to help anyone else who has the same issue. In “Edit” mode on the workflow you add the “Workflow control->Inputs” to use the api.

 

Final question....

Everything gets created and stored in the database, but the history column on the left of the browser doesn’t get updated. Can user see if the task has completed successfully and all files uploaded from the browser?

 

Thanks

Neil

 

From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Thursday, 9 May 2013 2:56 PM


To: [hidden email]
Subject: Getting example_watch_folder.py to work...

 

Hi,


       I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated.

I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file.

My xml file:
<tool id="CopyTool" name="Copy Tool">
  <description>Test</description>
  <command interpreter="perl">$__root_dir__/tools/copy/copy.pl
          --input_image "$inputImage"
          --output_html "$output"
  </command>

  <inputs>
    <param format="txt" name="inputImage" type="data" label="Input Image" />
  </inputs>
  <outputs>
    <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" />
  </outputs>
  <help>
Copy Tool

  </help>
</tool>

And copy.pl:
#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long;
my $input_image ;
my $output_html ;

# Get options from command line (i.e. galaxy)
my $result = GetOptions ( "input_image=s" => \$input_image,
                          "output_html=s" => \$output_html )
                          or die "Bad options";

print "input_image is: $input_image\n";

open FILE,"$input_image" or die $!;
my @lines = <FILE>;
close FILE;
my $numOfLines = scalar @lines;


# Create the output HTML, with links to the files and the square gif.
#
open HTML, ">", $output_html
        or die "Failed to create output HTML file '$output_html': $! ";

print HTML<<EOF;
<html>
<head>
<style>
iframe {
        border: 0px;
        background: #eeeeee ;
}
</style>
</head>
<body>
<h1>Copy Files tool</h1>
<h2>Generated $numOfLines files</h2>
EOF

# Put direct links to each output file
foreach my $sub_filename ( @lines )
{
  print HTML "Direct link to the $sub_filename file.<br/>\n";
}

print HTML "<br/><br/><br/>\n";

print HTML "</body></html>\n" ;

close HTML ;

So pretty basic.

I run:
./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e

and get the following output:
{'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'}

I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database:
/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat

and:
http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f6b131e891
shows the file has been uploaded:

{
    "data_type": "txt", 
    "date_uploaded": "2013-05-09T04:41:10.579602", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", 
    "file_size": 76, 
    "genome_build": "?", 
    "id": "1cd8e2f6b131e891", 
    "ldda_id": "1cd8e2f6b131e891", 
    "message": "", 
    "metadata_data_lines": 8, 
    "metadata_dbkey": "?", 
    "misc_blurb": "8 lines", 
    "misc_info": "uploaded txt file", 
    "model_class": "LibraryDataset", 
    "name": "sam2.txt", 
    "template_data": {}, 
    "uploaded_by": "[hidden email]", 
    "uuid": null
}

However, looking in the histories:
http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db


The output file is zero as the input file can not be found as shown below in bold ...

{
    "accessible": true, 
    "api_type": "file", 
    "data_type": "html", 
    "deleted": false, 
    "display_apps": [], 
    "display_types": [], 
    "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", 
    "file_ext": "html", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", 
    "file_size": 0, 
    "genome_build": "?", 
    "hid": 1, 
    "history_id": "33b43b4e7093c91f", 
    "id": "a799d38679e985db", 
    "metadata_data_lines": null, 
    "metadata_dbkey": "?", 
    "misc_blurb": "error", 
    "misc_info": "Thu May  9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", 
    "model_class": "HistoryDatasetAssociation", 
    "name": "Copy Tool on None", 
    "peek": null, 
    "purged": false, 
    "state": "error", 
    "uuid": null, 
    "visible": true, 
    "visualizations": []
}

Does anyone have any idea why the input_file is not passed in/obtained in the perl script?
as the script obviously copies it to the database so part of the script is working?

the file sam2.txt looks like this


Thanks again, sorry for swamping the list with this issue

Neil

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
Reply | Threaded
Open this post in threaded view
|

Re: Getting example_watch_folder.py to work...

Hakeem Almabrazi

In case someone has the same issue,

 

This is resolved by extending the sleep time found here http://gmod.827538.n3.nabble.com/Trouble-Shooting-example-watch-folder-py-td4030355.html

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Hakeem Almabrazi
Sent: Thursday, May 30, 2013 4:47 PM
To: Dannon Baker; [hidden email]
Cc: Galaxy Dev
Subject: Re: [galaxy-dev] Getting example_watch_folder.py to work...

 

Sorry for bringing this back but I am having an issue with the example_watch_folder.pl script.  After running in and put a fasta file, it gets copied over but it is empty.  Any idea how to fix this issue.

 

Uploaded by: galaxy@

Date uploaded: 2013-05-30

File size: 0 bytes

Data type: txt

Build: ?

Miscellaneous information: Uploaded temporary file (/tmp/API/myseq.fasta) does not exist.

empty

Database/Build: ?

Number of data lines: None

Disk file: /usr/local/galaxy/galaxy-dist/database/files/000/dataset_299.dat

 

 

Thank you for your help,

 

Regards

From: [hidden email] [mailto:[hidden email]] On Behalf Of Dannon Baker
Sent: Thursday, May 09, 2013 6:48 AM
To: [hidden email]
Cc: Galaxy Dev
Subject: Re: [galaxy-dev] Getting example_watch_folder.py to work...

 

Hey Neil,

 

A user browsing that history wouldn't automatically get the updates (unless the history was already refreshing for some reason), but they definitely would upon refresh.  Glad you got the rest working!

 

-Dannon

 

On Thu, May 9, 2013 at 2:02 AM, <[hidden email]> wrote:

Hi all,

          I finally resolved the issue I’ll mention it here to help anyone else who has the same issue. In “Edit” mode on the workflow you add the “Workflow control->Inputs” to use the api.

 

Final question....

Everything gets created and stored in the database, but the history column on the left of the browser doesn’t get updated. Can user see if the task has completed successfully and all files uploaded from the browser?

 

Thanks

Neil

 

From: Burdett, Neil (ICT Centre, Herston - RBWH)
Sent: Thursday, 9 May 2013 2:56 PM


To: [hidden email]
Subject: Getting example_watch_folder.py to work...

 

Hi,


       I'm still struggling to get the example_watch_folder.py to work. So any help much appreciated.

I've created a simple workflow, which essentially opens a text file and then writes out the data to a html file.

My xml file:
<tool id="CopyTool" name="Copy Tool">
  <description>Test</description>
  <command interpreter="perl">$__root_dir__/tools/copy/copy.pl
          --input_image "$inputImage"
          --output_html "$output"
  </command>

  <inputs>
    <param format="txt" name="inputImage" type="data" label="Input Image" />
  </inputs>
  <outputs>
    <data format="html" name="output" label="${tool.name} on #echo os.path.basename( str ( $inputImage.name ) )#" />
  </outputs>
  <help>
Copy Tool

  </help>
</tool>

And copy.pl:
#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long;
my $input_image ;
my $output_html ;

# Get options from command line (i.e. galaxy)
my $result = GetOptions ( "input_image=s" => \$input_image,
                          "output_html=s" => \$output_html )
                          or die "Bad options";

print "input_image is: $input_image\n";

open FILE,"$input_image" or die $!;
my @lines = <FILE>;
close FILE;
my $numOfLines = scalar @lines;


# Create the output HTML, with links to the files and the square gif.
#
open HTML, ">", $output_html
        or die "Failed to create output HTML file '$output_html': $! ";

print HTML<<EOF;
<html>
<head>
<style>
iframe {
        border: 0px;
        background: #eeeeee ;
}
</style>
</head>
<body>
<h1>Copy Files tool</h1>
<h2>Generated $numOfLines files</h2>
EOF

# Put direct links to each output file
foreach my $sub_filename ( @lines )
{
  print HTML "Direct link to the $sub_filename file.<br/>\n";
}

print HTML "<br/><br/><br/>\n";

print HTML "</body></html>\n" ;

close HTML ;

So pretty basic.

I run:
./example_watch_folder.py cce1b01926646d548f6ddc32ff01aa2e http://140.253.78.234/galaxy/api/ /home/galaxy/data_input /home/galaxy/data_output "My API Import" f2db41e1fa331b3e

and get the following output:
{'outputs': ['a799d38679e985db'], 'history': '33b43b4e7093c91f'}

I can see the sample.txt file I placed in the /home/galaxy/data_input has been put into the database:
/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat

and:
http://140.253.78.234/galaxy/api/libraries/f2db41e1fa331b3e/contents/1cd8e2f6b131e891
shows the file has been uploaded:

{
    "data_type": "txt", 
    "date_uploaded": "2013-05-09T04:41:10.579602", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_8.dat", 
    "file_size": 76, 
    "genome_build": "?", 
    "id": "1cd8e2f6b131e891", 
    "ldda_id": "1cd8e2f6b131e891", 
    "message": "", 
    "metadata_data_lines": 8, 
    "metadata_dbkey": "?", 
    "misc_blurb": "8 lines", 
    "misc_info": "uploaded txt file", 
    "model_class": "LibraryDataset", 
    "name": "sam2.txt", 
    "template_data": {}, 
    "uploaded_by": "[hidden email]", 
    "uuid": null
}



However, looking in the histories:

http://140.253.78.234/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db



 


The output file is zero as the input file can not be found as shown below in bold ...



{
    "accessible": true, 
    "api_type": "file", 
    "data_type": "html", 
    "deleted": false, 
    "display_apps": [], 
    "display_types": [], 
    "download_url": "/galaxy/api/histories/33b43b4e7093c91f/contents/a799d38679e985db/display", 
    "file_ext": "html", 
    "file_name": "/home/galaxy/galaxy-dist/database/files/000/dataset_9.dat", 
    "file_size": 0, 
    "genome_build": "?", 
    "hid": 1, 
    "history_id": "33b43b4e7093c91f", 
    "id": "a799d38679e985db", 
    "metadata_data_lines": null, 
    "metadata_dbkey": "?", 
    "misc_blurb": "error", 
    "misc_info": "Thu May  9 14:42:10 2013input_image is: None\nNo such file or directory at /home/galaxy/galaxy-dist/tools/copy/copy.pl line 27.\n", 
    "model_class": "HistoryDatasetAssociation", 
    "name": "Copy Tool on None", 
    "peek": null, 
    "purged": false, 
    "state": "error", 
    "uuid": null, 
    "visible": true, 
    "visualizations": []
}



Does anyone have any idea why the input_file is not passed in/obtained in the perl script? 

as the script obviously copies it to the database so part of the script is working?



the file sam2.txt looks like this



 


Thanks again, sorry for swamping the list with this issue



Neil

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

 


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  http://lists.bx.psu.edu/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/