Importing file via FTP can silently fail, [Errno ftp error] 550

classic Classic list List threaded Threaded
7 messages Options
| Threaded
Open this post in threaded view
|

Importing file via FTP can silently fail, [Errno ftp error] 550

Peter Cock
Hi all,

Using either the legacy upload tool, or the new upload widget, the
following FTP URL *appears* to load into Galaxy correctly giving
a green history entry.

However, on closer inspection it is an empty file with this peep text:

Unable to fetch
ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
[Errno ftp error] 550 pathogens: No such file or directory

Clearly this should have been treated as a failure (red history entry).

Should I file this as a bug on Trello?

It seems there is a problem with the Sanger folder permissions which
trips up Galaxy and curl (but does not bother Firefox and wget):

$ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (9) Server denied you to change to the given directory

Using curl -v for verbose mode it tries to walk the path (as per the
https://www.ietf.org/rfc/rfc1738.txt standard) and fails with the
command "CWD pathogens" matching the Galaxy error.

The alternative curl FTP methods work:

$ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
--ftp-method nocwd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.8M  100 21.8M    0     0  7656k      0  0:00:02  0:00:02 --:--:-- 7654k

and:

$ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
--ftp-method singlecwd
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.8M  100 21.8M    0     0  7767k      0  0:00:02  0:00:02 --:--:-- 7766k

You can also trick curl by replacing (some of the) slashes with
the escaped version %2f which has the result of combining the
CWD commands, e.g.

$ curl -O ftp://ftp.sanger.ac.uk/pub%2fpathogens%2fBursaphelenchus%2fxylophilus%2fAssembly-v1.2/BurXv1.2.supercontigs.fa.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21.8M  100 21.8M    0     0  6827k      0  0:00:03  0:00:03 --:--:-- 6829k

This encoding trick does not seem to work within Galaxy.

I plan to report this to Sanger tomorrow. Should they fix this quickly
additional test cases might be useful (does Galaxy have an FTP
server which might be used for testing data importing like this?).

Is this a common issue? Should Galaxy automatically use an FTP
CWD workaround in this situation?

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Importing file via FTP can silently fail, [Errno ftp error] 550

Peter Cock
On Wed, Feb 25, 2015 at 5:33 PM, Peter Cock <[hidden email]> wrote:

> Hi all,
>
> Using either the legacy upload tool, or the new upload widget, the
> following FTP URL *appears* to load into Galaxy correctly giving
> a green history entry.
>
> However, on closer inspection it is an empty file with this peep text:
>
> Unable to fetch
> ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
> [Errno ftp error] 550 pathogens: No such file or directory
>
> Clearly this should have been treated as a failure (red history entry).
>
> Should I file this as a bug on Trello?
>
> It seems there is a problem with the Sanger folder permissions which
> trips up Galaxy and curl (but does not bother Firefox and wget):
>
> $ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
> curl: (9) Server denied you to change to the given directory
>
> Using curl -v for verbose mode it tries to walk the path (as per the
> https://www.ietf.org/rfc/rfc1738.txt standard) and fails with the
> command "CWD pathogens" matching the Galaxy error.

The reply from Sanger was that "pathogens" is actually an alias
for "project/pathogens" and as a workaround this longer URL can
be used with curl or Galaxy:

ftp://ftp.sanger.ac.uk/pub/project/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz

The problem with Galaxy failing to detect the FTP error with the
shorter URL remains:

ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz

Regards,

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Importing file via FTP can silently fail, [Errno ftp error] 550

John Chilton-4
Yeah - that is unfortunate - I agree completely that the resulting
datasets should be red. I have created a Trello card here:

https://trello.com/c/A6LrdjUU

-John

On Thu, Feb 26, 2015 at 5:19 AM, Peter Cock <[hidden email]> wrote:

> On Wed, Feb 25, 2015 at 5:33 PM, Peter Cock <[hidden email]> wrote:
>> Hi all,
>>
>> Using either the legacy upload tool, or the new upload widget, the
>> following FTP URL *appears* to load into Galaxy correctly giving
>> a green history entry.
>>
>> However, on closer inspection it is an empty file with this peep text:
>>
>> Unable to fetch
>> ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>> [Errno ftp error] 550 pathogens: No such file or directory
>>
>> Clearly this should have been treated as a failure (red history entry).
>>
>> Should I file this as a bug on Trello?
>>
>> It seems there is a problem with the Sanger folder permissions which
>> trips up Galaxy and curl (but does not bother Firefox and wget):
>>
>> $ curl -O ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>>                                  Dload  Upload   Total   Spent    Left  Speed
>>   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
>> curl: (9) Server denied you to change to the given directory
>>
>> Using curl -v for verbose mode it tries to walk the path (as per the
>> https://www.ietf.org/rfc/rfc1738.txt standard) and fails with the
>> command "CWD pathogens" matching the Galaxy error.
>
> The reply from Sanger was that "pathogens" is actually an alias
> for "project/pathogens" and as a workaround this longer URL can
> be used with curl or Galaxy:
>
> ftp://ftp.sanger.ac.uk/pub/project/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>
> The problem with Galaxy failing to detect the FTP error with the
> shorter URL remains:
>
> ftp://ftp.sanger.ac.uk/pub/pathogens/Bursaphelenchus/xylophilus/Assembly-v1.2/BurXv1.2.supercontigs.fa.gz
>
> Regards,
>
> Peter
> ___________________________________________________________
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Importing file via FTP can silently fail, [Errno ftp error] 550

Peter Cock
On Thu, Feb 26, 2015 at 2:24 PM, John Chilton <[hidden email]> wrote:
> Yeah - that is unfortunate - I agree completely that the resulting
> datasets should be red. I have created a Trello card here:
>
> https://trello.com/c/A6LrdjUU
>
> -John

Thanks John,

Does the new "Download from URL or upload files from disk" interface
still end up calling the old Python script for the upload tool? i.e.

https://github.com/galaxyproject/galaxy/blob/dev/tools/data_source/upload.py

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Importing file via FTP can silently fail, [Errno ftp error] 550

John Chilton-4
Yes - I believe it does. The top level layer (controller) is a little
different because it is coming through the API - but the next 10 or so
layers related to uploading are the same with the new method :).

-John

On Thu, Feb 26, 2015 at 9:57 AM, Peter Cock <[hidden email]> wrote:

> On Thu, Feb 26, 2015 at 2:24 PM, John Chilton <[hidden email]> wrote:
>> Yeah - that is unfortunate - I agree completely that the resulting
>> datasets should be red. I have created a Trello card here:
>>
>> https://trello.com/c/A6LrdjUU
>>
>> -John
>
> Thanks John,
>
> Does the new "Download from URL or upload files from disk" interface
> still end up calling the old Python script for the upload tool? i.e.
>
> https://github.com/galaxyproject/galaxy/blob/dev/tools/data_source/upload.py
>
> Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Importing file via FTP can silently fail, [Errno ftp error] 550

Nicola Soranzo-2
In reply to this post by Peter Cock
Il 26.02.2015 15:57 Peter Cock ha scritto:

> On Thu, Feb 26, 2015 at 2:24 PM, John Chilton wrote:
>
>> Yeah - that is unfortunate - I agree completely that the resulting
>> datasets should be red. I have created a Trello card here:
>> https://trello.com/c/A6LrdjUU [1] -John
>
> Thanks John,
>
> Does the new "Download from URL or upload files from disk" interface
> still end up calling the old Python script for the upload tool? i.e.
>
>
https://github.com/galaxyproject/galaxy/blob/dev/tools/data_source/upload.py

Yes, I think so.

Nicola


Connetti gratis il mondo con la nuova indoona:  hai la chat, le
chiamate, le video chiamate e persino le chiamate di gruppo.
E chiami gratis anche i numeri fissi e mobili nel mondo!
Scarica subito l’app Vai su https://www.indoona.com/


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/
| Threaded
Open this post in threaded view
|

Re: Importing file via FTP can silently fail, [Errno ftp error] 550

Peter Cock
In reply to this post by John Chilton-4
On Thu, Feb 26, 2015 at 2:24 PM, John Chilton <[hidden email]> wrote:
> Yeah - that is unfortunate - I agree completely that the resulting
> datasets should be red. I have created a Trello card here:
>
> https://trello.com/c/A6LrdjUU
>
> -John

I just commented on the Trello issue - as John implied with
the issue title, this is more general than a rare FTP failure.

e.g. http://www.galaxyproject.org/test_does_not_exist.txt

Using this in Galaxy gives the 404 error page as a "green" HTML
file. It should be "red" in the error state (while showing the user
facing 404 message as the file contents is reasonable).

Peter
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/