brede_web_url

(export/brede/brede_web_url.m)

Function Synopsis

O = brede_web_url(url, varargin)

Help text

 brede_web_url        - Download a webpage

       function s = brede_web_url(url, ...
           'PropertyName', 'PropertyValue')  

       Input:    url        URL of the webpage
           
       Property: GetMethod  [ ExpectTelnet | {Java} | Perl | Telnet |
                            Wget | brede_web_url.pl ] Method to use
                            for the http download of the webpage. If
                            the argument is empty the default will be used. 
                 Info       [ {0} | Waitbar | an integer ] Amount of
                            debug information
                 Output     [ {string} | web ]

       Output:   O          The webpage as a string or a 'web'
                            structure 

       This function will download a webpage from the Internet. The
       entire webpage is returned as a string if 'output'='string'. 

       If 'output'='web' a 'web' will be returned with containing
       'url', 'html' and 'dateDownloaded' fields. 

       If the 'GetMethod'='java' then an empty string is returned if
       the response message from the web server is not 'OK'.
       For this GetMethod the Java facilities of Matlab is used
       (javaObject) together with objects java.net.URL and
       java.io.*. This should work on both Linux and Windows.
 
       The other 'GetMethods' might only work on UNIX systems and
       presently all the output is returned, ie, not just the webpage
       but also the system messages if any. Matlab does not directly
       support downloading of webpage, so it is done with a system
       call. 

       'GetMethod'='brede_web_url.pl' will use a Brede perl program
       from the perl directory. Perl and the LWP Perl library should
       be installed for this to work. This is presently only enabled
       for unix.

       'Telnet' will fetch the webpage only using telnet. On Linux
       systems this will probably not work and one has to use
       'ExpectTelnet'.

       'ExpectTelnet' will use a combination of expect and telnet to
       download the webpage. This option does not obtain redirect
       webpages from a 302 'error'. The 'Expect' program of Don Libes
       should be installed.

       If 'Perl' is used as the 'GetMethod' then Perl 5 should be
       installed including the 'LWP' library. Look at
       http://www.cpan.org/. The main Perl library that is required
       is 'libwww' and others need to be installed in order for this
       to work 
       http://cpan.valueclick.com/authors/id/G/GA/GAAS/Digest-MD5-2.13.tar.gz
       http://cpan.valueclick.com/authors/id/G/GA/GAAS/HTML-Parser-3.23.tar.gz
       http://cpan.valueclick.com/authors/id/G/GA/GAAS/MIME-Base64-2.12.tar.gz
       http://cpan.valueclick.com/authors/id/G/GA/GAAS/URI-1.12.tar.gz
       http://cpan.valueclick.com/authors/id/G/GA/GAAS/libwww-perl-5.53.tar.gz

       If 'Wget' is used as the 'GetMethod' the wget program should
       be installed. Look at ftp://prep.ai.mit.edu/pub/gnu
       wget is often installed by default on newer Linux systems. 

       Example:
         s = brede_web_url('http://www.dtu.dk')

         s = brede_web_url('http://www.dtu.dk', 'output', 'web')

       See also BREDE, BREDE_WEB, BREDE_WEB_ALLTHEWEB,
                BREDE_WEB_ALTAVISTA, BREDE_WEB_GOOGLE,
                BREDE_WEB_PMID, BREDE_STR_DEHTML.  

 $Id: brede_web_url.m,v 1.15 2006/03/24 15:34:50 fn Exp $ 

Cross-Reference Information

brede          export/brede/brede.m

brede_bm_get_paperexport/brede/brede_bm_get_paper.m
brede_web_allthewebexport/brede/brede_web_alltheweb.m
brede_web_altavistaexport/brede/brede_web_altavista.m
brede_web_googleexport/brede/brede_web_google.m
brede_web_pmid export/brede/brede_web_pmid.m
brede_web_pubmedexport/brede/brede_web_pubmed.m
brede_web_spmmailexport/brede/brede_web_spmmail.m
brede_web_wikipediaexport/brede/brede_web_wikipedia.m