Jun 212011
I recently encountered an issue while attempting to download a slow-loading web page where calls to the system.net.webclient class would time-out. As you may already know, the webclient class has no capability for increasing the time-out value. This functionality does exist in the slightly more complicated system.net.httpwebrequest class.
Here’s a sample script that shows how to do it:
?Download GetWebPage.ps1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | Param ([string]$WebPage="", [string]$FileName="", [switch]$Help,[switch]$History) $HelpText = @' ####################################################################################### Name : GetWebPage.ps1 Date : 03/04/2011 Author : xb90@PoshTips.com Purpose : Download a webpage and save to file Usage : ./GetWebPage -webpage <URL> -filename <filename> [-Help][-History] where -WebPage specifies the URL of a web page to download -Filename specifies the filename to which the web page contents will be saved -Help produces help output -History produces help output with maintenance history Example: ./GetWebPage -webpage http://posthtips.com -filename poshtips.html ####################################################################################### '@ $HistoryText = @' Maintenance Log Date By Updates (insert newest updates at top) ---------- ---- --------------------------------------------------------------------- 06/21/2011 xb90 New Script ####################################################################################### '@ if ($help -or $History){ $HelpText; if ($History){$HistoryText} exit } if ((!$WebPage) -or (!$FileName)){ Throw "Missing Required Parameters!" $HelpText; } ## ## Main Script ## "reading web page: $webpage" $req = [system.net.httpwebrequest]::create("$webpage") # uncomment the following lines if needed: # $req.proxy = new-object -typename system.net.webproxy -argumentlist "http://proxy-name.com:port" # $req.credentials = get-credential $req.timeout = 3600000 $res = $req.getresponse() if ($res.statuscode -eq "OK"){ $stream = $res.getresponsestream() $reader = new-object io.streamreader($stream) $webpage = $reader.readtoend() [string]$pagedata = $webpage $reader.close() $stream.close() $res.close() "writing to file: $FileName" $pagedata > $FileName } else { throw "Error accessing page: $WebPage" } |
Click HERE for a sample of how to use the simpler system.net.webclient class.