Do you need to download files from the web but hate repeatedly clicking links? If your job involves downloading files from the web regularly, you will probably want to automate the task. Why not use PowerShell to download files much like an alternative PowerShell wget?
Not a reader? Watch this related video tutorial!Windows PowerShell and PowerShell comes with file-download capabilities. Using PowerShell to download files is a matter of knowing which cmdlets and .NET classes to use and how to use them.
In this article, you’ll learn the various ways to use PowerShell to download files from the web.
Announcing a Free LIVE training – Starting your PowerShell Journey – presented by Johan Arwidmark. Understand how PowerShell skills enhance your IT career, learn where to start with PowerShell, build your first scripts, and ask Johan questions directly in a live training environment.
Prerequisites
Since this is a learning-by-doing article, there are some prerequisites to ensure that you can follow the examples. Below are the basic requirements.
- A computer that is running on Windows 10 or higher. This computer is where you will run the scripts/commands featured in this article.
- Windows PowerShell 5.1 or PowerShell 7.1 (recommended).
- Windows 10 already includes Windows PowerShell 5.1.
- A web site that hosts the files to download.
- For non-authenticated file downloads, consider using the Tele2 Speedtest site, which is free.
- If you want to test file downloads with authorization, you may have to build your HTTP file server. An example of a free HTTP file server is HFS by Rejetto.
Using PowerShell to Download Files from URLs: Four Ways
There are four methods to use PowerShell to download files that do not depend on third-party tools. These are:
Invoke-WebRequestInvoke-RestMethodStart-BitsTransfer- .NET WebClient Class.
Whichever one of these four methods you use, the logic and components to make them work are the same. There must be a source URL pointing to the file’s location and the destination path to save the downloaded files. If required by the webserver, you need to enter the credentials as well.
The next sections show each of these four methods. In the end, it’s up to you to decide which way you would adapt when using PowerShell to download files.
Using Invoke-WebRequest as a PowerShell wget Alternative
The first method in PowerShell to download files is by using the Invoke-WebRequest cmdlet. Perhaps the most used cmdlet in this article, Invoke-WebRequest, can download HTTP, HTTPS, and FTP links.
Whether the source location requires users to log in, the Invoke-WebRequest cmdlet can handle requests with credentials as well.
To download a file, the syntax below shows the minimum parameters required to achieve the desired outcome.
Invoke-WebRequest -Uri <source> -OutFile <destination>For example, the code below downloads a file with the name 10MB.zip from a website. Then it saves the downloaded file to C:\dload\10MB.zip. You may copy the code below and paste it into your PowerShell session to test.
# Source file location
$source = 'http://speedtest.tele2.net/10MB.zip'
# Destination to save the file
$destination = 'c:\dload\10MB.zip'
#Download the file
Invoke-WebRequest -Uri $source -OutFile $destinationThe demonstration below shows the expected result after running the code above in PowerShell. As you can see, the file download was successful.
How about if the source requires authentication before allowing access? For example, the code below downloads a file from a private website where users must log in.
$source = 'https://mirror.lzex.ml/100MB.zip'
$destination = 'c:\dload\100MB.zip'
Invoke-WebRequest -Uri $source -OutFile $destinationHowever, the download failed due to unauthorized access.
If authentication is required, you should add a credential to the request using the -Credential parameter. The first line in the code below prompts you to enter the credential (username and password) and stores it to the $credential variable.
$credential = Get-Credential
$source = 'https://mirror.lzex.ml/100MB.zip'
$destination = 'c:\dload\100MB.zip'
Invoke-WebRequest -Uri $source -OutFile $destination -Credential $credential
The demonstration below shows what you’d expect to see when you run the above code in PowerShell. As you can see, the Get-Credential cmdlet prompted a PowerShell credential request. This time, using the credential with Invoke-WebRequest resulted in a successful download.
Related: Using the PowerShell Get-Credential Cmdlet and all things credentials
Looking Out for Parsing Errors when using Invoke-WebRequest
A crucial thing to remember when using Invoke-WebRequest in Windows PowerShell is that, by default, this cmdlet uses the Internet Explorer engine to parse data. The error below may happen when using Invoke-WebRequest on computers without the Internet Explorer in it.
You’ll have to re-issue your command, but this time, include the -UseBasicParsing switch.
Invoke-WebRequest -Uri <source> -OutFile <destination> -UseBasicParsingIn Windows PowerShell, you may receive an error message: The response content cannot be parsed because the Internet Explorer engine is not available, or Internet Explorer’s first-launch configuration is not complete. Specify the UseBasicParsing parameter and try again.
Starting with PowerShell Core 6.0, the Invoke-WebRequest cmdlet uses basic parsing only. As such, the -UseBasicParsing parameter is no longer necessary.
Using Invoke-RestMethod
The Invoke-RestMethod cmdlet is more about sending an HTTP or HTTPS request to a RESTful web service. This cmdlet is more suited for requests that interact with REST APIs such as Microsoft Graph API.
When it comes to downloading files straight from the web, Invoke-RestMethod is an excellent contender. Do not be deceived into thinking otherwise. There is not much difference between using Invoke-RestMethod and Invoke-WebRequest when used for downloading files from a direct web link.
Downloading a File using Invoke-RestMethod
To download a file using Invoke-RestMethod, use the syntax below. You’ll notice that the command uses the same parameters as Invoke-WebRequest.
Invoke-RestMethod -Uri <source> -OutFile <destination>In the example code below, the file is downloaded from the URL value in the $source variable. Then, saved to the path defined in the $destination variable.
$source = 'http://speedtest.tele2.net/10MB.zip'
$destination = 'c:\dload\10MB.zip'
Invoke-RestMethod -Uri $source -OutFile $destinationIf the source requires authentication, you can pass the credentials using the -Credential parameter. The example below prompts for the credentials and stores it to the $credential variable. The value of the $credential variable is then passed to the -Credential parameter.
Also, since the file link is an HTTP source and not HTTPS, it means that you are sending an unencrypted authentication. Typically, you should avoid using HTTP sources for security. But if you must use an HTTP source, you need to add the -AllowUnencryptedAuthentication switch to your command.
$credential = Get-Credential
$source = 'http://speedtest.tele2.net/10MB.zip'
$destination = 'c:\dload\10MB.zip'
Invoke-RestMethod -Uri $source -OutFile $destination -Credential $credential -AllowUnencryptedAuthentication
Using Start-BitsTransfer
Start-BitsTransfer is designed specifically for transferring files between client and server computers. This PowerShell cmdlet is dependent on the Background Intelligent Transfer Service (BITS) that is native to the Windows operating system.
Because Start-BitsTransfer requires BITS to work means that this cmdlet is not available on non-Windows computers. On the flipside, Start-BitsTransfer enjoys the benefits of BITS itself. Some of these benefits are:
- Network bandwidth and usage awareness.
- Interruption handling (resume, auto-resume, pause, etc.)
- Downloading multiple files as background jobs.
- Ability to set download job priorities.
Downloading a File
The fundamental way to use Start-BitsTransfer in PowerShell to download a file is to specify a source and destination. Using the script below, you only need to change the $source and $destination values according to your requirements.
$source = 'http://speedtest.tele2.net/100MB.zip'
$destination = 'c:\dload\100MB.zip'
Start-BitsTransfer -Source $source -Destination $destinationAs you can see from the demo below, the file is downloaded to the path c:\dload\100MB.zip.
Suppose the destination is not specified,
Start-BitsTransferdownloads and saves the file to the current working directory. For example, if you runStart-BitsTransferfrom C:\dload, the file downloads to the same directory.
For downloads that require authentication, Start-BitsTransfer has a -Credential parameter that accepts a PSCredential object.
Downloading Multiple Files
To demonstrate downloading multiple files, you’ll need to create a CSV file with two columns. Name the file filelist.txt. The first column should contain the link to the source, while the second column must contain the destination path. The file contents would like the one below.
# source,destination
http://speedtest.tele2.net/1MB.zip,c:\dload\1MB.zip
http://speedtest.tele2.net/10MB.zip,c:\dload\10MB.zip
http://speedtest.tele2.net/100MB.zip,c:\dload\100MB.zipRelated: Managing CSV Files in PowerShell with Import-Csv
Once the CSV file is ready, use the command below to begin the file download. The command imports the CSV file using Import-Csv and passes the contents to Start-BitsTransfer.
Import-Csv .\filelist.csv | Start-BitsTransferRefer to the demo below to see how the code above works. As you can see, the download starts, and you see the download progress. The PowerShell prompt is not available during the download process.
Suppose you want to start the download process as a background job. To do so, you only have to add the -Asynchronous switch at the end of the Start-BitsTransfer command.
Import-Csv .\filelist.csv | Start-BitsTransfer -AsynchronousInitially, the state of each job would show connecting. The screenshot below shows each file download’s job id.
Now that you’ve started the download process, you’ll want to check whether the download has been completed. To check the download job status, use the Get-BitsTransfer cmdlet. As you can see below, the download jobs’ status has changed to Transferred.
Using WebClient Class and HttpClient Class (.NET Framework)
PowerShell is based on .NET, and its nature makes it capable of leveraging the power of .NET itself. There’s two .NET class you can use in PowerShell to download files; WebClient and HttpClient.
If you want to know more about these two .NET class in more development and technical way, you could start with → When to use WebClient vs. HttpClient vs. HttpWebRequest. In the next section, you will learn how to use WebClient and HttpClient in PowerShell to download files from the web.
Downloading a File using System.Net.WebClient
To use the WebClient class, you need to initiate an object as a System.Net.WebClient **type. In the example below, the $webClient is the new System.Net.WebClient object. Then, using the DownloadFile() method starts the download of the file from the source.
Related: Using PowerShell Data Types Accelerators to Speed up Coding
Please copy the code below and run it in your PowerShell session to test. Note that you will not see any progress or output on the screen unless there’s an error. However, the PowerShell prompt will be locked until the download is complete.
# Define the source link and destination path
$source = 'http://speedtest.tele2.net/10MB.zip'
$destination = 'c:\dload\10MB.zip'
# Create the new WebClient
$webClient = [System.Net.WebClient]::new()
# Download the file
$webClient.DownloadFile($source, $destination)If the source requires authentication to allow the file download, you can use the code below. The first line prompts for the credential and stores it to the $credentials variable. The value of $credential is then included in the file download request.
# Prompt for username and password
$credentials = Get-Credential
$source = 'http://speedtest.tele2.net/10MB.zip'
$destination = 'c:\dload\10MB.zip'
# Create the new WebClient
$webClient = [System.Net.WebClient]::new()
# Add the credential
$webClient.Credentials = $credentials
# Download the file
$webClient.DownloadFile($source, $destination)
According to this Microsoft document: “We don’t recommend that you use the WebClient class for new development. Instead, use the System.Net.Http.HttpClient class.”
It appears that the WebClient class is obsolete, and the new class that Microsoft is endorsing is the HttpClient class. Don’t worry, though. The next section talks about using the HttpClient class in PowerShell to download files from the web.
Downloading a File using System.Net.Http.HttpClient
Like the WebClient class, you need to create first the System.Net.Http.HttpClient. Using the code below downloads the file from the $source to the $destination. Refer to the comments above each line to know what each line of code does.
The code below is live, and you can test it by running it in your PowerShell session.
# Set the source and destination
$source = 'http://speedtest.tele2.net/10MB.zip'
$destination = 'c:\dload\10MB.zip'
# Create the HTTP client download request
$httpClient = New-Object System.Net.Http.HttpClient
$response = $httpClient.GetAsync($source)
$response.Wait()
# Create a file stream to pointed to the output file destination
$outputFileStream = [System.IO.FileStream]::new($destination, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write)
# Stream the download to the destination file stream
$downloadTask = $response.Result.Content.CopyToAsync($outputFileStream)
$downloadTask.Wait()
# Close the file stream
$outputFileStream.Close()In situations where downloading a file requires authentication, you need to add the credential to the HttpClient object. To include a credential to the file download request, create a new System.Net.Http.HttpClientHandler object to store the credentials.
You can copy the code below and run it in PowerShell to test. Or you can also run it as a PowerShell script. In this example, the code is saved as download-file.ps1.
# Set the source and destination
$source = 'http://speedtest.tele2.net/10MB.zip'
$destination = 'c:\dload\10MB.zip'
# Prompt for credentials
$credentials = Get-Credential
# Create the HTTP client download request with credentials
$handler = New-Object System.Net.Http.HttpClientHandler
$handler.Credentials = $credentials
$httpClient = New-Object System.Net.Http.HttpClient($handler)
$response = $httpClient.GetAsync($source)
$response.Wait()
# Create a file stream to pointed to the output file destination
$outputFileStream = [System.IO.FileStream]::new($destination, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write)
# Stream the download to the destination file stream
$downloadTask = $response.Result.Content.CopyToAsync($outputFileStream)
$downloadTask.Wait()
# Close the file stream
$outputFileStream.Close()The demo below shows the result when running the PowerShell script to download the file.
At the start, the directory only has the script file in it. There’s a prompt to enter the username and password. Then, the script proceeds to download the file. After downloading the file, you can see that the new file is now inside the destination directory.
Announcing a Free LIVE training – Starting your PowerShell Journey – presented by Johan Arwidmark. Understand how PowerShell skills enhance your IT career, learn where to start with PowerShell, build your first scripts, and ask Johan questions directly in a live training environment.
Conclusion
Windows PowerShell and PowerShell Core come with built-in capabilities to download files, acting as a PowerShell wget alternative! Whether downloading password-protected sources, single or multiple files – a PowerShell way is available to you.
The file download methods covered in this article works on both Windows PowerShell and PowerShell Core. This means that these methods apply to both Windows and Non-Windows systems, with the exclusion of Start-BitsTransfer.
And since PowerShell is more than a command prompt, you can translate what you learned into scripts. For you, that would mean an opportunity for automation. No more copying URLs, clicking links, and waiting for downloads manually.
