SteadyLatch

Guides · APIs and JSON

PowerShell API 429 Too Many Requests: No Partial Data

Published · Tested with PowerShell 7.6.6 and Windows PowerShell 5.1 (build 26100) on Windows 11 Pro

A script downloads a report from an API, one page at a time. On page 3 the server answers 429 Too Many Requests. The script was written not to crash on errors, so it keeps pages 1 and 2, saves them and prints “Saved 6 rows”. The report should have 15. Nobody sees a failure, and the next step in the chain works with a third of the data. This guide reproduces that against a small local server, shows what PowerShell 7's built-in retry does and does not cover, and gives a script that retries with Retry-After, caps the wait and writes the file only when every page arrived. If it cannot, it exits with code 2 and writes nothing.

Before you start

The script that reports success

The server has five pages of three items each, 15 in all. In this scenario it answers 429 with Retry-After: 1 every time page 3 is requested. The script is easy to write when the goal is “do not crash on an API error”:

# naive.ps1
param([string] $BaseUrl, [string] $OutPath)
$all = @()
$page = 1
while ($page) {
    try { $r = Invoke-RestMethod "${BaseUrl}?page=$page" }
    catch { break }
    $all += $r.items
    $page = $r.next
}
$all | Export-Csv $OutPath -NoTypeInformation
"Saved $($all.Count) rows to $OutPath"

The same output in PowerShell 7.6.6 and in Windows PowerShell 5.1, in each of three runs:

PS> pwsh -NoProfile -File .\naive.ps1 -BaseUrl http://localhost:8429/always -OutPath naive.csv
Saved 6 rows to naive.csv
PS> $LASTEXITCODE
0
PS> (Import-Csv naive.csv).Count
6

Retrying alone does not fix this. A script that retries three times and then gives up the same way still saves 6 rows. The delivery rule is the part that needs to change: the file is either complete or it does not exist.

What the built-in retry does

PowerShell 7 has retry parameters on Invoke-WebRequest and Invoke-RestMethod. Microsoft Learn describes them like this:

We tried it on one request with -MaximumRetryCount 2 -RetryIntervalSec 1, three runs per case in PowerShell 7.6.6:

Server answers 429 once, then 200
  Retry-After: 5                      ok after 5.2 s, 5.1 s, 5.1 s
  Retry-After as an HTTP date (+3 s)  ok after 1.1 s, 1.1 s, 1.1 s
  no Retry-After                      ok after 1.0 s (from the interval)
Server answers 404, always            failed after 2.1 s, 2.1 s, 2.2 s (two retries)
Server answers 429 with Retry-After: 3600
                                      still waiting after 12 s, in all three runs; we stopped it

So the built-in retry helps for a single call in PowerShell 7 with a numeric Retry-After. For a script that has to run in both versions, read dates and stop on a long wait, you write the loop yourself.

A script that delivers everything or nothing

The script does four things. It retries only 429, up to -MaxRetries times per page. It waits as long as Retry-After says, in seconds or as a date, or 1, 2, 4… seconds when the header is missing or unreadable. It stops the whole run when a wait would be longer than -MaxWaitSeconds. And it writes the CSV only after the last page arrived, through a temporary file that then replaces the real one. Any other failure (a 404, a timeout, a refused connection, invalid JSON, a page without an items field) also ends the run with nothing written. Save it as Get-AllPages.ps1:

# Get-AllPages.ps1: read every page of a paginated JSON API, or deliver nothing.
# Retries 429 (Too Many Requests) a few times, waits as long as Retry-After says (up to a cap),
# and writes the CSV only when every page arrived. Windows PowerShell 5.1 and PowerShell 7.
param(
    [Parameter(Mandatory)] [string] $BaseUrl,   # e.g. https://api.example.test/items
    [Parameter(Mandatory)] [string] $OutPath,
    [int] $MaxRetries = 3,          # retries per page, after the first try
    [int] $MaxWaitSeconds = 30,     # longest single wait we accept
    [int] $MaxPages = 1000,         # stops a "next" link that never ends
    [int] $TimeoutSec = 30
)
$ErrorActionPreference = 'Stop'

function Get-RetryAfterSeconds {
    # Retry-After is whole seconds or an HTTP date (RFC 9110, section 10.2.3). $null if absent or unreadable.
    param([string] $Value)
    if ([string]::IsNullOrWhiteSpace($Value)) { return $null }
    $seconds = [long] 0
    if ([long]::TryParse($Value.Trim(), [System.Globalization.NumberStyles]::None,
            [cultureinfo]::InvariantCulture, [ref] $seconds)) { return $seconds }
    $when = [datetime]::MinValue
    $styles = [System.Globalization.DateTimeStyles]::AdjustToUniversal -bor [System.Globalization.DateTimeStyles]::AssumeUniversal
    if ([datetime]::TryParse($Value, [cultureinfo]::InvariantCulture, $styles, [ref] $when)) {
        return [math]::Max(0, [int] [math]::Ceiling(($when - [datetime]::UtcNow).TotalSeconds))
    }
    return $null
}

function Get-HttpFailure {
    # Status code and Retry-After text of a failed request, in either PowerShell.
    param($ErrorRecord)
    $response = $ErrorRecord.Exception.Response
    if ($null -eq $response) { return $null }   # no HTTP response: timeout, refused, DNS
    $retryAfter = $null
    if ($response.Headers -isnot [System.Net.WebHeaderCollection]) {            # PowerShell 7
        $values = $null
        if ($response.Headers.TryGetValues('Retry-After', [ref] $values)) { $retryAfter = @($values)[0] }
    }
    else { $retryAfter = $response.Headers['Retry-After'] }                      # Windows PowerShell 5.1
    [pscustomobject]@{ Status = [int] $response.StatusCode; RetryAfter = $retryAfter }
}

function Get-Page {
    param([string] $Uri)
    for ($attempt = 0; ; $attempt++) {
        try {
            $reply = Invoke-WebRequest -Uri $Uri -UseBasicParsing -TimeoutSec $TimeoutSec
            return $reply.Content | ConvertFrom-Json
        }
        catch {
            $failure = Get-HttpFailure $_
            if ($null -eq $failure -or $failure.Status -ne 429) { throw }       # only 429 is retried
            if ($attempt -ge $MaxRetries) {
                throw "429 on $Uri after $($attempt + 1) tries"
            }
            $wait = Get-RetryAfterSeconds $failure.RetryAfter
            if ($null -eq $wait) { $wait = [math]::Min($MaxWaitSeconds, [math]::Pow(2, $attempt)) }   # 1, 2, 4... s
            if ($wait -gt $MaxWaitSeconds) {
                throw "429 on $Uri with Retry-After $($failure.RetryAfter): more than the $MaxWaitSeconds s we accept"
            }
            Write-Host "429 on $Uri, retry $($attempt + 1) of $MaxRetries in $wait s (Retry-After: '$($failure.RetryAfter)')"
            Start-Sleep -Seconds $wait
        }
    }
}

try {
    $rows = [System.Collections.Generic.List[object]]::new()
    $page = 1
    while ($page) {
        if ($page -gt $MaxPages) { throw "more than $MaxPages pages: stopped" }
        $reply = Get-Page "${BaseUrl}?page=$page"
        if ($null -eq $reply.PSObject.Properties['items']) { throw "page $page has no 'items' field" }
        $rows.AddRange(@($reply.items))
        $page = $reply.next
    }
    $tmp = "$OutPath.tmp"
    $rows | Export-Csv -LiteralPath $tmp -NoTypeInformation
    Move-Item -LiteralPath $tmp -Destination $OutPath -Force
    "OK: $($rows.Count) rows written to $OutPath"
}
catch {
    Write-Host "FAILED after $($rows.Count) rows: nothing written to $OutPath. $($_.Exception.Message)"
    exit 2
}

Run it

Each case below is the same call, .\Get-AllPages.ps1 -BaseUrl http://localhost:8429/<scenario> -OutPath out.csv, against a different behaviour of the server. The times are from the first run in PowerShell 7.6.6; Windows PowerShell 5.1 gave the same results, with times within 0.2 seconds.

one 429 with Retry-After: 5       exit 0, 15 rows, 5.6 s
429 once, no Retry-After header   exit 0, 15 rows, 1.5 s
429 once, Retry-After as a date   exit 0, 15 rows, 3.5 s
429 on every try                  exit 2, no file, 3.5 s
429 with Retry-After: 3600        exit 2, no file, 0.4 s
404 on every try                  exit 2, no file, 0.3 s
answer 200 that is not JSON       exit 2, no file, 0.5 s
nothing listening on the port     exit 2, no file

The messages of the two cases that matter most. The server answered 429 on every try for page 3, with Retry-After: 1:

429 on http://localhost:8429/always?page=3, retry 1 of 3 in 1 s (Retry-After: '1')
429 on http://localhost:8429/always?page=3, retry 2 of 3 in 1 s (Retry-After: '1')
429 on http://localhost:8429/always?page=3, retry 3 of 3 in 1 s (Retry-After: '1')
FAILED after 6 rows: nothing written to out.csv. 429 on http://localhost:8429/always?page=3 after 4 tries

And the server asking for an hour:

FAILED after 6 rows: nothing written to out.csv. 429 on http://localhost:8429/long-wait?page=3 with Retry-After 3600: more than the 30 s we accept

When the server answers 429 once and then recovers, the retry line is followed by OK: 15 rows written to out.csv. For the date form the line reads retry 1 of 3 in 3 s (Retry-After: 'Fri, 09 Oct 2026 15:21:42 GMT'), and without the header, (Retry-After: ''). In every failing case, $LASTEXITCODE was 2 and out.csv did not exist.

In a scheduled job, let that exit code reach the scheduler; Task Scheduler Says Success but Your PowerShell Script Failed shows how.

Differences between PowerShell 7 and 5.1

What we saw when a request got a 429, with Invoke-WebRequest and with Invoke-RestMethod (both gave the same):

The test server

A server on localhost built on System.Net.HttpListener, with no dependencies and no administrator rights. Run it in a second window with pwsh -File server.ps1 (we ran it in PowerShell 7.6.6). The first part of the URL path picks the behaviour, and /reset clears the counters between cases. Page 3 is the one that gets throttled:

# server.ps1: a tiny paginated API on localhost that answers 429 on page 3.
param([int] $Port = 8429)
$listener = [System.Net.HttpListener]::new()
$listener.Prefixes.Add("http://localhost:$Port/")
$listener.Start()
$hits = @{}   # "scenario/page" -> number of requests so far
$lastPage = 5
while ($listener.IsListening) {
    $ctx = $listener.GetContext()
    $req = $ctx.Request; $res = $ctx.Response
    $parts = $req.Url.AbsolutePath.Trim('/').Split('/')
    if ($parts[0] -eq 'stop') { $res.StatusCode = 200; $res.Close(); break }
    if ($parts[0] -eq 'reset') { $hits.Clear(); $res.StatusCode = 200; $res.Close(); continue }
    $scenario = $parts[0]
    $page = [int] $req.QueryString['page']
    $key = "$scenario/$page"
    $hits[$key] = 1 + [int] $hits[$key]
    $limited = $false; $retryAfter = $null
    if ($page -eq 3) {
        switch ($scenario) {
            'once-seconds' { $limited = $hits[$key] -eq 1; $retryAfter = '1' }
            'once-five'    { $limited = $hits[$key] -eq 1; $retryAfter = '5' }
            'once-bare'  { $limited = $hits[$key] -eq 1 }
            'always'       { $limited = $true; $retryAfter = '1' }
            'long-wait'    { $limited = $true; $retryAfter = '3600' }
            'once-date'    { $limited = $hits[$key] -eq 1
                             $retryAfter = [datetime]::UtcNow.AddSeconds(3).ToString('r', [cultureinfo]::InvariantCulture) }
        }
    }
    if ($scenario -eq 'missing') {
        $res.StatusCode = 404
        $body = '{"error":"not found"}'
    }
    elseif ($scenario -eq 'no-items') {
        $body = '{"data":[],"next":null}'
    }
    elseif ($scenario -eq 'not-json') {
        $body = '<html>maintenance</html>'
    }
    elseif ($limited) {
        $res.StatusCode = 429
        if ($retryAfter) { $res.AddHeader('Retry-After', $retryAfter) }
        $body = '{"error":"rate limited"}'
    }
    else {
        $items = (($page - 1) * 3 + 1)..($page * 3) | ForEach-Object { @{ id = $_; name = "item-$_" } }
        $next = if ($page -lt $lastPage) { $page + 1 } else { $null }
        $body = @{ items = @($items); next = $next } | ConvertTo-Json -Compress -Depth 4
    }
    $bytes = [System.Text.Encoding]::UTF8.GetBytes($body)
    $res.ContentType = 'application/json'
    $res.OutputStream.Write($bytes, 0, $bytes.Length)
    $res.Close()
}
$listener.Stop()

Limits

Just want the scripts? The Scripts Pack · €12 has the code of this guide and the other guides ready to run, with examples and tests. The code on this page stays free.

The code on this page was written for this guide and tested on the versions above. The test runs also set the culture to en-US, so that messages and numbers came out in English. The data is invented.