---
title: "n8n Web Scraping: HTTP Request and Proxy Settings"
description: "In n8n, a proxy is set in the HTTP Request node options or through environment variables. We cover setup, authentication and the most common errors."
url: https://proxynet.io/blog/n8n-proxy
date: 2026-09-19
author: "Acar Diveroli"
category: "Integration, Tutorial"
lang: en
---

# n8n Web Scraping: HTTP Request and Proxy Settings

The workflow you built in n8n opens ten product pages every morning and writes the prices to a sheet. The first week goes fine. Then the list grows to two hundred products, the workflow starts firing requests back to back from your server's single IP address, and the HTTP Request node turns red: first `429`, then `403`. Or the opposite case: the target API shows the right price only to requests from a specific country, and your n8n server sits in another one. The setting you need is the same in both cases: routing n8n's outgoing requests through a proxy.

This post covers the two places where a proxy is defined in n8n: the Proxy option on the HTTP Request node and the environment variables of a self-hosted install. We go through how credentials are written into the address, which setting overrides which, the difference between n8n Cloud and your own server, and the common errors. We also separate out the reverse proxy topic, which is what part of the people searching for "n8n proxy" are really after. The sample workflow tracks prices on a site published for scraping practice.

> **Note: Short answer**
>
> In the HTTP Request node, go to **Options → Add option → Proxy** and enter an address in the form `http://user:pass@pr.proxynet.io:8000`. This setting affects only that node and also works on n8n Cloud. On a self-hosted n8n, the `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` environment variables cover all nodes. If both are defined, the node setting overrides the environment variable. `N8N_PROXY_HOPS` is a different topic altogether: you need it when you put n8n itself behind a reverse proxy such as nginx.

## What is n8n and where does it fit in scraping?

n8n is an automation tool where you build workflows by connecting boxes (nodes) with lines. A trigger node starts the workflow (schedule, webhook, form), and the nodes after it fetch data, transform it and write it somewhere. You can use the tool on n8n's own cloud (n8n Cloud) or install it on your own server (self-hosted). That distinction matters a lot for proxies.

On the scraping side, two nodes do the work. The **HTTP Request** node sends a request to a URL and receives the response; the **HTML** node extracts the fields you want from that response with CSS selectors. This pair works well on pages whose content arrives ready from the server and on APIs that return JSON. On pages whose content is built in the browser with JavaScript, HTTP Request sees only an empty shell; we explained the difference in [Static vs Dynamic Pages](/blog/static-vs-dynamic-pages). For writing selectors, see [CSS Selectors vs XPath](/blog/css-selector-vs-xpath).

Where n8n is strong is in what happens to the data afterwards: writing it to a sheet, comparing it with the previous value, sending a notification on change. It is not the right tool for a crawl of thousands of pages; at that scale you need a framework like [Scrapy](/blog/scrapy-proxy) or a [data scraping](/data-scraping) infrastructure. We compared which method is enough in which case in [How to Extract Data From a Website](/blog/extract-data-from-website).

## "n8n proxy" is two separate topics: which one are you looking for?

In search suggestions, the query "n8n proxy" comes with the words `proxy hops`, `nginx` and `reverse`. These are not about the setting this post covers:

| | Forward proxy | Reverse proxy |
|---|---|---|
| Traffic direction | Requests going out from n8n | Requests coming in to n8n |
| What it does | Decides the IP address and country the request leaves from | Publishes n8n under a domain name with HTTPS |
| Typical tool | The proxy provider's endpoint | nginx, Caddy, Traefik |
| Setting in n8n | HTTP Request → Proxy, `HTTP_PROXY`, `HTTPS_PROXY` | `N8N_PROXY_HOPS`, the webhook URL variable |
| Symptom | `407`, `ECONNREFUSED`, `403` / `429` from the target site | Webhook URL showing `localhost`, wrong client IP |

If you run n8n behind nginx, the n8n documentation page on configuring webhook URLs behind a reverse proxy asks for two things: setting `N8N_PROXY_HOPS` to `1` (the default is `0`, and it tells n8n how many reverse proxies sit in front of it) and having the last proxy in the chain pass on the `X-Forwarded-For`, `X-Forwarded-Host` and `X-Forwarded-Proto` headers. These settings have no effect on your outgoing requests. You can find the difference between the two concepts and an nginx example in [Forward Proxy vs Reverse Proxy](/blog/forward-vs-reverse-proxy). The rest of this post is about the forward proxy.

## How do you set a proxy on the HTTP Request node?

The setting is not among the node's main fields but in the Options section at the very bottom, which is why it is easy to miss.

1. Open the HTTP Request node in your workflow and fill in **Method** and **URL**.
2. In the **Options** section at the bottom of the parameters, click **Add option**.
3. Add **Proxy** from the list.
4. Enter the proxy address in the new field, including the scheme: `http://user:pass@pr.proxynet.io:8000`
5. Run the node on its own with **Execute step** and check the output.

The field's placeholder text reads `e.g. http://myproxy:3128`, so n8n expects a full URL here. In the current version we installed locally for this post, an address written without a scheme (`user:pass@pr.proxynet.io:8000`) raised no error; the request simply went out directly without ever touching the proxy. The result was the same with an address starting with `socks5://`: the field is for HTTP proxies. The [HTTP Request node documentation](https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.httprequest/) states plainly that this option takes precedence over global settings made with `HTTP_PROXY`, `HTTPS_PROXY` and `ALL_PROXY`. So while a corporate proxy is defined on your server, you can still send a single node out through a different endpoint.

### Where do the username and password go?

The node has no separate username or password field for the proxy. Credentials go inside the address, in the form `username:password@`. The node's **Authentication** section is sent to the target site, not to the proxy; putting the proxy password there does not fix a `407`.

If your password contains characters such as `@`, `:`, `/` or `#`, the address gets split in the wrong place. Percent-encode them: `%40` for `@`, `%3A` for `:`. In our local test, the password `pa@ss:1` written as `pa%40ss%3A1` was decoded correctly at the proxy. The details of the two authentication methods are in [Proxy Authentication: User:Pass and IP Whitelist](/blog/proxy-authentication-methods).

One security note: the Proxy field is plain text and does not go into n8n's encrypted credential store (Credentials). When you export the workflow as JSON and share it, your proxy password travels inside the file; clear the field before sharing.

### Can you use an IP whitelist with n8n?

For a self-hosted n8n with a fixed IP address, yes: you add the server's IP to the allowed list in your proxy dashboard, and the address is written without a password as `http://pr.proxynet.io:8000`. On n8n Cloud this method is not reliable. On its Cloud IP addresses page, n8n says the outgoing IPs are not static and can change without warning. An address you whitelist today may be invalid tomorrow. On Cloud, connect with a username and password.

## How do environment variables work on a self-hosted install?

Instead of entering a proxy on every node, you can define one for the whole n8n process. n8n's [deployment environment variables](https://docs.n8n.io/deploy/host-n8n/configure-n8n/basic-configuration/use-environment-variables/deployment) page lists four variables:

| Variable | Purpose |
|---|---|
| `HTTP_PROXY` | Unencrypted HTTP traffic from nodes goes through this address |
| `HTTPS_PROXY` | TLS-encrypted (HTTPS) traffic from nodes goes through this address |
| `ALL_PROXY` | Used for both when the two more specific variables are not set |
| `NO_PROXY` | Comma-separated list of hosts to connect to directly, bypassing the proxy |

In a Docker Compose setup, the variables go into the service's `environment` section:

```yaml
services:
  n8n:
    image: docker.n8n.io/n8nio/n8n
    environment:
      - HTTP_PROXY=http://user:pass@pr.proxynet.io:8000
      - HTTPS_PROXY=http://user:pass@pr.proxynet.io:8000
      - NO_PROXY=localhost,127.0.0.1,postgres,redis
```

The `HTTPS_PROXY` value starting with `http://` is not a typo: the variable name says which traffic goes there, not the scheme of the proxy. We explained how these variables are defined at the operating system level in [How to Use a Proxy With wget](/blog/wget-proxy); here we only touch on three pitfalls specific to n8n.

**The lowercase variable overrides the uppercase one.** The same documentation page notes that in the `proxy-from-env` package n8n uses, lowercase names such as `http_proxy` take precedence over the uppercase ones when both are present. If someone else put a lowercase variable into your Docker image or server, your `HTTP_PROXY` value is silently ignored. Check both inside the container with `env | grep -i proxy`.

**Do not leave `NO_PROXY` empty.** If n8n also talks over HTTP to a database on the same network, to Redis or to an internal API, those requests go to the proxy too and will most likely get stuck there. Add internal hostnames and `localhost` to the list.

**Not every node honours these variables.** Nodes that use n8n's own HTTP helper pick up the setting. For some nodes that bring their own client library, there are issues reporting that the variable is ignored (for example, [issue #19652](https://github.com/n8n-io/n8n/issues/19652) in the n8n repository was opened in 2025 for the RSS Read node and has since been closed). Before putting a critical node into production, test it with the verification step below.

The variables are read when the n8n process starts; after changing them you need to restart the container or service.

## n8n Cloud vs self-hosted: which route is open where?

| | n8n Cloud | Self-hosted (Docker, npm) |
|---|---|---|
| HTTP Request → Proxy option | Available | Available |
| `HTTP_PROXY` / `HTTPS_PROXY` | Cannot be set, since the server environment is not yours | Can be set, affects the whole process |
| Proxy with IP whitelist | Not recommended: outgoing IPs change without warning | Works on a server with a fixed IP |
| Outgoing IP without a proxy | Variable addresses in n8n's cloud infrastructure | Your server's own address |

If you use Cloud, the node option is your only route. On a self-hosted install both are open; you can think of the environment variable as the "default exit" and the node option as "send this request out from somewhere else".

## Sample workflow: tracking the price of a product

We build the example on `books.toscrape.com`, a site published for scraping practice. In your own work, first check whether the target site has an official API or a seller dashboard; if it does, use that instead of parsing HTML. If not, read the site's `robots.txt` file and terms of use. How to read `robots.txt` rules is covered in [What Is a robots.txt File](/blog/robots-txt).

The workflow consists of six nodes:

1. **Schedule Trigger:** starts the workflow once a day. Price tracking rarely needs minute-level polling.
2. **Product list:** take the URLs from a Google Sheets table or an Edit Fields node. Each row carries a `url` field.
3. **Loop Over Items:** processes the list one by one. Leave Batch Size at `1`.
4. **HTTP Request:** put the expression `{{ $json.url }}` in the `URL` field, define the Proxy under Options, and leave the format as text in the Response option.
5. **HTML:** choose **Extract HTML Content** as the operation and define the fields to extract with CSS selectors.
6. **Wait:** waits a few seconds and returns to the start of the loop.

When the loop finishes, you write the resulting data to a sheet, compare it with the previous day's price in an **If** node, and send a notification if there is a difference.

This is what the HTTP Request node looks like in the exported workflow JSON:

```json
{
  "parameters": {
    "url": "={{ $json.url }}",
    "options": {
      "proxy": "http://user:pass@pr.proxynet.io:8000",
      "timeout": 20000,
      "response": {
        "response": { "fullResponse": true, "responseFormat": "text" }
      }
    }
  },
  "name": "HTTP Request",
  "type": "n8n-nodes-base.httpRequest",
  "typeVersion": 4.2
}
```

In the HTML node, three rows are enough for this site:

| Key | CSS Selector | Return Value |
|---|---|---|
| `title` | `h1` | Text |
| `price` | `p.price_color` | Text |
| `stock` | `p.availability` | Text |

Turn on **Trim Values** and **Clean Up Text** under Options; this removes the line breaks and extra spaces in the stock row. The output is a text such as `£51.77`; to turn it into a number, strip the currency symbol and fix the decimal separator in the next node.

We described a fuller setup with product matching, price history and threshold alerts in [Competitor Price Tracking in E-Commerce](/blog/competitor-price-tracking), and the product side on our [price monitoring solution](/price-monitoring) page.

## How do you set up rate limiting and retries in n8n?

When n8n receives a list, unless you tell it otherwise, it sends the request for all items back to back without waiting. Using a proxy does not make this behaviour polite; it only changes the address the requests leave from. Limiting the load the target site sees is up to you. The "Handle rate limits" page of the n8n documentation shows three built-in ways:

- **Batching (HTTP Request → Options):** `Items per Batch` sets how many requests go out at once, `Batch Interval (ms)` sets the wait between batches. It is the shortest route and needs no code.
- **Loop Over Items + Wait:** the setup we used in the example above. You see the wait after each request explicitly and can put other nodes in between.
- **Retry On Fail (the node's Settings tab):** retries a failed request; `Wait Between Tries (ms)` sets the time between attempts.

Turning on Retry On Fail for every error is not right. `429` and `503` get better with waiting; `403` and `407` do not, and sending the same request five times only produces needless traffic. We collected which codes to retry and which to stop on in the table in [HTTP Status Codes in Web Scraping](/blog/http-status-codes-web-scraping); the logic of rate limits is in [429 Too Many Requests](/blog/http-429-too-many-requests). If you want to branch on the status code, turn on **Include Response Headers and Status** and **Never Error** in the HTTP Request's Response option, then check the `statusCode` field in an If node.

## Do you need a separate node or code for rotation?

Some n8n scraping templates contain JavaScript snippets that keep a proxy list in a Code node and pick the next one on each request. If what you have is a list of individual IP addresses, you need that. With a rotating endpoint you do not: you connect to a single address such as `pr.proxynet.io:8000`, and the provider changes the exit IP on each new connection. On the n8n side, the address in the Proxy field never changes. The mechanism is explained in [What Is IP Rotation and How Does It Work?](/blog/ip-rotation-explained), the product side on the [Rotating Proxy](https://proxynet.io/rotating-proxy) page.

Conversely, there are jobs where the IP must never change. If a partner's API accepts requests only from whitelisted addresses and you are on n8n Cloud, going out through a fixed address such as [ISP Proxy](https://proxynet.io/static-isp-residential-proxy) instead of the variable Cloud IPs solves the problem. We covered this scenario in [Static IP for API Access](/blog/static-ip-for-api-access).

## How do you use a proxy with the AI Agent node?

You can attach HTTP Request to n8n's AI Agent node as a tool; the model calls it when needed to read a page or an API. An HTTP Request attached as a tool carries the same Options section, so the Proxy option works here in the same way. The documentation adds one more option for this use: **Optimize Response** filters JSON fields or extracts only the text from HTML before handing the response to the model, which lowers the number of tokens spent.

Watch two points in an agent setup. Restrict the addresses the agent can reach in the tool definition: instead of leaving the URL entirely to the model, define a fixed domain and a path parameter for the model to fill. Requests to the language model provider, on the other hand, do not pass through HTTP Request; putting them behind a proxy takes environment variables on a self-hosted install, and you need to test separately that the model node honours the variable. We explained how agents reach the web in [How AI Agents Work](/blog/how-ai-agents-work) and [Safe Web Access for LLMs](/blog/llm-safe-web-access). For pages that require a real browser, see [Playwright MCP](/blog/playwright-mcp).

## How do you verify that the proxy works?

Because a mistyped address can be ignored without an error, always test the setting:

1. Put two HTTP Request nodes in an empty workflow. Point both at a service that returns the IP the request came from (for example `https://api.ipify.org?format=json`).
2. Leave the Proxy option empty on the first and fill it in on the second.
3. Run both. The first node should show the IP of your n8n server (or of Cloud), the second the proxy's exit IP. If the two addresses are the same, the proxy is not in use.
4. If you use environment variables, do the same test with a node whose Proxy field is empty: if the IP changed, the variable is being read.

The steps are covered in detail in [How to Test a Proxy](/blog/how-to-test-a-proxy).

## Common errors and what they mean

| Error | Where it comes from | Likely cause | What to do |
|---|---|---|---|
| `ECONNREFUSED` | n8n server | Wrong proxy port, or a firewall blocking outbound traffic | Copy the address from the dashboard again, check outbound access to that port |
| `407 Proxy Authentication Required` | Proxy | Wrong password, special character not encoded, or IP missing from the whitelist | Put the credentials inside the address, percent-encoded |
| `400 Bad Request` (HTTPS targets only) | Proxy | The client sends the request to the proxy in plain form instead of opening a tunnel | Update n8n; see the note below |
| `ETIMEDOUT` / `ECONNRESET` | Network | The proxy is unreachable or the target is very slow | Raise Options → Timeout, pick a closer exit location |
| `ENOTFOUND` | DNS | Proxy hostname mistyped | Copy the name from the dashboard and enter it again |
| `403` / `429` from the target | Target site | The proxy works; the issue is rate or IP type | Slow down with Batching and Wait, diagnose the cause |

The `400` row has a history specific to n8n. The HTTP Request node uses the Axios library under the hood, and Axios's built-in proxy support is known for sending the request straight to the proxy on HTTPS targets instead of opening a [`CONNECT` tunnel](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Methods/CONNECT). [Issue #9169](https://github.com/n8n-io/n8n/issues/9169) in the n8n repository documents that this behaviour caused a `400` error in the node. The issue dates from 2024. In the current version we tested, the node opened a `CONNECT` tunnel at the proxy for an HTTPS target and fetched the page without trouble. If, on an old n8n version, HTTP URLs go through the proxy while HTTPS URLs return `400`, the first step is to update.

If you see a "proxy server is not responding" warning in your browser, the problem is unrelated to n8n; see [Proxy Server Is Not Responding](/blog/proxy-server-not-responding).

## Use cases

- **Price and stock tracking:** workflows that run once a day and notify on change. The setup is the same as the example above; the product side is on our [price monitoring](/price-monitoring) page.
- **Checking content that varies by location:** running the same node with proxies exiting from different countries to compare how the same page looks from each. For wide country coverage, [Residential Proxy](https://proxynet.io/residential-proxy) are used.
- **Small-scale content collection:** news headlines, listing counts, public catalogues. As scale grows, moving to a [web crawler](/web-crawler) solution makes more sense.

## Common mistakes

- **Putting the proxy password in the Authentication section.** That section goes to the target site. Proxy credentials live inside the address.
- **Leaving out the scheme.** Write `http://pr.proxynet.io:8000`, not `pr.proxynet.io:8000`; an address without a scheme can be ignored without an error.
- **Forgetting rate limits once a proxy is added.** A list of a hundred items sends a hundred requests at once if there is no Batching or Wait.
- **Automating pages that require login or contain personal data.** A proxy does not make a workflow legitimate if it breaks platform terms or data protection law; we summarised the legal framework in [Is Web Scraping Legal?](/blog/is-data-web-scraping-legal).

## Decision guide

| Need | Recommendation |
|---|---|
| I use n8n Cloud and want a single node to go out through a proxy | HTTP Request → Options → Proxy, with username and password |
| All nodes on a self-hosted n8n should go out through a proxy | `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY`; then verify node by node |
| A global setting exists but one node must exit from a different country | The Proxy option on that node; it overrides the global setting |
| A different IP on every request | A rotating endpoint; do not write rotation in a Code node |
| The other party will whitelist my IP | A fixed-IP proxy (ISP) or your own server with a fixed IP |
| The page loads with JavaScript and the HTML node returns nothing | HTTP Request is not enough; browser automation or the site's background JSON endpoint |

## Frequently asked questions

### Can you use a proxy on n8n Cloud?

Yes, the Proxy option on the HTTP Request node exists on Cloud too; environment variables are not available. Since Cloud's outgoing IPs are not fixed, connect to the proxy with a username and password rather than an IP whitelist.

### What does N8N_PROXY_HOPS do?

It tells n8n how many reverse proxies (nginx, Caddy, a cloud load balancer) sit in front of it; the default value is `0`. It has nothing to do with outgoing requests or the proxy setting in this post.

### Which takes priority, the node's Proxy setting or the environment variable?

The node setting. The n8n documentation says so, and in our local test a node with the Proxy option filled in went out through its own proxy while the environment variable pointed at a closed port.

### Does Zapier have the same setting?

No. Zapier's Webhooks step has URL, data, header and basic authentication fields; there is no proxy field. To route the request through a proxy you need to put a small service of your own in between: Zapier calls that service, and the service reaches the target through the proxy.

### Do I have to keep my proxy password in plain text inside the workflow?

If you use the node option, yes, the field is plain text. On a self-hosted install, the way to take the password out of the workflow is environment variables: the credentials stay in the server configuration and never enter the workflow JSON. If you have a server with a fixed IP, an IP whitelist removes the password altogether.

### Can HTTP Request pull data from any site?

No. The node does not run JavaScript, so on pages whose content is built in the browser the field you are looking for is not in the response. Sites that use bot protection may also return a verification page regardless of the proxy. In that case the thing to do is not to push against the protection but to look at the site's official API or data partnership options.

## Summary

In n8n a proxy is defined in two places: the Proxy field in the Options section of the HTTP Request node, and the `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` variables on a self-hosted install. Credentials go inside the address, the node setting overrides the global one, and `N8N_PROXY_HOPS` concerns only installs behind a reverse proxy. A proxy is no substitute for rate limiting: slow down with Batching or Wait, wait when you see a `429`, and choose the official API when there is one. You can compare the IP types that suit your workflows on our [proxy services](/proxy) page.