Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Grafana says connection refused to Prometheus on localhost:9090

ErrorErr reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused

Short answer

Inside a container, localhost means that container, not your machine or the other container. Grafana asking for localhost:9090 hits its own empty port 9090. Use the Prometheus service name on a user-defined network, or host.docker.internal for a Prometheus on the host. Prometheus needs the same fix for its Spring Boot target.

  • Applies to Grafana 12, Prometheus 3, Spring Boot 3.5, Docker Compose 2
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Find out where Prometheus really runs, then test the address from inside the Grafana container.

    docker compose ps
    docker compose exec grafana wget -qO- http://prometheus:9090/-/ready
  2. 2Match what you see, then jump to its fix

    1. 1The error names localhost:9090 or 127.0.0.1:9090 and Prometheus runs in another containerThe data source URL uses localhost from inside the Grafana containerFix
    2. 2The URL uses the service name but the error is a name resolution or connection error; containers were started with plain docker run on the default bridgeThe containers cannot resolve each other by nameFix
    3. 3Grafana connects, but the Prometheus Targets page shows the Spring job as down with connection refusedPrometheus scrapes localhost for the Spring Boot appFix

What does the error look like?

  • Err reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused

Cause 1 of 3

The data source URL uses localhost from inside the Grafana container

1Check

localhost refers to each container's own network namespace, so port 9090 inside the Grafana container is empty.

docker compose exec grafana sh -c 'wget -qO- http://localhost:9090/-/ready'

2Fix

Point the data source at the Prometheus service name from your compose file, or at host.docker.internal:9090 when Prometheus runs on the host.

# Grafana > Connections > Data sources > Prometheus > Prometheus server URL
http://prometheus:9090
# Prometheus on the host instead:
http://host.docker.internal:9090

3Verify

Click Save & test in the data source page; it reports success.

docker compose exec grafana wget -qO- http://prometheus:9090/-/ready

Confirmed by grafana.com

Did this fix it?

Cause 2 of 3

The containers cannot resolve each other by name

1Check

On the default bridge network containers reach each other only by IP address. Name resolution works on a user-defined bridge network.

docker network ls && docker inspect -f '{{json .NetworkSettings.Networks}}' grafana

2Fix

Put both containers on one user-defined network (Compose creates one for you automatically).

docker network create monitoring
docker run -d --name prometheus --network monitoring -p 9090:9090 prom/prometheus
docker run -d --name grafana --network monitoring -p 3000:3000 grafana/grafana

3Verify

The Grafana container reaches Prometheus by name.

docker exec grafana wget -qO- http://prometheus:9090/-/ready

Confirmed by docs.docker.com

Did this fix it?

Cause 3 of 3

Prometheus scrapes localhost for the Spring Boot app

1Check

The same localhost rule applies to the scrape target inside prometheus.yml. The Spring Boot endpoint is /actuator/prometheus, and it must be exposed.

docker compose exec prometheus wget -qO- http://app:8080/actuator/prometheus | head

2Fix

Use the app's service name and port as the target, and set the metrics path.

scrape_configs:
  - job_name: "spring"
    metrics_path: "/actuator/prometheus"
    static_configs:
      - targets: ["app:8080"]

3Verify

The target shows as UP on the Prometheus Targets page (port 9090, /targets).

curl -s http://localhost:9090/api/v1/targets | head -c 400

Confirmed by docs.spring.io

Did this fix it?

How do I stop it coming back?

  • Use service names in every container-to-container URL; keep localhost for what you open in your own browser.
  • Keep Grafana and Prometheus in one compose file so they share a network.
  • Provision the data source from a file so the URL is in version control.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.