Grafana says connection refused to Prometheus on localhost:9090
Err reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refusedShort answer
Inside a container, localhost means that container, not your machine or the other container. Grafana asking for localhost:9090 hits its own empty port 9090. Use the Prometheus service name on a user-defined network, or host.docker.internal for a Prometheus on the host. Prometheus needs the same fix for its Spring Boot target.
Fix it fast
1Find out which cause you have
Find out where Prometheus really runs, then test the address from inside the Grafana container.
docker compose ps docker compose exec grafana wget -qO- http://prometheus:9090/-/ready2Match what you see, then jump to its fix
- 1
The error names localhost:9090 or 127.0.0.1:9090 and Prometheus runs in another containerThe data source URL uses localhost from inside the Grafana containerFix - 2
The URL uses the service name but the error is a name resolution or connection error; containers were started with plain docker run on the default bridgeThe containers cannot resolve each other by nameFix - 3
Grafana connects, but the Prometheus Targets page shows the Spring job as down with connection refusedPrometheus scrapes localhost for the Spring Boot appFix
- 1
What does the error look like?
Err reading Prometheus: Post "http://localhost:9090/api/v1/query": dial tcp 127.0.0.1:9090: connect: connection refused
Cause 1 of 3
The data source URL uses localhost from inside the Grafana container
1Check
localhost refers to each container's own network namespace, so port 9090 inside the Grafana container is empty.
docker compose exec grafana sh -c 'wget -qO- http://localhost:9090/-/ready'2Fix
Point the data source at the Prometheus service name from your compose file, or at host.docker.internal:9090 when Prometheus runs on the host.
# Grafana > Connections > Data sources > Prometheus > Prometheus server URL
http://prometheus:9090
# Prometheus on the host instead:
http://host.docker.internal:90903Verify
Click Save & test in the data source page; it reports success.
docker compose exec grafana wget -qO- http://prometheus:9090/-/readyConfirmed by grafana.com
Cause 2 of 3
The containers cannot resolve each other by name
1Check
On the default bridge network containers reach each other only by IP address. Name resolution works on a user-defined bridge network.
docker network ls && docker inspect -f '{{json .NetworkSettings.Networks}}' grafana2Fix
Put both containers on one user-defined network (Compose creates one for you automatically).
docker network create monitoring
docker run -d --name prometheus --network monitoring -p 9090:9090 prom/prometheus
docker run -d --name grafana --network monitoring -p 3000:3000 grafana/grafana3Verify
The Grafana container reaches Prometheus by name.
docker exec grafana wget -qO- http://prometheus:9090/-/readyConfirmed by docs.docker.com
Cause 3 of 3
Prometheus scrapes localhost for the Spring Boot app
1Check
The same localhost rule applies to the scrape target inside prometheus.yml. The Spring Boot endpoint is /actuator/prometheus, and it must be exposed.
docker compose exec prometheus wget -qO- http://app:8080/actuator/prometheus | head2Fix
Use the app's service name and port as the target, and set the metrics path.
scrape_configs:
- job_name: "spring"
metrics_path: "/actuator/prometheus"
static_configs:
- targets: ["app:8080"]3Verify
The target shows as UP on the Prometheus Targets page (port 9090, /targets).
curl -s http://localhost:9090/api/v1/targets | head -c 400Confirmed by docs.spring.io
How do I stop it coming back?
- Use service names in every container-to-container URL; keep localhost for what you open in your own browser.
- Keep Grafana and Prometheus in one compose file so they share a network.
- Provision the data source from a file so the URL is in version control.
Sources
How was this page made?
I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.
Which stack is behind this page?
Projects of mine built on the same technologies:
Related fixes
Servers
A Java container dies at start with 'insufficient memory' or errno=12
os::commit_memory failed; error='Not enough space' (errno=12)
The JVM asked the operating system for more memory than the container is allowed to use.
3 causes, each with a fixServers
MongoDB says Authentication failed from a Java app against a Docker container
MongoCommandException: Command failed with error 18 (AuthenticationFailed): 'Authentication failed.' on server localhost:27017
Error 18 means MongoDB found no matching user in the database it checked.
4 causes, each with a fixServers
apk cannot find py-pip in an Alpine Docker image
py-pip (no such package):
On current Alpine releases there is no package called py-pip.
2 causes, each with a fixServers
ng serve runs in the container but localhost:4200 does not open
The browser cannot open localhost:4200 although ng serve is running in the container
ng serve listens on localhost by default, which inside a container is the container itself.
3 causes, each with a fixServers
Angular routes return 404 from nginx when you refresh or open a link
The home page loads, but refreshing /some/route shows an nginx 404
Angular routes like /users/42 are not files, so nginx looks for them on disk and returns 404.
3 causes, each with a fixServers
AH00558: Apache in Docker cannot reliably determine its fully qualified domain name
Set the 'ServerName' directive globally to suppress this message
AH00558 is a warning, not a crash.
2 causes, each with a fix