Maaz Zindani

Loading the index…

↑↓ move↵ openesc close4072 pages
Servers, nginx and Docker

Grafana does not see Loki in a Docker Compose setup

ErrorSave & test on the Loki data source fails with a connection error

Short answer

The Loki data source takes only a base URL. In Compose that is the service name and port, http://loki:3100, because localhost inside the Grafana container is Grafana itself. If the URL is right, check that Loki answers on /ready, that the URL has no API path, and that a multi-tenant Loki gets the X-Scope-OrgID header.

  • Applies to Grafana 12, Loki 3, Docker Compose 2
  • Updated
  • By Maaz Zindani, Senior Software Engineer

Fix it fast

  1. 1Find out which cause you have

    Ask Loki whether it is ready, from the host through the published port. If it is not, fix Loki first; if it is, the data source settings are the next suspect.

    curl -s http://localhost:3100/ready
  2. 2Match what you see, then jump to its fix

    1. 1connection refused to localhost:3100 in the Grafana log or in the Save & test messageThe data source URL uses localhostFix
    2. 2The URL ends in /loki/api/v1/push or another pathAn API path was added to the URLFix
    3. 3an authentication error, or queries that succeed but return no dataLoki runs with multi-tenancy and Grafana sends no tenant IDFix
    4. 4/ready does not answer OK, or the connection works but Explore has no labelsLoki is not ready, or no log shipper has sent anything yetFix

What does the error look like?

  • Save & test on the Loki data source fails with a connection error
  • Explore shows no data, but Loki is running
  • the data source URL is http://localhost:3100 and Grafana runs in a container

Cause 1 of 4

The data source URL uses localhost

1Check

Inside the Grafana container, localhost is that container, not Loki. Compose registers each service name in an internal DNS, so containers on one network reach each other by name.

docker compose ps
grep -rn "url:" grafana/provisioning/datasources/

2Fix

Use the Loki service name and the container port in the data source. The Loki Docker tutorial provisions exactly this URL.

# grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100

3Verify

Restart Grafana, open the data source and press Save & test.

docker compose restart grafana

Confirmed by grafana.com

Did this fix it?

Cause 2 of 4

An API path was added to the URL

1Check

That push path is what log shippers such as Grafana Alloy use to send logs in. Grafana queries Loki and needs only the base address.

grep -rn "url:" grafana/provisioning/datasources/

2Fix

Enter the base URL only: scheme, host and port.

url: http://loki:3100

3Verify

Press Save & test again; it queries Loki's labels and reports the result.

docker compose restart grafana

Confirmed by grafana.com

Did this fix it?

Cause 3 of 4

Loki runs with multi-tenancy and Grafana sends no tenant ID

1Check

When Loki has auth_enabled: true, every request needs a tenant. Look for the setting in Loki's config file.

grep -n "auth_enabled" loki-config.yaml

2Fix

In the data source settings, click Add header under HTTP headers, use X-Scope-OrgID as the name and your tenant ID as the value, then save.

# Connections > Data sources > Loki > HTTP headers
# name:  X-Scope-OrgID
# value: <your tenant ID>

3Verify

Run a label query in Explore with the Loki source selected.

curl -s http://localhost:3100/ready

Confirmed by grafana.com

Did this fix it?

Cause 4 of 4

Loki is not ready, or no log shipper has sent anything yet

1Check

Loki answers on /ready when it is up. Your queries also need a label selector that exists and a time range inside the retention period.

curl -s http://localhost:3100/ready
docker compose logs loki --tail 30

2Fix

Fix the error in the Loki log first (often its config file path or a mounted folder). Then check that your shipper is running and that the query starts with a label that exists, for example the container name.

docker compose logs alloy --tail 30

3Verify

In Explore, pick the Loki source, choose a label from the label browser and set the range to the last 15 minutes.

curl -s http://localhost:3100/ready

Confirmed by grafana.com

Did this fix it?

How do I stop it coming back?

  • Provision the data source from a file in the Compose project so the URL is the same on every machine.
  • Use service names for container-to-container traffic and published ports only for your browser.
  • Add a healthcheck or a depends_on condition so Grafana starts after Loki is ready.

Sources

How was this page made?

I drafted this page with AI assistance from the official documents listed under Sources. Every cause above names the document that confirms it. I have not reproduced this fix on a machine of my own yet, so read it as documented guidance, not a test report. Found a mistake? Tell me.

Which stack is behind this page?

Projects of mine built on the same technologies:

Related fixes

Next step

Want this handled for good?

I look after servers, apps and releases end to end. Tell me what keeps breaking.