Porthole: know why it’s slow, fix it fast

Service map · six teams · Mon Oct 19

Six teams, 46 services. One map.

Every service was discovered from its own traffic, with its owner, rate, p95 and errors. The checkout path runs through all six teams.

Checkout path: api-gateway → auth, checkout-svc
checkout-svc → inventory-svc, orders-db, payments-svc

Edge

Sam Whitaker · 8 services

api-gateway1,240 req/s · 38 ms · 0.02%
cdn-edge3,900 req/s · 12 ms · 0.00%
rate-limiter1,240 req/s · 2 ms · 0.00%

+5 more

Identity

Grace Liu · 6 services

auth410 req/s · 42 ms · 0.01%
sessions620 req/s · 9 ms · 0.00%
identity-verify3 req/s · 880 ms · 0.30%

+3 more

Checkout

Dana Brooks · 7 services

checkout-svc86 req/s · 390 ms · 0.04%
cart240 req/s · 31 ms · 0.01%
promotions150 req/s · 24 ms · 0.00%

+4 more

Inventory

Tomás Rivera · 9 services

inventory-svc520 req/s · 64 ms · 0.01%
search-api310 req/s · 88 ms · 0.02%
search-indexernightly job

+6 more

Data

Ben Okafor · 8 services

orders-db2,900 queries/s · 4 ms · 0.00%
kafka-main6,100 msgs/s · 3 ms · 0.00%
redis-cache18,000 ops/s · 1 ms · 0.00%

+5 more

Payments

Ana Ruiz · 8 services

payments-svc14 req/s · 180 ms · 0.05%
ledger14 req/s · 22 ms · 0.00%
billing-workerno traces yet

+5 more

46 services · 212 hosts · 9 clusters · 6 teams
POST /checkout 3.1 req/s · p95 420 ms · target 800 ms
45 of 46 sending traces · billing-worker logs only

Trace · Checkout and Data · Wed Oct 21 · 2:14 pm ET

2,310 ms. 1,840 in one query.

One click from the slow checkout to its trace, the query that held it, the plan and the deploy that changed it.

0 … 2,400 ms

Edgeapi-gateway · POST /checkout
2,310 ms
2,310 ms
Identityauth · verify session
40 ms
40 ms
Checkoutcheckout-svc · POST /checkout
2,250 ms
2,250 ms
Inventoryinventory-svc · reserve items
120 ms
120 ms
Dataorders-db · SELECT orders
1,840 msKeisha · Checkout
1,840 ms
Paymentspayments-svc · authorize
210 ms
210 ms

Top query on orders-db since 1:58 pm

SELECT id, status, total FROM orders
WHERE customer_id = $1 AND created_at > $2

Seq Scan on orders · 9,412,077 rows removed by filter · 3 returned

Ben · Data

2:14:07 pm · orders-db · duration: 1840.2 ms

traceparent 00-7f3a…c21

→ this trace

checkout-svc during this span

checkout-svc · POST /checkoutpgx Query

97% off-CPU · 82% in pgx Query (1,840 of 2,250 ms)

checkout-svc v412 · 1:58 pm · GitHub Actions

PR #3391 ‘drop unused orders indexes’

migration 0412 drops orders_customer_id_idx

Index gone in v412.

api-gateway 2,310 = auth 40 + checkout-svc 2,250 + edge 20
checkout-svc 2,250 = inventory 120 + payments 210 + Postgres 1,840 + own 80
1,840 of 2,310 ms (80%)

Slow traces since 1:58 pm share: db.statement orders by customer_id (100% vs 0% before) · service.version v412 (100%)

Incident · Checkout · Wed Oct 21 · 1:58 – 2:31 pm ET

Paged at 2:16. 410 ms by 2:31.

The agent tested five causes and posted the one that held, with its queries, three minutes after the page. It readied the rollback, and Dana approved it in Slack.

1:58 · v412 deployed · index dropped
2:06 · p95 610 ms · orders-db CPU 41% → 88%
2:14 · p95 840 ms · crossed 800 ms
2:16 · monitor fired · Oncall paged Keisha Grant
2:17 · Keisha acknowledged
2:19 · root cause posted, 6 queries cited
2:20 · rollback to v411 readied
2:22 · Dana Brooks approved · Slack
2:29 · index rebuilt
2:31 · p95 410 ms · resolved

POST /checkout p95

800 ms target1:58 v412
430 ms2:14 · 840 ms
crossed 8002:16 monitor2:19 root cause2:20 · 2.3 s peak2:22 approved2:28 · 2,150 ms2:29 index rebuilt2:31 · 410 ms

Agent · 2:19 pm · 5 causes tested · 6 queries cited

1 · v412 dropped orders_customer_id_idx · seq scan · 1,840 of 2,310 ms (80%) · heldQ1Q2Q3Q4
2 · Traffic surge · ruled out: 3.1 req/s, same as MondayQ5
3 · Payment provider slow · ruled out: calls unchangedQ1
4 · New errors · ruled out: no new Spool issuesQ6
5 · Pod pressure · ruled out: checkout pods at 38% CPUQ4

Rollback readied 2:20 pm

checkout-svc → v411 · reverts migration 0412

rebuilds orders_customer_id_idx without locking writes · about 7 min · no data changed

Dana Brooks approved · Slack · 2:22 pm

p95 2.3 s → 410 ms
SLO checkout 99.9% · 43.2 min budget − 11.4 used = 31.8 left
Lamp · Checkout · degraded 2:17 → operational 2:32

2:34 pm · follow-up CHK-482 filed in Vector · Thu Oct 22 · v413 shipped with the index kept

Kubernetes · Inventory · Tue Oct 20 · ET

Killed at 3:07 am. Resized by standup.

The nightly reindex outgrew its memory while search kept serving Monday’s index, so nobody was paged. The agent found why and opened the fix; Tomás merged it at 9:12.

Inventorysearch-indexer-7d9f6-x2kq4 · cluster prod-east-2 · namespace inventory

2:30 – 4:00 am

9:00 – 10:00 am

Kubernetes event · OOMKilled

3:07:41 am · in Porthole at 3:08:02

owner: Inventory

Agent · 3:12 am · posted to #inventory, not paged

Catalog grew 1.84M → 2.41M SKUs since Mon Sep 14 (+31%)

each run now peaks above the 2 GiB limit

search-api still serving Monday’s index · no customer impact

PR #2207 · search-indexer

memory limit 2 GiB → 3 GiB, request 1.5 → 2.5 GiB

node has 5.2 GiB unrequested

opened by the agent 3:14 am

Tomás Rivera merged · 9:12 am

Argo CD synced 9:14 am

Coverage and noise · Payments, Identity and Edge · Tue Oct 20 – Fri Oct 23

45 services traced. Then all 46.

billing-worker had sent logs but never a trace. The agent wrote the instrumentation as a pull request, and Ana merged it Thursday.

Wed Oct 21

Payments

billing-worker

Python · Kafka consumer

logs only · 0 spans since Mon Oct 12

PR #1182 · payments/billing-worker

Add OpenTelemetry tracing and Kafka context propagation

+18 −2 · opened by the agent Tue Oct 20, 4:10 pm · CI passed

requirements.txt+1

+opentelemetry-distro==0.48b0

worker/main.py+6 −2

worker/consumer.py+10

+from opentelemetry import trace+from opentelemetry.propagate import extract+tracer = trace.get_tracer("billing-worker") h = {k: v.decode() for k, v in msg.headers() or []}+ with tracer.start_as_current_span("invoice", context=extract(h)):

Dockerfile+1

+ENV OTEL_SERVICE_NAME=billing-worker

Ana Ruiz merged · Thu Oct 22, 10:40 am

Merged.

Thu Oct 22

Payments

billing-worker

38 jobs/s · p95 1.2 s · 0.02% errors

traces from 11:05 am

46 of 46 services traced

checkout-svc → payments-svc → invoices.created → billing-worker

Identity

auth p99 > 150 ms · fired 41 times Oct 12 – 22 · acted on 0

→ p99 > 250 ms for 10 min

replayed: 41 → 3, every real slowdown kept

PR #88 · Grace Liu merged Fri Oct 23, 2:10 pm

Edge

rate-limiter 5xx > 0.5% · flapped 23 times

→ judged over 5-minute windows

replayed: 23 → 1, the real one kept

PR #89 · Sam Whitaker merged Fri Oct 23, 2:25 pm

Cost control · Checkout and Inventory · Wed Oct 14 and Mon Oct 26 · ET

1.4M new series. Caught in 40 minutes.

Porthole watches what every team sends. A tag that multiplied one metric and a debug log nobody read were each caught, fixed by a pull request or a rule, and approved by their team.

Checkoutcheckout.latency · active series · Wed Oct 14

Agent · 11:04 am · PR #3307: group checkout.latency by route, keep user_id on spans

searching traces by user_id still works

Dana Brooks merged · 11:35 am

Inventorysearch-api · logs a day · Mon Oct 19 – Fri Oct 30

410 GB a day · 372 GB DEBUG · 0 searches on DEBUG lines Oct 12 – 25

Agent · Mon Oct 26, 9:05 am · drop DEBUG from search-api, keep every WARN and ERROR line

Tomás Rivera approved · 9:40 am

error lines kept

Logs a day by team · from Tue Oct 27 · 1,200 GB

Edge310 GB
Inventory260 GB (was 632)
Data190 GB
Payments180 GB
Identity140 GB
Checkout120 GB

was 1,572 GB

Logs · Payments and Identity · Mon Nov 16 · ET

Card numbers masked. Logs in your own bucket.

Every line is parsed, masked and linked to its trace on the way in. Fifteen days stay hot; the rest lives in your R2 bucket as Parquet, readable without Porthole.

Raw
Payments
payments-svc10:41:09 amcard verifycard 4111 1111 1111 1111customer 88412trace 5c2e…a90
Parsed
Payments
level infomsg card verifycard 4111 1111 1111 1111customer_id 88412trace_id 5c2e…a90
Masked
Payments
card •••• •••• •••• 1111masked before indexing
Hot index
Payments
searchable 15 days→ trace 5c2e…a90
Archive
Payments
r2://logs-archive/service=payments-svc/date=2026-11-16/Parquet

1.2 TB a day from 46 services
Mon Nov 16 · 21,340 card numbers and 2,906 SSNs masked
raw storage searched for card and SSN patterns: 0 found

Identity

Mon Nov 16, 10:00 am

SOC 2 auditor asks for failed admin logins, Tue Oct 13 – Thu Oct 15.

Identity

Grace Liu · 10:02 am

auth logs Oct 13 – 15 · 96 GB from R2.

searchable at 10:08:40 am · 6 min 40 s

auditor role: read-only, auth logs only.

Identity

DuckDB on the bucket.

the same days read straight from R2.

37 failed admin logins · same count in Porthole

sent to Evidence

Moving in · from Datadog · Mon Oct 12 – Fri Oct 16 · ET

Three lines on 212 hosts. 38 of 38 alerts matched.

Marcus brought the floor’s dashboards, monitors and SLOs across on Monday, and three lines in the Datadog Agent’s Helm values sent all 212 hosts to Porthole too. Every alert fired in both tools all week.

datadog-export · Mon Oct 12 · 20 dashboards + 50 monitors + 6 SLOs

{ "monitors": [
  { "name": "ledger sync lag",
    "type": "query alert",
    "query": "avg(last_5m):avg:ledger_sync.lag{service:ledger} > 30",
    "tags": ["service:ledger", "team:payments"] },
  { "name": "kafka-main consumer lag",
    "type": "query alert",
    "query": "avg(last_5m):max:kafka_lag.messages{service:kafka-main} > 10000",
    "tags": ["service:kafka-main", "team:data"] },
  … ],
  "widgets": [ { "q": "avg:ledger_sync.lag{service:ledger}" } ] }

67 of 70 translated · 3 flagged: 2 monitors on a custom Agent check, 1 dashboard widget

fixed by Marcus Reed Tue Oct 13

6 SLOs, one per team

payments-svc error rate > 1% · Thu Oct 15

Datadog: 11:02:05 am
Porthole: 11:02:31 am · 26 s apart

× 38

38 alerts fired in both, Mon Oct 12 – Fri Oct 16 · each within 1 min · widest gap 48 s

values.yaml · datadog-agent · datadog.env9 clusters · 212 hosts

 datadog:   env:+    - {name: DD_ADDITIONAL_ENDPOINTS,+       value: '{"https://porthole-intake": ["porthole"]}'}+    - {name: DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS,+       value: '[{"api_key": "porthole", "Host": "porthole-intake",+         "Port": 443}]'}+    - {name: DD_APM_ADDITIONAL_ENDPOINTS,+       value: '{"https://porthole-intake": ["porthole"]}'}

 datadog:   env:+  - {name: DD_ADDITIONAL_ENDPOINTS,+     value: '{"https://porthole-intake":+       ["porthole"]}'}+  - {name:+     DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS,+     value: '[{"api_key": "porthole",+       "Host": "porthole-intake",+       "Port": 443}]'}+  - {name: DD_APM_ADDITIONAL_ENDPOINTS,+     value: '{"https://porthole-intake":+       ["porthole"]}'}

Datadog Agent and services’ OpenTelemetry SDKs unchanged · rolled out Mon Oct 12, 11:20 – 11:48 am by Sam Whitaker

Fri Oct 16, 4:00 pm · Agent repointed to Porthole · Porthole’s monitors page through Oncall

$ npx -y @runeditrun/rer dev just-every/run-porthole
✓ Porthole is ready

New Relic and Grafana dashboards, monitors and SLOs come across the same way.

Know why it’s slow. Fix it fast.

Six teams, 46 services. One map.

2,310 ms. 1,840 in one query.

Top query on orders-db since 1:58 pm

2:14:07 pm · orders-db · duration: 1840.2 ms

checkout-svc during this span

checkout-svc v412 · 1:58 pm · GitHub Actions

Paged at 2:16. 410 ms by 2:31.

Agent · 2:19 pm · 5 causes tested · 6 queries cited

Rollback readied 2:20 pm

Killed at 3:07 am. Resized by standup.

Kubernetes event · OOMKilled

Agent · 3:12 am · posted to #inventory, not paged

PR #2207 · search-indexer

45 services traced. Then all 46.

billing-worker

PR #1182 · payments/billing-worker

billing-worker

auth p99 > 150 ms · fired 41 times Oct 12 – 22 · acted on 0

rate-limiter 5xx > 0.5% · flapped 23 times

1.4M new series. Caught in 40 minutes.

Logs a day by team · from Tue Oct 27 · 1,200 GB

Card numbers masked. Logs in your own bucket.

Raw

Parsed

Masked

Hot index

Archive

Mon Nov 16, 10:00 am

Grace Liu · 10:02 am

DuckDB on the bucket.

Three lines on 212 hosts. 38 of 38 alerts matched.

payments-svc error rate > 1% · Thu Oct 15

The busiest day. Every service in view.