Project alert rules evaluated every 5 minutes from stored metrics and audit events, incidents, and notification channels
Last updated November 21, 2026
Alerting
Flaggr alerts on project alert rules: conditions over the evaluation metrics and audit events the control plane stores, checked every 5 minutes. A breach opens an incident (an alert firing) and notifies the rule's channels. Incidents move Open → Investigating → Resolved and feed the project's status page.
On flaggr.dev, the scheduled checks start once alert evaluation is switched on (see Running the evaluator). Until then, rules are saved and listed, but no incidents open and rules send no notifications.
GET /api/alertswas retired and returns410 Gone: it evaluated one server instance's in-process counters. Use the project endpoints below, or the Prometheus metrics further down.
Project alert rules
curl -X POST https://flaggr.dev/api/projects/proj-123/alerts \
-H "Authorization: Bearer $FLAGGR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "Checkout error rate",
"severity": "critical",
"conditionType": "error_rate",
"threshold": 5,
"windowMinutes": 15,
"cooldownMinutes": 30,
"channels": [{ "channelId": "ch-oncall", "channelName": "On-call Slack" }]
}'conditionType | Threshold unit | Fires when |
|---|---|---|
error_rate | Percent (1 = 1%) | The share of evaluations that errored in the window is above the threshold; needs at least 20 evaluations |
p95_latency | Milliseconds | Latency in the window (each minute's stored p99, weighted by traffic) is above the threshold; needs at least 20 evaluations |
evaluation_count | Evaluations | More evaluations than the threshold were recorded in the window |
toggle_drift | Percentage points | The error rate in the window after a toggle (through the toggle endpoint or a trigger) rose by more than the threshold compared with the window before |
auto_rollback | Rollbacks | Automated rollbacks in the window exceed the threshold: the circuit breaker, site guardrail mitigation, rollout guardrails and canary aborts, manual ones included. 0 fires on any rollback |
new_issue | Issues | More error issues than the threshold were seen for the first time in the window, from browsers or servers. Issues someone has already resolved, ignored or reopened don't count. 0 fires on any new issue |
issue_regression | Issues | More resolved issues than the threshold were seen again after they were resolved. 0 fires on any regression |
issue_events | Events | The busiest issue had more events than the threshold in the window. Ignored issues don't count while they are ignored |
crash_free_rate | Percent (99.5 = 99.5%) | Fires below the threshold: the share of browser sessions in the window without an unhandled error dropped below it. It uses the sessions @flaggr/sdk's error tracking reports, or the analytics script's when there are none. Needs at least 20 sessions |
Severity is critical, warning or info. cooldownMinutes defaults to 15. List rules with GET /api/projects/{id}/alerts; update and delete them with PATCH and DELETE /api/projects/{id}/alerts/{ruleId}. Each rule in a response carries its latest scheduled check under evaluation. A project can have at most 100 enabled rules: creating or enabling one past that returns 400 (disabled rules don't count).
How evaluation works
- Metric conditions (
error_rate,p95_latency,evaluation_count) count only evaluations recorded by the control plane (flaggr.dev). Evaluations served by api.flaggr.dev or evaluated locally in an SDK aren't counted, so rules for those projects report no data. - Windows are at least 5 minutes (15 for the event conditions
toggle_drift,auto_rollback,new_issueandissue_regression) and end at the last fully recorded minute. - Event conditions count only events not yet reported, and catch up after delayed runs (up to 24 hours). Events that breach within the cooldown of a resolved incident are held, not dropped: every run reads them again and they open the next incident once the cooldown has passed. For this, a rule's event range can reach back as far as its window plus its cooldown plus an hour.
toggle_driftcounts a toggle as an event only when its error-rate rise is above the threshold. Healthy toggles, and toggles still waiting for post-toggle data, don't keep an incident open, and none is lost. Each is measured on every run while it's within the window, also after an incident fired or noted more events (they're kept as pending toggles; a self-hosted deployment needs migration 044 for this).- Each run checks rules round-robin across projects (every project's first rule before any project's second), so one project's rules can't use up the run.
- No data means no change: an incident is neither opened nor resolved.
Incidents
- A breach opens one incident and notifies the rule's channels once. An unresolved incident isn't notified again.
- A new incident for the same rule opens no sooner than
cooldownMinutesafter the last one.toggle_drift,auto_rollback,new_issueandissue_regressionevents in that time open it when the cooldown ends. - An Open incident resolves itself when the condition clears (event conditions: after a quiet window), with a timeline note and an audit entry.
- An Investigating incident only gets "Condition cleared" or "Condition breached again" notes, at most one per max(cooldown, 15) minutes.
- Resolved is final. Deleting or disabling a rule resolves its unresolved incidents.
- Open incidents of deleted or disabled rules that weren't resolved then are resolved by the next run; Investigating ones wait for a person.
- There are no recovery notifications yet.
GET /api/projects/{id}/alert-firings?state=open|investigating|resolved&severity=&ruleId=&limit=50 lists incidents newest first (limit 1–200). The filters apply before the limit; invalid values return 400.
PATCH /api/projects/{id}/alert-firings/{firingId} changes one incident:
{ "state": "investigating" }acknowledges it;"resolved"resolves it;"assigneeId"assigns it.
A state change that loses a race with an auto-resolve returns 409 (Incident was resolved meanwhile, with the stored firing and notes) and changes, notes and audits nothing. An assigneeId sent in the same request isn't applied either. An assignment on its own also applies to a resolved incident.
/alert-firings/{firingId}/notes holds the timeline.
GET /api/projects/{id}/alert-firings/{firingId} also returns the incident's impact. For error_rate rules it includes the browser errors your Sites' analytics recorded in the firing window (impact.browser: sessions, error events, top browser and page). It counts only events sent from the Site's own domain; impact.browserExcluded says how many from other origins it left out, and the console shows that number in the incident. Project members can count them too with ?includeUntrusted=1. Rules are evaluated from the control plane's evaluation metrics and audit events, never from browser analytics.
Legacy conditions
Existing grpc_errors and cache_hit_rate rules are kept but never evaluated; the console shows them as Not evaluated. New rules can't use them (400).
Running the evaluator
GET or POST /api/cron/alerts with Authorization: Bearer $CRON_SECRET evaluates every enabled rule once. It returns 503 when CRON_SECRET isn't set. A live run needs ?dryRun=0 (or false): without dryRun, or with ?dryRun=1, it evaluates without recording incidents, notifying or writing state. ?projectId= limits the run to one project. Any other parameter (dry_run, dryrun, projectID, projectIds, project …), other values, repeated parameters and an empty projectId return 400; an empty projectId never means every project. Vercel's x-vercel-protection-bypass parameters are ignored. A run in which a project or the orphan sweep failed, or every checked rule errored, returns 500 (error: projects_failed, sweep_failed or rules_failed) with the run's IDs, counts and codes.
On flaggr.dev, a step of the rollout-tick workflow runs it with ?dryRun=0 every 5 minutes once the repository variable ALERT_EVALUATION_ENABLED is true (dispatch alert-evaluation.yml with dry_run first); a failed run turns the workflow red. Self-hosted deployments should call it with ?dryRun=0 every 5 minutes.
Prometheus Metrics
To alert on platform-wide metrics, scrape /api/metrics and build alerts in Grafana, Datadog or any Prometheus-compatible tool. Prometheus scrapers present METRICS_BEARER_TOKEN as a bearer token; platform admins can also read it from a dashboard session.
/api/metrics is platform-wide (every project's flag keys appear as labels), so API tokens can't read it.
scrape_configs:
- job_name: flaggr
scheme: https
metrics_path: /api/metrics
authorization:
type: Bearer
credentials_file: /etc/prometheus/flaggr-metrics-token # contains METRICS_BEARER_TOKEN
static_configs:
- targets: ["flaggr.dev"]# Error rate
sum(rate(flaggr_evaluation_errors_total[5m])) / sum(rate(flaggr_evaluations_total[5m]))
# P95 latency
histogram_quantile(0.95, sum(rate(flaggr_http_request_duration_seconds_bucket[5m])) by (le))
# Cache hit rate
sum(rate(flaggr_cache_operations_total{result="hit"}[5m])) / sum(rate(flaggr_cache_operations_total[5m]))
# Active gRPC connections
flaggr_grpc_active_connectionsNotification Channels & Providers
Flaggr supports multi-channel notification routing at both the Organization and Project levels:
| Provider | Mechanism | Features |
|---|---|---|
| Slack | Incoming Webhook | Rich Block Kit formatting, status color-coding, interactive links |
| Discord | Webhook | Embed cards with severity colors, metric fields, and timestamps |
| PagerDuty | Events API v2 | Automatic event deduplication, incident trigger, and custom metric details |
| Direct / Resend | Automated email dispatch with styled HTML status cards and recipients list | |
| Webhook | HTTP POST | HMAC-SHA256 signature verification (X-Flaggr-Signature), JSON payload |
Organization-Level Inheritance
Configure global alert channels once in Organization Settings (/console/org/[slug]/settings). All projects within the organization automatically inherit these channels without duplicating secrets or webhook URLs.
- Inherited channels are marked with an
Org-widebadge in project alert dashboards. - Project-level rules can target either project-specific channels or inherited organization channels.
Channel secrets
Channel secrets — Slack and Discord webhook URLs, the generic webhook URL and signing secret, the PagerDuty routing key — are visible only to project admins/owners (org owners/admins for org channels) using the dashboard or an admin-tier credential. Members, viewers, read/write API tokens and visitors of the shared demo see [redacted].
Editing a channel without retyping its secret keeps the stored value. Changing where the channel sends it (its URL, recipients or type) while keeping the [redacted] placeholder needs the same manage-settings access; otherwise retype the secret. Visitors of the shared demo can't create, change, delete or test the demo project's channels.
Visitors of the shared demo (and showcase) can read alert rules and incidents but can't create, change, enable or delete rules, add incident notes or assign incidents (403); they can move an incident between states. Rules, incidents and timelines show other people as redacted / "Another member" — no names, emails or user ids — and hide other people's note text; the visitor's own entries and the alert evaluator's are shown. Visitors also see notification channels without their recipients: email addresses (recipients, to, cc, bcc, including channels the showcase inherits from its organization) and direct @-mentions read [redacted], as do webhook URLs and keys.
Delivery Reliability Engine
Flaggr's AlertNotifier protects downstream endpoints with enterprise delivery semantics:
- Retries: channel tests and direct operational alerts get 3 attempts (1s, 2s backoff). Scheduled rule notifications get 2 attempts with a 7-second timeout, and are re-sent on the next run if a run stopped before sending.
- Per-Channel Circuit Breakers: Automatically opens after 3 consecutive failures to prevent cascade exhaustion; half-opens after 5 minutes.
- One incident per breach: overlapping evaluations are serialised per rule (advisory lock and lease).
- Dead-Letter Audit Trail: Records delivery failures in
flaggr.alert_delivery_failuresfor post-incident analysis. - Destination limits: Slack headers are cut to 150 characters (the full rule name follows as plain text), Discord titles to 256 and PagerDuty summaries to 1024, so a long rule name never makes a destination reject the alert.
Automated Guardrail Circuit Breaking & Auto-Rollback
When a flag toggle causes an unexpected regression in production, Flaggr's circuit breaker triggers immediate automated mitigation:
POST /api/flags/{flagKey}/auto-rollback
Content-Type: application/json
{
"serviceId": "svc-checkout",
"environment": "production",
"maxErrorRateDelta": 0.02,
"maxLatencyDeltaMs": 200
}- Drift Analysis: Compares symmetric pre- and post-toggle evaluation metrics.
- Automated Rollback: If health is
degraded, or the error rate rose by at leastmaxErrorRateDelta(default 0.02, 2 percentage points) or P99 latency by at leastmaxLatencyDeltaMs(default 200 ms), Flaggr reverts the flag to its previous version snapshot. Rollout-engine rules are never brought back from the snapshot, and leftover ones of ended plans are removed (removedRolloutRuleIds). When a live rollout plan manages the flag in that environment, Flaggr cancels the plan instead, which removes only the plan's rule. In an environment that requires approval nothing is restored (approvalRequired: true); roll back to a version, which opens a change request. - Emergency Alert Broadcast: notifies the project's enabled alert channels. Organization channels are not included; route them with an
auto_rollbackalert rule.
Thresholds must be positive: maxErrorRateDelta above 0 and at most 1, maxLatencyDeltaMs above 0, and windowMinutes above 0 and at most 1440. Other values return 400.
Related
- Caching Strategy — Cache health and circuit breaker
- Rate Limiting — Rate limit monitoring
- Post-Toggle Safety & Drift Diagnostics — Post-toggle impact analysis
- REST API Reference — Alert endpoint details