Architecture

Deep dive into notifyd internals — how the queue works, how SSE broadcasts, and why we made these choices.

---

High-Level Overview

┌─────────────────────────────────────────────────────────────────┐
│                          notifyd                                 │
│                                                                  │
│  ┌──────────────┐    ┌────────────┐    ┌──────────────────────┐ │
│  │   REST API   │───→│   Queue    │───→│     Connectors       │ │
│  │  (Axum 0.7)  │    │ (Postgres) │    │                      │ │
│  └──────────────┘    └────────────┘    │  ┌──────────────┐    │ │
│         │                    │         │  │ Email (Resend)│    │ │
│         │                    │         │  └──────────────┘    │ │
│  ┌──────────────┐    ┌──────────────┐  │  ┌──────────────┐    │ │
│  │ SSE Broadcast│◄──→│   Worker    │  │  │ SMS (Twilio)  │    │ │
│  │(tokio channel)│   │ (tokio loop)│  │  └──────────────┘    │ │
│  └──────────────┘    └──────────────┘  │  ┌──────────────┐    │ │
│                                        │  │ Push (FCM)    │    │ │
│                                        │  └──────────────┘    │ │
│                                        │  ┌──────────────┐    │ │
│                                        │  │ In-App (DB)   │    │ │
│                                        │  └──────────────┘    │ │
│                                        └──────────────────────┘ │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │                   PostgreSQL 16                           │   │
│  │  jobs │ subscribers │ inbox_messages │ templates │ ...    │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

---

The Queue

Why Postgres Instead of Redis?

Redis + BullMQ is the standard for job queues. But for a notification service doing ~1000 sends/minute (which covers 99% of self-hosted use cases), Postgres is more than fast enough — and you already have it.

SELECT FOR UPDATE SKIP LOCKED is the key. It’s a Postgres feature that:

• Atomically claims rows for processing
• Skips rows already claimed by another worker
• Works like a distributed lock without a separate lock service

SELECT * FROM jobs
WHERE status IN ('pending', 'retry')
  AND scheduled_at <= now()
  AND NOT (channel = ANY($paused_channels)) -- channels a provider asked us to pause (429)
ORDER BY priority ASC, scheduled_at ASC    -- 10 critical … 50 normal … 80 bulk
LIMIT 50
FOR UPDATE SKIP LOCKED

Before every provider request the worker takes a token from the channel’s bucket (EMAIL_RATE_PER_SEC…, one token per call, a batch call counts once). A 429 pauses the channel for Retry-After and re-queues the job without consuming an attempt; the other channels keep flowing. Priorities do not bypass the pause: the provider limit is per account, so a critical email would be refused too. What priorities guarantee is the order at claim time, so when the channel resumes, critical leaves before bulk. With a fallback provider configured the breaker tries it first, and the pause only happens if both refuse.

This gives us:

• At-most-once processing per poll cycle
• Horizontal scaling — run multiple workers, they won’t double-process
• Scheduling — scheduled_at is just a WHERE clause
• Persistence — jobs survive restarts (Redis can’t guarantee this without AOF)

Retry Strategy

Every connector answers with a Delivery or a typed ProviderError (RateLimited, Transient, Permanent, Suppressed); the kind decides:

Kind	What the worker does
accepted	sent, provider + provider_message_id stored on the job
RateLimited (429)	retry without consuming an attempt; channel paused for Retry-After (default 2 s)
Transient (5xx, timeout, network, SMTP 4xx)	backoff below
Permanent (other 4xx, invalid recipient, unverified sender, SMTP 5xx, integrity violation)	failed at once
Suppressed (bounce/complaint list)	failed at once, provider never contacted

Transient backoff, ±20 % jitter:

Attempt	Delay	Total elapsed
1	30 seconds	30s
2	2 minutes	2m 30s
3	10 minutes	12m 30s
4	30 minutes	42m 30s
5	2 hours	2h 42m

After max_attempts (default 5), the job moves to failed status. Check via GET /v1/jobs/:id.

Job Lifecycle

pending → processing → sent ✓
                     → retry → processing → sent ✓
                                           → retry → ... → failed ✗

Cancelled via DELETE /v1/jobs/:id:

pending → cancelled ✓
scheduled → cancelled ✓
processing → (cannot cancel, already being sent)

---

SSE Broadcasting

Why SSE Over WebSocket?

	SSE	WebSocket
Direction	Server → Client	Bidirectional
Complexity	EventSource (5 lines)	ws library + reconnection logic
Proxy support	Works through all proxies	Some proxies break ws
Auto-reconnect	Built-in	Manual
Use case fit	Notifications (one-way)	Chat (two-way)

Notifications are inherently one-way: server pushes to client. SSE is the simpler, more reliable choice.

How It Works

                  ┌─────────────────────┐
                  │   SseBroadcaster    │
                  │                     │
  Worker ────────→│  HashMap<key, tx>   │
  (sends event)   │                     │
                  │  key = "project:sub"│
                  │  tx = broadcast::Tx │
                  └─────┬──────┬────────┘
                        │      │
                   ┌────▼┐  ┌──▼───┐
                   │ rx1 │  │ rx2  │  (multiple tabs/devices)
                   │(SSE)│  │(SSE) │
                   └─────┘  └──────┘

1. Client connects: GET /v1/inbox/:sub_id/stream?token=xxx
2. SseBroadcaster creates a tokio::sync::broadcast channel for this subscriber
3. When a notification is sent to this subscriber, the in-app connector calls broadcaster.send(), which publishes the event with Postgres NOTIFY notifyd_sse
4. Every replica runs a LISTEN task and delivers the event to the browsers connected to it — the tab may be attached to any replica

Cleanup: channels with no receivers are cleaned up periodically (every 2 minutes). Replicas: nothing about a connection lives in one process only (events go through NOTIFY, tickets through the sse_tickets table), so the service scales horizontally.

SSE Auth: One-Time Tickets

Putting a JWT in the URL (query param) is a security concern — it shows up in server logs, browser history, and referrer headers.

notifyd solves this with one-time tickets:

1. POST /v1/inbox/:sub_id/stream-ticket
   → {"ticket": "abc-123"}  (valid 60 seconds)

2. GET /v1/inbox/:sub_id/stream?ticket=abc-123
   → Ticket consumed, stream starts
   → Same ticket can't be reused

---

Connectors

Each connector implements a simple trait:

#[async_trait]
pub trait Connector: Send + Sync {
    async fn send(&self, request: SendRequest) -> Result<()>;
}

SendRequest contains:

• to: recipient (email, phone number, subscriber ID)
• subject: optional (email)
• body: rendered body (template vars already substituted)
• payload: raw JSON for connector-specific fields

Email: Resend

Simple HTTP POST to Resend API. Supports HTML (body_html) and plain text (body).

SMS: Twilio / Telnyx

Swappable via config — change provider = "twilio" to provider = "telnyx" and it works. Same interface, different HTTP calls internally.

Push: FCM

Firebase Cloud Messaging. Requires push token registration via POST /v1/push-tokens.

In-App

No external service. Writes to inbox_messages table and broadcasts via SseBroadcaster.

---

Workflow Engine

Lightweight event-driven workflows. Not Temporal — just what you need for notification sequences.

Workflow Definition

{
  "id": "welcome-series",
  "trigger_event": "user.signup",
  "steps": [
    {"type": "send", "channel": "email", "template": "welcome"},
    {"type": "delay", "duration": "24h"},
    {"type": "send", "channel": "email", "template": "getting_started"},
    {"type": "delay", "duration": "72h"},
    {"type": "condition", "check": "has_completed_onboarding", "if_false": [
      {"type": "send", "channel": "email", "template": "nudge"}
    ]}
  ]
}

Execution

1. Event arrives (POST /v1/workflows/trigger)
2. Engine finds matching workflows
3. Creates a workflow_run record
4. Executes steps sequentially
5. On delay steps: saves state, marks run as paused
6. Worker resumes paused runs when delay expires

State is persisted in Postgres — survives restarts. No in-memory state to lose.

---

Multi-Tenancy

notifyd is designed for running one instance across multiple projects (apps).

Isolation

• Every table has project_id as part of the primary key or foreign key
• API keys are project-scoped
• Subscribers, templates, inbox — all isolated by project
• A subscriber user-1 in project square is different from user-1 in project clozup

Key Rotation

Zero-downtime key rotation:

1. POST /v1/admin/projects/:id/rotate-key — issues a new key; the previous one keeps working as the secondary key until you revoke it
2. Update your app to use the new key
3. POST /v1/admin/projects/:id/revoke-secondary — revoke the old key

Keys are shown once, in the response that issued them. The database holds only their SHA-256 hashes (api_key_hash, secondary_api_key_hash, migration 020); a database dump does not leak project keys. Projects declared in notifyd.toml are compared in constant time from the config file.

---

Security

PII Masking

The pii.rs module masks sensitive data in logs:

• Email: u***@example.com
• Phone: +336***678
• Names: first 2 chars + ***

Audit Log

Every API mutation is logged to audit_log:

• Who (API key / project)
• What (action)
• When (timestamp)
• From where (IP, request ID)

Rate Limiting

In-memory sliding window, per-project. Default 100 req/min. Configurable.

// Middleware checks before every request
if !rate_limiter.check(&project.id, project.rate_limit).await {
    return Err(StatusCode::TOO_MANY_REQUESTS);
}

---

Performance

Benchmarks (single instance, 1 vCPU, 1GB RAM)

Metric	Value
Send throughput	~500 notifications/second
Queue latency (enqueue→process)	<100ms
SSE connection overhead	~2KB per connection
Memory footprint	~15MB idle, ~50MB under load
Cold start	<2 seconds

Scaling

For most use cases (< 10K notifications/hour), a single instance is plenty.

If you need more:

• Run multiple instances pointing to the same Postgres (queue handles coordination)
• Scale Postgres reads with replicas
• For >100K/hour sustained, consider moving to Redis-backed queue (but you probably don’t need this)