Projo FHIR Documentation - Local Development build (v0.1.0) built by the FHIR (HL7® FHIR® Standard) Build Tools. See the Directory of published versions

Operations

Operations

Queue architecture

13 queues (6 main + 7 poison):

• fhir-events, cr-status-events, identity-resolution-events (projo)
• cr-events, wa-inbound, wa-status, wa-reminder-checks (sidecar)
• plus each *-poison

Timer functions

• sweeper: every 5 min (both apps, different responsibilities)
  • Projo: subscription reconciliation, CR revocation, extraction self-healing
  • Sidecar: reminder checks, CR pickup
• carePlanMaintenance: daily at 2:00 AM (renews due-event subscriptions, 30-day window)

Alerts → Teams routing

Azure Monitor → Action Group (webhook) → Logic App (transforms to MessageCard) → Teams Power Automate webhook → Teams channel. Per-env Logic App + PA webhook. Theme colors by severity (Sev0 red, Sev1 orange-red, Sev2 dark orange, Sev3+ blue, Resolved green).

Alert catalogue

• Queue & backlog: sweeper-cr-backlog, sweeper-reminder-backlog, clinical-extraction-stuck, queue-saturation-exhausted, sidecar-missed-cr-over-age-cap, queue-depth-backlog (>500), queue-message-age (>900s), poison-queue, storage-queue-count
• PostgreSQL: cpu, memory, storage, connections (>50), long-query (>60s), deadlocks, iops
• Functions: high-error-rate, fire-arrow-http-5xx, fire-arrow-response-time, fire-arrow-http-429, orchestrator-heartbeat, whatsapp-failures, fhir-unavailable, sweeper-failures
• Control plane: function-delete, service-health

App Insights caveats

Node v4 Flex Consumption Function Apps do NOT emit requests rows (no public HTTP server on worker). Reconstruct invocations from traces + dependencies + customMetrics. Classic Application Insights uses lowercase table names.

Common issues

• FHIR unavailable: Fire Arrow 5xx or 429; saturation-retry handles transient; persistent → check Postgres/HikariCP
• Reverse CR stuck: identity-resolution-events queue backlog; check identityResolution handler
• Extraction stuck: completed QRs lacking Provenance; self-healing sweep handles; alert after threshold
• WhatsApp send failures: sidecar logs; check WABA status, template approval, 24h window
• Poison queue: inspect via manage-poison-queues.sh; fix root cause before replay