Building callduty: Phone-Call Alerting Without the PagerDuty Bill

The Problem

Recently we got a PagerDuty bill of $10k/year, which is not much if you are a big org. But it is huge for a small team, and that was exactly the scenario for us. And this kindled the spark to create an internal tool for similar functionality.

Now, PagerDuty as a product is quite simple:

Alert → Phone call → Acknowledgement

For us on-call rotation wasn't that important, so we just needed a basic service which could alert us with a call in odd hours, whenever there was a critical alert.

The Existing Setup

We had an alerting flow already in place, which means:

Service (any log/metric generator) → SigNoz, ELK (any logging platform) → Slack, email (notification channel)

To enable a call as a notification channel we require a CPaaS — i.e. a Communication Platform as a Service.

For that I used Twilio, which was exactly a fit for my use case:

• Low cost
• Easy code integration
• Easy to manage

Now once the CPaaS is in place we can go forward with the tooling, which is going to utilize Twilio for calling.

Service Architecture

We are going to deploy this service and let it poll alert status (firing / healthy) from our logging platform.

Key requirements for this service are:

• API to fetch alert status
• Alert to monitor
• Alert to POC (person of contact) relation
• POC to phone number relation
• Twilio config

Put together, the service is just a small loop. Every minute it asks the logging platform which alerts are firing, and for each alert we care about it checks how long it has been firing. Once that crosses a threshold we set (say 5 minutes), it places a Twilio call to the person on call for that alert, then posts to Slack whether the call was picked up.

Two of those requirements are just small config files — one maps an alert to a person and a wait time, the other maps a person to a phone number. The rest (Twilio keys, the Slack webhook, the logging platform API key) are environment variables.

The One Tricky Part

The part worth care is not calling someone twice for the same incident. So before it dials, it records that a call is about to happen; if it crashes mid-call it won't dial again. When the alert resolves the state resets, so the next time it fires it is treated as a fresh incident.

The Cost

Instead of a flat $10k/year, we now pay Twilio per call — a number is about $1/month and a call is a few cents. For a small team that gets paged a handful of times a month, that is basically nothing.

Open Source

I have open sourced the tool, callduty, here:

github.com/lakshayyyyyyy/callduty

It is a single Python service, runs as a container or straight on a host, and only needs a Twilio account and a logging platform with an alerts API (we use SigNoz).