daemon package - github.com/a-holm/paceq/internal/daemon - Go Packages

daemon

package
v0.0.0-...-aa3f79c Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Sep 4, 2026 License: Apache-2.0 Imports: 38 Imported by: 0

Documentation

Overview

Package daemon is the long lived process behind paceq serve: one process, one state directory, and a fixed family of loops that share a database.

The rules the package lives by:

  • There is no second execution path. Work runs through engine.ExecuteRun, the same code paceq run uses in its own process.
  • Every timing decision comes from a clock.Clock. No loop reads the wall clock directly, which the architecture guard enforces.
  • The notify bus makes waking fast, never correct. Every loop keeps its ticker, so a lost wake costs latency and nothing else (05 section 3.2).
  • A clean stop hands work back instead of inventing verdicts for it. Steps cut short by a stop go back to pending with their attempt restored, and claimed runs go back to the queue without counting a crash.

Index

Constants

View Source
const DefaultNotifierConfigDir = "/etc/paceq"

DefaultNotifierConfigDir is where a system-wide configuration looks when the state directory carries none.

View Source
const DeliveryBatch = 20

DeliveryBatch is how many notifications one claim hands out. Twenty matches the issue sketch: a batch is small enough that one slow target cannot monopolise a wake, large enough that bursts do not tick by.

View Source
const ExitHardStop = 130

ExitHardStop is the exit code of a daemon that got a second stop request and answered it by killing every process group at once. It sits above the shell signal band, so "killed by insisting" can never be confused with the exit codes paceq hands out on its own (03 section 7.2).

View Source
const MaxDrainBatches = 8

MaxDrainBatches bounds how much work ONE wake may start. The claim's visibility window returns anything unfinished, so stopping here is safe: the next wake picks the rest up exactly where the bookkeeping says.

View Source
const NotifierFileName = "config.yaml"

NotifierFileName is the configuration file that names notifiers and defaults. It lives in the state directory first (portable per project) and falls back to the system configuration directory (/etc/paceq).

Variables

This section is empty.

Functions

func HostName

func HostName() string

HostName captures the machine identity stamped into payloads once, so a churning hostname mid-process cannot make two alerts disagree about where they came from. Empty stays empty in the payload: the field documents the machine when it knows its own name.

func SensorSpecFromRow

func SensorSpecFromRow(row store.SensorSummary) (sensor.Spec, error)

SensorSpecFromRow turns a sensor row into the runnable shape the evaluator needs. exec_json holds the exec adapter's configuration as JSON: either the frozen M3 contract object {"run":[...], "working_dir":..., "env":{...}} or a bare argv array for rows seeded before that object form.

The whole declared contract travels, not just the command: the frozen sensor contract (docs/reference/sensor-contract.md) promises that any variable the sensor declared is visible to the subprocess and that workdir is where it starts. Dropping either would evaluate a different program than the one applied.

func Serve

func Serve(ctx context.Context, cfg Config, clk clock.Clock) error

Serve runs the daemon until the context is cancelled or a loop fails, then stops cleanly. The startup order is the one the reliability plan fixes (02 section 5.9): lock, verify pragmas and migrate, open the session row, converge what a crash left behind, start the loops, report ready.

func ValidateMetricsListen

func ValidateMetricsListen(addr string) error

ValidateMetricsListen refuses any --metrics-listen address that is not loopback (#40, 08 section 6). The metrics endpoint is an opt-in TCP surface: an operator who asks for one asked for loopback or made a mistake, and a mistake here must end in an explanation, not in a daemon listening on 0.0.0.0. Hostnames are resolved, so "localhost" works and anything that resolves partly off the loopback is refused.

Types

type Config

type Config struct {
	// Version lands in the session row and in /readyz, so an operator can
	// tell which binary has been running since when.
	Version string

	// StateDir holds the lock file, the database and the logs. It is the only
	// required field.
	StateDir string

	// JobsDir is where job files live. The scheduler reads it from M2-05 on;
	// until then it is accepted and carried.
	JobsDir string

	// ConfigDir is the directory paceq reads configuration files from.
	// Empty means /etc/paceq when running under systemd, or the working
	// directory otherwise.
	ConfigDir string

	// RuntimeDir is where transient runtime files live (the unix socket).
	// Empty means /run/paceq when running under systemd, falling back to
	// the state directory.
	RuntimeDir string

	// SocketPath enables the health endpoints over a unix socket when not
	// empty. Empty means disabled for now: the runtime directory contract
	// arrives with the systemd work, and nothing else should invent paths.
	SocketPath string

	// MetricsListen is the opt-in TCP bind for /metrics (#40). Empty means
	// the endpoint answers on the unix socket only, which is the default
	// the security plan fixes (08 section 6: no TCP listener in the MVP
	// without an explicit operator decision). When set, it is validated to
	// loopback before anything starts; anything else is refused with an
	// explanation, never bound.
	MetricsListen string

	// Workers is how many runs may execute at once. Zero means
	// runtime.NumCPU(). One dispatcher hands work out no matter what; this
	// caps the executors behind it.
	Workers int

	// DrainTimeout bounds phase two of the shutdown: how long executors may
	// keep their process groups while they finish. Zero means 30s.
	DrainTimeout time.Duration

	// KillGrace is the SIGTERM to SIGKILL gap inside every step's process
	// group during the drain. Zero leaves the runner's own default.
	KillGrace time.Duration

	// Shadow turns the whole instance into a recorder (#32): the scheduler
	// still plans, claims fire-times and advances cursors, but no run is
	// ever materialised and nothing executes. The marker is persisted on
	// startup so status and reports in other processes see it too.
	Shadow bool

	// Observe names where observed cron behaviour comes from (#32):
	// none (default), journald or file=<path>. Only read when Shadow is
	// set; observation capture never runs for a normal instance.
	Observe string

	// SensorMaxParallel is the global cap on concurrent sensor evaluations.
	// Zero means four.
	SensorMaxParallel int

	// TickInterval is the safety net period shared by the loops: how often a
	// loop looks at the world without being woken. Zero means 1s.
	TickInterval time.Duration

	// HeartbeatEvery is how often the session row's last_seen_at moves
	// forward. Zero means 10s.
	HeartbeatEvery time.Duration

	// LeaseTTL is the lease an executor claims per run. Zero means sixty
	// seconds, renewed every twenty by the daemon's renewal loop. Tests
	// shorten it; production should not.
	LeaseTTL time.Duration

	// RenewInterval overrides how often held leases renew. Zero means a
	// third of the ttl.
	RenewInterval time.Duration

	// ClockSkewAllowance is how long past the expiry the reaper waits before
	// it takes a run. Zero means ten seconds.
	ClockSkewAllowance time.Duration

	// RequeueBackoff is how long a reaped run waits before it is due again.
	// Zero means thirty seconds.
	RequeueBackoff time.Duration

	// ReapEvery is how often the reaper sweep looks for expired leases.
	// Zero means ten seconds.
	ReapEvery time.Duration

	// ReconcileEvery is the safety-net cadence for periodic reconciliation
	// (issue #62), riding under the reaper's role lease. Zero means thirty
	// seconds.
	ReconcileEvery time.Duration

	// MaxCrashCount is the poison quarantine line for runs that keep dying
	// with their executor. Zero means five.
	MaxCrashCount int

	// Owner names the claim holder in the database. Empty means serve:<pid>.
	Owner string

	// DisableNotifyBus is the --no-notify-bus switch. With it set the loops
	// run on their tickers alone, which is the standing proof that the bus is
	// an optimisation and never a dependency.
	DisableNotifyBus bool

	// Policies carries the retention configuration keys (issue #36). Zero
	// fields fall back to the shipped defaults, so a config that says
	// nothing keeps the documented horizons.
	Policies store.Policies

	// Limits carries the disk-guard's four configuration keys (#44) from
	// config.yaml's limits section. Zero fields fall back to the shipped
	// defaults; the guard applies them.
	Limits obs.DiskLimits

	// NightlyHour is the local hour the maintenance cycle aims for. Zero
	// means 03:00 (07 section 6.5). Tests move it to make a slot due.
	NightlyHour int

	// Signals carries copies of the process signals. When set, the daemon
	// watches it for the second stop request: two signals mean the operator
	// insists, and every process group gets SIGKILL before ExitHardStop. Nil
	// disables the watcher.
	Signals <-chan os.Signal

	// OnHardStop ends the process after the hard kill. Tests record the call;
	// production leaves it nil for os.Exit(ExitHardStop).
	OnHardStop func()

	// Logger receives the structured lines. JSON on stderr is what journald
	// wants (06 section 5); nil means slog's default.
	Logger *slog.Logger
}

Config is everything Serve needs beyond the state directory. The zero value runs with defaults: workers equal to the CPU count, a 30 second drain, and tickers at one second.

type Heart

type Heart struct {
	// contains filtered or unexported fields
}

Heart tracks the scheduler loop aliveness. Beat records each real tick. The watchdog loop reads the last beat time and only sends WATCHDOG=1 when fresh enough. Slow operations (backup, migrations, GC) must never touch this Heart: only the scheduler loop updates it, so a slow backup can never cause a watchdog timeout restart loop.

The clock is injected so the Heart never calls time.Now or time.Since directly.

func NewHeart

func NewHeart(clk clock.Clock) *Heart

NewHeart returns a Heart for the given clock.

func (*Heart) Beat

func (h *Heart) Beat()

Beat records one scheduler tick. Called from the scheduler loop on every real decision (a Tick that materialises work or decides nothing to do).

func (*Heart) LastTick

func (h *Heart) LastTick() time.Time

LastTick returns the wall time of the last beat. Zero if never beaten.

func (*Heart) SinceLastBeat

func (h *Heart) SinceLastBeat() time.Duration

SinceLastBeat returns the elapsed wall time since the last beat.

func (*Heart) TickCount

func (h *Heart) TickCount() int64

TickCount returns the total number of beats since start.

type LoopStatus

type LoopStatus struct {
	Name     string
	Ticks    int64
	LastTick time.Time
}

LoopStatus is one loop's line in a snapshot.

type NotificationConfig

type NotificationConfig struct {
	Notifiers map[string]*notify.ExecNotifier // type=exec targets
	Stderr    []string                        // names bound to stderr output

	Entries  map[string]notifierFileEntry
	Defaults model.NotifyDefaults

	// Limits carries the disk-guard's four configuration keys (#44) from
	// the same file. Zero values mean the shipped defaults.
	Limits obs.DiskLimits

	// Timeouts mirrors Entries keyed by name, parsed and defaulted.
	Timeouts map[string]time.Duration
}

NotificationConfig is the whole [notifiers + defaults] document.

func LoadNotificationConfig

func LoadNotificationConfig(stateDir, configDir string) (*NotificationConfig, error)

LoadNotificationConfig reads the notification configuration from stateDir/config.yaml or, failing that, configDir/config.yaml. A missing file everywhere is NOT an error: it means notifications are simply not configured, which is every installation's honest starting point. Bad syntax or bad values refuse loudly, because a silently dropped alert is the exact failure this milestone exists to prevent.

type Notifications

type Notifications struct {
	Store *store.Store
	Clock clock.Clock
	Log   *slog.Logger

	Config *NotificationConfig

	Host string

	StderrOut io.Writer
	// contains filtered or unexported fields
}

Notifications carries everything the dispatch + SLA loops need. It fails closed: an unknown target becomes permanently failed history (visible in `notifications list`, counted by pulseq_notifications_failed_total), never silence (#29 AC five).

func NewNotifications

func NewNotifications(st *store.Store, clk clock.Clock, log *slog.Logger,
	cfg *NotificationConfig, errOut io.Writer,
) *Notifications

NewNotifications builds the service from loaded configuration. A nil cfg means nothing is configured: the loops still run, find no rows the CLI paths could write anyway, and cost one indexed probe per tick.

func (*Notifications) SendTest

func (n *Notifications) SendTest(ctx context.Context, target string) (string, error)

SendTest delivers one synthetic event through the named target without touching the outbox. It is what `paceq notifications test` runs: the same registry, timeout and wire envelope the dispatcher uses, so a passing test proves the wiring rather than a copy of it. The payload comes back so both output modes can show exactly what left the building.

func (*Notifications) SyntheticTestMessage

func (n *Notifications) SyntheticTestMessage(target string) notify.OutboxMsg

SyntheticTestMessage builds the event `notifications test` sends: every contract field filled with an honest "this is synthetic" value.

type ScheduleSource

type ScheduleSource interface {
	Tick(ctx context.Context) error
}

ScheduleSource is the seam M2-05 fills: whatever turns due schedules into triggers. Nil means idle, which is correct for this milestone: nothing is materialised until that lands, and the loop still ticks so the topology, the heartbeat and the health surface are all live before then. It stays an interface here so the daemon never learns cronx's shape early.

type SensorCommitResult

type SensorCommitResult struct {
	store.SensorTickCommitResult

	Outcome   string
	Truncated bool
	Dropped   int
}

SensorCommitResult is what one recorded evaluation did: the store's answer, plus the two facts the caller cannot recover from it. Outcome is the word written to ticks.outcome, and the truncation pair says whether the ceiling cut this batch, so a forced tick can tell the operator it was partial.

func CommitSensorEvaluation

func CommitSensorEvaluation(ctx context.Context, st SensorCommitter, in SensorEvaluation) (SensorCommitResult, error)

CommitSensorEvaluation translates one evaluation into the store's commit input and writes it. It is the single translation from a sensor.Result into tick, trigger, run and cursor rows, so a daemon evaluation and a forced `paceq sensors tick` can never disagree about an outcome, a reason code, a skip reason, the trigger ceiling or when the sensor is due again.

The ceiling is applied here rather than by either caller (#215). It is the one point both evaluations pass through, and applying it twice would erase its own record: the second pass sees a batch inside the budget and reports nothing was dropped.

type SensorCommitter

type SensorCommitter interface {
	CommitSensorTick(ctx context.Context, in store.SensorTickCommitInput) (store.SensorTickCommitResult, error)
}

SensorCommitter is the write half of one finished sensor evaluation.

type SensorEvaluation

type SensorEvaluation struct {
	Row    store.SensorSummary
	Begin  store.BeginSensorTickResult
	Result sensor.Result
	Now    time.Time
}

SensorEvaluation is one finished evaluation, ready to be recorded: the sensor row the runs are written against, the intention row it is closed against, and what the evaluation decided.

type WatchdogNotifier

type WatchdogNotifier interface {
	Send(msg string) error
}

WatchdogNotifier is the seam for production to send watchdog pings.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL