Beacon reference
An ultra-lightweight time-series database for IoT, edge devices and homelabs. This page is the complete reference: build, ingest, query, stream, and what the storage layer does underneath.
- Binary
- 2.89 MB
- Embedded UI
- 35.1 KB
- Tests
- 71/71
- License
- MIT
Overview
Beacon is a small time-series database you can drop onto anything with a CPU. It ingests sensor readings over HTTP, compresses them heavily, and serves them back with live streaming — all from a single executable with no runtime and no external services.
Reach for it when the data is small, the machine is small and the network may not be there: ESP32 fleets, home-lab monitoring, offline edge gateways, bench instrumentation. It is a deliberate trade: a compact single-node store with numeric fields only, not a distributed analytics cluster.
What you get out of the box
- Ingest — InfluxDB-compatible line protocol, single points or multi-line batches, validated atomically before anything is written.
- Store — Gorilla compression (delta-of-delta timestamps + XOR floats) inside immutable segments, protected by a CRC-framed write-ahead log.
- Query — time-range reads with optional
avg/min/max/count, plus block-skip pruning by time range. - Stream — server-sent events push every ingested point to browsers, with a lag notice instead of back-pressure.
- Watch — a 35.1 KB dashboard compiled into the binary, served at
/. No CDN, no npm, no fonts.
One process owns one data directory: a write-ahead log and immutable Gorilla-compressed segments. The HTTP API, the SSE stream and the dashboard all live in that same process.
| Route | Method | Purpose |
|---|---|---|
/api/v1/write | POST | Ingest line protocol, single line or batch. |
/api/v1/query | GET | Time-range read with optional aggregation. |
/api/v1/metrics | GET | Every known series with counts and bounds. |
/api/v1/stats | GET | Engine statistics and storage footprint. |
/api/v1/health | GET | Liveness probe. |
/api/v1/stream | GET | Server-Sent Events of ingested points. |
/ | GET | Embedded dashboard (404 with --no-ui). |
Errors are JSON with a single error string. Protocol and query problems return 400; anything else is500.
Installation & prerequisites
There is nothing to set up beyond getting the binary on the machine. Pick the path that matches the box you are deploying to.
| Requirement | Required for | Notes |
|---|---|---|
Rust 1.85+ via rustup.rs | Methods 1 · 2 · 3 | Any source-based path. Edition 2024 is required. |
Cargo | Methods 1 · 2 · 3 | Ships with rustup, so installing Rust covers this. |
git | Method 2 | Or download the source archive from Releases. |
| Nothing — prebuilt binary | Skipping the toolchain | Windows x64, Linux x64, macOS arm64 from GitHub Releases. |
| Nothing at runtime | Any method | No runtime, no Node, no CDN, no external assets. |
1. Install with Cargo RECOMMENDED
Pulls the crate from GitHub and drops a beaconexecutable into ~/.cargo/bin:
cargo install --git https://github.com/prxmel4/beacon2. Build from source
git clone https://github.com/prxmel4/beacon.git && cd beacon
cargo build --release
# the binary lands at:
# Windows .\target\release\beacon.exe
# Linux / macOS ./target/release/beacon3. Run without installing
Straight from the crate, no install step. Handy while iterating:
cargo run --release -- serve --port 8086 --data-dir ./beacon-dataConfirm it worked
Any of the three paths should answer with the version embedded at build time:
beacon --version
# -> beacon 0.1.0Prefer a prebuilt binary
Download the archive for your platform fromGitHub Releases, extract it, and runbeacon serve. No toolchain, no compilation — the only file you ship is the executable.
Your first 60 seconds
Four steps, in order. Each one takes a few seconds; the whole walkthrough is under a minute on a laptop.
- Start the server. Pick a port and a data directory:Shell
beacon serve --port 8086 --data-dir ./beacon-data # -> listening on http://0.0.0.0:8086 # -> dashboard: http://0.0.0.0:8086/ - Open the console. Browse tohttp://127.0.0.1:8086. The KPI strip shows
TOTAL POINTS 0— nothing has been written yet, which is correct. - Press MOCK SENSOR in the top-right corner. The button turns red and three synthetic series (temperature, humidity, CPU) start arriving every 500 ms: the KPI counters climb, the LIVE dot pulses per frame, and the canvas draws a 60fps trace. Leave it running — it is also your live data source for the next step.
- Write something yourself, then read it back. This is the same request path a real device would use:curl
curl -sS -X POST http://127.0.0.1:8086/api/v1/write \ -H 'Content-Type: text/plain' \ --data-binary 'temperature,device=bench-01,room=lab value=21.5 1732250000000' # -> {"points_written":1}curlcurl -sS 'http://127.0.0.1:8086/api/v1/query?metric=temperature&tag=device=bench-01' # -> {"series":[{"measurement":"temperature", # "tags":[["device","bench-01"],["room","lab"]], # "field":"value", # "points":[{"timestamp":1732250000000,"value":21.5}]}]}
That is the full loop: write → store → query. Ctrl+C on the server triggers a graceful flush; the next start replays the WAL and picks up where it left off.
CLI reference
Two subcommands: serve and bench.
beacon serve
| Flag | Default | Description |
|---|---|---|
--port <u16> | 8086 | TCP port to listen on. |
--host <addr> | 0.0.0.0 | Bind address. Use 127.0.0.1 to stay loopback-only. |
--data-dir <path> | ./beacon-data | Directory for the WAL and segment files. |
--flush-threshold <n> | 1000 | Flush the MemTable to a segment after n buffered points. |
--no-ui | off | Headless: GET / returns 404, the embedded UI is never served. |
beacon bench
| Flag | Default | Description |
|---|---|---|
--points <n> | 1000000 | Total points to ingest across all series. |
--devices <n> | 8 | Simulated devices; each produces temperature, humidity and cpu. |
--batch-size <n> | 1000 | Lines per write batch — one WAL fsync per batch. |
--flush-threshold <n> | 50000 | MemTable flush threshold during the benchmark run. |
--queries <n> | 200 | Random 1%-window aggregation queries for the latency report. |
--data-dir <path> | temp | Keep the generated data here instead of a cleaned-up temp dir. |
Headless mode
Pass --no-ui and GET /returns 404. The API, the SSE stream and the storage engine are untouched — only the dashboard route is disabled. Use it on fleet nodes where the UI is noise, or where a reverse proxy owns the root path.
beacon serve --host 127.0.0.1 --port 8086 --data-dir /var/lib/beacon --no-uiLine protocol
InfluxDB-compatible syntax, numeric fields only:
measurement[,tag=value...] field=value[,field=value...] [timestamp_ms]
# timestamps are Unix milliseconds.
# omit the timestamp and the server fills in wall-clock now().Tags are sorted canonically, so a=1,b=2 andb=2,a=1 address the same series. A batch is validated in full before anything is written: one bad line rejects the entire request and nothing is broadcast to subscribers.
# measurement only
cpu usage=0.64 1732250000000
# tags, multiple fields
temperature,device=sensor-01,room=kitchen value=28.5,humidity=61i 1732250000000
# no timestamp: the server stamps it with wall-clock now()
temperature,device=sensor-01 value=28.5
# comment lines and blank lines are ignored
# this whole batch is validated before any point is written
humidity,device=sensor-01 value=61i 1732250001000Field types
| Literal | Stored as | Notes |
|---|---|---|
28.5, -3.2e2 | f64 | Standard float syntax. |
28i | f64 | Signed integer suffix, parsed as i64. |
28u | f64 | Unsigned integer suffix, parsed as u64. |
true / t | 1.0 | Also True, TRUE, T. |
false / f | 0.0 | Also False, FALSE, F. |
"text" | — | Rejected: string fields are not supported. |
NaN, inf | — | Rejected: non-finite values never enter the WAL. |
Escaping & errors
Backslash escapes \,, \ ,\= and \\ are honoured in measurement names, tag keys, tag values and field keys:
weather\,station,city=New\ York,sensor=a\=b temp\=out=21.5 1732250000000Parse failures name the offending line, 1-based, inside the batch:line 2: invalid numeric field 'bad'. Blank lines and lines starting with # are skipped.
POST /api/v1/write
Body is raw line protocol as text/plain — one line or a multi-line batch. Every acknowledged write is fsynced to the WAL before the response is sent.
curl -sS -X POST http://127.0.0.1:8086/api/v1/write \
-H 'Content-Type: text/plain' \
--data-binary $'temperature,device=sensor-01,room=kitchen value=28.5 1732250000000\nhumidity,device=sensor-01 value=61i 1732250000000'{"points_written": 2}Parsed lines are also broadcast to every SSE subscriber. When no subscriber is connected, the ingest path skips the broadcast entirely and does zero extra work.
GET /api/v1/query
| Parameter | Required | Description |
|---|---|---|
metric | required | Measurement name, e.g. temperature. |
tag | optional | One filter as key=value, e.g. device=sensor-01. |
field | optional | Restrict to a single field key. |
from | optional | Inclusive lower bound, Unix milliseconds. |
to | optional | Inclusive upper bound, Unix milliseconds. |
agg | optional | avg · min · max · count. Presence switches the response to a scalar aggregate. |
Without agg the response carries rawpoints. With agg it carries a single aggregation object instead —points is omitted entirely.
# raw points in a 60-second window
curl -sS 'http://127.0.0.1:8086/api/v1/query?metric=temperature&from=1732250000000&to=1732250060000'{
"series": [
{
"measurement": "temperature",
"tags": [["device", "sensor-01"], ["room", "kitchen"]],
"field": "value",
"points": [
{ "timestamp": 1732250000000, "value": 28.5 },
{ "timestamp": 1732250010000, "value": 28.7 }
]
}
]
}# same window, aggregated
curl -sS 'http://127.0.0.1:8086/api/v1/query?metric=temperature&tag=device=sensor-01&agg=avg'
{
"series": [
{
"measurement": "temperature",
"tags": [["device", "sensor-01"], ["room", "kitchen"]],
"field": "value",
"aggregation": { "function": "avg", "value": 28.6, "point_count": 2 }
}
]
}from greater than to is a400. Series with no points in range are omitted from the response rather than returned empty. Blocks whose[min_ts, max_ts] does not overlap the range are skipped without touching the disk.
GET /api/v1/metrics
Every known series, with point counts and time bounds. This is what the dashboard's explorer is built from.
{
"series": [
{
"measurement": "temperature",
"tags": [["device", "sensor-01"], ["room", "kitchen"]],
"field": "value",
"points": 2,
"min_ts": 1732250000000,
"max_ts": 1732250010000
}
]
}GET /api/v1/stats
Live engine statistics. The dashboard polls this every two seconds.
{
"ingested_points": 1000000,
"stored_points": 1000000,
"series_count": 24,
"memtable_points": 0,
"flushed_points": 1000000,
"raw_bytes": 16000000,
"compressed_bytes": 1533456,
"compression_ratio": 10.4352,
"segment_count": 1,
"wal_bytes": 0,
"disk_usage_bytes": 1625984
}| Field | Meaning |
|---|---|
ingested_points | Lifetime acknowledged points, restored from disk on startup. |
stored_points | Points currently retained: segments + MemTable. |
series_count | Distinct (measurement, tags, field) keys known to the engine. |
memtable_points | Points buffered in RAM, not yet flushed. |
flushed_points | Points living in immutable compressed segments. |
raw_bytes | stored_points × 16 — the uncompressed cost. |
compressed_bytes | Gorilla payload bytes across all segments. |
compression_ratio | flushed raw ÷ compressed; 0.0 before the first flush. |
segment_count | Open segment files. |
wal_bytes | Current write-ahead log length. |
disk_usage_bytes | Segments + WAL on disk. |
GET /api/v1/health
Constant-time liveness probe. It does not touch the engine or the disk.
{"status": "ok", "service": "beacon"}GET /api/v1/stream
Server-Sent Events. Each ingested line becomes a pointevent carrying the parsed line as JSON: measurement,tags as [[key, value]],fields as [[key, number]], and timestamp in Unix milliseconds.
GET /api/v1/stream
Accept: text/event-stream
event: point
data: {"measurement":"temperature","tags":[["device","sim-01"],["location","edge"]],"fields":[["value",23.9]],"timestamp":1732250000123}
event: point
data: {"measurement":"humidity","tags":[["device","sim-01"],["location","edge"]],"fields":[["value",55.0]],"timestamp":1732250000123}
event: lagged
data: {"dropped":37}
: keep-aliveFrames are fanned out through a bounded broadcast channel of 1024 entries. A subscriber that cannot keep up does not stall ingestion: it receives alagged event with the number of dropped frames and should re-read the range it cares about. A keep-alive comment is emitted every 15 seconds so proxies hold the connection open.
const es = new EventSource("/api/v1/stream");
es.addEventListener("point", (e) => {
const { measurement, tags, fields, timestamp } = JSON.parse(e.data);
console.log(measurement, fields, new Date(timestamp).toISOString());
});
// The browser was too slow: buffered frames were dropped. Re-sync history.
es.addEventListener("lagged", async (e) => {
const { dropped } = JSON.parse(e.data);
console.warn(dropped, "frames dropped — reloading range");
await reloadHistory();
});The dashboard uses exactly this contract: pointframes append to the live canvas, lagged triggers a history re-sync.
Gorilla compression
Timestamps are encoded with delta-of-delta: the second difference between consecutive samples is bucket-coded into a variable number of bits, which collapses a fixed-cadence IoT stream to a handful of bits per point. Values are encoded with XOR: XOR the f64 against its predecessor, then store the leading and trailing zero counts plus only the significant bits.
All bit packing runs through a hand-written BitWriter /BitReader — no compression crate, no unsafe. A raw point costs 16 bytes (8-byte timestamp + 8-byte f64); the benchmark measures 1.53 B/point on realistic sensor data, a 10.44× reduction.
WAL & recovery
The write-ahead log is append-only with CRC framing. Records are written and fsynced before the HTTP response acknowledges the write.
record := payload_len: u32le | crc32: u32le | payload
^ CRC32 covers the payload only; a bad or short record ends replay.On startup the log is scanned from offset zero. The first short header, short payload, CRC mismatch or implausible length (over 64 MiB) ends the replay; the file is truncated back to the last valid boundary. A crash can therefore lose only an unacknowledged tail — never an acknowledged write.
1. open WAL read/write, scan records from offset 0
2. stop at the first short header, short payload, or CRC mismatch
3. truncate the file back to the last valid record boundary
4. replay every valid record into the MemTable
5. scan segments, rebuild SeriesKey -> Vec<BlockMeta>, truncate a torn tail blockFlush ordering matters: compress → append blocks → fsync the segment → truncate the WAL. A crash in the middle replays points that are already durable; queries and the next flush deduplicate by timestamp, so recovery is idempotent.
Segments & index
block := magic: u32le ("BEAC") | header_len: u32le | header_json | payload | crc32: u32le
header_json := { series_key, min_ts, max_ts, point_count, payload_len }The JSON header carries the series key and the block's time range, so the in-memory index (SeriesKey → Vec<BlockMeta>) can skip blocks without reading them. The index is rebuilt on startup by scanning every segment; a torn tail block from a crash mid-flush is detected by CRC and truncated. The active segment rolls to a new file at 64 MiB.
Benchmark
beacon bench simulates temperature, humidity and CPU streams with delivery jitter, pushes them through the real engine (WAL fsync + Gorilla flush) and reports throughput, compression and query latency percentiles. The PRNG is deterministic, so runs are comparable.
beacon bench --points 1000000
# keep the generated data for inspection
beacon bench --points 1000000 --data-dir ./bench-dataReference machine, three runs: 552K / 608K / 615Kpoints/sec, 10.44× compression, avg 52.1 µs (p50 26 µs, p99 71 µs) over 200 random 1%-window aggregations.
Verification
The checks this project holds itself to:
cargo build --release
cargo test # 71 tests
cargo clippy --all-targets -- -D warnings
cargo run --release -- bench --points 500000
powershell -NoProfile -ExecutionPolicy Bypass -File scripts\smoke-test.ps1Current state: 71/71 tests passing, 0 clippy warnings with-D warnings, 0 unsafe blocks insrc/.