Docs / Contact / sdks/micropython/microtetherdb

MicroTetherDB: Embedded Database

MicroTetherDB is a lightweight embedded database built for microcontrollers. It gives you persistent key-value storage with TTL expiration, tag-based querying, and batch operations, all designed to run within the RAM constraints of an ESP32.

The Tendrl MicroPython Client uses MicroTetherDB internally for offline message storage and local device state. You can also use it directly in your application for any on-device data that needs to survive reboots or be queried locally.

Why Use It

• Built for microcontrollers: small memory footprint, configurable page sizes, and an optional record cap to keep storage bounded
• TTL expiration: data automatically expires and cleans up, no manual housekeeping
• Tag-based queries: tag your data and query it back without building indexes
• Field queries with operators: $gt, $lt, $in, $contains, and more
• Offline resilience: file-backed mode persists data across reboots on a best-effort basis (durability)
• Batch operations: insert or delete many records efficiently in a single call

Quick Start

from tendrl.lib.microtetherdb import MicroTetherDB

# In-memory (fast, lost on reboot)
db = MicroTetherDB(in_memory=True)

# File-backed (slower, usually survives reboot - see Durability below)
db = MicroTetherDB(filename="mydata.db", in_memory=False)

Store Data

# Store a reading
key = db.put({"temperature": 25.5, "location": "room-1"})

# Store with tags
key = db.put(
    {"temperature": 25.5, "location": "room-1"},
    tags=["sensor", "temperature"]
)

# Store with TTL (auto-expires after 1 hour)
key = db.put(
    {"alert": "high_temp", "value": 45.0},
    ttl=3600,
    tags=["alert"]
)

Retrieve Data

# Get by key
data = db.get(key)
# {"temperature": 25.5, "location": "room-1", "_tags": ["sensor", "temperature"]}

# Returns None if not found or expired
data = db.get("nonexistent_key")  # None

Query Data

# Find by exact field value
results = db.query({"location": "room-1"})

# Find by tag
results = db.query({"_tags": "temperature"})

# Comparison operators
results = db.query({"temperature": {"$gt": 30}})

# Combine conditions (AND)
results = db.query({
    "location": "room-1",
    "temperature": {"$gte": 20, "$lte": 30}
})

# Limit results
results = db.query({"_tags": "sensor", "$limit": 10})

Delete Data

# Delete one record
db.delete(key)

# Purge entire database
db.delete(purge=True)

Batch Operations

# Insert many at once
items = [
    {"sensor": "temp", "value": 25.5},
    {"sensor": "humidity", "value": 60},
    {"sensor": "pressure", "value": 1013},
]
keys = db.put_batch(items, ttls=3600)  # All expire in 1 hour

# Different TTL per item
keys = db.put_batch(items, ttls=[3600, 7200, 0])  # 0 = no expiry

# Delete many at once
count = db.delete_batch(keys)

Query Operators

Nested Fields

Use dot notation to query nested objects:

db.put({"sensor": {"type": "dht22", "location": "outdoor"}})

results = db.query({"sensor.type": "dht22"})
results = db.query({"sensor.location": {"$ne": "indoor"}})

TTL (Time-To-Live)

Data with a TTL automatically expires and is cleaned up in the background:

# Expires in 60 seconds
db.put({"event": "motion_detected"}, ttl=60)

# Expires in 24 hours
db.put({"daily_report": {...}}, ttl=86400)

# No expiration (default)
db.put({"config": "permanent"})

# Manually trigger cleanup (normally automatic)
deleted_count = db.cleanup()

The TTL system uses a min-heap for efficient expiration checking: it doesn't scan all records, just checks the next-to-expire item. Cleanup runs automatically every 60 seconds by default.

Bounded Growth (Size Cap)

TTL bounds data by age. For unattended devices you often also want to bound it by count so the database can never grow without limit, for example if data is produced faster than it expires, or never expires at all. Set max_records:

# Keep at most 1000 records; oldest are evicted first when the cap is hit
db = MicroTetherDB(filename="readings.db", in_memory=False, max_records=1000)

# Unlimited (default); only do this when growth is otherwise bounded
db = MicroTetherDB(max_records=None)

Behavior:

• The cap is on record count, not bytes. Combined with the 8 KB per-record

limit it gives a predictable worst-case footprint.

• Eviction is oldest-first: keys are time-ordered, so the newest data is

always preserved. This makes it ideal for rolling buffers and offline queues.

• It works alongside TTL: TTL removes expired data, the cap enforces a hard

ceiling regardless of age. Using both is the strongest guarantee against runaway storage.

• Eviction happens lazily on insert (put/put_batch), so there's no

background scanning cost when you're under the cap.

Configuration

Tuning

The defaults are already tuned, and they key off the backend, not the board. Page size is the only parameter that materially affects throughput, and what it should be depends on where the pages live:

• File-backed → large pages (4096). Pages live in flash, so bigger pages cost

no RAM and make batched writes dramatically faster: on an ESP32-S3, a 100-item batch delete went from 39.2 s at 512 bytes to 5.5 s at 4096, about 7×.

• In-memory → small pages (1024). Pages are charged to the heap, so page

waste is real memory. At 4096, small records cost 3.3× more RAM (81 → 270 bytes per record).

Because of that second point, a board with more RAM should not use larger in-memory pages, since it would waste more, not less. There is no board-size tuning to do here.

btree_cachesize is not worth touching: raising it from 32 to 4096 changed file delete time by 1.3%, which is measurement noise.

Constrained boards (ESP32 without PSRAM): reach for max_records, not page size: capping the record count is what actually bounds RAM.

db = MicroTetherDB(
    in_memory=True,
    max_records=500,   # bounds RAM; oldest records evicted first
)

Persistent storage (file-backed):

db = MicroTetherDB(
    filename="/data/readings.db",
    in_memory=False,
    # btree_pagesize omitted on purpose: file-backed already defaults to 4096,
    # which is the fast setting. Passing 1024 here would make batched writes
    # ~7x slower.
    max_records=5000      # Bound flash usage
)

In-Memory vs File-Backed

Durability and flash wear

How It Works with the Tendrl Client

The MicroPython Client creates two MicroTetherDB instances automatically:

1. Offline storage (tether.db, file-backed): queues messages when MQTT is disconnected, replays them on reconnect. Bounded by default (1000 records, oldest-first eviction) via the client's offline_max_records parameter so a long outage can't fill flash.
2. Client database (client_db.db, in-memory by default): your application's local data store, accessed via client.db_put(), client.db_get(), etc.

# These client methods use MicroTetherDB under the hood:
client.db_put({"temp": 25.5}, tags=["sensor"])
client.db_get(key)
client.db_query({"_tags": "sensor"})
client.db_delete(key)

You can also create standalone MicroTetherDB instances for your own data needs since it's not tied to the Tendrl Client.

Practical Examples

Sensor Reading Buffer

Buffer readings locally and query recent data without a network call:

db = MicroTetherDB(in_memory=True)

# Store readings with TTL
def store_reading(temp, humidity):
    db.put(
        {"temp": temp, "humidity": humidity},
        ttl=3600,  # Keep for 1 hour
        tags=["reading"]
    )

# Get recent high-temp readings
hot = db.query({"temp": {"$gt": 35}, "$limit": 20})

Offline Command Queue

Queue commands for execution even if the main loop is busy:

db = MicroTetherDB(filename="commands.db", in_memory=False)

# Receive and store command
def on_command(msg):
    db.put(msg["data"], tags=["pending"])

# Process pending commands
pending = db.query({"_tags": "pending"})
for cmd in pending:
    execute(cmd)
    db.delete(cmd["_key"])

Device Configuration Store

Persist configuration across reboots:

db = MicroTetherDB(filename="config.db", in_memory=False)

# Save config
db.put({"interval": 30, "sensors": ["dht22", "bmp280"]}, key="app_config")

# Load config on boot
config = db.get("app_config")
if config:
    interval = config["interval"]

Limits

• Maximum value size: 8KB per record (JSON-encoded)
• Maximum operation queue: 50 pending operations
• Record cap: Unlimited by default; set max_records for oldest-first eviction
• Query method: Full table scan (no secondary indexes), best for small-to-medium datasets
• Key format: Auto-generated as {timestamp}:{ttl}:{id}, or specify your own with the key parameter

← Back to Contact docs