Docs / Contact / sdks/micropython/vision

MicroPython Client: Vision

tendrl.vision turns a camera into a sensor.

Instead of uploading video, the device runs a detector on each frame, reduces it to a state, and publishes a small message only when that state changes:

{"state": "empty", "prev": "full", "conf": 0.81}

While the scene is stable, nothing is sent at all. A couple of hundred bytes cross the network when reality changes, not a video feed. That is what makes a camera cheap enough to leave running forever, and it means the footage never leaves the device.

Install

Provision from Tendrl (recommended). In the Contact Device tab, or the standalone tendrl-dev-mcp console, pick the vision tier. The SDK is pushed to the board over serial as precompiled .mpy (into /flash/lib/tendrl), no mip involved. See Getting started.

Host-side mpremote. mpremote runs mip on your computer and copies the files to the board over USB, so the board itself needs neither mip nor Wi-Fi:

mpremote mip install https://app.tendrl.com/api/public/sdk/v1/latest/mpy/package-vision.json

OpenMV IDE. Copy the tendrl/ package onto the board from the editor's file browser.

Vision is an add-on: it layers on top of either the full or the minimal SDK install.

Quick start

from tendrl import Client
from tendrl.vision import Camera, Watch, roi_brightness

client = Client(mode="sync")
client.start()
time.sleep(5)

cam = Camera()  # RGB565, ready for detection and clips

watch = Watch(
    client,
    detect=roi_brightness(roi=(120, 80, 60, 90), threshold=0.45,
                          states=("full", "empty")),
    debounce=5,          # consecutive agreeing frames before a change is believed
    cooldown_s=30,       # minimum gap between published changes
    tags=["coffee"],     # routes to a flow / Strand workflow
)

while True:
    watch.update(cam.snapshot())   # publishes ONLY on a real transition
    time.sleep_ms(500)

The tags are the whole integration story: a flow or a Strand workflow keyed on coffee picks the message up and turns it into a Slack post, a log entry, or a page.

Detectors

A detector is a function detect(img) -> (state, confidence). Each one owns its own state labels. A detector may return (None, 0.0) to mean "no verdict this frame"; Watch ignores those.

Every detector takes a states argument: a 2-tuple (negative, positive). states[0] is the resting / "not happening" state; states[1] is the fired state (above the threshold, a blob present, motion detected, a person seen). So for motion you'd write states=("still", "motion"): resting first, fired second. Get this order backwards and your labels are inverted.

Budgeting heap for the person detector

person() uses OpenMV's YOLO LC model, which ships in flash (/rom/yolo_lc_192.tflite). Its weights are read in place rather than copied into the heap, which is what lets it run on boards that could never fit a custom model. But loading the model still allocates a tensor arena on the heap.

Measured on an OpenMV RT1062:

(The model reports its own arena as model.ram: ~270 KB here.)

So budget two things for person detection, not one: this ~275 KB arena and the camera framebuffer (VGA windowed to 400×400 is another ~280 KB), call it ~550 KB free before you start. If you are tight on memory, shrink the framesize first; the arena is fixed, so on a very small board a smaller model may be the only lever left.

from tendrl.vision import Camera, Watch, person

cam = Camera(framesize="VGA", window=(400, 400))
watch = Watch(client, detect=person(min_conf=0.4), debounce=2, tags=["person-detected"])

Faces and hands (MediaPipe)

Three more models ship in OpenMV flash, from Google's MediaPipe family. Same economics as person() (weights read in place, tensor arena on the heap), and all three want a square input, so build the camera as Camera(framesize="VGA", window=(400, 400)).

Each has a state form for Watch and a geometry form for drawing:

face is a 128×128 model and the cheapest of the three. palm (192×192) and hand_landmarks (224×224) want an NPU to run in real time.

Draw the results with draw_boxes, draw_keypoints, and draw_skeleton; all three write into the pixels, so the overlay survives JPEG compression and shows up in a stream, a clip, or a preview.frame():

from tendrl.vision import Camera, HAND_LINES, draw_boxes, draw_skeleton, hand_landmarks
from tendrl import preview

cam = Camera(framesize="VGA", window=(400, 400))
detect = hand_landmarks(min_conf=0.4)

while True:
    img = cam.snapshot()
    for x, y, w, h, score, kps, label in detect(img):
        draw_boxes(img, [(x, y, w, h)], label=label)
        draw_skeleton(img, kps, HAND_LINES)
    preview.frame(img)          # draw FIRST: this compresses img in place

Joint order follows the MediaPipe convention: 0 is the wrist, then four joints per digit running base → tip (thumb 1-4, index 5-8, middle 9-12, ring 13-16, pinky 17-20), so 4/8/12/16/20 are the fingertips. PALM_LINES and HAND_LINES are the connection tables for draw_skeleton.

Measured on an OpenMV AE3 (which has an NPU):

When the model isn't there

The in-flash models come from OpenMV firmware, not from the Tendrl SDK, so which ones a board has varies by board and firmware version. Asking for one that isn't installed raises a VisionError naming the file:

VisionError: /rom/blazeface_front_128.tflite is not in this firmware's flash.
List /rom to see which models this board ships, or update OpenMV firmware.

To see what your board actually has:

print([f for f in os.listdir('/rom') if f.endswith('.tflite')])

Presence isn't sufficiency: a model that exists still has to fit in free heap alongside the camera framebuffer. Check gc.mem_free() if a load fails on a small board.

Picking a threshold

roi_brightness works by watching a rectangle of the image (the "ROI", or region of interest) and asking one question: is it lighter or darker than a cutoff? That cutoff is the threshold. Two things to pick, and this section walks through both.

First, some vocabulary:

• ROI: the box you care about, given as (x, y, w, h) in pixels: the top-left corner

(x, y) and the box's width and height. At QVGA the image is 320×240, so (120, 80, 60, 90) is a 60×90 box starting a bit right-of-center. You don't have to be exact; just cover the thing that changes (the coffee in the carafe, the liquid in a tank) and not much else.

• level: how bright that box is, as a number from 0.0 (black) to 1.0 (white).

roi_level(img, roi) gives you this number.

• threshold: the cutoff. When the level is at or above the threshold, the detector

reports states[1]; below it, states[0].

Now find the threshold: you measure it, you don't guess it. Run this on the device with the camera aimed at your scene:

from tendrl.vision import Camera, roi_level

cam = Camera()
ROI = (120, 80, 60, 90)   # the box to watch; adjust to cover your thing

while True:
    print("%.3f" % roi_level(cam.snapshot(), ROI))
    time.sleep_ms(500)

It prints the box's brightness twice a second. Now change the thing you care about and watch the two numbers it settles at. For a coffee pot:

0.31   0.30   0.31    ← pot is FULL: dark coffee fills the box, low number
0.30   0.62   0.63    ← you pour it out; empty glass is bright, number jumps up
0.62   0.61   0.62    ← pot is EMPTY: settles high

You saw it settle around 0.31 when full and 0.62 when empty. Pick a threshold halfway between, 0.45, so full sits comfortably below it and empty comfortably above:

roi_brightness(roi=(120, 80, 60, 90), threshold=0.45, states=("full", "empty"))

The bigger the gap between your two numbers, the more reliable the detector. If they're too close (say 0.48 vs 0.52), move or resize the ROI to a spot where the change is more dramatic, or improve the lighting, then re-measure.

Detecting a colour (color_blob)

color_blob fires when a patch of a specific colour appears: a lit red lamp, a green status LED, an amber stack light. It takes:

• thresholds: a list of colour ranges to look for. Each range is a 6-number tuple

in LAB colour space: (L_min, L_max, A_min, A_max, B_min, B_max). L is lightness (0–100), A is green↔red, B is blue↔yellow (both roughly −128 to 127). You don't compute these by hand; use the OpenMV IDE's built-in Threshold Editor (Tools → Machine Vision → Threshold Editor): point it at your scene, drag the sliders until only the colour you care about is highlighted, and it prints the tuple. Paste that in.

• roi: the (x, y, w, h) box to search, or None for the whole frame (same idea as

in Picking a threshold above).

• min_pixels: how many matching pixels count as a real blob (default 50). Raise it to

ignore small specks; lower it to catch a tiny LED.

• states: (off, on); fires states[1] when a blob of at least min_pixels is found.

from tendrl.vision import Camera, Watch, color_blob

# A red LAB range from the OpenMV Threshold Editor:
RED = [(15, 60, 30, 90, 10, 60)]

watch = Watch(
    client,
    detect=color_blob(RED, roi=(200, 40, 40, 40), min_pixels=40,
                      states=("off", "lit")),
    debounce=3, tags=["fault-lamp"],
)

Self-illuminated targets (lamps, LEDs) are the easy case: they glow at the camera, so ambient light barely matters.

The other detectors, briefly

• motion(threshold=0.02, scale=0.25, states=("still","motion")): frame-to-frame

differencing. threshold is the mean pixel change between frames as a 0.0–1.0 fraction (higher = needs more movement to fire). It's a whole-frame average, so a small moving object barely moves it. Measured on an RT1062: a dead-still scene is ~0.00, a person or hand moving through the frame is ~0.02, and a near-total change (lens covered, lights, a pan) is ~0.10+. The default 0.02 catches someone moving through the frame; raise it toward 0.05 for only dramatic changes, lower it toward 0.01 for small or distant motion. In a dim or noisy scene, print the confidence for a few still seconds first and set the threshold above that floor. scale downscales the frame it keeps as a reference so the comparison is cheap: 0.25 makes that copy 16× smaller; leave it unless you're very tight on heap. Returns (None, 0.0) on the very first frame (no reference yet).

• model_detect(path, min_conf=0.5, postprocess=None, states=("absent","present")): runs

a TFLite model and reduces it to present/absent by comparing the best class score to min_conf. Detection models (FOMO, YOLO, which output boxes) need a postprocess object (e.g. from ml.postprocessing.darknet import YoloLC); classification models leave postprocess=None — their predict() output is one (1, N) ndarray of class scores, which model_detect flattens before it reads it. If you just want people, use person() instead; it wires up the built-in model for you. path can be a /rom model or a model file on /flash, an SD card or /app (after an OTA deploy): it is opened with load_model(path, kw), the SDK's ml.Model wrapper. Call tendrl.vision.load_model yourself, instead of ml.Model, for any file-based model you run directly — on the OpenMV N6 (firmware 5.0.0) a bare ml.Model("/flash/…") can fail with Failed to load network, and load_model is the fix (details in Custom vision models). On every other board it is exactly ml.Model. A model with more than two outcomes (say idle / ok / unknown / nobody / violation) is a plain detect(img) -> (state, conf) function of your own around load_model; Watch never inspects the detector. Training, export, loading and pushing a retrained model over the air** are covered in Custom vision models.

Camera modes

Camera wraps the OpenMV sensor. Its constructor:

Camera(mode="detect", framesize="QVGA", quality=90, window=None, skip_ms=2000)

The two modes configure the sensor for different pixel formats. Camera makes that explicit rather than letting you discover it the hard way.

cam.set_mode("stream")
client.start_streaming(capture_frame_func=cam.jpeg)   # detection is stopped

cam.jpeg() returns hardware-encoded JPEG when the sensor has an encoder, and otherwise compresses in software, so STREAM mode works on sensors without hardware JPEG (like the OpenMV AE3's PAG7936) at some CPU cost. cam.hw_jpeg records which path was chosen.

Streaming frames you have detected on

You can stream frames you ran a detector on; you just don't do it from STREAM mode. Stay in DETECT, draw onto the RGB565 frame, and compress it yourself inside the capture function. start_streaming accepts any callable that returns JPEG bytes, and the draw helpers write into the pixels, so the overlay survives compression:

from tendrl.vision import Camera, face_boxes, draw_boxes, draw_keypoints

cam = Camera(mode="detect", framesize="QVGA")
detect = face_boxes(min_conf=0.5)

def annotated_jpeg():
    img = cam.snapshot()
    dets = detect(img)
    draw_boxes(img, dets, label="face")
    for d in dets:
        draw_keypoints(img, d[5])
    return img.compress(quality=70).bytearray()

client.start_streaming(capture_frame_func=annotated_jpeg, target_fps=10)

Measured on an OpenMV AE3 at QVGA: one snapshot → detect → draw → compress pass:

Watch

A raw detector flips on noise: one stray reflection, a passing hand, a single bad exposure. Watch is what turns a detector into something you can leave running.

Watch(client, detect, debounce=3, cooldown_s=0, tags=None,
      on_change=None, entity="", publish=True, emit_initial=True, debug=False)

watch.update(img) returns the current confirmed state. watch.reset() forgets it, so the next confirmation publishes again.

The on_change callback

If you pass an on_change function, Watch calls it each time the state changes, right after it publishes. Your function always receives three positional arguments: (new state, previous state, image):

def on_change(state, prev, img):
    ...

Why this matters for your bill

Every file upload is malware-scanned and costs one Surface scan, charged to the sender (see File transfer). Video is expensive; a state change is not. The local detector's real job is deciding which rare moments are worth the bytes, which is exactly what debounce and cooldown_s control.

Sending a clip on a change

This uses the on_change callback (see The on_change callback above): Watch calls it after each transition with (state, prev, img): the new state, the state before it, and the single frame from that moment. Here we record a clip only when a person arrives (state == "person"), not when they leave.

from tendrl.vision import Camera, Watch, person

cam = Camera(framesize="VGA", window=(400, 400))

def on_change(state, prev, img):   # state = new state, prev = state before, img = the frame
    if state != "person":
        return                      # departures publish, but don't need video
    cam.lock_exposure(True)         # stop auto-exposure pumping mid-clip
    try:
        # send_clip_burst takes a capture FUNCTION (cam.snapshot); it calls it
        # repeatedly to record the burst. Not the single `img` above.
        client.send_clip_burst(cam.snapshot, duration_s=3, fps=5,
                               tags=["person-detected"])
    finally:
        cam.lock_exposure(False)

watch = Watch(client, detect=person(), debounce=2, cooldown_s=30,
              tags=["person-detected"], on_change=on_change)

The clip lands in the Contact clip gallery, transcoded to MP4 with playback controls.

Previewing frames while you build

When you're tuning a detector or a custom capture loop, you usually just want to see what the camera sees, without standing up a full stream. tendrl.preview prints frames and short clips in the marker format the Contact Device tab and the standalone tendrl-dev-mcp console recognise, so while your script is Run from the Files tab (or the Terminal), the frames render live in the browser. Run the same script headless and those are just harmless print lines that nothing consumes.

from tendrl.vision import Camera
from tendrl import preview

cam = Camera()   # RGB565, detection and preview

while True:
    img = cam.snapshot()
    # ... run your detector on img here ...
    preview.frame(img)          # -> live view; the stream panel opens itself

preview.frame(img, quality=80) JPEG-compresses the frame in place (the AE3 has no hardware encoder, and the non-destructive path doesn't render there), so run any detection on the frame before you call it. To assemble a short playable clip in the console's Clips view instead:

preview.clip([f1, f2, f3], meta={"label": "dog", "conf": 0.92})

preview ships in the Full (default) install and the Vision add-on, so from tendrl import preview works wherever you're already running detection. It's a development aid: for a real video feed to the dashboard use streaming, and to store an event use motion clips.

Examples

Requirements & limits

• OpenMV board with csi (newer firmware) or sensor (legacy). Camera detects which.
• ml module for model_detect(), load_model() and person() only. The classical

detectors and Watch import fine without it.

• Memory. The dominant cost is camera framebuffers, not models: VGA windowed to

400×400 is about 280 KB of heap, while the in-flash person() model is 0. Shrink the framesize first. motion() additionally keeps one downscaled reference frame and allocates one more per call; scale=0.25 keeps those copies 16× smaller than the live frame. Custom .tflite models (unlike person()) do load into heap and realistically want a more capable OpenMV board; the N6 (about 25 MB heap, 800 MHz) has by far the most headroom, then the RT1062 and AE3.

• Pixel format changes the numbers you get. OpenMV reports GRAYSCALE statistics on

0–255, but colour (RGB565) statistics come from the LAB L channel, which tops out at 100. roi_level() normalises both to 0.0–1.0, so a threshold means the same thing in either format, but if you call get_statistics() yourself, mind the scale.

• Lighting. Detection thresholds are scene-specific. Tune with roi_level() in the

environment the camera will actually live in, not on your desk.

Related

• Streaming: live JPEG video, when you do want the feed
• File transfer: clips, scanning, and credits
• Strand integration: turning a tag into a workflow

← Back to Contact docs