Kinex v1.0.0 docs: Single-page view

Kinex documentation → Page

Kinex is a general harness for arbitrary robots with coding agents, built on the Pi coding agent. It gives an agent three layers to work with:

Kinex lets you work with real hardware the way you already work with a coding agent. Describe the task in plain language and the robot carries it out.

Install and run

curl -fsSL https://raw.githubusercontent.com/stacx-lab/kinex/main/install.sh | bash
cd /path/to/robot-workspace
kinex

Then /login to connect a model provider and /model to choose a model.

Already use Pi, Claude Code or Codex? Kinex plugins currently support these three harnesses: see Plugin systems.

Pick a path

Understand the design of Kinex

• Architecture — skills, tools, devices and who owns what.
• Devices — types, profiles, drivers, frames and clocks.
• Tools & skills — the built-in tools and skills, and how custom ones are discovered.
• Runtime — what Kinex handles for you: devices, discovery, permissions, frames, time and context.

Go further

Reference: CLI, slash commands, troubleshooting.

About this version

These pages describe Kinex 1.0.0, bundling the Pi coding agent 1.2.0 (@earendil-works/pi-coding-agent). No separate Pi install is needed. Install on macOS and Ubuntu. Node.js ≥ 22.19 is required.

The documentation tracks source commit afeddc6 (2026-10-14). Source references are recorded in each page’s metadata. Kinex and its core/setup packages use the Apache License, Version 2.0. Bundled third-party notices are listed in THIRD_PARTY_NOTICES.txt.

Onboarding real robots → Page

Kinex drives real hardware through the same device interface it uses for simulators. You describe your robot and what you want it to do; the agent writes the device profiles and drivers, brings the hardware up and checks it, calibrating only what the task needs. Supported embodiments include arms, lifts, mobile bases, quadrupeds, grippers, suction, dexterous hands, cameras, IMUs and F/T sensors (see Devices).

Assumes Kinex is installed (see Install and run).

1. Start Kinex in a fresh workspace

mkdir -p robot-workspace
cd robot-workspace
kinex

If you haven’t yet, connect a model with /login and /model.

2. Onboard your robot

Describe the hardware, how it connects, and the task:

/onboarding Set up my robot arm, gripper, and overhead camera to pick objects from the table.

Onboarding is an interview. The agent inspects what is already there, asks only for missing facts, and records a plan in .kinex/setup/plan.json. It then prepares profiles and drivers, loads the devices and verifies them, covering only what this task needs. Unknowns stay ???; nothing is invented.

You can also register devices one at a time with /register <type-or-template>:

/register rgbd
/register grounding-dino

Each sets up a driver and writes an unloaded profile, validating it without touching the hardware; run /load to bring it up. ROS hardware is routed through onboard-ros2 automatically (see ROS 2 onboarding).

3. Calibrate

Calibration pins down the physical facts the task depends on but the agent can’t know yet, such as camera intrinsics, where a camera sits relative to the arm (extrinsics), the tool-tip (TCP) offset, or joint limits. It is handled by Kinex’s calibration skills. Describe what you want calibrated, or leave it blank:

/calibrate camera-to-arm transform for picking
/calibrate

The agent works out what needs calibrating and which procedure to follow, guides you through collecting samples, and writes the result into the device profile. See Onboarding & calibration for the bundled procedures.

4. Bring it up and give a task

/devices
/load all
/ping all

Then give the task in plain language. Useful during work:

Workspace layout

Start Kinex from your project folder. Directories are created as needed.

robot-workspace/
├── .kinex/                     # robotics workspace
│   ├── workspace.yaml          # robot and workspace settings
│   ├── kinex.json              # permission rules
│   ├── devices/<name>/         # one folder per device
│   │   ├── profile.yaml        # connection and calibration
│   │   ├── driver.py           # Python driver, when used
│   │   └── out/                # images and other outputs
│   ├── lib/                    # shared Python helpers
│   ├── tools/                  # callable agent tools
│   ├── skills/                 # task procedures (SKILL.md)
│   ├── setup/plan.json         # onboarding plan
│   ├── python/                 # managed Python environment
│   ├── recordings/             # recorded runs
│   └── agent/settings.json     # agent settings and packages
└── .pi/                        # Pi agent resources

• .kinex/ holds robot configuration. devices/ contains each device’s profile, driver, and outputs. See Devices.
• .kinex/lib/ holds shared Python helpers that drivers and tools can import. tools/ and skills/ add callable operations and task procedures. See Custom tools & skills.
• .pi/ holds Pi agent resources; .kinex/agent/ holds project agent settings and package choices. See Plugin systems.
• .kinex/recordings/ holds recorded runs. See Recording. Conversation sessions are stored separately, outside the workspace; see Working with the agent.

Playing with simulators → Page

RoboEnv runs simulator tasks (RoboCasa, LIBERO, RoboTwin, RoboDojo, BEHAVIOR, Isaac Lab, SIMPLE, ABC) and exposes their robots as Kinex devices. Kinex itself ships no simulator engine — just the device contracts RoboEnv implements.

Assumes Kinex is installed (see Install and run).

1. Launch a simulator

RoboEnv runs from a source checkout on Linux with Git and uv. Clone and bootstrap it once:

git clone https://github.com/stacx-lab/RoboEnv.git
cd RoboEnv
bash tools/bootstrap.sh
source .venv/bin/activate

Bootstrap installs the roboenv CLI into a Python 3.12 environment. Activate it in each new shell, or call .venv/bin/roboenv directly.

Choose a task and its paired robot/device configuration:

roboenv tasks list --simulator robocasa
roboenv robots list --simulator robocasa

Both commands print one ID per line. Omit --simulator to see all entries, or add --json for structured output. launch takes a task ID; setup takes its robot ID, which exports the robot’s full device collection. Read robot: in RoboEnv’s catalog/tasks/<task-id>/task.yaml for the exact pairing. A simulator may have several robots or scene configurations, so sharing a simulator name is not enough.

Prepare the simulator, then launch the task. This page uses the RoboCasa pair:

roboenv prepare robocasa
roboenv launch robocasa/load-condiments-in-fridge --port 8123

prepare installs the simulator’s isolated interpreter and downloads its assets (RoboCasa needs about 10 GB). It is only needed once per simulator. Wait for Ready and leave launch running; Ctrl+C stops the environment.

For another simulator, run roboenv prepare <name> and pick a task from roboenv tasks list --simulator <name>. GPU prerequisites and per-simulator assets are in RoboEnv’s setup guide.

2. Create a Kinex workspace

In a second terminal, activate the same environment and create a workspace:

source /path/to/RoboEnv/.venv/bin/activate         # put the roboenv CLI on PATH
mkdir -p kinex-sim-workspace                       # the Kinex workspace directory
cd kinex-sim-workspace
roboenv setup robocasa/robocasa-panda-omron         # export the robot into .kinex/
export ROBOENV_ENDPOINT=http://127.0.0.1:8123       # where the drivers find the running scene
roboenv describe --endpoint "$ROBOENV_ENDPOINT"   # read the initialized task instruction
kinex

roboenv setup <simulator/robot> copies a ready-made robot configuration from RoboEnv’s catalog into the current directory’s .kinex/, creating it if needed:

• workspace.yaml: the robot manifest.
• devices/: one profile and thin driver per device. For the RoboCasa Panda–Omron that is arm, base, torso, gripper, three cameras and world.

setup exports devices without installing or launching a simulator. The configuration is task-independent, so one workspace serves every task with the same robot ID. roboenv robots list --simulator robocasa lists robot IDs for RoboCasa; roboenv setup --list remains available for listing all robots. For a different robot ID, create a new workspace or save custom configuration before roboenv setup ROBOT_ID --force, which replaces the manifest and entire device collection.

ROBOENV_ENDPOINT tells the drivers which running scene to connect to; match the launch port.

3. Inspect devices

Inside the Kinex session:

/devices
/load all
/ping all

/devices shows saved activation intent (loaded), not liveness. /ping checks live status without starting anything.

4. Give it a task

Give Kinex the running task’s task.instruction and task.context from roboenv describe. For the example above, ask it to inspect the scene and complete that fridge task using the paired robot’s devices. To inspect the scene first:

Look through the robot's cameras and describe what is in the scene.

Approve the device read when asked (or switch with /mode; see Approval modes). Camera images come back attached to the tool result. Press Ctrl+O to expand tool calls and check the raw observation against the agent’s answer.

5. Record the episode

Add --record when you launch to film the episode from RoboEnv’s spectator cameras:

roboenv launch robocasa/load-condiments-in-fridge --port 8123 --record                         # under the run's evidence/recording/
roboenv launch robocasa/load-condiments-in-fridge --port 8123 --record ./recordings/episode-1  # into this directory

You get one MP4 per view; it needs ffmpeg on PATH. More options: RoboEnv’s spectator recording guide.

6. Finish and Reconnect

Leave Kinex with /quit. Then stop the simulator: press Ctrl+C in the launch terminal, or, if you launched with --detach, run roboenv stop <run-id> from the RoboEnv checkout with the run ID that launch printed (add --project /path/to/RoboEnv when running it elsewhere).

After restarting the simulator or changing profiles, reconnect the devices in Kinex:

/unload all
/load all

After upgrading Kinex, refresh the copied RoboEnv profiles and drivers together (roboenv setup <simulator/robot> --force) and restart the simulator.

More tasks and connection details: RoboEnv’s Kinex walkthrough.

Architecture → Page

Architecture overview

Kinex connects a coding agent to robots and simulators through three layers.

Devices: typed physical or virtual resources

A device is anything the agent can read from or command: an arm, a camera, a gripper, but also a detection model or the simulator’s clock. Each device has a type, which fixes its operations and data conventions; a profile, which records how to reach it and its specifications; and a driver, which talks to the real SDK, simulator or service. Because real hardware and simulators share the same types, the agent works with both the same way. → Devices

Tools: executable operations over devices and computation

Tools bundle computation or a sequence of device calls into one named operation, so the agent doesn’t have to orchestrate every step itself. For example:

• locate_object_3d combines a camera image, a detection or segmentation model and depth to estimate where an object is in 3D.
• servo plans a joint trajectory from measured state, sends it and waits its planned duration by default. Arrival still needs fresh feedback.
• exec runs Python source or script files to combine device calls, bounded feedback loops and computation, returning structured results and selected camera images for the agent’s next decision.

You can add your own as a tool.yaml plus tool.py. Python tools never load or unload devices; only the device tool and your own commands can. → Tools & skills

Skills: task procedures the agent loads on demand

A skill is a written procedure, a SKILL.md with optional references and scripts, that tells the agent how to approach a kind of task, such as setting up a device or calibrating a camera. The agent loads one when it needs it. Loading a skill only adds instructions to the conversation; it runs nothing and grants no extra permissions. → Tools & skills

The coding-agent analogy

Packages

Kinex is split into three packages that ship together at the same version:

• @kinex/core is the robotics layer: devices and drivers, built-in tools and the runtime Python. It doesn’t depend on any agent or UI, so other harnesses can use it too.
• @kinex/setup adds onboarding, calibration and ROS 2 setup on top of core.
• kinex is the command-line app you run. It combines the two with the Pi agent, which provides the model connection, the conversation loop, sessions and the terminal UI. It also decides what needs your approval.

For the full contracts, see the design docs.

Devices → Page

Kinex treats a robot as a collection of independent devices rather than one monolithic interface: a mobile base, arms, grippers and cameras. The agent coordinates them through tools and code for each task, adapting the workflow to the capabilities available. For example, if a camera is missing, it can use another viewpoint where one is available.

A device is anything the agent can read from or command: a piece of hardware, part of a simulator, or a service such as a detection model. Every device has three parts:

• a type, which defines its operations and data conventions;
• a profile, which records how to reach it and its specifications;
• a driver, which talks to the actual SDK, simulator or service.

Real and simulated devices use the same types, so the agent works with both the same way.

Types

A type describes what a device can do, not who made it. A robot is usually several devices: an arm, a gripper, a few cameras.

Some types can stand in for others: an rgbd camera works wherever an rgb camera is required, and a segmentation model works wherever a detection model is.

Profiles

Each workspace keeps its devices under .kinex/:

.kinex/
├── workspace.yaml              # the robot as a whole: identity, simulator details, safety settings
└── devices/<name>/
    ├── profile.yaml            # this device's profile
    └── driver.py               # its driver (or a pinned ROS-generated driver)

A profile contains:

A value of ??? means the fact is still unknown, and needs_calibration marks a value that calibration still has to fill in. Change profiles through the agent’s profile_write tool rather than editing the YAML by hand during a session.

Drivers

A driver should expose only the device’s basic, atomic operations, such as reading its state or sending one command. Higher-level behavior like planning a trajectory, running a feedback loop or chaining several calls belongs in tools and skills, which build on those operations.

Every driver supports load, unload, status and read. Actuators also support write and stop. Arms and hands can provide fk and ik if their profile declares them, world adds wait, and models add their own inference operations. A driver can run once per call, or stay running as a persistent process (started with --serve) that exchanges JSON lines over stdin and stdout and writes diagnostics to stderr.

Connecting, stopping and disconnecting

/devices            # list devices and whether each should be loaded
/load arm           # connect and wait until the driver is ready
/ping arm           # check status without starting anything
/stop arm           # stop its motion right away (or /stop for every device)
/unload arm         # disconnect and release its resources
/unregister arm     # after confirmation, unload and delete the profile, driver and files

To restart a device, run /unload arm and then /load arm (or use all). If a device’s driver fails, Kinex doesn’t restart it automatically; reads and writes report the error until you run /unload and /load again.

/stop skips the approval prompt and also cancels the agent’s current response. What stopping means depends on the device, and each driver declares its own stop behavior. For example, an arm drops its queued motion, a mobile base comes to a halt, and a gripper keeps holding whatever it has grasped rather than opening.

Reading devices and camera images

device calls one device; devices calls several concurrently and returns results in input order. Batch entries use deviceID, action and args. Each result has its own ok status and data or error; the batch is neither atomic nor a simultaneous sample. Successful camera images survive a partial failure. The agent-facing batch is marked as an error only when every call fails.

Direct camera calls through device or devices attach RGB images automatically. Inside Python, camera reads default to NumPy arrays for computation and do not display images to the agent. Request output="file", then return selected paths with finish(result, file=[...]). Keep camera names and acquisition timestamps with the images; returning several together does not synchronize their clocks. See Returning camera observations.

Spatial and temporal relationships

Space: frames and extrinsics

Every device has its own coordinate frame, and some devices have more than one. An arm, for instance, has frames for its base, each link and its flange, computed from its joint model. A device’s extrinsics say where it is mounted relative to another device’s frame. A wrist camera mounted on an arm’s flange looks like this:

extrinsics:
  mount:
    parent: arm              # the device it is attached to
    parent_frame: flange     # which frame on that device
    translation_m: [0.04, 0, 0.02]
    rotation_xyzw: [0, 0, 0, 1]

Linking devices this way builds a tree of frames. As the arm moves, Kinex follows the chain through the arm’s measured joints, so it can work out where the camera is at any moment. That is how relative_pose turns a point that locate_object_3d finds in the camera’s frame into a target in the arm’s frame.

Some rules for frames:

• Poses are written as [x, y, z, qx, qy, qz, qw]: position in metres and orientation as a unit quaternion. Cameras use optical axes (X right, Y down, Z forward).
• Unknown values stay ??? until they’re measured. Filling them in is what calibration does (see Onboarding & calibration).
• There is not always a single tree. If two frames aren’t linked, Kinex treats their relationship as unknown rather than assuming they line up.
• A shared world frame is optional. Mobile and legged bases don’t know their position in the world, so relating them to the world needs an explicit estimate.

Time: the world clock

Each device timestamps its readings with its own clock, in seconds. These clocks are not synchronized with each other automatically.

The world device provides the clock the task runs on. On real hardware it follows real (monotonic) time; in a simulator, the simulator’s driver supplies its own clock.

• world.read returns the current time time_s, and may also report an episode budget as budget: {used_s, max_s}. Kinex shows the budget but doesn’t enforce it.
• world.wait({duration_s}) lets time pass and returns the new time_s and how long it actually waited (waited_s).

Simulators run in one of two modes:

• Realtime: the simulation keeps running on its own, including while the agent is thinking.
• Stepped: simulated time only moves forward during world.wait. Reads never advance it, and a host sleep doesn’t either.

So in a stepped simulator, commanding a motion and then reading the result isn’t enough on its own: call world.wait in between to let the motion happen. Blocking servo already waits its planned duration on a loaded world clock, so do not wait that duration twice. It does not verify arrival. With block=False, or asynchronous device writes and run_policy, scripts must wait explicitly and then check fresh feedback.

Tools & skills → Page

Tools execute; skills instruct. Both complement the host’s coding tools (read, write, edit, bash, grep, find, ls).

Tools

Use bounded Python scripts in exec to combine control, fresh feedback and camera observations. The former closed_loop and settled built-ins have been removed; put their loop state and task-specific stopping checks in Python. See Feedback control.

More detail: Using vision models as tools, Using VLAs as tools.

Bundled skills

Discovery

Kinex looks for tools and skills in a .kinex/ folder, searching these places in order:

1. the .kinex/ nearest the current directory;
2. .kinex/ folders in enclosing directories further up;
3. your global config, ~/.config/kinex/;
4. the tools and skills built into Kinex.

When two places define the same name, the first one found wins. A project can therefore override a global or built-in Python tool or skill by defining one with the same name.

Custom discovery uses the plural tools/ and skills/ directories, including in global config. Move older custom content out of singular tool/ and skill/ directories. Packaged assets/builtin/tool/ and assets/builtin/skill/ paths are unchanged.

Tools

.kinex/tools/
├── my_tool/
│   ├── tool.yaml     # name, kind, description, parameters (JSON Schema)
│   └── tool.py       # implementation
└── helpers.ts        # JS/TS tools, one or more per file

• Python tools live in their own folder with a tool.yaml manifest and a tool.py. The tool’s name comes from name in the manifest.
• JS/TS tools are .ts or .js files placed directly in tools/. Each export with a description and an execute function (plus an optional Zod parameters schema) becomes a tool. A default export is named after the file; any other export is named <file>_<export>.
• Tools are rescanned on every agent step, so a tool you write or edit mid-task is available at the next step.
• The core tools listed above (device, exec, skill and so on) always take priority. A custom tool with the same name is skipped, and Kinex reports the conflict.

Skills

.kinex/skills/
└── pick-and-place/
    ├── SKILL.md      # frontmatter (name, description), then instructions
    └── references/   # optional supporting files

• Kinex finds every SKILL.md under skills/, at any depth. The skill’s name comes from name in the frontmatter, or the folder name if that is missing.
• The agent sees each skill’s name and description up front. When a description matches the task, it loads the full instructions with the skill tool.
• Write the description carefully. It is what the agent uses to decide whether to load the skill.

Write your own: Custom tools & skills.

Kinex runtime → Page

The Kinex runtime (@kinex/core) sits between the agent and the robot. Every robotics tool call the agent makes, and every device command you type (/load, /stop, /record…), passes through it. The runtime holds no model, conversation or UI; those come from the host (standalone Kinex, or a plugin for another harness). What it does provide is everything needed to use devices safely and consistently.

Device management

Each device runs on its own. Its driver is a separate process with its own resources, such as an SDK connection, a camera stream or a model loaded on a GPU, so one device failing or hanging doesn’t take down the others.

• Loading in parallel. /load all starts every driver at once and waits for each independently, so a slow model server doesn’t hold up the arm.
• Scheduling per device. Writes to one device are serialized; a new write supersedes older pending commands. Reads can run alongside writes, and different devices work concurrently. devices uses this to read a camera and an arm in one call.
• Clean release. /unload, quitting or switching workspace shuts down the processes Kinex started. Services it only connected to, like a simulator you launched yourself, are left running.
• Surviving session changes. Starting a new conversation or resuming an old one in the same workspace keeps devices connected. In-flight calls are cancelled, session approvals are reset and any active recording ends.
• No silent restarts. If a driver fails, it stays failed until you reload it with /unload and /load.

See Devices for types, profiles and drivers.

Tool and skill discovery

The runtime rescans tools and skills on every agent step. Drop a new Python tool into .kinex/tools/ or a new SKILL.md into .kinex/skills/, and the agent can use it at the next step, without restarting Kinex. The agent can do the same: it can write itself a tool mid-task and then call it. Project, global and built-in locations are searched in a fixed order, so a project can override a shared tool or skill. See Tools & skills.

Permissions

Every tool call the agent makes is checked before it runs. Your own slash commands, like /load or /record, are trusted and never checked. How often you’re asked depends on the approval mode; in the default ask mode, Kinex follows these permissions:

To change the defaults, set rules in .kinex/kinex.json for one project or ~/.config/kinex/kinex.json for all of them. Each value is allow, ask or deny; the project file wins.

{
  "permission": {
    "device_read": "allow",
    "device_act": "ask",
    "shell": "ask"
  }
}

A few rules always hold, whatever the mode:

• /stop and device_stop never need approval. /stop also cancels the agent’s response. A direct model device call with action: "stop" still requests device_act.
• One approval for exec covers every device call that script makes.
• Only your commands and the device tool can load or unload devices. Python tools and scripts cannot.
• Standalone model-tool batches containing Kinex tools run in order, so denying a call stops later calls without rolling back completed work. Inside one devices call, device requests run concurrently and report independent outcomes.

Approvals are a safeguard, not a sandbox. They don’t isolate the agent from your system, and they don’t make motion safe on their own: some operations can’t be interrupted by their driver, so know where your hardware emergency stop is.

Spatial relationships

The runtime keeps a graph of every frame it knows: each device’s base, the frames computed from an arm’s joint model, and the mounts declared in each profile’s extrinsics. Arm frames are recomputed from measured joint positions, so the graph follows the robot as it moves.

relative_pose asks this graph a question: where is one frame, or a pose attached to it, relative to another? For example, the pose of an object seen by the wrist camera, expressed in the arm’s base frame:

{
  "child":  {"frame": "wrist_cam", "pose": [0.02, -0.01, 0.35, 0, 0, 0, 1]},
  "parent": {"frame": "arm"}
}

The runtime finds the path between the two frames, chains the transforms along it, and returns the pose together with the path it used. If any link on that path is unknown, the call fails rather than guessing. You can pass an estimate for a missing link in the call itself; it applies only to that call and is never saved.

An arm’s frames and its forward and inverse kinematics both come from its joint model; servo calls the arm’s inverse kinematics to turn a pose target into a joint trajectory. See Devices.

Time and motion

The runtime owns the link between motion and the world clock. By default, servo waits its planned trajectory duration on a loaded world clock. That wait does not confirm arrival; check fresh measured feedback afterward. For replanning during motion, use block=False in Python and choose an explicit interval with device("world", "wait", {"duration_s": interval_s}). A stepped simulator advances only during world waits.

Use bounded loops in exec for feedback control. Scripts and their composed tools share motion ownership. On failure, timeout or cancellation, Kinex attempts to stop motion still owned by that invocation; cleanup failures are logged while the original error is returned. Successful execution may leave motion running. Call stop_owned() when the loop must stop on exit and surface any cleanup error. Ownership checks preserve other calls’ replacements, subject to the driver’s physical stop scope. See Feedback control.

Python environment

By default, drivers, Python tools and exec scripts run in one Python ≥ 3.11 environment per workspace, .kinex/python/, managed with uv. Kinex creates it the first time it is needed and never touches your own Python projects. Inside it, the kinex library gives scripts the same device access the agent has:

from kinex import device, devices

frame = device("camera", "read", {"output": "file"})

Explicit interpreters use their own dependencies. Ordinary shell tools retain their existing environment; Kinex does not activate .kinex/python/ in them. Existing managed environments are reused without automatic dependency synchronization. See Custom tools & skills.

Context for the agent

Before each request, the runtime gives the agent an up-to-date picture of the workspace: which devices exist and which are loaded, how they’re mounted, what still needs calibration, the usage guide for each loaded device type, shared conventions such as frames and units, the available skills and the Python environment. Changes are appended to the conversation, not rewritten, and this context is restored after the conversation is compacted. See Working with the agent.

Working with the agent → Page

Kinex is built on the Pi coding agent (@earendil-works/pi-coding-agent, bundled at version 1.2.0, so there’s nothing extra to install). Pi provides the agent itself: the model connection, the conversation loop, the terminal UI, sessions, compaction and the everyday coding tools (read, write, edit, bash, grep, find, ls). Kinex adds the robotics on top: devices, robotics tools and skills, permissions and recording.

Apart from approvals and devices, everything on this page works the way it does in Pi. If you’ve used Pi or a similar coding agent before, it will feel familiar. For more detail, see Pi’s documentation.

Choosing a model

The agent model is the LLM that reads your requests and decides what to do. It’s separate from the perception and policy models your robot uses, which are devices (see Using vision models as tools and Using VLAs as tools). Slash commands like /ping don’t need a model at all.

/login              # connect a provider, with an OAuth subscription or an API key (or set one of the API-key environment variables Pi recognizes)
/model              # pick a model
/thinking high      # set reasoning effort

You can also choose from the shell:

kinex --model provider/model
kinex --model sonnet --provider anthropic     # match a model name within one provider
kinex --thinking xhigh                        # off, minimal, low, medium, high, xhigh or max

Local or custom models

To use a model you host yourself, add it to ~/.config/kinex/agent/models.json:

{
  "providers": {
    "local": {
      "api": "openai-completions",
      "baseUrl": "http://localhost:11434/v1",
      "apiKey": "local",
      "models": [{ "id": "your-model-name" }]
    }
  }
}

Start the model server, then choose local/your-model-name with /model. See Pi’s custom provider docs for all options.

Where credentials are kept

Credentials live in ~/.local/share/kinex/agent/auth.json, with a link at ~/.config/kinex/agent/auth.json so every part of Kinex shares one login. If that location already holds different credentials, Kinex stops with AuthPathConflict rather than overwrite them; merge the two files and restart. Keep agent credentials out of device profiles. Model devices read their own keys from environment variables.

Talking to the agent

Type a request and press Enter. While the agent is working, you can keep talking to it:

A few other commands help day to day: /copy copies the agent’s last reply, /settings opens preferences and /hotkeys lists every shortcut. /quit exits. Escape only stops the agent’s response; to stop the robot as well, use /stop.

Kinex also bundles two Pi extensions: pi-web-search, which lets the agent search the web, and pi-btw, for side conversations that don’t interrupt the main one. Side conversations use Pi’s regular coding tools only, without your devices or Kinex’s approval checks. You can add more Pi packages; see Plugin systems.

Approval modes

Before the agent runs a tool, such as reading a device, moving the robot or running a shell command, Kinex checks whether it needs your approval. The approval mode sets how much it asks:

Switch with /mode (or /mode auto), or start in a mode with kinex --auto or kinex --yolo. The current mode is shown in the footer.

When Kinex asks, you can allow once, allow for the rest of the session, or deny. Denying stops the rest of that turn, so related calls don’t run without you. Switching mode clears the approvals you’ve granted for the session.

In scripted runs without a terminal, nobody is there to answer, so in ask mode anything that would prompt is denied and the agent’s questions are answered from the evidence it already has. See Kinex runtime for what each permission covers and how to set rules.

Sessions

A session is one conversation: your messages, the agent’s replies and every tool call. It’s separate from the workspace, the .kinex/ folder that holds your devices, tools, skills and recordings. Switching sessions never changes the workspace, and vice versa.

Each session is saved automatically. When you quit, Kinex prints the command to come back to it. To return later:

kinex --continue              # the latest session in this workspace (-c)
kinex --resume                # choose from a list (-r)
kinex --session 3f2a          # a specific session, by ID or the start of one

Inside Kinex, /new starts a fresh session and /resume switches to another. Sessions are stored as JSONL files under ~/.local/share/kinex/agent/sessions/, grouped by workspace.

Branching

Pi keeps each session as a tree, so you can go back and try something different without losing what came before:

What happens to your devices

Changing session within the same workspace (/new, /resume, /fork) keeps your devices connected, so you don’t have to load them again. It does cancel any call still running, reset the permissions you granted for the session and finish any active recording. Quitting Kinex or switching to another workspace disconnects the devices Kinex started; services you started yourself, like a simulator, keep running.

Context and compaction

Long conversations eventually fill the model’s context window. Compaction summarizes the conversation so far so you can keep going:

/compact                                      # summarize now
/compact keep the calibration results in detail   # tell it what to preserve

Kinex makes sure compaction doesn’t lose track of the robot. Before every request it gives the agent a fresh picture of the workspace: loaded devices, how to use them, and the Python environment. That picture is rebuilt in full after compaction. The robotics guidance asks the agent to avoid unnecessary tool calls and motion, account for intended contact, and check the full path for unintended collisions. Command acceptance or arrival alone does not establish task success. Old observations in the history are still just history, though. When the current state matters, the agent should read the device again.

The footer shows how long you’ve been actively working in the session, not counting idle time.

Exporting a conversation

To save or share a conversation:

/export [path]        # save this session as HTML or JSONL

kinex --export session.jsonl [output.html]    # convert a saved session, no model needed

Exports include file paths, tool output and images, so review them before sharing. To capture what the robot saw and did, use Recording instead.

Onboarding & calibration → Page

Getting a new robot ready means two things: describing the hardware so Kinex can talk to it (onboarding), and measuring the physical facts a task depends on (calibration). Both are guided by the agent, and both are driven by your task: Kinex prepares and measures only what that task needs.

Onboarding

Start Kinex in your robot workspace, then describe your hardware and goal in your own words. For example:

I have a robot arm, a gripper, and an overhead camera. Help me set them up to pick objects from the table.

Share any connection details, manuals, or SDKs you already have. The agent checks the workspace, asks for missing information, and prepares the device connections and drivers. It then checks that the devices respond and identifies any calibration the task needs.

You can also ask it to prepare the hardware before choosing a task. When you add a new goal later, it reuses the setup and measurements that still apply.

One device at a time

To add or remove a device, tell the agent what changed:

Add this depth camera to my robot. Here is its connection information and documentation.

I no longer use the wrist camera. Remove it from this workspace.

The agent prepares and checks new devices before use. Removing a device requires confirmation because it deletes that device’s configuration and files. See Devices for more detail.

Calibration

Calibration measures facts such as where a camera sits, where the tool tip is, or how two arms are positioned relative to each other. Explain what you want the robot to do, or what has changed:

Help the arm pick objects using the overhead camera. Check what needs calibrating first.

I moved the wrist camera. Update its calibration before we continue.

The agent chooses a suitable procedure and guides you through it:

1. Prepare. Reuse reliable manufacturer or robot-model information and identify the measurements still needed.
2. Measure. Guide robot movements or ask you to position a target, then collect images and readings.
3. Check. Calculate the result and assess whether it meets the task’s accuracy needs.
4. Apply. Save an accepted result to the device configuration. If the measurements are insufficient, explain what needs improving.

If you choose to keep an approximate result, the agent retains its measured error and quality assessment. A result that failed a quality check is still recorded as such.

Sampling and evidence

The agent checks that the robot has reached the intended position before collecting measurements. It saves calibration images, readings, and results so they can be reviewed later. When hardware moves or changes, it reassesses the affected measurements.

Bundled calibration skills

Describe the outcome you need; the agent selects the procedure.

For custom procedures, see Tools & skills.

Calibration in practice

These recordings show Kinex calibrating a simulated YAM dual-arm robot in Isaac Lab, through RoboEnv. Each plays at 5× speed.

All set!

Ask the agent to connect the devices and check that they respond, then describe your first task. For example:

Check that the arm, gripper, and camera are ready, then pick up the blue block from the table.

If you prefer command shortcuts, see Slash commands.

ROS 2 onboarding → Page

Reference onboarding video

Supported setup

Use your robot’s existing ROS 2 installation. Kinex runs ROS drivers in a separate process with the matching ROS environment and Python interpreter—for example, Jazzy on Ubuntu 24.04 with Python 3.12, or Humble on Ubuntu 22.04 with Python 3.10. Onboarding handles the additional compatibility setup needed for ROS interpreters older than Python 3.11.

Before onboarding, install Kinex and open a robot workspace.

Where ROS 2 fits

ROS 2 connects through Kinex’s device layer. Adapters translate between Kinex device operations and ROS interfaces for joint trajectories, velocity commands, cameras and IMUs.

What onboarding handles

The standard onboarding workflow uses the onboard-ros2 skill to:

• Reuse packages. Prefer installed ROS packages, then official apt packages. Build from source only for a specific need or your request.
• Resolve dependencies. If installation requires sudo, provide the commands and target host for you to run them on.
• Prepare drivers. Write drivers and ROS configuration in each device’s directory, reusing existing nodes and a shared ROS backend in .kinex/lib/ where appropriate.
• Verify interfaces. Check live ROS interfaces and report observations and failed checks. Apply measured camera calibration updates to device profiles.

Start the robot’s ROS stack before loading devices; unloading a Kinex device leaves the stack running.

Continue with calibration for the measurements your task needs.

Custom tools & skills → Page

There are three ways to extend what Kinex can do, from quickest to most reusable:

• Run a Python script for a one-off job, like inspecting a device or trying out a motion.
• Add a tool when an operation is worth reusing: the agent can then call it by name like any built-in tool.
• Add a skill to teach the agent how to approach a kind of task.

The agent can write these for you: ask it to “turn that into a tool” or “write a skill for this”. You can also write them by hand. Either way they’re files in your workspace, so they’re easy to review, edit and share.

Running Python

The agent runs Python through the exec tool. Supply exactly one of python (source) or file (a script path). No manifest, main() function or kind is needed. Optional description gives the invocation a short display label, such as inspect arm state.

A snippet that reads the arm and a camera together:

from kinex import devices

for result in devices(calls=[{"deviceID": "arm"}, {"deviceID": "camera"}])["results"]:
    print(result["deviceID"], result["ok"])

Or an existing script, with arguments:

{"file": "scripts/inspect.py", "argv": ["--samples", "10"], "args": {"device": "arm"}}

Each invocation runs in a fresh process from the runtime’s working directory. args is available through kinex.args(); argv supplies sys.argv[1:]. Files support script-local imports and __name__ == "__main__". Optional interpreter selects another Python executable.

The result contains exit_code, bounded combined stdout/stderr in output (up to the last 16 KB), and result (normally null). finish(value) supplies a structured result; it can also attach one or more images. Scripts time out after 120 wall-clock seconds unless given a different timeout_s; composed tools also enforce their own timeouts within the remaining time.

A few things to know:

• One approval covers the script. You approve exec once, and that covers every device call the script makes.
• Scripts can’t load or unload devices or start recordings; those stay with you and the agent.
• Choose when motion stops. On failure, timeout or cancellation, Kinex attempts to stop motion still owned by the script. Cleanup failures are logged while the original error is returned. Successful scripts can leave motion running; call stop_owned() when the exit condition requires a stop, or use /stop yourself.
• Inspect batch errors. devices() returns per-entry results. An entry with ok: false does not raise an exception by itself; check it before continuing control.

The kinex library

Scripts and tools use the kinex Python library to talk to the runtime:

from kinex import call, device

state = device("arm")
call("servo", device="arm", joints=[0, -0.5, 0, 1.2, 0, 0.8, 0], block=False)

Pass tool parameters as keywords: call("servo", **params), not call("servo", params). In contrast, device("world", "wait", {"duration_s": 0.1}) takes an argument dictionary. A failed call raises kinex.RPCError, which carries a code and details.

Returning camera observations

Python camera reads default to arrays and do not attach images automatically. For an agent decision that needs visual input, request files and return them explicitly:

from kinex import device, finish

observations = []
images = []
for camera in ["head_camera", "wrist_camera"]:
    frame = device(camera, args={"output": "file"})
    observations.append({"deviceID": camera, "timestamp": frame["timestamp"]})
    images.append(frame["file"])
finish({"observations": observations}, file=images)

file accepts one path or an ordered list. When attaching files, the result must be a JSON object (or omitted); Kinex includes the paths in its file field. Printing paths or returning base64 text does not attach images. Camera names and timestamps identify the separate observations.

Feedback control

Use bounded Python loops in exec for control, measured feedback and task-specific stopping checks. The closed_loop and settled built-ins are no longer provided. For an arm with bounded revolute joints, the following script takes target joints and a positive tolerance_rad through exec’s args:

from kinex import args, call, device, finish, stop_owned

params = args()
try:
    initial = device("arm")
    if initial["faults"]:
        raise RuntimeError(str(initial["faults"]))
    previous_stamp = initial["timestamp"]
    motion = call("servo", device="arm", joints=params["joints"])
    # Blocking servo already waited the planned duration. Check fresh feedback.
    for _ in range(20):
        feedback = device("arm")
        if feedback["faults"]:
            raise RuntimeError(str(feedback["faults"]))
        if feedback["timestamp"] < previous_stamp:
            raise RuntimeError("device clock moved backwards")
        fresh = feedback["timestamp"] > previous_stamp
        previous_stamp = feedback["timestamp"]
        error = max(abs(q - goal) for q, goal in
                    zip(feedback["positions"], motion["joints"], strict=True))
        if fresh and error <= params["tolerance_rad"]:
            break
        device("world", "wait", {"duration_s": 0.1})
    else:
        raise TimeoutError("joint target not confirmed")
finally:
    stop_owned()

frame = device("wrist_camera", args={"output": "file"})
finish({"feedback": device("arm"), "camera": "wrist_camera",
        "timestamp": frame["timestamp"]}, file=frame["file"])

Adapt the names, units and tolerances to the workspace. Continuous joints need angular error modulo a full turn. Position agreement alone does not establish stability, grasp success or task completion; check the feedback the task needs. Missing feedback and faults must stop the loop.

For replanning during motion, use block=False and explicit world waits between observations. Blocking servo already waits its planned duration on a loaded world clock; waiting that duration again is unnecessary. When further motion depends on visual interpretation, return selected images to the agent before continuing. stop_owned() checks ownership before stopping, subject to the driver’s declared physical stop scope.

The Python environment

By default, scripts, tools and drivers run in one shared Python ≥ 3.11 environment per workspace, in .kinex/python/. A driver or tool can use its own interpreter when it needs isolated dependencies, as model devices often do. Kinex creates it with uv the first time it’s needed, and never touches your own Python projects. To add a package:

uv add --project .kinex/python opencv-python-headless

Adding a tool

Turn something into a tool when you or the agent will want it again. A tool has a name, a description and declared inputs, so the agent can find it and call it correctly without re-reading any code.

To add a Python tool, create a folder under .kinex/tools/ with two files. Here is a tool that estimates whether a closed gripper stopped on an object instead of closing all the way:

.kinex/tools/grasp_check/
├── tool.yaml
└── tool.py

tool.yaml describes the tool to the agent:

name: grasp_check
kind: perception        # required by the manifest schema
description: Estimate whether a closed gripper is holding an object, from its measured opening.
parameters:
  type: object
  properties:
    gripper: {type: string, description: "Gripper device name"}
    min_aperture: {type: number, default: 0.005}
  required: [gripper]

tool.py does the work, reading its inputs with args() and returning with finish():

from kinex import args, device, fail, finish

a = args()
aperture = device(a["gripper"])["aperture_m"]
if aperture is None:
    fail("this gripper doesn't report its opening")
finish({"holding": aperture > a.get("min_aperture", 0.005), "aperture_m": aperture})

That’s all. Kinex picks up new and changed tools at the agent’s next step, without a restart, so the agent can write a tool and use it in the same task. The quickest way to start is to copy one of the built-in tools and adapt it.

You can also write tools in TypeScript or JavaScript: put a .ts or .js file directly in .kinex/tools/ that exports a description, parameters (a Zod schema) and an execute function.

Adding a skill

A skill is a written procedure: how to approach a kind of task, what to check, and what to watch out for. Add one when you find yourself giving the agent the same guidance again and again.

Create a folder under .kinex/skills/ with a SKILL.md:

---
name: pick-and-place
description: Pick an object from the table and place it in a target, using the wrist camera to locate both.
---

# Pick and place

1. Read the wrist camera and locate the object with `locate_object_3d`.
2. Convert its position into the arm's frame with `relative_pose`.
3. ...

The agent always sees each skill’s name and description, and loads the full instructions only when a description matches the task. So write the description to say clearly when the skill applies. You can keep supporting material, such as reference notes or helper scripts, in the same folder and mention it in the instructions.

Where to put them

Tools and skills in a project’s .kinex/ folder apply to that project. To make one available everywhere, put it in ~/.config/kinex/tools/ or ~/.config/kinex/skills/ instead. When two have the same name, the one closest to your project wins; see Tools & skills for the details.

Custom tools are code that runs on your machine with access to your robot. Review them as you would any other code, especially ones written by the agent or shared by others.

Using VLAs as tools → Page

A vision-language-action model (VLA) takes camera images and a task description and predicts the robot’s next moves. In Kinex, a VLA is not the agent; it’s a tool the agent uses. The agent decides when to run the policy, on which devices, for how long, and when to stop.

This takes three pieces:

1. The VLA becomes a device. It’s registered like a camera or an arm, as a model device with an infer operation.
2. run_policy runs it once. It sends one observation to the model and passes the predicted actions to the robot.
3. Python exec coordinates the cycle. It gathers fresh observations, runs bounded policy chunks, checks feedback and returns images for the agent to assess.

This page uses StarVLA on a RoboTwin dual-arm robot as the example.

1. Register the VLA as a device

Let the agent set it up. Tell /register which model you want and what robot it’s for:

/register model StarVLA/Qwen3-VL-OFT-RoboTwin2-All policy for the dual-arm RoboTwin robot

The agent follows the built-in setup-device skill. It reads the model’s documentation, downloads and pins the weights, builds an isolated Python environment for the model, writes a driver and a profile, and checks what it can without loading the model. The device is left unloaded. Loading it puts the model on the GPU, so load it only when you need it:

/load starvla

The result is an ordinary device in .kinex/devices/starvla/. Its profile records where the model runs and, most importantly, what it takes in and gives back:

name: starvla
type: model
access:
  connection: {kind: process, compute: {kind: cuda, index: 3}}
intrinsics:
  family: starVLA
  variant: Qwen3-VL-OFT-RoboTwin2-All
  operations:
    - name: infer
      input:   # three camera images (file paths) and a task instruction
        obs: {head_image, left_image, right_image}
        task: string
      output:  # 50 steps × 14 values
        actions: [[left_arm ×6, left_gripper, right_arm ×6, right_gripper], ...]

(Abridged; the real profile spells this out as JSON Schema.)

Each prediction is a chunk of 50 actions. Each action holds 14 numbers: six joint positions and a gripper value for each arm. The model only predicts. It never moves the robot itself.

2. Run the policy once with run_policy

run_policy connects the model to the robot. It sends the observation and task to the model, splits each predicted action across the devices you list, and sends the result to them:

{
  "model": "starvla",
  "observation": {
    "head_image": "/tmp/obs/head.png",
    "left_image": "/tmp/obs/left.png",
    "right_image": "/tmp/obs/right.png"
  },
  "task": "place the bottle in the basket",
  "command": [
    {"device": "left_arm"},
    {"device": "left_gripper"},
    {"device": "right_arm"},
    {"device": "right_gripper"}
  ],
  "action_hz": 25,
  "max_actions": 25
}

• command lists the devices in the same order as the model’s output. Here that’s 6 values for left_arm, 1 for left_gripper, and so on. Values are sent as absolute joint positions by default. A model that predicts end-effector poses or pose changes can set "space": "pose" or "pose_delta", and Kinex converts them to joint positions with inverse kinematics.
• action_hz is how fast the actions are played back.
• max_actions limits how much of the chunk is used. Here only the first 25 of the 50 actions, one second of motion, are sent. The robot then gets a fresh prediction instead of following a stale plan.

run_policy returns as soon as the actions are sent, reporting how many went out and for how long. It doesn’t wait for the motion to finish; that’s the loop’s job.

3. Observe, act and inspect with exec

Use Python variables and a bounded loop to gather observations and run policy chunks. This example runs at most three chunks, then stops owned motion and returns all three camera views for the agent’s next decision:

from kinex import call, device, finish, stop_owned

cameras = {"head_image": "head_camera", "left_image": "left_camera",
           "right_image": "right_camera"}
actuators = ["left_arm", "left_gripper", "right_arm", "right_gripper"]
commands = [{"device": name} for name in actuators]
chunks = []
try:
    for _ in range(3):
        for name in actuators:
            state = device(name)
            if state["faults"]:
                raise RuntimeError(f"{name}: {state['faults']}")
        observation = {
            key: device(camera, args={"output": "file"})["file"]
            for key, camera in cameras.items()
        }
        chunk = call("run_policy", model="starvla", observation=observation,
                     task="place the bottle in the basket", command=commands,
                     action_hz=25, max_actions=25)
        if chunk["ik_failure"] or chunk["actions_submitted"] == 0:
            raise RuntimeError(f"policy chunk rejected: {chunk}")
        chunks.append(chunk)
        device("world", "wait", {"duration_s": chunk["duration_s"]})
finally:
    stop_owned()

feedback = {name: device(name) for name in actuators}
for name, state in feedback.items():
    if state["faults"]:
        raise RuntimeError(f"{name}: {state['faults']}")
observations, images = [], []
for camera in cameras.values():
    frame = device(camera, args={"output": "file"})
    observations.append({"deviceID": camera, "timestamp": frame["timestamp"]})
    images.append(frame["file"])
finish({"reason": "inspection required", "chunks": chunks,
        "feedback": feedback, "observations": observations}, file=images)

Adapt the camera names, action order and feedback checks to the actual model and robot. The three-chunk limit bounds this example; it does not establish task success. If each chunk needs visual interpretation, run just one and return images before continuing. Ask the agent to run a short policy segment, inspect whether the bottle reached the basket, and decide what comes next.

• Timing. run_policy returns immediately with actions_submitted, duration_s, execution_ids and ik_failure. Wait on the world clock for the returned duration before assessing the resulting state. In a stepped simulator, inference time does not advance the world.
• Rejected actions. IK rejection can return normally with a shortened or empty valid prefix. Inspect ik_failure; a successful tool call does not imply the whole chunk was accepted.
• Replanning. A new chunk can replace the remaining references of the previous one. Choose an explicit shorter world-wait interval if the task needs replanning during execution.
• Cleanup. stop_owned() in finally requests a stop on every exit. Runtime failure, timeout or cancellation also triggers cleanup attempts; successful execution alone does not stop motion.

The former closed_loop and settled tools are no longer bundled. See Feedback control for a measured joint-target check, and vision models for perception-driven motion.

Using vision models as tools → Page

Vision models answer questions like “where is the red mug?” by finding objects in camera images. In Kinex, a vision model is not the agent; it’s a tool the agent uses. The agent decides what to look for, which camera to use, and what to do with the answer.

This takes three pieces:

1. The vision model becomes a device. It’s registered like a camera or an arm, as a segmentation or detection device with a detect operation.
2. locate_object_3d uses it once. It takes a camera image, asks the model to find the object, and turns the result into a 3D position.
3. Python exec combines perception and action. It converts the position for the arm, commands motion and returns fresh observations for the agent to check.

This page uses SAM3, an open-vocabulary segmentation model, as the example. Grounding DINO works the same way and is also built in.

1. Register the vision model as a device

Let the agent set it up:

/register sam3

The agent follows the built-in setup-device skill and SAM3’s own setup guide. It asks whether to run on a GPU or the CPU, checks your Hugging Face access, installs the model into its own environment inside .kinex/devices/sam3/, downloads the weights, and writes a driver and a profile. The device is left unloaded. Loading it puts the model on the GPU, so load it only when you need it:

/load sam3

SAM3’s weights are gated on Hugging Face. Request access at facebook/sam3 and make a token available (for example as HF_TOKEN) before registering.

The result is an ordinary device. Its profile records where the model runs and what it can do:

name: sam3
type: segmentation
access:
  connection: {kind: process, compute: {kind: cuda, index: 0}}
intrinsics:
  family: sam3
  operations:
    - segment_point   # a clicked point → masks
                      # detect (text prompt → boxes, scores, labels and masks)
                      # comes with the segmentation type

(Abridged; the real profile spells out each operation as JSON Schema.)

Every segmentation and detection device answers the same detect call: give it an image and a text prompt, and it returns the matching objects. Each object has a bounding box in image pixels, a score and a label; segmentation models also return a mask. It may also return an overlay image showing what it found; SAM3 always does. The model only looks at the image you give it. It never takes a picture itself.

2. Locate an object with locate_object_3d

locate_object_3d connects the vision model to a camera. It reads the camera, asks the model to find the object, then uses the depth image to work out how far away it is:

{
  "camera": "wrist_cam",
  "model": "sam3",
  "prompt": "red mug"
}

• camera must be calibrated (see Onboarding & calibration). An RGB-D camera supplies depth itself. For a plain RGB camera, pass a known distance as depth.
• model is any detection or segmentation device.
• prompt describes the object in plain words.
• Optional: min_score ignores weak matches (default 0.25), and instance picks another match when there are several (0 is the best).

The result is a point in metres, in the camera’s frame (X right, Y down, Z forward). It comes with the detection score, how the depth was measured and the overlay image, which the agent can look at to check it found the right thing:

{
  "frame": "wrist_cam",
  "point": [0.04, -0.02, 0.31],
  "label": "red mug",
  "detection_score": 0.91,
  "reference": "visible_surface"
}

With a segmentation model like SAM3, depth is measured inside the object’s mask, so the point lies on the object’s visible surface. A detection-only model gives just a box, so depth comes from the central half of the box and may include some background.

The point is in the camera’s frame. To use it with the arm, convert it with relative_pose, which works through the camera’s mount (see Spatial relationships).

3. Combine perception and motion with exec

Use a Python script to locate the mug, transform its position and command an approach. This example assumes the arm’s base Z axis points up and quaternion [1, 0, 0, 0] points its controlled tool frame down. Review that orientation, the clearance and the full path for your setup before motion.

from kinex import call, device, finish, stop_owned

try:
    initial = device("arm")
    if initial["faults"]:
        raise RuntimeError(str(initial["faults"]))
    found = call("locate_object_3d", camera="wrist_cam", model="sam3",
                 prompt="red mug")
    x, y, z = found["point"]
    mug = call("relative_pose",
               child={"frame": found["frame"], "pose": [x, y, z, 0, 0, 0, 1]},
               parent={"frame": "arm"})["pose"]
    target = [mug[0], mug[1], mug[2] + 0.10, 1, 0, 0, 0]
    motion = call("servo", device="arm", pose=target)
    # Servo waited its planned duration; inspect measured state next.
    feedback = device("arm")
    if feedback["faults"]:
        raise RuntimeError(str(feedback["faults"]))
    if feedback["timestamp"] <= initial["timestamp"]:
        raise RuntimeError("fresh arm feedback unavailable")
finally:
    stop_owned()

frame = device("wrist_cam", args={"output": "file"})
finish({"target": target, "motion": motion, "feedback": feedback,
        "reason": "inspection required", "camera": "wrist_cam",
        "timestamp": frame["timestamp"]}, file=frame["file"])

This commands an approach and returns evidence; it does not claim the arm arrived or the mug was grasped. The agent checks measured state and the new image before continuing. For a numerical arrival check, use the bounded feedback-control example with task-appropriate tolerances. For stability, also check fresh velocity feedback or a sequence of fresh position samples.

If the mug cannot be found, locate_object_3d raises an error before servo runs. When the next decision depends on confirming the detection visually, split the operation: return its overlay first, let the agent inspect it, then command motion. Images obtained inside Python must be returned through finish(..., file=...); printing their paths does not display them.

For moving targets, implement a bounded Python loop that re-detects, transforms, commands and checks feedback. Use block=False with servo and an explicit device("world", "wait", {"duration_s": interval_s}) between observations. Check faults, stale samples and elapsed time, and call stop_owned() when leaving the loop. The old closed_loop and settled built-ins are no longer provided.

The same pattern runs a policy model instead of a vision model; see Using VLAs as tools.

Bring your own model

To use a vision model that isn’t built in, register it from a type template and describe it to the agent:

/register segmentation my-segmenter served at http://gpu-box:8000, API key in SEG_API_KEY

As long as it answers detect in the standard form, with boxes and masks in the original image’s pixels, locate_object_3d and every loop built on it work unchanged.

Recording → Page

A recording captures what your devices observed and what they were commanded to do while Kinex works on a task, together with the conversation that drove it. Use it to review a run, compare trials, or collect demonstrations.

How to record

From the TUI

Load the devices first (/load arm); unloaded devices are refused. Then start a recording with the devices you want and how often to sample each one (in Hz). Anything after the rates is the task instruction:

/record start arm=10 gripper=10 wrist_cam=5 pick up the red mug and place it in the bin

Then let Kinex work, or steer it yourself. The recording keeps running across turns and while the agent is thinking. Along the way:

/record snap wrist_cam grasp attempt     # mark a moment and save a full-quality image
/record status                           # check rates, frame counts and any problems

When the task is done, stop and label how it went (success, failure or partial), with an optional note:

/record stop success Mug placed in the bin on the first try

The device list and rates are fixed once a recording starts. To change them, stop and start a new one.

In scripted runs

Pass the devices and rates to a headless run; the recording covers the whole task:

kinex -p --record "arm=10 gripper=10 wrist_cam=5" "Pick up the red mug and place it in the bin"

Letting the agent record

Start Kinex with --enable-recording to give the agent the record tool. It can then start, snap and stop recordings itself, for example to capture each attempt in a series of trials.

Camera recording needs ffmpeg on your PATH.

What you get

Each recording is a folder under .kinex/recordings/:

.kinex/recordings/rec_00mv475jct0000ur/
├── recording.json          # summary: task, devices, rates, counts, outcome
├── timeline.jsonl          # what happened when: start, tool calls, notes (including snaps), stop
├── trace.json              # the conversation: messages and tool calls (no image data)
├── profile/                # device profiles and workspace.yaml as they were at the start
├── devices/
│   ├── wrist_cam/
│   │   ├── stream/
│   │   │   ├── video.mp4   # the camera stream
│   │   │   ├── depth.mkv   # lossless depth, for RGB-D cameras
│   │   │   └── frames.jsonl  # timestamp of every video frame
│   │   ├── frames/         # full-quality images from /record snap
│   │   └── events.jsonl    # every call made to this device
│   └── arm/
│       ├── stream/state.jsonl  # sampled joint state
│       └── events.jsonl        # every call, including the commands sent
└── passive/                # other devices the agent used but you didn't list

The main files:

Some sample lines, from an arm’s state stream and a command sent to it:

{"positions": [-0.207, 0.473, 0.994, -0.704, -0.308, -0.329], "velocities": [0, 0, 0, 0, 0, 0], "timestamp": 0, "t_ns": 1791267198127000000}
{"t": 649, "seq": 4, "action": "write", "args": {"actions": [[-0.207, 0.473, 0.994, -0.704, -0.308, -0.329], ...], "duration_s": 0.5}}

Every stream carries timestamps, so cameras, joint states and commands can be aligned afterwards even though they were sampled at different rates.

Checking quality

recording.json reports, per device, how many samples were taken, the rate actually achieved (effectiveHz), missed capture slots (skipped) and read errors. Video plays at a fixed frame rate; when a camera can’t keep up, the last image is repeated so elapsed time stays correct, and repeated counts those frames. Check these before relying on a recording. A device that fails doesn’t stop the others.

Recordings end when you stop them, when the size limit is reached, when you start a new session, or when you change workspace. Unloading a device ends its part of the recording.

Using recordings

Each recording keeps the instruction, the outcome, the conversation and the exact device data, which makes it useful beyond review:

• Refining skills and tools. Point the agent at past recordings, for example “review the last three failed runs and update the pick-and-place skill”. It can compare failed and successful attempts and turn what it finds into better skills, tools or parameters.
• Training models. Observations, commanded actions and outcome labels are already in the shape of robot-learning episodes, so recordings can be turned into training data for policies that then run through run_policy.

To save just the conversation, use /export instead; see Working with the agent.

Plugin systems → Page

Plugins work in two directions with Kinex:

• Into Kinex. Kinex is built on Pi, so it can install and run plugins from Pi’s ecosystem: extensions, skills, prompt templates and themes.
• Out of Kinex. Kinex’s robotics layer is itself packaged as plugins, so you can use your devices, tools and skills from Pi, Claude Code or Codex instead of the Kinex app.

Installing Pi plugins

Pi plugins are distributed as packages, from npm, a Git repository or a local folder. Kinex manages them with the same commands as Pi. Browse what’s available at pi.dev/packages.

Kinex already bundles two: pi-web-search for web search and pi-btw for side conversations.

For example, to pin web search to a specific version, check it, and later go back to the bundled one:

kinex install npm:pi-web-search@1.7.0      # install a specific version, replacing the bundled copy
kinex list                                 # see installed and bundled packages
kinex update --extensions                  # update installed packages, keeping version pins
kinex remove npm:pi-web-search             # remove it; the bundled version takes over again

Other sources work the same way:

kinex install git:github.com/me/my-pi-package@v1    # from a Git repository, at a tag
kinex install ./my-pi-package -l                    # from a local folder, for this project only
kinex update npm:pi-web-search                      # update a single package

Packages install for your user by default; add -l to install for the current project only. Restart Kinex after installing, updating or removing a package so the change takes effect. See Pi’s package documentation to write your own.

Using Kinex from other harnesses

The kinex-plugins repository currently supports Pi, Claude Code and Codex. It brings Kinex’s devices, robotics tools, skills and workspace context into those hosts. Your harness keeps its own coding tools, sessions, credentials and approval UI. You don’t need the standalone Kinex app installed. OpenCode is not currently supported.

All three use the shared @kinex/mcp adapter and packed @kinex/core/@kinex/setup snapshots. Plugin releases are versioned separately; updating standalone Kinex does not update a plugin’s snapshot. Check the plugin’s bundled version before relying on newly added tools or runtime behavior. Hosts can use the same .kinex/ workspace in separate sessions; independent hosts do not coordinate concurrent control of the same physical devices.

Installing a plugin

Install from a local checkout on Linux or macOS; npm publication is not required. The plugins need Node.js 22.22.2 or newer. Build with the repository’s selected Node version:

git clone https://github.com/stacx-lab/kinex-plugins.git
cd kinex-plugins
nvm use
npm ci
npm run build
export KINEX_SOURCE="$PWD"

For Pi, install the package directly from this checkout:

pi install "$KINEX_SOURCE/packages/pi"

For Claude Code and Codex, follow the local marketplace setup. It copies the generated bundles outside the checkout and points their MCP configurations at the built local server with host-managed approvals. Install kinex@kinex-plugins through the selected host. Keep the checkout and its dependencies in place; the installed plugins use its built files. Packed tarball installation is also covered in the shared server README.

Using it

Open your robot workspace and launch a new host session:

cd /path/to/robot-workspace
export KINEX_WORKSPACE="$PWD"
# Launch pi, claude, or codex.

Connecting the plugin leaves robotics inactive. Use these lifecycle commands or skills:

Claude Code and Codex include the corresponding kinex-start, kinex-status and kinex-end skills. In Codex, explicitly ask “Start Kinex”; installing or enabling the plugin does not activate robotics. Starting after an end creates a fresh runtime. Normal host exit also disposes its runtime; borrowed external services remain running.

All three hosts discover current tools, skills, workflows and context through kinex_tools, and execute robotics tools through kinex_call. The shared server also exposes kinex_start, kinex_status, kinex_shutdown and kinex_stop. Pi’s direct lifecycle commands run without a model round trip.

The plugins use host-managed approvals by default. Explicit allow, ask and deny rules in kinex.json still apply; setup questionnaires still require answers. See the plugin permissions documentation for details and how to restore Kinex-managed approvals.

Building your own integration

To bring Kinex to a host that isn’t covered, build on @kinex/core directly. It has no agent, UI or model dependency: you create a runtime for a workspace, register its tools with your host, and supply a callback that asks the user for permission.

import { createKinexRuntime } from "@kinex/core"
import { coreContribution, composeContributions } from "@kinex/core/integration"

const runtime = createKinexRuntime({ directory: process.cwd(), host: { ask: async request => { /* allow, or throw to deny */ } } })
const capabilities = composeContributions([coreContribution(runtime)])

Add @kinex/setup for onboarding and calibration. The core README and setup README cover the API.

CLI reference → Page

Kinex’s command line follows the syntax of the Pi coding agent it’s built on, with robotics options added and a few Pi options left out. For commands you type inside a session, see Slash commands.

Commands

Options

Running

Sessions

Model

Approvals

--auto and --yolo can’t be combined. See Approval modes.

Recording

Other

Input and prompts

• Text piped into kinex is added to the start of the first prompt.
• Put -- before a prompt that starts with -, so it isn’t read as an option.
• Unknown options are rejected. Pi’s RPC mode, @file attachments and its advanced tool and resource flags aren’t supported.

Examples

kinex "Inspect the workspace"                           # start a session with a first prompt
kinex run "/ping arm"                                   # run a slash command and exit
kinex -p --mode json "Describe the scene" > events.jsonl
kinex -c --model provider/model                         # continue, with a different model
kinex -p --record "arm=10 camera=2" "Pick the red mug"  # scripted run, recorded
kinex --export session.jsonl out.html
kinex update --all

Changed in 1.0.0

Install and update

Install, or reinstall, with:

curl -fsSL https://raw.githubusercontent.com/stacx-lab/kinex/main/install.sh | bash

The installer:

• uses your existing Node.js ≥ 22.19 and npm, or asks before installing them;
• checks the downloaded release against its published checksums;
• installs into ~/.local/share/kinex/install and adds a kinex launcher in ~/.local/bin;
• asks before adding ~/.local/bin to your PATH;
• refuses to overwrite a kinex it didn’t install, or to hide another kinex already on your PATH.

kinex update only updates installs made by the installer, and never downgrades. After updating, refresh any drivers and tools you copied into a workspace (for example RoboEnv profiles); updates don’t rewrite those copies.

Build from source

git clone https://github.com/stacx-lab/kinex.git
cd kinex
nvm use
npm ci
npm run build
npm run dev          # run Kinex from source

Before submitting changes, run the full check (needs Python 3 on Linux or macOS):

python3 -m pip install -r packages/core/test/requirements.txt -r packages/setup/test/requirements.txt
npm run check

See AGENTS.md for contribution guidelines.

Slash commands → Page

Type / in a session to search every available command. Device names complete with Tab. Device and recording commands run immediately, without involving the model; they also work from the shell, such as kinex run "/ping arm".

Devices

To restart a device, run /unload and then /load.

Recording

Add --verbose right after the action (for example /record status --verbose) to include empty fields. See Recording.

Setup

These hand a guided task to the agent, so they need a model.

See Onboarding & calibration.

Approvals

See Approval modes.

Agent and sessions

These come from Pi, the coding agent Kinex is built on. See Working with the agent.

Installed extensions and prompt templates can add their own commands. Start a line with ! to run it as a shell command.

Keyboard shortcuts

Troubleshooting → Page

First, check which version you’re running and which workspace and devices Kinex sees:

kinex --version
kinex run "/devices"

Installation and startup

Models and approvals

Devices

Python

Reporting an issue

Open an issue at stacx-lab/kinex/issues and include:

• the output of kinex --version, your OS, and your Node.js and Python versions;
• the command you ran, what you expected, and what happened, including the error;
• whether the device is simulated or real.

Remove credentials and any private images first.