vision-alignment/Readme.md

# gSender Vision Alignment Plugin (FluidNC)

## Overview
The **Vision Alignment Plugin** is a custom extension for gSender (v1.7.0-Edge-1 and later) running on **Linux**. It interfaces with FluidNC-based CNC controllers and uses a live camera feed with a digital cross-hair overlay for highly accurate stock alignment.

By guiding the user to locate three known fiducial markers on their stock, the plugin mathematically determines the exact X/Y workspace origin and the angular deviation of the material. Since FluidNC does not natively support hardware-level coordinate rotation (`G68`), the plugin automatically sets the work zero and **transforms the loaded G-code in software**—applying a rotation matrix to compensate for the skewed stock—completely eliminating the need for perfect physical stock alignment.

## Features
* **Live Camera Feed:** Streams real-time video directly within the gSender interface using any standard USB webcam or microscope camera.
* **Software Cross-Hair Overlay:** Provides a customizable, digital center-target for high-precision visual probing.
* **Three-Point Fiducial Registration:**
  * Captures machine coordinates at three distinct points on the stock.
  * Point 1 establishes the reference origin.
  * Points 2 and 3 establish the true X/Y axes vectors and verify scale/skew.
* **Software-Side Skew Compensation:** Calculates the exact rotational angle (θ) of the stock. The plugin then parses your loaded G-code and applies a rotational matrix (adjusting all `X`, `Y`, `I`, and `J` values) to dynamically match the skew of your stock before sending the job to the controller.
* **Automated Work Zeroing:** Automatically calculates and applies camera-to-spindle offsets and issues `G10 L20` commands to establish the new Work Coordinate System (WCS).

## Prerequisites
* **gSender:** v1.7.0-Edge-1 or later on Linux (plugin system required).
* **Controller:** A CNC machine running **FluidNC** firmware (or any Grbl / grblHAL-based firmware).
* **Hardware:** A rigidly mounted camera (e.g., attached to the spindle or Z-axis carriage) with calibrated X/Y offsets.
* **Development Environment:** Node.js 18+ and yarn (or npm).

## Installation (Linux end users)

gSender loads plugins from:

```text
~/.config/gSender/plugins/
```

(The exact path is also shown under **Tools → Plugins** inside the app.)

1. Obtain a **built** plugin folder (one that already contains `gsender-plugin.json` **and** a `ui/` directory with `index.html`).
2. Copy that folder into `~/.config/gSender/plugins/`, **or** use **Tools → Plugins → Import** and select the folder.
3. Restart gSender. The plugin appears on the **Tools** page.

> Plugins with a `com.sienci.*` id are labelled “Sienci official”; all others appear as “Community”.

## Usage Guide
1. Open the **Tools** page and launch **Vision Alignment**.
2. Select your camera device from the dropdown to initialize the feed.
3. **Calibrate Camera Offset (if not already done):** Enter the physical X and Y distance between your spindle center and camera center.
4. **Point 1 (Origin):** Jog the machine until the cross-hair is perfectly centered on your first fiducial. Click **"Mark Point 1"**.
5. **Point 2 (Axis Reference):** Jog to the second fiducial (defines your primary axis line) and click **"Mark Point 2"**.
6. **Point 3 (Verification):** Jog to the third fiducial and click **"Mark Point 3"**.
7. Click **"Align & Zero Workspace"**.
   * The plugin processes the affine transformation math.
   * It zeros the workspace to the calculated origin point.
   * It applies a rotation matrix to the loaded G-code so the toolpath matches the rotational skew of your stock.

---

## Plugin architecture (gSender 1.7+ on Linux)

gSender plugins are self-contained SPA folders. Each plugin **must** contain:

* `gsender-plugin.json` – manifest
* `ui/` – **built** frontend (produced by Vite; not present until you build)

### Manifest (`gsender-plugin.json`)

| Field              | Required | Description |
|--------------------|----------|-------------|
| `id`               | yes      | Unique reverse-DNS id (e.g. `com.yourname.vision-alignment`). |
| `name`             | yes      | Display name shown on the Tools page. |
| `version`          | yes      | Semver string. |
| `description`      | no       | Short blurb on the plugin card. |
| `engine`           | no       | Compatible gSender version range (e.g. `>=1.7.0`). |
| `ui.entry`         | yes      | Path to the built entry HTML (usually `ui/index.html`). |
| `ui.contributions` | no       | Array of `{ slot, route, label }` – most plugins use `"slot": "tools-page"`. |
| `capabilities`     | no       | Bridge permissions the plugin is allowed to use. |

Example skeleton:

```json
{
  "id": "com.yourname.vision-alignment",
  "name": "Vision Alignment",
  "description": "Camera-based stock alignment and software skew compensation for FluidNC.",
  "version": "0.1.0",
  "engine": ">=1.7.0",
  "ui": {
    "entry": "ui/index.html",
    "contributions": [
      {
        "slot": "tools-page",
        "route": "vision-alignment",
        "label": "Vision Alignment"
      }
    ]
  },
  "capabilities": {
    "requestTypes": ["gcode:load:to:visualizer"],
    "topics": ["workspace"],
    "allowedFunctions": ["gcode", "machine", "useWorkspaceState"]
  }
}
```

The bridge **denies** any call the plugin was not granted.

### Official examples & SDK

Official examples live in the gSender repo under [`plugins/`](https://github.com/Sienci-Labs/gsender/tree/v1.7.0-Edge-1/plugins):

| Folder            | Stack                          | Demonstrates |
|-------------------|--------------------------------|--------------|
| `example-hello/`  | Plain JS + Vite                | Bridge client, subscriptions |
| `react-ts-app/`   | React + TypeScript + Vite      | React hooks |
| `example-viewer/` | Plain JS + Vite                | Embedded G-code preview |
| `basic-cam/`      | React + TypeScript + Vite + Tailwind | Full reference CAM plugin |

SDK package: `@sienci/gsender-plugin-sdk` (source lives in `packages/plugin-sdk`).

| Import | Purpose |
|--------|---------|
| `@sienci/gsender-plugin-sdk` | Framework-agnostic bridge |
| `@sienci/gsender-plugin-sdk/react` | React hooks |
| `@sienci/gsender-plugin-sdk/viewer` | G-code viewer (`@sienci/gviewer`) |
| `@sienci/gsender-plugin-sdk/vite` | Vite plugin for correct externalisation & import map |

Always build plugins with the SDK’s Vite helper:

```ts
// vite.config.ts
import gsenderPlugin from "@sienci/gsender-plugin-sdk/vite";
import react from "@vitejs/plugin-react";
import { defineConfig } from "vite";

export default defineConfig({
  plugins: [react(), gsenderPlugin()],
  base: "./",
  build: { outDir: "ui", emptyOutDir: true },
});
```

Dark mode is applied by gSender as the class `html.dark` on the plugin iframe—use class-based dark styles (or Tailwind `dark:` with class strategy), not `prefers-color-scheme`.

---

## Building plugins (Linux development)

### Important: build the SDK first

The example plugins depend on the local SDK via `file:../../packages/plugin-sdk`. That package’s `package.json` points at `./dist/…`, which **does not exist until you build it**. If you skip this step you will see:

```text
Failed to resolve entry for package "@sienci/gsender-plugin-sdk"
```

### Step-by-step

From the **gSender repo root** (`~/proj/gsender` or similar):

```bash
# 1. Build the plugin SDK (creates packages/plugin-sdk/dist/)
cd packages/plugin-sdk
yarn install
yarn build

# 2. Build the example plugin(s) you need
cd ../../plugins/example-hello
yarn install
yarn build          # creates ui/index.html

# Repeat for other examples if desired:
# cd ../react-ts-app && yarn install && yarn build
# cd ../example-viewer && yarn install && yarn build
# cd ../basic-cam && yarn install && yarn build
```

Or build everything in one go:

```bash
cd ~/proj/gsender

(cd packages/plugin-sdk && yarn install && yarn build)

for d in plugins/example-hello plugins/react-ts-app plugins/example-viewer plugins/basic-cam; do
  echo "=== Building $d ==="
  (cd "$d" && yarn install && yarn build)
done
```

After a successful build you should have:

```text
plugins/example-hello/ui/index.html
```

### “UI entry not found: ui/index.html”

This red message in **Tools → Plugins** means the plugin folder exists but has not been built (no `ui/` directory). Run `yarn build` inside the plugin folder (after the SDK is built) and refresh/restart gSender.

### Watch mode while developing

```bash
cd plugins/example-hello   # or your own plugin
yarn build -- --watch
```

### Create a prod package

```bash
cd gsender
git apply plugins/vision-alignment/allow_camera.patch
yarn build-prod
yarn start-electron
```

gSender (when started with `yarn dev` / `NODE_ENV=development`) watches each plugin’s `ui/` directory and reloads the iframe automatically.

### Starting from an official template

1. Copy the closest example (`example-hello`, `react-ts-app`, `example-viewer`, or `basic-cam`).
2. Edit `gsender-plugin.json` – change `id`, `name`, `description`, `route`, and `label`.
3. Rename the folder if desired (the manifest `id` is what matters).
4. Ensure the SDK is built (`packages/plugin-sdk` → `yarn build`).
5. `yarn install && yarn build` inside the new plugin folder.
6. Place the folder in `~/.config/gSender/plugins/` **or** keep it under the gSender repo’s `plugins/` for local dev.

### Local development inside the gSender source tree

When gSender runs in development (`NODE_ENV=development`, e.g. `yarn dev` or `yarn electron:hot`):

* Plugins are loaded from **both** the repo’s `plugins/` folder **and** `~/.config/gSender/plugins/`.
* Repo plugins take precedence when ids collide.
* Extra search paths: `GSENDER_PLUGINS_DIRS` (colon-separated on Linux).
* Adding a brand-new plugin folder requires a one-time server restart; subsequent edits hot-reload once `ui/` is present.

---

## Development & Testing Workflow

### Leveraging the FluidNC PC Simulation Feature

When developing a CNC plugin that manipulates Work Coordinate Systems and modifies G-code on the fly, testing on a physical machine carries a high risk of accidental crashes. **Because you are developing on Linux, the FluidNC PC Simulation feature is highly recommended.**

FluidNC provides a **PC port/simulation mode** that compiles and runs the firmware as a native executable, completely bypassing the need for an ESP32 microcontroller or physical steppers.

#### Why it is critical for this plugin’s development
1. **Zero Hardware Risk:** Safely test the trigonometry for software-side G-code transformation (linear moves and arcs) and workspace zeroing (`G10 L20`) without risk of plunging a real endmill into the spoilboard.
2. **Rapid Iteration:** Run gSender Edge and the FluidNC simulator side-by-side on the same Linux machine—no workshop computer or hardware cabling required for UI/math work.
3. **Toolpath Verification:** The simulator behaves identically to real FluidNC firmware. Send the dynamically rotated G-code from the plugin and query the controller (`?` or `$#`) to confirm the virtual machine executes the skewed toolpath correctly.

#### Setup Guide for the Simulator
See the [Development](Development.md) guide.

## License
MIT License. See [LICENSE](LICENSE) for more information.