Headless CLI for Browser Tests

Rostyman's Web Intelligence tests can run from the command line — handy for CI, scheduled smoke checks, and reproducing flaky failures with the same engine the app uses.

The CLI is called rostyman-test. It drives the installed Rostyman binary in headless mode, runs the listed tests, and writes a JSON summary + a JUnit XML report that any CI (GitHub Actions, GitLab, CircleCI, Jenkins) can ingest directly.

Install

rostyman-test ships inside the desktop install. After running Rostyman once you'll have a dist-cli/rostyman-test.js next to the app. The simplest way to expose it on $PATH:

# From the Rostyman source checkout
yarn build:cli
chmod +x dist-cli/rostyman-test.js
ln -s "$(pwd)/dist-cli/rostyman-test.js" /usr/local/bin/rostyman-test

Set ROSTYMAN_BIN=/path/to/Rostyman.exe (or the macOS .app/Contents/MacOS/Rostyman / Linux /opt/Rostyman/rostyman) if the CLI can't auto-find your install.

Usage

rostyman-test run <test-id-1>,<test-id-2> [options]

Flag	Default	What it does
--env=<env-id>	none	Resolve {{vars}} against this environment before running
--out=<dir>	./rostyman-results	Where reports are written
--format=<list>	json,junit	Comma-separated subset of json, junit
--timeout=<ms>	600000	Hard cap per run — fails with exit code 2 if exceeded
--electron=<path>	auto-detect	Explicit binary path (overrides ROSTYMAN_BIN)

Exit codes follow CI convention:

• 0 — every test passed
• 1 — at least one test failed (steps had an assertion or selector miss)
• 2 — driver error (binary not found, timeout, renderer crash)

Finding test IDs

Open the app, go to Browser → Tests, right-click any test, and copy its ID. Or run an export of the workspace and grep the IDs out of the JSON.

A future release will add rostyman-test list to dump IDs from the active workspace without opening the app.

What the reports look like

results-latest.json

{
  "version": 1,
  "generator": "rostyman",
  "startedAt": 1715800000000,
  "endedAt":   1715800040000,
  "durationMs": 40000,
  "counts":   { "total": 3, "passed": 2, "failed": 1, "stopped": 0 },
  "results":  [ /* one entry per test with full stepResults */ ]
}

The results[].stepResults array carries the same { stepId, status, actual, error, durationMs, screenshot } you see in the in-app Test Run History panel — including the pre-action JPEG thumbnails captured by Pillar 1 when assertions fail.

junit-latest.xml

Standard JUnit <testsuites> schema — one <testsuite> per Rostyman test, one <testcase> per step, <failure> elements where they failed. GitHub Actions parses this directly and produces an annotated check.

CI integration

GitHub Actions

# .github/workflows/web-tests.yml
name: Web Intelligence tests
on:
  push:    { branches: [main] }
  pull_request:
  schedule: [{ cron: "0 6 * * *" }]   # daily 06:00 UTC

jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Rostyman
        run: |
          curl -sL https://rostyman.com/api/download/linux -o rostyman.AppImage
          chmod +x rostyman.AppImage
          echo "ROSTYMAN_BIN=$PWD/rostyman.AppImage" >> $GITHUB_ENV

      - name: Run the smoke tests
        run: |
          rostyman-test run ${{ vars.SMOKE_TEST_IDS }} \
            --env=${{ vars.PROD_ENV_ID }} \
            --out=./test-out

      - name: Upload JSON + JUnit artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: rostyman-results
          path: test-out/

      - name: Publish JUnit summary
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name:   Web Intelligence
          path:   test-out/junit-latest.xml
          reporter: jest-junit

If you'd like a reusable composite action, drop the snippet above into .github/actions/rostyman-test/action.yml with inputs: for the test IDs, env, and output dir — the implementation is identical.

GitLab CI

web-tests:
  image: rostyman/ubuntu:latest
  script:
    - rostyman-test run "$SMOKE_TEST_IDS" --env="$ENV_ID" --out=out
  artifacts:
    when: always
    paths: [out/]
    reports:
      junit: out/junit-latest.xml

Headed runs while debugging

The CLI launches the bundled Chromium with show:false — you won't see a window. To watch the test play in a visible window while reproducing a failure, run from the source tree:

yarn transpile:electron
ROSTYMAN_CLI_HEADED=1 yarn dev   # then click Run inside the app

In other words: the CLI itself is headless by design, but Web Intelligence tests are first-class inside the app — use the app for debugging, use the CLI for CI.

Limitations & known gaps

• rostyman-test list is not implemented yet — copy IDs from the app for now.
• The CLI requires a workspace to already exist; it doesn't create one. Run Rostyman once before plumbing into CI.
• Visual-regression baselines are stored per-workspace under userData/ — for CI you'll typically commit baselines to git and ROSTYMAN_USERDATA_DIR=$CI_WORKSPACE/.rostyman before invoking the CLI. (That env var is honored by Electron's app.setPath('userData', …) shim.)
• Cross-run perf history (Pillar 7-c) is not surfaced in JUnit yet — JSON has full step durations; JUnit only reports pass/fail.

See Browser Automation for the broader Web Intelligence overview, the seven pillars, and the in-app workflow.