AI Business Maturity Model
Certifications
Find a CoachFind a SpeakerSign In

Testing

Our multi-layer testing approach: fast type checks, linting, E2E browser testing with Playwright, and API integration tests. Catches issues at every level before they reach production.

Testing Philosophy

Testing happens in layers, from fastest to slowest. Every layer catches different types of bugs. The key insight: most issues are caught by the fast layers (type checking and linting), which take seconds. The slow layers (E2E and API tests) catch the subtle integration bugs that only appear at runtime.

Layer 1: Quick Check

~15 seconds

Catches: TypeScript errors, type mismatches, missing imports, invalid syntax

Command: npm run quick-check

When: After every code change

Layer 2: Full Check

~60 seconds

Catches: Linting violations, formatting inconsistencies, security anti-patterns, code quality issues

Command: npm run check

When: Before commits

Layer 3: Build Check

~5 minutes

Catches: SSR rendering errors, prerender failures, missing environment variables, broken imports at build time

Command: npm run build

When: Before deployment (part of BobTheBuilder)

Layer 4: E2E & API Tests

~5-10 minutes

Catches: Broken user journeys, API contract violations, wrong response shapes, authentication failures, cross-page navigation issues

Command: Playwright + TypeScript API runner (tsx)

When: After deploying to dev, before production

Layer 1: Quick Check (Required After Every Change)

The quick check runs TypeScript compilation without emitting files. It catches type errors in ~15 seconds vs the 5-minute full build. This is the single most important testing command — it should be run after every meaningful code change.

Quick Check Setup

json
// In package.json scripts:
{
  "quick-check": "tsc --noEmit",
  "type-check": "tsc --noEmit"
}

For non-TypeScript projects, the equivalent is whatever static analysis runs fastest:cargo check for Rust, mypy for Python, go vet for Go.

Layer 2: Full Check (Lint + Format + Types)

The full check combines type checking, linting, and formatting into one command. It includes an auto-fix variant that resolves most issues automatically.

Full Check Setup

json
// In package.json scripts:
{
  "check": "npm run type-check && npm run lint && npm run format:check",
  "check:fix": "npm run lint:fix && npm run format && npm run type-check",
  "lint": "eslint . --max-warnings 0",
  "lint:fix": "eslint . --fix",
  "format": "prettier --write .",
  "format:check": "prettier --check ."
}

Our ESLint configuration includes these key plugins:

  • eslint-plugin-security: Catches common security anti-patterns (eval, non-literal require, etc.)
  • eslint-plugin-sonarjs: Code quality rules — cognitive complexity, duplicate strings, dead stores
  • typescript-eslint: TypeScript-specific rules — no-any, consistent type imports, explicit return types
  • eslint-config-prettier: Disables ESLint rules that conflict with Prettier formatting

Layer 3: E2E Browser Tests (Playwright)

Playwright tests run against the deployed dev environment. They simulate real user journeys through the application — logging in, navigating portals, filling forms, verifying content.

e2e/playwright.config.ts — Key Configuration

typescript
import { defineConfig, devices } from '@playwright/test';

const BASE_URL = process.env['TEST_BASE_URL'] || 'https://dev.aibmm.ai';

export default defineConfig({
  testDir: './tests',
  fullyParallel: true,
  forbidOnly: !!process.env['CI'],
  retries: process.env['CI'] ? 2 : 0,
  workers: process.env['CI'] ? 1 : 4,

  reporter: [
    ['list'],
    ['html', { outputFolder: 'playwright-report', open: 'never' }],
    ['json', { outputFile: 'test-results/results.json' }],
  ],

  use: {
    baseURL: BASE_URL,
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },

  projects: [
    // Smoke tests — quick health checks
    {
      name: 'smoke',
      testMatch: '**/smoke/**/*.spec.ts',
      use: { ...devices['Desktop Chrome'] },
    },
    // Full E2E tests by portal
    {
      name: 'coach',
      testMatch: '**/coach/**/*.spec.ts',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'ascent',
      testMatch: '**/ascent/**/*.spec.ts',
      use: { ...devices['Desktop Chrome'] },
    },
  ],
});

Test organization follows the portal structure:

E2E Test File Structure

text
e2e/
├── playwright.config.ts
├── tests/
│   ├── smoke/              # Quick health checks (~30 seconds)
│   │   └── health.spec.ts  # Page loads, API responds, auth works
│   ├── coach/              # Coach portal user journeys
│   │   └── workflows.spec.ts
│   ├── ascent/             # Ascent portal user journeys  
│   │   └── assessment.spec.ts
│   └── helpers/            # Shared test utilities
│       └── auth.ts         # Login helpers, test user management
├── playwright-report/      # HTML test reports
└── test-results/           # JSON results for CI

Layer 4: API Integration Tests (TypeScript Runner)

API tests use a TypeScript runner (scripts/tests/index.ts) executed with tsx. This is a step up from bash/curl scripts: you get type-safe test definitions, assertion helpers, auth fixtures, structured JSON reporting, and test filtering — all without pulling in a heavy framework like Jest or Vitest.

The key insight: bash + curl works fine for under 10 endpoints. Once you have 20+ routes, you need type-safe fixtures so tests don't silently pass on wrong shapes, assertion helpers so failures tell you what was wrong, and structured output so CI can parse results. A lightweight custom runner with tsx gives you all of this without framework overhead.

scripts/tests/lib/types.ts — Core Interfaces

typescript
export interface TestCase {
  name: string;
  endpoint: string;
  method: 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE';
  expectedStatus: number;
  acceptableStatuses?: number[];   // for idempotent tests
  body?: Record<string, unknown>;
  headers?: Record<string, string>;
  requiresAuth?: boolean;
  validate?: (response: Response, body: unknown) => ValidationResult;
  suggestedFix?: string;
}

export interface TestSuite {
  name: string;
  tests: TestCase[];
}

export interface ValidationResult {
  pass: boolean;
  message?: string;
}

export interface TestResult {
  suite: string;
  test: string;
  passed: boolean;
  status?: number;
  error?: string;
  durationMs: number;
}

scripts/tests/lib/runner.ts — Lightweight Runner

typescript
import type { TestCase, TestResult, TestSuite } from './types';
import { getAuthCookies } from './auth';

export class TestRunner {
  private results: TestResult[] = [];

  constructor(
    private baseUrl: string,
    private suites: TestSuite[],
  ) {}

  async run(): Promise<TestResult[]> {
    const authCookies = await getAuthCookies(this.baseUrl);

    for (const suite of this.suites) {
      for (const test of suite.tests) {
        const result = await this.runTest(suite.name, test, authCookies);
        this.results.push(result);
      }
    }
    return this.results;
  }

  private async runTest(
    suiteName: string,
    test: TestCase,
    authCookies: string,
  ): Promise<TestResult> {
    const start = Date.now();
    try {
      const headers: Record<string, string> = {
        'Content-Type': 'application/json',
        ...(test.requiresAuth ? { Cookie: authCookies } : {}),
        ...test.headers,
      };

      const response = await fetch(`${this.baseUrl}${test.endpoint}`, {
        method: test.method,
        headers,
        body: test.body ? JSON.stringify(test.body) : undefined,
      });

      const acceptable = test.acceptableStatuses ?? [test.expectedStatus];
      const statusPass = acceptable.includes(response.status);

      let validationPass = true;
      let validationMessage: string | undefined;

      if (statusPass && test.validate) {
        const body = await response.json().catch(() => null);
        const result = test.validate(response, body);
        validationPass = result.pass;
        validationMessage = result.message;
      }

      return {
        suite: suiteName,
        test: test.name,
        passed: statusPass && validationPass,
        status: response.status,
        error: !statusPass
          ? `Expected ${test.expectedStatus}, got ${response.status}`
          : validationMessage,
        durationMs: Date.now() - start,
      };
    } catch (err) {
      return {
        suite: suiteName,
        test: test.name,
        passed: false,
        error: err instanceof Error ? err.message : String(err),
        durationMs: Date.now() - start,
      };
    }
  }
}

The runner supports three test scopes via CLI flags:

  • Full: All registered suites (~100+ tests across all routes)
  • Key Systems: Core CRUD operations only (fast, ~30 tests)
  • User Story: Tests that map to specific US-XXX user stories

How to Build This (AI Guide)

Step 1: Quick Check

Add a fast static analysis command that runs in under 30 seconds:

For TypeScript projects

bash
# Add to package.json scripts
"quick-check": "tsc --noEmit"

# For Python: mypy .
# For Rust: cargo check
# For Go: go vet ./...

Step 2: Linting + Formatting

Install ESLint + Prettier (TypeScript)

bash
npm install -D eslint prettier typescript-eslint \
  eslint-plugin-security eslint-plugin-sonarjs \
  eslint-config-prettier eslint-plugin-prettier

# Add scripts to package.json:
"lint": "eslint . --max-warnings 0",
"lint:fix": "eslint . --fix",
"format": "prettier --write .",
"format:check": "prettier --check .",
"check": "npm run type-check && npm run lint && npm run format:check",
"check:fix": "npm run lint:fix && npm run format && npm run type-check"

Step 3: E2E Tests with Playwright

Playwright Setup

bash
# Install Playwright
npm install -D @playwright/test
npx playwright install chromium

# Create config: e2e/playwright.config.ts
# Create test directory: e2e/tests/
# Create first test: e2e/tests/smoke/health.spec.ts

# Add to package.json:
"test:e2e": "npx playwright test --config=e2e/playwright.config.ts",
"test:smoke": "npx playwright test --config=e2e/playwright.config.ts --project=smoke"

Example smoke test: e2e/tests/smoke/health.spec.ts

typescript
import { test, expect } from '@playwright/test';

test('homepage loads', async ({ page }) => {
  await page.goto('/');
  await expect(page).toHaveTitle(/Your App Name/);
});

test('API health check', async ({ request }) => {
  const response = await request.get('/api/health');
  expect(response.ok()).toBeTruthy();
  const body = await response.json();
  expect(body.status).toBe('ok');
});

Step 4: TypeScript API Test Runner

Start with a bash/curl script if you have fewer than 10 endpoints. Once you scale past that, migrate to a TypeScript runner. Here's the minimal setup:

Install tsx (TypeScript runner)

bash
npm install -D tsx

# Add to package.json scripts:
"test:api": "tsx scripts/tests/index.ts",
"test:api:key": "tsx scripts/tests/index.ts --scope=key-systems",
"test:api:story": "tsx scripts/tests/index.ts --scope=user-story"

scripts/tests/index.ts — Entry Point

typescript
import { TestRunner } from './lib/runner';
import { Reporter } from './lib/reporter';
import { healthSuite } from './suites/health';
import { authSuite } from './suites/auth';
// import more suites...

const BASE_URL = process.env['TEST_BASE_URL'] ?? 'https://dev.yourapp.com';
const scope = process.argv.find((a) => a.startsWith('--scope='))?.split('=')[1] ?? 'full';

const ALL_SUITES = [healthSuite, authSuite /*, ...more */];

const suites =
  scope === 'key-systems'
    ? ALL_SUITES.filter((s) => s.tags?.includes('key-system'))
    : ALL_SUITES;

const runner = new TestRunner(BASE_URL, suites);
const results = await runner.run();

const reporter = new Reporter(results);
reporter.printConsole();
reporter.saveJson('scripts/tests/results/');

process.exit(results.some((r) => !r.passed) ? 1 : 0);

scripts/tests/suites/health.ts — Example Suite

typescript
import type { TestSuite } from '../lib/types';

export const healthSuite: TestSuite = {
  name: 'Health',
  tags: ['key-system'],
  tests: [
    {
      name: 'Health endpoint returns 200',
      endpoint: '/api/health',
      method: 'GET',
      expectedStatus: 200,
      validate: (_, body) => {
        const b = body as Record<string, unknown>;
        return {
          pass: b['status'] === 'healthy',
          message: `Expected status "healthy", got "${b['status']}"`,
        };
      },
    },
    {
      name: 'Health endpoint includes build commit',
      endpoint: '/api/health',
      method: 'GET',
      expectedStatus: 200,
      validate: (_, body) => {
        const b = body as Record<string, unknown>;
        const build = b['build'] as Record<string, unknown> | undefined;
        return {
          pass: typeof build?.['commit'] === 'string' && build['commit'] !== 'unknown',
          message: 'Build commit should be a real git hash, not "unknown"',
        };
      },
    },
  ],
};

Step 5: Wire Into BobTheBuilder

Add the test commands as menu options in your BobTheBuilder script (see the Bob the Builder resource). The pre-deployment check should run layers 1-3 automatically. E2E and API tests are separate menu options because they require a running dev server.

← Back to all resources