-
Notifications
You must be signed in to change notification settings - Fork 0
Testing
This guide covers testing strategies, tools, and practices for TMI development including unit tests, integration tests, API tests, and end-to-end tests.
- Testing Philosophy
- Unit Testing
- Integration Testing
- API Testing
- End-to-End Testing
- WebSocket Testing
- CATS Security Fuzzing
- Coverage Reporting
- TMI-UX Testing Utilities
TMI follows a comprehensive testing approach:
- Unit Tests - Fast tests with no external dependencies
- Integration Tests - Tests with real database and services
- API Tests - Complete API workflow testing with Postman/Newman
- E2E Tests - Full user journey testing with Cypress
/\
/E2E\ Few, slow, expensive
/------\
/ API \ Some, medium speed
/----------\
/Integration\ More, medium speed
/--------------\
/ Unit Tests \ Many, fast, cheap
/------------------\
- Test business logic thoroughly - Unit test all business rules
- Test integration points - Verify components work together
- Test user workflows - Ensure complete features work end-to-end
- Automate everything - All tests should be automated
- Fast feedback - Unit tests run in seconds
- Realistic testing - Integration tests use real databases
TMI server uses Go's built-in testing framework.
# Run all unit tests
make test-unit
# Run specific test
go test -v ./api -run TestCreateThreatModel
# Run with coverage
make test-coverage-unitTest File Naming: *_test.go
Example Test:
// api/threat_model_test.go
package api
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestCreateThreatModel(t *testing.T) {
// Arrange
tm := ThreatModel{
Name: "Test Threat Model",
Description: stringPtr("Test description"),
}
// Act
result, err := createThreatModelLogic(tm)
// Assert
assert.NoError(t, err)
assert.NotEmpty(t, result.ID)
assert.Equal(t, tm.Name, result.Name)
}Table-Driven Tests:
func TestAuthorizationRoles(t *testing.T) {
tests := []struct {
name string
role string
canRead bool
canWrite bool
canDelete bool
}{
{"owner", "owner", true, true, true},
{"writer", "writer", true, true, false},
{"reader", "reader", true, false, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.canRead, canRead(tt.role))
assert.Equal(t, tt.canWrite, canWrite(tt.role))
assert.Equal(t, tt.canDelete, canDelete(tt.role))
})
}
}Mocking External Dependencies:
type MockDatabase struct {
mock.Mock
}
func (m *MockDatabase) GetThreatModel(id string) (*ThreatModel, error) {
args := m.Called(id)
return args.Get(0).(*ThreatModel), args.Error(1)
}
func TestWithMock(t *testing.T) {
// Create mock
mockDB := new(MockDatabase)
mockDB.On("GetThreatModel", "123").Return(&ThreatModel{
ID: "123",
Name: "Test",
}, nil)
// Use mock in test
tm, err := mockDB.GetThreatModel("123")
assert.NoError(t, err)
assert.Equal(t, "123", tm.ID)
mockDB.AssertExpectations(t)
}TMI-UX uses Vitest for unit testing.
# Run all tests
pnpm run test
# Run in watch mode
pnpm run test:watch
# Run with UI
pnpm run test:ui
# Run specific test
pnpm run test -- src/app/pages/tm/tm.component.spec.ts
# Coverage report
pnpm run test:coverageTest File Naming: *.spec.ts
Example Component Test (using Vitest):
// src/app/pages/tm/tm.component.spec.ts
import '@angular/compiler';
import { vi, expect, beforeEach, describe, it } from 'vitest';
import { ComponentFixture, TestBed } from '@angular/core/testing';
import { TmComponent } from './tm.component';
import { ApiService } from '../../core/services/api.service';
import { of } from 'rxjs';
describe('TmComponent', () => {
let component: TmComponent;
let fixture: ComponentFixture<TmComponent>;
let mockApiService: {
getThreatModels: ReturnType<typeof vi.fn>;
};
beforeEach(async () => {
vi.clearAllMocks();
// Create mock using Vitest
mockApiService = {
getThreatModels: vi.fn()
};
await TestBed.configureTestingModule({
imports: [TmComponent],
providers: [
{ provide: ApiService, useValue: mockApiService }
]
}).compileComponents();
fixture = TestBed.createComponent(TmComponent);
component = fixture.componentInstance;
});
it('should create', () => {
expect(component).toBeTruthy();
});
it('should load threat models on init', () => {
// Arrange
const mockThreatModels = [
{ id: '1', name: 'TM 1' },
{ id: '2', name: 'TM 2' }
];
mockApiService.getThreatModels.mockReturnValue(of(mockThreatModels));
// Act
component.ngOnInit();
// Assert
expect(mockApiService.getThreatModels).toHaveBeenCalled();
expect(component.threatModels).toEqual(mockThreatModels);
});
});Service Test:
// src/app/core/services/api.service.spec.ts
import { TestBed } from '@angular/core/testing';
import { HttpClientTestingModule, HttpTestingController } from '@angular/common/http/testing';
import { ApiService } from './api.service';
describe('ApiService', () => {
let service: ApiService;
let httpMock: HttpTestingController;
beforeEach(() => {
TestBed.configureTestingModule({
imports: [HttpClientTestingModule],
providers: [ApiService]
});
service = TestBed.inject(ApiService);
httpMock = TestBed.inject(HttpTestingController);
});
afterEach(() => {
httpMock.verify();
});
it('should fetch threat models', () => {
const mockThreatModels = [{ id: '1', name: 'TM 1' }];
service.getThreatModels().subscribe(tms => {
expect(tms).toEqual(mockThreatModels);
});
const req = httpMock.expectOne('/api/threat_models');
expect(req.request.method).toBe('GET');
req.flush(mockThreatModels);
});
});For DFD services with complex interdependent behavior, TMI-UX uses an integration testing approach with Vitest instead of mocks that would require duplicating business logic.
Key Principles:
- Use Real Service Instances: Replace mocks with actual service instances for business logic services
-
Mock Cross-Cutting Concerns: Keep
LoggerServicemocked since it's a cross-cutting concern - Test Real Integration: Verify actual service integration and behavior
- Eliminate Logic Duplication: No need to replicate service logic in mocks
Example - Testing DFD Edge Service:
```typescript // src/app/pages/dfd/infrastructure/services/infra-edge.service.spec.ts import { Graph } from '@antv/x6'; import { InfraEdgeService } from './infra-edge.service'; import { InfraEdgeQueryService } from './infra-edge-query.service'; import { InfraPortStateService } from './infra-port-state.service'; import { InfraX6CoreOperationsService } from './infra-x6-core-operations.service'; import { createTypedMockLoggerService, type MockLoggerService } from '../../../../../testing/mocks';
describe('InfraEdgeService - X6 Integration Tests', () => { let service: InfraEdgeService; let queryService: InfraEdgeQueryService; let portStateManager: InfraPortStateService; let x6CoreOps: InfraX6CoreOperationsService; let mockLogger: MockLoggerService;
beforeEach(() => { // Create mock for LoggerService only (cross-cutting concern) mockLogger = createTypedMockLoggerService();
// Create REAL service instances for integration testing
queryService = new InfraEdgeQueryService(mockLogger as unknown as LoggerService);
portStateManager = new InfraPortStateService(
queryService,
mockLogger as unknown as LoggerService,
);
x6CoreOps = new InfraX6CoreOperationsService(mockLogger as unknown as LoggerService);
// Create service under test with real dependencies
service = new InfraEdgeService(
mockLogger as unknown as LoggerService,
portStateManager,
x6CoreOps,
);
}); }); ```
Service Dependency Chain:
``` InfraEdgeService +-- InfraPortStateService (REAL) | +-- InfraEdgeQueryService (REAL) | +-- LoggerService (MOCK) +-- InfraX6CoreOperationsService (REAL) | +-- LoggerService (MOCK) +-- LoggerService (MOCK) ```
When to Use Integration Testing:
- Services have complex interdependent behavior
- Mocking would require duplicating significant business logic
- You need to verify actual integration between services
- The services are part of the same bounded context (e.g., DFD infrastructure layer)
When to Use Unit Testing with Mocks:
- Testing isolated units of functionality
- Dependencies are simple or cross-cutting concerns (like logging)
- You want to test error conditions that are hard to reproduce with real services
- Performance is a concern for test execution speed
Testing Strategy Decision Tree:
``` Is this a high-level orchestrator/coordinator? +- YES: Mock all dependencies (test orchestration logic only) | Example: AppDfdOrchestrator with 12 mocked dependencies +- NO: Use integration testing approach +- Create real service instances +- Only mock cross-cutting concerns (LoggerService) +- Test actual service integration Example: InfraEdgeService with real InfraPortStateService ```
Angular + Vitest Setup:
The test setup in `src/test-setup.ts` handles Angular JIT compilation globally via `src/testing/compiler-setup.ts`. Zone.js is loaded globally via `src/testing/zone-setup.ts`. TestBed state is NOT serializable across Vitest's forked processes, so each test file must initialize TestBed in its own `beforeAll()` hook if needed.
For complete implementation examples, see:
- `src/app/pages/dfd/infrastructure/services/infra-edge.service.spec.ts`
- `src/app/pages/dfd/infrastructure/services/infra-port-state.service.spec.ts`
Integration tests verify that components work correctly with real databases and services.
# Run all integration tests (automatic setup and cleanup)
make test-integration
# This automatically:
# 1. Starts PostgreSQL container
# 2. Starts Redis container
# 3. Runs migrations
# 4. Starts server
# 5. Runs tests
# 6. Cleans up everythingIntegration tests use dedicated ports to avoid conflicts:
- PostgreSQL: Port 5434 (vs 5432 for development)
- Redis: Port 6381 (vs 6379 for development)
- Server: Port 8080
Test File Naming: *_integration_test.go
Example:
// api/threat_model_integration_test.go
package api
import (
"testing"
"net/http"
"net/http/httptest"
"github.com/stretchr/testify/assert"
)
func TestDatabaseThreatModelIntegration(t *testing.T) {
suite := SetupIntegrationTest(t)
defer suite.TeardownIntegrationTest(t)
// Create threat model
threatModelData := map[string]interface{}{
"name": "Integration Test TM",
"description": "Test with real database",
}
req := suite.makeAuthenticatedRequest("POST", "/threat_models", threatModelData)
w := suite.executeRequest(req)
assert.Equal(t, http.StatusCreated, w.Code)
// Verify in database
var tm ThreatModel
err := suite.db.First(&tm).Error
assert.NoError(t, err)
assert.Equal(t, "Integration Test TM", tm.Name)
}Predictable Test Users (using login hints):
func createTestUser(hint string) (*User, string) {
// Create specific test user 'alice@test.tmi' instead of random
resp, _ := http.Get(
"http://localhost:8080/oauth2/authorize?idp=test&login_hint=" + hint
)
// Parse token from response
token := parseTokenFromResponse(resp)
return &User{Email: hint + "@test.tmi"}, token
}
func TestMultiUserScenario(t *testing.T) {
alice, aliceToken := createTestUser("alice")
bob, bobToken := createTestUser("bob")
// Test with both users
}Complete Entity Lifecycle:
func TestThreatModelLifecycle(t *testing.T) {
suite := SetupIntegrationTest(t)
defer suite.TeardownIntegrationTest(t)
// 1. Create
createReq := suite.makeAuthenticatedRequest("POST", "/threat_models", data)
createW := suite.executeRequest(createReq)
assert.Equal(t, http.StatusCreated, createW.Code)
tmID := parseID(createW.Body)
// 2. Read
getReq := suite.makeAuthenticatedRequest("GET", "/threat_models/" + tmID, nil)
getW := suite.executeRequest(getReq)
assert.Equal(t, http.StatusOK, getW.Code)
// 3. Update
updateReq := suite.makeAuthenticatedRequest("PUT", "/threat_models/" + tmID, updatedData)
updateW := suite.executeRequest(updateReq)
assert.Equal(t, http.StatusOK, updateW.Code)
// 4. Delete
deleteReq := suite.makeAuthenticatedRequest("DELETE", "/threat_models/" + tmID, nil)
deleteW := suite.executeRequest(deleteReq)
assert.Equal(t, http.StatusNoContent, deleteW.Code)
// 5. Verify deletion
verifyReq := suite.makeAuthenticatedRequest("GET", "/threat_models/" + tmID, nil)
verifyW := suite.executeRequest(verifyReq)
assert.Equal(t, http.StatusNotFound, verifyW.Code)
}Authorization Testing:
func TestAuthorizationMatrix(t *testing.T) {
suite := SetupIntegrationTest(t)
defer suite.TeardownIntegrationTest(t)
alice, aliceToken := createTestUser("alice")
bob, bobToken := createTestUser("bob")
// Alice creates threat model
tm := createThreatModel(aliceToken)
// Test reader permissions
addAuthorization(tm.ID, bob.Email, "reader", aliceToken)
// Bob can read
getReq := makeRequestWithToken("GET", "/threat_models/" + tm.ID, nil, bobToken)
assert.Equal(t, http.StatusOK, suite.executeRequest(getReq).Code)
// Bob cannot write
updateReq := makeRequestWithToken("PUT", "/threat_models/" + tm.ID, data, bobToken)
assert.Equal(t, http.StatusForbidden, suite.executeRequest(updateReq).Code)
// Bob cannot delete
deleteReq := makeRequestWithToken("DELETE", "/threat_models/" + tm.ID, nil, bobToken)
assert.Equal(t, http.StatusForbidden, suite.executeRequest(deleteReq).Code)
}TMI uses an OpenAPI-driven integration test framework located in test/integration/. The framework provides:
- OAuth Authentication: Automated OAuth flows via the OAuth callback stub
- Request Building: Type-safe request construction with fixtures
- Response Validation: OpenAPI schema validation for all responses
- Assertion Helpers: Specialized assertions for API responses
test/integration/
├── framework/
│ ├── client.go # HTTP client with authentication
│ ├── oauth.go # OAuth authentication utilities
│ ├── fixtures.go # Test data fixtures
│ ├── assertions.go # Test assertion helpers
│ └── database.go # Database utilities
├── spec/
│ ├── schema_loader.go # OpenAPI schema loading
│ └── openapi_validator.go # Response validation
└── workflows/
├── example_test.go # Framework demonstration
├── oauth_flow_test.go # OAuth tests
├── threat_model_crud_test.go # Threat model CRUD
├── diagram_crud_test.go # Diagram CRUD
├── user_operations_test.go # User operations
├── user_preferences_test.go # User preferences
└── admin_promotion_test.go # Admin promotion
package workflows
import (
"os"
"testing"
"github.com/ericfitz/tmi/test/integration/framework"
)
func TestResourceCRUD(t *testing.T) {
// Skip if not running integration tests
if os.Getenv("INTEGRATION_TESTS") != "true" {
t.Skip("Skipping integration test")
}
serverURL := os.Getenv("TMI_SERVER_URL")
if serverURL == "" {
serverURL = "http://localhost:8080"
}
// Ensure OAuth stub is running
if err := framework.EnsureOAuthStubRunning(); err != nil {
t.Fatalf("OAuth stub not running: %v", err)
}
// Authenticate
userID := framework.UniqueUserID()
tokens, err := framework.AuthenticateUser(userID)
framework.AssertNoError(t, err, "Authentication failed")
// Create client
client, err := framework.NewClient(serverURL, tokens)
framework.AssertNoError(t, err, "Client creation failed")
// Use subtests for each operation
t.Run("Create", func(t *testing.T) {
fixture := framework.NewThreatModelFixture().
WithName("Test Model")
resp, err := client.Do(framework.Request{
Method: "POST",
Path: "/threat_models",
Body: fixture,
})
framework.AssertNoError(t, err, "Request failed")
framework.AssertStatusCreated(t, resp)
})
}TMI aims for 100% API coverage (178 operations across 92 paths) organized in three tiers:
| Tier | Purpose | Run Frequency | Time Budget |
|---|---|---|---|
| Tier 1 | Core workflows (OAuth, CRUD) | Every commit | < 2 min |
| Tier 2 | Feature tests (metadata, webhooks, addons) | Nightly | < 10 min |
| Tier 3 | Edge cases & admin operations | Weekly | < 15 min |
For the complete integration test plan including implementation roadmap and coverage tracking, see the source documentation at docs/migrated/developer/testing/integration-test-plan.md.
TMI uses Postman collections and Newman for comprehensive API testing.
# Run all API tests
make test-api
# Or run manually
cd postman
./run-tests.shLocated in /postman directory:
-
comprehensive-test-collection.json- Main test suite -
unauthorized-tests-collection.json- 401 error testing -
threat-crud-tests-collection.json- Threat CRUD operations -
metadata-tests-collection.json- Metadata operations -
permission-matrix-tests-collection.json- Authorization testing -
bulk-operations-tests-collection.json- Batch operations
API tests cover:
- 70+ endpoints
- 91 workflow methods
- All HTTP status codes (200, 201, 204, 400, 401, 403, 404, 409, 422, 500)
- Authentication and authorization
- CRUD operations for all entities
- Metadata operations
- Batch operations
- Error scenarios
A comprehensive coverage analysis tracks test coverage across all 41 threat model paths (105 operations):
| Metric | Value |
|---|---|
| Total Threat Model Paths | 41 |
| Total Operations | 105 |
| Operations with Success Tests | ~85 (81%) |
| Operations with 401 Tests | ~25 (24%) |
| Operations with 403 Tests | ~15 (14%) |
| Operations with 404 Tests | ~35 (33%) |
| Operations with 400 Tests | ~30 (29%) |
Key Gap Areas:
- Sub-resource 401 tests (threats, diagrams, metadata endpoints)
- Authorization 403 tests for writer/reader role scenarios
- Rate limit (429) and server error (500) tests
Collection Files (test/postman/):
-
comprehensive-test-collection.json- Full workflow tests -
unauthorized-tests-collection.json- 401 authentication tests -
permission-matrix-tests-collection.json- Multi-user authorization -
threat-crud-tests-collection.json- Threat entity CRUD -
collaboration-tests-collection.json- WebSocket collaboration -
advanced-error-scenarios-collection.json- 409, 422 edge cases
For complete coverage matrix, gap analysis, and implementation recommendations, see docs/migrated/developer/testing/postman-threat-model-coverage.md.
Basic Test:
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
pm.test("Response has threat models", function () {
const response = pm.response.json();
pm.expect(response).to.be.an('array');
pm.expect(response.length).to.be.above(0);
});Advanced Test with Setup:
// Pre-request Script
const data = {
name: "Test Threat Model",
description: "Created by test"
};
pm.collectionVariables.set("threat_model_data", JSON.stringify(data));
// Test Script
pm.test("Threat model created", function () {
pm.response.to.have.status(201);
const response = pm.response.json();
pm.expect(response).to.have.property('id');
pm.expect(response.name).to.equal("Test Threat Model");
// Save ID for subsequent tests
pm.collectionVariables.set("threat_model_id", response.id);
});TMI-UX uses Cypress for E2E testing.
# Run all E2E tests
pnpm run test:e2e
# Open Cypress GUI
pnpm run test:e2e:open
# Run specific spec
pnpm run test:e2e -- --spec="cypress/e2e/login.cy.ts"Test File Naming: *.cy.ts
Example Login Test:
// cypress/e2e/login.cy.ts
describe('Login Flow', () => {
beforeEach(() => {
cy.visit('/');
});
it('should display login page', () => {
cy.contains('Sign In').should('be.visible');
});
it('should login with test provider', () => {
cy.contains('Test Login').click();
cy.url().should('include', '/dashboard');
cy.contains('Threat Models').should('be.visible');
});
});Example Diagram Test:
// cypress/e2e/diagram.cy.ts
describe('Diagram Editor', () => {
beforeEach(() => {
cy.login(); // Custom command
cy.visit('/threat-models/123/diagrams/456');
});
it('should add process to diagram', () => {
// Open shape palette
cy.get('[data-cy=shape-palette]').click();
// Select process shape
cy.get('[data-cy=shape-process]').click();
// Click on canvas to add
cy.get('[data-cy=diagram-canvas]').click(200, 200);
// Verify process added
cy.get('[data-shape=process]').should('exist');
});
it('should edit process label', () => {
cy.get('[data-shape=process]').first().dblclick();
cy.get('[data-cy=label-input]').clear().type('Authentication Service');
cy.get('[data-cy=label-save]').click();
cy.get('[data-shape=process]').first()
.should('contain', 'Authentication Service');
});
});This section defines a comprehensive browser-based integration test plan for the DFD (Data Flow Diagram) component. The plan prioritizes catching selection styling persistence issues while providing complete coverage of all DFD features.
Note: This is a planned test suite. Existing DFD integration tests use Vitest and are located in src/app/pages/dfd/integration/. Those tests are currently skipped pending conversion to browser-based E2E tests due to Angular CDK JIT compilation issues in the Vitest environment.
The DFD component requires browser-first integration testing because:
- Real browser environments test actual user interactions
- DOM verification inspects actual SVG elements and CSS properties
- Visual regression detection catches styling issues that unit tests miss
- Performance monitoring measures actual browser rendering performance
| Priority | Issue | Impact | Test Focus |
|---|---|---|---|
| 1 (Highest) | Selection Styling Persistence | After undo operations, restored cells retain selection styling (glow effects, tools) | Verify clean state restoration after undo/redo |
| 2 | Visual Effects State Management | Visual effects may accumulate or persist across operations | State transitions maintain correct styling throughout workflows |
| 3 | History System Integration | Visual effects may pollute undo/redo history | History contains only structural changes, not visual effects |
``` cypress/e2e/dfd/ ├── critical/ │ ├── selection-styling-persistence.cy.ts # Priority 1 bug │ ├── visual-effects-consistency.cy.ts # Visual state management │ └── history-system-integrity.cy.ts # History filtering ├── core-features/ │ ├── node-creation-workflows.cy.ts # Node creation and styling │ ├── edge-creation-connections.cy.ts # Edge creation and validation │ ├── drag-drop-operations.cy.ts # Movement and positioning │ └── port-management.cy.ts # Port visibility and connections ├── user-workflows/ │ ├── complete-diagram-creation.cy.ts # End-to-end workflows │ ├── context-menu-operations.cy.ts # Right-click operations │ ├── keyboard-interactions.cy.ts # Keyboard shortcuts │ └── multi-user-collaboration.cy.ts # Real-time collaboration ├── advanced-features/ │ ├── z-order-embedding.cy.ts # Layer management │ ├── export-functionality.cy.ts # Export to various formats │ ├── label-editing.cy.ts # In-place text editing │ └── performance-testing.cy.ts # Large diagram performance └── browser-specific/ ├── responsive-behavior.cy.ts # Window resize, zoom, pan ├── cross-browser-compatibility.cy.ts # Browser-specific behaviors └── accessibility-testing.cy.ts # Keyboard navigation, a11y ```
The highest priority test verifies that deleted and restored cells have clean state:
```typescript describe('Selection Styling Persistence Bug', () => { it('should restore deleted nodes without selection styling', () => { // Create node cy.dfdCreateNode('actor', { x: 100, y: 100 }); cy.dfdGetNode('actor').should('be.visible');
// Select node and verify selection styling
cy.dfdSelectNode('actor');
cy.dfdVerifySelectionStyling('actor', true);
cy.dfdVerifyTools('actor', ['button-remove', 'boundary']);
// Delete selected node
cy.dfdDeleteSelected();
cy.dfdGetNodes().should('have.length', 0);
// Undo deletion - CRITICAL VERIFICATION
cy.dfdUndo();
cy.dfdGetNodes().should('have.length', 1);
// VERIFY: No selection styling artifacts
cy.dfdVerifySelectionStyling('actor', false);
cy.dfdVerifyTools('actor', []);
cy.dfdVerifyCleanState('actor');
// VERIFY: Graph selection is empty
cy.dfdGetSelectedCells().should('have.length', 0);
}); }); ```
Tests should use centralized styling constants from `src/app/pages/dfd/constants/styling-constants.ts`:
| Constant | Value | Usage |
|---|---|---|
| `DFD_STYLING.SELECTION.GLOW_COLOR` | `rgba(255, 0, 0, 0.8)` | Selection glow effect |
| `DFD_STYLING.HOVER.GLOW_COLOR` | `rgba(255, 0, 0, 0.6)` | Hover glow effect |
| `DFD_STYLING.CREATION.GLOW_COLOR` | `rgba(0, 150, 255, 0.9)` | Creation highlight (blue) |
| `DFD_STYLING.CREATION.FADE_DURATION_MS` | `500` | Fade animation duration |
| `DFD_STYLING.SELECTION.GLOW_BLUR_RADIUS` | `8` | Selection blur radius |
| Phase | Week | Focus |
|---|---|---|
| 1 | Week 1 | Selection styling persistence tests, visual effects state management, basic Cypress infrastructure |
| 2 | Week 2 | Node/edge creation workflows, history system integration, port management and connections |
| 3 | Week 3 | Complete user workflows, performance testing, browser-specific behaviors |
| 4 | Week 4 | Multi-user collaboration, export functionality, cross-browser compatibility |
- Selection styling persistence eliminated - no selection artifacts after undo
- Visual effects state management - clean state transitions
- History system integrity - only structural changes in history
- Complete workflow testing - end-to-end diagram creation
- Performance validation - large diagrams handle smoothly
- Real DOM verification - actual SVG and CSS inspection
For the complete test plan with all test scenarios and implementation details, see `docs/migrated/agent/dfd-integration-test-plan.md`.
TMI provides a WebSocket test harness for manual testing:
# Build test harness
make build-wstest
# Run 3-terminal test (alice as host, bob and charlie as participants)
make wstest
# Run monitor mode
make monitor-wstest
# Clean up
make clean-wstestTest File: postman/collaboration-tests-collection.json
Tests WebSocket functionality:
- Session creation and joining
- Diagram operations broadcast
- Presenter mode
- Cursor sharing
- User join/leave events
CATS (Contract-driven Automatic Testing Suite) is a security fuzzing tool that tests API endpoints for vulnerabilities and spec compliance.
CATS automatically generates, runs, and reports tests with minimum configuration and no coding effort. Tests are self-healing and do not require maintenance.
Features:
- Boundary testing (very long strings, large numbers)
- Type confusion testing
- Required field validation
- Authentication bypass testing
- Malformed input handling
# Full fuzzing with OAuth authentication
make cats-fuzz
# Fuzz with specific user
make cats-fuzz-user USER=alice
# Fuzz specific endpoint
make cats-fuzz-path ENDPOINT=/addons
# Analyze results
make analyze-cats-resultsTMI has 17 public endpoints (OAuth, OIDC, SAML) that are intentionally accessible without authentication per RFC specifications:
Public Endpoint Categories:
| Category | Count | RFC/Standard |
|---|---|---|
| OIDC Discovery | 5 | RFC 8414 |
| OAuth Flow | 6 | RFC 6749 |
| SAML Flow | 6 | SAML 2.0 |
Implementation:
- Marked with
x-public-endpoint: truevendor extension in OpenAPI spec - CATS uses
--skipFuzzersForExtension=x-public-endpoint=true:BypassAuthenticationto skip auth bypass tests - All other security fuzzers still run on these endpoints
Cacheable Endpoints:
6 discovery endpoints use Cache-Control: public, max-age=3600 (intentionally cacheable per RFC 8414/7517/9728):
- Marked with
x-cacheable-endpoint: truevendor extension - CATS uses
--skipFuzzersForExtension=x-cacheable-endpoint=true:CheckSecurityHeaders
IDOR False Positive Handling:
Filter parameters (like threat_model_id, addon_id) are not IDOR vulnerabilities - they narrow results, not authorize access. The results parser marks these as false positives.
For complete documentation, see docs/migrated/developer/testing/cats-public-endpoints.md.
Results are stored in test/outputs/cats/:
-
cats-results.db- SQLite database of parsed results -
report/- Detailed HTML and JSON reports
Test results are saved as individual JSON files (Test*.json) in test/outputs/cats/report/. Each result has one of these statuses:
-
error- Test failed with unexpected behavior -
warn- Test produced warnings -
success- Test passed
After running CATS, parse results into a SQLite database for analysis:
# Parse results into SQLite
make parse-cats-results
# Query parsed results (excludes false positives)
make query-cats-results
# Full analysis pipeline
make analyze-cats-resultsWhen reviewing CATS results, categorize each finding as:
| Category | Description | Action |
|---|---|---|
| Should Fix (High) | Security vulnerability | Fix immediately |
| Should Fix (Medium) | API contract violation | Fix in next sprint |
| Should Fix (Low) | Minor compliance issue | Add to backlog |
| Should Investigate | Unclear behavior | Review with team |
| False Positive | Expected/correct behavior | Mark as ignored |
| Should Ignore | By design or not applicable | Document reason |
Example Classifications:
-
False Positive: Server returns 200 for
GET /without authorization - this endpoint is intentionally public (security: []) - Should Investigate: Unexpected Accept-Language header handling - needs design decision
- Should Fix (Low): Server returns 400 instead of 405 for unsupported HTTP method
- Should Fix (Medium): Response Content-Type doesn't match OpenAPI schema
AI-Assisted Analysis: For large result sets, use AI agents to analyze test/outputs/cats/report/ files, classifying each error/warning as "should fix", "should ignore", "false positive", or "should investigate" with priority and reasoning.
A comprehensive security analysis report from a CATS fuzzing run (November 2025, CATS v13.3.2) is available at docs/migrated/security/cats-fuzzer-analysis.md. This report demonstrates the analysis approach for CATS results, including:
- Executive summary with test statistics and key findings
- Prioritized findings categorized by severity (Must Fix, Should Fix, Ignore)
- False positive identification with RFC references for expected behavior
- Recommended action plan with time estimates and file locations
The report analyzed 35,680 test results and identified 2 actionable patterns (security headers and error response content types) while documenting 7 false positive categories.
Key Security Headers Identified:
-
Cache-Control: no-store- Prevents caching of sensitive data -
X-Content-Type-Options: nosniff- Prevents MIME-type sniffing attacks -
X-Frame-Options: DENY- Prevents clickjacking attacks -
Content-Security-Policy: frame-ancestors 'none'- Modern clickjacking protection
See OWASP HTTP Headers Cheat Sheet for current header recommendations.
Early CATS fuzzing identified several important issues that have been resolved:
-
CheckDeletedResourcesNotAvailable Bug -
DeleteGroupAndDatawas looking up groups by name instead ofinternal_uuid, causing incorrect deletions when multiple groups shared the same name. Fixed inauth/repository/deletion_repository.go. -
RemoveFields on oneOf Schemas - CATS doesn't fully understand
oneOfconstraints. When removing fields fromCreateAdministratorRequest(which requires exactly one ofemail,provider_user_id, orgroup_name), the API correctly returns 400. -
XSS on Query Parameters - XSS warnings on GET requests are false positives for JSON APIs. TMI returns
application/json, not HTML, so XSS payloads in query parameters are not exploitable. Theparse-cats-results.pyscript now automatically flags these.
For the complete historical analysis and bug traces, see docs/migrated/developer/testing/cats-findings-plan.md.
CATS may flag legitimate API responses as "errors" due to expected behavior patterns. These are not security vulnerabilities - they are correct, RFC-compliant responses or expected API behavior.
The scripts/parse-cats-results.py script automatically detects and marks false positives using the is_false_positive() function, which handles 16+ categories:
| Category | Response Codes | Description |
|---|---|---|
| OAuth/Auth | 401, 403 | Expected authentication failures during fuzzing |
| Rate Limiting | 429 | Infrastructure protection, not API behavior |
| Validation | 400 | API correctly rejects malformed input |
| Not Found | 404 | Expected when fuzzing with random/invalid resource IDs |
| IDOR | 200 | Filter parameters and admin endpoints behave correctly |
| HTTP Methods | 400, 405 | Correct rejection of unsupported methods |
| Response Contract | Various | Header mismatches are spec issues, not security issues |
| Conflict | 409 | Duplicate name conflicts from fuzzed values |
| Content Type | 400 | Go HTTP layer transport errors (text/plain) |
| Injection | Various | JSON API data reflection is not XSS |
| Header Validation | 400 | Correct rejection of malformed headers |
| Transfer Encoding | 501 | Correct rejection per RFC 7230 |
Using Filtered Results:
The database provides two views:
-
test_results_view- All tests withis_oauth_false_positiveflag -
test_results_filtered_view- Excludes false positives (recommended)
-- Query actual errors (excluding false positives)
SELECT * FROM test_results_filtered_view WHERE result = 'error';
-- View false positives separately
SELECT * FROM test_results_view WHERE is_oauth_false_positive = 1;Quick Reference:
| Scenario | Is False Positive? | Reason |
|---|---|---|
| 401 with "invalid_token" | Yes | Correct OAuth error response |
| 403 with "forbidden" | Yes | Correct permission denied |
| 409 on POST /admin/groups | Yes | Duplicate name from fuzzed values |
| 400 from header fuzzers | Yes | Correct header validation |
| 429 rate limit | Yes | Infrastructure protection |
| 404 from boundary fuzzers | Yes | Expected with invalid IDs |
| 500 with "NullPointerException" | No | Actual server error |
For complete details on all false positive categories, see docs/migrated/developer/testing/cats-oauth-false-positives.md.
The following are documented false positives with detailed explanations:
1. PrefixNumbersWithZeroFields (400 Bad Request)
- CATS sends numeric values as strings with leading zeros (e.g.,
"0095") - JSON numbers with leading zeros are invalid per RFC 8259
- The API correctly rejects these malformed inputs
2. NoSQL Injection Detection (201 Created)
- CATS reports "NoSQL injection vulnerability detected" when payloads like
{ $where: function() { return true; } }are stored - TMI uses PostgreSQL, not MongoDB - NoSQL operators have no effect
- The payload is stored as a literal string, not executed
3. POST /admin/administrators Validation (400 Bad Request)
- This endpoint uses
oneOfschema requiring exactly one of:email,provider_user_id, orgroup_name - CATS generates bodies that may not satisfy the oneOf constraint
- The API correctly validates and returns 400 for invalid combinations
4. Connection Errors (Response Code 999)
- HTTP 999 is not a real status code - it indicates connection errors
- Often occurs with URL encoding issues (e.g., trailing
%) - This is a CATS/network issue, not an API bug
5. StringFieldsLeftBoundary on Optional Fields (201 Created)
- CATS sends empty strings for optional fields and expects 4XX
- Empty strings on optional fields are valid input
- The API correctly creates resources with empty optional fields
6. GET Filter Parameters Returning Empty Results (200 OK)
- CATS sends fuzzing values as filter parameters
- Returning empty results for non-matching filters is standard REST behavior
- Endpoints:
/admin/groups,/admin/users,/admin/administrators
7. XSS on Query Parameters (200 OK)
- TMI is a JSON API, not an HTML-rendering application
- XSS requires HTML context to execute - JSON responses don't render HTML
- Client applications handle output encoding, not the API
8. POST /admin/groups Duplicate Rejection (409 Conflict)
- CATS's boundary fuzzers may trigger duplicate group creation
- 409 Conflict is proper REST semantics for duplicate resources
9. POST /admin/administrators User/Group Not Found (404)
- CATS generates random values for reference fields
- 404 is correct when referenced users/groups don't exist
The following fuzzers were previously skipped due to CATS 13.5.0 bugs but are now re-enabled:
-
MassAssignmentFuzzer: Was crashing with
JsonPath.InvalidModificationExceptionon array properties (#191) -
InsertRandomValuesInBodyFuzzer: Was crashing with
IllegalArgumentException: count is negativeduring HTML report generation (#193)
Ensure you're running CATS 13.6.0 or later to avoid these issues.
TMI validates addon name and description fields for template injection patterns (defense-in-depth):
| Pattern | Description | Example |
|---|---|---|
{{ / }}
|
Handlebars, Jinja2, Angular, Go templates | {{constructor.constructor('alert(1)')()}} |
${ |
JavaScript template literals, Freemarker | ${alert(1)} |
<% / %>
|
JSP, ASP, ERB server templates | <%=System.getProperty('user.home')%> |
#{ |
Spring EL, JSF EL expressions | #{T(java.lang.Runtime).exec('calc')} |
${{ |
GitHub Actions context injection | ${{github.event.issue.title}} |
NoSQL syntax is allowed since it's harmless in a SQL (PostgreSQL) context.
TMI maintains a detailed remediation plan documenting the analysis and resolution of all CATS fuzzing findings. The plan tracks:
- 24,211 successes (99.4%), 116 errors (0.5%), 39 warnings (0.2%)
- Issue categories: False positives, OpenAPI spec issues, potential security issues, SAML documentation, input validation edge cases
- All issues resolved with documented resolutions
Key resolutions include:
-
IDOR on DELETE /addons/{id}: False positive - admin-only endpoint by design (see
api/addon_handlers.goline 207) -
Admin endpoint HappyPath failures: OpenAPI spec updated with
oneOfandmaximumconstraints -
WebhookQuota schema mismatch: Added
created_at/modified_atfields to schema -
SAML 400 responses: Added to OpenAPI spec for
/saml/{provider}/loginand/saml/{provider}/metadata
Skipped fuzzers (false positives):
-
DuplicateHeaders- Server correctly ignores duplicate headers per HTTP spec -
LargeNumberOfRandomAlphanumericHeaders- Server correctly ignores extra headers -
EnumCaseVariantFields- Server correctly uses case-sensitive enum validation
For complete remediation details including OpenAPI changes and expected metrics, see docs/migrated/developer/testing/cats-remediation-plan.md.
CATS fuzzing can report false positives when testing endpoints that require prerequisite objects (e.g., testing GET /threat_models/{id}/threats/{threat_id} fails with 404 when no threat model exists). TMI addresses this by pre-creating a complete object hierarchy before fuzzing.
threat_model (root)
├── threats
│ └── metadata
├── diagrams
│ └── metadata
├── documents
│ └── metadata
├── assets
│ └── metadata
├── notes
│ └── metadata
├── repositories
│ └── metadata
└── metadata
addons (independent root)
└── invocations
webhooks (independent root)
└── deliveries
client_credentials (independent root)
Test data is created automatically when running make cats-fuzz:
# Full fuzzing (includes test data creation via cats-seed)
make cats-fuzz
# Or manually create test data
make cats-create-test-data TOKEN=eyJhbGc... SERVER=http://localhost:8080 USER=aliceThe scripts/cats-create-test-data.sh script:
- Authenticates via OAuth (uses the OAuth callback stub)
- Creates one of each object type with stable IDs
- Stores IDs in a reference file (
test/outputs/cats/cats-test-data.json) - Generates YAML reference data for CATS (
test/outputs/cats/cats-test-data.yml)
CATS uses the --refData parameter to substitute path parameters with real IDs. The YAML file uses the all: key for global substitution:
# CATS Reference Data - Path-based format for parameter replacement
all:
id: <threat_model_uuid>
threat_model_id: <threat_model_uuid>
threat_id: <threat_uuid>
diagram_id: <diagram_uuid>
document_id: <document_uuid>
asset_id: <asset_uuid>
note_id: <note_uuid>
repository_id: <repository_uuid>
webhook_id: <webhook_uuid>
addon_id: <addon_uuid>
client_credential_id: <credential_uuid>
key: cats-test-key
# Admin resource identifiers
group_id: <group_uuid>
internal_uuid: <user_internal_uuid>For complete reference data format documentation, see CATS Reference Data File.
The TMI server provides comprehensive test coverage reporting with both unit and integration test coverage.
# Generate full coverage report (unit + integration + merge + reports)
make test-coverage
# Run only unit tests with coverage
make test-coverage-unit
# Run only integration tests with coverage
make test-coverage-integration
# Generate reports from existing profiles
make generate-coverageCoverage Directory (coverage/):
-
unit_coverage.out- Raw unit test coverage data -
integration_coverage.out- Raw integration test coverage data -
combined_coverage.out- Merged coverage data -
unit_coverage_detailed.txt- Detailed unit test coverage by function -
integration_coverage_detailed.txt- Detailed integration test coverage -
combined_coverage_detailed.txt- Detailed combined coverage -
coverage_summary.txt- Executive summary with key metrics
HTML Reports Directory (coverage_html/):
-
unit_coverage.html- Interactive unit test coverage report -
integration_coverage.html- Interactive integration test coverage report -
combined_coverage.html- Interactive combined coverage report
View HTML Report:
open coverage_html/combined_coverage.html- Unit Tests: Target 80%+ coverage for core business logic
- Integration Tests: Target 70%+ coverage for API endpoints and workflows
- Combined: Target 85%+ overall coverage
High priority areas for coverage:
- API Handlers - All HTTP endpoints should be tested
- Business Logic - Core threat modeling functionality
- Authentication & Authorization - Security-critical code
- Database Operations - Data persistence and retrieval
- Cache Management - Performance-critical caching logic
- Go 1.25 or later
- Docker (for integration tests with PostgreSQL and Redis)
-
gocovmergetool (automatically installed if missing)
Coverage integration tests use dedicated ports (see config/coverage-report.yml):
- PostgreSQL: localhost:5434 (container: tmi-coverage-postgres)
- Redis: localhost:6381 (container: tmi-coverage-redis)
These ports avoid conflicts with development databases (5432, 6379).
Docker Not Available:
# Start Docker on macOS
open -a Docker
# Verify Docker is running
docker infoDatabase Connection Issues:
# Clean up any existing containers
make clean-everything
# Or manually clean up
docker stop tmi-coverage-postgres tmi-coverage-redis 2>/dev/null
docker rm tmi-coverage-postgres tmi-coverage-redis 2>/dev/nullCoverage Tool Missing:
go install github.com/wadey/gocovmerge@latestCustom Coverage Profiles:
# Test specific packages
go test -coverprofile=custom.out ./api/...
# Test with race detection
go test -race -coverprofile=race.out ./...
# Generate HTML from custom profile
go tool cover -html=custom.out -o custom.htmlCoverage Analysis:
# Find functions with zero coverage
go tool cover -func=coverage/combined_coverage.out | awk '$3 == "0.0%" {print $1}'
# Show files sorted by coverage
go tool cover -func=coverage/combined_coverage.out | sort -k3 -n# Generate coverage report
pnpm run test:coverage
# View report
open coverage/index.htmlCoverage Configuration: vitest.config.ts
export default defineConfig({
test: {
coverage: {
provider: 'v8',
reporter: ['text', 'html', 'lcov'],
exclude: [
'node_modules/',
'src/**/*.spec.ts',
'src/environments/'
]
}
}
});- One test file per source file
- Group related tests with describe blocks
- Use clear, descriptive test names
- Follow AAA pattern: Arrange, Act, Assert
- Use factories for test data
- Create minimal test data
- Clean up after tests
- Use predictable test users (login hints)
- Tests should be independent
- Don't rely on test order
- Clean up between tests
- Mock external dependencies
- Test one thing per test
- Use specific assertions
- Test both happy path and error cases
- Verify side effects
- Keep unit tests fast (<1s each)
- Use before/after hooks efficiently
- Parallelize tests when possible
- Cache test fixtures
- DRY - Don't Repeat Yourself
- Use helper functions
- Keep tests simple
- Update tests with code
Tests run automatically on:
- Pull requests
- Pushes to main branch
- Scheduled nightly builds
Workflow (.github/workflows/test.yml):
name: Tests
on: [push, pull_request]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- run: make test-unit
integration-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- run: make test-integration
api-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- run: npm install -g newman
- run: make test-api# Clean everything and retry
make clean-everything
make test-integration
# Check container logs
docker logs tmi-integration-postgres
docker logs tmi-integration-redis
# Verify ports are free
lsof -ti :5434 # PostgreSQL
lsof -ti :6381 # Redis# Check server is running
curl http://localhost:8080/
# Check authentication
curl -H "Authorization: Bearer TOKEN" http://localhost:8080/threat_models
# Run specific collection
newman run postman/comprehensive-test-collection.json# Clear Cypress cache
pnpm run test:e2e:clean
# Run in headed mode to see what's happening
## TMI-UX Testing Utilities
<!-- Migrated from: docs/testing/UNIT_TEST_CHECKLIST.md on 2026-01-25 -->
TMI-UX provides standardized testing utilities in the \`src/testing/\` directory to make testing easier, more consistent, and more maintainable.
### Unit Test Implementation Checklist
## TMI-UX Testing Utilities
<!-- Migrated from: docs/testing/UNIT_TEST_PROGRESS.md on 2026-01-25 -->
TMI-UX uses Vitest with promise-based async patterns (no `done()` callbacks). This section documents established patterns and utilities for service unit testing.
### Service Test Coverage
<!-- Updated from: docs/testing/UNIT_TEST_PLAN_SUMMARY.md verification on 2026-01-25 -->
| Metric | Value |
|--------|-------|
| Total Services | 72 |
| Services with Tests | 66 (91.7%) |
| Services Needing Tests | 6 (8.3%) |
**Services still requiring tests:**
1. `client-credential.service.ts` - Client credential management
2. `user-preferences.service.ts` - User preferences storage
3. `threat-model-report.service.ts` - Report generation
4. `import-orchestrator.service.ts` - Multi-step import workflow (high complexity)
5. `app-websocket-event-processor.service.ts` - WebSocket event processing
6. `ui-presenter-cursor-display.service.ts` - Cursor rendering
Unit tests have been implemented across the following categories:
**Core Services** (`src/app/core/services/`):
- Dialog direction, theme, operator, server connection
- Addon, administrator, quota, webhook management
- Collaboration session and DFD collaboration state
**TM Services** (`src/app/pages/tm/services/`):
- Authorization checking and role management
- Import utilities: ID translation, field filtering, reference rewriting
- Provider adapters and authorization preparation
**DFD Application Services** (`src/app/pages/dfd/application/services/`):
- Diagram loading, state management, history (undo/redo)
- Export and SVG optimization
- Operation state management and broadcasting
- Event handling and rejection recovery
**DFD Presentation Services** (`src/app/pages/dfd/presentation/services/`):
- Tooltip content and positioning
- Presenter coordination with WebSocket
- Cursor tracking and selection broadcasting
**Shared Services** (`src/app/shared/services/`):
- Notification with spam prevention
- Form validation with multiple validators
- Framework JSON loading
- Cell data extraction from X6 and threat models
**I18N Services** (`src/app/i18n/`):
- Language switching and direction management
- Translation file loading via HTTP
### Key Testing Patterns
#### Mock Setup
- Use typed mocks with `ReturnType<typeof vi.fn>`
- Cast to service types with `as unknown as Type`
- For properties (not methods), include them directly in mock object
```typescript
let mockAuthService: {
userEmail: string;
isAuthenticated: ReturnType<typeof vi.fn>;
};
beforeEach(() => {
mockAuthService = {
userEmail: 'test@example.com', // property, not vi.fn()
isAuthenticated: vi.fn().mockReturnValue(true)
};
});-
vi.useFakeTimers()in beforeEach -
vi.useRealTimers()in afterEach -
vi.advanceTimersByTimeAsync(1)to trigger scheduled operations -
Avoid
vi.runAllTimersAsync()(causes infinite loops with intervals)
| Issue | Problem | Solution |
|---|---|---|
| Timer Infinite Loops |
vi.runAllTimersAsync() causes infinite recursion |
Use vi.advanceTimersByTimeAsync(milliseconds)
|
| Mock Property Access | Services accessing properties not methods | Include properties directly in mock object |
| Event Handler Types | ESLint complains about Function type |
Use explicit signature: (event: EventType) => void
|
Shared mock factories in src/app/pages/dfd/application/services/test-helpers/mock-services.ts:
- LoggerService, AppStateService, AppHistoryService
- AppOperationStateManager, AppDiagramService
- InfraNodeConfigurationService, InfraX6GraphAdapter
- Graph (with batchUpdate), DfdCollaborationService
- InfraDfdWebsocketAdapter, AppDiagramResyncService
- Contributing - Learn contribution workflow
- Getting Started - Set up dev environment
- API Integration - Learn API patterns
- Using TMI for Threat Modeling
- Accessing TMI
- Creating Your First Threat Model
- Understanding the User Interface
- Working with Data Flow Diagrams
- Managing Threats
- Collaborative Threat Modeling
- Using Notes and Documentation
- Metadata and Extensions
- Planning Your Deployment
- Deploying TMI Server
- OCI Container Deployment
- Deploying TMI Web Application
- Setting Up Authentication
- Database Setup
- Component Integration
- Post-Deployment
- Monitoring and Health
- Database Operations
- Security Operations
- Performance and Scaling
- Maintenance Tasks