feat: add Support Vector Classifier (SVC) implementation

[AliAlimohammadi]

Description

This PR adds a Support Vector Classifier (SVC) implementation to the machine learning module. The SVC is a powerful binary classification algorithm that finds the optimal hyperplane to separate two classes.

Features Implemented

• Linear Kernel: Standard linear classification for linearly separable data
• RBF Kernel: Radial Basis Function kernel for non-linearly separable data
• Soft Margin Support: Configurable regularization parameter ($C$) for handling overlapping classes
• Dual Formulation: Implements the Wolfe dual optimization problem
• Comprehensive Tests: 7 unit tests covering various scenarios

Algorithm Details

The implementation uses the dual formulation of the SVM optimization problem:

Maximize:

$$L(\lambda) = \sum_i \lambda_i - \frac{1}{2} \sum_i \sum_j \lambda_i \lambda_j y_i y_j K(x_i, x_j)$$

Subject to:

$$0 \leq \lambda_i \leq C \quad \text{(for all } i\text{)}$$

$$\sum_i \lambda_i y_i = 0$$

Where:

• $\lambda$ (lambda) are the Lagrange multipliers
• $y$ are the class labels
• $K$ is the kernel function
• $C$ is the regularization parameter

Solver Implementation:

• Uses gradient descent with adaptive learning rate
• Iterative constraint projection (up to 10 iterations per step)
• Corrected bias calculation using support vectors
• Tight convergence tolerance (1e-8)

Production Note: For large-scale or production use, integrate with a proper QP solver (OSQP, Clarabel) or use SMO algorithm.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have run cargo fmt --all
• I have run cargo clippy --all-targets --all-features -- -D warnings
• The algorithm is based on a well-known algorithm or paper (SVM is a foundational ML algorithm)

Testing

All tests pass successfully:

cargo test support_vector_classifier

Test coverage includes:

• ✅ Linear kernel with simple dataset
• ✅ RBF kernel with XOR-like pattern
• ✅ Invalid gamma parameter handling
• ✅ Invalid regularization parameter handling
• ✅ Empty data validation
• ✅ Dimension mismatch validation
• ✅ Support vector counting

Code Quality

• ✅ All functions are documented with doc comments
• ✅ All doctests pass with proper imports
• ✅ Code formatted with cargo fmt
• ✅ No clippy warnings
• ✅ Follows Rust naming conventions
• ✅ Uses idiomatic Rust patterns (Result types, iterators, etc.)
• ✅ Uses clone_from() for efficient cloning

Dependencies

Added to Cargo.toml:

[dependencies]
ndarray = "0.17.2"

Note: The repository already uses nalgebra for linear algebra operations. This PR adds ndarray which is specifically needed for the SVC implementation and may be useful for future machine learning algorithms.

Examples

All usage examples are included in the documentation (doctests). These demonstrate:

• Basic linear classification
• RBF kernel for non-linear patterns
• Soft margin configuration

View documentation:

cargo doc --open

Run doc tests:

cargo test --doc

Example Code

Example 1: Linear Kernel (Linearly Separable Data)

use ndarray::array;
use the_algorithms_rust::machine_learning::{Kernel, SVC};

// Create a simple linearly separable dataset
// Class 1: points on the left (x < 0.5)
// Class -1: points on the right (x > 0.5)
let observations = vec![
    array![0.0, 1.0],
    array![0.0, 2.0],
    array![0.2, 1.5],
    array![1.0, 1.0],
    array![1.0, 2.0],
    array![0.8, 1.5],
];
let classes = array![1.0, 1.0, 1.0, -1.0, -1.0, -1.0];

// Create SVC with linear kernel and hard margin (infinite regularization)
let mut svc = SVC::new(Kernel::Linear, f64::INFINITY).unwrap();

// Train the classifier
svc.fit(&observations, &classes).unwrap();

println!("Number of support vectors: {}", svc.n_support_vectors());

// Test predictions
let test_points = vec![
    (array![0.1, 1.0], 1.0),
    (array![0.9, 1.0], -1.0),
    (array![0.5, 1.5], 1.0),
];

for (point, expected) in test_points {
    let predicted = svc.predict(&point);
    println!("  {:?} -> predicted: {}, expected: {}", 
             point.as_slice().unwrap(), predicted, expected);
}

Example 2: RBF Kernel (Non-linearly Separable Data)

use ndarray::array;
use the_algorithms_rust::machine_learning::{Kernel, SVC};

// XOR-like pattern: not linearly separable
// Class 1: corners (0,0) and (1,1)
// Class -1: corners (0,1) and (1,0)
let observations = vec![
    array![0.0, 0.0],
    array![0.1, 0.1],
    array![0.9, 0.9],
    array![1.0, 1.0],
    array![0.0, 1.0],
    array![0.1, 0.9],
    array![0.9, 0.1],
    array![1.0, 0.0],
];
let classes = array![1.0, 1.0, 1.0, 1.0, -1.0, -1.0, -1.0, -1.0];

// Use RBF kernel with gamma = 2.0
let mut svc = SVC::new(Kernel::Rbf { gamma: 2.0 }, 1.0).unwrap();
svc.fit(&observations, &classes).unwrap();

println!("Number of support vectors: {}", svc.n_support_vectors());

// Test on corner points and center
let test_points = vec![
    (array![0.05, 0.05], 1.0),  // Near (0,0) - class 1
    (array![0.95, 0.95], 1.0),  // Near (1,1) - class 1
    (array![0.05, 0.95], -1.0), // Near (0,1) - class -1
    (array![0.95, 0.05], -1.0), // Near (1,0) - class -1
    (array![0.5, 0.5], 1.0),    // Center
];

for (point, expected) in test_points {
    let predicted = svc.predict(&point);
    println!("  {:?} -> predicted: {}, expected: {}", 
             point.as_slice().unwrap(), predicted, expected);
}

Example 3: Soft Margin Classification

use ndarray::array;
use the_algorithms_rust::machine_learning::{Kernel, SVC};

// Create data with some overlap between classes
let observations = vec![
    // Class 1 cluster (mostly left side)
    array![0.0, 0.5],
    array![0.1, 0.6],
    array![0.2, 0.4],
    array![0.5, 0.5], // Outlier
    // Class -1 cluster (mostly right side)
    array![0.8, 0.5],
    array![0.9, 0.6],
    array![1.0, 0.4],
    array![0.4, 0.5], // Outlier
];
let classes = array![1.0, 1.0, 1.0, 1.0, -1.0, -1.0, -1.0, -1.0];

// Compare hard margin vs soft margin
println!("Hard Margin (C = ∞):");
let mut svc_hard = SVC::new(Kernel::Linear, f64::INFINITY).unwrap();
svc_hard.fit(&observations, &classes).unwrap();
println!("  Support vectors: {}", svc_hard.n_support_vectors());

println!("Soft Margin (C = 1.0):");
let mut svc_soft = SVC::new(Kernel::Linear, 1.0).unwrap();
svc_soft.fit(&observations, &classes).unwrap();
println!("  Support vectors: {}", svc_soft.n_support_vectors());

// Test on points between clusters
let test_point = array![0.5, 0.5];
println!("Prediction for point in overlap region {:?}:", 
         test_point.as_slice().unwrap());
println!("  Hard margin: {}", svc_hard.predict(&test_point));
println!("  Soft margin: {}", svc_soft.predict(&test_point));

Examples Output

=== Support Vector Classifier Examples ===

Example 1: Linear Kernel (Linearly Separable Data)
---------------------------------------------------
Training completed!
Number of support vectors: 4

Predictions:
  [0.1, 1.0] -> predicted: 1, expected: 1 ✓
  [0.9, 1.0] -> predicted: -1, expected: -1 ✓
  [0.5, 1.5] -> predicted: 1, expected: 1 ✓

Example 2: RBF Kernel (Non-linearly Separable Data)
----------------------------------------------------
Training completed!
Number of support vectors: 8

Predictions:
  [0.05, 0.05] -> predicted: 1, expected: 1 ✓
  [0.95, 0.95] -> predicted: 1, expected: 1 ✓
  [0.05, 0.95] -> predicted: -1, expected: -1 ✓
  [0.95, 0.05] -> predicted: -1, expected: -1 ✓
  [0.5, 0.5] -> predicted: 1, expected: 1 ✓

Example 3: Soft Margin (Overlapping Classes)
----------------------------------------------

Hard Margin (C = ∞):
  Support vectors: 8

Soft Margin (C = 1.0):
  Support vectors: 8

Prediction for point in overlap region [0.5, 0.5]:
  Hard margin: 1
  Soft margin: 1

Future Enhancements

Potential improvements for future PRs:

• Integration with a proper QP solver (OSQP, Clarabel) for better optimization
• Additional kernels (polynomial, sigmoid)
• Multi-class classification support
• Probability estimates using Platt scaling

References

• Support Vector Machines - Wikipedia
• RBF Kernel - Wikipedia

---

Note to Reviewers: The optimization uses a simplified gradient descent approach for educational clarity. While this works well for small datasets, a production implementation would benefit from a proper QP solver. I'm open to suggestions on whether to integrate a more robust solver in this PR or leave it for a future enhancement.

[AliAlimohammadi]

@siriak, this is ready to be merged.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 96.07843% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.06%. Comparing base (4402903) to head (b29707b).

Files with missing lines	Patch %	Lines
src/machine_learning/support_vector_classifier.rs	96.07%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1005      +/-   ##
==========================================
- Coverage   96.06%   96.06%   -0.01%     
==========================================
  Files         371      372       +1     
  Lines       25943    26147     +204     
==========================================
+ Hits        24922    25117     +195     
- Misses       1021     1030       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.