Color Extraction Guide

Learn how VideoIntel extracts dominant colors from videos using K-means clustering, smart sampling, and optimized pixel analysis for fast and accurate results.

Overview

Color extraction identifies the most dominant colors in a video by analyzing frames and grouping similar colors together. This is useful for:

  • UI Theming: Generate color schemes matching video content
  • Video Categorization: Classify videos by color palette
  • Design Systems: Create brand-aligned color palettes
  • Thumbnail Matching: Ensure UI colors complement video colors
  • Mood Analysis: Understand emotional tone through color
// Extract 5 dominant colors from a video
const colors = await videoIntel.extractColors(video, {
  count: 5,
  sampleFrames: 10,
  quality: 'balanced'
});

// Use the colors
colors.forEach(color => {
  console.log(`${color.hex} - ${color.percentage}% of video`);
  // Output example: "#3B82F6 - 32.5% of video"
});

How It Works

VideoIntel's color extraction uses a sophisticated multi-stage pipeline:

Step 1: Frame Sampling

Extract frames at evenly distributed intervals throughout the video. Default: 10 frames from start to end, ensuring representative sampling of the entire video.

Step 2: Pixel Extraction

Extract pixel data from each frame and sample based on quality setting. Balanced quality samples 30% of pixels for good accuracy with fast processing.

Step 3: K-Means Clustering

Group similar colors together using K-means clustering algorithm. This finds the N most dominant colors by iteratively improving cluster centers.

Step 4: Color Conversion

Convert cluster centers to multiple color formats (hex, RGB, HSL) and calculate dominance percentages for each color.

K-Means Clustering Algorithm

K-means is an unsupervised machine learning algorithm that groups similar colors:

// Simplified K-means pseudo-code
function kMeansClustering(pixels: RGB[], k: number) {
  // 1. Initialize K random cluster centers (K-means++ for better results)
  let centers = initializeKMeansPlusPlus(pixels, k);
  
  for (let iteration = 0; iteration < maxIterations; iteration++) {
    // 2. Assign each pixel to nearest cluster center
    const assignments = pixels.map(pixel => 
      findNearestCenter(pixel, centers)
    );
    
    // 3. Recalculate centers as mean of assigned pixels
    const newCenters = calculateClusterMeans(pixels, assignments, k);
    
    // 4. Check convergence (centers stop moving)
    if (hasConverged(centers, newCenters)) {
      break;
    }
    
    centers = newCenters;
  }
  
  // 5. Calculate dominance percentage for each cluster
  const percentages = calculatePercentages(assignments, k);
  
  return centers.map((center, i) => ({
    color: center,
    percentage: percentages[i]
  }));
}

K-Means++ Initialization

VideoIntel uses K-means++ for smarter initial cluster placement, which:

  • Spreads initial centers far apart in color space
  • Converges 2-3x faster than random initialization
  • Produces more consistent results across runs
  • Avoids poor local minima
// K-means++ initialization
function initializeKMeansPlusPlus(pixels: RGB[], k: number) {
  const centers: RGB[] = [];
  
  // 1. Choose first center randomly
  centers.push(pixels[randomIndex()]);
  
  // 2. Choose remaining centers based on distance
  for (let i = 1; i < k; i++) {
    // Calculate distance from each pixel to nearest existing center
    const distances = pixels.map(pixel => {
      const minDist = Math.min(...centers.map(c => 
        colorDistance(pixel, c)
      ));
      return minDist * minDist;  // Square for probability weighting
    });
    
    // 3. Choose next center with probability proportional to distance
    const nextCenter = weightedRandomChoice(pixels, distances);
    centers.push(nextCenter);
  }
  
  return centers;
}

Frame Sampling Strategy

VideoIntel samples frames evenly throughout the video to ensure representative color coverage:

// Example: 60-second video, 10 frames
const interval = 60 / 10;  // 6 seconds
const timestamps = [3, 9, 15, 21, 27, 33, 39, 45, 51, 57];
//                   ↑ Start at 0.5 × interval to avoid black frames

// Distribution ensures:
// ✓ Start, middle, and end are all represented
// ✓ No clustering in one section
// ✓ Captures color changes throughout video

Pixel Extraction & Sampling

Analyzing every pixel would be slow. VideoIntel uses adaptive sampling based on quality:

QualitySampling RatePixels AnalyzedProcessing Time
fast10%~200K pixels1-2 seconds
balanced ⭐30%~600K pixels3-5 seconds
best100%~2M pixels10-15 seconds
// Pixel extraction with sampling
function extractPixels(frame: Canvas, samplingRate: number) {
  const ctx = frame.getContext('2d');
  const imageData = ctx.getImageData(0, 0, frame.width, frame.height);
  const pixels: RGB[] = [];
  
  // Calculate step size based on sampling rate
  // 30% sampling = step every ~3 pixels
  const step = Math.round(1 / samplingRate);
  
  for (let i = 0; i < imageData.data.length; i += 4 * step) {
    const r = imageData.data[i];
    const g = imageData.data[i + 1];
    const b = imageData.data[i + 2];
    const a = imageData.data[i + 3];
    
    // Skip fully transparent pixels
    if (a > 0) {
      pixels.push([r, g, b]);
    }
  }
  
  return pixels;
}

Color Clustering Process

The clustering process iteratively refines color groups:

// Example clustering progression for 5 colors

Iteration 0 (Initial):
Centers: [Random colors from K-means++ initialization]

Iteration 1:
Centers: [Updated based on pixel assignments]
Movement: Large (centers adjust to data)

Iteration 5:
Centers: [Refined positions]
Movement: Moderate (converging)

Iteration 10:
Centers: [Nearly final positions]
Movement: Small (<1% change)

Iteration 12:
Centers: [Converged]
Movement: Negligible → STOP

Result:
Color 1: #2563EB (Blue)   - 35.2%
Color 2: #10B981 (Green)  - 28.7%
Color 3: #F59E0B (Orange) - 18.3%
Color 4: #6B7280 (Gray)   - 12.1%
Color 5: #1F2937 (Dark)   - 5.7%

Configuration Options

Fine-tune color extraction for your specific needs:

const colors = await videoIntel.extractColors(video, {
  // Number of colors to extract (2-10)
  count: 5,             // Default: 5
  
  // Number of frames to sample from video
  sampleFrames: 10,     // Default: 10
  
  // Sampling quality
  quality: 'balanced',  // 'fast' | 'balanced' | 'best'
});

// Output format
interface Color {
  hex: string;            // "#3B82F6"
  rgb: [number, number, number];  // [59, 130, 246]
  hsl: [number, number, number];  // [217, 91, 60]
  percentage: number;     // 32.5 (percentage of video)
}

Choosing Color Count

The optimal number of colors depends on your use case:

// Minimalist palette (2-3 colors)
const minimal = await videoIntel.extractColors(video, { count: 3 });
// Use case: Simple color scheme, accent colors

// Standard palette (4-5 colors)
const standard = await videoIntel.extractColors(video, { count: 5 });
// Use case: UI theming, brand colors

// Rich palette (6-8 colors)
const rich = await videoIntel.extractColors(video, { count: 8 });
// Use case: Detailed analysis, design systems

// Comprehensive (9-10 colors)
const comprehensive = await videoIntel.extractColors(video, { count: 10 });
// Use case: Color grading, mood boards

Best Practices

1. Match Quality to Use Case

// Real-time preview (prioritize speed)
const previewColors = await videoIntel.extractColors(video, {
  count: 3,
  sampleFrames: 5,
  quality: 'fast'
});

// Production color palette (balance speed & accuracy)
const productionColors = await videoIntel.extractColors(video, {
  count: 5,
  sampleFrames: 10,
  quality: 'balanced'  // ⭐ Recommended
});

// High-precision analysis (prioritize accuracy)
const precisionColors = await videoIntel.extractColors(video, {
  count: 8,
  sampleFrames: 20,
  quality: 'best'
});

2. Filter Out Near-White/Black (Optional)

For UI theming, you might want to exclude neutral colors:

function filterNeutrals(colors: Color[], threshold = 15) {
  return colors.filter(color => {
    const [r, g, b] = color.rgb;
    const max = Math.max(r, g, b);
    const min = Math.min(r, g, b);
    
    // Skip near-white colors (high brightness, low saturation)
    if (max > 240 && (max - min) < threshold) {
      return false;
    }
    
    // Skip near-black colors (low brightness)
    if (max < 20) {
      return false;
    }
    
    return true;
  });
}

// Usage
const allColors = await videoIntel.extractColors(video, { count: 8 });
const vibrantColors = filterNeutrals(allColors);

3. Sort by Dominance or Brightness

const colors = await videoIntel.extractColors(video, { count: 5 });

// Already sorted by dominance (most common first)
const mostDominant = colors[0];

// Sort by brightness (for gradients)
const byBrightness = [...colors].sort((a, b) => {
  const brightnessA = (a.rgb[0] + a.rgb[1] + a.rgb[2]) / 3;
  const brightnessB = (b.rgb[0] + b.rgb[1] + b.rgb[2]) / 3;
  return brightnessA - brightnessB;  // Dark to light
});

// Sort by saturation (most vibrant first)
const bySaturation = [...colors].sort((a, b) => {
  return b.hsl[1] - a.hsl[1];  // High to low saturation
});

Performance Optimization

Benchmarks

ConfigurationPixels AnalyzedTime (720p)Accuracy
5 frames, fast~100K0.5-1s85%
10 frames, balanced~600K3-5s95%
20 frames, best~4M12-18s98%

💡 Performance Tip

The 'balanced' quality setting provides 95% accuracy at 5x the speed of 'best' quality. Unless you need pixel-perfect precision, balanced is the recommended choice.

Common Use Cases

UI Theming

async function generateTheme(videoFile: File) {
  const colors = await videoIntel.extractColors(videoFile, {
    count: 5,
    quality: 'balanced'
  });
  
  // Sort by dominance
  const [primary, secondary, accent, ...rest] = colors;
  
  return {
    primary: primary.hex,        // Most dominant color
    secondary: secondary.hex,    // Second most dominant
    accent: accent.hex,          // Third for highlights
    
    // Create variations
    primaryLight: lighten(primary.hex, 20),
    primaryDark: darken(primary.hex, 20),
    
    // Use in CSS
    css: `
      --color-primary: ${primary.hex};
      --color-secondary: ${secondary.hex};
      --color-accent: ${accent.hex};
    `
  };
}

Color Palette Generator

async function createPalette(videoFile: File) {
  const colors = await videoIntel.extractColors(videoFile, {
    count: 8,
    quality: 'best'
  });
  
  // Generate palette UI
  const palette = document.createElement('div');
  palette.className = 'color-palette';
  
  colors.forEach(color => {
    const swatch = document.createElement('div');
    swatch.className = 'color-swatch';
    swatch.style.backgroundColor = color.hex;
    swatch.style.width = `${color.percentage}%`;
    swatch.title = `${color.hex} - ${color.percentage}%`;
    
    // Add color info
    const info = document.createElement('div');
    info.innerHTML = `
      <strong>${color.hex}</strong>
      <span>RGB(${color.rgb.join(', ')})</span>
      <span>HSL(${color.hsl.join(', ')})</span>
      <span>${color.percentage}%</span>
    `;
    
    swatch.appendChild(info);
    palette.appendChild(swatch);
  });
  
  return palette;
}

Video Mood Analysis

async function analyzeVideoMood(videoFile: File) {
  const colors = await videoIntel.extractColors(videoFile, {
    count: 5,
    quality: 'balanced'
  });
  
  // Analyze color characteristics
  const avgBrightness = colors.reduce((sum, c) => 
    sum + (c.rgb[0] + c.rgb[1] + c.rgb[2]) / 3, 0
  ) / colors.length;
  
  const avgSaturation = colors.reduce((sum, c) => 
    sum + c.hsl[1], 0
  ) / colors.length;
  
  // Determine mood
  let mood = 'neutral';
  if (avgBrightness > 180 && avgSaturation > 50) {
    mood = 'vibrant';  // Bright and saturated
  } else if (avgBrightness < 100) {
    mood = 'dark';     // Low brightness
  } else if (avgSaturation < 20) {
    mood = 'muted';    // Low saturation
  } else if (avgBrightness > 150) {
    mood = 'light';    // High brightness
  }
  
  return {
    mood,
    brightness: avgBrightness,
    saturation: avgSaturation,
    colors: colors.map(c => c.hex),
  };
}

Gradient Generation

async function createGradient(videoFile: File) {
  const colors = await videoIntel.extractColors(videoFile, {
    count: 5,
    quality: 'balanced'
  });
  
  // Sort by brightness for smooth gradient
  const sorted = [...colors].sort((a, b) => {
    const brightnessA = (a.rgb[0] + a.rgb[1] + a.rgb[2]) / 3;
    const brightnessB = (b.rgb[0] + b.rgb[1] + b.rgb[2]) / 3;
    return brightnessA - brightnessB;
  });
  
  // Create CSS gradient
  const stops = sorted.map((color, i) => 
    `${color.hex} ${(i / (sorted.length - 1)) * 100}%`
  ).join(', ');
  
  return {
    linear: `linear-gradient(to right, ${stops})`,
    radial: `radial-gradient(circle, ${stops})`,
    colors: sorted.map(c => c.hex),
  };
}

🚀 Next Steps