Paletteyes, Semantic Colour Palette Generator

1. Context

Paletteyes began as an experiment in semantic colour generation, turning natural language into coherent colour palettes.

It evolved into a full system:

• A PyTorch transformer
• Custom colour distance metrics
• A refined RAG pipeline
• A frontend tool to extract palettes from images
• A publishable Python package (Pyletteyes)

The goal was not AI art. It was a system for reasoning about colour.

2. The Problem

Colour palettes are both:

• structured (mathematically definable), and
• subjective (semantic, emotional, cultural).

Users want to say:

• “Coastal sunset”
• “Energetic sci-fi neon”
• “Soft vintage sepia”

…and instantly get a palette that feels right.

The challenge:

How do you turn ambiguous language into specific, coherent colour sets?

3. Constraints

• The transformer originally output only 1 palette per query
• Risk of generating strange or “unusable” results
• Needed multiple interpretable palettes
• Needed to run on limited compute (GCP serverless)
• Language meanings are fuzzy, and mapping them to RGB is non-trivial
• Must handle both image extraction and semantic generation

4. The System I Designed

A hybrid system combining ML, RAG, clustering, and colour science.

Components

• Pyletteyes PyPI package: colour classes and palette objects
• Palette extractor: modified K-Means with preprocessing
• Semantic palette generator: transformer and custom loss
• RAG retriever: returns top 5 closest palettes from DB
• Flask API: image upload and text query endpoints
• Frontend UI: palette cards, download, colour codes
• Training DB: curated palette dataset for semantic similarity

How It Works

1. User provides a text prompt OR an image.
2. If text:
   • Transformer generates an embedding.
   • RAG module retrieves the 5 closest palettes from dataset.
   • These are returned as the “semantic matches.”
3. If image:
   • Palette extractor runs modified k-means and generates a palette.
4. All palettes are represented by Palette objects (class-based abstraction).
5. System returns structured colour values + metadata.

The move from pure model output to a hybrid RAG system improved:

• reliability
• interpretability
• user trust

5. Before vs After

Before	After
Transformer outputs a single palette	RAG returns top 5 best-fit palettes
High risk of odd outputs	Curated, realistic results
No abstraction layer	Colour & Palette classes standardize everything
Only generation	Full pipeline for both generation & extraction
ML-only	Hybrid ML + retrieval + colour harmony

6. Stack & Architecture

• PyTorch for transformer embeddings
• Scikit-Learn for clustering (k-means)
• Custom loss functions for palette smoothness
• RAG pipeline pulling from palette DB
• Flask backend and frontend
• GCP Cloud Run for serverless deployment
• Pyletteyes PIP installable package for internal consistency

Architecture theme: Hybrid intelligence: ML, retrieval, and deterministic tools.

7. Impact

• Achieved expressive, usable palette generation
• Reliable outputs for creative or brand use
• Provided a polished, hosted tool + an installable Python package
• Demonstrated your ability to combine:
  • ML
  • Colour theory
  • Retrieval systems
  • Software UX
  • Deployment pipelines

It’s a perfect example of your “systems, not point models” identity.

8. What I’d Build Next (v2)

• Fine-tuned transformer with more nuanced semantic space