Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT)
I got tired of rewriting the same dataset-prep scripts before every LoRA/Dreambooth run — resize, dedupe, toss the blurry ones, caption, split — so I cleaned mine up and open-sourced it.
Proof1 community thread
Key points
- It scores each image on classical-CV signals (sharpness, exposure/clipping, contrast, color cast, noise, composition), flags the weak ones, removes exact + near-duplicates, and exports a Kohya-style folder with caption sidecars.
- Near-duplicates get grouped into the same train/val/test split so they can’t leak across.
- Pure NumPy/Pillow, no OpenCV, runs from a CLI or as a library.
- Would love feedback on the plugin API and which verticals are worth building next.
Sources (1)
- [1]Open-sourced my image-dataset prep tool — quality scoring, dedupe, leak-safe splits, Kohya export (pure-Python, MIT)r/StableDiffusion (top, day) · Oct 12, 05:46 AM
I got tired of rewriting the same dataset-prep scripts before every LoRA/Dreambooth run — resize, dedupe, toss the blurry ones, caption, split — so I cleaned mine up and open-sourced it.
It scores each image on classical-CV signals (sharpness, exposure/clipping, contrast, color cast, noise, composition), flags the weak ones, removes exact + near-duplicates, and exports a Kohya-style folder with caption sidecars.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 12, 2026Inside the AI Economy: What Stripe's Data Reveals — Nilofer Rajpurkar, Stripe
- Oct 12, 2026epGen: open LoRA for LTX-2.5 that gives every sound in the video its own audio track (generation + separation, weights on HF)
- Oct 11, 2026MCP - Qwen2.1 Turbo + Kandinsky Lite Video Gen on 8GB VRAM
- Oct 11, 2026SonarQube + OpenAI: Agentic Development — Killian Carlsen-Phelan, Sonar
- Oct 11, 2026I am building a web application to transform images into Gaussian Splats
- Oct 9, 2026openai/codex rust-v0.162.1: 0.162.1
