altmake

Projection Mapping With AI: My End-to-End Workflow (2025)

A two-story house at night washed in red light, with a tall shadowy figure projected across the upper story and a lit living-room scene projected onto a screen over the garage door.
Contents
  1. tl;dr — The Key Steps
  2. Key Terms
  3. Components
    1. Software & Services
    2. Projectors
    3. Mounting & Rigging
  4. Planning (theme → profile)
    1. Ideate with ChatGPT
    2. Collect Inspiration Images
    3. Create a Theme Profile (YAML) with Gemini
  5. Designing the Scene
    1. Capture the Structure Reference Image
    2. Generate the Scene Base Image
    3. Generate the Alignment Image
  6. (Optional) AI Video Experiments
  7. Projection Alignment
  8. Place, Align, Lock in LumaMap
  9. Playback Improvements (beyond LumaMap)
    1. Raspberry Pi setup (mpv)
  10. Important Notes
  11. Demo

A practical pipeline for capturing, aligning, generating, and projecting AI-assisted visuals — ending with a Raspberry Pi playback loop.

tl;dr — The Key Steps

Skip to the end of the article for a video demo.

  1. Capture Images — Create a Structure Reference image (photo taken during daylight) and Projection Boundary image (photo taken at night, white frame to identify projection area).
  2. Generate Images — Scene Base Image (styled to theme) and Alignment Image (B/W highlighting key edges of structure to aid in projection alignment).
  3. Align & Lock Projection — Place Alignment Image on facade and Lock transform in LumaMap.
  4. Animate the Scene — In Veo 3 (if stable) or in CapCut (composite animations over Scene Base Image).
  5. Export & Play — Final video played on Raspberry Pi + mpv loop.

Key Terms

  • Structure Reference Image (aka “Reference Image”) — Daylight photo taken from the projector location. Provides geometry and visual context to guide AI.
  • Projection Boundary Image (aka “Canvas Capture”, “White-frame photo”) — Night photo with a full-white projection. Clearly shows the exact area the projector covers.
  • Alignment Image (aka “Structural Outline”) — High-contrast black-and-white image (generated via Stable Diffusion Control Structure) emphasizing edges of walls, roofline, windows, and doors. Used to verify and fine-tune alignment on the real surface.
  • Scene Base Image — The stylized, on-brand visual we’ll animate (directly in an AI video model or later in an editor like CapCut).
  • Inspiration Images — A small set of images that define the theme, palette, motifs, and texture language. Used to build a reusable “profile” for Stable Diffusion Control Structure (SDCS).

Components

Affiliate links: Links to Amazon are affiliate links. As an Amazon Associate I earn from qualifying purchases, at no extra cost to you.

Software & Services

  • ChatGPT — concept and technical planning
  • Google Images — inspiration assets
  • Stable Diffusion Control Structure (SDCS) — image generation and structural control
  • Google Gemini — generates theme profiles (YAML) for SDCS
  • sdcs-wrapper — my CLI to apply SDCS profiles to input images
  • LumaMap — simple alignment & playback for stills (good for quick tests)
  • CapCut (or any NLE) — motion layers and finishing

Projectors

Mounting & Rigging

If you want a semi-permanent mounting solution, a chain-link fence post works great. I put mine two feet into the ground with a PVC pipe as a ground sleeve. This means I can take the pole out while I’m not using the setup. At the top of the pole I used a chain-link floor flange. This is normally used to mount the pole to the ground but I’m using it to transition from the pole to the projector mount. I have an older projector mounted to the pole so I needed an older style projector mount.

A lot of newer projectors use a 1/4”-20 thread screw which means you could get away with using a t-nut and a bolt to connect the floor flange to the projector.

An Epson projector on a low-profile mount, bolted through a wooden plate to a floor flange on top of a galvanized fence post, against a cloudy sky.

Mounting for projector on chain-link fence pole

Tip: Before buying hardware, estimate your throw distance vs. image width using the projector’s throw ratio so you know you can fill the facade you want.

Planning (theme → profile)

Ideate with ChatGPT

“Give me 5 Halloween themes that are relevant this year. Focus on motifs that read well on light stone and stucco.”

Collect Inspiration Images

Build a small mood board: textures, motifs, characters, architectural treatments that match your facade. Pinterest works well for this if you are collaborating with other people.

A mood board with five columns of reference images labeled Scary Carnival, Rick and Morty, Beetlejuice, Wicked, and Stranger Things.

Mood Board — Halloween 2025

Create a Theme Profile (YAML) with Gemini

Upload a few inspiration images to Gemini and ask it to output a single YAML profile for SDCS (prompt, negative prompt, control strength, style preset, output format).

  • Keep the prompt verbose and concrete (composition, materials, lighting, mood).
  • Keep the negative prompt comprehensive (block low-quality traits and off-theme elements).
  • Default control_strength: 0.8, style_preset: cinematic|photographic, output_format: jpeg.

Gemini Prompt for Theme Profile:

You are an AI assistant, and I need your help to create highly detailed configuration profiles for a Stable Diffusion API.
Your task is to analyze the reference image(s) I provide and generate a single, complete YAML configuration file that can be used to create a new, very similar image. You must follow these specific formatting rules for all of your responses.
Your output must be only a YAML configuration file, structured exactly like this template:

```yaml
# Profile configuration for [A brief, descriptive title of the scene]
prompt: "A single, highly-detailed, photorealistic, and extremely verbose string. This description must be meticulous.
You must describe all key subjects, the position of every major element, the environment, all textures (e.g., 'vintage floral wallpaper', 'worn plaid fabric', 'wet biomechanical vines'), the lighting (e.g., 'primary light from multi-colored string lights', 'eerie blue backlight', 'intense crimson red glow'), the atmosphere ('nostalgic', 'eerie', 'ominous'), and the overall artistic style ('cinematic', 'horror', 'Stranger Things aesthetic').
Be as specific as possible to leave no room for misinterpretation."
negative_prompt: "A single string of comma-separated keywords. This must always include a base set of 'blurry, low quality, noisy, grainy, cartoon, anime, illustration, painting, sketch, ugly, deformed, watermark, text, signature'. You must also add specific negative keywords that are the *opposite* of the prompt's main subjects and mood. For example, if the prompt is 'dark, night, horror', the negative prompt must also include 'bright, daytime, cheerful, happy, clean, modern'."
control_strength: 0.8
style_preset: "cinematic"
output_format: "jpeg"
```

Key Instructions to Always Follow:
* prompt: The description must be as long and verbose as possible. It is the most important part.
* negative_prompt: Must be a comprehensive list to block common errors and unwanted themes.
* control_strength: You must always include this field. The default value must be 0.8. Do not change it unless I explicitly ask you to.
* style_preset: You must always include this field. Default to cinematic or photographic, whichever best fits the image.
* output_format: You must always include this field, and its value must always be jpeg.
My first request is to describe the image(s) I have attached to this message. Please analyze them and generate the YAML profile now.

Designing the Scene

Capture the Structure Reference Image

  • Daylight or golden hour.
  • Shoot from the same location you’ll place the projector.
  • Capture more of the building than the final projection — extra context helps AI.

A daylight photo of a two-story stone and siding house with the sky masked out in solid blue and the garage door masked in black.

Structure Reference Image — Masking is optional but improves results

Generate the Scene Base Image

  • Use sdcs-wrapper with your theme profile derived from Gemini.
  • This becomes the hero still you’ll animate (either in-AI or later in CapCut).

An AI-generated version of the house as a weathered Victorian farmhouse, bathed in red light under a dark red sky.

Scene Base Image — Themed as Vecna’s House from Stranger Things

Generate the Alignment Image

  • Use a profile that forces bold, high-contrast edges (no color).

Example profile for Alignment Image:

# Profile configuration for creating alignment image
prompt: "Transform this image into bold, high-contrast black and white with thick outlines, clean architectural edges (walls, roofline, windows, doors), flat white fill, no color, and minimal shading. Technical schematic feel."
negative_prompt: "color, gradients, shading, noise, blur, low quality, text, logo, watermark, people, figures, background clutter"
control_strength: 0.95
style_preset: "photographic"
output_format: "jpeg"

A black line drawing of the house on a flat background, tracing the roofline, windows, doors, garage, and stonework.

Alignment Image — Emphasizing edges

Why two images? The Alignment Image gives you crisp edges to match on the facade; the Scene Base Image looks good once mapped. You lock the geometry once, then swap content without re-doing alignment.

(Optional) AI Video Experiments

Veo 3 can generate animated shots, but it may drift (camera moves) even when instructed not to. Preface your prompts with:

STATIC SHOT // NO CAMERA MOVEMENT
The camera must be completely fixed and stationary. NO pan, NO zoom, NO dolly, NO tilt, and NO change in perspective. The camera is fixed to a pole in concrete.
STATIC ARCHITECTURE
All architectural elements and structures must remain completely static and unchanged throughout the video.
PERMITTED MOVEMENT
Characters, objects, and environmental/atmospheric elements (lighting, shadows, weather, fog) may move.
---

If the AI video model won’t respect a locked camera, animate in CapCut instead:

  • Use your Scene Base Image as the bottom layer.
  • Render separate foreground animations on green or transparent background and key/compose them over the base.
  • Mask (black-out) any areas you never want projected.

Export from the editor at your projector resolution (or higher).

The CapCut editor with the red Scene Base Image in the preview and several stacked animation tracks on the timeline.

CapCut — Asset layering

Projection Alignment

  1. Wait for darkness (earlier if your projector is very bright).
  2. Reset projector settings, max brightness, no keystone.
  3. Send a full-white frame (LumaMap has a tool for this).
  4. Physically position the projector so the white area neatly covers the target surface.
  5. Mount the projector (tripod, pipe, ceiling mount, etc.).
  6. Capture the Projection Boundary Image: a photo of the illuminated area from the projector location.

The front of the house at night lit by a full-white projection, with a gray box covering part of the image.

Projection Boundary Image (gray area added for privacy)

Place, Align, Lock in LumaMap

  1. Load the Alignment Image into LumaMap.
  2. Use the Projection Boundary Image as a visual reference to roughly place the Alignment Image on canvas.
  3. Fine-tune alignment by eye, using edges (roof, windows, door frames) as hard references. Tip: Add a grid overlay or flash white/black to check edge bleed.
  4. Lock the alignment once edges are crisp and stable.
  5. Swap in your Scene Base Image or animated video and verify fit. The need for minor adjustments is normal; do not keystone on the projector — adjust the mapping instead.
  6. Save your alignment/project.

LumaMap showing the red scene placed on the canvas with corner handles for adjusting the mapping, at a 1920 by 1200 output size.

LumaMap Scene Alignment

Playback Improvements (beyond LumaMap)

LumaMap is perfect for stills and quick tests, but long video loops can stutter on some machines. For rock-solid playback:

Export your video from LumaMap.

Note on exclusion zones: LumaMap includes them in image exports, not video. Workaround: Export your video normally. Export a blank image that does contain the exclusions. In CapCut, stack the blank image on top of your video and use it as a mask layer. (There are lossless/ffmpeg alternatives, but this is the fastest.)

Raspberry Pi setup (mpv)

  1. Copy your video to /home/pi/videos/loop.mp4.

  2. Test playback:

    mpv --fs --no-osd --really-quiet --hwdec=auto --loop-file /home/pi/videos/loop.mp4

  3. If running over SSH:

    sudo -u pi DISPLAY=:0 mpv --fs --no-osd --really-quiet --hwdec=auto --loop-file /home/pi/videos/loop.mp4

  4. Create a systemd service:

    sudo nano /etc/systemd/system/videoloop.service

    With the following contents:

    [Unit]
Description=Seamless Video Loop (MPV)
After=multi-user.target

[Service]
User=pi
ExecStart=/usr/bin/mpv --fs --no-osd --really-quiet --hwdec=auto --loop-file /home/pi/videos/loop.mp4
Restart=always
StandardOutput=null
StandardError=null

[Install]
WantedBy=multi-user.target

  5. Enable the service so it starts at boot:

    sudo systemctl daemon-reload
sudo systemctl enable videoloop.service

  6. Reboot to confirm it loads as expected.

Note: If you are having performance issues, try updating /boot/firmware/config.txt (/boot/config.txt on Raspberry Pi OS releases before Bookworm) with the following (remember to make a backup first):

hdmi_force_hotplug=1
hdmi_group=1
# 1080p60
hdmi_mode=16
hdmi_drive=2
disable_overscan=1
gpu_mem=256

Note: The hdmi_* settings above are options for the legacy display driver. Raspberry Pi OS Bookworm and later use the KMS driver by default, which ignores most of them, so they may have no effect on a current install.

Note: For older Raspberry Pis, consider videolooper.de — lean and purpose-built but doesn’t work with most recent versions of Raspberry Pi.

Important Notes

  • Don’t keystone on the projector. Fix geometry in the mapping tool; hardware keystone compounds errors.
  • Tripod discipline. Any position change between white-frame capture and alignment will cost time.
  • Bright, light-biased art wins. Dark art disappears outdoors; stay high-key unless your surface is very bright.
  • Match projector resolution. Export at exact output res to avoid resampling blur.
  • Mask early. Black out areas you never want projected (windows, neighbor walls, trees).
  • Protect your gear. If you don’t have an outdoor projector you will want to build or buy a projector enclosure.

Demo