"Does the AI version keep the same camera moves as the original?" is one of the first things people ask about photo-to-video meme generators. The Tiny Desk Outstanding studio makes a strong claim here. Its about page says a photo "does not change the length, the camera, or the song already in the clip", and every scene page calls the source "a 15-second vertical clip with the original soundtrack".
That's a testable claim. The home page has eight public sample files: the "Whole Band reference video" and seven generated or alternate cuts. So I measured two things in each one: where the hard cuts are and how much the whole picture moves between frames. I used NumPy and ffmpeg only, no OpenCV. Everything below was run on the box on November 14, 2026 (Asia/Shanghai), and the outputs are pasted as they came out.
Why phase correlation
When a camera pans, most of the frame shifts by the same amount. When you shift an image, its Fourier transform keeps the same magnitude and only the phase changes. If you divide the cross-power spectrum of two frames by its own magnitude and take the inverse FFT, you get one sharp peak at the shift. That peak's height also tells you something useful:
- high peak (0.2 to 0.98): same shot, and the peak's location is the global shift in pixels
- very low peak (about 0.03): the two frames don't share structure, which means it's a hard cut
So the same 10 lines give you a pan/tilt estimate and a cut detector.
The script
camera_motion.py decodes the clip at 12 fps, 144×256 grayscale (aspect forced to 9:16 so clips of different sizes compare), and runs phase correlation between consecutive frames:
#!/usr/bin/env python3
"""Global camera motion + cut map for a short vertical clip (NumPy + ffmpeg only).
usage: python3 camera_motion.py clip.mp4 [more.mp4 ...] [--fps 12] [--csv out.csv]
"""
import argparse, csv, subprocess, sys
import numpy as np
W, H = 144, 256 # analysis size (9:16); small is enough for global motion
def frames(path, fps):
cmd = ["ffmpeg", "-v", "error", "-i", path, "-vf",
f"fps={fps},scale={W}:{H}:force_original_aspect_ratio=disable,format=gray",
"-f", "rawvideo", "-"]
raw = subprocess.run(cmd, capture_output=True, check=True).stdout
return np.frombuffer(raw, np.uint8).reshape(-1, H, W).astype(np.float32)
WIN = np.outer(np.hanning(H), np.hanning(W)).astype(np.float32)
def phase_corr(a, b):
"""Return (dx, dy, peak) so that b is roughly a shifted by (dx, dy)."""
A = np.fft.fft2((a - a.mean()) * WIN)
B = np.fft.fft2((b - b.mean()) * WIN)
R = np.conj(A) * B
R /= np.abs(R) + 1e-9
r = np.fft.ifft2(R).real
y, x = np.unravel_index(np.argmax(r), r.shape)
if y > H // 2: y -= H
if x > W // 2: x -= W
return float(x), float(y), float(r.max())
def analyse(path, fps, cut_peak=0.06):
f = frames(path, fps)
rows = []
for i in range(1, len(f)):
dx, dy, peak = phase_corr(f[i - 1], f[i])
rows.append({"t": round(i / fps, 3), "dx": dx, "dy": dy,
"peak": round(peak, 3), "cut": peak < cut_peak})
return rows
def summary(rows):
cuts = [r["t"] for r in rows if r["cut"]]
# merge cut flags closer than 0.25 s (one cut can flag two pairs)
merged = []
for t in cuts:
if not merged or t - merged[-1] > 0.25:
merged.append(t)
still = [r for r in rows if not r["cut"]]
pan = sum(abs(r["dx"]) for r in still)
tilt = sum(abs(r["dy"]) for r in still)
moving = sum(1 for r in still if abs(r["dx"]) + abs(r["dy"]) >= 1) / max(len(still), 1)
return merged, pan, tilt, moving
if __name__ == "__main__":
ap = argparse.ArgumentParser()
ap.add_argument("clips", nargs="+")
ap.add_argument("--fps", type=float, default=12)
ap.add_argument("--csv")
a = ap.parse_args()
out = []
print(f"{'clip':<14}{'cuts (s)':<34}{'pan px':>8}{'tilt px':>8}{'moving':>8}")
for c in a.clips:
rows = analyse(c, a.fps)
cuts, pan, tilt, moving = summary(rows)
name = c.rsplit("/", 1)[-1][:12]
print(f"{name:<14}{', '.join(f'{t:.2f}' for t in cuts):<34}{pan:>8.0f}{tilt:>8.0f}{moving:>8.0%}")
out += [{"clip": name, **r} for r in rows]
if a.csv:
with open(a.csv, "w", newline="") as fh:
w = csv.DictWriter(fh, fieldnames=list(out[0]))
w.writeheader(); w.writerows(out)
Does it measure what it says? A self-test
Before pointing it at real clips, I cut two windows out of the same blurred noise texture with a known offset, and also compared the frame with unrelated noise:
import numpy as np, camera_motion as cm
# smooth random texture, then cut two windows offset by a known shift
rng = np.random.default_rng(0)
big = rng.random((cm.H + 40, cm.W + 40)).astype(np.float32)
for _ in range(3): # cheap blur so it looks like an image, not noise
big = (big + np.roll(big, 1, 0) + np.roll(big, 1, 1) + np.roll(big, -1, 0) + np.roll(big, -1, 1)) / 5
a = big[20:20 + cm.H, 20:20 + cm.W]
for dx, dy in [(3, 0), (-2, 5), (7, -4)]:
b = big[20 - dy:20 - dy + cm.H, 20 - dx:20 - dx + cm.W]
x, y, peak = cm.phase_corr(a, b)
print("true", (dx, dy), "measured", (x, y), "peak", round(peak, 3))
other = rng.random((cm.H, cm.W)).astype(np.float32)
print("unrelated frame peak", round(cm.phase_corr(a, other)[2], 3))
true (3, 0) measured (3.0, 0.0) peak 0.977
true (-2, 5) measured (-2.0, 5.0) peak 0.972
true (7, -4) measured (7.0, -4.0) peak 0.936
unrelated frame peak 0.03
The shift comes back exactly, matching frames peak at 0.94 to 0.98, and unrelated frames peak at 0.03. In the real clips, cuts landed at 0.027 to 0.045 and ordinary frames had a median peak of 0.23 to 0.31, so a 0.06 threshold sits in the gap.
Results on the eight public samples
python3 camera_motion.py samples/*.mp4 --csv motion.csv
clip cuts (s) pan px tilt px moving
0299078c3f43 5.33, 11.67, 15.00 152 160 57%
632c0129d45a 5.33, 11.67, 15.00 248 212 71%
8256cd873b7e 5.25, 11.42, 14.75 225 203 60%
875d2458181e 102 60 94%
947f5eb2a447 1.50, 7.00, 13.67, 17.33 257 258 59%
db033545db00 1.58, 4.75, 7.83 160 145 64%
db09b1c54359 5.17, 11.17 260 164 74%
f7ff52686113 1.50, 6.92, 10.08, 13.17, 16.50 273 274 62%
(pan px / tilt px add up the absolute per-frame shift at 144×256 within shots, and moving is the share of frame pairs that shifted by at least 1 px.) db09b1c54359 is the reference. 0299…, 632c… and 8256… are the three full recasts (632c… is the cat version the home page also shows). The others are alternate or partial cuts the site labels as such (a 10-second take, a longer cut, a 4-second close-up).
To line the recasts up against the reference shot by shot, shot_compare.py splits each clip at its cuts and reports the mean shift per frame in each shot:
#!/usr/bin/env python3
"""Compare each clip's shot list and per-shot camera motion with a reference.
usage: python3 shot_compare.py reference.mp4 clip.mp4 [clip.mp4 ...]
"""
import sys
import numpy as np
from camera_motion import analyse, summary
FPS = 12
def shots(path):
rows = analyse(path, FPS)
cuts, *_ = summary(rows)
end = round((len(rows) + 1) / FPS, 2)
edges = [0.0] + cuts + [end]
out = []
for a, b in zip(edges, edges[1:]):
seg = [r for r in rows if a < r["t"] < b and not r["cut"]]
mag = np.mean([np.hypot(r["dx"], r["dy"]) for r in seg]) if seg else 0.0
out.append((a, b, mag))
return out
def fmt(s):
return " | ".join(f"{a:5.2f}-{b:5.2f}s {m:4.2f}px/f" for a, b, m in s)
ref = shots(sys.argv[1])
print("reference ", fmt(ref))
for clip in sys.argv[2:]:
s = shots(clip)
name = clip.rsplit("/", 1)[-1][:10]
print(f"{name:<10} ", fmt(s))
n = min(len(ref), len(s))
drift = [s[i][1] - ref[i][1] for i in range(n - 1)] # cut-time differences
print(f"{'':<11} shots {len(s)} vs {len(ref)}; cut drift vs reference: "
+ ", ".join(f"{d:+.2f}s" for d in drift))
python3 shot_compare.py samples/db09b1c5…mp4 samples/0299078c…mp4 samples/632c0129…mp4 samples/8256cd87…mp4
reference 0.00- 5.17s 0.94px/f | 5.17-11.17s 2.59px/f | 11.17-14.50s 2.58px/f
0299078c3f 0.00- 5.33s 0.22px/f | 5.33-11.67s 2.36px/f | 11.67-15.00s 1.42px/f | 15.00-17.58s 0.41px/f
shots 4 vs 3; cut drift vs reference: +0.17s, +0.50s
632c0129d4 0.00- 5.33s 0.68px/f | 5.33-11.67s 2.91px/f | 11.67-15.00s 2.50px/f | 15.00-17.67s 0.74px/f
shots 4 vs 3; cut drift vs reference: +0.17s, +0.50s
8256cd873b 0.00- 5.25s 0.13px/f | 5.25-11.42s 3.37px/f | 11.42-14.75s 1.87px/f | 14.75-17.42s 0.66px/f
shots 4 vs 3; cut drift vs reference: +0.08s, +0.25s
What the numbers say
- The shot list is kept. The reference has three shots: the desk trio around the red keyboard (0 to 5.17 s), the lead singer's laughing close-up (to 11.17 s) and the sax solo (to 14.5 s). Each full recast has the same three shots in the same order, and the cuts land within +0.08 to +0.50 s of the reference's.
- But the recasts run longer and add a fourth shot. All three full recasts cut back to the desk at 14.75 to 15.0 s and keep going for another 2.6 s, ending at 17.4 to 17.7 s. I checked a frame strip to confirm: it really is a return to the desk trio. So "15 seconds" is the reference, not a promise about every output file.
- The camera's motion profile is similar, not identical. The close-up is the busiest shot everywhere (2.36 to 3.37 px/frame vs 2.59 in the reference), and the desk shot is the calmest. The opening shot is steadier in the recasts (0.13 to 0.68 vs 0.94 px/frame). That fits how reference-video models work: the reference drives the camera path and timing, and the model re-renders each frame, so small handheld wobble isn't copied pixel for pixel.
Caveats: phase correlation measures the single dominant shift, so a laughing head in a close-up counts as "motion" even if the camera is still. It doesn't detect zoom or rotation. Use it to compare clips against each other, not as an absolute camera tracker. A 12 fps analysis can't place a cut more precisely than about 0.08 s.
Why it matters if you edit the clip
If you plan to cut your own clip on the beat, add a slow-motion replay, or line up captions with the laugh, don't hard-code the reference's timestamps. Run camera_motion.py on your file and use its cut list. The shots are in the same order, but a recast can be up to half a second late and about 3 seconds longer.
The scene choices (Just Me for the close-up, Back Row for the two standing singers in the opening shot, Whole Band for all five seats) are compared on the scenes page. Every scene uses the same reference, so these shot timings apply to all three.
Top comments (0)