Huh?
The next piece of software on my quest for the perfect music video. This program allows me to map video keyframes to midi files. Boring, slow, unrythmic videos can be re-mapped to simple rythms, perfect for found-footage video generation.
MIDI Generation
Depending on if the music was home-made, the vicinity to the producer or the production workflow itself one might have a hard time getting usable midi data. In this case, the spurious process of reverse-engineering music begins:
First the track needs to be split up into individual instruments. Open source projects accomplishing this exist. Not wanting to set this up I used this ridiculously undersold conversion tool.
Afterwards the split tracks can be converted to midi by filtering for relevant frequencies and setting thresholds to trigger midi notes. Reaper offers a wide range of control for doing this. Other digital audio workstations are available.
Mechanizm
Next the source videos are imported into mechanizm and rythmic points can be defined.
These "clips" can then be mapped to the MIDI files from above. Here I mapped the kick to increase the position to the next key frame and the snare to do the opposite:
Composition & Post-Processing
After the mapped clips are exported from mechanizm, they can be composited and assembled in a video editor (or Reaper, for that matter). Since editing and compositing videos and audio is an "ESP" (Extremely Solved Problem) I won't go into detail here.
Then we can do whatever else we want. The first video is additionally colorized with a neural network.