> We found that the model's modulation weights (~40% of the total parameters) could be pruned and replaced with a functionally equivalent lookup table, dramatically shrinking the memory footprint with no loss in output quality.
Is this a common approach to reducing weights with "no loss in output quality", assuming this is true? Seems almost too simple to work. If this is doable, would this be applicable to LLMs as well?
Neat with native frame-to-frame generation, but wonder how easy it is to "link" together clips at the intersection, typically the models kind of lose the "momentum" across these stiches, being able to merge things with frame-to-frame between clips might help with this it feels like.
"Modulation weights" here refers to weights used to adjust layer normalization depending on the task (adaLN). General-purpose LLMs generally don't have those in the first place.
Also begs the question whether this is applicable for high-throughput applications on FPGAs, which are to my novice mind basically LUTs, right?
I remember a paper which was posted on HN a few weeks ago where somebody implemented KAN networks in FPGAs, since those can readily be approximated as LUTs.
I am particularly curious how multimodal models will work with types of knowledge that are inherently non-text. For example, SOTA LLMs really suck at electronics, especially analog electronics.
Is MiniMax H3 capable of logical / technical reasoning, or is it purely art oriented?
FWIW on a 6000 Pro it takes 68 seconds for 10 seconds 480p video, cold startup, same demo workflow as you used. Set "megapixels" to 2.0 (1920x1088) and same video seems to take 5+ minutes, not sure if everything is right/correct at the moment.
As far as I can tell, the current ComfyUI nodes don't even do compilation, and I haven't looked into what attention mechanism they're using, but I'm sure with time these durations will come down even more.
> The result gives a total memory footprint reduced by 66%, from 123.6 GB in full precision to 42.5 GB with the smallest models variants. Combining this with our dynamic VRAM offloading enables a next-generation 2K video model to run locally on a GPU like the RTX 3060.
Pretty cool.
But assuming you have a 16GB 3060, how long would it take to generate a 15 second clip?
The mouse render is surprisingly good. Several of those clips stood out to be a pretty big leap in terms of current SOTA models.
The only one that looks "off" is the beverage ad video during the can opening clip, it still has that "AI smoothening" effect. Good thing this can be done pretty well using traditional rendering.
I feel like for a good while now we'll transition into a process that uses traditional "close-up" rendering/shots + AI generated wide-shots or quick cuts.
Exciting, but also troubling. This being open-weights is a massive win for the community though.
I remember reading a report where people running AI-model instagram account were using insanely long and detailed prompts about the setting, lighting, makeup, pose, disposition, clothing, etc. about their models. Presumably with some reference image of the face / body to remain consistent across images.
It‘s not clear to me whether a sufficiently detailed prompt can generate actually interesting video with a natural ”texture” (for lack of a better word).
I've said it before and I'll say it again, human directors are still valuable, as they use AI video editing tools to generate the shots they want and put them together in a cohesive way. Previously they might've used film and actors but if they can just prompt the AI (or create workflows as seen with ComfyUI) then they arrange them together just like how an EDM producer doesn't actually play the instruments but instead the creativity is in the arrangement.
I suspect it'll be quite a while until AI gets a good enough aesthetic sense to do this, as even with static HTML websites humans can easily see that it's AI slop.
"Regions such as the EU, UK, South Korea, and the US are currently developing or enforcing AI-related regulations that may have specific implications for generative video models"
The example video just looks like the highly produced art (TV, commercials, games) other people have created. I find it impossible to believe this wasn't trained on other people's work, and there is no protection for it. Terribly sad. A lack of original thinking is coming.
As with the arts, 99.9% of people can't use these models to express vision, get attention, or achieve distribution.
The game is the same as it has always been. You still need hard work, taste, something important to say, the ability to articulate it, good timing, and luck.
Nothing has changed. We can just build faster.
What this does enable is for more to be created that caters to a wider variety of interests. It disrupts existing structures of capital allocation, production, and distribution and gives new players a chance to reshape the game.
The bar will rise and people will still be running at the same pace on the treadmill. There will be more to see, but less time to see it.
> We found that the model's modulation weights (~40% of the total parameters) could be pruned and replaced with a functionally equivalent lookup table, dramatically shrinking the memory footprint with no loss in output quality.
Is this a common approach to reducing weights with "no loss in output quality", assuming this is true? Seems almost too simple to work. If this is doable, would this be applicable to LLMs as well?
Neat with native frame-to-frame generation, but wonder how easy it is to "link" together clips at the intersection, typically the models kind of lose the "momentum" across these stiches, being able to merge things with frame-to-frame between clips might help with this it feels like.
"Modulation weights" here refers to weights used to adjust layer normalization depending on the task (adaLN). General-purpose LLMs generally don't have those in the first place.
Also begs the question whether this is applicable for high-throughput applications on FPGAs, which are to my novice mind basically LUTs, right?
I remember a paper which was posted on HN a few weeks ago where somebody implemented KAN networks in FPGAs, since those can readily be approximated as LUTs.
Im running this on my 4070ti super (16 gb vram), and it takes 10 minutes for a 10-seconds 480p video. but the results are spectacular.
I am particularly curious how multimodal models will work with types of knowledge that are inherently non-text. For example, SOTA LLMs really suck at electronics, especially analog electronics.
Is MiniMax H3 capable of logical / technical reasoning, or is it purely art oriented?
FWIW on a 5080 16GB it takes 3 minutes for 10 seconds 480p video (the mouse video workflow with length changed from 5 seconds to 10 seconds)
FWIW on a 6000 Pro it takes 68 seconds for 10 seconds 480p video, cold startup, same demo workflow as you used. Set "megapixels" to 2.0 (1920x1088) and same video seems to take 5+ minutes, not sure if everything is right/correct at the moment.
As far as I can tell, the current ComfyUI nodes don't even do compilation, and I haven't looked into what attention mechanism they're using, but I'm sure with time these durations will come down even more.
If you wouldn't mind sharing, what's your Comfy workflow for this? I have the same video card setup and would like to give it a shot.
just the default one in the link for image-to-video
1 minute for a second of footage, that is awesome! Thanks for sharing.
> The result gives a total memory footprint reduced by 66%, from 123.6 GB in full precision to 42.5 GB with the smallest models variants. Combining this with our dynamic VRAM offloading enables a next-generation 2K video model to run locally on a GPU like the RTX 3060.
Pretty cool.
But assuming you have a 16GB 3060, how long would it take to generate a 15 second clip?
The mouse render is surprisingly good. Several of those clips stood out to be a pretty big leap in terms of current SOTA models.
The only one that looks "off" is the beverage ad video during the can opening clip, it still has that "AI smoothening" effect. Good thing this can be done pretty well using traditional rendering.
I feel like for a good while now we'll transition into a process that uses traditional "close-up" rendering/shots + AI generated wide-shots or quick cuts.
Exciting, but also troubling. This being open-weights is a massive win for the community though.
Learned something, upvoted
Any tutorial for me to learn how to use
On one hand: impressive. On the other hands aesthetically it all looks painfully bland and generic.
I agree. But is that the model or the prompt?
I remember reading a report where people running AI-model instagram account were using insanely long and detailed prompts about the setting, lighting, makeup, pose, disposition, clothing, etc. about their models. Presumably with some reference image of the face / body to remain consistent across images.
It‘s not clear to me whether a sufficiently detailed prompt can generate actually interesting video with a natural ”texture” (for lack of a better word).
This, today, is the absolute worse this model will ever be. Chill.
I've said it before and I'll say it again, human directors are still valuable, as they use AI video editing tools to generate the shots they want and put them together in a cohesive way. Previously they might've used film and actors but if they can just prompt the AI (or create workflows as seen with ComfyUI) then they arrange them together just like how an EDM producer doesn't actually play the instruments but instead the creativity is in the arrangement.
I suspect it'll be quite a while until AI gets a good enough aesthetic sense to do this, as even with static HTML websites humans can easily see that it's AI slop.
I saw the samples people have posted. Immediately deleted LTX2 and WAN folders. Those are completely worthless now.
There is some debate on the license for those in the US, UK, EU, plus… no comment other than whew those samples though!
"Regions such as the EU, UK, South Korea, and the US are currently developing or enforcing AI-related regulations that may have specific implications for generative video models"
You just have to pinkie promise you won't make disney mad and they will send you a licence https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/docs/Q...
I do work animations for fun and internal use only (mostly jokes). I MAY reach out to them.
Hollywood and the film industry on red alert. Too bad.
This is AGI.
The example video just looks like the highly produced art (TV, commercials, games) other people have created. I find it impossible to believe this wasn't trained on other people's work, and there is no protection for it. Terribly sad. A lack of original thinking is coming.
> A lack of original thinking is coming.
As with the arts, 99.9% of people can't use these models to express vision, get attention, or achieve distribution.
The game is the same as it has always been. You still need hard work, taste, something important to say, the ability to articulate it, good timing, and luck.
Nothing has changed. We can just build faster.
What this does enable is for more to be created that caters to a wider variety of interests. It disrupts existing structures of capital allocation, production, and distribution and gives new players a chance to reshape the game.
The bar will rise and people will still be running at the same pace on the treadmill. There will be more to see, but less time to see it.
Can't wait for netflix to be topped on unimaginative nonsense meant to have in the background while you scroll your phone.
Unlimited slop "content" to fill decomposed brain shaped vessels, the future is bright!
Unfortunately Hollywood been feeding us human generated slop for a long time already. Transition won't be hard.