Skip to main content

在 Three.js 中将视频用作纹理

如果你想在 Three.js 中将视频作为材质嵌入，你有多种选择。

🌐 If you want to embed a video as a texture in Three.js, you have multiple options.

使用 @remotion/mediav4.0.387​

🌐 Using @remotion/mediav4.0.387

我们建议你以headless模式挂载<Video>标签，并使用onVideoFrame属性在绘制新帧时更新 Three.js 纹理。

🌐 We recommend that you mount a <Video> tag in headless mode and use the onVideoFrame prop to update the Three.js texture whenever a new frame is being drawn.

VideoTexture.tsx
import {useThree} from '@react-three/fiber';
import {Video} from '@remotion/media';
import {ThreeCanvas} from '@remotion/three';
import React, {useCallback, useState} from 'react';
import {useRemotionEnvironment, useVideoConfig} from 'remotion';
import {CanvasTexture} from 'three';

const videoSrc = 'https://remotion.media/video.mp4';

const videoWidth = 1920;
const videoHeight = 1080;
const aspectRatio = videoWidth / videoHeight;
const scale = 3;
const planeHeight = scale;
const planeWidth = aspectRatio * scale;

const Inner: React.FC = () => {
  const [canvasStuff] = useState(() => {
    const canvas = new OffscreenCanvas(videoWidth, videoHeight);
    const context = canvas.getContext('2d')!;
    const texture = new CanvasTexture(canvas);
    return {canvas, context, texture};
  });

  const {invalidate, advance} = useThree();
  const {isRendering} = useRemotionEnvironment();

  const onVideoFrame = useCallback(
    (frame: CanvasImageSource) => {
      canvasStuff.context.drawImage(frame, 0, 0, videoWidth, videoHeight);
      canvasStuff.texture.needsUpdate = true;
      if (isRendering) {
        // ThreeCanvas's ManualFrameRenderer already calls advance() in a
        // useEffect on frame change, but video frame extraction is async
        // (BroadcastChannel round-trip) and resolves after that useEffect.
        // So by the time onVideoFrame fires, the scene was already rendered
        // with the stale texture. We need a second advance() here to
        // re-render the scene now that the texture is actually updated.
        advance(performance.now());
      } else {
        // During preview with the default frameloop='always', the texture
        // is picked up automatically. This is only needed if
        // frameloop='demand' is passed to <ThreeCanvas>.
        invalidate();
      }
    },
    [canvasStuff.context, canvasStuff.texture, invalidate, advance, isRendering],
  );

  return (
    <>
      <Video src={videoSrc} onVideoFrame={onVideoFrame} muted headless />
      <mesh>
        <planeGeometry args={[planeWidth, planeHeight]} />
        <meshBasicMaterial color={0xffffff} toneMapped={false} map={canvasStuff.texture} />
      </mesh>
    </>
  );
};

export const RemotionMediaVideoTexture: React.FC = () => {
  const {width, height} = useVideoConfig();

  return (
    <ThreeCanvas style={{backgroundColor: 'white'}} linear width={width} height={height}>
      <Inner />
    </ThreeCanvas>
  );
};

注意​

🌐 Notes

  • 通过使用 headless 属性，<Video> 标签将不会返回任何内容，因此它可以在 <ThreeCanvas> 内挂载而不影响渲染。
  • 在渲染过程中，<ThreeCanvas> 设置了 frameloop='never'，这意味着场景仅在需要时重新渲染。请在 onVideoFrame 内使用 advance() 来在截图之前同步重新渲染场景。使用 invalidate() 只会安排异步重新渲染，这可能导致过时的帧——特别是在并发大于 1 的情况下。
  • 在预览期间，invalidate() 足够，因为帧循环会持续运行。

示例​

🌐 Examples

使用 <OffthreadVideo>​

🌐 Using <OffthreadVideo>

已弃用，建议使用 @remotion/media

🌐 deprecated in favor of using @remotion/media

你可以使用 @remotion/three 的 useOffthreadVideoTexture() 钩子从视频中获取纹理。

🌐 You can use the useOffthreadVideoTexture() hook from @remotion/three to get a texture from a video.

缺点：

  • 必须先将整个视频下载到磁盘上，然后才能提取帧
  • 它在客户端渲染中不起作用
  • 它为每一帧创建一个新纹理，这比使用 @remotion/media 的效率低

因此，这个 API 已被弃用，建议使用上面提到的推荐方法。

🌐 This API is therefore deprecated in favor of using the recommended approach mentioned above.

使用 <Html5Video>​

🌐 Using <Html5Video>

已弃用，建议使用 @remotion/media

🌐 deprecated in favor of using @remotion/media

你可以使用 @remotion/three 的 useVideoTexture() 钩子从视频中获取纹理。

🌐 You can use the useVideoTexture() hook from @remotion/three to get a texture from a video.

它具有 <Html5Video> 标签的所有缺点，因此已 不推荐使用，建议使用上述提到的推荐方法。

🌐 It has all the drawbacks of the <Html5Video> tag and is therefore deprecated in favor of using the recommended approach mentioned above.

另请参阅​

🌐 See also

Last updated on