在 JavaScript 中从视频中提取帧

从视频文件中提取帧，例如在编辑界面中显示胶片条，可以使用 Mediabunny 来完成。

🌐 Extracting frames from a video file, for example to display a filmstrip in an editing interface, can be done using Mediabunny.

这是一个你可以复制粘贴到你项目中的 extractFrames() 函数：

🌐 Here's a extractFrames() function you can use copy and paste into your project:

extract-frames.ts
import {ALL_FORMATS, Input, InputDisposedError, UrlSource, VideoSample, VideoSampleSink} from 'mediabunny';

type Options = {
  track: {width: number; height: number};
  container: string;
  durationInSeconds: number | null;
};

export type ExtractFramesTimestampsInSecondsFn = (options: Options) => Promise<number[]> | number[];

export type ExtractFramesProps = {
  src: string;
  timestampsInSeconds: number[] | ExtractFramesTimestampsInSecondsFn;
  onVideoSample: (sample: VideoSample) => void;
  signal?: AbortSignal;
};

export async function extractFrames({src, timestampsInSeconds, onVideoSample, signal}: ExtractFramesProps): Promise<void> {
  using input = new Input({
    formats: ALL_FORMATS,
    source: new UrlSource(src),
  });

  const [durationInSeconds, format, videoTrack] = await Promise.all([input.computeDuration(), input.getFormat(), input.getPrimaryVideoTrack()]);
  if (!videoTrack) {
    throw new Error('No video track found in the input');
  }
  if (signal?.aborted) {
    throw new Error('Aborted');
  }

  const timestamps =
    typeof timestampsInSeconds === 'function'
      ? await timestampsInSeconds({
          track: {
            width: videoTrack.displayWidth,
            height: videoTrack.displayHeight,
          },
          container: format.name,
          durationInSeconds,
        })
      : timestampsInSeconds;

  if (timestamps.length === 0) {
    return;
  }

  if (signal?.aborted) {
    throw new Error('Aborted');
  }

  const sink = new VideoSampleSink(videoTrack);

  for await (using videoSample of sink.samplesAtTimestamps(timestamps)) {
    if (signal?.aborted) {
      break;
    }

    if (!videoSample) {
      continue;
    }

    onVideoSample(videoSample);
  }
}

示例：在特定时间提取帧

🌐 Example: Extract frames at specific times

await extractFrames({
  src: 'https://remotion.media/video.mp4',
  timestampsInSeconds: [0, 1, 2, 3, 4],
  onVideoSample: (sample) => {
    // Draw frame to canvas
    const canvas = document.createElement('canvas');
    canvas.width = sample.displayWidth;
    canvas.height = sample.displayHeight;
    const ctx = canvas.getContext('2d');
    sample.draw(ctx!, 0, 0);
  },
});

示例：创建影片条

🌐 Example: Create a filmstrip

根据视频的纵横比，在画布中提取尽可能多的帧：

🌐 Extract as many frames as fit in a canvas based on the video's aspect ratio:

extract-frames.ts
const canvasWidth = 500;
const canvasHeight = 80;
const fromSeconds = 0;
const toSeconds = 10;

await extractFrames({
  src: 'https://remotion.media/video.mp4',
  timestampsInSeconds: async ({track, durationInSeconds}) => {
    const aspectRatio = track.width / track.height;
    const amountOfFramesFit = Math.ceil(canvasWidth / (canvasHeight * aspectRatio));
    const segmentDuration = toSeconds - fromSeconds;
    const timestamps: number[] = [];

    for (let i = 0; i < amountOfFramesFit; i++) {
      timestamps.push(fromSeconds + (segmentDuration / amountOfFramesFit) * (i + 0.5));
    }

    return timestamps;
  },
  onVideoSample: (sample) => {
    // Process the sample
    console.log(`Frame at ${sample.timestamp}s`);

    // Draw to canvas or process as needed
    const canvas = document.createElement('canvas');
    canvas.width = sample.displayWidth;
    canvas.height = sample.displayHeight;
    const ctx = canvas.getContext('2d');
    sample.draw(ctx!, 0, 0);
  },
});

内存管理

🌐 Memory management

在上面的示例中，using 关键字用于在 VideoSample 和 Input 对象超出作用域时自动关闭它们。确保不要保留对它们的引用，以确保正确清理。

🌐 In the example above, the using keyword is used to automatically close the VideoSample and Input objects when they go out of scope. Make sure to keep no references to them to ensure proper cleanup.

中止帧提取

🌐 Abort frame extraction

传递一个 AbortSignal 来取消帧提取：

🌐 Pass an AbortSignal to cancel frame extraction:

extract-frames.ts
const controller = new AbortController();

// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000);

try {
  await extractFrames({
    src: 'https://remotion.media/video.mp4',
    timestampsInSeconds: [0, 1, 2, 3, 4],
    onVideoSample: (sample) => {
      using frame = sample;
      const canvas = document.createElement('canvas');
      canvas.width = frame.displayWidth;
      canvas.height = frame.displayHeight;
      const ctx = canvas.getContext('2d');
      frame.draw(ctx!, 0, 0);
    },
    signal: controller.signal,
  });

  console.log('Frame extraction complete!');
} catch (error) {
  console.error('Frame extraction was aborted or failed:', error);
}

设置超时

🌐 Setting a timeout

以下是设置提取帧最大持续时间的方法：

🌐 Here is how you can set a maximum duration for extracting frames:

Fail if not able to extract within 10 seconds
const controller = new AbortController();

const timeoutPromise = new Promise<never>((_, reject) => {
  const timeoutId = setTimeout(() => {
    controller.abort();
    reject(new Error('Frame extraction timed out after 10 seconds'));
  }, 10000);

  controller.signal.addEventListener('abort', () => clearTimeout(timeoutId), {once: true});
});

try {
  await Promise.race([
    extractFrames({
      src: 'https://remotion.media/video.mp4',
      timestampsInSeconds: [0, 1, 2, 3, 4],
      onVideoSample: (sample) => {
        using frame = sample;
        const canvas = document.createElement('canvas');
        canvas.width = frame.displayWidth;
        canvas.height = frame.displayHeight;
        const ctx = canvas.getContext('2d');
        frame.draw(ctx!, 0, 0);
      },
      signal: controller.signal,
    }),
    timeoutPromise,
  ]);

  console.log('Frame extraction complete!');
} catch (error) {
  console.error('Frame extraction was aborted or failed:', error);
}

另请参阅

🌐 See also

示例：在特定时间提取帧​

示例：创建影片条​

内存管理​

中止帧提取​

设置超时​

另请参阅​