在 SQS 中使用 Lambda
这个过程有一个缺陷：队列会在渲染被触发后立即继续，而不是在渲染完成后继续。这意味着渲染不会全部按顺序执行。我们正在研究一种改进的 Lambda 排队策略，同时请注意这种行为。
本指南将向你展示如何使用 Amazon SQS（简单队列服务）和 Remotion 的 renderMediaOnLambda() API。
🌐 This guide will show you how to use Amazon SQS (Simple Queue Service) and Remotion's renderMediaOnLambda() API.
队列用于将请求排队，以为底层资源应对需求腾出空间。由于 AWS Lambda 受并发限制的影响，你可以使用 SQS 在后台排队渲染，并在渲染过程完成后通过发送电子邮件或使用其他通知方式通知用户。
🌐 Queues are used to park requests to make way for the underlying resources to cope up demand. Since AWS Lambda is subject to a concurrency limit, you can use SQS to queue renders in the background and you notify the user when the render process is completed by sending them an email or using other means of notification.
为了补充本指南，已经创建了两个项目：
🌐 To supplement this guide, two projects have been created:
- remotion-app 包含一个 Remotion 组合以及用于在 AWS 中部署和删除 Remotion Lambda 基础设施的实用脚本。请注意，这与 Serverless Framework 指南 中的应用相同。
- apigw-sqs-app（API Gateway SQS）包含一个 CDK 项目，该项目部署了两个 Lambda 函数。enqueue-function 函数在被调用时，会将用户发送的 JSON 数据加入队列；这些数据可以作为 Remotion Lambda 函数的输入负载。render-lambda-function 函数监听来自 SQS 的队列消息，处理消息并渲染视频。
enqueue-function 配置为通过 API Gateway 调用，并由 Cognito 进行保护。API Gateway 和 Cognito 的设置会在执行 cdk deploy 时由 CDK Stack 部署自动创建。
🌐 The enqueue-function is configured to be invoked through API Gateway and is secured by Cognito. The API Gateway and Cognito setup is automatically created by the CDK Stack deployment upon execution of cdk deploy.
本指南假设你具备使用 CDK 与 TypeScript 以及 AWS SQS 知识 的经验。之所以选择 AWS 云开发工具包 (CDK) 来提供基础设施，是因为它是来自 AWS 的官方库。
🌐 This guide assumes that you have knowledge in using CDK with TypeScript and AWS SQS knowledge. The AWS Cloud Development Kit (CDK) has been chosen to provision infrastructure as it is an official library from AWS.
remotion-app
- 按照 remotion-app 指南 的相同设置说明进行操作，因为我们将重复使用该应用。
apigw-sqs-app
在接下来的步骤中，Lambda 函数 enqueue-function 和 render-lambda-function 将部署到你的 AWS 账户。本指南旨在在你的本地计算机上执行。
🌐 In the following step, the Lambda functions enqueue-function and render-lambda-function will be deployed to your AWS account. This guide is designed to be executed on your local machine.
该项目将创建 CDK 堆栈中定义的所有资源，包括为项目的身份验证和授权系统设置 Cognito、上传 Lambda 代码、生成并关联 IAM 角色到你的 AWS 账户。
🌐 The project will create all the resources defined by the CDK stack, including setting up Cognito for the project's authentication and authorization system, uploading Lambda code, generating and associating IAM roles to your AWS account.
先决条件
🌐 Prerequisites
- 在本地计算机上配置 AWS 部署配置文件，在本地计算机上配置 AWS 部署配置文件。
- 一个名为
remotion-executionrole-policy的 AWS 策略是根据这个指南创建的。 - AWS CDK 应该在你的本地计算机上全局安装。如果尚未安装，请按照“安装 AWS CDK”部分的说明进行操作 这里 。
设置
🌐 Setup
1. 克隆或下载项目
🌐 1. Clone or download the project
该项目可以在 remotion-serverless project 找到。如果在上一步骤中尚未完成，请使用以下命令克隆项目：
🌐 The project can be found at remotion-serverless project. If not done in the previous step, clone the project using:
git clone https://github.com/alexfernandez803/remotion-serverless2. 转到 remotion-serverless 并进入 apigw-sqs-app 目录
🌐 2. Go to remotion-serverless and traverse to apigw-sqs-app directory
cd remotion-serverless && cd apigw-sqs-app3. 安装依赖
🌐 3. Install dependencies
- npm
- yarn
- pnpm
npm ipnpm iyarn4. 创建移除政策
🌐 4. Create the Remotion policy
remotion-executionrole-policy应该已经创建，如果没有，请按照此指南进行设置。
remotion-executionrole-policy 引用自 这里。CDK 堆栈为 render-lambda-function 创建了一个带有 remotion-executionrole-policy 的角色。
🌐 The remotion-executionrole-policy is referenced from here. The CDK stack creates a role for render-lambda-function with the remotion-executionrole-policy.
// 👇 Create a role with custom name
const renderFunctionLambdaRole = new Role(this, 'remotionSQSLambdaRole', {
roleName: 'remotionSQSLambdaRole',
assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
managedPolicies: [
ManagedPolicy.fromAwsManagedPolicyName(
'service-role/AWSLambdaBasicExecutionRole',
),
ManagedPolicy.fromManagedPolicyName(
this,
'remotion-executionrole-policy',
'remotion-executionrole-policy',
),
],
});这将在 AWS 中创建一个名为 remotionSQSLambdaRole 的角色，并附加 2 个策略：
🌐 This creates a role in AWS with the name of remotionSQSLambdaRole with 2 policies attached:
service-role/AWSLambdaBasicExecutionRole授权 Lambda 函数与 CloudWatch 进行交互并记录信息。该策略是 AWS 策略库的一部分。remotion-executionrole-policy策略授予 Lambda 函数与 Remotion Lambda 需要访问的 AWS 服务交互的权限，以便渲染视频。此策略与此指南中的策略完全相同。
5. 可选 - 合成
🌐 5. Optional - Synthesize
在 apigw-sqs-app 目录下，执行 cdk synthesize 命令。
🌐 From the apigw-sqs-app directory, execute the cdk synthesize command.
cdk synthesize此命令将显示 CDK 将在你的 AWS 账户中执行的 CloudFormation 模板。
🌐 This command will show the CloudFormation Template that CDK will execute to your AWS account.
6. 部署 apigw-sqs-app 项目
🌐 6. Deploy the apigw-sqs-app project
在 apigw-sqs-app 目录中，执行 cdk deploy 命令。
🌐 From the apigw-sqs-app directory, execute the cdk deploy command.
cdk deploy这将协调生成 CDK 堆栈中定义的资源，这些资源在后台是 CloudFormation 模板。
🌐 This will orchestrate the generation of resources defined in the CDK stack, which are CloudFormation templates in the background.
Deploy outputBundling asset apigw-sqs-app-stack/enqueue-function/Code/Stage... cdk.out/bundling-temp-a813aece2454684086de775f918faac45b1b77c67fff24ec6aad4bff8c978ebe/index.js 881.5kb ⚡ Done in 72ms Bundling asset apigw-sqs-app-stack/render-function/Code/Stage... cdk.out/bundling-temp-e7d973ee34691a8e6a2ceda969fbf59380866bb486be333238b7e554907f7b95/index.js 2.6kb ⚡ Done in 2ms added 279 packages, and audited 280 packages in 2s 21 packages are looking for funding run `npm fund` for details found 0 vulnerabilities ✨ Synthesis time: 7.97s apigw-sqs-app-stack: building assets... [0%] start: Building 8efaff13bbe794558db1f1cb8f506bc13b87d7ab3e568ebc324bac680da3a75d:XXXXXXXXXX-ap-southeast-2 [0%] start: Building 3b7a9f596977e2db94a676c6c89c99dd7eb87a5985f97a11ff23b9f338027764:XXXXXXXXXX-ap-southeast-2 [0%] start: Building cf7f13fe5c0ff3b22e7352152a554dd8a4767f6a5e2285e6bf353fc42070e697:XXXXXXXXXX-ap-southeast-2 [33%] success: Built 8efaff13bbe794558db1f1cb8f506bc13b87d7ab3e568ebc324bac680da3a75d:XXXXXXXXXX-ap-southeast-2 [66%] success: Built 3b7a9f596977e2db94a676c6c89c99dd7eb87a5985f97a11ff23b9f338027764:XXXXXXXXXX-ap-southeast-2 [100%] success: Built cf7f13fe5c0ff3b22e7352152a554dd8a4767f6a5e2285e6bf353fc42070e697:XXXXXXXXXX-ap-southeast-2 apigw-sqs-app-stack: assets built This deployment will make potentially sensitive changes according to your current security approval level (--require-approval broadening). Please confirm you intend to make the following modifications: IAM Statement Changes ┌───┬─────────────────────────────┬────────┬─────────────────────────────┬─────────────────────────────┬────────────────────────────────┐ │ │ Resource │ Effect │ Action │ Principal │ Condition │ ├───┼─────────────────────────────┼────────┼─────────────────────────────┼─────────────────────────────┼────────────────────────────────┤ │ + │ ${api-integration-role.Arn} │ Allow │ sts:AssumeRole │ Service:lambda.amazonaws.co │ │ │ │ │ │ │ m │ │ ├───┼─────────────────────────────┼────────┼─────────────────────────────┼─────────────────────────────┼────────────────────────────────┤ │ + │ ${enqueue-function.Arn} │ Allow │ lambda:InvokeFunction │ Service:apigateway.amazonaw │ "ArnLike": { │ │ │ │ │ │ s.com │ "AWS:SourceArn": "arn:${AWS: │ │ │ │ │ │ │ :Partition}:execute-api:ap-sou │ │ │ │ │ │ │ theast-2:XXXXXXXXXX:${apiC85 │ │ │ │ │ │ │ 50315}/*/*/enqueue" │ │ │ │ │ │ │ } │ │ + │ ${enqueue-function.Arn} │ Allow │ lambda:InvokeFunction │ Service:apigateway.amazonaw │ "ArnLike": { │ │ │ │ │ │ s.com │ "AWS:SourceArn": "arn:${AWS: │ │ │ │ │ │ │ :Partition}:execute-api:ap-sou │ │ │ │ │ │ │ theast-2:XXXXXXXXXX:${apiC85 │ │ │ │ │ │ │ 50315}/*/*/enqueue" │ │ │ │ │ │ │ } │ ├───┼─────────────────────────────┼────────┼─────────────────────────────┼─────────────────────────────┼────────────────────────────────┤ │ + │ ${queue.Arn} │ Allow │ sqs:ChangeMessageVisibility │ AWS:${remotionSQSLambdaRole │ │ │ │ │ │ sqs:DeleteMessage │ } │ │ │ │ │ │ sqs:GetQueueAttributes │ │ │ │ │ │ │ sqs:GetQueueUrl │ │ │ │ │ │ │ sqs:ReceiveMessage │ │ │ │ + │ ${queue.Arn} │ Allow │ sqs:GetQueueAttributes │ AWS:${api-integration-role} │ │ │ │ │ │ sqs:GetQueueUrl │ │ │ │ │ │ │ sqs:SendMessage │ │ │ ├───┼─────────────────────────────┼────────┼─────────────────────────────┼─────────────────────────────┼────────────────────────────────┤ │ + │ ${remotionSQSLambdaRole.Arn │ Allow │ sts:AssumeRole │ Service:lambda.amazonaws.co │ │ │ │ } │ │ │ m │ │ └───┴─────────────────────────────┴────────┴─────────────────────────────┴─────────────────────────────┴────────────────────────────────┘ IAM Policy Changes ┌───┬──────────────────────────┬────────────────────────────────────────────────────────────────────────────────┐ │ │ Resource │ Managed Policy ARN │ ├───┼──────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤ │ + │ ${api-integration-role} │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole │ ├───┼──────────────────────────┼────────────────────────────────────────────────────────────────────────────────┤ │ + │ ${remotionSQSLambdaRole} │ arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole │ │ + │ ${remotionSQSLambdaRole} │ arn:${AWS::Partition}:iam::XXXXXXXXXX:policy/remotion-executionrole-policy │ └───┴──────────────────────────┴────────────────────────────────────────────────────────────────────────────────┘ (NOTE: There may be security-related changes not in this list. See https://github.com/aws/aws-cdk/issues/1299) Do you wish to deploy these changes (y/n)? y
选择 'y' 作为提示的答案，CDK 将继续部署该堆栈，并带有两个功能。
🌐 Select 'y' for the answer of the prompt and CDK will continue to deploy the Stack, with the 2 function.
final outputapigw-sqs-app-stack: deploying... [1/1] [0%] start: Publishing 8efaff13bbe794558db1f1cb8f506bc13b87d7ab3e568ebc324bac680da3a75d:XXXXXXXXXX-ap-southeast-2 [0%] start: Publishing 3b7a9f596977e2db94a676c6c89c99dd7eb87a5985f97a11ff23b9f338027764:XXXXXXXXXX-ap-southeast-2 [0%] start: Publishing cf7f13fe5c0ff3b22e7352152a554dd8a4767f6a5e2285e6bf353fc42070e697:XXXXXXXXXX-ap-southeast-2 [33%] success: Published 3b7a9f596977e2db94a676c6c89c99dd7eb87a5985f97a11ff23b9f338027764:XXXXXXXXXX-ap-southeast-2 [66%] success: Published 8efaff13bbe794558db1f1cb8f506bc13b87d7ab3e568ebc324bac680da3a75d:XXXXXXXXXX-ap-southeast-2 [100%] success: Published cf7f13fe5c0ff3b22e7352152a554dd8a4767f6a5e2285e6bf353fc42070e697:XXXXXXXXXX-ap-southeast-2 apigw-sqs-app-stack: creating CloudFormation changeset... ✅ apigw-sqs-app-stack ✨ Deployment time: 158.36s Outputs: apigw-sqs-app-stack.apiUrl = https://6mvgq2iad9.execute-api.ap-southeast-2.amazonaws.com/ apigw-sqs-app-stack.queuearn = arn:aws:sqs:ap-southeast-2:XXXXXXXXXX:remotion_queue apigw-sqs-app-stack.queuename = remotion_queue apigw-sqs-app-stack.queueurl = https://sqs.ap-southeast-2.amazonaws.com/XXXXXXXXXX/remotion_queue apigw-sqs-app-stack.region = ap-southeast-2 apigw-sqs-app-stack.userPoolClientId = 5d88adjpffj314pm4pot8g292i apigw-sqs-app-stack.userPoolId = ap-southeast-2_vzSlhO9O0 Stack ARN: arn:aws:cloudformation:ap-southeast-2:XXXXXXXXXX:stack/apigw-sqs-app-stack/acb8b8f0-a52a-11ed-a440-024da00b5a8e
该部署将提供诸如 apigw-sqs-app-stack.region、apigw-sqs-app-stack.userPoolClientId 和 apigw-sqs-app-stack.userPoolId 等变量。
🌐 The deployment will provide variables such as apigw-sqs-app-stack.region, apigw-sqs-app-stack.userPoolClientId, and apigw-sqs-app-stack.userPoolId.
8. 如果不再需要，从你的 AWS 账户中删除 apigw-sqs-app
🌐 8. Remove the apigw-sqs-app from your AWS account, if not needed anymore
来自 apigw-sqs-app 目录。
🌐 From the apigw-sqs-app directory.
cdk destroy将 API 网关、Remotion Lambda 和 SQS 组合在一起。
🌐 Combining API Gateway, Remotion Lambda and SQS together.
以下是关于 IAM 角色如何被函数使用的重要信息，以及启用 render-lambda-function 消费 SQS 消息的说明。
🌐 These are important information on how IAM roles are used by the function and instructions to enable the render-lambda-function to consume SQS messages.
从 CDK 堆栈代码 中，这里是需要注意的重要部分。
🌐 From CDK stack code, here are the important parts to take note of.
- 堆栈创建了一个名为
remotion_queue的队列。
create queue// 👇 create the queue const remotionQueue = new sqs.Queue(this, 'queue', { encryption: sqs.QueueEncryption.KMS_MANAGED, queueName: 'remotion_queue', });
-
remotion_queue授予访问 2 个 Lambda 函数的权限以与其交互。每个单独的角色都分配给它们各自的 Lambda 函数。apiIntegrationRole// 👇 create the apiIntegrationRole role const apiIntegrationRole = new IAM.Role(this, 'api-integration-role', { assumedBy: new IAM.ServicePrincipal('lambda.amazonaws.com'), managedPolicies: [ ManagedPolicy.fromAwsManagedPolicyName( 'service-role/AWSLambdaBasicExecutionRole', ), ], }); -
此角色被分配给
enqueue-function，以允许它写入 CloudWatch 日志。remotionSQSLambdaRole// 👇 create a role with custom name const renderFunctionLambdaRole = new Role(this, 'remotionSQSLambdaRole', { roleName: 'remotionSQSLambdaRole', assumedBy: new ServicePrincipal('lambda.amazonaws.com'), managedPolicies: [ ManagedPolicy.fromAwsManagedPolicyName( 'service-role/AWSLambdaBasicExecutionRole', ), ManagedPolicy.fromManagedPolicyName( this, 'remotion-executionrole-policy', 'remotion-executionrole-policy', ), ], }); -
render-lambda-function已被分配此角色，该角色包括访问 Cloudwatch 的权限以及按照remotion-executionrole-policy中指定的权限访问其他 AWS 服务的权限。Grant access to the queue// 👇 Grant permission to publish to the queue remotionQueue.grantSendMessages(apiIntegrationRole); // 👇 grant permission to consume messages from the queue remotionQueue.grantConsumeMessages(renderFunctionLambdaRole); -
允许渲染函数监听队列。
Listen to queueremotionRenderFunction.addEventSource( new SqsEventSource(remotionQueue, { batchSize: 1, maxBatchingWindow: Duration.minutes(5), reportBatchItemFailures: true, // default to false }), );
与 API 交互
🌐 Interacting with the API
该 API 需要授权令牌才能进行交互。要获取该令牌：
🌐 The API requires an authorization token to interact with it. To obtain the token:
- 部署成功后，你将获得诸如
apigw-sqs-app-stack.region、apigw-sqs-app-stack.userPoolClientId和apigw-sqs-app-stack.userPoolId的输出，这些用于与 Cognito 进行身份验证。 - 如果你没有需要用户登录的前端，你可以按照此指南为 API 手动创建用户和身份验证令牌。
根据指南，YOUR_USER_POOL_CLIENT_ID 是 apigw-sqs-app-stack.userPoolClientId，YOUR_USER_POOL_ID 是 apigw-sqs-app-stack.userPoolId，步骤应按照获取 IdToken 为止执行。
🌐 From the guide, YOUR_USER_POOL_CLIENT_ID is apigw-sqs-app-stack.userPoolClientId and YOUR_USER_POOL_ID is the apigw-sqs-app-stack.userPoolId, the steps should be followed up to retrieving the IdToken.
基础 API URL 的格式是来自输出 apigw-sqs-app-stack.apiUrl 的 https://25w651t09g.execute-api.ap-southeast-2.amazonaws.com/dev/enqueue。
🌐 The base API URL has the format of https://25w651t09g.execute-api.ap-southeast-2.amazonaws.com/dev/enqueue from the output apigw-sqs-app-stack.apiUrl.
触发视频生成请求
🌐 Trigger a video generation request
render videocurl --location --request POST 'https://xxxxxxxx.execute-api.ap-southeast-2.amazonaws.com/dev/enqueue' \ --header 'Authorization: Bearer eyJraWQiOiJMVVVVZGtIQ1JXWEEyWEEXXXXXXXXXjMKR1t5S-oA' \ --data-raw '{ "message": "Hello" }'
response{ {"message":"Message Send to SQS- Here is MessageId: a6abd0bc-b838-48b5-a562-4c511fac5b2f"} }
这将通过获取 JSON 并将其放入队列中来启动视频的渲染请求。
在另一端，render-lambda-function 将通过 SQS 接收到这是一个视频渲染请求的通知，它将从渲染请求中获取数据并按照指令生成视频，该函数在此过程中执行 renderMediaOnLambda()。
🌐 This will initiate the render request of a video by taking the JSON and putting it in the queue.
On the other end, render-lambda-function will be notified by SQS that is a video render request, it will consume the data from the render request and generate the video as per instruction, the function executes renderMediaOnLambda() as part of the process.
注意
🌐 Notes
- Remotion Lambda 的部署配置为仅部署到
ap-southeast-2区域，以简化项目，可在region.ts的代码中进行调整。 apigw-sqs-app的部署配置为在ap-southeast-2区域进行，以简化项目，在代码的remotion-cdk-starter.ts中进行调整。- 部署时，Remotion 包应打包在函数内部，你可以在
bundling对象的nodeModules属性中进行设置，相关代码位于 这里。
另请参阅
🌐 See also