# Computer Vision Pipeline Architecture: A Tutorial # 计算机视觉流水线架构：入门教程 **Build a simple, reusable foundation for your video processing apps and alleviate the common challenges of video manipulation.** **构建简单、可重用的视频处理应用基础，解决视频操作中的常见挑战。** Video and all of its moving parts can be a lot for a developer to contend with. An expert developer’s [deep understanding of data structures](https://www.toptal.com/java/the-trie-a-neglected-data-structure), encoding techniques, and image and signal processing plays a major role in the outcomes of allegedly simple, everyday video processing tasks such as [compression or editing](https://www.toptal.com/marketing/video-editors). 对于开发者而言，视频及其所有动态组成部分可能非常难以应付。在处理诸如[压缩或编辑](https://www.toptal.com/marketing/video-editors)等看似简单的日常视频任务时，开发者对[数据结构](https://www.toptal.com/java/the-trie-a-neglected-data-structure)、编码技术以及图像和信号处理的深刻理解，将在很大程度上决定最终的效果。 To work effectively with video content, you must understand the properties and distinctions between its primary file formats (e.g., .mp4, .mov, .wmv, .avi) and their version-specific codecs (e.g., H.264, H.265, VP8, VP9). The tools necessary for effective video processing are seldom neatly packaged as comprehensive libraries, leaving the developer to navigate the vast, intricate ecosystem of open-source tools to deliver engaging computer vision applications. 为了有效地处理视频内容，你必须了解主要文件格式（如 .mp4、.mov、.wmv、.avi）及其特定版本的编解码器（如 H.264、H.265、VP8、VP9）之间的属性和区别。有效的视频处理工具很少被整齐地打包成综合库，这使得开发者必须在庞大而复杂的开源工具生态系统中摸索，才能开发出引人入胜的计算机视觉应用。 ## Computer Vision Applications Explained ## 计算机视觉应用详解 Computer vision applications are based on the implementation of the spectrum of techniques—from simple heuristics to [complex neural networks](https://www.toptal.com/neural-network)—by which we feed an image or video to a computer as input and produce meaningful output, such as: 计算机视觉应用基于一系列技术的实现——从简单的启发式算法到[复杂的神经网络](https://www.toptal.com/neural-network)。通过这些技术，我们将图像或视频作为输入提供给计算机，并产生有意义的输出，例如： * **Facial recognition features** in smartphone cameras, useful for organizing and searching photo albums and for tagging individuals in social media apps. 智能手机相机中的**人脸识别功能**，用于整理和搜索相册，或在社交媒体应用中标记个人。 * **Road marking detection**, as implemented in self-driving cars moving at high speeds. 高速行驶的自动驾驶汽车中实现的**道路标线检测**。 * **Optical character recognition (OCR)** technology that allows visual search apps (like Google Lens) to recognize the shapes of text characters in photos. **光学字符识别 (OCR)** 技术，使视觉搜索应用（如 Google Lens）能够识别照片中的文字形状。 The preceding examples are as different as can be, each showcasing an entirely distinctive function, but sharing one simple commonality: [Images are their primary input.](https://www.toptal.com/machine-learning/machine-learning-video-analysis) Each application transforms unstructured—sometimes chaotic—images or frames into intelligible and ordered data that provides a benefit to end users. 上述例子千差万别，展示了完全不同的功能，但它们有一个简单的共同点：**[图像是它们的主要输入。](https://www.toptal.com/machine-learning/machine-learning-video-analysis)** 每个应用都将非结构化的——有时是混乱的——图像或帧转化为可理解的、有序的数据，从而为最终用户提供价值。 ## Size Matters: Common Challenges of Working With Video ## 大小很关键：处理视频时的常见挑战 An end user who views a video may regard it as a single entity. But a developer must approach it as a collection of individual, sequential frames. For example, before an engineer writes a program to detect real-time traffic patterns in a video of moving vehicles, they must first extract individual frames from that video, and then apply an algorithm that detects the cars on the road. 观看视频的最终用户可能将其视为一个整体。但开发者必须将其视为一系列独立的、连续的帧的集合。例如，在工程师编写程序来检测视频中移动车辆的实时交通模式之前，他们必须首先从视频中提取单独的帧，然后应用算法来检测道路上的汽车。 In its raw state, a video file is huge in size, making it too large to hold in a computer’s memory, unwieldy for the developer to handle, difficult to share, and costly to store. A single minute of 60 frames per second (fps) raw, uncompressed video requires more than 22 gigabytes of storage space, for example: 原始状态下的视频文件体积巨大，不仅大到无法完全放入计算机内存，而且难以处理、难以分享，存储成本高昂。例如，一分钟 60 帧每秒 (fps) 的原始未压缩视频需要超过 22 GB 的存储空间： > **60** seconds * **1080** px (height) * **1920** px (width) * **3** bytes per pixel * **60** fps = **22.39** GB > **60** 秒 * **1080** px (高) * **1920** px (宽) * **3** 字节/像素 * **60** fps = **22.39** GB Video is, therefore, compressed before it is processed, as a matter of course. But there is no guarantee that an individual compressed video frame will display an image in its entirety. This is because the parameters applied at compression time define the quality and detail a video’s individual frame will retain. While the compressed video, as a whole, may play well enough to provide a terrific viewing experience, that is not the same as the individual frames comprising it being interpretable as complete images. 因此，视频在处理前通常会被压缩。但并不能保证单个压缩视频帧能显示完整的图像。这是因为压缩时应用的参数定义了单个视频帧保留的质量和细节。虽然压缩视频作为一个整体可能播放流畅，提供极佳的观看体验，但这并不意味着构成它的单个帧可以被解读为完整的图像。 In this tutorial, we will use popular open-source computer vision tools to solve some basic challenges of video processing. This experience will position you to customize a computer vision pipeline to your exact use cases. (To keep things simple, we will not describe the audio components of video in this article.) 在本教程中，我们将使用流行的开源计算机视觉工具来解决视频处理的一些基本挑战。这一经验将帮助你根据具体用例定制计算机视觉流水线。（为了保持简单，本文将不涉及视频的音频部分。） ## A Simple Computer Vision App Tutorial: Calculating Brightness ## 一个简单的计算机视觉应用教程：计算亮度 To deliver a computer vision application, an [engineering team](https://www.toptal.com/developers) develops and implements an efficient and powerful computer vision pipeline whose architecture includes, at a minimum: 为了交付计算机视觉应用，[工程团队](https://www.toptal.com/developers)需要开发并实现一个高效且强大的计算机视觉流水线，其架构至少包括： | Step / 步骤 | Description / 描述 | | :--- | :--- | | **Step 1: Image acquisition**
**步骤 1：图像采集** | Images or videos can be acquired from a wide range of sources, including cameras or sensors, digital videos saved on disk, or videos streamed over the internet.
图像或视频可以从多种来源获取，包括摄像头或传感器、存储在磁盘上的数字视频，或通过互联网流传输的视频。 | | **Step 2: Image preprocessing**
**步骤 2：图像预处理** | The developer chooses preprocessing operations, such as denoising, resizing, or conversion into a more accessible format. These are intended to make the images easier to work with or analyze.
开发者选择预处理操作，如去噪、调整大小或转换为更易访问的格式。这些操作旨在使图像更易于处理或分析。 | | **Step 3: Feature extraction**
**步骤 3：特征提取** | In the representation or extraction step, information in the preprocessed images or frames is captured. This information may consist of edges, corners, or shapes, for instance.
在表示或提取步骤中，预处理后的图像或帧中的信息被捕获。例如，这些信息可能包括边缘、角点或形状。 | | **Step 4: Interpretation, analysis, or output**
**步骤 4：解释、分析或输出** | In the final step we accomplish the task at hand.
在最后一步，我们完成手头的任务。 | Let’s imagine you were hired to build a tool that calculates the brightness of a video’s individual frames. We’ll align the project’s pipeline architecture to match the simple computer vision model shared above. 假设你被聘请构建一个工具来计算视频单个帧的亮度。我们将调整项目的流水线架构，使其符合上述简单的计算机视觉模型。 The program we will produce in this tutorial has been included as an [example within Hypetrigger,](https://github.com/nathanbabcock/hypetrigger-rust/blob/main/examples/rust-simple/src/main.rs) an open-source Rust library I developed. Hypetrigger includes everything you’d need to run a computer vision pipeline on streaming video from the internet: [TensorFlow](https://www.tensorflow.org/) bindings for image recognition, [Tesseract](https://github.com/tesseract-ocr/tesseract) for optical character recognition, and support for using GPU-accelerated video decoding for a 10x speed boost. To install, clone the [Hypetrigger repo](https://github.com/nathanbabcock) and run the command `cargo add hypetrigger`. 我们在本教程中编写的程序已包含在我开发的开源 Rust 库 [Hypetrigger 的示例](https://github.com/nathanbabcock/hypetrigger-rust/blob/main/examples/rust-simple/src/main.rs)中。Hypetrigger 包含了在互联网流视频上运行计算机视觉流水线所需的一切：用于图像识别的 [TensorFlow](https://www.tensorflow.org/) 绑定、用于光学字符识别的 [Tesseract](https://github.com/tesseract-ocr/tesseract)，以及支持 GPU 加速视频解码以实现 10 倍速度提升。要安装它，请克隆 [Hypetrigger 仓库](https://github.com/nathanbabcock) 并运行命令 `cargo add hypetrigger`。 To maximize the learning and experience to be gained, we will construct a computer vision pipeline from scratch in this tutorial, rather than implementing user-friendly Hypetrigger. 为了最大化学习效果和经验积累，在本教程中我们将从头构建一个计算机视觉流水线，而不是直接使用用户友好的 Hypetrigger。 ### Our Tech Stack ### 我们的技术栈 For our project, we will use: 对于本项目，我们将使用： | Tool / 工具 | Description / 描述 | | :--- | :--- | | **[FFmpeg](https://www.toptal.com/ffmpeg)** | Touted as one of the best tools out there for working with video, [FFmpeg](https://ffmpeg.org/)—the Swiss Army knife of video—is an open-source library written in C and used for encoding, decoding, conversion, and streaming. It is used in enterprise software like [Google Chrome](https://support.google.com/webdesigner/answer/10151269?hl=en#zippy=%2Cffmpeg), [VLC Media Player](https://wiki.videolan.org/FFmpeg/), and [OBS](https://github.com/obsproject/obs-studio/search?q=ffmpeg), among others.
被誉为处理视频的最佳工具之一，[FFmpeg](https://ffmpeg.org/)——视频界的“瑞士军刀”——是一个用 C 语言编写的开源库，用于编码、解码、转换和流传输。它被用于 [Google Chrome](https://support.google.com/webdesigner/answer/10151269?hl=en#zippy=%2Cffmpeg)、[VLC 播放器](https://wiki.videolan.org/FFmpeg/) 和 [OBS 直播软件](https://github.com/obsproject/obs-studio/search?q=ffmpeg) 等企业级软件中。 | | **[Rust](https://www.toptal.com/rust)** | A major strength of Rust is its ability to detect memory errors (e.g., null pointers, segfaults, dangling references) at compile time. Rust offers high performance with guaranteed memory safety, and is also highly performant, making it a good choice for video processing.
Rust 的主要优势在于能够在编译时检测内存错误（例如空指针、段错误、悬垂引用）。Rust 在保证内存安全的同时提供高性能，非常适合视频处理。 | ### Step 1: Image Acquisition ### 步骤 1：图像采集 In this scenario, a previously acquired [animated sample video](http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4) is ready to be processed. 在这个场景中，一个预先获取的[动画示例视频](http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4)已准备好进行处理。 ### Step 2: Image Preprocessing ### 步骤 2：图像预处理 For this project, image preprocessing consists of converting the video from its H.264 encoded format to raw [RGB](https://www.techtarget.com/whatis/definition/RGB-red-green-and-blue), a format that is much easier to work with. 对于本项目，图像预处理包括将视频从 H.264 编码格式转换为原始 [RGB](https://www.techtarget.com/whatis/definition/RGB-red-green-and-blue) 格式，后者更容易处理。 Let’s decompress our video using FFmpeg’s portable, executable command-line tool from inside a Rust program. The Rust program will open and convert our sample video to RGB. For optimal results, we’ll append the appropriate [FFmpeg syntax](https://ffmpeg.org/ffmpeg.html) to the `ffmpeg` command: 我们将在 Rust 程序中调用 FFmpeg 的便携式命令行工具来解压视频。该 Rust 程序将打开示例视频并将其转换为 RGB。为了获得最佳效果，我们将向 `ffmpeg` 命令添加适当的 [FFmpeg 参数](https://ffmpeg.org/ffmpeg.html)： | Argument*
参数* | Description
描述 | Use Case
用例 | | :--- | :--- | :--- | | `-i` | Indicates the file name or URL of the source video.
指定源视频的文件名或 URL。 | [The URL of the sample video](http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4) or another of your choosing
[示例视频的 URL](http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4) 或你选择的其他视频 | | `-f` | Sets the output format.
设置输出格式。 | The `rawvideo` format to obtain raw video frames
`rawvideo` 格式以获取原始视频帧 | | `-pix_fmt` | Sets the pixel format.
设置像素格式。 | `rgb24` to produce RGB color channels with eight bits per channel
`rgb24` 以生成每个通道 8 位的 RGB 颜色通道 | | `-r` | Sets the output frame rate.
设置输出帧率。 | `1` to produce one frame per second
`1` 以产生每秒一帧 | | `` | Tells FFmpeg where to send output; it is a required final argument.
告诉 FFmpeg 输出发送到哪里；这是必需的最后一个参数。 | The identifier `pipe:1` to output on [standard output](http://www.linfo.org/standard_output.html) instead of to file
标识符 `pipe:1` 以输出到[标准输出](http://www.linfo.org/standard_output.html)而不是文件 | *For a complete screen listing of arguments, enter `ffmpeg -help`.* *要查看完整的参数列表，请输入 `ffmpeg -help`。* These arguments combined at the command line or terminal give us `ffmpeg -i input_video.mp4 -f rawvideo -pix_fmt rgb24 pipe:1` and serve as our starting point to process the video’s frames: 将这些参数在命令行或终端中组合起来，我们就得到了 `ffmpeg -i input_video.mp4 -f rawvideo -pix_fmt rgb24 pipe:1`，以此作为处理视频帧的起点： ```rust use std::{ io::{BufReader, Read}, process::{Command, Stdio}, }; fn main() { // Test video provided by https://gist.github.com/jsturgis/3b19447b304616f18657. // 测试视频来源：https://gist.github.com/jsturgis/3b19447b304616f18657 let test_video = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4"; // Video is in RGB format; 3 bytes per pixel (1 red, 1 blue, 1 green). // 视频为 RGB 格式；每个像素 3 字节（1 红，1 蓝，1 绿）。 let bytes_per_pixel = 3; let video_width = 1280; let video_height = 720; // Create an FFmpeg command with the specified arguments. // 使用指定参数创建 FFmpeg 命令。 let mut ffmpeg = Command::new("ffmpeg") .arg("-i") .arg(test_video) // Specify the input video (指定输入视频) .arg("-f") // Specify the output format (raw RGB pixels) (指定输出格式：原始 RGB 像素) .arg("rawvideo") .arg("-pix_fmt") .arg("rgb24") // Specify the pixel format (RGB, 8 bits per channel) (指定像素格式：RGB，每通道8位) .arg("-r") .arg("1") // Request rate of 1 frame per second (请求每秒 1 帧的速率) .arg("pipe:1") // Send output to the stdout pipe (将输出发送到标准输出管道) .stderr(Stdio::null()) .stdout(Stdio::piped()) .spawn() // Spawn the command process (启动命令进程) .unwrap(); // Unwrap the result (i.e., panic and exit if there was an error) (处理结果，如果出错则恐慌并退出) } ``` Our program will receive one video frame at a time, each decoded into raw RGB. To avoid accumulating huge volumes of data, let’s allocate a frame-sized buffer that will release memory as it finishes processing each frame. Let’s also add a loop that fills the buffer with data from FFmpeg’s standard output channel: 我们的程序将一次接收一个视频帧，每一帧都解码为原始 RGB。为了避免积累大量数据，我们分配一个帧大小的缓冲区，该缓冲区将在处理完每一帧后释放内存。我们再添加一个循环，用 FFmpeg 标准输出通道的数据填充缓冲区： ```rust fn main() { // … (前文代码) // Read the video output into a buffer. // 将视频输出读取到缓冲区。 let stdout = ffmpeg.stdout.take().unwrap(); let buf_size = video_width * video_height * bytes_per_pixel; let mut reader = BufReader::new(stdout); let mut buffer = vec![0u8; buf_size]; let mut frame_num = 0; while let Ok(()) = reader.read_exact(buffer.as_mut_slice()) { // Retrieve each video frame as a vector of raw RGB pixels. // 将每个视频帧检索为原始 RGB 像素向量。 let raw_rgb = buffer.clone(); } } ``` Notice that the `while` loop contains a reference to `raw_rgb`, a variable which contains a full RGB image. 注意 `while` 循环中引用了 `raw_rgb`，这个变量包含了一张完整的 RGB 图像。 ### Step 3: Feature Extraction ### 步骤 3：特征提取 To calculate the average brightness of each frame preprocessed in Step 2, let’s add the following function to our program (either before or after the `main` method): 为了计算步骤 2 中预处理后的每一帧的平均亮度，我们在程序中添加以下函数（位于 `main` 方法之前或之后均可）： ```rust /// Calculate the average brightness of an image, /// returned as a float between 0 and 1. /// 计算图像的平均亮度，返回 0 到 1 之间的浮点数。 fn average_brightness(raw_rgb: Vec) -> f64 { let mut sum = 0.0; for (i, _) in raw_rgb.iter().enumerate().step_by(3) { let r = raw_rgb[i] as f64; let g = raw_rgb[i + 1] as f64; let b = raw_rgb[i + 2] as f64; let pixel_brightness = (r / 255.0 + g / 255.0 + b / 255.0) / 3.0; sum += pixel_brightness; } sum / (raw_rgb.len() as f64 / 3.0) } ``` Then, at the end of the `while` loop, we can calculate and print the frames’ brightness to the console: 然后，在 `while` 循环的末尾，我们可以计算并将帧亮度打印到控制台： ```rust fn main() { // … while let Ok(()) = reader.read_exact(buffer.as_mut_slice()) { // Retrieve each video frame as a vector of raw RGB pixels. // 将每个视频帧检索为原始 RGB 像素向量。 let raw_rgb = buffer.clone(); // Calculate the average brightness of the frame. // 计算帧的平均亮度。 let brightness = average_brightness(raw_rgb); println!("frame {frame_num} has brightness {brightness}"); frame_num += 1; } } ``` The code, at this point, will match this [example file](https://github.com/nathanbabcock/hypetrigger-rust/blob/main/examples/rust-simple/src/main.rs). 至此，代码将与此[示例文件](https://github.com/nathanbabcock/hypetrigger-rust/blob/main/examples/rust-simple/src/main.rs)一致。 And now we run the program on our sample video to produce the following output: 现在我们在示例视频上运行该程序，将产生以下输出： ```txt frame 0 has brightness 0.055048076377046 frame 1 has brightness 0.467577447011064 frame 2 has brightness 0.878193112575386 frame 3 has brightness 0.859071674156269 frame 4 has brightness 0.820603467400872 frame 5 has brightness 0.766673757205845 frame 6 has brightness 0.717223347005918 frame 7 has brightness 0.674823835783496 frame 8 has brightness 0.656084418402863 frame 9 has brightness 0.656437488652946 [500+ more frames omitted] ``` ### Step 4: Interpretation ### 步骤 4：解释 Here’s a graphical representation of these numbers: 以下是这些数字的图表表示： ![Average Brightness Level Over Time / 随时间变化的平均亮度水平](https://assets.toptal.io/images?url=https%3A%2F%2Fbs-uploads.toptal.io%2Fblackfish-uploads%2Fpublic-files%2FUntitled-ff1259714bda4791a6c3dc1b6204690d.png) In the preceding graph, note the plotted line that represents our video’s brightness. Its sharp peaks and valleys represent the dramatic transitions in brightness that occur between consecutive frames. The brightness of frame 0, depicted at the graph’s far left, measures at 5% (i.e., quite dark) and peaks sharply at 87% (i.e., remarkably bright), just two frames later. Equally prominent transitions occur around 5:00, 8:00, and 9:40 minutes into the video. In this case, such intense variations in brightness represent normal movie scene transitions, as seen in the video. 在上图中，请注意代表视频亮度的曲线。其尖锐的峰谷代表了连续帧之间发生的剧烈亮度转换。图表最左侧的第 0 帧亮度仅为 5%（即相当暗），仅仅两帧之后就急剧飙升至 87%（即非常亮）。同样显著的转换出现在视频的 5:00、8:00 和 9:40 左右。在本例中，这种剧烈的亮度变化代表了正常的电影场景切换，正如视频中所见。 ## Real-world Use Cases for Calculating Brightness ## 计算亮度的真实应用场景 In the real world, we would likely continue to analyze the brightness levels detected and, conditionally, trigger an action. In true postproduction processing, the filmmaker, videographer, or video editor would analyze this data and retain all frames whose values for brightness fall within the project’s agreed-upon range. Alternatively, a professional may pull and review frames whose brightness values are iffy, and may ultimately approve, re-render, or exclude individual frames from the video’s final output. 在现实世界中，我们可能会继续分析检测到的亮度水平，并根据条件触发操作。在实际的后期制作处理中，电影制作人、摄像师或视频编辑会分析这些数据，并保留亮度值在项目约定范围内的所有帧。或者，专业人员可能会提取并审查亮度值有问题的帧，最终决定批准、重新渲染或从视频的最终输出中排除个别帧。 Another interesting use case for analyzing frame brightness can be illustrated by considering a scenario that involves security camera footage from an office building. By comparing the frames’ brightness levels to the building’s in/out logs, we can determine whether the last person to leave actually shuts off the lights as they are supposed to. If our analysis indicates that lights are being left on after all people have gone for the day, we could send reminders encouraging folks to turn off the lights when they leave in order to conserve energy. 分析帧亮度的另一个有趣用例可以通过办公楼监控摄像头的场景来说明。通过将帧亮度水平与大楼的进出日志进行比较，我们可以确定最后离开的人是否按规定关了灯。如果分析显示在所有人下班离开后灯仍然亮着，我们可以发送提醒，鼓励大家离开时关灯以节约能源。 This tutorial details some basic computer vision processing and lays the foundation for more advanced techniques, such as graphing multiple features of the input video to correlate using more advanced statistical measures. Such analysis marks a crossing from the world of video into the domain of statistical inference and machine learning—the essence of computer vision. 本教程详细介绍了一些基本的计算机视觉处理方法，并为更先进的技术奠定了基础，例如绘制输入视频的多个特征图表，以使用更高级的统计指标进行关联。这种分析标志着从视频世界跨越到统计推断和机器学习领域——这正是计算机视觉的精髓。 By following the steps laid out in this tutorial and leveraging the tools presented, you can minimize the obstacles (large file sizes or complicated video formats) that we commonly associate with decompressing video and interpreting RGB pixels. And when you’ve simplified working with video and computer vision, you can better focus on what matters: delivering intelligent and robust [video capabilities](https://www.toptal.com/php/online-video-wowza-amazon-elastic-transcoder) in your applications. 通过遵循本教程中列出的步骤并利用所介绍的工具，你可以最大限度地减少通常与解压视频和解释 RGB 像素相关的障碍（如大文件尺寸或复杂的视频格式）。当你简化了视频和计算机视觉的处理工作后，你就能更好地专注于重要的事情：在你的应用中交付智能且强大的[视频功能](https://www.toptal.com/php/online-video-wowza-amazon-elastic-transcoder)。 --- *The editorial team of the Toptal Engineering Blog extends its gratitude to [Martin Goldberg](https://www.toptal.com/resume/martin-goldberg-ph-d) for reviewing the code samples and other technical content presented in this article.* *Toptal 工程博客编辑团队向 [Martin Goldberg](https://www.toptal.com/resume/martin-goldberg-ph-d) 致谢，感谢他审阅本文中的代码示例和其他技术内容。*