Slowfast行为识别部署

一、简介

SlowFast 是一个用于视频行为识别的深度学习模型框架，由 Facebook AI Research (FAIR) 提出。它主要用于理解视频中的动作和行为，广泛应用于视频分析、监控、运动分析等领域。

SlowFast 模型采用了一种独特的双路径网络结构，包括一个慢路径（Slow pathway）和一个快路径（Fast pathway）：

慢路径：以较低的帧率捕捉视频中的空间信息（如形状、环境等），使用较深的网络结构来提取高级的空间特征。
快路径：以较高的帧率捕捉视频中的动态信息（如快速运动、微小变化等），使用较浅的网络结构来迅速响应动态变化。

这种结构使得 SlowFast 能够同时捕获视频中的精细动态和关键空间信息，提高了对复杂动作的识别能力和效率。

使用的技术

双流网络：结合了慢速和快速处理路径，优化了信息的捕获和处理。
3D 卷积：利用 3D ConvNets 处理视频数据，能够更好地理解时间维度上的信息。
融合策略：在网络的某些阶段将快路径和慢路径的特征进行融合，以强化模型的判别能力。

SlowFast 是一个在视频理解领域具有重要意义的技术，通过其创新的网络结构设计，提供了对动态视觉内容高效且深入的分析能力。

提高精确度：通过同时分析视频的空间和时间特征，SlowFast 能够更精确地识别和理解视频中的复杂行为和动作。
效率提升：快路径的设计使得模型能够以较低的计算成本快速响应，同时慢路径确保了深入的特征提取，平衡了速度和精度。
广泛应用：可用于安全监控、体育分析、健康监测、自动驾驶等多种场景，具有广泛的应用前景和实际价值。

二、环境安装

https://github.com/facebookresearch/SlowFast


conda create -n action python==3.9activate actionconda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidiapip install numpypip install fvcorepip install simplejsonpip install PyAVpip install psutilpip install opencv-pythonpip install tensorboardpip install moviepypip install pytorchvideogit clone https://github.com/facebookresearch/detectron2.git
pip install -e detectron2

git clone https://github.com/facebookresearch/SlowFast.git
cd SlowFast
python setup.py build develop

三、文件准备

1.下载SLOWFAST_32x2_R101_50_50.pkl

SlowFast/MODEL_ZOO.md at main · facebookresearch/SlowFast · GitHub

下载第三个。

2.新建 ava.json:

{"bend/bow (at the waist)": 0,"crawl": 1,"crouch/kneel": 2,"dance": 3,"fall down": 4,"get up": 5,"jump/leap": 6,"lie/sleep": 7,"martial art": 8,"run/jog": 9,"sit": 10,"stand": 11,"swim": 12,"walk": 13,"answer phone": 14,"brush teeth": 15,"carry/hold (an object)": 16,"catch (an object)": 17,"chop": 18,"climb (e.g., a mountain)": 19,"clink glass": 20,"close (e.g., a door, a box)": 21,"cook": 22,"cut": 23,"dig": 24,"dress/put on clothing": 25,"drink": 26,"drive (e.g., a car, a truck)": 27,"eat": 28,"enter": 29,"exit": 30,"extract": 31,"fishing": 32,"hit (an object)": 33,"kick (an object)": 34,"lift/pick up": 35,"listen (e.g., to music)": 36,"open (e.g., a window, a car door)": 37,"paint": 38,"play board game": 39,"play musical instrument": 40,"play with pets": 41,"point to (an object)": 42,"press": 43,"pull (an object)": 44,"push (an object)": 45,"put down": 46,"read": 47,"ride (e.g., a bike, a car, a horse)": 48,"row boat": 49,"sail boat": 50,"shoot": 51,"shovel": 52,"smoke": 53,"stir": 54,"take a photo": 55,"text on/look at a cellphone": 56,"throw": 57,"touch (an object)": 58,"turn (e.g., a screwdriver)": 59,"watch (e.g., TV)": 60,"work on a computer": 61,"write": 62,"fight/hit (a person)": 63,"give/serve (an object) to (a person)": 64,"grab (a person)": 65,"hand clap": 66,"hand shake": 67,"hand wave": 68,"hug (a person)": 69,"kick (a person)": 70,"kiss (a person)": 71,"lift (a person)": 72,"listen to (a person)": 73,"play with kids": 74,"push (another person)": 75,"sing to (e.g., self, a person, a group)": 76,"take (an object) from (a person)": 77,"talk to (e.g., self, a person, a group)": 78,"watch (a person)": 79}

3.在SlowFast下新建input ，output文件夹，将视频1.MP4 放入input下。修改：SLOWFAST_32x2_R101_50_50.yaml：

TRAIN:ENABLE: FalseDATASET: avaBATCH_SIZE: 16EVAL_PERIOD: 1CHECKPOINT_PERIOD: 1AUTO_RESUME: TrueCHECKPOINT_FILE_PATH: '/home/ps/ycc/SlowFast-main/demo/AVA/SLOWFAST_32x2_R101_50_50.pkl'  #path to pretrain modelCHECKPOINT_TYPE: pytorch
DATA:NUM_FRAMES: 32SAMPLING_RATE: 2TRAIN_JITTER_SCALES: [256, 320]TRAIN_CROP_SIZE: 224TEST_CROP_SIZE: 256INPUT_CHANNEL_NUM: [3, 3]
DETECTION:ENABLE: TrueALIGNED: False
AVA:BGR: FalseDETECTION_SCORE_THRESH: 0.8TEST_PREDICT_BOX_LISTS: ["person_box_67091280_iou90/ava_detection_val_boxes_and_labels.csv"]
SLOWFAST:ALPHA: 4BETA_INV: 8FUSION_CONV_CHANNEL_RATIO: 2FUSION_KERNEL_SZ: 5
RESNET:ZERO_INIT_FINAL_BN: TrueWIDTH_PER_GROUP: 64NUM_GROUPS: 1DEPTH: 101TRANS_FUNC: bottleneck_transformSTRIDE_1X1: FalseNUM_BLOCK_TEMP_KERNEL: [[3, 3], [4, 4], [6, 6], [3, 3]]SPATIAL_DILATIONS: [[1, 1], [1, 1], [1, 1], [2, 2]]SPATIAL_STRIDES: [[1, 1], [2, 2], [2, 2], [1, 1]]
NONLOCAL:LOCATION: [[[], []], [[], []], [[6, 13, 20], []], [[], []]]GROUP: [[1, 1], [1, 1], [1, 1], [1, 1]]INSTANTIATION: dot_productPOOL: [[[2, 2, 2], [2, 2, 2]], [[2, 2, 2], [2, 2, 2]], [[2, 2, 2], [2, 2, 2]], [[2, 2, 2], [2, 2, 2]]]
BN:USE_PRECISE_STATS: FalseNUM_BATCHES_PRECISE: 200
SOLVER:MOMENTUM: 0.9WEIGHT_DECAY: 1e-7OPTIMIZING_METHOD: sgd
MODEL:NUM_CLASSES: 80ARCH: slowfastMODEL_NAME: SlowFastLOSS_FUNC: bceDROPOUT_RATE: 0.5HEAD_ACT: sigmoid
TEST:ENABLE: FalseDATASET: avaBATCH_SIZE: 8
DATA_LOADER:NUM_WORKERS: 2PIN_MEMORY: TrueNUM_GPUS: 1
NUM_SHARDS: 1
RNG_SEED: 0
OUTPUT_DIR: .
# TENSORBOARD:
#   MODEL_VIS:
#     TOPK: 2
DEMO:ENABLE: TrueLABEL_FILE_PATH: '/home/ps/ycc/SlowFast-main/demo/AVA/ava.json' # Add local label file path here.INPUT_VIDEO: "/home/ps/ycc/SlowFast-main/input/1.mp4"OUTPUT_FILE: "/home/ps/ycc/SlowFast-main/output/1.mp4"#WEBCAM: 0DETECTRON2_CFG: "COCO-Detection/faster_rcnn_R_50_FPN_3x.yaml"DETECTRON2_WEIGHTS: detectron2://COCO-Detection/faster_rcnn_R_50_FPN_3x/137849458/model_final_280758.pkl

四、运行以及报错处理

在根目录下运行：

python tools/run_net.py --cfg demo/AVA/SLOWFAST_32x2_R101_50_50.yaml

报错1：

(action) ps@ps:~/ycc/SlowFast-main$ python tools/run_net.py --cfg demo/AVA/SLOWFAST_32x2_R101_50_50.yaml
Traceback (most recent call last):File "/home/ps/ycc/SlowFast-main/tools/run_net.py", line 6, in <module>from slowfast.utils.misc import launch_jobFile "/home/ps/ycc/SlowFast-main/SlowFast/slowfast/utils/misc.py", line 13, in <module>import slowfast.utils.logging as loggingFile "/home/ps/ycc/SlowFast-main/SlowFast/slowfast/utils/logging.py", line 16, in <module>import slowfast.utils.distributed as duFile "/home/ps/ycc/SlowFast-main/SlowFast/slowfast/utils/distributed.py", line 13, in <module>from pytorchvideo.layers.distributed import (  # noqa
ImportError: cannot import name 'cat_all_gather' from 'pytorchvideo.layers.distributed' (/home/ps/anaconda3/envs/action/lib/python3.9/site-packages/pytorchvideo/layers/distributed.py)

解决1：下载pytorchvidio后pip安装：

GitHub - facebookresearch/pytorchvideo: A deep learning library for video understanding research.

cd pytorchvideo-main
pip install -e .

问题2：

(action) ps@ps:~/ycc/SlowFast-main$ python tools/run_net.py --cfg demo/AVA/SLOWFAST_32x2_R101_50_50.yaml
/home/ps/anaconda3/envs/action/lib/python3.9/site-packages/torchvision/transforms/_functional_video.py:6: UserWarning: The 'torchvision.transforms._functional_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms.functional' module instead.
warnings.warn(
/home/ps/anaconda3/envs/action/lib/python3.9/site-packages/torchvision/transforms/_transforms_video.py:22: UserWarning: The 'torchvision.transforms._transforms_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms' module instead.
warnings.warn(
Traceback (most recent call last):
File "/home/ps/ycc/SlowFast-main/tools/run_net.py", line 8, in
from vision.fair.slowfast.tools.demo_net import demo
ModuleNotFoundError: No module named 'vision'

解决2：将涉及vison的代码删除，修改如下：其他代码同理。

from demo_net import demo
from test_net import test
from train_net import train
from visualization import visualize