MOT_Challenge官方代码TrackEval介绍
MOT_Challenge官方代码TrackEval介绍
作者声明
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
原文链接：凤⭐尘 》》https://www.cnblogs.com/phoenixash/
MOTChallenge 介绍：
MOTChallenge是多目标跟踪领域最为常用的benchmark
CLEAR MOT Metrics认为一个好的多目标跟踪器应该有如下三点特性：
1.所有出现的目标都要能够及时找到（检测的性能）
2.找到目标位置要尽可能可真实目标位置一致（检测的性能）
3.保持追踪一致性，避免跟踪目标的跳变 （匹配的性能）
MOTChallenge的评价指标一共有十一个，分别是：
| Measure | Better | Perfect | Description |
|---|---|---|---|
| MOTA | higher | 100% | 跟踪的准确度，和出现FN，FP，IDs的数量负相关，可能出现负值。 |
| MOTP | higher | 100% | 跟踪的精度，GT和检测的bbox的匹配交叠 |
| IDF1 | higher | 100% | 引入track ID的F1 |
| FAF | lower | 0 | 每帧的平均误报警数 |
| MT | higher | 100% | 命中的轨迹占总轨迹的占比，定义命中的轨迹为长度小于ground truth 80%的轨迹 |
| ML | lower | 0 | 丢失的轨迹占总轨迹的占比，定义丢失轨迹为长度小于ground truth 20%的轨迹 |
| FP | lower | 0 | FP的总数量，false positives也就是误检 |
| FN | lower | 0 | FN的总数量，false negatives也就是漏检 |
| IDs | lower | 0 | ID改变的总数量 |
| Frag | lower | 0 | 轨迹被打断的总数量 |
| Hz | higher | Inf | 处理速度，不包括检测器的耗时，而且这个指标由作者提供，MOTChallenge是计算不出来的，因为递交的是offline文件。 |
TrackEval 介绍:
provides code for a number of different tracking evaluation metrics.
源码地址：https://github.com/JonathonLuiten/TrackEval
安装环境：常见的数据处理环境
The code is written 100% in python with only numpy and scipy as minimum requirements.
数据格式：官网上下载data样例，放在TrackEval/下，从样例中可以看到评估支持多种常用的跟踪数据集
TrackEval 使用 - MOTA评估
以MOTA指标为例，通过自己的验证集，介绍TrackEval的使用。
首先data文件夹下，gt包含每个数据集的标注，gt/mot_challenge中包含训练集、验证集的标注，测试集没有标注。
data
|-- gt
| |-- bdd100k
| |-- mot_challenge
| |-- ......
| `-- youtube_vis
`-- trackers
|-- bdd100k
|-- mot_challenge
|-- ......
`-- youtube_vis
data/gt/mot_challenge/seqmaps里面存储的是每个数据需要测试的视频序列名称，因为mot17包含3种检测器的视频序列，通常我们只使用一个检测器的视频序列，因为gt的标注都一样。这里如果只使用SDP检测的视频序列，当评估mot17验证集时，需要把其他检测器的视频序列名称删除。
1、修改需要评估的视频序列名称
例如：
修改源文件MOT17-train内容：data/gt/mot_challenge/seqmaps/MOT17-train.txt
name
MOT17-02-SDP
MOT17-04-SDP
MOT17-05-SDP
MOT17-09-SDP
MOT17-10-SDP
MOT17-11-SDP
MOT17-13-SDP
我的是：
name
001
002
003
004
...
011
2、添加验证集标签
因为不想过多修改原代码，这里删除MOT16-val下的文件，放入自己的验证集真值
data
|-- gt
|-- mot_challenge
|-- MOT16-val
|-- 001
| |-- gt.txt
| `-- seqinfo.ini
|-- 002
| |-- ......
|...
|-- 011
|-- gt.txt
`--seqinfo.ini
3、添加跟踪的结果
在data/trackers/mot_challenge/MOT16-val文件夹下创建yourtrack/data文件夹保存每个视频跟踪的结果。例如：验证集的结果放入data/trackers/mot_challenge/MOT16-val/ICPR2022/data
trackers
|-- mot_challenge
|-- MOT16-val
|-- ICPR2022
|-- data
|-- 001.txt
|-- ...
|-- 010.txt
`-- 011.txt
4、运行检测脚本
进入TrackEval/scripts
python run_mot_challenge.py --BENCHMARK MOT17 --TRACKERS_TO_EVAL yourtracker --METRICS CLEAR Identity --USE_PARALLEL True --GT_LOC_FORMAT {gt_folder}/{seq}/gt/gt_val_half.txt
5、修改检测脚本
也可以自己根据自己的数据集格式修改检测脚本：
TrackEval/scripts的每个文件头部都有Command Line Arguments: Defaults介绍参数含义。
在TrackEval/trackeval/datasets/文件夹下复制一个自己的数据集对应配置文件如sat_challenge_2d_box.py并修改：
1、默认配置函数：
CLASSES_TO_EVAL，BENCHMARK，SPLIT_TO_EVAL, GT_LOC_FORMAT等
@staticmethod
def get_default_dataset_config():
"""Default class config values"""
code_path = utils.get_code_path()
default_config = {
'GT_FOLDER': os.path.join(code_path, 'data/gt/mot_challenge/'), # Location of GT data
'TRACKERS_FOLDER': os.path.join(code_path, 'data/trackers/mot_challenge/'), # Trackers location
'OUTPUT_FOLDER': None, # Where to save eval results (if None, same as TRACKERS_FOLDER)
'TRACKERS_TO_EVAL': None, # Filenames of trackers to eval (if None, all in folder)
'CLASSES_TO_EVAL': ['car'], # Valid: ['pedestrian']
'BENCHMARK': 'MOT16', # Valid: 'MOT17', 'MOT16', 'MOT20', 'MOT15'
'SPLIT_TO_EVAL': 'val', # Valid: 'train', 'test', 'all'
'INPUT_AS_ZIP': False, # Whether tracker input files are zipped
'PRINT_CONFIG': True, # Whether to print current config
'DO_PREPROC': True, # Whether to perform preprocessing (never done for MOT15)
'TRACKER_SUB_FOLDER': 'data', # Tracker files are in TRACKER_FOLDER/tracker_name/TRACKER_SUB_FOLDER
'OUTPUT_SUB_FOLDER': '', # Output files are saved in OUTPUT_FOLDER/tracker_name/OUTPUT_SUB_FOLDER
'TRACKER_DISPLAY_NAMES': None, # Names of trackers to display, if None: TRACKERS_TO_EVAL
'SEQMAP_FOLDER': None, # Where seqmaps are found (if None, GT_FOLDER/seqmaps)
'SEQMAP_FILE': None, # Directly specify seqmap file (if none use seqmap_folder/benchmark-split_to_eval)
'SEQ_INFO': None, # If not None, directly specify sequences to eval and their number of timesteps
'GT_LOC_FORMAT': '{gt_folder}/{seq}/gt/gt.txt', # '{gt_folder}/{seq}/gt/gt.txt'
'SKIP_SPLIT_FOL': False, # If False, data is in GT_FOLDER/BENCHMARK-SPLIT_TO_EVAL/ and in
# TRACKERS_FOLDER/BENCHMARK-SPLIT_TO_EVAL/tracker/
# If True, then the middle 'benchmark-split' folder is skipped for both.
}
return default_config
2、初始化函数：
self.valid_classes = ['car','airplane','ship','train']
self.class_name_to_class_id = {'car': 1, 'airplane': 2, 'ship': 3, 'train': 4}
def __init__(self, config=None):
"""Initialise dataset, checking that all required files are present"""
super().__init__()
# Fill non-given config values with defaults
self.config = utils.init_config(config, self.get_default_dataset_config(), self.get_name())
self.benchmark = self.config['BENCHMARK']
gt_set = self.config['BENCHMARK'] + '-' + self.config['SPLIT_TO_EVAL']
self.gt_set = gt_set
if not self.config['SKIP_SPLIT_FOL']:
split_fol = gt_set
else:
split_fol = ''
self.gt_fol = os.path.join(self.config['GT_FOLDER'], split_fol)
self.tracker_fol = os.path.join(self.config['TRACKERS_FOLDER'], split_fol)
self.should_classes_combine = False
self.use_super_categories = False
self.data_is_zipped = self.config['INPUT_AS_ZIP']
self.do_preproc = self.config['DO_PREPROC']
self.output_fol = self.config['OUTPUT_FOLDER']
if self.output_fol is None:
self.output_fol = self.tracker_fol
self.tracker_sub_fol = self.config['TRACKER_SUB_FOLDER']
self.output_sub_fol = self.config['OUTPUT_SUB_FOLDER']
# Get classes to eval
self.valid_classes = ['pedestrian']
self.class_list = [cls.lower() if cls.lower() in self.valid_classes else None
for cls in self.config['CLASSES_TO_EVAL']]
if not all(self.class_list):
raise TrackEvalException('Attempted to evaluate an invalid class. Only pedestrian class is valid.')
self.class_name_to_class_id = {'pedestrian': 1, 'person_on_vehicle': 2, 'car': 3, 'bicycle': 4, 'motorbike': 5,
'non_mot_vehicle': 6, 'static_person': 7, 'distractor': 8, 'occluder': 9,
'occluder_on_ground': 10, 'occluder_full': 11, 'reflection': 12, 'crowd': 13}
self.valid_class_numbers = list(self.class_name_to_class_id.values())
# Get sequences to eval and check gt files exist
self.seq_list, self.seq_lengths = self._get_seq_info()
if len(self.seq_list) < 1:
raise TrackEvalException('No sequences are selected to be evaluated.')
# Check gt files exist
for seq in self.seq_list:
if not self.data_is_zipped:
curr_file = self.config["GT_LOC_FORMAT"].format(gt_folder=self.gt_fol, seq=seq)
if not os.path.isfile(curr_file):
print('GT file not found ' + curr_file)
raise TrackEvalException('GT file not found for sequence: ' + seq)
if self.data_is_zipped:
curr_file = os.path.join(self.gt_fol, 'data.zip')
if not os.path.isfile(curr_file):
print('GT file not found ' + curr_file)
raise TrackEvalException('GT file not found: ' + os.path.basename(curr_file))
# Get trackers to eval
if self.config['TRACKERS_TO_EVAL'] is None:
self.tracker_list = os.listdir(self.tracker_fol)
else:
self.tracker_list = self.config['TRACKERS_TO_EVAL']
if self.config['TRACKER_DISPLAY_NAMES'] is None:
self.tracker_to_disp = dict(zip(self.tracker_list, self.tracker_list))
elif (self.config['TRACKERS_TO_EVAL'] is not None) and (
len(self.config['TRACKER_DISPLAY_NAMES']) == len(self.tracker_list)):
self.tracker_to_disp = dict(zip(self.tracker_list, self.config['TRACKER_DISPLAY_NAMES']))
else:
raise TrackEvalException('List of tracker files and tracker display names do not match.')
for tracker in self.tracker_list:
if self.data_is_zipped:
curr_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol + '.zip')
if not os.path.isfile(curr_file):
print('Tracker file not found: ' + curr_file)
raise TrackEvalException('Tracker file not found: ' + tracker + '/' + os.path.basename(curr_file))
else:
for seq in self.seq_list:
curr_file = os.path.join(self.tracker_fol, tracker, self.tracker_sub_fol, seq + '.txt')
if not os.path.isfile(curr_file):
print('Tracker file not found: ' + curr_file)
raise TrackEvalException(
'Tracker file not found: ' + tracker + '/' + self.tracker_sub_fol + '/' + os.path.basename(
curr_file))
并在__init__.py里添加from .sat_challenge_2d_box import SatChallenge2DBox
3、修改运行脚本
复制一份TrackEval/scripts/run_mot_challenge.py为run_sat_challenge.py修改：
dataset_list = [trackeval.datasets.SatChallenge2DBox(dataset_config)]
freeze_support()
# Command line interface:
default_eval_config = trackeval.Evaluator.get_default_eval_config()
default_eval_config['DISPLAY_LESS_PROGRESS'] = False
default_dataset_config = trackeval.datasets.MotChallenge2DBox.get_default_dataset_config()
default_metrics_config = {'METRICS': ['HOTA', 'CLEAR', 'Identity'], 'THRESHOLD': 0.5}
config = {**default_eval_config, **default_dataset_config, **default_metrics_config} # Merge default configs
parser = argparse.ArgumentParser()
for setting in config.keys():
if type(config[setting]) == list or type(config[setting]) == type(None):
parser.add_argument("--" + setting, nargs='+')
else:
parser.add_argument("--" + setting)
args = parser.parse_args().__dict__
for setting in args.keys():
if args[setting] is not None:
if type(config[setting]) == type(True):
if args[setting] == 'True':
x = True
elif args[setting] == 'False':
x = False
else:
raise Exception('Command line parameter ' + setting + 'must be True or False')
elif type(config[setting]) == type(1):
x = int(args[setting])
elif type(args[setting]) == type(None):
x = None
elif setting == 'SEQ_INFO':
x = dict(zip(args[setting], [None]*len(args[setting])))
else:
x = args[setting]
config[setting] = x
eval_config = {k: v for k, v in config.items() if k in default_eval_config.keys()}
dataset_config = {k: v for k, v in config.items() if k in default_dataset_config.keys()}
metrics_config = {k: v for k, v in config.items() if k in default_metrics_config.keys()}
# Run code
evaluator = trackeval.Evaluator(eval_config)
dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
metrics_list = []
for metric in [trackeval.metrics.HOTA, trackeval.metrics.CLEAR, trackeval.metrics.Identity, trackeval.metrics.VACE]:
if metric.get_name() in metrics_config['METRICS']:
metrics_list.append(metric(metrics_config))
if len(metrics_list) == 0:
raise Exception('No metrics selected for evaluation')
evaluator.evaluate(dataset_list, metrics_list)
制作自己的MOT
https://blog.csdn.net/gubeiqing/article/details/123648141
