使用rdfind删除重复文件

很多工具，为了安全和方便，自带了很多库文件。安装软件多之后，系统中有很多重复文件。可以使用rdfind创建硬链接，删除重复文件，节省硬盘空间。

下面检查库libboost_system，根据inode，有六个独立的文件libboost_system.so，而且他们的MD5校验和也都一样。 反复执行命令“rdfind -makehardlinks true /opt/Xilinx/”后，所有3.6KB的文件libboost_system.so，都具有相同的inode，说明只有一份文件了。

下面是操作前的记录。简单检查，可以看到，前三行的文件大小一样，md5sum，inode不一样。

/opt$ locate libboost_system | xargs -n 1 ls -li -h
18883079 -rwxr-xr-x 1 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/DocNav/lib/lnx64.o/libboost_system.so
18628522 -rwxr-xr-x 1 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/PetaLinux_2021.2/lib/lnx64.o/libboost_system.so
18884625 -rwxr-xr-x 6 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/Vitis_2021.2/lib/lnx64.o/libboost_system.so
23734716 -rwxr-xr-x 1 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/xic/lib/lnx64.o/libboost_system.so
18884625 -rwxr-xr-x 6 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Model_Composer/2021.2/lib/lnx64.o/libboost_system.so
18884625 -rwxr-xr-x 6 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vitis/2021.2/aietools/lib/lnx64.o/libboost_system.so
18884625 -rwxr-xr-x 6 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vitis/2021.2/lib/lnx64.o/libboost_system.so
18753508 -rwxr-xr-x 1 hankf hankf 5.3K 10月 21 20:49 /opt/Xilinx/Vitis/2021.2/lib/lnx64.o/libboost_system.so.1.72.0
18884625 -rwxr-xr-x 6 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vitis_HLS/2021.2/lib/lnx64.o/libboost_system.so
19405685 -rwxr-xr-x 1 hankf hankf 17K 11月 23  2019 /opt/Xilinx/Vivado/2021.2/ids_lite/ISE/lib/lin64/libboost_system-gcc41-mt-p-38.so.1.38.0
18884625 -rwxr-xr-x 6 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vivado/2021.2/lib/lnx64.o/libboost_system.so
21511619 -rwxr-xr-x 1 hankf hankf 3.6K 10月  7 11:17 /opt/Xilinx/peta/2021.2/tools/xsct/lib/lnx64.o/libboost_system.so
21511621 -rwxr-xr-x 1 hankf hankf 5.3K 10月  7 10:40 /opt/Xilinx/peta/2021.2/tools/xsct/lib/lnx64.o/libboost_system.so.1.72.0
18875997 -rwxr-xr-x 1 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/xic/lib/lnx64.o/libboost_system.so
22556163 -rw-r--r-- 1 root root 63K 3月   6  2018 /usr/lib/x86_64-linux-gnu/libboost_system.a
22556164 lrwxrwxrwx 1 root root 25 3月   6  2018 /usr/lib/x86_64-linux-gnu/libboost_system.so -> libboost_system.so.1.65.1
22552303 -rw-r--r-- 1 root root 19K 3月   6  2018 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1

$ locate libboost_system | xargs -n 1 md5sum
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/.xinstall/DocNav/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/.xinstall/PetaLinux_2021.2/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/.xinstall/Vitis_2021.2/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/.xinstall/xic/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/Model_Composer/2021.2/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/Vitis/2021.2/aietools/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/Vitis/2021.2/lib/lnx64.o/libboost_system.so
098f53c5d0200370ef57b8bfe5f350a6  /opt/Xilinx/Vitis/2021.2/lib/lnx64.o/libboost_system.so.1.72.0
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/Vitis_HLS/2021.2/lib/lnx64.o/libboost_system.so
d9f2e3e299c3135b4a4bbca3f0aabef5  /opt/Xilinx/Vivado/2021.2/ids_lite/ISE/lib/lin64/libboost_system-gcc41-mt-p-1_38.so.1.38.0
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/Vivado/2021.2/lib/lnx64.o/libboost_system.so
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/peta/2021.2/tools/xsct/lib/lnx64.o/libboost_system.so
098f53c5d0200370ef57b8bfe5f350a6  /opt/Xilinx/peta/2021.2/tools/xsct/lib/lnx64.o/libboost_system.so.1.72.0
5b70ad53fe2f665e8dd02cebe56a0126  /opt/Xilinx/xic/lib/lnx64.o/libboost_system.so
03bfbbff05f7d342169d9fc4c967fa01  /usr/lib/x86_64-linux-gnu/libboost_system.a
316aef962f4e810b304d7d38a652036b  /usr/lib/x86_64-linux-gnu/libboost_system.so
316aef962f4e810b304d7d38a652036b  /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1

下面是操作后的记录。可以看到，前三行的文件大小一样，inode一样。

$ locate libboost_system | xargs -n 1 ls -li -h
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/DocNav/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/PetaLinux_2021.2/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/Vitis_2021.2/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/.xinstall/xic/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Model_Composer/2021.2/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vitis/2021.2/aietools/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vitis/2021.2/lib/lnx64.o/libboost_system.so
18753508 -rwxr-xr-x 2 hankf hankf 5.3K 10月 21 20:49 /opt/Xilinx/Vitis/2021.2/lib/lnx64.o/libboost_system.so.1.72.0
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vitis_HLS/2021.2/lib/lnx64.o/libboost_system.so
19405685 -rwxr-xr-x 1 hankf hankf 17K 11月 23  2019 /opt/Xilinx/Vivado/2021.2/ids_lite/ISE/lib/lin64/libboost_system-gcc41-mt-p-1_38.so.1.38.0
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/Vivado/2021.2/lib/lnx64.o/libboost_system.so
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/peta/2021.2/tools/xsct/lib/lnx64.o/libboost_system.so
18753508 -rwxr-xr-x 2 hankf hankf 5.3K 10月 21 20:49 /opt/Xilinx/peta/2021.2/tools/xsct/lib/lnx64.o/libboost_system.so.1.72.0
18875997 -rwxr-xr-x 11 hankf hankf 3.6K 10月 21 21:06 /opt/Xilinx/xic/lib/lnx64.o/libboost_system.so
22556163 -rw-r--r-- 1 root root 63K 3月   6  2018 /usr/lib/x86_64-linux-gnu/libboost_system.a
22556164 lrwxrwxrwx 1 root root 25 3月   6  2018 /usr/lib/x86_64-linux-gnu/libboost_system.so -> libboost_system.so.1.65.1
22552303 -rw-r--r-- 1 root root 19K 3月   6  2018 /usr/lib/x86_64-linux-gnu/libboost_system.so.1.65.1

为了简化操作，准备了脚本，执行上述操作。

#!/bin/bash

echo -e "Begin to execute $0 in dir:\n"
pwd

remove_duplicate_dir="/opt/Xilinx/"

if [ "$1" != "" ]; then
    echo "No directory is specified in the command."
    echo "Use default directory:$remove_duplicate_dir."
    remove_duplicate_dir=$1
fi

echo "Try to remove Xilinx tool duplicate files in  directory:$remove_duplicate_dir."

if [ ! -d  $remove_duplicate_dir ]; then 
    echo "No directory: $remove_duplicate_dir. "
    echo "Exit."
    exit 1
fi 

echo "Content in its parent directory: "
ls -l $remove_duplicate_dir/../

echo "Content in target directory: "
ls -l $remove_duplicate_dir/

cd $remove_duplicate_dir/../
rm -f results-*.txt 
for i in {1..100}; do

    echo "Remove Xilinx tool duplicate files for the No.$i time in directory /opt/Xilinx/."
    rdfind -makehardlinks true  $remove_duplicate_dir
    du -h --max-depth=3 | grep G
    df -h | grep -v "/snap" | grep -v "/run"
    mv results.txt results-$i.txt 
done