从APNIC提取IP信息

https://blog.csdn.net/nullzeng/article/details/17538009

Apnic介绍
简而言之，Apnic是全球5个地区级的Internet注册机构（RIR）之一，负责亚太地区的以下一些事务：
（1）分配IPv4和IPv6地址空间，AS号
（2）为亚太地区维护Whois数据库
（3）反向DNS指派
（4）在全球范围内作为亚太地区的Internet社区的代表
更详细的Apnic介绍文档请参见：
http://www.apnic.net/about-APNIC/organization
或者
http://en.wikipedia.org/wiki/Asi ... _Information_Centre

相关网址：
https://www.apnic.net/
https://www.nro.net/

Apanic IP地址分配信息总表
Apanic提供了每日更新的亚太地区IPv4，IPv6，AS号分配的信息表，访问url是
http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest
该文件的格式与具体内容参加
ftp://ftp.apnic.net/pub/apnic/stats/apnic/README.TXT

通过该文件我们能够得到Apnic辖下IPv4地址空间的分配情况。但从该文件中我们只能得到
apnic|TW|asn|1768|2|20020801|allocated
apnic|CN|ipv4|1.2.2.0|256|20110331|assigned
等级机构|获得该IP段的国家/组织|资源类型|起始IP|IP段长度|分配日期|分配状态

Format:
        registry|cc|type|start|value|date|status[|extensions...]
    Where:
        registry The registry from which the data is taken.
                For APNIC resources, this will be:
                 apnic
        cc ISO 3166 2-letter code of the organisation to
         which the allocation or assignment was made. 
         May also include the following non-ISO 3166
         code: 
         
                 AP - networks based in more than one 
                 location in the Asia Pacific region
        type Type of Internet number resource represented
                in this record. One value from the set of 
                defined strings:
                 {asn,ipv4,ipv6}
        start In the case of records of type 'ipv4' or
                'ipv6' this is the IPv4 or IPv6 'first
                address' of the range.
        value In the case of IPv4 address the count of
                hosts for this range. This count does not 
                have to represent a CIDR range.

但关于省、运营商、地址、路由等信息仍然无法获取
信息总表的扩展
在获取上述总表的基础上，我们通过不断的进行whois，就能得到更多的信息
例如，这对起始IP 1.4.4.0，我们运行whois 1.4.4.0，得到如下的信息

whois 1.4.4.0
[Querying whois.arin.net]
[Redirected to whois.apnic.net]
[Querying whois.apnic.net]
[whois.apnic.net]
% [whois.apnic.net node-1]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html

inetnum: 1.4.4.0 – 1.4.4.255
netname: KNET
descr: KNET Techonlogy (BeiJing) Co.,Ltd.
descr: 4,South 4th treet, Zhongguancun,Haidian District,Beijing
country: CN
admin-c: HS1165-AP
tech-c: WL1818-AP
mnt-by: MAINT-CNNIC-AP
mnt-routes: MAINT-CNNIC-AP
mnt-irt: IRT-CNNIC-CN
status: ASSIGNED PORTABLE
changed: hm-changed@apnic.net 20110331
source: APNIC

person: Hanhui Sun
address: 4,South 4th treet, Zhongguancun,Haidian District,Beijing
country: CN
phone: +86-010-58813102
fax-no: +86-010-58812666-126
e-mail: sunhanhui@knet.cn
nic-hdl: HS1165-AP
mnt-by: MAINT-CNNIC-AP
changed: ipas@cnnic.net.cn 20110315
source: APNIC

person: Wenzhe Lu
address: 4,South 4th treet, Zhongguancun,Haidian District,Beijing
country: CN
phone: +86-010-58813017
fax-no: +86-010-58812666-126
e-mail: lwz@knet.cn
nic-hdl: WL1818-AP
mnt-by: MAINT-CNNIC-AP
changed: ipas@cnnic.net.cn 20110315
source: APNIC

其中我们比较关注的信息有：
inetnum：起始和终止IP地址
netname：网络名（运营商/Org等）
country：国家
descr：描述信息（往往含有地域、ISP等信息）
address: 地址信息

通过不断对上述的信息总表的起始IP地址进行whois，我们可以得到详细的IP地址对应的地域、ISP、Org等信息

实现时，在对上述信息总表的起始IP地址进行whois时，需要考虑到两种情况：
（1）对信息总表起始IP地址进行whois时，得到的inetnum中包含的IP地址数目经常会小于信息总表中对应的IP数，因此往往需要多次whois才能得到最终的结果
（2）对信息总表起始IP地址进行whois时，得到的inetnum中的起始IP有可能会小于信息总表中的起始IP，这时也需要进行处理，否则可能使得最终的结果出现IP段交叉的情况

Apnic whois库处理流程总结
为了获取Apnic whois库的完整结果，可以遵循下面的流程：
（1）通过http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest获取Apanic负责分配的所有IPv4地址段的起始地址和长度
（2）通过对上述IP段进行whois操作，得到该IP段的相关信息
（3）whois结果的信息提取、处理和规范化。常见的处理包括国家、省、市、ISP、机构等信息的提取，英文与中文的转化等

通过上述三步，我们就能够得到whois库的结果

#!/bin/bash
#include
#define
TMP=/tmp/apnic_file
FILE=$1
DIR=APNIC
PROG1="whois.sh"
PROG2="merge.sh"
THREAD=30
#function
#main
if [[ -z $1 ]]; then
    echo "$(basename $0) "
    exit
fi
which whois &>/dev/null
if [[ $? -ne 0 ]]; then
    echo "Please install whois(apt-get install whois)"
    exit 1
fi
rm -rf $DIR [0-9]* $PROG 2>/dev/null
tail -n +$(awk '/^#!/{if(i){print NR;exit}i++}' $0) "$0" > $PROG1
tail -n +$(awk '/^#!/{if(i==2){print NR;exit}i++}' $0) "$0" > $PROG2
chmod +x $PROG1 $PROG2 2>/dev/null
awk -F"[|]" '/apnic\|CN\|ipv4\|/{print $4,$5}' $FILE > $TMP
awk -vp=$THREAD 'BEGIN{while(getline i}' $TMP
for BLOCK in $(ls [0-9]*); do
    ./$PROG1 $BLOCK &
done
wait
echo "The whois query is completed"
mkdir -p $DIR/CHINANET $DIR/UNICOM
for dir in $(ls [0-9]*); do
    cd $DIR/$dir
    for i in $(find . -type f); do
        file=${i#*/}
        cat $file >> ../$file
    done
    cd ../..
done
rm -rf $DIR/[0-9]* 2>/dev/null
rm -rf [0-9]* 2>/dev/null
for file in $(find $DIR -type f); do
    case $file in
        *bug|*error|*print)
            echo "ignore $file"
            ;;
        *)
            ./merge.sh $file
            ;;
    esac
done
rm $PROG1 $PROG2 2>/dev/null
rm /tmp/whois_*
echo "$(basename $0) Completed"
exit 0
#!/bin/bash
#################################################
# 主机数 = 2 ^ (32-mask)
# 所以以2为底取主机数的对数, 就是该mask的值.
#
#MASK=$(cat <<EOF | bc | tail -1
#pow=32;
#define log2(x) {
# if (x<2) return (pow);
# pow--;
# return(log2(x/2));
#}
#log2($CNT)
#EOF
#)
#MASK=$(pow=32;for((i=$CNT;i>1;i=i/2)); do :; ((pow--)); done;echo $pow)
#MASK=$(awk -v c=$CNT 'function log2(x){if(x<2)return(pow);pow--;return(log2(x/2))}BEGIN{pow=32;print log2(c)}')
#################################################
#include
#define
FILE=$1
WHOIS=/tmp/whois_$FILE
DIR=APNIC/$FILE
#function
province(){
    case $4 in
        FJ*|fj*|FuZhou|fuzhou)
            echo "$2/$3" >> $DIR/$1/fujian
            ;; 
        GD*)
            echo "$2/$3" >> $DIR/$1/guangdong
            ;;
        NM)
            echo "$2/$3" >> $DIR/$1/neimenggu
            ;;
        GZ)
            echo "$2/$3" >> $DIR/$1/guizhou
            ;;
        NX|NINGXIA)
            echo "$2/$3" >> $DIR/$1/ningxia
            ;;
        HL*)
            echo "$2/$3" >> $DIR/$1/heilongjiang
            ;;
        SX|TY)
            echo "$2/$3" >> $DIR/$1/shanxi
            ;;
        SN|SHAANXI)
            echo "$2/$3" >> $DIR/$1/shannxi
            ;;
        HA)
            echo "$2/$3" >> $DIR/$1/henan
            ;;
        BJ)
            echo "$2/$3" >> $DIR/$1/beijing
            ;;
        CQ)
            echo "$2/$3" >> $DIR/$1/chongqing
            ;;
        KM|YN)
            echo "$2/$3" >> $DIR/$1/yunan
            ;;
        HB|DIAQOS1)
            echo "$2/$3" >> $DIR/$1/hubei
            ;;
        XZ)
            echo "$2/$3" >> $DIR/$1/xizang
            ;;
        HE)
            echo "$2/$3" >> $DIR/$1/hebei
            ;;
        SD)
            echo "$2/$3" >> $DIR/$1/shandong
            ;;
        GS)
            echo "$2/$3" >> $DIR/$1/gansu
            ;;
        AH|Anhui)
            echo "$2/$3" >> $DIR/$1/anhui
            ;;
        LN)
            echo "$2/$3" >> $DIR/$1/liaoning
            ;;
        HN|HUNAN)
            echo "$2/$3" >> $DIR/$1/hunan
            ;;
        JS|SZ)
            echo "$2/$3" >> $DIR/$1/jiangsu
            ;;
        XJ)
            echo "$2/$3" >> $DIR/$1/xinjiang
            ;;
        JX)
            echo "$2/$3" >> $DIR/$1/jiangxi
            ;;
        JL)
            echo "$2/$3" >> $DIR/$1/jilin
            ;;
        SH|INSURANCE)
            echo "$2/$3" >> $DIR/$1/shanghai
            ;;
        GX)
            echo "$2/$3" >> $DIR/$1/guangxi
            ;;
        HI)
            echo "$2/$3" >> $DIR/$1/hainan
            ;;
        TJ)
            echo "$2/$3" >> $DIR/$1/tianjin
            ;;
        SC)
            echo "$2/$3" >> $DIR/$1/sichuan
            ;;
        QH|GEERMU)
            echo "$2/$3" >> $DIR/$1/qinghai
            ;;
        HK)
            echo "$2/$3" >> $DIR/$1/xianggang
            ;;
        ZJ)
            echo "$2/$3" >> $DIR/$1/zhejiang
            ;;
        *)
            echo "$2/$3" >> $DIR/$1/_other
            ;;
    esac
}
whois_query(){
    echo -e "Process[$FILE]\twhois [$1]"
    whois $1 > $WHOIS
    return $?
}
ntoa(){
    awk '{c=256;print int($0/c^3)"."int($0%c^3/c^2)"."int($0%c^3%c^2/c)"."$0%c^3%c^2%c}' <<<$1
}
aton(){
    awk '{c=256;split($0,ip,".");print ip[4]+ip[3]*c+ip[2]*c^2+ip[1]*c^3}' <<<$1
}
add_network(){
    echo "$2/$3 $1 $4" >> $DIR/print
    case $1 in
        CHINANET)
            province $1 $2 $3 $4
            ;;
        UNICOM)
            province $1 $2 $3 $4
            ;;
        CMNET)
            echo "$2/$3" >> $DIR/$1
            ;;
        CTTNET)
            echo "$2/$3" >> $DIR/$1
            ;;
        CERNET)
            echo "$2/$3" >> $DIR/$1
            ;;
        *)
            echo "$2/$3 $1 $4" >> $DIR/bug
            echo "$2/$3" >> $DIR/others
            ;;
    esac
}
bool_sub(){
    START=$HEAD
    MASK=32
    local NET
    local i=$((~0))
    while [[ $START -lt $TAIL ]]; do
        ((i<<=1))
        NET=$((HEAD&i))
        START=$((~(NET^i)))
        ((MASK--))
        if [[ $START -eq $TAIL ]]; then
            return 0
        fi
    done
    return 1
}
do_whois(){
    local NET
    local i=$((~0))
    local j
    eval $(awk 'BEGIN{i=256}/^inetnum:/{split($4,ipe,".");ipt=ipe[4]+ipe[3]*i+ipe[2]*i^2+ipe[1]*i^3}END{print "TAIL="ipt}' $WHOIS)
    eval $(awk '/^$/{if(i)exit;}\
                /^netname:/{i++;split($2,a,"-");isp=a[1];area=a[2];if(isp=="CNC"||isp=="UNI"||isp=="uni")isp="UNICOM";\
                    if((isp=="UNICOM"&&length(area)) || (isp=="CHINANET"&&length(area)))exit}\
                /^mnt-by:.*CNCGROUP/{n=split($2,a,"-");isp="UNICOM";for(x=1;x<=n;x++){if(a[x]=="CNCGROUP"){area=a[x+1];break}};exit}\
                /^mnt-by:.*CHINANET/{n=split($2,a,"-");isp="CHINANET";for(x=1;x<=n;x++){if(a[x]=="CHINANET"){area=a[x+1];break}};exit}\
                /^mnt-by:.*CERNET/{n=split($2,a,"-");isp="CERNET";for(x=1;x<=n;x++){if(a[x]=="CERNET"){area=a[x+1];break}};exit}\
                /^mnt-by: *MAINT-CN-SNXIAN/{isp="CHINANET";area="SN";exit}\
                /^netname: *guangzhou-.*-corp/{isp="UNICOM";area="GD";exit}\
                /^mnt-lower:.*CERNET/{isp="CERNET";exit}\
                /^mnt-lower:.*CHINANET/{n=split($2,a,"-");isp="CHINANET";for(x=1;x<=n;x++){if(a[x]=="CHINANET"){area=a[x+1];break}};exit}\
                END{print "ISP="isp";AREA="area}' $WHOIS)
    HEAD=$(aton $IP)
    bool_sub
    if [[ $? -eq 0 ]]; then
        add_network $ISP $IP $MASK $AREA
    else
        j=$((32-MASK))
        ((i<<=j))
        while [[ $NET -ne $HEAD ]]; do
            ((i>>=1))
            NET=$((HEAD&i))
            ((MASK++))
        done
        IP=$(ntoa $HEAD)
        add_network $ISP $IP $MASK $AREA
        TAIL=$((~(NET^i)))
        ((TAIL++))
        IP=$(ntoa $TAIL)
        whois_query $IP
        if [[ $? -eq 0 ]]; then
            do_whois
        else
            echo "$IP/$MASK" >> $DIR/error
        fi
    fi
}
#main
FILE=$1
rm -rf $DIR 2>/dev/null
mkdir -p $DIR/CHINANET $DIR/UNICOM
while read IP CNT; do
    START=$(aton $IP)
    END=$((START+CNT-1))
    TAIL=0
    MASK_MAX=$(pow=32;for((i=$CNT;i>1;i>>=1)); do :; ((pow--)); done;echo $pow)
    while [[ $TAIL -lt $END ]]; do
        whois_query $IP
        if [[ $? -eq 0 ]]; then
            do_whois
            ((TAIL++))
            IP=$(ntoa $TAIL)
        else
            echo "$IP/$MASK" >> $DIR/error
        fi
    done
done < $FILE
exit 0
#!/bin/bash
if [[ -z $1 ]]; then
    echo "file not found"
    exit
fi
if [[ ! -f $1 ]]; then
    echo "$1 is not exsit"
    exit
fi
TMP=/tmp/merge
while :; do
    awk -F"/" '
    function ntoa(n){c=256;return int(n/c^3)"."int(n%c^3/c^2)"."int(n%c^3%c^2/c)"."n%c^3%c^2%c}
    function aton(d){c=256;split(d,ip,".");return ip[4]+ip[3]*c+ip[2]*c^2+ip[1]*c^3}
    function ntobc(a,b){e=compl(0);f=lshift(e,32-b);s=and(a,f);return compl(xor(s,f))}
    function ntosub(j,k){g=compl(0);h=lshift(g,32-k);return and(j,h)}
    NR>1{
     if($1==ntoa(bc+1) && $2==mask && ip_int==ntosub(ip_int,$2-1)){
            mask=$2-1;bc=ntobc(ip_int,mask);
            next;
     }else{
            print add"/"mask
     }
    }
    {add=$1;ip_int=aton($1);mask=$2;bc=ntobc(ip_int,mask)}
    END{print add"/"mask}
    ' $1 > $TMP
    FILE_SIZE=$(ls -l $1 2>/dev/null | awk '{print $5}')
    TMP_SIZE=$(ls -l $TMP 2>/dev/null | awk '{print $5}')
    if [[ $FILE_SIZE -eq $TMP_SIZE ]]; then
        break
    fi
    cp $TMP $1
done
exit 0

==============================

#!/bin/bash
# download from apnic
rm -f delegated-apnic-latest
wget http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest

# IPs allocated to china.
grep 'apnic|CN|ipv4|' delegated-apnic-latest | cut -f 4 -d'|' > delegated-apnic-CN

# get detail of echo IP from apnic database.
rm -f apnic_CN.txt
while read ip
do
    # query apnic database
    echo "query who is $ip"
    whois -h whois.apnic.net $ip > tmp.txt
    grep inetnum  tmp.txt >> apnic_CN.txt          # IP range
    grep netname  tmp.txt >> apnic_CN.txt          # netname which include sp information  
    grep descr    tmp.txt >> apnic_CN.txt          # description which include province information
    echo ""  >> apnic_CN.txt           
done < delegated-apnic-CN

# clean up
rm -f tmp.txt
rm -f delegated-apnic-latest
rm -f delegated-apnic-CN

APNIC是管理亚太地区IP地址分配的机构，它有着丰富准确的IP地址分配库，同时这些信息也是对外公开的！

一、在Linux下获得一些电信运营商的IP地址段的情况：

1、编译安装软件

shell> wget http://ftp.apnic.net/apnic/dbase/tools/ripe-dbase-client-v3.tar.gz

shell> tar xzvf ripe-dbase-client-v3.tar.gz

shell> cd whois-3.1

shell> ./configure

shell> make

2、开始获取IP地址段

中国网通：

shell> ./whois3 -h whois.apnic.net -l -i mb MAINT-CNCGROUP > /root/cncrange.txt

中国电信：

shell> ./whois3 -h whois.apnic.net -l -i mb MAINT-CHINANET > /root/chinanetrange.txt

中国铁通：

shell> ./whois3 -h whois.apnic.net -l -i mb MAINT-CN-CRTC > /root/crtcrange.txt

专门获取中国网通路由表

shell> ./whois3 -h whois.apnic.net -l -i mb MAINT-CNCGROUP-RR > /root/cncrr.txt

注：MAINT-CNCGROUP 为中国网通在apnic的自治系统号

如果想得到具体的服务商比如山东网通的就用MAINT-CNCGROUP-SD

具体查看http://www.apnic.net/index.html

其他查询选项请查看 http://www.apnic.net/apnic-info/whois_search

3、过滤提取IP段

以中国网通为例:

shell> cat cncip.txt | grep “inetnum” | sed ‘s/inetnum: //g’

中国网通路由表

shell> cat /root/cnciprt.txt | grep route | sed ‘s/route://g’ | sed ‘s/. //g’

用下面的工具进行整理更方便LFHIPConverter subnettools102

如果做智能dns的话用下面的方法：

获取中国CNC网通地址列表字段

./whois3 -h whois.apnic.net -l -i mb MAINT-CNCGROUP | grep “descr” | grep “Reverse” | awk -F “for” ‘{if ($2!=”") print $2}’| sort -n | awk ‘BEGIN{print “acl /”CNC/” ‘{‘”}{print $1″;”}END{print “‘}’;”}’ > /var/named/cnc_acl.conf

获取中国电信CTC 地址列表字段

./whois3 -h whois.apnic.net -l -i mb MAINT-CHINANET | grep “descr” | grep “Reverse” | awk -F “for” ‘{if ($2!=”") print $2}’| sort -n | awk ‘BEGIN{print “acl /”CTC/” ‘{‘”}{print $1″;”}END{print “‘}’;”}’ > /var/named/ctc_acl.conf

二、使用ripe-whois3获得电信，网通等运营商的ip地址（以下方法没有测试，摘抄于网络）

Linux中下载安装

http://ftp.apnic.net/apnic/dbase/tools/ripe-whois-tools-2.3.tar.gz

tar xzvf ripe-whois-tools-2.3.tar.gz

cd /usr/ports/net/ripe-whois3

make install clean

rehash

查询:

中国网通: whois3 -h whois.apnic.net -l -i mb MAINT-CNCGROUP

中国电信: whois3 -h whois.apnic.net -l -i mb MAINT-CHINANET

中国铁通: whois3 -h whois.apnic.net -l -i mb MAINT-CNC-CRTC

三、自动获取任意国家地区ip段（需要自己根据实际情况修改）

#!/bin/sh
FILE=./ip_apnic
rm -f $FILE
wget http://ftp.apnic.net/apnic/stats/apnic/delegated-apnic-latest -O $FILE
grep ‘apnic|CN|ipv4|’ $FILE | cut -f 4,5 -d’|'|sed -e ‘s/|/ /g’ | while read ip cnt
do
echo $ip:$cnt
mask=$(cat << EOF | bc | tail -1
pow=32;
define log2(x) {
if (x<=1) return (pow);
pow–;
return(log2(x/2));
}
log2($cnt)
EOF)
echo $ip/$mask>> cn.net
NETNAME=`whois $ip@whois.apnic.net | sed -e ‘/./{H;$!d;}’ -e ‘x;/netnum/!d’ |grep ^netname | sed -e ‘s/.*:      /(.*/)//1/g’ | sed -e ‘s/-.*//g’`
case $NETNAME in
CHINANET|CNCGROUP)
echo $ip/$mask >> $NETNAME
;;
#如果你还要其他 ISP , 请在这边加上去即可,透过 apnic whois , 你可以知道他的 NETNAME OTHER_NETNAME_here)
;;
esac
done

================== End

posted @ 2018-08-01 14:27 lsgxeva 阅读(6413) 评论(0) 编辑收藏举报

刷新页面返回顶部

云水

MyGitee - https://gitee.com/lsgx/
MyGithub - https://github.com/lsgxeva/
Download - https://d.serctl.com
Proxy - tcp://frpgz1.idcfengye.com:10000

从APNIC提取IP信息

从APNIC提取IP信息

公告

云水

MyGitee - https://gitee.com/lsgx/ MyGithub - https://github.com/lsgxeva/ Download - https://d.serctl.com Proxy - tcp://frpgz1.idcfengye.com:10000

从APNIC提取IP信息

从APNIC提取IP信息

公告

MyGitee - https://gitee.com/lsgx/
MyGithub - https://github.com/lsgxeva/
Download - https://d.serctl.com
Proxy - tcp://frpgz1.idcfengye.com:10000