webrtc笔记(5): 基于kurento media server的多人视频聊天示例

这是kurento tutorial中的一个例子(groupCall)，用于多人音视频通话，效果如下：

登录界面：

聊天界面：

运行方法：

2、idea里启用这个项目

3、浏览器里输入https://localhost:8443/ 输入用户名、房间号，然后再开一个浏览器tab页，输入一个不同的用户名，房间号与第1个tab相同，正常情况下，这2个tab页就能聊上了，还可以再加更多tab模拟多人视频(注：docker容器性能有限，mac本上实测，越过4个人，就很不稳定了）

下面是该项目的一些代码和逻辑分析：

一、主要模型的类图如下：

点击查看原图

UserSession类：代表每个连接进来的用户会话信息。

Room类：即房间，1个房间可能有多个UserSession实例。

RoomManager类：房间管理，用于创建或销毁房间。

UserRegistry类：用户注册类，即管理用户。

二、主要代码逻辑：

1、创建房间入口

public Room getRoom(String roomName) {
  log.debug("Searching for room {}", roomName);
  Room room = rooms.get(roomName);
 
  if (room == null) {
    log.debug("Room {} not existent. Will create now!", roomName);
    room = new Room(roomName, kurento.createMediaPipeline());
    rooms.put(roomName, room);
  }
  log.debug("Room {} found!", roomName);
  return room;
}

注：第7行，每个房间实例创建时，都绑定了一个对应的MediaPipeline（用于隔离不同房间的媒体信息等）

2、创建用户实例入口

public UserSession(final String name, String roomName, final WebSocketSession session,
                   MediaPipeline pipeline) {
 
    this.pipeline = pipeline;
    this.name = name;
    this.session = session;
    this.roomName = roomName;
    this.outgoingMedia = new WebRtcEndpoint.Builder(pipeline).build();
 
    this.outgoingMedia.addIceCandidateFoundListener(event -> {
        JsonObject response = new JsonObject();
        response.addProperty("id", "iceCandidate");
        response.addProperty("name", name);
        response.add("candidate", JsonUtils.toJsonObject(event.getCandidate()));
        try {
            synchronized (session) {
                session.sendMessage(new TextMessage(response.toString()));
            }
        } catch (IOException e) {
            log.debug(e.getMessage());
        }
    });
}

UserSession的构造函数上，把房间实例的pipeline做为入参传进来，然后上行传输的WebRtcEndPoint实例outgoingMedia又跟pipeline绑定(第8行)。这样："用户实例--pipeline实例--房间实例" 就串起来了。

用户加入房间的代码：

public UserSession join(String userName, WebSocketSession session) throws IOException {
    log.info("ROOM {}: adding participant {}", this.name, userName);
    final UserSession participant = new UserSession(userName, this.name, session, this.pipeline);
 
    //示例工程上，没考虑“相同用户名”的人进入同1个房间的情况，这里加上了“用户名重名”检测
    if (participants.containsKey(userName)) {
        final JsonObject jsonFailMsg = new JsonObject();
        final JsonArray jsonFailArray = new JsonArray();
        jsonFailArray.add(userName + " exist!");
        jsonFailMsg.addProperty("id", "joinFail");
        jsonFailMsg.add("data", jsonFailArray);
        participant.sendMessage(jsonFailMsg);
        participant.close();
        return null;
    }
 
    joinRoom(participant);
    participants.put(participant.getName(), participant);
    sendParticipantNames(participant);
    return participant;
}

原代码没考虑到用户名重名的问题，我加上了这段检测，倒数第2行代码，sendParticipantNames在加入成功后，给房间里的其它人发通知。

3、SDP交换的入口

kurento-group-call/src/main/resources/static/js/conferenceroom.js 中有一段监听websocket的代码：

ws.onmessage = function (message) {
    let parsedMessage = JSON.parse(message.data);
    console.info('Received message: ' + message.data);
 
    switch (parsedMessage.id) {
        case 'existingParticipants':
            onExistingParticipants(parsedMessage);
            break;
        case 'newParticipantArrived':
            onNewParticipant(parsedMessage);
            break;
        case 'participantLeft':
            onParticipantLeft(parsedMessage);
            break;
        case 'receiveVideoAnswer':
            receiveVideoResponse(parsedMessage);
            break;
        case 'iceCandidate':
            participants[parsedMessage.name].rtcPeer.addIceCandidate(parsedMessage.candidate, function (error) {
                if (error) {
                    console.error("Error adding candidate: " + error);
                    return;
                }
            });
            break;
        case 'joinFail':
            alert(parsedMessage.data[0]);
            window.location.reload();
            break;
        default:
            console.error('Unrecognized message', parsedMessage);
    }
}

服务端在刚才提到的sendParticipantNames后，会给js发送各种消息，existingParticipants(其它人加入)、newParticipantArrived(新人加入) 这二类消息，就会触发generateOffer，开始向服务端发送SDP

function onExistingParticipants(msg) {
    const constraints = {
        audio: true,
        video: {
            mandatory: {
                maxWidth: 320,
                maxFrameRate: 15,
                minFrameRate: 15
            }
        }
    };
    console.log(name + " registered in room " + room);
    let participant = new Participant(name);
    participants[name] = participant;
    let video = participant.getVideoElement();
 
    const options = {
        localVideo: video,
        mediaConstraints: constraints,
        onicecandidate: participant.onIceCandidate.bind(participant)
    };
    participant.rtcPeer = new kurentoUtils.WebRtcPeer.WebRtcPeerSendonly(options,
        function (error) {
            if (error) {
                return console.error(error);
            }
            this.generateOffer(participant.offerToReceiveVideo.bind(participant));
        });
 
    msg.data.forEach(receiveVideo);
}

4、服务端回应各种websocket消息

org.kurento.tutorial.groupcall.CallHandler#handleTextMessage 信令处理的主要逻辑，就在这里：

@Override
public void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
    final JsonObject jsonMessage = gson.fromJson(message.getPayload(), JsonObject.class);
 
    final UserSession user = registry.getBySession(session);
 
    if (user != null) {
        log.debug("Incoming message from user '{}': {}", user.getName(), jsonMessage);
    } else {
        log.debug("Incoming message from new user: {}", jsonMessage);
    }
 
    switch (jsonMessage.get("id").getAsString()) {
        case "joinRoom":
            joinRoom(jsonMessage, session);
            break;
        case "receiveVideoFrom":
            final String senderName = jsonMessage.get("sender").getAsString();
            final UserSession sender = registry.getByName(senderName);
            final String sdpOffer = jsonMessage.get("sdpOffer").getAsString();
            user.receiveVideoFrom(sender, sdpOffer);
            break;
        case "leaveRoom":
            leaveRoom(user);
            break;
        case "onIceCandidate":
            JsonObject candidate = jsonMessage.get("candidate").getAsJsonObject();
 
            if (user != null) {
                IceCandidate cand = new IceCandidate(candidate.get("candidate").getAsString(),
                        candidate.get("sdpMid").getAsString(), candidate.get("sdpMLineIndex").getAsInt());
                user.addCandidate(cand, jsonMessage.get("name").getAsString());
            }
            break;
        default:
            break;
    }
}

其中user.receiveVideoFrom方法，就会回应SDP

public void receiveVideoFrom(UserSession sender, String sdpOffer) throws IOException {
    log.info("USER {}: connecting with {} in room {}", this.name, sender.getName(), this.roomName);
 
    log.trace("USER {}: SdpOffer for {} is {}", this.name, sender.getName(), sdpOffer);
 
    final String ipSdpAnswer = this.getEndpointForUser(sender).processOffer(sdpOffer);
    final JsonObject scParams = new JsonObject();
    scParams.addProperty("id", "receiveVideoAnswer");
    scParams.addProperty("name", sender.getName());
    scParams.addProperty("sdpAnswer", ipSdpAnswer);
 
    log.trace("USER {}: SdpAnswer for {} is {}", this.name, sender.getName(), ipSdpAnswer);
    this.sendMessage(scParams);
    log.debug("gather candidates");
    this.getEndpointForUser(sender).gatherCandidates();
}