LitServe fastapi endpoint注册简单说明

LitServe 是基于fastapi 开发的，因为服务ai 模型的特殊性（一般都比较耗时，运行时间会比较长），LitServe 对于api endpoint 的注册并不是简单的处理，而是包含了一个自己的spec，基于此spec 实现灵活的定义（比如支持openai 兼容模式）

参考注册

• spec 接口定义

class LitSpec:
    """Spec will have its own encode, and decode."""

    def __init__(self):
        self._endpoints = []

        self._server: "LitServer" = None

    def setup(self, server: "LitServer"):
        self._server = server
    # 此处可以自定义endpoint  
    def add_endpoint(self, path: str, endpoint: Callable, methods: List[str]):
        """Register an endpoint in the spec."""
        self._endpoints.append((path, endpoint, methods))

    @property
    def endpoints(self):
        return self._endpoints.copy()

    @abstractmethod
    def decode_request(self, request, meta_kwargs):
        """Convert the request payload to your model input."""
        pass

    @abstractmethod
    def encode_response(self, output, meta_kwargs):
        """Convert the model output to a response payload.

        To enable streaming, it should yield the output.

        """
        pass

• 标准模式 参数不包含spec 的，会提供一个标准的 /predict endpoint，当然此地址是可以调整的 参考处理

#  具体是在LitServer init 方法中的self.setup_server()
if not self._specs:
    stream = self.lit_api.stream
    # In the future we might want to differentiate endpoints for streaming vs non-streaming
    # For now we allow either one or the other
    endpoint = self.api_path
    methods = ["POST"]
    # fastapi的add_api_route 添加路由
    self.app.add_api_route(
        endpoint,
        stream_predict if stream else predict,
        methods=methods,
        dependencies=[Depends(self.setup_auth())],
    )

• 包含spec 的处理

for spec in self._specs:
    spec: LitSpec
    # TODO check that path is not clashing
    # 不同spec 提供的能力endpoints
    for path, endpoint, methods in spec.endpoints:
        # fastapi的add_api_route 添加路由
        self.app.add_api_route(
            path, endpoint=endpoint, methods=methods, dependencies=[Depends(self.setup_auth())]
        )

• openai spec 对的注册

openai 的endpoint api 是基于spec 处理的

class OpenAISpec(LitSpec):
    def __init__(
        self,
    ):
        super().__init__()
        # register the endpoint, 实现openai 兼容的处理
        self.add_endpoint("/v1/chat/completions", self.chat_completion, ["POST"])
        self.add_endpoint("/v1/chat/completions", self.options_chat_completions, ["OPTIONS"])

说明

对于fastapi endpoint 的注册实际上就是标准的fastapi 玩法，只是LitServe 提供了一个抽象可以方便自定义以及标准化

参考资料

https://github.com/Lightning-AI/LitServe