Python log在fastapi中的全局配置
在FastAPI中使用日志功能，实现日志切割。
原由
日志在未实现切割以及回滚时候会将所有的日志记录写入同一个地方，这样就会使日志文件特别大，如果该项目的访问量很大，然后运行时间长了之后还有可能因为日志文件过大，造成服务器因存储空间不足而宕机，所以需要将日志进行切割以及回滚。
实现
目录结构
注释：
- conf文件主要放置项目参数配置文件以及日志配置文件
- logging.ini为日志的参数配置文件
- test.ini 为项目的参数配置文件
- app主要就是项目以及一些相关设置
- api 项目的接口文件
- routers文件（使用 APIRouter是更具有层次性）
- core项目的配置文件
- defines.py中主要配置各种固定参数，或者文件名
- setting.py中为项目的配置文件
- utils项目独立函数文件
- parser.py 常用文件或字符串解析函数
- server.py为项目的服务主入口文件
- api 项目的接口文件
- logs为日志输出文件
- default.log 访问日志
- error.log 错误日志
- main.py为项目的启动文件
部分代码以及注释
-
按照时间切割
cinf/logging.ini
[loggers] keys=root,test [handlers] keys=rotatingFileHandler,streamHandler,errorHandler [formatters] keys=simpleFmt, errorFmt, consoleFmt [logger_root] level=DEBUG handlers=rotatingFileHandler,streamHandler,errorHandler [logger_test] level=DEBUG qualname=test handlers=rotatingFileHandler,streamHandler,errorHandler [handler_rotatingFileHandler] class=handlers.TimedRotatingFileHandler level=INFO formatter=simpleFmt args=(os.path.join(sys.path[0], "logs/default.log"),"M", 1, 6,'utf-8') [handler_errorHandler] class=handlers.TimedRotatingFileHandler level=ERROR formatter=errorFmt args=(os.path.join(sys.path[0], "logs/error.log"), "M", 1, 6,'utf-8') [handler_streamHandler] level=INFO class=StreamHandler formatter=consoleFmt args=(sys.stdout,) [formatter_consoleFmt] format=%(asctime)s.%(msecs)03d [%(levelname)s] [%(name)s] %(message)s [formatter_simpleFmt] format=%(asctime)s %(pathname)s(%(lineno)d): [%(levelname)s] [%(name)s] %(message)s [formatter_errorFmt] format=%(asctime)s %(pathname)s(%(lineno)d): [%(levelname)s] [%(name)s] %(message)s
args参数(filename, when, interval, backupCount, encoding) filename：日志文件的地址+文件名 when: 参数是一个字符串。表示时间间隔的单位,不区分大小写 S 秒 M 分 H 小时 D 天 W 每星期（interval==0时代表星期一） midnight 每天凌晨 interval: 时间间隔 eg:args=(os.path.join(sys.path[0], "logs/error.log"), "M", 1, 6,'utf-8') 1分钟切割一次
-
按照大小切割
cinf/logging.ini
[loggers] keys=root,test [handlers] keys=rotatingFileHandler,streamHandler,errorHandler [formatters] keys=simpleFmt, errorFmt, consoleFmt [logger_root] level=DEBUG handlers=rotatingFileHandler,streamHandler,errorHandler [logger_test] level=DEBUG qualname=test handlers=rotatingFileHandler,streamHandler,errorHandler [handler_rotatingFileHandler] class=handlers.RotatingFileHandler level=INFO formatter=simpleFmt args=(os.path.join(sys.path[0], "logs/default.log"), 'a', 2*1024*10, 5, 'utf-8') [handler_errorHandler] class=handlers.RotatingFileHandler level=ERROR formatter=errorFmt args=(os.path.join(sys.path[0], "logs/error.log"), 'a', 2*1024*10, 5, 'utf-8') [handler_streamHandler] level=INFO class=StreamHandler formatter=consoleFmt args=(sys.stdout,) [formatter_consoleFmt] format=%(asctime)s.%(msecs)03d [%(levelname)s] [%(name)s] %(message)s [formatter_simpleFmt] format=%(asctime)s %(pathname)s(%(lineno)d): [%(levelname)s] [%(name)s] %(message)s [formatter_errorFmt] format=%(asctime)s %(pathname)s(%(lineno)d): [%(levelname)s] [%(name)s] %(message)s
args参数(filename, mode, maxBytes, backupCount, encoding) filename：日志文件的地址+文件名 mode: 对日志文件的操作方式
maxBytes：日志文件的最大存储
backupCount：回滚数量
encoding：编码格式
eg:args=(os.path.join(sys.path[0], "logs/error.log"), "a", 10*2*1024, 6,'utf-8') 20KB切割一次
- 公用注释
日志等级：使用范围 FATAL：致命错误 CRITICAL：特别糟糕的事情，如内存耗尽、磁盘空间为空，一般很少使用 ERROR：发生错误时，如IO操作失败或者连接问题 WARNING：发生很重要的事件，但是并不是错误时，如用户登录密码错误 INFO：处理请求或者状态变化等日常事务 DEBUG：调试过程中使用DEBUG等级，如算法中每个循环的中间状态format参数中可能用到的格式化串： %(name)s Logger的名字 %(levelno)s 数字形式的日志级别 %(levelname)s 文本形式的日志级别 %(pathname)s 调用日志输出函数的模块的完整路径名，可能没有 %(filename)s 调用日志输出函数的模块的文件名 %(module)s 调用日志输出函数的模块名 %(funcName)s 调用日志输出函数的函数名 %(lineno)d 调用日志输出函数的语句所在的代码行 %(created)f 当前时间，用UNIX标准的表示时间的浮 点数表示 %(relativeCreated)d 输出日志信息时的，自Logger创建以 来的毫秒数 %(asctime)s 字符串形式的当前时间。默认格式是 “2003-07-08 16:49:45,896”。逗号后面的是毫秒 %(thread)d 线程ID。可能没有 %(threadName)s 线程名。可能没有 %(process)d 进程ID。可能没有 %(message)s用户输出的消息
注解：日志切割的时候，在win下会出现文件被占用的情况，从而使用切割是会出现错误
在centos下就能规避该问题，从而正常的使用切割功能
conf/test.ini
[SERVER] API_PREFIX=/api SERVER_HOST=0.0.0.0 SERVER_PORT=8080 WORKERS_COUNT=4
routers/test.py
# Copyright (c) 2020—2021 Toplinker Corporation. All rights reserved. """ 用户管理相关路由 """ from fastapi import APIRouter router = APIRouter() @router.get('/error') def error( a: int = None, b: int = None ): # if b == 0: # raise TimpHTTPException(status_code=status.HTTP_400_BAD_REQUEST,err_code=400,err_msg='b can`t be 0') return a/b
core/defines.py
# Copyright (c) 2020—2021 Toplinker Corporation. All rights reserved. """ 全局静态变量定义 """ # 配置文件路径环境变量名 ENV_TIMPSTACK_CONFIG_DIR = "SYCC_CONFIG_DIR" # 服务配置文件名 SERVER_CONFIG_FILENAME = "test.ini" # 日志配置文件名 LOGGING_CONFIG_FILENAME = "logging.ini"
core/setting.py
# Copyright (c) 2020—2021 Toplinker Corporation. All rights reserved. """ 管理TIMP-STACK后台服务配置信息 功能特点: * 基于pydantic进行有效性验证. * 全局单例模式,各模块共享配置. * 可从文件载入配置信息 """ import os from pathlib import Path from typing import List from pydantic import BaseModel, AnyHttpUrl from app.core.defines import ENV_TIMPSTACK_CONFIG_DIR, LOGGING_CONFIG_FILENAME class Settings(BaseModel): # HTTP服务的主机地址, 默认为任意地址 server_host: str = "0.0.0.0" # HTTP服务端口, 默认为8000 server_port: int = 8000 # Uvicorn进程数量 workers_count: int = 4 # 数据表前缀 table_prefix: str = "" # API路由前缀 api_prefix: str = "/api" # 跨域设置, 兼容json格式list字符串 # 如: '["http://localhost", "http://localhost:4200", "http://localhost:3000"]' backend_cors_origins: List[AnyHttpUrl] = [] # 日志配置文件 logging_config_file: str = None settings = Settings() def load_settings(): """ 初始化配置信息 由于uvicorn在开启reload模式或开启多个workers时会重新开启新的进程, 需重新载入配置信息。 要想多个进程之前用的是同一套配置文件。因此通过'TIMPSTACK_CONFIG_DIR'环境变量来传递配置文件路径。 """ app_root_path = Path(__file__).parent.parent.parent.absolute() if not os.getenv(ENV_TIMPSTACK_CONFIG_DIR): conf_dir = app_root_path.joinpath('conf') else: conf_dir = Path(os.getenv(ENV_TIMPSTACK_CONFIG_DIR)).absolute() # 日志配置文件路径 settings.logging_config_file = str(conf_dir.joinpath(LOGGING_CONFIG_FILENAME))
utils/parser.py
# Copyright (c) 2020—2021 Toplinker Corporation. All rights reserved. """ Parser - 包含常用文件或字符串解析函数 功能特点: * read_ini_file - 读取ini文件函数, 并返回为dict结构. """ from configparser import ConfigParser, ExtendedInterpolation def read_ini_file(file: str, pre_sections: dict = None, optionxform: callable = None, default_section_name: str = "DEFAULT" ) -> dict: """ 读取ini文件, 转化为dict结构; 并且可预传入一系列值初始值, 用将ini文件中`%{SomeSection:somekey}` 替换为实际传入的值。如下文件:: ``` [LOG] LOG_PATH=${APP:path}/${APP:name}/logs/log.txt ``` res = read_ini_file(file, {'APP':{'path': '/opt', 'name': 'timpstack'}}) 相当于把ini文件修改为如下: ``` [APP] root_path=/opt name=timpstack [LOG] PATH=${APP:root_path}/${APP:name}/logs/log.txt ``` #=> ress返回值为 {'LOG': {'path': '/opt/timpstack/logs/log.txg'}} :param file: ini文件路径 :param pre_sections: 预传入的Section dict对象 :param optionxform: key值转换函数, 默认为str.lower; 即将所有key转为小写 :param default_section_name: 默认Section名称, 缺省为DEFAULT :return: 返回解析后的字典对象 """ parser = ConfigParser(interpolation=ExtendedInterpolation(), default_section=default_section_name) if optionxform: parser.optionxform = optionxform ini_lines = [] if pre_sections and isinstance(pre_sections, dict): for sec_key, sec_values in pre_sections.items(): if sec_values and isinstance(sec_values, dict): ini_lines.append(f'[{sec_key}]\n') for k, v in sec_values.items(): ini_lines.append(f'{k}={v}\n') with open(file) as f: for ln in f.readlines(): ini_lines.append(ln) parser.read_string("".join(ini_lines)) result = {} for sn in parser.sections(): section = parser[sn] section_values = {} for k, v in section.items(): if v == "": v = None section_values[k] = v result[sn] = section_values return result
main.py
import fire import uvicorn import os from pathlib import Path from app.core import settings, load_settings from app.core.defines import ENV_TIMPSTACK_CONFIG_DIR class Command(object): def start(self, confdir: str = None, reload: bool = False) -> None: if not confdir: conf_directory = Path(__file__).parent.absolute().joinpath('conf') else: conf_directory = Path(confdir).absolute() os.putenv(ENV_TIMPSTACK_CONFIG_DIR, str(conf_directory)) load_settings() uvicorn.run('app.server:app', host=settings.server_host, port=settings.server_port, workers=settings.workers_count, log_config=settings.logging_config_file, reload=reload) if __name__ == '__main__': command = Command() fire.Fire(command) command.start()
异常及原因
--- Logging error --- Traceback (most recent call last): File "C:\Program Files (x86)\Python37-32\lib\logging\handlers.py", line 70, in emit self.doRollover() File "C:\Program Files (x86)\Python37-32\lib\logging\handlers.py", line 394, in doRollover self.rotate(self.baseFilename, dfn) File "C:\Program Files (x86)\Python37-32\lib\logging\handlers.py", line 111, in rotate os.rename(source, dest) PermissionError: [WinError 32] 另一个程序正在使用此文件，进程无法访问。: 'G:\\python_object\\test6_22\\logs\\default.log' -> 'G:\\python_object\\test6_22\\logs\\default.log.2021-06-22_11-26' Call stack: File "<string>", line 1, in <module> File "C:\Program Files (x86)\Python37-32\lib\multiprocessing\spawn.py", line 105, in spawn_main exitcode = _main(fd) File "C:\Program Files (x86)\Python37-32\lib\multiprocessing\spawn.py", line 118, in _main return self._bootstrap() File "C:\Program Files (x86)\Python37-32\lib\multiprocessing\process.py", line 297, in _bootstrap self.run() File "C:\Program Files (x86)\Python37-32\lib\multiprocessing\process.py", line 99, in run self._target(*self._args, **self._kwargs) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\subprocess.py", line 61, in subprocess_started target(sockets=sockets) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\server.py", line 49, in run loop.run_until_complete(self.serve(sockets=sockets)) File "C:\Program Files (x86)\Python37-32\lib\asyncio\base_events.py", line 566, in run_until_complete self.run_forever() File "C:\Program Files (x86)\Python37-32\lib\asyncio\base_events.py", line 534, in run_forever self._run_once() File "C:\Program Files (x86)\Python37-32\lib\asyncio\base_events.py", line 1771, in _run_once handle._run() File "C:\Program Files (x86)\Python37-32\lib\asyncio\events.py", line 88, in _run self._context.run(self._callback, *self._args) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 396, in run_asgi result = await app(self.scope, self.receive, self.send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__ return await self.app(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 61, in __call__ await self.app(scope, inner_receive, inner_send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\fastapi\applications.py", line 199, in __call__ await super().__call__(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\applications.py", line 111, in __call__ await self.middleware_stack(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__ await self.app(scope, receive, _send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\middleware\cors.py", line 78, in __call__ await self.app(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\exceptions.py", line 71, in __call__ await self.app(scope, receive, sender) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\routing.py", line 566, in __call__ await route.handle(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\routing.py", line 227, in handle await self.app(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\routing.py", line 44, in app await response(scope, receive, send) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\responses.py", line 136, in __call__ "headers": self.raw_headers, File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\exceptions.py", line 68, in sender await send(message) File "G:\python_object\tset_log\backend\venv\lib\site-packages\starlette\middleware\errors.py", line 156, in _send await send(message) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\middleware\message_logger.py", line 55, in inner_send await send(message) File "G:\python_object\tset_log\backend\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 467, in send status_code, Message: '%s - "%s %s HTTP/%s" %d' Arguments: ('127.0.0.1:58497', 'GET', '/api/openapi.json', '1.1', 200) 2021-06-22 13:34:37,344.344 [INFO] [uvicorn.access] 127.0.0.1:58497 - "GET /api/openapi.json HTTP/1.1" 200
原因：logs文件夹里面的文件日志文件一直在被监控着，当写的时候发现大小或者时间满足的切割的时候，文件已经被打开，进入了写入状态，然后将该文件备份的事就就会造成无法完成该动作的错误，在centos中就不会出现该问题，可以正常的切割以及回滚
样板代码离线包，百度盘地址
链接：https://pan.baidu.com/s/1Fr6AHYISyPqWZUrf-LKdEw
提取码：k6n8
