本地顯卡部署olmocr Docker鏡像+本地預訓練模型

1.0 allenai/olmocr

git clone https://github.com/CaldeDaniele/olmo-ocr-docker

　　下載olmocr項目。官網：https://github.com/allenai/olmocr

2.0 下載大模型

cd /home/user/modal
git lfs clone https://huggingface.co/allenai/olmOCR-7B-0225-preview
git lfs clone https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct

　　olmocr是基於Qwen2.5-VL多模態視覺大模型，

3.0 修改app.py

model = Qwen2VLForConditionalGeneration.from_pretrained(
    pretrained_model_name_or_path="/modal/olmOCR-7B-0225-preview", torch_dtype=torch.bfloat16, local_files_only=True,
).eval()

　　line18~20：pretrained_model_name_or_path + local_files_only=True

processor = AutoProcessor.from_pretrained(pretrained_model_name_or_path="/modal/Qwen2.5-VL-7B-Instruct", local_files_only=True,)

　　line21：pretrained_model_name_or_path + local_files_only=True

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

　　line22：cuda:0

4.0 部署Docker鏡像

sudo docker build -t olmocr/cuda12.4:1.2 .

　　建立鏡像

sudo docker run --name olmo_orc --rm -p 7861:7861 --gpus all -v /home/user/modal:/modal -v ~/PycharmProjects/olmo-ocr-docker/app:/app olmocr/cuda12.4:1.2

　　啟動鏡像

需修改參數	值	Description
-v	/home/user/modal:/modal	下載大模型
-v	~/PycharmProjects/olmo-ocr-docker/app:/app	用數據卷/app

 

5.0 總結

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacity of 9.77 GiB of which 24.44 MiB is free. Process 1960837 has 318.00 MiB memory in use. Process 2681121 has 672.00 MiB memory in use. Process 1056088 has 214.00 MiB memory in use. Process 3551268 has 1.08 GiB memory in use. Process 3804120 has 1.73 GiB memory in use. Process 967149 has 5.71 GiB memory in use. Of the allocated memory 5.36 GiB is allocated by PyTorch, and 150.30 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

　　OOM。後來發現至少需要20G顯存。

https://github.com/allenai/olmocr/blob/main/README.md