ProductivitySuite OPEA部署
简介
GenAIExamples/ProductivitySuite at main · opea-project/GenAIExamples (github.com)
Productivity Suite, a tool designed to streamline your workflow and boost productivity! Our application leverages the power of OPEA microservices to deliver a comprehensive suite of features tailored to meet the diverse needs of modern enterprises.
现状
OPEA的E是企业级的意思。
简单的总结:
1. OPEA目前有15个例子,其中ProductivitySuite功能比较综合,其他都是简单的demo。 并没有达到号称的Enterprise的E级,距离文心一言的差距很大。
2. 大模型依赖于GPU,NPU。 但是GPU、NPU的管理和调度功能,目前没有。后面OPEA的大模型和向量数据库等组件需要这些加速技术在k8s上的管理调度。
3. OPEA只是单模态的功能。 定位企业级的话,最终实行还是得多模态multimodal吧。
4. Malini(OPEA的 chair)说了,GMC可以现实水平扩展。 可以扩缩多个应用共享一个大模型。 个人觉得也可以根据业务,扩缩大模型。
5. Rollout和rolling upgrade,这是E级基本功能吧。 目前还没有。
6. AI的评估模块是有的,但是没有尝试。 基于metric,实现业务的自动缩扩容是没有的。
NOTE:对于k8s的部署,目前只支持helm charts,GMC的部署还在开发中。 GMC的优势是可以无缝切换多种模型的。
GenAIExamples/ProductivitySuite/kubernetes/intel/README.md at main · opea-project/GenAIExamples (github.com)
In ProductivitySuite, it consists of following pipelines/examples and components:
- productivity-suite-react-ui
- chatqna
- codegen
- docsum
- faqgen
- dataprep via redis
- chat-history
- prompt-registry
- mongo
- keycloak
K8s 部署步骤
set value
git clone https://github.com/opea-project/GenAIExamples.git cd GenAIExamples/ProductivitySuite/kubernetes/intel/cpu/xeon/manifest/ export HUGGINGFACEHUB_API_TOKEN="YourOwnToken" sed -i "s/insert-your-huggingface-token-here/${HUGGINGFACEHUB_API_TOKEN}/g" *.yaml sed -i HUGGINGFACEHUB_API_TOKEN
HFTOKEN='HUGGINGFACEHUB_API_TOKEN: "'${HUGGINGFACEHUB_API_TOKEN}'"'
conf=chatqna-retriever-usvc-config
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${HFTOKEN}"'|' chatqna.yaml
sed -i -e "s|\(https_proxy:\)\s*\"\"|\1 \"$https_proxy\"|g" *.yaml sed -i -e "s|\(http_proxy:\)\s*\"\"|\1 \"$http_proxy\"|g" *.yaml sed -i -e "s|\(no_proxy:\)\s*\"\"|\1 \"$no_proxy\"|g" *.yaml
linux - find matching text and replace next line - Stack Overflow
sed -i '/https_proxy/{n;s|\(value:\)\s.*""|\1 "'"$https_proxy"'"|g}' *.yaml sed -i '/http_proxy/{n;s|\(value:\)\s.*|\1 "'"$http_proxy"'"|}' *.yaml sed -i '/no_proxy/{n;s|\(value:\)\s.*|\1 "'"$no_proxy"'"|}' *.yaml
Access Services Running on Clusters | Kubernetes
# kubectl exec -n opea-chatqna b-d34095a3-6847-559a-8a3c-692b04ec9c86-chatqna-ui-f497d75fvggs6 -- ping b-d34095a3-6847-559a-8a3c-692b04ec9c86-chatqna.opea-chatqna.svc.cluster.local
Once the service is deployed in your cluster you should be able to contact the service using its name, and Kube-DNS
will answer with the correct ClusterIP
to speak to your final pods. ClusterIPs are governed by IPTables rules created by kube-proxy on Workers that NAT your request to the final container’s IP.
The Kube-DNS naming convention is service.namespace.svc.cluster-domain.tld
and the default cluster domain is cluster.local
.
For example, if you want to contact a service called mysql
in the db
namespace from any namespace, you can simply speak to mysql.db.svc.cluster.local
.
set ENDPOINT
sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 sudo chmod a+x /usr/local/bin/yq
# sudoapt
install
build-essential
make
git curl jq yq yamllint python3-pip -y
get mega endpoint
git grep APP_.*ENDPOINT ns= name=chatqna file=chatqna.yaml svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name}.${ns:-default}.svc.cluster.local:${port} echo $url name=codegen file=codegen.yaml svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name}.${ns:-default}.svc.cluster.local:${port} echo $url # https://blog.51cto.com/hmtk520/2074219 for i in $(grep -oP "(?<=APP_BACKEND_SERVICE_ENDPOINT_).*" *.yaml); do echo $i name=${i##*:} file=${name,,}.yaml svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name,,}.${ns:-default}.svc.cluster.local:${port} echo $url
sed -i -e '/APP_BACKEND_SERVICE_ENDPOINT_'"$name"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml
done
get data prepare endpoint
name=chatqna-data-prep file=chatqna.yaml svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name}.${ns:-default}.svc.cluster.local:${port} echo $url for i in $(grep -oP "(?<=APP_)DATAPREP.*(?=_ENDPOINT)" *.yaml); do echo $i curd=${i##*:}; sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml; done
get CHAT_HISTORY endpoint
for i in $(grep -oP "(?<=APP_)CHAT_HISTORY.*(?=_ENDPOINT)" *.yaml); do echo $i; curd=${i##*:}; name=${curd%_*}; file=${name,,}.yaml; name=${name/_/-}; svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name,,}.${ns:-default}.svc.cluster.local:${port}; echo $url; sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml; done
get PROMPT endpoint
for i in $(grep -oP "(?<=APP_)PROMPT_SERVICE.*(?=_ENDPOINT)" *.yaml); do echo $i; curd=${i##*:}; curdr=${curd/SERVICE/REGISTRY}; name=${curdr%_*}; file=${name,,}.yaml; name=${name/_/-}; svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name,,}.${ns:-default}.svc.cluster.local:${port}; echo $url; sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml ; done
get keycloak
name=keycloak file=keycloak_install.yaml svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name}.${ns:-default}.svc.cluster.local:${port} echo $url sed -i -e '/APP_KEYCLOAK_SERVICE_ENDPOINT/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml
configure mirror
bash - How to add line before match within given line range/patterns using sed? - Stack Overflow
MIRROR='HF_ENDPOINT: "https://hf-mirror.com"' conf=chatqna-tgi-config sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${MIRROR}"'|' chatqna.yaml conf=chatqna-tei-config
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${MIRROR}"'|' chatqna.yaml conf=chatqna-teirerank-config
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${MIRROR}"'|' chatqna.yaml
conf=codegen-tgi-config
file=codegen.yaml
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${MIRROR}"'|' $file
conf=docsum-tgi-config
file=docsum.yaml
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${MIRROR}"'|' $file
conf=faqgen-tgi-config
file=faqgen.yaml
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${MIRROR}"'|' $file
configure model ID
conf=chatqna-tgi-config model=test_id set_model_id() { conf=$1 file=${1%%-*} sed '/name: '"${conf}"'/,/---/s|\(MODEL_ID:\).*|\1 "'"${2}"'"|' ${file}.yaml }
get model ID
json - JQ: Filtering for keys - Stack Overflow
yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_ID": .data.MODEL_ID}' *.yaml
get and set model ID
set_model_id() { if [ -z "$1" ] && [ -z "$2" ]; then yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_ID": .data.MODEL_ID}' *.yaml echo "usage:" echo " set_model_id \${ConfigMap} \${MODEL_ID}" return fi conf=$1 file=${1%%-*} sed -i '/name: '"${conf}"'/,/---/s|\(MODEL_ID:\).*|\1 "'"${2}"'"|' ${file}.yaml }
all codes
set_model_id() { if [ -z "$1" ] && [ -z "$2" ]; then yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_ID": .data.MODEL_ID}' *.yaml echo "usage:" echo " set_model_id \${ConfigMap} \${MODEL_ID}" return fi conf=$1 file=${1%%-*} sed -i '/name: '"${conf}"'/,/---/s|\(MODEL_ID:\).*|\1 "'"${2}"'"|' ${file}.yaml } set_model_mirror() { if [ -z "$1" ] ; then yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_MIRROR": .data.HF_ENDPOINT}' *.yaml echo "usage:" echo " set_model_mirror \${MODEL_MIRROR}" return fi cm=$(yq -r -o json '.| select(.data | has("MODEL_ID"))| .metadata.name' *.yaml) mirror=$1 for i in $cm; do conf=$i file=${i%%-*} echo "ConfigMap: $conf set mirror as $mirror" has_mirror=$(yq -r -o json '.| select(.metadata.name == "'"${conf}"'")| .data.HF_ENDPOINT' ${file}.yaml) if [ "$has_mirror" == "null" ]; then sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n HF_ENDPOINT: "'"${mirror}"'"|' ${file}.yaml else sed -i '/name: '"${conf}"'/,/---/s|\(HF_ENDPOINT:\).*|\1 "'"${1}"'"|' ${file}.yaml fi done } set_hf_token() { if [ -z "$1" ] ; then echo "usage:" echo " set_hf_token \${HF_TOKEN}" return fi sed -i "s/\(HF_TOKEN:\).*/\1 \"${1}\"/g" *.yaml sed -i "s/\(HUGGINGFACEHUB_API_TOKEN:\).*/\1 \"${1}\"/g" *.yaml sed -i "s/\(HUGGING_FACE_HUB_TOKEN:\).*/\1 \"${1}\"/g" *.yaml } set_https_proxy() { if [ -z "$1" ] ; then echo "usage:" echo " set_https_proxy \${https_proxy}" return fi https_proxy=$1 sed -i -e "s|\(https_proxy:\)\s*\"\"|\1 \"$https_proxy\"|g" *.yaml sed -i '/https_proxy/{n;s|\(value:\)\s.*""|\1 "'"$https_proxy"'"|g}' *.yaml } set_http_proxy() { if [ -z "$1" ] ; then echo "usage:" echo " set_http_proxy \${http_proxy}" return fi http_proxy=$1 sed -i -e "s|\(http_proxy:\)\s*\"\"|\1 \"$http_proxy\"|g" *.yaml sed -i '/http_proxy/{n;s|\(value:\)\s.*""|\1 "'"$http_proxy"'"|g}' *.yaml } set_no_proxy() { if [ -z "$1" ] ; then echo "usage:" echo " set_no_proxy \${no_proxy}" return fi no_proxy=$1 sed -i -e "s|\(no_proxy:\)\s*\"\"|\1 \"$no_proxy\"|g" *.yaml sed -i '/no_proxy/{n;s|\(value:\)\s.*""|\1 "'"$no_proxy"'"|g}' *.yaml } set_backend_service_endpoint() { for i in $(grep -oP "(?<=APP_BACKEND_SERVICE_ENDPOINT_).*" *.yaml); do echo $i name=${i##*:} file=${name,,}.yaml svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name,,}.${ns:-default}.svc.cluster.local:${port} echo $url sed -i -e '/APP_BACKEND_SERVICE_ENDPOINT_'"$name"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml done } set_dataprep_service_endpoint() { name=chatqna-data-prep file=chatqna.yaml svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name}.${ns:-default}.svc.cluster.local:${port} echo $url for i in $(grep -oP "(?<=APP_)DATAPREP.*(?=_ENDPOINT)" *.yaml); do echo $i curd=${i##*:}; sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml; done } set_chat_history_endpoint() { for i in $(grep -oP "(?<=APP_)CHAT_HISTORY.*(?=_ENDPOINT)" *.yaml); do echo $i; curd=${i##*:}; name=${curd%_*}; file=${name,,}.yaml; name=${name/_/-}; svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name,,}.${ns:-default}.svc.cluster.local:${port}; echo $url; sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml; done } set_prompt_service_endpoint() { for i in $(grep -oP "(?<=APP_)PROMPT_SERVICE.*(?=_ENDPOINT)" *.yaml); do echo $i; curd=${i##*:}; curdr=${curd/SERVICE/REGISTRY}; name=${curdr%_*}; file=${name,,}.yaml; name=${name/_/-}; svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name,,}.${ns:-default}.svc.cluster.local:${port}; echo $url; sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml ; done } set_keycloak_service_endpoint() { name=keycloak file=keycloak_install.yaml svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file) port=$(jq .spec.ports[0].port <<< $svc) url=http://${name}.${ns:-default}.svc.cluster.local:${port} echo $url sed -i -e '/APP_KEYCLOAK_SERVICE_ENDPOINT/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml } set_services_endpoint() { set_backend_service_endpoint set_keycloak_service_endpoint set_chat_history_endpoint set_prompt_service_endpoint set_dataprep_service_endpoint }
and submit a PR: https://github.com/opea-project/GenAIExamples/pull/919
deployment
kubectl apply -f .
five pods failed.
查看code
GenAIComps/comps/llms/text-generation/native/langchain/requirements.txt at 405a2fc68ae590740dd70f78df83b1ca2976360f · opea-project/GenAIComps (github.com)
需要安装 langchain_core.
docker file 采用的是python:3.11-slim
GenAIComps/comps/llms/text-generation/tgi/Dockerfile at 405a2fc68ae590740dd70f78df83b1ca2976360f · opea-project/GenAIComps (github.com)
使用同样类型的pod,发现应该是代理的问题。
kubectl exec chatqna-embedding-usvc-6556d4bbd7-9f7xk -it -- env |grep proxy
kubectl exec chatqna-embedding-usvc-6556d4bbd7-9f7xk -it -- pip install --upgrade langchain_core --proxy=http://proxy-country.company.com:8080
相应的code:
GenAIComps/comps/llms/text-generation/native/langchain/llm.py at 405a2fc68ae590740dd70f78df83b1ca2976360f · opea-project/GenAIComps (github.com)
Build Docker Images
# ghp_eKZfZasrUxblfTZBBfRpmGpQqtdbUObfUt
# ghp_kOFRJkamkPBDwrkMsdKLcoLtXzzmQlWATp
# hf_T3Bp07Akonutn2NSLgnoUs9PIKDJpB2yaPpO
https://github.com/opea-project/GenAIInfra/issues/367