ProductivitySuite OPEA部署

简介

GenAIExamples/ProductivitySuite at main · opea-project/GenAIExamples (github.com)

Productivity Suite, a tool designed to streamline your workflow and boost productivity! Our application leverages the power of OPEA microservices to deliver a comprehensive suite of features tailored to meet the diverse needs of modern enterprises.

 生产力套件,一款旨在简化您的工作流程并提高生产力的工具!我们的应用程序利用 OPEA 微服务的强大功能来提供一套全面的功能,以满足现代企业的多样化需求。

 现状 

OPEA的E是企业级的意思。

简单的总结:
1. OPEA目前有15个例子,其中ProductivitySuite功能比较综合,其他都是简单的demo。 并没有达到号称的Enterprise的E级,距离文心一言的差距很大。  
2. 大模型依赖于GPU,NPU。 但是GPU、NPU的管理和调度功能,目前没有。后面OPEA的大模型和向量数据库等组件需要这些加速技术在k8s上的管理调度。
3. OPEA只是单模态的功能。 定位企业级的话,最终实行还是得多模态multimodal吧。 
4. Malini(OPEA的 chair)说了,GMC可以现实水平扩展。 可以扩缩多个应用共享一个大模型。 个人觉得也可以根据业务,扩缩大模型。
5. Rollout和rolling upgrade,这是E级基本功能吧。 目前还没有。 
6. AI的评估模块是有的,但是没有尝试。 基于metric,实现业务的自动缩扩容是没有的。

NOTE:对于k8s的部署,目前只支持helm charts,GMC的部署还在开发中。 GMC的优势是可以无缝切换多种模型的。
GenAIExamples/ProductivitySuite/kubernetes/intel/README.md at main · opea-project/GenAIExamples (github.com)

In ProductivitySuite, it consists of following pipelines/examples and components:

- productivity-suite-react-ui
- chatqna
- codegen
- docsum
- faqgen
- dataprep via redis
- chat-history
- prompt-registry
- mongo
- keycloak

K8s 部署步骤

 set value 

git clone https://github.com/opea-project/GenAIExamples.git

cd GenAIExamples/ProductivitySuite/kubernetes/intel/cpu/xeon/manifest/


export HUGGINGFACEHUB_API_TOKEN="YourOwnToken"
sed -i "s/insert-your-huggingface-token-here/${HUGGINGFACEHUB_API_TOKEN}/g" *.yaml
sed -i HUGGINGFACEHUB_API_TOKEN
HFTOKEN='HUGGINGFACEHUB_API_TOKEN: "'${HUGGINGFACEHUB_API_TOKEN}'"'
conf=chatqna-retriever-usvc-config sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n '"${HFTOKEN}"'|' chatqna.yaml
sed -i -e "s|\(https_proxy:\)\s*\"\"|\1 \"$https_proxy\"|g" *.yaml
sed -i -e "s|\(http_proxy:\)\s*\"\"|\1 \"$http_proxy\"|g" *.yaml
sed -i -e "s|\(no_proxy:\)\s*\"\"|\1 \"$no_proxy\"|g" *.yaml

 linux - find matching text and replace next line - Stack Overflow

sed -i '/https_proxy/{n;s|\(value:\)\s.*""|\1 "'"$https_proxy"'"|g}' *.yaml
sed -i '/http_proxy/{n;s|\(value:\)\s.*|\1 "'"$http_proxy"'"|}' *.yaml
sed -i '/no_proxy/{n;s|\(value:\)\s.*|\1 "'"$no_proxy"'"|}' *.yaml

 Access Services Running on Clusters | Kubernetes

docker - How to access the service deployed on one pod via another pod in Kubernetes? - Stack Overflow

#  kubectl exec -n opea-chatqna b-d34095a3-6847-559a-8a3c-692b04ec9c86-chatqna-ui-f497d75fvggs6 -- ping b-d34095a3-6847-559a-8a3c-692b04ec9c86-chatqna.opea-chatqna.svc.cluster.local

Once the service is deployed in your cluster you should be able to contact the service using its name, and Kube-DNS will answer with the correct ClusterIP to speak to your final pods. ClusterIPs are governed by IPTables rules created by kube-proxy on Workers that NAT your request to the final container’s IP.

The Kube-DNS naming convention is service.namespace.svc.cluster-domain.tld and the default cluster domain is cluster.local.

For example, if you want to contact a service called mysql in the db namespace from any namespace, you can simply speak to mysql.db.svc.cluster.local.

 set ENDPOINT 

sudo wget -qO /usr/local/bin/yq https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 
 
sudo chmod a+x /usr/local/bin/yq 
# sudo apt install build-essential make git curl jq yq yamllint python3-pip -y

 get mega endpoint

git grep APP_.*ENDPOINT

ns=

name=chatqna
file=chatqna.yaml
svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file)
port=$(jq .spec.ports[0].port <<< $svc)

url=http://${name}.${ns:-default}.svc.cluster.local:${port}

echo $url

name=codegen
file=codegen.yaml
svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file)
port=$(jq .spec.ports[0].port <<< $svc)

url=http://${name}.${ns:-default}.svc.cluster.local:${port}

echo $url

# https://blog.51cto.com/hmtk520/2074219
for i in $(grep -oP "(?<=APP_BACKEND_SERVICE_ENDPOINT_).*"  *.yaml); do
  echo $i
  name=${i##*:}
  file=${name,,}.yaml
  svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file)
  port=$(jq .spec.ports[0].port <<< $svc)

  url=http://${name,,}.${ns:-default}.svc.cluster.local:${port}
  echo $url
sed -i -e '/APP_BACKEND_SERVICE_ENDPOINT_'"$name"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml
done

 get data prepare endpoint

name=chatqna-data-prep
file=chatqna.yaml
svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file)
port=$(jq .spec.ports[0].port <<< $svc)
url=http://${name}.${ns:-default}.svc.cluster.local:${port}
echo $url
for i in $(grep -oP "(?<=APP_)DATAPREP.*(?=_ENDPOINT)" *.yaml); do
  echo $i
  curd=${i##*:};
  sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml;
done

get CHAT_HISTORY endpoint

for i in $(grep -oP "(?<=APP_)CHAT_HISTORY.*(?=_ENDPOINT)" *.yaml); do
  echo $i;
  curd=${i##*:};
  name=${curd%_*};
  file=${name,,}.yaml;
  name=${name/_/-};
  svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file)
  port=$(jq .spec.ports[0].port <<< $svc)
  url=http://${name,,}.${ns:-default}.svc.cluster.local:${port};
  echo $url;
  sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml;
done

  get PROMPT endpoint

for i in $(grep -oP "(?<=APP_)PROMPT_SERVICE.*(?=_ENDPOINT)" *.yaml); do
  echo $i;
  curd=${i##*:};
  curdr=${curd/SERVICE/REGISTRY};
  name=${curdr%_*};
  file=${name,,}.yaml;
  name=${name/_/-};
  svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file)
  port=$(jq .spec.ports[0].port <<< $svc)
  url=http://${name,,}.${ns:-default}.svc.cluster.local:${port};
  echo $url;
  sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml ;
done

 get keycloak

name=keycloak
file=keycloak_install.yaml
svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file)
port=$(jq .spec.ports[0].port <<< $svc)
url=http://${name}.${ns:-default}.svc.cluster.local:${port}
echo $url
sed -i -e '/APP_KEYCLOAK_SERVICE_ENDPOINT/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml

 configure mirror

bash - How to add line before match within given line range/patterns using sed? - Stack Overflow

MIRROR='HF_ENDPOINT: "https://hf-mirror.com"'

conf=chatqna-tgi-config
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  '"${MIRROR}"'|' chatqna.yaml

conf=chatqna-tei-config
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  '"${MIRROR}"'|' chatqna.yaml

conf=chatqna-teirerank-config
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  '"${MIRROR}"'|' chatqna.yaml


conf=codegen-tgi-config
file=codegen.yaml
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  '"${MIRROR}"'|' $file

conf=docsum-tgi-config
file=docsum.yaml
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  '"${MIRROR}"'|' $file

conf=faqgen-tgi-config
file=faqgen.yaml
sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  '"${MIRROR}"'|' $file

configure model ID

conf=chatqna-tgi-config
model=test_id
set_model_id() {
  conf=$1
  file=${1%%-*}
  sed '/name: '"${conf}"'/,/---/s|\(MODEL_ID:\).*|\1 "'"${2}"'"|' ${file}.yaml
}

 get model ID

json - JQ: Filtering for keys - Stack Overflow

yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_ID": .data.MODEL_ID}'  *.yaml

get and set model ID

set_model_id() {
  if  [ -z "$1" ] && [ -z "$2" ]; then
    yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_ID": .data.MODEL_ID}'  *.yaml
    echo "usage:"
    echo "  set_model_id \${ConfigMap} \${MODEL_ID}"
    return
  fi
  conf=$1
  file=${1%%-*}
  sed -i '/name: '"${conf}"'/,/---/s|\(MODEL_ID:\).*|\1 "'"${2}"'"|' ${file}.yaml
}

all codes 

set_model_id() {
  if  [ -z "$1" ] && [ -z "$2" ]; then
    yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_ID": .data.MODEL_ID}'  *.yaml
    echo "usage:"
    echo "  set_model_id \${ConfigMap} \${MODEL_ID}"
    return
  fi
  conf=$1
  file=${1%%-*}
  sed -i '/name: '"${conf}"'/,/---/s|\(MODEL_ID:\).*|\1 "'"${2}"'"|' ${file}.yaml
}

set_model_mirror() {
  if  [ -z "$1" ] ; then
    yq -o json '.| select(.data | has("MODEL_ID"))| {"ConfigMap": .metadata.name, "MODEL_MIRROR": .data.HF_ENDPOINT}'  *.yaml
    echo "usage:"
    echo "  set_model_mirror \${MODEL_MIRROR}"
    return
  fi
  cm=$(yq -r -o json '.| select(.data | has("MODEL_ID"))| .metadata.name'  *.yaml)
  mirror=$1
  for i in $cm; do
    conf=$i
    file=${i%%-*}
    echo "ConfigMap: $conf set mirror as $mirror"
    has_mirror=$(yq -r -o json '.| select(.metadata.name == "'"${conf}"'")| .data.HF_ENDPOINT' ${file}.yaml)
    if [ "$has_mirror" == "null" ]; then
      sed -i '/name: '"${conf}"'/,/---/s|\(data:\)|\1\n  HF_ENDPOINT: "'"${mirror}"'"|' ${file}.yaml
    else
      sed -i '/name: '"${conf}"'/,/---/s|\(HF_ENDPOINT:\).*|\1 "'"${1}"'"|' ${file}.yaml
    fi
  done
}

set_hf_token() {
  if  [ -z "$1" ] ; then
    echo "usage:"
    echo "  set_hf_token \${HF_TOKEN}"
    return
  fi
  sed -i "s/\(HF_TOKEN:\).*/\1 \"${1}\"/g" *.yaml
  sed -i "s/\(HUGGINGFACEHUB_API_TOKEN:\).*/\1 \"${1}\"/g" *.yaml
  sed -i "s/\(HUGGING_FACE_HUB_TOKEN:\).*/\1 \"${1}\"/g" *.yaml
}

set_https_proxy() {
  if  [ -z "$1" ] ; then
    echo "usage:"
    echo "  set_https_proxy \${https_proxy}"
    return
  fi
  https_proxy=$1
  sed -i -e "s|\(https_proxy:\)\s*\"\"|\1 \"$https_proxy\"|g" *.yaml
  sed -i '/https_proxy/{n;s|\(value:\)\s.*""|\1 "'"$https_proxy"'"|g}' *.yaml
}

set_http_proxy() {
  if  [ -z "$1" ] ; then
    echo "usage:"
    echo "  set_http_proxy \${http_proxy}"
    return
  fi
  http_proxy=$1
  sed -i -e "s|\(http_proxy:\)\s*\"\"|\1 \"$http_proxy\"|g" *.yaml
  sed -i '/http_proxy/{n;s|\(value:\)\s.*""|\1 "'"$http_proxy"'"|g}' *.yaml
}

set_no_proxy() {
  if  [ -z "$1" ] ; then
    echo "usage:"
    echo "  set_no_proxy \${no_proxy}"
    return
  fi
  no_proxy=$1
  sed -i -e "s|\(no_proxy:\)\s*\"\"|\1 \"$no_proxy\"|g" *.yaml
  sed -i '/no_proxy/{n;s|\(value:\)\s.*""|\1 "'"$no_proxy"'"|g}' *.yaml
}

set_backend_service_endpoint() {
  for i in $(grep -oP "(?<=APP_BACKEND_SERVICE_ENDPOINT_).*"  *.yaml); do
    echo $i
    name=${i##*:}
    file=${name,,}.yaml
    svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file)
    port=$(jq .spec.ports[0].port <<< $svc)

    url=http://${name,,}.${ns:-default}.svc.cluster.local:${port}
    echo $url
    sed -i -e '/APP_BACKEND_SERVICE_ENDPOINT_'"$name"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml
  done
}


set_dataprep_service_endpoint() {
  name=chatqna-data-prep
  file=chatqna.yaml
  svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file)
  port=$(jq .spec.ports[0].port <<< $svc)
  url=http://${name}.${ns:-default}.svc.cluster.local:${port}
  echo $url
  for i in $(grep -oP "(?<=APP_)DATAPREP.*(?=_ENDPOINT)" *.yaml); do
    echo $i
    curd=${i##*:};
    sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml;
  done
}


set_chat_history_endpoint() {
  for i in $(grep -oP "(?<=APP_)CHAT_HISTORY.*(?=_ENDPOINT)" *.yaml); do
    echo $i;
    curd=${i##*:};
    name=${curd%_*};
    file=${name,,}.yaml;
    name=${name/_/-};
    svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file)
    port=$(jq .spec.ports[0].port <<< $svc)
    url=http://${name,,}.${ns:-default}.svc.cluster.local:${port};
    echo $url;
    sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml;
  done
}


set_prompt_service_endpoint() {
  for i in $(grep -oP "(?<=APP_)PROMPT_SERVICE.*(?=_ENDPOINT)" *.yaml); do
    echo $i;
    curd=${i##*:};
    curdr=${curd/SERVICE/REGISTRY};
    name=${curdr%_*};
    file=${name,,}.yaml;
    name=${name/_/-};
    svc=$(yq -o json '. | select(.metadata.name == "'"${name,,}"'" and .kind=="Service")' $file)
    port=$(jq .spec.ports[0].port <<< $svc)
    url=http://${name,,}.${ns:-default}.svc.cluster.local:${port};
    echo $url;
    sed -i -e '/'"$curd"'/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml ;
  done
}

set_keycloak_service_endpoint() {
  name=keycloak
  file=keycloak_install.yaml
  svc=$(yq -o json '. | select(.metadata.name == "'"$name"'" and .kind=="Service")' $file)
  port=$(jq .spec.ports[0].port <<< $svc)
  url=http://${name}.${ns:-default}.svc.cluster.local:${port}
  echo $url
  sed -i -e '/APP_KEYCLOAK_SERVICE_ENDPOINT/{n;s|\(value:\)\s.*|\1 "'"$url"'"|}' productivity_suite_reactui.yaml
}

set_services_endpoint() {
  set_backend_service_endpoint
  set_keycloak_service_endpoint
  set_chat_history_endpoint
  set_prompt_service_endpoint
  set_dataprep_service_endpoint
}
View Code

 

and submit a PR: https://github.com/opea-project/GenAIExamples/pull/919

 

deployment

kubectl apply -f .

 five pods  failed.

 

 查看code
GenAIComps/comps/llms/text-generation/native/langchain/requirements.txt at 405a2fc68ae590740dd70f78df83b1ca2976360f · opea-project/GenAIComps (github.com)
需要安装 langchain_core. 

docker file 采用的是python:3.11-slim
GenAIComps/comps/llms/text-generation/tgi/Dockerfile at 405a2fc68ae590740dd70f78df83b1ca2976360f · opea-project/GenAIComps (github.com)

使用同样类型的pod,发现应该是代理的问题。 

kubectl exec chatqna-embedding-usvc-6556d4bbd7-9f7xk  -it  -- env |grep proxy

kubectl exec chatqna-embedding-usvc-6556d4bbd7-9f7xk  -it  -- pip install --upgrade langchain_core --proxy=http://proxy-country.company.com:8080

相应的code:

GenAIComps/comps/llms/text-generation/native/langchain/llm.py at 405a2fc68ae590740dd70f78df83b1ca2976360f · opea-project/GenAIComps (github.com)
Build Docker Images

GenAIExamples/ProductivitySuite/docker_compose/intel/cpu/xeon/README.md at main · opea-project/GenAIExamples (github.com)

# ghp_eKZfZasrUxblfTZBBfRpmGpQqtdbUObfUt
# ghp_kOFRJkamkPBDwrkMsdKLcoLtXzzmQlWATp
# hf_T3Bp07Akonutn2NSLgnoUs9PIKDJpB2yaPpO

https://github.com/opea-project/GenAIInfra/issues/367
posted @ 2024-09-29 11:21  lvmxh  阅读(30)  评论(0编辑  收藏  举报