hasura graphql-engine + pgspider citus 加强hasura的大数量graphql 处理能力

以前有写过比较简单的关于hasura graphql-engine 集成 citus的，以下是一个尝试，将pgspider 同时也集成起来
增强hasura graphql-engine 的大数据量以及多中数据源的并行处理能力

环境准备

  • pgspider citus 9.1 docker 镜像
 
FROM dalongrong/pgspider:base as build
WORKDIR /app
RUN apt-get update && apt-get install -y cmake automake autoconf libcurl4-openssl-dev libtool pkg-config libssl-dev
RUN wget https://github.com/citusdata/citus/archive/v9.1.0.tar.gz && tar zxvf v9.1.0.tar.gz && cp -rf citus-9.1.0 /app/postgresql-11.6/contrib/citus
RUN cd /app/postgresql-11.6/contrib/citus && ./configure && make && make install
​
FROM debian:stretch-slim
ENV GOSU_VERSION 1.11
RUN apt-get update && apt-get install -y wget openssl libcurl4-openssl-dev libreadline-dev
# explicitly set user/group IDs
RUN set -eux; \
    groupadd -r postgres --gid=999; \
# https://salsa.debian.org/postgresql/postgresql-common/blob/997d842ee744687d99a2b2d95c1083a2615c79e8/debian/postgresql-common.postinst#L32-35
    useradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres; \
# also create the postgres user's home directory with appropriate permissions
# see https://github.com/docker-library/postgres/issues/274
    mkdir -p /var/lib/postgresql; \
    chown -R postgres:postgres /var/lib/postgresql
​
RUN wget -O /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture)" \
   && chmod +x /usr/local/bin/gosu \
   && gosu nobody true
# make the "en_US.UTF-8" locale so postgres will be utf-8 enabled by default
RUN set -eux; \
   if [ -f /etc/dpkg/dpkg.cfg.d/docker ]; then \
   # if this file exists, we're likely in "debian:xxx-slim", and locales are thus being excluded so we need to remove that exclusion (since we need locales)
   grep -q '/usr/share/locale' /etc/dpkg/dpkg.cfg.d/docker; \
   sed -ri '/\/usr\/share\/locale/d' /etc/dpkg/dpkg.cfg.d/docker; \
   ! grep -q '/usr/share/locale' /etc/dpkg/dpkg.cfg.d/docker; \
   fi; \
   apt-get update; apt-get install -y locales; rm -rf /var/lib/apt/lists/*; \
   localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
ENV LANG en_US.utf8
​
# install "nss_wrapper" in case we need to fake "/etc/passwd" and "/etc/group" (especially for OpenShift)
# https://github.com/docker-library/postgres/issues/359
# https://cwrap.org/nss_wrapper.html
RUN set -eux; \
   apt-get update; \
   apt-get install -y --no-install-recommends libnss-wrapper; \
   rm -rf /var/lib/apt/lists/*
​
COPY --from=build /usr/local/pgspider /usr/local/pgspider
​
RUN sed -ri "s!^#?(listen_addresses)\s*=\s*\S+.*!\1 = '*'!" /usr/local/pgspider/share/postgresql/postgresql.conf.sample; \
   grep -F "listen_addresses = '*'" /usr/local/pgspider/share/postgresql/postgresql.conf.sample
RUN echo "shared_preload_libraries='citus'" >> /usr/local/pgspider/share/postgresql/postgresql.conf.sample
RUN mkdir -p /var/run/postgresql && chown -R postgres:postgres /var/run/postgresql && chmod 2777 /var/run/postgresql
ENV PATH $PATH:/usr/local/pgspider/bin
ENV PGDATA /var/lib/postgresql/data
RUN mkdir -p "$PGDATA" && chown -R postgres:postgres "$PGDATA" && chmod 777 "$PGDATA"
VOLUME /var/lib/postgresql/data
COPY docker-entrypoint.sh /usr/local/bin/
RUN ln -s usr/local/bin/docker-entrypoint.sh / # backwards compat
ENTRYPOINT ["docker-entrypoint.sh"]
​
EXPOSE 5432
CMD ["postgres"]
  • docker-compose 文件
version: "3"
services: 
  graphql-engine:
    image: hasura/graphql-engine:v1.1.0
    ports:
    - "8080:8080"
    environment:
      HASURA_GRAPHQL_DATABASE_URL: postgres://postgres:dalong@pg-citus-master:5432/postgres
      HASURA_GRAPHQL_ENABLE_CONSOLE: "true" # set to "false" to disable console
      HASURA_GRAPHQL_ENABLED_LOG_TYPES: startup, http-log, webhook-log, websocket-log, query-log
  pg-citus-master:
    container_name: pg-citus-master
    image: dalongrong/pgspider:citus-9.1
    volumes: 
    - "./csvfiles:/opt/csv"
    ports: 
    - "5432:5432"
    environment: 
    - "POSTGRES_PASSWORD=dalong"
  pg-citus-worker:
    container_name: pg-citus-worker
    image: dalongrong/pgspider:citus-9.1
    volumes: 
    - "./csvfiles:/opt/csv"
    ports: 
    - "5433:5432"
  • 启动
docker-compose up -d
  • 加载扩展
    目前制作的镜像，没像官方的可以直接启用扩展的，需要自己手工创建，注意多有节点都需要添加
create extension citus;
  • 添加节点
    因为使用的是社区版本，权限管理上是不支持的,只对于master 进行了认证处理，此操作在master 执行
 
SELECT master_add_node('pg-citus-worker', '5432');
SELECT master_add_node('pg-citus-worker2', '5432');
  • 数据初始化
CREATE TABLE companies (
    id bigint NOT NULL,
    name text NOT NULL,
    image_url text,
    created_at timestamp without time zone NOT NULL,
    updated_at timestamp without time zone NOT NULL
);
​
CREATE TABLE campaigns (
    id bigint NOT NULL,
    company_id bigint NOT NULL,
    name text NOT NULL,
    cost_model text NOT NULL,
    state text NOT NULL,
    monthly_budget bigint,
    blacklisted_site_urls text[],
    created_at timestamp without time zone NOT NULL,
    updated_at timestamp without time zone NOT NULL
);
​
CREATE TABLE ads (
    id bigint NOT NULL,
    company_id bigint NOT NULL,
    campaign_id bigint NOT NULL,
    name text NOT NULL,
    image_url text,
    target_url text,
    impressions_count bigint DEFAULT 0,
    clicks_count bigint DEFAULT 0,
    created_at timestamp without time zone NOT NULL,
    updated_at timestamp without time zone NOT NULL
);
​
ALTER TABLE companies ADD PRIMARY KEY (id);
ALTER TABLE campaigns ADD PRIMARY KEY (id, company_id);
ALTER TABLE ads ADD PRIMARY KEY (id, company_id);
​
SET citus.replication_model = 'streaming';
SELECT create_distributed_table('companies', 'id');
SELECT create_distributed_table('campaigns', 'company_id');
SELECT create_distributed_table('ads', 'company_id');
  • 加载数据
    数据行citus 官方下载的测试数据
    下载
curl https://examples.citusdata.com/tutorial/companies.csv > csvfiles/scompanies.csv
curl https://examples.citusdata.com/tutorial/campaigns.csv > csvfiles/campaigns.csv
curl https://examples.citusdata.com/tutorial/ads.csv > csvfiles/ads.csv

加载数据，需要在容器内部（master）

\copy companies from 'companies.csv' with csv
\copy campaigns from 'campaigns.csv' with csv
\copy ads from 'ads.csv' with csv

hasura graphql-engine 使用

这个就比较简单了，我们需要的就是使用鼠标点击了

  • ui 操作

 

 

  • 数据查询

 

 

说明

当前测试的citus 最新版本9.2 有问题，使用9.1 可以测试通过,具体原因待定，可能是hasura对于citus 支持的问题

参考资料

https://github.com/rongfengliang/pgspider-docker
https://hub.docker.com/repository/docker/dalongrong/pgspider
http://docs.citusdata.com/en/v9.2/use_cases/multi_tenant.html
https://github.com/citusdata/citus

posted on 2020-03-05 19:32  荣锋亮  阅读(384)  评论(0)  编辑  收藏  举报

导航