A Complete Guide to Setting up Apache ShardingSphere-Proxy/

...

/

Docker Setup

Docker Setup

We'll cover the following...

Docker on the Educative Platform

We use Docker to allow authors to create their own custom environments to accommodate any language that is not supported by our platform.

Authors can create custom docker images that deploy on the Google Cloud and allow end-users to directly have access to these customized environments through our platform.

The Jargon

Base Image: An OS userspace.

Image: A read-only template with instructions for creating a Docker container.

Dockerfile: A simple file with no extension, named as Dockerfile is used to builds a Docker image, and specifies everything that will go in the application’s environment setup.

Container: An image when run, builds a container. This contains your application environment, basically where your application will be run.

Demo

Before we start, let’s see what our resultant Apache ShardingSphere-Proxy would look like.

Click on the Run button and wait for the environment to set up.

schemaName: sharding_db

dataSources:
 ds_0:
   url: jdbc:postgresql://127.0.0.1:5432/demo_db
   username: postgres
   password: postgres
   connectionTimeoutMilliseconds: 30000
   idleTimeoutMilliseconds: 60000
   maxLifetimeMilliseconds: 1800000
   maxPoolSize: 50
   minPoolSize: 1
 ds_1:
   url: jdbc:postgresql://127.0.0.1:5432/demo_db_1
   username: postgres
   password: postgres
   connectionTimeoutMilliseconds: 30000
   idleTimeoutMilliseconds: 60000
   maxLifetimeMilliseconds: 1800000
   maxPoolSize: 50
   minPoolSize: 1

rules:
- !SHARDING
 tables:
   t_order:
     actualDataNodes: ds_${0..1}.t_order_${0..4}
     tableStrategy:
       standard:
         shardingColumn: order_id
         shardingAlgorithmName: t_order_inline
     keyGenerateStrategy:
         column: order_id
         keyGeneratorName: snowflake
   t_order_item:
     actualDataNodes: ds_${0..1}.t_order_item_${0..4}
     tableStrategy:
       standard:
         shardingColumn: order_id
         shardingAlgorithmName: t_order_item_inline
     keyGenerateStrategy:
       column: order_item_id
       keyGeneratorName: snowflake
 bindingTables:
   - t_order,t_order_item
 defaultDatabaseStrategy:
   standard:
     shardingColumn: user_id
     shardingAlgorithmName: database_inline
 defaultTableStrategy:
   none:
 
 shardingAlgorithms:
   database_inline:
     type: INLINE
     props:
       algorithm-expression: ds_${user_id % 2}
   t_order_inline:
     type: INLINE
     props:
       algorithm-expression: t_order_${order_id % 2}
   t_order_item_inline:
     type: INLINE
     props:
       algorithm-expression: t_order_item_${order_id % 2}

 keyGenerators:
   snowflake:
     type: SNOWFLAKE
     props:
       worker-id: 123

 scalingName: default_scaling
 scaling:
   default_scaling:
     input:
       workerThread: 40
       batchSize: 1000
       rateLimiter:
         type: QPS
         props:
           qps: 50
     output:
       workerThread: 40
       batchSize: 1000
       rateLimiter:
         type: TPS
         props:
           tps: 2000
     streamChannel:
       type: MEMORY
       props:
         block-queue-size: 10000
     completionDetector:
       type: IDLE
       props:
         incremental-task-idle-minute-threshold: 30
     dataConsistencyChecker:
       type: DATA_MATCH
       props:
         chunk-size: 1000
Demo application

Please ensure that the proxy has started. Once verified, we can execute the following command:

psql -h 127.0.0.1 -p 8080 -U postgres sharding_db
Postgres CLI

Create the tables using the following SQL:

create table t_order(order_id int8 primary key, user_id int8, info text, c1 int, crt_time timestamp);  
create table t_order_item(order_item_id int8 primary key, order_id int8, user_id int8, info text, c1 int, c2 int, c3 int, c4 int, c5 int, crt_time timestamp);

Now, if we were to execute \dt in the terminal, we would see five copies of these two tables like this:

We can go ahead and insert data into the tables and the data will be distributed among the above tables. We can insert random data by executing the following command:

insert into t_order (user_id, info, c1, crt_time) values (0,'a',1,now());
Insert orders

Try executing this multiple times and verify if the rows have been added by entering this command:

select * from t_order;
Query orders

The SPA widget contains three files:

  • server.yaml: We have used the default configuration and allowed the appropriate users in the config to interact with the nodes.

  • config-sharding.yaml: Used to configure sharding with PostgreSQL. We have used the default configuration, but you can go ahead and try out a few others as well.

  • config-readwrite-splitting.yaml: Just like the config-sharding.yaml file, we are using the default configuration. We can try out the same commands and verify if they are working or not.

Dockerfile

Press + to interact
FROM ubuntu:latest
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Europe/Moscow
# Install the appropriate packages for postgres and java
RUN apt-get update && apt-get install -y libgtk2.0-0 libgtk-3-0 libgbm-dev
    libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2 libxtst6 xauth 
    xvfb gnupg curl wget git vim sudo postgresql postgresql-contrib tzdata 
    zip unzip openjdk-11-jdk ca-certificates-java && \
    apt-get clean && \
    update-ca-certificates -f
# Get the tar binary for the sharding sphere proxy
RUN wget https://dlcdn.apache.org/shardingsphere/5.1.0/apache-shardingsphere-5.1.0-shardingsphere-proxy-bin.tar.gz 
    && tar -xvf apache-shardingsphere-5.1.0-shardingsphere-proxy-bin.tar.gz
# Install MySQL and create a user
RUN apt-get update && apt-get install -y mysql-server mysql-client 
    && service mysql start && mysql -u root -e "Create User 'user'@'localhost' 
    identified by 'user'; GRANT ALL PRIVILEGES ON *.* TO 'user'@'localhost'; 
    Flush Privileges;" && service mysql stop
# Create the appropriate databases a
RUN service postgresql start && \
    su - postgres -c "psql -U postgres -d postgres -c \"alter user postgres with password 'postgres';\"" && \
    su - postgres -c "psql -U postgres -d postgres -c \"Create Database demo_db;\"" && \
    su - postgres -c "psql -U postgres -d postgres -c \"Create Database demo_db_1;\"" && \ 
    su - postgres -c "psql -U postgres -d postgres -c \"Create Database replica_db;\"" && \ 
    su - postgres -c "psql -U postgres -d postgres -c \"Create Database replica_db_1;\"" && \ 
    su - postgres -c "psql -U postgres -d postgres -c \"Create Database replica_db_2;\"" && \
    service postgresql stop

Line 1: FROM ubuntu:latest

The FROM command sets the Base Image for the rest of the instructions. This command must be on top of the Dockerfile.

Lines 7-32:

The RUN command is used to execute commands that build up the Docker image, adding to the build time.

Note: Each RUN command builds a layer on top of the base image, which results in a longer build time. Hence, chaining your commands in a single run makes things faster.

We are performing the following tasks with the Run command:

  • Installing the required packages/services.
  • Downloading the required folders/files.
  • Installing the required packages. Creating a user, and granting it the required privileges and permissions.
  • Starting the postgresql service, altering the user, and creating some replica (demo) databases.

Once we have created a Dockerfile we need to upload it on the Educative platform.

Docker Configuration

On the Collection Editor page, scroll down to find the “Docker Container” section. This is where most of the setup takes place.

Uploading the Dockerfile

To upload your Dockerfile, compress it into a tarball first. In the terminal, move to the directory where the Dockerfile is present.

Note: The Dockerfile needs to be at the root of this directory.

The following command can be used to create a tarball:

Press + to interact
# The dot . at the end tells the tar command to create a zip file with
# all the contents of current directory. 
tar -czvf tarball.tar.gz .

You can rename tarball to whatever you prefer. Make sure the tarball extension is tar.gz.

Building the Docker Image

Once the file has been uploaded, click on the Save button in the top-right corner to build the image. This step may take a while, especially if the image is large. You will see the following loading screen:

If the image is built successfully, a message will be prompted on the screen:

If there is an error in the Dockerfile, the image building process will fail. This can happen due to a number of reasons. To check the details of the issue, click the “Download Build Logs” button:

You can download the tarball that we created for this tutorial from the widget below:

proxy.tar.gz

In the next lesson, we will look into the rest of the steps to run the application on the platform.