Start and stop Saagie

Follow this process to start and stop Saagie. It is important to follow the orders listed below. Essentially, you will start Saagie in one order and stop Saagie in the opposite order.

1. Start Saagie

Start Saagie in the following order. Some components have prerequisites, so it important to follow the steps.

Start by declaring the variables that will be used by the scripts. These variables need to be declared exactly as they appear below.

declare -a toPatch=("saagie-common-phagocytor")
declare dbs="mongo|postgresql|schema-registry"
declare -a statefulsets=("saagie-kafka-cp-zookeeper" "saagie-kafka-cp-kafka")

Next, start Saagie by launching the components in the following order:

  1. Launch Zookeeper and Kafka, each running three pods.

    for statefulset in "${statefulsets[@]}"
    do
        echo "Starting $statefulset..."
        kubectl scale statefulset.apps -n saagie-common $statefulset --replicas=3
    done
  2. Launch database-related deployments, containing MongoDB, PostgreSQL, and schema-registry.

    for db_deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -E $dbs | grep -Ev $excludes`
    do
        echo "Starting $db_deployment..."
        kubectl scale deployment.extensions -n saagie-common --replicas 1 $db_deployment
    done
  3. Launch product-related deployments.

    for deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -Ev $dbs | grep -Ev $excludes`
    do
        echo "Starting $deployment..."
        kubectl scale deployment.extensions -n saagie-common --replicas 1 $deployment
    done
  4. Resume Phagocytor cron job to enable job and pipeline cleaning.

    for cronJob in "${toPatch[@]}"
    do
        echo "Starting $cronJob..."
        kubectl patch cronjob.batch -n saagie-common $cronJob -p '{"spec" : {"suspend" : false }}'
    done

2. Stop Saagie

When stopping Saagie product, some components must be stopped before others in order to avoid unexpected behaviors.

Start by declaring the variables that will be used by the scripts. These variables need to be declared exactly as they appear below.

declare -a toPatch=("saagie-common-phagocytor")
declare dbs="mongo|postgresql|schema-registry"
declare -a statefulsets=("saagie-kafka-cp-zookeeper" "saagie-kafka-cp-kafka")

Next, stop Saagie by stopping the components in the following order:

  1. Suspend Phagocytor cron job and pipeline cleaning.

    for cronJob in "${toPatch[@]}"
    do
        echo "Shutting down $cronJob..."
        kubectl patch cronjob.batch -n saagie-common $cronJob -p '{"spec" : {"suspend" : true }}'
    done
  2. Stop product-related deployments.

    for deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -Ev $dbs | grep -Ev $excludes`
    do
        echo "Shutting down $deployment..."
        kubectl scale deployment.extensions -n saagie-common --replicas 0 $deployment
    done
  3. Stop database-related deployments.

    for db_deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -E $dbs | grep -Ev $excludes`
    do
        echo "Shutting down $db_deployment..."
        kubectl scale deployment.extensions -n saagie-common --replicas 0 $db_deployment
    done
  4. Stop Kafka and Zookeeper.

    for statefulset in "${statefulsets[@]}"
    do
        echo "Shutting down $statefulset..."
        kubectl scale statefulset.apps -n saagie-common $statefulset --replicas=0
    done

3. Script to start Saagie

startCluster.sh
#!/bin/bash

declare -a toPatch=("saagie-common-phagocytor")
declare dbs="mongo|postgresql|schema-registry"
declare -a statefulsets=("saagie-kafka-cp-zookeeper" "saagie-kafka-cp-kafka")

# Start kafka & zookeeper

for statefulset in "${statefulsets[@]}"
do
    echo "Starting $statefulset..."
    kubectl scale statefulset.apps -n saagie-common $statefulset --replicas=3
done

# Start every saagie-common databases deployments

for db_deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -E $dbs | grep -Ev $excludes`
do
    echo "Starting $db_deployment..."
    kubectl scale deployment.extensions -n saagie-common --replicas 1 $db_deployment
done

# Start every saagie-common deployments (except databases & schema-registry)

for deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -Ev $dbs | grep -Ev $excludes`
do
    echo "Starting $deployment..."
    kubectl scale deployment.extensions -n saagie-common --replicas 1 $deployment
done

# Resume saagie-common cronJobs

for cronJob in "${toPatch[@]}"
do
    echo "Starting $cronJob..."
    kubectl patch cronjob.batch -n saagie-common $cronJob -p '{"spec" : {"suspend" : false }}'
done

4. Script to stop Saagie

stopCluster.sh
#!/bin/bash

declare -a toPatch=("saagie-common-phagocytor")
declare dbs="mongo|postgresql|schema-registry"
declare -a statefulsets=("saagie-kafka-cp-zookeeper" "saagie-kafka-cp-kafka")

# Suspend saagie-common cronJobs

for cronJob in "${toPatch[@]}"
do
    echo "Shutting down $cronJob..."
    kubectl patch cronjob.batch $cronJob -n saagie-common -p '{"spec" : {"suspend" : true }}'
done

# Stop every saagie-common deployments (except databases & schema-registry)

for deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -Ev $dbs | grep -Ev $excludes`
do
    echo "Shutting down $deployment..."
    kubectl scale deployment.extensions -n saagie-common --replicas 0 $deployment
done

# Stop every saagie-common databases deployments

for db_deployment in `kubectl get deployment.extensions -n saagie-common -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | grep -E $dbs | grep -Ev $excludes`
do
    echo "Shutting down $db_deployment..."
    kubectl scale deployment.extensions -n saagie-common --replicas 0 $db_deployment
done

# Stop kafka & zookeeper

for statefulset in "${statefulsets[@]}"
do
    echo "Shutting down $statefulset..."
    kubectl scale statefulset.apps -n saagie-common $statefulset --replicas=0
done