Leveraging Yandex Pandora: Stress Testing GRPC and Flatbuffer Services

Hi there!

This is a continuation of my first article on how to use Yandex Tank for stress testing your services. The documentation mentions that the main engine for load is Phantom, but it can be replaced with Pandora if you need to create unique test scenarios or work with other types of requests (such as grpc or any other binaries). In this article, we will explore how to integrate Yandex Pandora for two cases:

1. GRPC
2. flatbuffer (over http)

As I mentioned before, our main goal is to use a simple tool with straightforward documentation that doesn’t require hiring a bunch of people to set up the test. Pandora, like Tank, has very simple and understandable documentation, does exactly what we need, and has a low entry threshold. In my case, the primary stack is Golang, and I choose solutions around it.

Let’s start by referring to the documentation and see what Pandora is all about.
“Pandora is a high-performance load generator in the Go language. It has built-in HTTP(S) and HTTP/2 support, and you can write your own load scenarios in Go, compiling them just before your test.”

Since we are interested in scenarios, let’s take a look at the documentation:

As we can see from the documentation, there is an excellent example of how to write our “gun” to create our load engine. Great, let’s take this example and copy-paste it into our repository. Next, let’s look at our service contracts:

syntax = "proto3";

option go_package = "/docs";

service DocumentService {
 rpc GetAllByLimitAndOffset(GetAllRequest) returns (GetAllResponse) {}
 rpc Save(SaveRequest) returns (SaveResponse) {}
 rpc Validate(ValidateRequest) returns (ValidateResponse) {}
}

Our task involves covering two methods (yes, we’ll skip save method for current case):

• Save
• Validate

Create a Yandex-tank folder, create a GRPC folder inside it, move our PROTO file there, and generate contracts with the command:

protoc - go_out=./docs - go_opt=paths=source_relative \
 - go-grpc_out=./docs - go-grpc_opt=paths=source_relative docs.proto

After that, we will get files as shown in the screenshot.
And now, let’s start to write our own gun. We have to add a huge number of imports and create a struct for Custom params:

package main

import (
 "context"
 "log"
 "strconv"
 "time"

 _ "github.com/golang/protobuf/ptypes/timestamp"
 "github.com/spf13/afero"
 "google.golang.org/grpc"
 pb "grpc-gun/docs"

 "github.com/yandex/pandora/cli"
 "github.com/yandex/pandora/components/phttp/import"
 "github.com/yandex/pandora/core"
 "github.com/yandex/pandora/core/aggregator/netsample"
 "github.com/yandex/pandora/core/import"
 "github.com/yandex/pandora/core/register"
)

type Ammo struct {
 Tag    string
 Param1 string
 Param2 string
 Param3 string
}

Now, we should add GunConfig and Gun, where we can add our grpc client (just copy and paste from official docs)

type GunConfig struct {
 Target string `validate:"required"` // Configuration will fail, without target defined
}

type Gun struct {
 // Configured on construction.
 client grpc.ClientConn
 conf   GunConfig
 // Configured on Bind, before shooting
 aggr core.Aggregator // Maybe your custom Aggregator.
 core.GunDeps
}

And add new func for Gun, almost done!

func NewGun(conf GunConfig) *Gun {
   return &Gun{conf: conf}
}

We can see, that we should add a little bit more for running:

– Bind method. We just bind our client and args:

– override Shoot method

func (g *Gun) Bind(aggr core.Aggregator, deps core.GunDeps) error {
 // create gRPC stub at gun initialization
 conn, err := grpc.Dial(
  g.conf.Target,
  grpc.WithInsecure(),
  grpc.WithTimeout(time.Second),
  grpc.WithUserAgent("load test, pandora custom shooter"))
 if err != nil {
  log.Fatalf("FATAL: %s", err)
 }
 g.client = *conn
 g.aggr = aggr
 g.GunDeps = deps
 return nil
}

func (g *Gun) Shoot(ammo core.Ammo) {
 customAmmo := ammo.(*Ammo)
 g.shoot(customAmmo)
}

It’s time to add switch method for SaveCase and ValidateCase requests and to add methods, as example for ValidateCase:

func (g *Gun) caseValidate(client pb.DocumentServiceClient, ammo *Ammo) int {
 code := 0
 docName := ammo.Param1

 out, err := client.Validate(
  context.TODO(), &pb.ValidateRequest{Document: createDoc(docName)},
 )

 if err != nil {
  log.Printf("FATAL: %s", err)
  code = 500
 }

 if out != nil {
  code = 200
 }
 return code
}

Full code here:

Alright, pay close attention!

func main() {
     // Standard imports.
     fs := afero.NewOsFs()
     coreimport.Import(fs)
     // May not be imported, if you don't need http guns, etc.
     phttp.Import(fs)
    // Custom imports. Integrate your custom types into configuration system.
     coreimport.RegisterCustomJSONProvider("grpc_provider", func() core.Ammo { return &Ammo{} })
    register.Gun("grpc_gun", NewGun, func() GunConfig {
     return GunConfig{
     Target: "localhost:84",
     }
     })
    cli.Run()
}

When creating our gun, we must specify everything to compile and run it.

• We create a provider, specifying the name “grpc_provider”
• We create a gun and specify the name “grpc_gun”
• In the target, we specify**“localhost:84”** the location where our service is accessible.

Okay, let’s continue. We have successfully copied and adjusted the code for the gun. Now, how do we run it, and what do we need? First, let’s create ammo again. This time, we’ll simply describe them like this:

{"tag": "/ValidateCase", "Param1": "validate_doc_grpc_pandora"}

This is necessary so that, in the method, we can retrieve the parameters needed for the test. In our case, we will retrieve the document name specified in Param1. Our code will automatically pass this parameter to the ValidateCase method, and we’ll be able to retrieve it.

Great, the ammo is ready. Now, let’s configure the test itself:

pools:
  - id: HTTP pool
    gun:
      type: grpc_gun   # custom gun name (specified at `register.Gun("my_custom_gun_name", ...`)
      target: "localhost:84"
    ammo:
      type: grpc_provider
      source:
        type: file
        path: validate-json.ammo
    result:
      type: phout
      destination: ./phout.log
    rps: { duration: 60s, type: const, ops: 1000 }
    startup:
      type: once
      times: 10

Here, we specify our ammo as “validate-json.ammo”
Their type is “grpc_provider”
Our gun is “grpc_gun”

The load is 1000 rps for 60 seconds:

rps: { duration: 60s, type: const, ops: 1000 }

And the output goes to the file “./phout.log” — Yandex Tank understands this.

The finishing straight — let’s now perform three simple actions:

1. Build our gun.
2. Assemble a new configurator for the tank.
3. Run tests and check the results.

Building it is very straightforward — just like a simple Go application:

GOOS=linux GOARCH=amd64 go build

It’s important to specify `GOOS=linux GOARCH=amd64`; otherwise, you might encounter errors when launching the tank. That’s it; the binary file is ready. Let’s update another configurator:

phantom:
  enabled: false
pandora:
  enabled: true
  package: yandextank.plugins.Pandora
  pandora_cmd: ./grpc-gun # Pandora executable path
  config_file: ./validate-load.yml # Pandora config path
overload:
  enabled: true
  package: yandextank.plugins.DataUploader
  job_name: "grpc validate report"
  token_file: "env/token.txt"
telegraf:
  enabled: false
autostop:
  autostop:
    - time(1s,10s) # if request average > 1s
    - http(5xx,100%,1s) # if 500 errors > 1s
    - http(4xx,25%,10s) # if 400 > 25%
    - net(xx,25,10) # if amount of non-zero net-codes in every second of last 10s period is more than 25

Specify where our gun and its settings are located. Leave everything else as before and run it all with this script:

echo "run tank"
docker run \
 -v $(pwd):/var/loadtest \
 - net="host" \
 -it direvius/yandex-tank -c validate-tank-load.yml

After that, we will see our results.

Special gun for flatbuffer!

For flatbuffer, the story repeats. All you need to do is adjust your client code from grpc to http. Here’s an example:

type Gun struct {
   // Configured on construction.
   client http.Client
   conf GunConfig
   // Configured on Bind, before shooting.
   aggr core.Aggregator // Maybe your custom Aggregator.
   core.GunDeps
   files [][]byte
}

func NewGun(conf GunConfig) *Gun {
   return &Gun{conf: conf}
}

func (g *Gun) Bind(aggr core.Aggregator, deps core.GunDeps) error {
   c := http.Client{Timeout: time.Duration(1) * time.Second}
   g.client = c
   g.GunDeps = deps
   g.aggr = aggr
}

And then create similar methods and describe their behavior:

func (g *Gun) caseValidate(client *http.Client) int {
  host := g.conf.Target
  response, err := client.Post(host+"/report/validate", "application/octet-stream", bytes.NewBuffer(g.files[2]))
  if err != nil {
   log.Printf("FATAL: %s", err)
   return 500
  }
  return response.StatusCode
}

That’s it!

In conclusion

We’ve explored a new tool for performance testing called Yandex Pandora. You can run it as a standalone load engine if you don’t need fancy metrics. Alternatively, you can run it based on Yandex Tank, and in this case, you’ll get beautiful graphs and all the features of this cool tool!

That’s all, I hope it was helpful! Thank you.

Full code here:

Also published here.