How To Run Cloud Tests From the Command Line

What is a CLI

A command line interface (or CLI) is a text-based interface used for entering commands. In the early days of computing, before the mouse, it was the standard way to interact with a computer. While the graphic user interface (GUI) has largely replaced CLIs, they are still included with several major operating systems, such as Windows and OS X.

There are many different types of command line interfaces, but the two most popular ones are DOS (for Windows) and the bash/zsh shell (for Linux and OS X). Each CLI uses its own command syntax, but they all function in a similar way. For example, every CLI has a command prompt, which is displayed when the interface is ready to accept a command. When you type a command, it shows up next to the command prompt, and when you press Enter, the command is executed.

source = TechTerms

Difference between GUI and CLI

The operating system of any device is nothing but an interface between user and hardware components of that device. The operating system offers a GUI vs CLI user interface to interact with an electronic device. Some operating systems provide GUI and CLI while others offer only CLI. GUI means a Graphical user interface while CLI means the command line interface. As the name suggests, one has to write commands to perform a certain task in the CLI system. On the other hand, GUI offers graphics that consists of icons and images that enable a user to do a task directly. CLI requires expertise in commands for performing a certain task while GUI can be operated by a beginner.

For its working, GUI uses computer graphics. One can click on the icon; drag the object with help of mouse. There is no need to remembering commands in this. GUI comes with numerous components. With the help of a textbox editor, one can enter input data in this. A menu offers a list of options to choose from. Buttons enable the user to select a particular option. The checkbox element enables the user to select an option to choose from multiple alternatives. Linux and Windows use a Graphical User Interface. It consists of icons, search boxes, windows, menus, and many other graphical elements.

source = EDUCBA

When to use the CLI

It’s not one tool is better than the other one (CLI vs GUI) in general but we have to put them in context and see the requirements. One tool might be better for this situation but the other one can be the best tool for another occasion.

• Do things at scale — A simple CLI command can easily adjust configurations for a large group of systems at once
• Something needs to be scripted and automated
• You need greater control over system functions
• Use NPM for Package Installs as Node Package Manager is easily the most popular tool for modern developers and it does not have a GUI
• Utilize Git Version Control
• For less memory usage
• Use CI/CD systems

source = TablePlus

CI/CD compatibility

In software engineering, CI/CD or CICD generally refers to the combined practices of continuous integration and either continuous delivery or continuous deployment.

CI/CD bridges the gaps between development and operation activities and teams by enforcing automation in building, testing and deployment of applications. Modern day DevOps practices involve continuous development, continuous testing, continuous integration, continuous deployment and continuous monitoring of software applications throughout its development life cycle. The CI/CD practice or CI/CD pipeline forms the backbone of modern day DevOps operations.

Hence, CI/CD is the perfect use-case for the CLI as it allows the user to create automated workflows that can define, scale and capture exceptions for a deployment and testing process of an application from the point of its inception (infrastructure dependencies) to the point of going live (final live test was successful).

source = Rungutan — What is a CI/CD pipeline?

How do I download the Rungutan CLI

The Python Package Index (PyPI) is a repository of software for the Python programming language.

Check Python version

Before you go any further, make sure you have Python and that the expected version is available from your command line. You can check this by running:

mariusmitrofan@Rungutan ~ >> python --version
Python 2.7.16
mariusmitrofan@Rungutan ~ >> python3 --version
Python 3.8.2

Due to the way most Linux and MacOS distributions are handling the Python 3 migration, which is my case as well, your default version of Python installed would be 2.7.* and the Python3 version would only be available through the python3 binary.

Due to the following 2 reasons, the Rungutan CLI would only work with the Python3 version:

• Python 2 is no longer supported — Python2 was released in the year 2000 and Python3 was started in 2006, and while they both received continuous update over the years, recent developments have pointed the community to deprecate Python2 in 2015 and drop support altogether on the 1st of January 2020.
• Python 3 is the de-facto version installed on new operating systems after 2020, and although Mac OS is a bit lagging behind this specific change (unlike the Linux community), they have officially communicated that they will stop shipping Python 2 as well with newer versions of the distro.

Install/Upgrade PIP

The following commands show ensure that pip is installed through the standard library and then updated to the latest version.

mariusmitrofan@Rungutan ~ >> python3 -m ensurepip --default-pip
mariusmitrofan@Rungutan ~ >> pip3 install --upgrade pip

In some cases, your system might output an error, for one or both of the above commands, saying that the current does not have permissions to properly process these commands.

In that case, the solution is to prepend “sudo -H “ to the command, and input your OS password, as so:

mariusmitrofan@Rungutan ~ >> sudo -H pip3 install --upgrade pip
Password:
Collecting pip
  Downloading https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)
     |████████████████████████████████| 1.5MB 2.3MB/s
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-21.0.1

Install/Upgrade the Rungutan CLI

Now that your system is up and running with the latest (and stable) version of Python3 and PIP3, you can proceed to actually install the Rungutan CLI as so:

mariusmitrofan@Rungutan ~ >> sudo -H pip3 install --upgrade rungutan
Collecting rungutan
  Using cached rungutan-1.6.3-py3-none-any.whl (16 kB)
Requirement already satisfied: simplejson in /private/var/root/Library/Python/3.8/lib/python/site-packages (from rungutan) (3.17.2)
Installing collected packages: rungutan
Successfully installed rungutan-1.6.3

After you have successfully installed it, you should be able to easily check the current installed version and see the help menu, as so:

mariusmitrofan@Rungutan ~ >> rungutan version
1.6.3
mariusmitrofan@Rungutan ~ >>  rungutan help
usage: rungutan  []
To see help text, you can run:
    rungutan help
    rungutan version
    rungutan configure --help
    rungutan domains --help
    rungutan team --help
    rungutan results --help
    rungutan raw_results --help
    rungutan tests --help
    rungutan templates --help
    rungutan crons --help
    rungutan notifications --help
    rungutan vault --help
Rungutan CLI utility for interacting with https://rungutan.com
positional arguments:
  command     Command to run
optional arguments:
  -h, --help  show this help message and exit

Authenticating with the Rungutan CLI

The Rungutan CLI supports the following methods of authentication:

• CLI arguments that are served on your local system in your user’s home folder
• Environment variables

The local user profile takes precedence over any defined environment variables, meaning that if you already authenticated using CLI arguments at any time in the past (and did not delete or overwrite your profile), then those specific credentials are used, regardless of any injected environment variables.

How do I authenticate?

In order to authenticate against the Rungutan platform, you need to provide 2 values:

• Team ID — which is the team name under which you are executing the commands
• API Key — which is the Rungutan API key generated at sign up for all users on the platform

In case your API Key gets leaked, use the “Refresh API Key” button to delete the old one and generate a new one for your account!

Authenticate using CLI arguments

In order to use the CLI arguments to authenticate, you would run the following command and input the parameters as they are being requested:

mariusmitrofan@Rungutan ~ >> rungutan configure 
TEAM_ID: rungutan 
API_KEY: ENShJ5w1Jw8yaIcYdYLDy8WWkAEPIjNL7Cn1cl4F

Once you have authenticated, you should be able to run any supported command in the list, such as listing templates for example:

mariusmitrofan@Rungutan ~ >> rungutan templates list
{
  "Templates": [
    {
      "template_id": "some-template-id-here",
      "test_name": "DEMO - Launch tests in APAC",
      "template_name": "DEMO - Launch tests in APAC",
      "domain_name": "some-domain-name",
      "threads_per_region": 2,
      "num_clients": 250,
      "hatch_rate": 10,
      "created_date": "2021-02-10T06:16:14Z",
      "run_time": 30,
      "member_email": "some-email-address",
      "test_region": "ap-northeast-1,ap-northeast-2"
    }
  ]
}

Authenticate using environment variables

In order to use the environment variables, you have to make sure the following two environment key variables are defined:

The only difference is that instead of running the “rungutan configure” command, you would simply skip it, and directly run the command that you desire, by ensuring that the environment variables are present, as following:

mariusmitrofan@Rungutan ~ >> export RUNGUTAN_TEAM_ID=rungutan
mariusmitrofan@Rungutan ~ >> export RUNGUTAN_API_KEY=ENShJ5w1Jw8yaIcYdYLDy8WWkAEPIjNL7Cn1cl4F
mariusmitrofan@Rungutan ~ >> rungutan templates list
{
  "Templates": [
    {
      "template_id": "some-template-id-here",
      "test_name": "DEMO - Launch tests in APAC",
      "template_name": "DEMO - Launch tests in APAC",
      "domain_name": "some-domain-name",
      "threads_per_region": 2,
      "num_clients": 250,
      "hatch_rate": 10,
      "created_date": "2021-02-10T06:16:14Z",
      "run_time": 30,
      "member_email": "some-email-address",
      "test_region": "ap-northeast-1,ap-northeast-2"
    }
  ]
}

When should I use which?

Usually, environment variable authentication is used in systems in which the user does not have direct terminal access, does not need to access it directly or should not be accessed directly.

The most basic use case for using environment variables is in defining CI/CD pipelines, in which usually those environment variables are defined at the project/sub-project levels as secrets which are injected automatically into any new job.

For more details on how to do this with GitHub Actions, see our CI/CD article.

Can I use the CLI with multiple teams?

Of course, but it of course requires that you authenticate with each of those teams separately and it is only available through the CLI arguments logic.

So, instead of simply running “rungutan configure”, you could authenticate twice as so:

mariusmitrofan@Rungutan ~ >> rungutan configure --profile rungutan
TEAM_ID: rungutan
API_KEY: ENShJ5w1Jw8yaIcYdYLDy8WWkAEPIjNL7Cn1cl4F

mariusmitrofan@Rungutan ~ >> rungutan configure --profile rungutan2
TEAM_ID: rungutan2
API_KEY: ENShJ5w1Jw8yaIcYdYLDy8WWkAEPIjNL7Cn1cl4F

The only caveat is that now you would have to always define the profile you would like to use when using commands, in order to instruct the CLI which authentication chain to use, as so:

mariusmitrofan@Rungutan ~ >> rungutan templates list --profile rungutan 
{
  "Templates": [
    {
      "template_id": "some-template-id-here",
      "test_name": "DEMO - Launch tests in APAC",
      "template_name": "DEMO - Launch tests in APAC",
      "domain_name": "some-domain-name",
      "threads_per_region": 2,
      "num_clients": 250,
      "hatch_rate": 10,
      "created_date": "2021-02-10T06:16:14Z",
      "run_time": 30,
      "member_email": "some-email-address",
      "test_region": "ap-northeast-1,ap-northeast-2"
    }
  ]
}

If no profile is defined when running a command (through the “ — profile”) option, then the default profile is used, which would of course be the one that was defined through the simple “rungutan configure” command.

Using the Rungutan CLI

The Rungutan CLI offers the exact same experience that you would get through the web platform, directly in your terminal, without the flashiness :)

Every menu, command, option that is available for your user role in the web platform, you are able to replicate it via the CLI, and of course an API call.

For instance, running a test from a template but modifying its name first, waiting for it to finish and then checking the summary results looks like this:

mariusmitrofan@Rungutan ~ >> rungutan templates list
{
    "Templates": [
        {
            "template_id": "some-template-id-here",
            "test_name": "DEMO - Launch tests in APAC",
            "template_name": "DEMO - Launch tests in APAC",
            "domain_name": "some-domain-name",
            "threads_per_region": 2,
            "num_clients": 250,
            "hatch_rate": 10,
            "created_date": "2021-02-10T06:16:14Z",
            "run_time": 30,
            "member_email": "some-email-address",
            "test_region": "ap-northeast-1,ap-northeast-2"
        }
    ]
}
mariusmitrofan@Rungutan ~ >> rungutan tests add --template_id some-template-id-here --test_name "Test ran from the CLI" --wait_to_finish
{
    "test_id": "a-new-test-id-here",
    "test_name": "Test ran from the CLI",
    "message": "Successfully created new test"
}
Waiting for test to finish...
{
    "Results": {
        "region": "overall",
        "failures": [],
        "public_ips": [
            "3.35.12.2",
            "15.164.226.216",
            "52.199.126.145",
            "13.230.29.159"
        ],
        "requests": [
            {
                "Method": "GET",
                "Path": "/",
                "num_requests": 16092,
                "min_response_time": 13,
                "median_response_time": 48,
                "avg_response_time": 59,
                "max_response_time": 653,
                "response_time_percentiles": {
                    "55": 50,
                    "65": 57,
                    "75": 65,
                    "85": 83,
                    "95": 135
                },
                "total_rps": 537,
                "total_rpm": 32220
            },
            {
                "Method": "GET",
                "Path": "/blog/",
                "num_requests": 16053,
                "min_response_time": 13,
                "median_response_time": 44,
                "avg_response_time": 50,
                "max_response_time": 738,
                "response_time_percentiles": {
                    "55": 45,
                    "65": 49,
                    "75": 55,
                    "85": 66,
                    "95": 90
                },
                "total_rps": 536,
                "total_rpm": 32160
            }
        ]
    },
    "ResultsMetadata": {
        "team_id": "rungutan",
        "test_id": "a-new-test-id-here"
    }
}

Extending the Rungutan CLI

Besides the obvious use case of deploying the Rungutan CLI as part of your CI/CD pipeline to execute Performance Testing after every (minor and) major release of your platform, you also have the possibility to script it to automatically generate reports that can be delivered to key stakeholders.

As also mentioned in our Documentation, because out platform was built using the Serverless Technology from the ground up, you have the possibility to perform the same actions with the same confidence trough any of the following methods:

• Web platform
• CLI
• API call

This gives you the opportunity to script your user experience and perform any of the following maintenance tasks:

• Run Performance Test jobs as part of CI/CD pipelines
• Export result data to be stored in your own preferred on-premise/hybrid/cloud database
• Parse result data and analyse for abrupt and/or anomalies in response times

And many many more …

Final thoughts

Regardless of using Rungutan via the web platform, CLI or through direct API calls, your experience is 100% the same and you won’t feel any difference in how you interact with your tests and results!

Previously published at https://rungutan.com/blog/run-tests-cli/