A special kind of testing…

Not at the code level

Automatic application interaction

Ensure correct behaviour

… under heavy load !

Analyse performances and limitations

Optimise and check for regression

Validate behaviour

Identify critical scenarios

How ?

For HTTP: ApacheBench, Siege, …

And for the others ?It depends 😄

What about WebSockets ?

Standard real-time protocol

RFC6455 released in 2011

Full-Duplex: Client to server and server to client

« Just » a protocol, language agnostic

Only transfer data when useful

A persistent connection

Different paradigm, different problems

Memory management, long running process, shared state between clients…

🤔 how to test like HTTP ?

« Just » simulate client interaction

artillery.io

Open source & easy-to-use performance testing toolkit.

Installing

# Install
npm install artillery

# Verify it works
npm exec artillery dino

Hello Dino !

 ------------
< Artillery! >
 ------------
           \
               __
              / _)
     _/\/\/\_/ /
   _|         /
 _|  (  | (  |
/__.-'|_|--|_|

Your first scenario…

config:
  target: "https://example.org/"
  phases:
    #…
scenarios:
  - name: "Look at some page"
    flow:
      - get:
          url: "/"

Understanding virtual users

How much « users » for how long…

config:
  #[...]
  phases:
    - duration: 5
      arrivalRate: 5
      rampTo: 10
      name: "Say hello"
#[...]

Simple report

Elapsed time: 5 seconds
  Scenarios launched:  25
  Scenarios completed: 25
  Requests completed:  25
  Mean response/sec: 4.75

Response time (msec):
  min: 425.7
  max: 711.9
  median: 428.7
  p95: 665.5
  p99: 711.9
Codes:
  200: 25

Now with WebSockets

config:
  target: "wss://echo.wss-websocket.net"
  phases:
    - duration: 5
      arrivalRate: 5
scenarios:
  - engine: ws
    flow:
      - send: "hello"
      - think: 1
      - send: "world"

Go further, Processor

Enrich artillery with your own code.

Hooks (http): beforeRequest,afterResponse,{before,after}Scenario

Your function as step

config:
  processor: "./my-functions.js"
scenarios:
  - flow:
      - function: doSomething
      - send: "hello"

Some JavaScript

module.exports = {
    doSomething: doSomething
}

function doSomething(context, ee, next) {
    context.ws /* <= Actual WebSocket connection */

    /* MUST be called for the scenario to continue */
    return next();
}

Environment configuration

config:
  target: "wss://echo.wss-websocket.net"
  variables:
    httpUrl: http://localhost:8000
  environments:
    staging:
      target: "wss://custom.staging.url"
      variables:
        httpUrl: https://example.org/
    prod:
      target: "wss://custom.staging.url"
      variables:
        httpUrl: https://example.com/

Environment configuration

                            npm exec artillery run -e staging
                        

💡 Quick overview

YAML configuration

Clearly define tests steps

Configure and adapt per environment

Well documented !

Extend with code

Hooks and context during execution

Simple CLI interface

npm exec artillery run file.yaml

Runtime flags for grained control (-e, -v, …)

Limitations ?

There are no real users 🤖

Scenarios are just sequence of tasks

Not easy to act on received message