The Markets Are Out of Order. How To #BreakTheCycle

Last week’s markets were a whirlwind making history for not one, but two crashes now called Black Monday (2020) and Black Thursday (2020). On each of these days, market-wide curbs, also called trading circuit breakers, kicked in to prevent panic selling and rapid market decline when the markets dropped more than 7% after the opening bell.

You can read more about these market-wide circuit breakers from NYSE here.

A Quick Note: Whether the coronavirus is a black swan or gray rhino, we at PEAK6 hope you and yours are staying healthy and positive during this crisis. We also hope you can join us in acting now to flatten the curve. As each of us are impacted by quarantines, school shutdowns, work from home, and other social distancing, remember that our adaptability is part of our genius. Be who you ought to be and have compassion for those who are in tough spots both now and in the coming weeks.

Seeing as these events happen so infrequently, I thought I’d capture some
detail around how our systems at PEAK6 experienced them. I won’t go into
system volume or resiliency again as I talked about those in Indebted to the Tech.

However, the graphs below show how our feed systems recorded these events with a nice big gap during the 15-minute breaks.

Flag on the Play, False Start...

When a market-wide circuit breaker is triggered, there’s a bit of a scramble. Similar to a false start in football with some confusion and players going back to starting positions, market participants quickly run through their order books acknowledging cancels, busts, and any other system anomalies before the markets reopen after the 15-minute pause.

Unlike football, however, when the markets come back, there is actually a bit of hesitancy that’s different from a normal open. Not only do we see wide markets with large spreads on most names, but there’s almost a trickle-in effect until market activity picks back up.

You can see this in our market viewing tools below. This first view shows
what we’d normally expect as markets are moving and bid/asks change.

However, this next view shows the market stagger with trades coming in after the circuit breaker.

Both of the days we saw market curbs, we also saw markets stabilize
after the break without further drops or need to trigger additional curbs later in the day. Meaning they did their job. Cool!

Breaking the Cycle

The circuit breaker pattern is widely used in software, just as in exchanges, to reduce harm and allow systems to stabilize. Michael Nygard’s Release It! popularized the approach back in 2007. Quick aside, Nygard also has a great talk on real-world software architecture called Architecture without an End State, which you should definitely check out.

What’s neat about circuit breakers is they acknowledge that there are times when further interaction with a failing system only increases the chance of overall system failure.

In this way, circuit breakers insulate a system while also allowing the system’s users a chance to fail-fast or accommodate for the failure.

As a quick example, consider a simple 3-tier architecture with a UI routing traffic to three instances of a backend service.

If service A3 starts experiencing issues at the data layer, it could use
a circuit breaker and open the circuit, disabling the connection from
the UI. This lets the UI know that the service is unhealthy while also
removing stress from DB A3.

If DB A3 stabilizes, then service A3 can either half-open (partially allow use while checking for errors) or close the circuit to re-enable operation. For a full rundown on the pattern, check out Martin Fowler’s write up here.

Breaking Some Code

There are many implementations of the circuit breaker pattern across languages, libraries, and frameworks. Netflix’s Hystrix library codified a Java version of the approach back in 2012 and libraries like resilience4j and goresilience are also great examples. In the code below from the goresilience library, a function called errorOnOddMinute will be called wrapped within a circuit breaker. As this is example code, the circuit will fluctuate from open to closed based on the provided config as errorOnOddMinute succeeds or fails every other minute.

func errorOnOddMinute(ctx context.Context) error {
 minute := time.Now().Minute()
 if minute%2 != 0 {
  return fmt.Errorf("error because %d minute is odd", minute)
 } return nil
}

func main() {
 runner := circuitbreaker.New(circuitbreaker.Config{
    //ErrorPercentThresholdToOpen:        50,
    //MinimumRequestToOpen:               20,
    //SuccessfulRequiredOnHalfOpen:       1,
    //WaitDurationInOpenState:            5 * time.Second,
    //MetricsSlidingWindowBucketQuantity: 10,
    //MetricsBucketDuration:              1 * time.Second,
 })for {
    time.Sleep(75 * time.Millisecond)    err := runner.Run(context.TODO(), errorOnOddMinute)
    if err != nil {
      if err == errors.ErrCircuitOpen {
        fmt.Println("[!] circuit open")
      } else {
        fmt.Printf("[-] execution error: %s\n", err)
      }
    } else {
      fmt.Printf("[+] good\n")
    }
  }
}

Breaking Your Own Circuits

One last thought on circuit breaking hits a little closer to home. The social distancing many of us are following through on because of the coronavirus is also a kind of circuit breaker. Instead of DB A3, we’re thinking about how our local hospitals and at-risk communities might be insulated from an overwhelming amount of new cases. We open the circuit between healthy, susceptible individuals and newly infected individuals without symptoms by distancing ourselves. If done effectively, it will relieve stress on our entire hospital system so it remains stable and can endure the load.

I hope this sheds some light on circuit breaking as a pattern, how it shows up in the wild, and how you might use it in your own systems. If you’re interested in patterns like this and writing resilient code, check out our open positions. We’re always looking for passionate, curious engineers who like working on hard problems!