Jason Howard from Sheffield is part of a team rowing 3,000 miles from the Canary Islands to Antigua.

'Most crews make it, some don’t' - Yahoo Sports

Canary Technology Solutions has acquired Layer 8 Security, enhancing its cybersecurity offerings and promising a comprehensive approach to digital protection.

Canary Technology acquires Layer 8 Security to expand

Canary

canary

In a few words, the idea behind canary releases is to deliver a new software version to only a fraction of the users, analyze the results, and decide whether to proceed further or not. If results are not aligned with expectations, roll back; if they are, increase the number of users exposed until all users benefit from the new version. In this post, I'd like to detail this introduction briefly, explain different ways to define the fraction, and show how to execute it with Apache APISIX. Introduction to canary releases The term "canary" originates from the coal mining industry. When mining, it's not uncommon to release toxic gases: In a small enclosed space, it can mean quick death. Worse, the gas may be odorless, so miners would breathe it until it was too late to leave. Carbon monoxide is quite common in coal mines and is not detectable by human senses. For this reason, miners brought canaries with them underground. If the canary suddenly dropped dead, chances were high that such a gas pocket had been breached, and it was high time to leave the place. Years ago, we brought this approach to releasing a new software version. The analogy goes like this: miners are the Ops team deploying the version, the canary consists of all tools to measure the impact of the release, and the gas is a (critical) bug. The most crucial part is that of the release, including failure rates, HTTP status codes, etc., and compare them with those of the previous version. It's outside the scope of this post, but again, it's critical if you want to benefit from canary releases. The second most important part is the ability to roll back fast if the new version is buggy. you need to measure the impact Canary releases vs. feature flags Note that canary releases are not the only way to manage the risk of releasing new code. For example, feature flags are another popular way: The canary approach delivers the complete set of features in the new component version Feature flags deploy the component as well, but dedicated configuration parameters allow activating and deactivating each feature individually Feature flags represent a more agile approach (in the true sense of the word) toward rollbacks. If one feature out of 10 is buggy, you don't need to undeploy the new version; you only deactivate the buggy feature. However, this superpower comes at the cost of additional codebase complexity, regardless of whether you rely on third-party products or implement it yourself. On the other hand, canary requires a mature deployment pipeline to be able to deploy and undeploy at will. Approaches to canary releases The idea behind canary releases is to allow only a fraction of users to access the new version. Most canary definitions only define "fraction" as a percentage of users. However, there's more to it. The first step may be to allow only vetted users to check that the deployment in the production environment works as expected. In this case, you may forward only a specific set of internal users, , testers, to the new version. If you know the people in advance, and the system authenticates users, you can configure it by identity; if not, you need to fallback to some generic way, , HTTP headers - . e.g. e.g. X-Canary: Let-Me-Go-To-v2 Remember that we must monitor the old and the new systems to look at discrepancies. If nothing shows up, it's an excellent time to increase the pool of users forwarded to the new version. I assume you eat your own dog food, ; team members use the software they're developing. If you don't, for example, an e-commerce site for luxury cars, you're welcome to skip this section. i.e. To enlarge the fraction of users while limiting the risks, we can now indiscriminately provide the new version to internal users. We can configure the system to forward to the new version based on the client IP to do this. At a time when people were working on-site, it was easy as their IPs were in a specific range. Remote doesn't change much since users probably access the company's network via a VPN. Again, monitor and compare at this point. The whole nine yards At this point, everything should work as expected for internal users, either a few or all. But just as no plan survives contact with the enemy, no usage can mimic the whole diversity of a production workload. In short, we need to let regular users access the new version, but in a controlled way, just as we gradually increased the number of users so far: start with a small fraction, monitor it, and if everything is fine, increase the fraction. Here's how to do it with Apache APISIX. Apache APISIX offers a plugin-based architecture and provides a plugin that caters to our needs, namely the plugin. traffic-split The Plugin can be used to dynamically direct portions of traffic to various Upstream services. traffic-split This is done by configuring , which are custom rules for splitting traffic, and which is a set of Upstreams to direct traffic to. match weighted_upstreams -- traffic-split Let's start with some basic upstreams, one for each version: upstreams:
  - id: v1
    nodes:
      "v1:8080": 1
  - id: v2
    nodes:
      "v2:8080": 1 We can use the plugin to forward most of the traffic to v1 and a fraction to v2: traffic-split routes:
  - id: 1
    uri: "*"                                                     #1
    upstream_id: v1           
    plugins:           
      traffic-split:           
        rules:           
          - weighted_upstreams:                                  #2
            - upstream_id: v2                                    #3
              weight: 1                                          #3
            - weight: 99                                         #3 Define a catch-all route Configure how to split traffic; here, weights Forward 99% of the traffic to v1 and 1% to v1. Note that the weights are relative to each other. To achieve 50/50, you can set weights 1 and 1, 3 and 3, 50 and 50, etc. Again, we monitor everything and make sure results are as expected. Then, we can increase the fraction of the traffic forwarded to v2, : e.g. routes:
  - id: 1
    uri: "*"
    upstream_id: v1
    plugins:
      traffic-split:
        rules:
          - weighted_upstreams:
            - upstream_id: v2
              weight: 5                                          #1
            - weight: 95                                         #1 Increase the traffic to v2 to 5% Note that Apache APISIX reloads changes to the file above every second. Hence, you split traffic in near-real time. Alternatively, you can use the Admin API to achieve the same. More controlled releases In the above, I moved from internal users to a fraction of external users. Perhaps releasing to every internal user is too big a risk in your organization, and you need even more control. Note that you can further configure the plugin in this case. traffic-split routes:
  - id: 1
    uri: /*
    upstream_id: v1
    plugins:
      traffic-split:
        rules:
          - match:
            - vars: [["http_X-Canary","~=","Let-Me-Go-To-v2"]]   #1
          - weighted_upstreams:
            - upstream_id: v2
              weight: 5
            - weight: 95 Only split traffic if the HTTP header has the configured value. X-Canary The following command always forwards to v1: curl http://localhost:9080 The following command also always forwards to v1: curl -H 'X-Canary: Let-Me-Go-To-v1' http://localhost:9080 The following command splits the traffic according to the configured weights, , 95/5: i.e. curl -H 'X-Canary: Let-Me-Go-To-v2' http://localhost:9080 Conclusion This post explained canary releases and how you can configure one via Apache APISIX. You can start with several routes with different priorities and move on to the plugin.  The latter can even be configured further to allow more complex use cases. traffic-split The complete source code for this post can be found on . GitHub To go further: Implementation of canary release solution based on Apache APISIX Canary Release in Kubernetes With Apache APISIX Ingress Smooth Canary Release Using APISIX Ingress Controller with Flagger Originally published at on December 3rd, 2023 A Java Geek

Implementing Canary Releases with Apache APISIX

Apache APISIX এর সাথে ক্যানারি রিলিজ বাস্তবায়ন করা

Implementieren von Canary-Releases mit Apache APISIX

Implementación de versiones Canary con Apache APISIX

Implémentation des versions Canary avec Apache APISIX

Apache APISIX के साथ कैनरी रिलीज़ को कार्यान्वित करना

Apache APISIX を使用したカナリア リリースの実装

Apache APISIX를 사용하여 Canary 릴리스 구현

Implementando versões Canary com Apache APISIX

Реализация Canary-релизов с помощью Apache APISIX

Canary Sürümlerini Apache APISIX ile Uygulamak

Triển khai các bản phát hành Canary với Apache APISIX

使用 Apache APISIX 实施 Canary 版本

DUCKS—GOOSE—PEACOCK—TURKEY—GUINEA-FOWL—CANARY-BIRD—GOLD-FISH—HIVE-BEES—SILK-MOTHS

Introduction Kusama is a decentralized network of nodes that can be used to build and run decentralized applications (dApps). Kusama uses the same system architecture as Polkadot, which makes it a suitable testing ground for new features and functionality before they are integrated into the main network. Kusama is designed to be a platform for experimentation, allowing developers to try out new ideas in a live environment. Kusama is still in development, it is subject to change and may not be fully stable. What is Kusama? Kusama is a blockchain platform that enables developers to build and launch decentralized applications. The Kusama network uses the same technology as Polkadot, which makes it easy for developers to get started. Kusama also has its own native currency, called KSM, which can be used to purchase goods and services on the platform or traded on cryptocurrency exchanges. As a playground for developers, Kusama offers a safe environment in which they can experiment with new ideas and test their applications. Key features of Kusama Kusama is a decentralized network designed for canary releases, rapid development cycles, and experimentation for Polkadot. Kusama uses Substrate, an open-source framework for building blockchain applications, to create a platform that is censorship-resistant and scalable. Changes to the network must be approved by a two-thirds supermajority of the validators in order to be implemented. The native token of Kusama is called KSM and it is used for staking, paying transaction fees, and voting on governance proposals. Who are the developers behind Kusama? Kusama is a decentralized network designed to incentivize early adoption of Polkadot and its ecosystem. Kusama is owned by the Web3 Foundation, a Swiss non-profit foundation that promotes decentralized technologies. The Foundation provides grants and funding to support the development of the Kusama network and ecosystem projects. Kusama also has a governance model that allows for community involvement in decision-making processes. The native currency of the Kusama network is KSM, which is used for staking and transaction fees. Why Kusama is important for the innovation of the Polkadot ecosystem? Kusama is a Polkadot-based canary network that allows developers to experiment with new features and scalability solutions before they are implemented on Polkadot. This helps to ensure that the platform remains scalable and reliable and that the user experience is consistent across all devices. Kusama is also known as the because it's a place where development can happen more quickly and with fewer constraints than on mainnet. This makes Kusama ideal for testing new ideas and solutions before they are implemented on Polkadot's mainnet. "wilderness" or "parachain testnet" Anyone can join in on the fun by using Kusama parachains to access its unique features without having to be a developer or know about blockchain technology. Simply sign up, stake your tokens, and get started! To get started, check out where I explain how to create a Polkadot account. Making a Kusama account is the same as making a account. Once populated one can easily toggle any KSM or DOT parachain. Once set up a user can seamlessly, without leaving one central control board, utilize the whole ecosystem. my other article on Polkadot Polkadot Also published . here

What Is Kusama and Why Is It the Canary Network to Polkadot?

Is Continuous Integration enough for testing? With Continuous Integration, every code from developers’ workstations gets incorporated into a shared repository or a central repository, after which with automated build and tests, these codes are checked and merged. What we notice here is that these are synthetic tests, in simpler words, the scripts on the basis of which these tests happen only emulate user experience. They cannot help the IT team understand the actual end-user experience. This also doesn’t give insights on device resources and health state, which can also affect the application performance. Understanding “Canary” of Canary Deployment Before diving into the fact of - how Canary Deployment helps us in understanding the actual end-user experience, let’s begin our discussion by learning the significance of its name. Well, if somewhere your mind is comparing it with “Canary birds,” then you’re on the right track! In British Mining History, these humble birds were used as “toxic gas detectors.” While mining for coals, if there is any way emission of toxic gasses like Carbon Monoxide, Nitrogen dioxide, etc, these birds alerted the miners about its presence. These birds are more sensitive to airborne toxins as compared to human beings. The miners filled those caves with these soft-singing birds, and the moment they notice them falling sick or dying, they evacuate the place immediately before it was too late. Similarly, the DevOps Engineers perform a canary deployment analysis of their code in CI/CD pipeline to gauge any possible errors present. However, here the figurative canaries are a small set of users, who will experience all the glitches present in the update. Let’s define Canary Deployment! Canary deployment is a process or technique of controlled rolling of a software update to a small batch of users, before making it available to everyone. Thereby, reducing the chances of widescale faulty user experience. When the update is examined and feedback is taken, this feedback is again applied and then released on a larger scale. Steps Involved in Canary Deployment Strategy To keep our customers happy and engaged, it’s important to roll out new updates from time to time. Not ignoring the fact that every new change introduced, might have an error or two attached to it, we need a Canary Release Deployment analysis before releasing it to all our customers. With the level of competition in the market, any bug left unattended is going to attract customers’ displeasure and might cause a good loss of the Company’s reputation. Start off by understanding the basic structure of the canary deployment strategy. We can elucidate it under the following headings: Creation Analysis Roll-out/Roll-back Creation To begin with, we need to create a canary infrastructure where our newest update gets deployed. Now, we need to direct a small amount of traffic to this newly created canary instance. Well, the rest would be still continuing with the older version of our software model. Analysis Now, it’s showtime for our DevOps team! Here they need to constantly monitor the performance insights received — data collected from network traffic monitors, synthetic transaction monitors, and all possible resources linked to the canary instance. After the very awaited data gets collected, the DevOps teams then start comparing it with the baseline version’s data. Roll-out/Roll-back After the analysis is done, it’s time to think over the results of the comparative data, and whether rolling out a new feature is a good decision, or is it better to stick back to our baseline state and roll back the update. Well, then how are we benefitted? Zero Production Downtime You know it when there’s small traffic, and the canary instance is not performing as expected, you can simply reroute them to your baseline version. When the engineers are conducting all sorts of tests at this point, they can easily pinpoint the source of error, and can effectively fix it, or roll back the entire update and prepare for a new one. Cost-Efficient — Friendly with smaller Infrastructure The goal of Canary Release Deployment analysis is to drive a tiny amount of your customers to the newly created canary instance — where the new update is deployed. This means you’re using a little extra of your infrastructure to facilitate the entire process. In addition to that, if we compare it with the blue-green deployment strategy, it requires an entire application hosting environment for deploying the new application. As compared to blue-green deployment, we don’t really have to put our efforts into operating and maintaining the environment, in canary deployment, it’s easier to enable and/or disable any particular feature based on any criteria. Room for Constant Innovation The flexibility of testing new features with a small subset of users, and being able to receive end-user experience immediately, is what motivates the dev team to bring in constant improvements/updates. We can increase the load of the canary instance up to 100% and can keep track of the production stability of the enrolled features. Do we have any Limitations? Well, everything has limitations. What’s important for us is to understand how to counteract them. Time-consuming and Prone to Errors Enterprises executing a canary deployment strategy, perform the deployments in a siloed fashion. Then a DevOps Engineer is assigned to collect the data and analyze it manually. This is quite time-consuming as it is not scalable and hinders rapid deployments in CI/CD processes. There might be some cases where the analysis might go wrong and we might roll back a good update or roll forward a wrong one. On-Premise Applications are difficult to update Canary Deployment looks like an appropriate and quite a possible approach when it comes to applications present in Cloud. It is something to think about — when the applications are installed on personal devices. Even then, we can have a way around it by setting up an auto-update environment for end-users. Implementations might require some skill! We are focussing right now, is on the flexibility it offers to test different versions of our application, but we should also bring our attention to managing the databases associated with all these instances. For performing a proper canary deployment and to be able to compare the old version with the new one, we need to modify the schema of the database to support more than one version of the application. Thereby, allowing the old and new versions to run simultaneously. Wrapping Up The essence of the “canary in a coal mine” has been captured so well in strategizing the modern deployment strategy - Canary Deployment Strategy. Back in the old days the major concern was - “ ” to the present-day concept saying - “ ,” our lives have evolved much. Technology is the reason for this change. That which began with simplifying our lives is now aiming at giving us a personalized experience. save the human lives save the human interests With an increasing interest of Enterprises to perform canary deployment analysis, it is to note that we need to counteract the limitations and make processes smoother. We need some good ( , , etc) to automate certain functionalities to keep errors at bay, and also integrate security at every stage of development. continuous delivery solution providers or Managed Kubernetes orchestrators Buildpiper Platform9 Also Published here

Does your Business Need Canary Deployment?

New service releases deployed into DoorDash’s microservice architecture immediately begin processing and serving their entire volume of production traffic. If one of those services is buggy, however, customers may have a degraded experience or the site may go down completely. Although we currently have a traffic management solution under development for gradual service rollouts as a long-term solution, we needed an interim solution we could implement quickly. Our short-term strategy involved developing an in-house deployment process powered by a custom controller. Getting this solution in place gives our engineers the ability to gradually release code while monitoring for issues. With this short-term solution, we were able to ensure the best possible user experience while still ramping up the speed of our development team. Canary Kubernetes The problems with our code release process Our code release process is based on the approach implemented using . This setup gives us quick rollbacks. However, the drawback is that the entire customer base is immediately exposed to new code. If there are problems with the new release, all our customers will find a degraded experience. blue-green deployment Argo Rollouts Our long-term answer to this problem consists of a service mesh that can provide more gradual control of the traffic flow and a special purpose deployment tool with a set of custom pipelines. This solution would allow us to release new code gradually so that any defects do not affect our entire customer base. However, this new system will take several cycles to build, which meant we needed to come up with a short-term solution to manage this problem in the meantime. A quick overview of the blue-green deployment pattern At DoorDash, we release new versions of our code following the blue-green (B/G) deployment pattern, as shown in Figure 1, below. B/G reduces the risk of downtime by keeping two versions of the same deployment running simultaneously. The blue version is stable and serves production traffic. The green version is idle and considered the new release candidate. The actual release happens when the production traffic is switched from the blue version to the green version. After the switch, the blue version is still kept on hold as a backup in case there are problems with the green version. Rolling back to the blue version is instantaneous and immediately resolves any issues coming from the green version. Utilizing Argo Rollouts for B/G deployments Argo Rollouts is a framework that provides a set of enhanced deployment strategies, including B/G deployments, and is powered by a Kubernetes controller and some (CRDs). The Kubernetes objects created as part of the Argo Rollouts B/G CRD are called Rollouts. Rollouts are almost identical to , to which they add more advanced functionalities needed in scenarios such as B/G deployments. Kubernetes Custom Resource Definitions Kubernetes Deployment objects Figure 1: The blue-green (B/G) deployment process deploys the release candidate, called the green version, alongside the stable version, called blue, which is serving all production traffic. The green version is promoted by switching traffic from the blue version to the green version. The blue version is kept around idle for some time to allow for quick rollbacks. The Argo Rollouts B/G implementation in detail The Argo Rollouts B/G implementation is based on the Rollouts Kubernetes controller, a Kubernetes Rollout object, and two : one named Active Service and another named Preview Service. There are two separate versions of the Rollout: the blue and the green. Each version comes with a label that stores the (unique) hash of its specs and is computed by the controller. Kubernetes Services rollouts-pod-template-hash The controller injects a selector into the Active Service with a value that matches the blue hash, effectively causing the Active Service to forward production traffic to the blue version. In the same way, the Preview Service can be used to target the green version in order to perform all tests needed to qualify it as production-ready. Once the green version is deemed ready, the Active Service selector value is updated to match the green hash, redirecting all production traffic to the green version. The blue version is then kept idle for some time in case a rollback is needed. The ability to almost instantaneously switch the incoming traffic from the blue version to the green version has two important consequences: Rolling back is very fast (as long as the blue version is still around). The traffic switch exposes the production traffic to the green version all at once. In other words, the entire user base is affected if the green version is buggy. Our objective is a gradual release process that prevents a bug in the new release candidate code from negatively affecting the experience of our entire customer base. A gradual deployment strategy requires a long-term effort The overall solution to the deployment process problem we faced was to pursue a next-generation deployment platform and a traffic management system that can provide us with more granular control of the traffic flow. However, the scope of such a workstream and the potential impact over our current setup make these changes only feasible within a time range spanning multiple quarters. In the meantime, there were many factors pushing us to find a short-term solution to improve the current process. Why we needed to improve the current process While our long-term strategy for gradual deployments was in the works, our systems were still at risk of disappointing all our customers at every deployment. We needed to ensure that customers, merchants, and Dashers would not be negatively affected by future outages that could erode their trust in our platform. At the same time, our strong customer obsession pushed many of our teams towards adjusting their deployment schedules to minimize the potential customer impact of a buggy release. Late-night deployments became almost a standard practice, forcing our engineers to babysit their new production instances until early morning. These and other motivations pushed us towards finding a short-term solution to provide our engineers with gradual deployments capabilities, without having to utilize non-optimal working requirements. Our short-term strategy for gradual code releases: an in-house Canary solution A Canary solution would be a fast, easy-to-build short-term solution that would allow teams to monitor gradual rollouts of their services so that buggy code would have minimal impacts and could be rolled back easily. This process involved figuring out our functional requirements, configuring our Kubernetes controller, and avoiding obstacles from Argo Rollouts. Short-term solution requirements We started by laying down a list of functional and non-functional requirements for our solution. It had to: leverage native Kubernetes resources as much as possible to reduce the risk and overhead of introducing, installing, and maintaining external components use exclusively in-house components to minimize the chance of our schedule getting delayed by factors, such as external dependencies not under our control provide an interface that is easy to use and integrated with tools our engineers were already familiar with provide powerful levers to control gradual deployments, but at the same time put guardrails in place to prevent potential issues like excessive stress on the Kubernetes cluster lay out a streamlined adoption process that is quick, easy, and informed avoid the additional risks of changing the existing deployment process, and instead integrating it as an additional step Core idea: leverage the Kubernetes Services forwarding logic Our solution was based on the creation of an additional Kubernetes Deployment, which we called Canary, right next to the Kubernetes Rollout running the blue version serving our production traffic. We could sync the labels of the Canary Deployment with the selectors of the Kubernetes Active Service, causing the latter to start forwarding a portion of production traffic to the Canary Deployment. That portion could be estimated starting from the fact that a Kubernetes Service load balances the incoming traffic among all the instances with labels matching its selectors. Therefore, the approximate percentage of incoming traffic hitting the Canary Deployment was determined by the following formula: where #canary and #production represent, respectively, the number of instances belonging to the Canary Deployment and to the existing production Rollout. Our engineers can increase and decrease the number of instances in the Canary Deployment, respectively increasing and decreasing the portion of traffic served by the Canary Deployment. This solution was elegant in that it was simple and satisfied all our requirements. The next step was to figure out how to sync the labels of the Canary Deployment with the selectors of the Active Kubernetes Service in order to implement the Canary solution. Building a custom Kubernetes controller to power our Canary logic One challenge to implementing our solution was to come up with a way to inject the Canary Deployment with the right labels at runtime. Our production Rollouts were always created with a predefined set of labels which we had full control over. However, there was an additional label injected by the Rollouts controller as part of the Argo Rollout B/G implementation. Therefore, we needed to retrieve the value of these labels from the Kubernetes Active Service and attach them to our Canary Deployment. Attaching the labels forces the Kubernetes Active Service to start forwarding a portion of the production traffic to the Canary Deployment. In line with the , we wrote a custom Kubernetes controller to perform the label-syncing task, as shown in Figure 2, below: rollouts-pod-template-hash Kubernetes controller pattern Figure 2: Our custom Kubernetes controller listens to Kubernetes Events related to the creation or update of Canary Deployments. Once a matching Event is received, the controller updates the labels of the corresponding Canary Deployment so that the Kubernetes Service will start forwarding a portion of production traffic to it. This portion can be increased by scaling up the Canary Deployment. A Kubernetes controller is a component that watches the state of a Kubernetes cluster and performs changes on one or more resources when needed. The controller does so by registering itself as a listener for generated by actions ( , , ) targeting a given Kubernetes resource (Kubernetes Pod, Kubernetes ReplicaSet, and so on). Once an event is received, the controller parses it and, based on its contents, decides whether to execute actions involving one or more resources. Kubernetes events create update delete The controller we wrote registered itself as a listener to and events involving resources of type Kubernetes pods with a label set to . After an event of this kind is received, the controller takes the following actions: create update canary true unwraps the event and retrieves the pod labels stops handling the event if a label is already present rollouts-pod-template-hash constructs the Kubernetes Active Service name given metadata provided in the pod labels sends a message to the Kubernetes API server asking for the specs of a Service with a matching name retrieves the selector value from the Service specs if present rollouts-pod-template-hash injects the value as a label to the pod rollouts-pod-template-hash Once the design of our Canary deployment process was concluded, we were ready to put ourselves in the shoes of our customers, the DoorDash Engineering team, and make it easy for them to use. Creating a simple but powerful process to control Canary Deployments With the technical design finalized, we needed to think about the simplest interface that could make our Canary Deployment process easy to use for the Engineering team we serve. At DoorDash, our deployment strategy is based on the model and is powered by a called / . Our engineers type build and deploy commands in Slack channels where multiple people can observe and contribute to the deployment process. ChatOps Slackbot ddops We created three additional /ddops commands: , to create a new Canary Deployment with a single instance only. This would immediately validate that the new version was starting up correctly without any entry point issues. canary-promote , to grow or shrink the size of the Canary Deployment. This step provides a means to test out the new version by gradually exposing more and more customers to it. canary-scale , to bring down the entire Canary Deployment. canary-destroy Once the interface was ready to use, we thought about how to prevent potential problems that may come up as a result of our new Canary Deployment process and build guardrails. Introducing guardrails During the design phase, we identified a few potential issues and decided that some guardrails would need to be put in place as a prevention mechanism. The first guardrail was a hard limit on the number of instances a Canary Deployment could be scaled to. Our Kubernetes clusters are already constantly under high stress because the B/G process effectively doubles the number of instances deployed for a given service while the deployment is in progress. Adding Canary pods would only increase this stress level, which could lead to catastrophic failures. We looked at the number of instances in our services deployments and chose a number that would allow about 80% of services to run a Canary Deployment big enough to achieve an even split between the Canary and non-Canary traffic. We also included an override mechanism to satisfy special requests that would need more instances than we allowed. Another guardrail we put in place prevented any overlap between a new B/G deployment and a Canary Deployment. We embedded checks into our deployment process so that a new B/G deployment could not be started if there were any Canary pods around. Now that the new Canary Deployment process was shaping up to be both powerful and safe, we moved on to consider how to empower our engineers with enough visibility into the Canary metrics. Canary could not exist without proper observability Our new Canary Deployment allows our engineers to assess whether a new release of their service is ready for prime time using a portion of real-life production traffic. This qualification process is essentially a comparison between metrics coming from the blue version serving production traffic and metrics coming from the Canary Deployment running the new release candidate. Looking at the two different categories of metrics, our engineers can evaluate whether the Canary version should be scaled up further to more customer traffic or should be torn down if it’s deemed buggy. The implementation of this observability aspect was completely in the hands of our engineers, as each service had its own naming conventions and success metrics that could vary substantially based on the nature of the service. A clear separation between Canary and non-Canary metrics was possible thanks to specific labels injected only to the Canary pods. This way, teams could just replicate the queries powering their dashboard, adding an additional filter. We designed adoptions to be easy, quick, and informed The next and final step was to plan for helping the Engineering team adopt the Canary Deployments. As always, our objective was a quick and painless adoption. At the same time, we were providing our engineers with a powerful tool that was using our production traffic as a testbed, so we needed them to be empowered with knowledge about potential pitfalls and how to act in case things went wrong. To onboard engineers quickly and safely, we designed a four-step adoption process that they could follow without any external help, as it was simply re-using concepts they were already familiar with. In this case, we tried to stay away from fully automating the adoption process as we wanted them to understand its details. We also added a validation step at the end of the process that required explicit approval. This validation step enabled us to sit down with first-time users and guide them through their first Canary Deployment, explaining what was happening under the hood and what to do in case of failure. The adoption campaign went better than anticipated Initially, we prioritized adoptions for about a dozen services on the golden path of our workflows. Reducing the chances of those services suffering from a full outage meant improving by a great deal the reliability of the core customer flows that power our business. We had a lot of positive feedback about the adoption procedure and the new Canary process overall, and what we observed was that additional teams onboarded their services and scheduled time with us to review their configuration. We ended up onboarding about twice the number of services we anticipated before the end of the year, which was a huge success, taking into account a deploy freeze during the holiday season. Conclusion Releasing new code carries risk and associated costs that can be mitigated using a gradual rollout strategy. However, Kubernetes does not provide this capability out of the box. All the existing solutions require some form of traffic management strategy, which involves significant changes at the infrastructure level and an engineering effort that can last multiple quarters. On the other hand, the Kubernetes controller pattern is a simple but extremely powerful technique that can be used to implement custom behavior for Kubernetes resources, like advanced deployment strategies. Thanks to our new Canary Deployment process powered by a custom Kubernetes controller and other company-wide efforts towards improving the reliability of DoorDash services, our customers, merchants, and Dashers can put their trust in our platform while our engineers work with greater velocity. Interested in taking on challenging problems with practical solutions in a collaborative team environment? ! Consider joining DoorDash engineering Acknowledgments The author would like to thank Adam Rogal, Ivar Lazzaro, Luigi Tagliamonte, Matt Ranney, Sebastian Yates, and Stephen Chu for their advice and support during this project. Previously published at https://doordash.engineering/2021/04/14/gradual-code-releases-using-an-in-house-kubernetes-canary-controller/

Gradual Code Releases: An In-House Kubernetes Canary Controller

Perform Canary Deployments with AWS App Mesh on Amazon ECS Fargate

The Canary Dies

In this post we’ll walk through different AWS services and features that enable canary deployments of Lambda Functions , although you can check the Canary Deployments Serverless Plugin if you just want to safely deploy your functions and you are not interested about the details. Deployment in a Serverless application is an all-at-once process, when we release a new version of any of our functions, every single user will hit the new version. We must be really confident about the new version, because if anything goes wrong and the function contains an error, all of our users will be experiencing ugly issues. However, AWS recently introduced a new feature that can make our deployment process much more reliable and secure: traffic shifting using aliases. How can alias traffic shifting help us? Usually, deploying a Lambda function involves that all the function invocations will execute the new code, either because we are updating or because we are pointing an alias to a new version. That means that if anything goes wrong, 100% of the invocations will be errored and we should quickly roll back to the previous version or, what is even worse, we might not notice the bug and we leave our system in an inconsistent state. However, with the introduction of alias traffic shifting, we can now specify on an alias, so that not all the invocations hit the new release, but only a certain amount of traffic is routed to the latest version. This means that Lambda will automatically between the two latest versions, allowing us to check how the new release behaves before completely replacing the previous one, minimizing the impact of a possible bug. $Latest version weights load balance requests As we see that the new version behaves correctly, we could then gradually update its weight, increasing the load it receives. We can do that in the AWS Console, through the or with some , but there’s a better way to handle Lambda function deployments. CLI open source tools that handle weight updates automatically Automating the deployment process Even though being able to do traffic shifting is a huge leap forward, let’s admit that having to update weights manually (or deploying our own management system) is not really convenient. Don’t worry, as usual AWS has us covered. With we can just specify how we want traffic to be shifted over the time and it the . There are three different types of deployment preferences: CodeDeploy automatically adjusts weights : we specify the percentage of traffic we want to shift and the time we want the deployment to last. So, if we pass 10% and 30 minutes, for example, the 10% of the traffic will be routed to the new version during half an hour. When that time has passed, all the traffic will be shifted to the new version. Canary : the amount of the traffic routed to the new version will be incremented according to the provided percentage and interval. So, if we configure it to increment a 10% of the traffic every 5 minutes, CodeDeploy will update the alias weights adding a 10% to the new version in intervals of 5 minutes, until all the traffic has been shifted. Linear : all the traffic is shifted to the new version straight away. All-at-once This way, we let CodeDeploy do all the heavy lifting in the deployment process, and change alias weights according to our preferences. Traffic will be shifted gradually and, in the meantime, we can check if our new function is behaving correctly and cancel the deployment if we see anything weird. How could we automate the roll back process, so that we don’t have to manually check how the system is performing? CodeDeploy has thought about that as well. Rolling back to the previous Lambda Function version CodeDeploy allows us to configure a and a traffic shifting , which are in fact Lambda functions that are triggered before and after the traffic shifting process. They’re suited for performing tasks like running . CodeDeploy expects to get notified about the success or failure of the hooks , otherwise it’ll assume they failed. In any case, if the hook failed either for not calling CodeDeploy or for explicitly calling it with a failure response, the deployment will be aborted and all the traffic will be shifted to the old function version. pre post hook integration tests within one hour Hooks are not the only way we can check that our function is behaving as expected, since we can provide CodeDeploy with a list of to monitor the deployment process. As the traffic shifting begins, CodeDeploy will track those alarms, cancelling the deployment and to the previous function version if any of them is . CloudWatch Alarms rolling back triggered Deployment process with CodeDeploy and Lambda weighted alias Hooks and alarms allow us to monitor the whole deployment process. We can perform some tests before routing any traffic to the new function version, track alarms during the traffic shifting process and run more tests right after all the traffic is hitting the new version. If CodeDeploy notices something wrong in any of those steps, it will automatically roll back to the old, stable function version. CloudFormation all the things All that sounds really good, but how do we set it up? Well, doing it manually doesn’t seem to be a reasonable option, but fortunately AWS has an awesome service for defining and provisioning infrastructure: . This tool gives us a way to model our system in YAML or JSON templates, which we can use to create a collection of related AWS resources. As you can guess, the syntax of a template where we can define almost any imaginable resource in the AWS ecosystem and its configurations it’s intricate. They even built a simplified syntax to define Serverless applications, the , although it only supports a tiny subset of the AWS resources. The best way to deal with CloudFormation in a Serverless environment is, hands down, the , that provides a nice an easy DSL, which then turns into a CloudFormation template, so that we don’t have to deal with its complexity. However, when the DSL falls short, we can still include chunks of CloudFormation template syntax to create the resources we need. It turns out that the framework has not implemented the canary deployments feature, so we’ll have to specify the resources ourselves. So, we’ll need to include the following: CloudFormation Serverless Application Model Serverless framework A . CodeDeploy::Application An with and permissions for CodeDeploy. IAM::Role AWSCodeDeployRoleForLambda AWSLambdaFullAccess A for every function, where we’ll specify the deployment preference type and alarms. CodeDeploy::DeploymentGroup A for every function including , where we specify the CodeDeploy Application and DeploymentGroup it belongs to, and the associated hooks. Lambda::Alias CodeDeployLambdaAliasUpdate The Serverless Framework always triggers the Lambda function version upon any event, so we must replace any reference to the function by the newly created alias in the event sources. $Latest If this sounds like a lot of hassle… it’s just because it is. Luckily, the Serverless Framework is really modular and there are tons of plugins to complement its features, so you can use the to create all those resources in a much more convenient way (Note: I’m the author of the plugin, any contribution, comment or feature request is welcome). Serverless Plugin Canary Deployments Happy safe deployments! : Reference _By default, an alias points to a single Lambda function version. When the alias is updated to point to a different…_docs.aws.amazon.com Traffic Shifting Using Aliases - AWS Lambda _This post courtesy of Ryan Green, Software Development Engineer, AWS Serverless The concepts of blue/green and canary…_aws.amazon.com Implementing Canary Deployments of AWS Lambda Functions with Alias Traffic Shifting | Amazon Web… _If you use AWS SAM to create your serverless application, it comes built-in with AWS CodeDeploy for safe Lambda…_docs.aws.amazon.com Gradual Code Deployment - AWS Lambda

Canary deployments in Serverless applications

Safely and easily deploying new version of your web app is the technique that we use to “softly” deploy a new version of an into Production. It consists of letting only a part of the audience get access to the new version of the app, while the rest still access the “old” version one. This is very useful when we want to be sure about stability in case of any changes which may be breaking, and have big side effects. Canary Release application The point is: canary release has never been easy to be put into practice. Depending on the environment we have, it can take so long to be put in place that we often prefer to leave this away. However, with Docker containers and Kubernetes orchestration it is quite friendly to do that. Here I am going to show how to do a small canary deployment of a new version of a dummy website using this tool. Those are the versions of our website: On the left, musicstore v1; on the right, the new version v2 Setup environment In order to use Kubernetes locally, we are going to install and . We are going also need to install . minikube kubectl Docker With everything installed, we launch minikube in order to have Kubernetes up and running: minikube start As a response, the terminal will display: Starting local Kubernetes cluster...Starting VM...SSH-ing files into VM...Setting up certs...Starting cluster components...Connecting to cluster...Setting up kubeconfig...Kubectl is now configured to use the cluster. Second step: call this command in order to run Docker inside Minikube: eval $(minikube docker-env) Setting up DB and backend For this app, the backend side will be an API developped in Go, querying against a Mongo DB. We are not going into details about this part, as it is not the goal here to show . how to develop a REST API in Go To deploy the backend, and build the docker image with the command below: clone the repository docker build -t api-musicstore . Optional step: insert some data in the DB: Enter the docker container with the following command: docker exec -t -i <mongo_container> /bin/bash where is the id of the container running against the mongo image. You can discover it after running <mongo_container> docker ps CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS               NAMES3304ee6ab94e        api-musicstore:v1                                            "/go/bin/musicstore"     3 minutes ago       Up 4 minutes                            k8s_backend.72a7365d_api-musicstore-931708012-g4grg_musicstore_442ad9d4-123c-11e7-ae23-0800275d1cac_9f1e978d**75b2eca8cf1f        mongo **                                                       "docker-entrypoint..."   4 minutes ago       Up 4 minutes                            k8s_db.b9d32dcd_api-musicstore-931708012-g4grg_musicstore_442ad9d4-123c-11e7-ae23-0800275d1cac_67653438 In this case, <mongo_container> is . So we run: 75b2eca8cf1f docker exec -t -i /bin/bash 75b2eca8cf1f Inside the container, we open mongo command-line tool with the following command: mongo Once we are inside mongo, copy-paste the following contents and hit Enter: That’s it! Now we have some seed data in our DB, let’s now deploy our REST API that will communicate with the website! Deploying the backend: REST API Normally, I like to organize my deployments and services inside namespaces. Here, I create a namespace musicstore to deploy my Music Store dummy application. To create this namespace, locally and run the following command: copy this file kubectl create -f namespace.yaml After, and run the following command in order to create the API/DB deployment in Kubernetes: copy these files kubectl create -f deployment.json Note: configuration files for Kubernetes can be created in both structures JSON and YAML. In order to create replica controllers, run this command: kubectl create -f rc.yaml Finally, we can create the service, and our backend is ready to run: kubectl create -f service.json We can see that our API is well deployed consulting the URL and opening it in a web browser: minikube service api-musicstore --url --namespace=musicstore http://<url>:32495 Don’t forget to specify the namespace musicstore, otherwise kubectl is going to search for the service in the default namespace and it is not going to find it! Deploying version 1 of the Music Store dummy website Now we are going to build the image for the version 1 of the website. In order to do this, you are going to need to clone and run the following command: this repository docker build -t musicstore:v1 . With our container built, we can proceed with the deployment. First, we create our service: kubectl -f service.yaml This will create the musicstore-website service. We are going to create the deployments, so that we can deploy our code to be served by this service. Let’s get its URL and keep it for later testing: minikube service musicstore-website --url --namespace=musicstore http://<url>:32000 Let’s take a look at the deployment config file: we are going to ask Kubernetes to create 3 replicas of this website in order to handle traffic and load balancing. All of the replicas will be, in a first moment, pointing to the image corresponding to the image of version v1 of the musicstore-website: Let’s deploy it: kubectl create -f deployment-without-canary.json Now, if you open the website you are going to see its version 1: Website after first deployment Deploying version 2 of the Music Store dummy website using canary deployment First step in order to deploy version 2 is to create a docker image for it, just like we did for version 1. So, and run the following command in a terminal: copy the contents from here docker build -t musicstore:v2 . As seen before, we created 3 replicas of our front-end service. As explained before, the idea of canary deployment is getting a part of the traffic and redirect it to a new version. We are going to achieve this pointing one of the replicas of our service to the new version of the Music Store. Like this, most part of the traffic will still be redirected to version 1. To achieve this, we run the following commands: kubectl apply deployment.jsonkubectl apply deployment-canary.json With this command, we tell Kubernetes to apply this deployment to our environment. If we take a look in the file, we see that the only thing it changes regarding the deployment of version 1 is the label “canary” (the other version is considered “stable”), and the number of replicas (1 to the new version, and 2 serving the old one). And of course, it points to the Docker image : musicstore:v2 And here is how the deployment of version 1 looks like, with 2 replicas pointing to the stable version: Now, we can see that some of the traffic are already seeing the version 2 of our Music Store dummy app: Musicstore version 2 Rolling out the new version Now we feel safe and sound to go for the new version. With the canary delpoyment, this is kind of easy: we just need to set all of the replicas to point to a stable version running the image of the version 2 of the Music Store: kubectl apply deployment-new-version.json This is how our deployment-new-version.json will look like: Very good! Now, everybody will be able to see the new version of the Music Store web site! All the files used in this tutorial can be found . here A great example of Canary Release can be found as well. here

Canary Release with Kubernetes

Engineering teams building high-throughput systems know that scaling infrastructure is never just about speed, it’s about resilience. At extreme volumes, even widely adopted tools can become bottlenecks. That’s exactly what we ran into while developing a high-volume surveillance platform designed to handle sudden spikes of over 300,000 requests per second. Our team was developing a platform for a VIP surveillance service.  The goal was to ensure that busy clients — like Wall Street executives — weren’t bombarded with 101 notifications just because a squirrel ran past their house a few times a day. We had to build a system from scratch that could handle massive request volumes, process them very fast, and know where to direct them. On a typical day, the system handled a large load. During certain events, such as Halloween, when many people triggered motion sensors, traffic increased to 300,000 requests per second. A single query could take 20 to 50 milliseconds or more. To address this, we implemented an additional caching layer in front of Redis. That’s where eBPF came in. How Did We Find It? How Did We Find It? When we were exploring new cache solutions, there weren’t many real-world implementations with eBPF. We came across a research paper on how eBPF could improve caching at scale. While the academic writing style was hard to grasp, the core idea was something new. a research paper a research paper Unfortunately, we didn’t have PhD-level engineers on our team. But we wanted to make this work, and make it work fast. So we dug into the paper, broke down its key insights, and figured out how to apply them to our own system. It wasn’t easy, but once we successfully implemented it, the results spoke for themselves. It ended up changing how we approached caching overall. So, What Exactly Is eBPF and Why Did We Use It? eBPF is a Linux technology that allows small, high-performance programs to run directly inside the kernel. Normally, application code runs in user space, while the Linux kernel handles system operations like networking, disk access, and process management in kernel space. The problem is that switching between these two spaces — sending a request from an application to the kernel and waiting for a response — creates overhead. eBPF allowed us to execute custom code directly in kernel space, avoiding the back-and-forth delays between user space and the kernel. This meant we could store the hottest cache lookups inside eBPF itself and eliminate unnecessary Redis queries altogether. How Exactly a Cache in Front of the Cache Worked? So, instead of hitting Redis for every request, we used small eBPF-based caches to store our most frequently accessed keys. These caches sat in front of Redis, catching the hottest data before it ever needed to reach our main caching layer. Thanks to this, the 99th percentile latency reached 800 ms; the 95th percentile response time was also reduced. This led to lower latency, reduced infrastructure usage, and a more consistent experience. But There Was Another Challenge Even with this optimization in cache, we still had latency spikes at the 99th percentile. Our backend was written in Go, which, despite being performant, has a garbage collector that can introduce unpredictable pauses. We were monitoring everything with Victoria Metrics and had detailed Grafana dashboards tracking latency distribution, so the impact was obvious — whenever the garbage collector kicked in, we saw latency spikes. Since optimizing around garbage collection didn't get us very far, we made the call to rewrite one of the most latency-sensitive services in Rust. Rust has no garbage collector, which made it a better fit for ultra-low-latency workloads. We initially rolled out the Rust-based service as a canary release. Then, we gradually increased traffic while monitoring for performance improvements. Once we confirmed that the latency spikes were gone, we fully replaced the Go with Rust. To Sum Up This led to a change in how we structure our cache design. Redis is no longer the first destination for every cache request; it now functions as a fallback layer. If a response is available through eBPF, the request does not reach Redis. This reduces Redis load and avoids repeated lookups. The use of eBPF for caching goes way beyond our specific use case. As more companies run into Redis scaling challenges, I wouldn’t be surprised to see more teams adopting this approach — or even someone turning it into an open-source project. In this case, the implementation reduced latency and infrastructure usage. The solution did not require additional servers, Redis instances, or engineering capacity. It just reused existing data with a different processing approach.

Stories, News & Mentions about CANARY

Categories

Trending Topics

Stories, News & Mentions about CANARY

Canary’s Stories on HackerNoon

Canary's News & Mentions

Categories

Trending Topics