Cybersecurity researchers at Aqua Nautilus have discovered a new hacking campaign by Adept Libra (aka TeamTNT), targeting exposed Docker daemons to deploy Sliver malware, a cyber worm, and cryptominers.

TeamTNT Exploits 16 Million IPs in Malware Attack on Docker Clusters

Since <strong>Docker</strong> is a US company, we must comply with US export control regulations. In an effort to comply with these, we now block all IP addresses that are located in Cuba, Iran, North Korea, Republic of Crimea, Sudan, and Syria. If you are not in one of these cities, countries, or regions and ...

Docker hub перестал работать в России — Разработка на vc.ru

Docker

docker

KEY TAKEAWAYS WasmEdge functions as a stand-alone option. It replaces traditional OCI runtime systems with its own faster and lighter solution.
These containers provide faster startups and reduced resource consumption. So, the performance is faster alongside minimal overhead expenses.
The secure sandboxing of WasmEdge containers creates an improved security environment that cuts down system vulnerabilities while reducing the exposed attack area.
WasmEdge containers can run on various hardware architectures. Some of the common include x86 together with ARM and RISC-V while maintaining full portability without needing any modifications.
WasmEdge allows users to perform simplified Kubernetes deployments that work perfectly with edge computing alongside serverless applications. WasmEdge functions as a stand-alone option. It replaces traditional OCI runtime systems with its own faster and lighter solution. These containers provide faster startups and reduced resource consumption. So, the performance is faster alongside minimal overhead expenses. The secure sandboxing of WasmEdge containers creates an improved security environment that cuts down system vulnerabilities while reducing the exposed attack area. WasmEdge containers can run on various hardware architectures. Some of the common include x86 together with ARM and RISC-V while maintaining full portability without needing any modifications. WasmEdge allows users to perform simplified Kubernetes deployments that work perfectly with edge computing alongside serverless applications. INTRODUCTION The application running landscape has undergone a complete revolution through docker containers. These create isolated systems that can be easily managed. Kubernetes workloads traditionally depend on OCI runtimes, including containers and CRI-O, for executing container operations. A newer and more efficient solution known as WebAssembly (Wasm) containers with WasmEdge technology emerges as an alternative. In this guide, I will explain the practical implementation of running WebAssembly-based containers on Kubernetes through WasmEdge by avoiding traditional container runtimes. Moreover, it covers complete explanations, starting from configuration deployment to debugging tasks and performance evaluations. You will end up with complete knowledge about deploying WasmEdge to implement containers that deliver high efficiency and speed alongside architectural portability. WHY REPLACE OCI RUNTIMES WITH WEBASSEMBLY Before jumping into the setup and practical steps, let’s first understand why you might want to replace OCI runtimes with WasmEdge. PERFORMANCE GAINS WITH WASMEDGE WasmEdge provides superior performance. It should be the primary reason to choose the platform. The initialization and resource usage of traditional containers last a long time. WasmEdge exists for fast operation. These containers begin operations in milliseconds while requiring fewer system resources. So they become suitable for time-sensitive applications and devices that operate under resource limitations, such as edge systems, and IoT components. When developing an application that needs speed-based scaling according to user demand, you should consider it. WasmEdge containers can establish themselves practically instantaneously compared to how long traditional Docker containers take to begin their operation. They offer a breakthrough benefit for systems that must respond in critical periods like serverless computing and edge computing deployments. SECURITY Standard containers provide running security issues when deployed in operational systems. The main vulnerability of running your container environment exposes applications to potential attacks that could harm neighboring applications sharing the same infrastructure. Each WebAssembly workload receives an isolated sandbox environment through WasmEdge execution. A new security layer is created through WasmEdge implementation effectively. It minimizes the vulnerable points in your system. Each WasmEdge container provides a secure environment that makes it harder for malicious code to spread. Therefore, the host system remains secure. In addition, the security model proves critical for multi-tenant cloud deployments and microservice development. Since it provides necessary isolation between various applications. The implementation of WasmEdge benefits developers who aim to establish application resistance against attacks. CROSS-PLATFORM COMPATIBILITY Among the useful features of WasmEdge is its capability to work across multiple platforms. The normal deployment of containers between different platforms like x86 and ARM requires modifications either in container images/platform-specific configuration implementation. WasmEdge shows different behavior in this respect. The container engine operates without issues on multiple processor systems that include x86 ARM and RISC-V. WasmEdge operates containers on cloud servers, edge devices, and Raspberry Pi without any changes to the initial configuration. The high level of portability reduces deployment complexity. Because your application needs to function within diverse execution environments. IDEAL FOR EDGE AND SERVERLESS COMPUTING WasmEdge demonstrates exceptional performance within edge and serverless computing domains. It's compatible with handling small and quick booting containers. WasmEdge serves efficient workloads because engineers developed it specifically for lightweight, small systems that need fewer resources than traditional containers. WasmEdge works flawlessly in situations where equipment has restricted processing power, such as smart sensors, edge devices, and IoT setups. WasmEdge serves developers who construct microservices or serverless applications through its optimal features. The quick boot times of containers match their need for scarce resources while operating. Such performance attributes make WasmEdge ideal for applications that require fast responsiveness under various traffic conditions, such as event-driven workloads or stateless applications. SETTING UP KUBERNETES FOR WASMEDGE CONTAINERS This section will explain the necessary steps for WasmEdge installation on Kubernetes platforms, although the Kubernetes experience is beneficial. PREREQUISITES The following items should be checked before starting: A Kubernetes 1.26+ cluster. WasmEdge deployment requires a Kubernetes service provider.


Your cluster needs Kubectl installation with optimal configuration to access it from any location.


All Kubernetes nodes must have WasmEdge runtime running.


Container with Wasm support, and CRI-O with WasmEdge integration. A Kubernetes 1.26+ cluster. WasmEdge deployment requires a Kubernetes service provider. A Kubernetes 1.26+ cluster. WasmEdge deployment requires a Kubernetes service provider. Your cluster needs Kubectl installation with optimal configuration to access it from any location. Your cluster needs Kubectl installation with optimal configuration to access it from any location. All Kubernetes nodes must have WasmEdge runtime running. All Kubernetes nodes must have WasmEdge runtime running. Container with Wasm support, and CRI-O with WasmEdge integration. Container with Wasm support, and CRI-O with WasmEdge integration. Install the WasmEdge on Kubernetes nodes by using the command below: curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash curl -sSf https://raw.githubusercontent.com/WasmEdge/WasmEdge/master/utils/install.sh | bash After installation, make sure that WasmEdge is installed rightly: wasmedge --version wasmedge --version This should return the version of WasmEdge you’ve installed. CONFIGURING KUBERNETES FOR WASMEDGE Your Kubernetes environment requires registration of WasmEdge as its runtime for containers. The process needs you to establish WasmEdge Runtime Class configuration for Kubernetes usage. WasmEdge can be used by Kubernetes instead of the default container, and CRI-O runtimes through this configuration. Create a RuntimeClass YAML file for WasmEdge: apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
runtimeHandler: wasmedge apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmedge
runtimeHandler: wasmedge Apply this configuration using: kubectl apply -f runtimeclass-wasmedge.yaml kubectl apply -f runtimeclass-wasmedge.yaml This tells Kubernetes to use WasmEdge as a valid runtime for your containers. DEPLOYING A WEBASSEMBLY (WASM) CONTAINER FOR KUBERNETES Now, let’s discuss, how can you deploy a WebAssembly (Wasm) Container on Kubernetes: CREATING A WASMEDGE-BASED DOCKER IMAGE First, convert your application into a WebAssembly (Wasm) module. For example, try a Rust-based HTTP server and convert it into a Wasm module. Next, install the wasm-pack tool. It helps you compile Rust code into WebAssembly: cargo install wasm-pack cargo install wasm-pack Build the Rust app into a Wasm module: wasm-pack build --release --target=wasm32-wasi wasm-pack build --release --target=wasm32-wasi The Wasm module requires setup before it can be deployed to Docker Hub and ORAS container registries. Kubernetes can obtain Wasm images through this mechanism during its deployment process. Example: docker push myrepo/wasm-app:latest docker push myrepo/wasm-app:latest DEPLOYING THE WASMEDGE CONTAINER Now, you’re ready to deploy the Wasm-based container on Kubernetes. Create a deployment YAML file for Kubernetes to use: apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wasm-app
  template:
    metadata:
      labels:
        app: wasm-app
    spec:
      runtimeClassName: wasmedge
      containers:
        - name: wasm-container
          image: myrepo/wasm-app:latest
          command: ["/wasm-app.wasm"] apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: wasm-app
  template:
    metadata:
      labels:
        app: wasm-app
    spec:
      runtimeClassName: wasmedge
      containers:
        - name: wasm-container
          image: myrepo/wasm-app:latest
          command: ["/wasm-app.wasm"] Now, apply this deployment file to Kubernetes: kubectl apply -f wasm-deployment.yaml kubectl apply -f wasm-deployment.yaml The WasmEdge container receives deployment through Kubernetes with two exact duplicates enabled. The deployment status can be checked by running this command: kubectl get pods -l app=wasm-app kubectl get pods -l app=wasm-app The above command provides a quick overview of the deployment status. RUNNING AND DEBUGGING WASM CONTAINERS Once deployed, it's imperative to verify that everything is working as intended. Let’s talk about some practical commands to monitor and troubleshoot your WasmEdge containers. VERIFYING THE DEPLOYMENT To check if your WasmEdge container is running, use: kubectl get pods -l app=wasm-app kubectl get pods -l app=wasm-app If everything is running in the right manner, this should show the pods associated with your deployment. To check the logs for a specific pod: kubectl logs <pod-name> kubectl logs <pod-name> TROUBLESHOOTING If you face any issues with your container, use the Kubectl describe pod to see detailed information: kubectl describe pod <pod-name> kubectl describe pod <pod-name> The command will show you the events related to the pod. It can help you debug issues such as image pull errors or misconfigurations. Additionally, WasmEdge provides a CLI tool to validate your Wasm module. To certify that your Wasm module is accurately built and ready to run, use wasmedge --validate my-wasm-app.wasm wasmedge --validate my-wasm-app.wasm This will check for any potential issues with the Wasm module before you deploy it. BENCHMARKING WASMEDGE VS. OCI CONTAINERS It’s important to compare the performance of WasmEdge containers with traditional OCI containers. Let’s look at a few performance benchmarks to see how WasmEdge performs. STARTUP TIME COMPARISON One of the most significant benefits of WasmEdge is the startup time. To evaluate the startup latency, run both WasmEdge and OCI containers using the time command: time docker run myrepo/wasm-app
time wasmedge run myrepo/wasm-app.wasm time docker run myrepo/wasm-app
time wasmedge run myrepo/wasm-app.wasm MEMORY & CPU USAGE To compare memory and CPU usage, use the kubectl top pods command: kubectl top pods kubectl top pods This will show you how much CPU and memory your containers are consuming. WasmEdge containers should consume far fewer resources than their OCI counterparts. For even more detailed profiling, use wasm-stats: wasm-stats my-wasm-app.wasm wasm-stats my-wasm-app.wasm It will help you understand how efficiently WasmEdge handles your workload. CONCLUSION WasmEdge offers users lightweight and secure container runtime functions. It replaces existing traditional OCI runtimes effectively. OCI runtime containers will become more effective when you migrate to WasmEdge because your applications will start faster and use fewer resources, while the sandbox security measures improve system safety. WasmEdge operates across multiple platforms. So, it means your containers can function on x86 devices as well as ARM and RISC-V devices without changing any configurations. The guide presents simple deployment and management instructions for WasmEdge containers on Kubernetes systems. WasmEdge proves to be an ideal solution for developers who create microservices and serverless applications as well as edge deployments.

Running Docker Containers on Kubernetes Without a Container Runtime Using WasmEdge

💡 Introduction 💡 Introduction Welcome Dev to the world of testing and Containerisation. Today, we will be diving into an exciting project, where we will be testing a Node Js application using cypress and this whole project will be inside a docker container. So without further ado, Let’s get started! 💡 Pre-requisites 💡 Pre-requisites Before we delve into the project, we need to ensure that we have following requirements met in our system: Docker and Docker-Compose: Since we are working inside a container, we will be needing docker and docker-compose for that.
Node installed: We are testing a node application, so we would be needing node to run in it.
Basic Understanding of Docker: We will be creating a dockerfile and Docker-compose file, a prior understanding of both would be appreciated. Docker and Docker-Compose: Since we are working inside a container, we will be needing docker and docker-compose for that. Docker and Docker-Compose: Node installed: We are testing a node application, so we would be needing node to run in it. Node installed: Basic Understanding of Docker: We will be creating a dockerfile and Docker-compose file, a prior understanding of both would be appreciated. Basic Understanding of Docker: 💡 Setting up the project 💡 Setting up the project We will start the project by creating a project dir named book-app-testing, inside the directory, we will be creating a docker-compose file with the following content: services:
 book-reader-app:
   image: rajudandigam/book-reader-node:v1
   container_name: book-reader-app
   ports:
     - "3000:3000" 
   networks:
     - test-network


 cypress:
   build:
     context: ./cypress-test
     dockerfile: Dockerfile
   container_name: cypress
   depends_on:
    - book-reader-app


   environment:
     - CYPRESS_baseUrl=http://book-reader-app:3000 
   volumes:
     - ./cypress-test:/e2e 
   networks:
     - test-network
   command: ["npx", "cypress", "run", "--spec", "cypress/e2e/book-app-heading.cy.js"]


networks:
 test-network:
   driver: bridge services:
 book-reader-app:
   image: rajudandigam/book-reader-node:v1
   container_name: book-reader-app
   ports:
     - "3000:3000" 
   networks:
     - test-network


 cypress:
   build:
     context: ./cypress-test
     dockerfile: Dockerfile
   container_name: cypress
   depends_on:
    - book-reader-app


   environment:
     - CYPRESS_baseUrl=http://book-reader-app:3000 
   volumes:
     - ./cypress-test:/e2e 
   networks:
     - test-network
   command: ["npx", "cypress", "run", "--spec", "cypress/e2e/book-app-heading.cy.js"]


networks:
 test-network:
   driver: bridge Now inside the project dir, we will create a new dir, named cyprus-test. We will navigate inside it and run the following commands: npm init -y
npm install cypress --save-dev
npx cypress open npm init -y
npm install cypress --save-dev
npx cypress open This will open a new Window. Here Select E2e Testing and then click on continue and start testing E2E testing in Electron. Now, Click on Create a new Spec File and in the name put cypress/e2e/book-app-heading.cy.js and click Okay and run the Spec file. cypress/e2e/book-app-heading.cy.js This will run a demo test. Now go to thebook-app-testing/cypress-test/cypress/e2e and here you will have your book-app-heading.cy.js file, remove the existing content and put the full information in it: book-app-testing/cypress-test/cypress/e2e book-app-heading.cy.js describe('Book Reader Node App', () => {
 it('should have a heading that includes "Book App"', () => {
   // Use the baseUrl from environment variable (preferred approach)
   cy.visit('/'); // This appends to CYPRESS_baseUrl (http://book-reader-app:3000)


   // Alternatively, hardcode the service name (less flexible):
   // cy.visit('http://book-reader-app:3000');


   // Check if any heading (h1, h2, etc.) contains "Book App"
   cy.get('h1, h2, h3, h4, h5, h6').should('contain.text', 'Book App');
 });
}); describe('Book Reader Node App', () => {
 it('should have a heading that includes "Book App"', () => {
   // Use the baseUrl from environment variable (preferred approach)
   cy.visit('/'); // This appends to CYPRESS_baseUrl (http://book-reader-app:3000)


   // Alternatively, hardcode the service name (less flexible):
   // cy.visit('http://book-reader-app:3000');


   // Check if any heading (h1, h2, etc.) contains "Book App"
   cy.get('h1, h2, h3, h4, h5, h6').should('contain.text', 'Book App');
 });
}); This is a simple test which will test whether the application heading has Book App in it. 💡 Creating the Dockerfile Now, we will be creating a dockerfile for Cypress with our Test in it, for that, go inside the  book-app-testing/cypress-test and create a Dockerfile and put the following content it: book-app-testing/cypress-test #Use Cypress base image #Use Cypress base image FROM cypress/included:13.0.0


WORKDIR /e2e


# Copy project files
COPY . .


# Install dependencies
RUN npm install


# Run Cypress tests
CMD ["npx", "cypress", "run"] FROM cypress/included:13.0.0


WORKDIR /e2e


# Copy project files
COPY . .


# Install dependencies
RUN npm install


# Run Cypress tests
CMD ["npx", "cypress", "run"] Now, we will navigate to our project dir and run the following command to run the docker-container: docker-compose up –build docker-compose up –build This will give you the following output and in that, we can see that the test we provided is passed. To enhance the testing process, you can add following test in the same file or create separate file for each test inside the cypress-test/cypress/e2e: cypress-test/cypress/e2e: // Render Header component Properly
describe('Book Reader Node App', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should render the header component', () => {
   cy.get('header').should('be.visible');
 });


 it('should render the sidebar component', () => {
   cy.get('aside').should('be.visible');
 });
}); // Render Header component Properly
describe('Book Reader Node App', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should render the header component', () => {
   cy.get('header').should('be.visible');
 });


 it('should render the sidebar component', () => {
   cy.get('aside').should('be.visible');
 });
}); // Check Animation Tap and Hover
describe('Book Reader Node App - Animations', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should scale up on hover', () => {
   cy.get('ul li').first().trigger('mouseover')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });


 it('should scale down on tap', () => {
   cy.get('ul li').first().trigger('mousedown')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });
}); // Check Animation Tap and Hover
describe('Book Reader Node App - Animations', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should scale up on hover', () => {
   cy.get('ul li').first().trigger('mouseover')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });


 it('should scale down on tap', () => {
   cy.get('ul li').first().trigger('mousedown')
     .should('have.css', 'transform')
     .and('contain', 'matrix');
 });
}); // Check Test Responses in various Devices
describe('Book Reader Node App - Responsiveness', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 const sizes = [
   { width: 1280, height: 720 },  // Desktop
   { width: 768, height: 1024 },  // Tablet
   { width: 375, height: 812 }    // Mobile
 ];


 sizes.forEach((size) => {
   it(`should display correctly on ${size.width}x${size.height}`, () => {
     cy.viewport(size.width, size.height);
     cy.get('main').should('be.visible');
   });
 });
}); // Check Test Responses in various Devices
describe('Book Reader Node App - Responsiveness', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 const sizes = [
   { width: 1280, height: 720 },  // Desktop
   { width: 768, height: 1024 },  // Tablet
   { width: 375, height: 812 }    // Mobile
 ];


 sizes.forEach((size) => {
   it(`should display correctly on ${size.width}x${size.height}`, () => {
     cy.viewport(size.width, size.height);
     cy.get('main').should('be.visible');
   });
 });
}); // Test Naviation
describe('Book Reader Node App - Navigation', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should navigate to book details page on click', () => {
   cy.get('ul li a').first().click();
   cy.url().should('include', '/book/');
 });


 it('should display book details on the details page', () => {
   cy.get('ul li a').first().click();
   cy.get('h1').should('be.visible');
   cy.get('img').should('be.visible');
 });
}); // Test Naviation
describe('Book Reader Node App - Navigation', () => {
 beforeEach(() => {
   cy.visit('/');
 });


 it('should navigate to book details page on click', () => {
   cy.get('ul li a').first().click();
   cy.url().should('include', '/book/');
 });


 it('should display book details on the details page', () => {
   cy.get('ul li a').first().click();
   cy.get('h1').should('be.visible');
   cy.get('img').should('be.visible');
 });
}); You can also visit __http://localhost:300__0 where our application is running. http://localhost:300 💡 Conclusion 💡 Conclusion This marks the end of our blog. In today’s blog, we learnt how to use Cypress to test our Web Application built using Node. We started with a basic test to verify the heading contains "Book App" and gradually saw how we can expand our testing coverage with more complex and meaningful tests. By incorporating Cypress, we can ensure our app’s functionality, UI interactions, and navigation work as expected, making our web applications more robust and reliable.

Running Cypress Tests for Node.js Apps Inside a Docker Container

Your Docker Containers Are Probably Full of Security Holes—Here’s How to Fix Them Now

Firms increasingly make use of artificial intelligence (AI) infrastructures to host and manage autonomous workloads. Consequently, there has been significant demand for scalable as well as resilient infrastructures that will be able to meet heterogeneous application or cloud requirements. Organizations use Kubernetes and Docker to meet such needs because firms realize that both are highly effective use cases that deliver scalable AI infrastructures. Deploying AI infrastructure typically provides adequate computation power to execute and process large datasets. These demands can translate into the need for scalable methods that enable AI models to run on large workloads without hurting performance. Why Companies Need to Scale Up Their AI Infrastructure AI systems, nonetheless, are also resource-intensive, normally demanding both high computing capacity and the ability to process high levels of data. As more advanced AI applications and a larger scale become required, scalability becomes more critical. Scalability ensures that AI systems can handle increasing workloads without any loss of performance. Expanding Data Volumes The growing amount of data is a concern for AI systems in many facets. Most AI models, especially those based on deep learning, heavily depend on large amounts of data during training and inference. However, without adequate scalable infrastructure, processing and interpreting such enormous quantities of data is a roadblock. Optimized Performance Scalable AI hardware supports reliable and stable performance despite drastically overwhelming computational loads. With Kubernetes, horizontal scaling of AI jobs is a breeze, and the dynamic resizing of replica numbers can be done as a function of necessity. In contrast, Docker containers support lean, isolated environments for running AI models where resource conflict is not a performance bottleneck. Effective Resource Management Efficient use of resources is the key to cost-effective and sustainable AI deployment. Kubernetes' resource requests and limits allow for fine-grained CPU and memory resource management by avoiding underprovisioning and overprovisioning. Docker's resource management fills the gap by isolating container resources. Scaling AI Infrastructure With Kubernetes and Docker Containerization is one of the milestones in the evolution of scalable artificial intelligence infrastructure. Containerization of the AI application and its dependencies in a Docker container ensures consistency throughout the development, testing, and deployment environments. First, you must define a Dockerfile in order to install the environment. The Dockerfile is a series of instructions about how to build a Docker image. It declares a base image, the dependencies required, and the initial setup commands that apply to your app. The following is a basic Dockerfile for a Python machine-learning model: # Use an official Python runtime as a parent image
FROM python:3.9-slim
 
# Set the working directory in the container
WORKDIR /usr/src/app
 
# Copy the current directory contents into the container
COPY . .
 
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
# Expose the port the app runs on
EXPOSE 5000
 
# Define environment variable
ENV NAME World
 
# Run the app
CMD ["python", "./app.py"] If the Dockerfile is ready, then you can build the Docker image and run the container. Run the following commands: # Build the Docker image
docker build -t ml-model:latest .
 
# Run the container
docker run -p 5000:5000 ml-model:latest Deploying the Dockerized AI Model to Kubernetes Kubernetes provides a wide range of orchestration features that enable efficient application management in the containerized infrastructure. Deployment of the Docker image on Kubernetes ensures that a specified number of application replicas is always running. The following is an example of deployment.yaml file that you can use to deploy your Dockerized machine learning model: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3  
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000 The above code snippet shows how to deploy the AI model, but you also need to make the model externally accessible. You will need to expose it by defining a Kubernetes Service. The service.yaml below illustrates an example: apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
spec:
  selector:
	app: ml-model
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 5000
  type: LoadBalancer Use the kubectl command-line tool to apply the deployment and service configurations: # Deploy the application
kubectl apply -f deployment.yaml
 
# Expose the service
kubectl apply -f service.yaml Scaling With Kubernetes Kubernetes provides excellent scaling capabilities to AI environments, maximizing resource utilization and performance. Horizontal scaling is done by adding additional containers, and vertical scaling involves adding additional resources like CPU or memory to a container. Horizontal Scaling Horizontal scaling is used to scale up the number of replicas (Pods) of an AI system to handle a higher workload. The process requires enabling dynamic scaling depending on the number of replicas. The command used to enable such a process is `kubectl scale`. The particular command is used to set up the deployment to function up to a maximum of five replicas: `kubectl scale --replicas=5 deployment/ml-model-deployment` The command scales up the ml-model-deployment to use five replicas of the machine-learning model container. The system dynamically provisions more Pods to meet the required number afterward. Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Kubernetes facilitates auto-scaling using the Horizontal Pod Autoscaler (HPA). The HPA dynamically adjusts the number of replicas based on resource use, i.e., CPU or memory, in relation to set limits. The YAML configuration shown below is a relevant example of an HPA that dynamically scales for ml-model-deployment in response to CPU use: apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50 In this setup, scaleTargetRef is used to define the Deployment to be scaled, i.e., ml-model-deployment. The minimum replica count is set using MinReplicas, while the maximum replica count is controlled using maxReplicas. In addition, the CPU utilization percentage is set using targetCPUUtilizationPercentage, i.e., to 50%. CPU utilization of more than 50% across all Pods results in scaling up the replica count to a maximum of 10 automatically. As soon as CPU utilization drops below the set percentage, Kubernetes automatically reduces the replica count in order to release resources. Vertical Scaling Horizontal scaling is mainly to cope with more traffic, whereas vertical scaling provides more resources (such as CPU or memory) to existing containers. The process is to scale up or down resource requests and limits in the Kubernetes Deployment. In order to scale up the CPU and memory limits of the ml-model-deployment, one would need to open the deployment.yaml file: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000
    	resources:
      	requests:
        	cpu: "1"
        	memory: "2Gi"
      	limits:
        	cpu: "2"
        	memory: "4Gi" In this updated configuration: requests specify the minimum resources required for the container.limits define the maximum resources the container can use. Firms increasingly make use of artificial intelligence (AI) infrastructures to host and manage autonomous workloads. Consequently, there has been significant demand for scalable as well as resilient infrastructures that will be able to meet heterogeneous application or cloud requirements. Organizations use Kubernetes and Docker to meet such needs because firms realize that both are highly effective use cases that deliver scalable AI infrastructures. there has been significant demand for scalable Kubernetes Docker Deploying AI infrastructure typically provides adequate computation power to execute and process large datasets. These demands can translate into the need for scalable methods that enable AI models to run on large workloads without hurting performance. Why Companies Need to Scale Up Their AI Infrastructure Why Companies Need to Scale Up Their AI Infrastructure AI systems , nonetheless, are also resource-intensive, normally demanding both high computing capacity and the ability to process high levels of data. As more advanced AI applications and a larger scale become required, scalability becomes more critical. Scalability ensures that AI systems can handle increasing workloads without any loss of performance. AI systems Expanding Data Volumes Expanding Data Volumes The growing amount of data is a concern for AI systems in many facets. Most AI models, especially those based on deep learning, heavily depend on large amounts of data during training and inference. However, without adequate scalable infrastructure, processing and interpreting such enormous quantities of data is a roadblock . enormous quantities of data is a roadblock Optimized Performance Optimized Performance Scalable AI hardware supports reliable and stable performance despite drastically overwhelming computational loads. With Kubernetes, horizontal scaling of AI jobs is a breeze, and the dynamic resizing of replica numbers can be done as a function of necessity. In contrast, Docker containers support lean, isolated environments for running AI models where resource conflict is not a performance bottleneck. Effective Resource Management Effective Resource Management Efficient use of resources is the key to cost-effective and sustainable AI deployment. Kubernetes' resource requests and limits allow for fine-grained CPU and memory resource management by avoiding underprovisioning and overprovisioning. Docker's resource management fills the gap by isolating container resources. Scaling AI Infrastructure With Kubernetes and Docker Scaling AI Infrastructure With Kubernetes and Docker Containerization is one of the milestones in the evolution of scalable artificial intelligence infrastructure. Containerization of the AI application and its dependencies in a Docker container ensures consistency throughout the development, testing, and deployment environments. First, you must define a Dockerfile in order to install the environment. The Dockerfile is a series of instructions about how to build a Docker image. It declares a base image, the dependencies required, and the initial setup commands that apply to your app. The following is a basic Dockerfile for a Python machine-learning model: # Use an official Python runtime as a parent image
FROM python:3.9-slim
 
# Set the working directory in the container
WORKDIR /usr/src/app
 
# Copy the current directory contents into the container
COPY . .
 
# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
 
# Expose the port the app runs on
EXPOSE 5000
 
# Define environment variable
ENV NAME World
 
# Run the app
CMD ["python", "./app.py"] If the Dockerfile is ready, then you can build the Docker image and run the container. Run the following commands: # Build the Docker image
docker build -t ml-model:latest .
 
# Run the container
docker run -p 5000:5000 ml-model:latest Deploying the Dockerized AI Model to Kubernetes Deploying the Dockerized AI Model to Kubernetes Kubernetes provides a wide range of orchestration features that enable efficient application management in the containerized infrastructure. Deployment of the Docker image on Kubernetes ensures that a specified number of application replicas is always running. The following is an example of deployment.yaml file that you can use to deploy your Dockerized machine learning model : Kubernetes deploy your Dockerized machine learning model apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3  
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000 The above code snippet shows how to deploy the AI model, but you also need to make the model externally accessible. You will need to expose it by defining a Kubernetes Service. The service.yaml below illustrates an example: apiVersion: v1
kind: Service
metadata:
  name: ml-model-service
spec:
  selector:
	app: ml-model
  ports:
	- protocol: TCP
  	port: 80
  	targetPort: 5000
  type: LoadBalancer Use the kubectl command-line tool to apply the deployment and service configurations: # Deploy the application
kubectl apply -f deployment.yaml
 
# Expose the service
kubectl apply -f service.yaml Scaling With Kubernetes Scaling With Kubernetes Kubernetes provides excellent scaling capabilities to AI environments, maximizing resource utilization and performance. Horizontal scaling is done by adding additional containers, and vertical scaling involves adding additional resources like CPU or memory to a container. Horizontal Scaling Horizontal Scaling Horizontal scaling is used to scale up the number of replicas (Pods) of an AI system to handle a higher workload. The process requires enabling dynamic scaling depending on the number of replicas. The command used to enable such a process is `kubectl scale`. The particular command is used to set up the deployment to function up to a maximum of five replicas: `kubectl scale --replicas=5 deployment/ml-model-deployment` The command scales up the ml-model-deployment to use five replicas of the machine-learning model container. The system dynamically provisions more Pods to meet the required number afterward. Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Automatic Scaling using the Horizontal Pod Autoscaler (HPA) Kubernetes facilitates auto-scaling using the Horizontal Pod Autoscaler (HPA). The HPA dynamically adjusts the number of replicas based on resource use, i.e., CPU or memory, in relation to set limits. The YAML configuration shown below is a relevant example of an HPA that dynamically scales for ml-model-deployment in response to CPU use: apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: ml-model-hpa
spec:
  scaleTargetRef:
	apiVersion: apps/v1
	kind: Deployment
	name: ml-model-deployment
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 50 In this setup, scaleTargetRef is used to define the Deployment to be scaled, i.e., ml-model-deployment. The minimum replica count is set using MinReplicas, while the maximum replica count is controlled using maxReplicas. In addition, the CPU utilization percentage is set using targetCPUUtilizationPercentage, i.e., to 50%. CPU utilization of more than 50% across all Pods results in scaling up the replica count to a maximum of 10 automatically. As soon as CPU utilization drops below the set percentage, Kubernetes automatically reduces the replica count in order to release resources. Vertical Scaling Vertical Scaling Horizontal scaling is mainly to cope with more traffic, whereas vertical scaling provides more resources (such as CPU or memory) to existing containers. The process is to scale up or down resource requests and limits in the Kubernetes Deployment. In order to scale up the CPU and memory limits of the ml-model-deployment, one would need to open the deployment.yaml file: apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-model-deployment
spec:
  replicas: 3
  selector:
	matchLabels:
  	app: ml-model
  template:
	metadata:
  	labels:
    	app: ml-model
	spec:
  	containers:
  	- name: ml-model-container
    	image: ml-model:latest
    	ports:
    	- containerPort: 5000
    	resources:
      	requests:
        	cpu: "1"
        	memory: "2Gi"
      	limits:
        	cpu: "2"
        	memory: "4Gi" In this updated configuration: requests specify the minimum resources required for the container. limits define the maximum resources the container can use. requests specify the minimum resources required for the container. limits define the maximum resources the container can use.

How to Scale AI Infrastructure With Kubernetes and Docker

Com escalar la infraestructura d'IA amb Kubernetes i Docker

Πώς να κλιμακώσετε την υποδομή AI με το Kubernetes και το Docker

Cómo escalar la infraestructura de IA con Kubernetes y Docker

چگونه زیرساخت هوش مصنوعی را با Kubernetes و Docker مقیاس بندی کنیم

Kako skalirati AI infrastrukturu s Kubernetesom i Dockerom

Kubernetes と Docker で AI インフラストラクチャを拡張する方法

Kaip padidinti AI infrastruktūrą naudojant „Kubernetes“ ir „Docker“.

Kubernetes සහ Docker සමඟ AI යටිතල පහසුකම් පරිමාණය කරන්නේ කෙසේද?

Hur man skalar AI-infrastruktur med Kubernetes och Docker

Paano I-scale ang AI Infrastructure Gamit ang Kubernetes at Docker

کبرنیٹس اور ڈوکر کے ساتھ اے آئی انفراسٹرکچر کی پیمائش کیسے کریں۔

Cách mở rộng cơ sở hạ tầng AI với Kubernetes và Docker

Optimizing Docker Images is More Than Just a One and Done Thing

Optimitzar les imatges de Docker és més que una cosa feta

Optimizar las imágenes de Docker es más que una tarea que se realiza una sola vez

بهینه سازی تصاویر Docker چیزی فراتر از یک کار انجام شده است

Docker イメージの最適化は一度で終わるものではありません

Ny fanatsarana ny sary Docker dia mihoatra noho ny zavatra iray sy vita

Докерын зургуудыг оновчтой болгох нь зөвхөн нэг бөгөөд хийсэн зүйл биш юм

डकर छविहरू अप्टिमाइज गर्नु केवल एक र सम्पन्न चीज भन्दा बढी हो

Otimizar imagens do Docker é mais do que algo feito uma vez e pronto

Docker Imahinakuna allinchayqa Aswan Hukllamanta chaymanta Rurasqamanta

Hagaajinta Sawirrada Docker waa in ka badan hal iyo wax la sameeyay

Docker suratlaryny optimizirlemek diňe bir we ýerine ýetirilen zat däl

Ang Pag-optimize sa Mga Imahe ng Docker ay Higit pa sa Isa at Tapos na Bagay

This is the third part of the series Container Networking. I will explain a little bit about the docker container networking in this blog post. I followed the steps mentioned in this to install docker on Ubuntu. Post docker install, you can see the docker0 device in the list ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff Let’s create a busybox container. docker run --name bb -dt busybox docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb In the device list, you see the veth7b01920@if16 interface is created with the master as docker0. ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 Let’s check the network namespace. ip netns list And you won’t see any network namespaces…How come??? This is because: ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible If you want ip netns list to show the namespace name docker has created, then follow the below steps export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name Now, you can see the namespace name. ip netns list
bb (id: 0) sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 The following diagram will help you visualize it better. With this, I’m signing off. I hope this blog series helps to de-clutter container networking. This is the third part of the series Container Networking . I will explain a little bit about the docker container networking in this blog post. Container Networking I followed the steps mentioned in this to install docker on Ubuntu. this Post docker install, you can see the docker0 device in the list docker0 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff Let’s create a busybox container. docker run --name bb -dt busybox docker run --name bb -dt busybox docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED              STATUS              PORTS    NAMES
02429964e449   busybox   "sh"      About a minute ago   Up About a minute             bb In the device list, you see the veth7b01920@if16 interface is created with the master as docker0. veth7b01920@if16 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 02:fd:4d:34:55:76 brd ff:ff:ff:ff:ff:ff
11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:cf:b4:c1:a8 brd ff:ff:ff:ff:ff:ff
17: veth7b01920@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether e2:15:50:88:11:03 brd ff:ff:ff:ff:ff:ff link-netnsid 2 Let’s check the network namespace. ip netns list ip netns list And you won’t see any network namespaces…How come??? This is because: ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible ip netns list command looks up network namespaces file in the 
/var/run/netns directory.

However, the Docker daemon doesn’t create a reference of
the network namespace file in the /var/run/netns directory
after the creation. Therefore, ip netns ls cannot resolve the
network namespace file

Ref: https://www.baeldung.com/linux/docker-network-namespace-invisible If you want ip netns list to show the namespace name docker has created, then follow the below steps ip netns list export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name export container_name=bb
container_pid=$(sudo docker inspect -f '' $container_name)
echo $container_pid

sudo touch /var/run/netns/$container_name
sudo mount -o bind /proc/$container_pid/ns/net /var/run/netns/$container_name Now, you can see the namespace name. ip netns list
bb (id: 0) ip netns list
bb (id: 0) sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 sudo ip netns exec bb ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 The following diagram will help you visualize it better. With this, I’m signing off. I hope this blog series helps to de-clutter container networking.

A Guide to Container Networking: Part III - Docker Container Networking

When building Python applications with Poetry in a Docker container, we sometimes encounter issues accessing private packages stored in the Google Artifact Registry (GAR). Locally, this challenge arises because docker build cannot directly handle the Google Cloud credentials in the same way as our CI/CD pipeline, where we leverage service accounts and the Kaniko action for secure builds. To resolve this for local development, we'll configure Google Cloud credentials within Docker using Docker Compose, enabling secure access to the GAR repository during the build process. Non-Containerized Pre-requirements Python ^3.12
Poetry ^1.7.1
Google Cloud SDK ^489.0.0
Keyring ^24.0.0
keyrings.google-artifactregistry-auth ^1.0.0
Read access to your GAR Python repository
Logged into GCP (gcloud auth application-default login) Setup Keyring Setup
Install keyring support for Google Artifact Registry:
pip install keyring
pip install keyrings.google-artifactregistry-auth Connect Poetry to your GAR repo
Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running:
poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Be sure to append /simple to the repository URL for compatibility. Now you can install packages from your private repo: poetry add --source <PACKAGE_NAME> some-package@1.0.0 Notes In some cases, accessing the repository may require setting an explicit OAuth token for authentication in Poetry. Use the following command to configure this globally in Poetry: poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) Containerized Pre-requirements Docker ^20.10
Docker Compose ^2.0
Google Cloud SDK ^489.0.0
Read access to your GAR Python repository
Logged into GCP (gcloud auth application-default login) Setup Secrets Configuration
First, define a secret in docker-compose.yaml using the local path to your credentials file:
secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json



We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data.




Dockerfile Adjustments
In the Dockerfile, we handle credentials with the following setup:
ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS}



--mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container.




Service Configurations
Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration.
some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Usage Just run: docker-compose up --build Conclusion This approach allows local Docker builds to access private GAR resources securely, ensuring that credentials are handled appropriately and remain protected. When building Python applications with Poetry in a Docker container, we sometimes encounter issues accessing private packages stored in the Google Artifact Registry (GAR). Locally, this challenge arises because docker build cannot directly handle the Google Cloud credentials in the same way as our CI/CD pipeline, where we leverage service accounts and the Kaniko action for secure builds. To resolve this for local development, we'll configure Google Cloud credentials within Docker using Docker Compose, enabling secure access to the GAR repository during the build process. docker build Docker Non-Containerized Pre-requirements Python ^3.12 Poetry ^1.7.1 Google Cloud SDK ^489.0.0 Keyring ^24.0.0 keyrings.google-artifactregistry-auth ^1.0.0 Read access to your GAR Python repository Logged into GCP (gcloud auth application-default login) Python ^3.12 Poetry ^1.7.1 Google Cloud SDK ^489.0.0 Keyring ^24.0.0 keyrings.google-artifactregistry-auth ^1.0.0 Read access to your GAR Python repository Logged into GCP ( gcloud auth application-default login ) gcloud auth application-default login Setup Keyring Setup
Install keyring support for Google Artifact Registry:
pip install keyring
pip install keyrings.google-artifactregistry-auth Keyring Setup Install keyring support for Google Artifact Registry: pip install keyring
pip install keyrings.google-artifactregistry-auth Keyring Setup Keyring Setup Install keyring support for Google Artifact Registry: keyring pip install keyring
pip install keyrings.google-artifactregistry-auth pip install keyring
pip install keyrings.google-artifactregistry-auth Connect Poetry to your GAR repo
Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running:
poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Connect Poetry to your GAR repo Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running: poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Connect Poetry to your GAR repo Connect Poetry to your GAR repo Access to the private repository in the Google Artifact Registry can be managed through Poetry. First, configure a custom source in Poetry for the GAR repository by running: poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple poetry source add --priority=explicit <PACKAGE_NAME> https://<REGION>-python.pkg.dev/<PROJECT>/<REGISTRY>/simple Be sure to append /simple to the repository URL for compatibility. Be sure to append /simple to the repository URL for compatibility. /simple Now you can install packages from your private repo: poetry add --source <PACKAGE_NAME> some-package@1.0.0 poetry add --source <PACKAGE_NAME> some-package@1.0.0 Notes In some cases, accessing the repository may require setting an explicit OAuth token for authentication in Poetry. Use the following command to configure this globally in Poetry: poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) poetry config http-basic.<PACKAGE_NAME> oauth2accesstoken $(gcloud auth print-access-token) Containerized Pre-requirements Docker ^20.10 Docker Compose ^2.0 Google Cloud SDK ^489.0.0 Read access to your GAR Python repository Logged into GCP (gcloud auth application-default login) Docker ^20.10 Docker Compose ^2.0 Google Cloud SDK ^489.0.0 Read access to your GAR Python repository Logged into GCP ( gcloud auth application-default login ) gcloud auth application-default login Setup Secrets Configuration
First, define a secret in docker-compose.yaml using the local path to your credentials file:
secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json



We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Dockerfile Adjustments
In the Dockerfile, we handle credentials with the following setup:
ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS}



--mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. Service Configurations
Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration.
some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Secrets Configuration First, define a secret in docker-compose.yaml using the local path to your credentials file: secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml:


This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Secrets Configuration Secrets Configuration First, define a secret in docker-compose.yaml using the local path to your credentials file: docker-compose.yaml secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json secrets:
  gcloud_credentials:
    file: ~/.config/gcloud/application_default_credentials.json We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml: This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml: We define gcloud_credentials.file as a consistent path for Unix-like environments in docker-compose.yaml : gcloud_credentials.file docker-compose.yaml This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. This configuration securely passes the credentials file from your local machine to the build context without exposing sensitive data. Dockerfile Adjustments In the Dockerfile, we handle credentials with the following setup: ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process.


GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. Dockerfile Adjustments Dockerfile Adjustments In the Dockerfile, we handle credentials with the following setup: ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} ARG GOOGLE_APPLICATION_CREDENTIALS
ENV GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}

RUN --mount=type=secret,id=gcloud_credentials \
    mkdir -p $(dirname ${GOOGLE_APPLICATION_CREDENTIALS}) && \
    cp /run/secrets/gcloud_credentials ${GOOGLE_APPLICATION_CREDENTIALS} --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process. GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. --mount=type=secret,id=gcloud_credentials: Securely mounts the credentials during the build process. --mount=type=secret,id=gcloud_credentials : Securely mounts the credentials during the build process. --mount=type=secret,id=gcloud_credentials GOOGLE_APPLICATION_CREDENTIALS: Specifies the credential file's path within the container. GOOGLE_APPLICATION_CREDENTIALS : Specifies the credential file's path within the container. GOOGLE_APPLICATION_CREDENTIALS Service Configurations Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration. some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Service Configurations Service Configurations Here, gcloud_credentials is the secret mounted at build time, as specified in the secrets configuration. gcloud_credentials some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials some-service:
  build:
    context: .
    dockerfile: ./Dockerfile.local
    args:
      GOOGLE_APPLICATION_CREDENTIALS: /tmp/application_default_credentials.json
    secrets:
      - gcloud_credentials Usage Just run: docker-compose up --build docker-compose up --build Conclusion This approach allows local Docker builds to access private GAR resources securely, ensuring that credentials are handled appropriately and remain protected. Docker

Accessing Private Google Artifact Registry with Poetry: Local and Docker Setup

How to Use Docker to Template Local Database

Are you here because of the error message above when building the Dockerfile? You're not alone. This is a common issue that many developers encounter. Understanding the Issue Dockerfile keywords can be written in uppercase or lowercase, but it’s best not to mix them for clarity and consistency. The error message "WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match" specifically points to a mismatch in the casing of the 'FROM' and 'AS' keywords. This rule highlights instances where mixed case is used in a FROM instruction that includes the AS keyword to specify a stage name. For example, FROM ubuntu as a builder would trigger this warning because 'FROM' is in uppercase while 'as' is in lowercase. Why Casing Matters Consistency in casing improves readability and reduces the potential for errors. When working in teams or on open-source projects, maintaining a standard practice for casing can prevent misunderstandings and ensure that scripts and Dockerfiles are more predictable and easier to manage. In Docker, the 'FROM' keyword is used to specify the base image, and the 'AS' keyword is used to name the build stage in multi-stage builds. The inconsistency in casing can lead to warnings and, in some cases, errors that can disrupt your build process. Recent Changes in Docker Some recent versions of Docker enforce stricter guidelines for keyword casing. This means that Docker now expects the 'FROM' and 'AS' keywords to match in casing, typically both being in uppercase. This enforcement helps maintain a standard across Dockerfiles and aids in avoiding potential issues that arise from mixed casing. Quick Fix To resolve this issue, you need to ensure that both 'FROM' and 'AS' keywords are in the same case. The recommended approach is to use uppercase for both, as shown below: # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder This will prevent the warning from appearing during your build process. Additional Tips Consistency Across the Dockerfile: Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN, CMD, COPY, and ENV.
Linting Tools: Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early.
Documentation and Team Standards: Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards.
Updating Existing Dockerfiles: If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Consistency is key, and following these guidelines will help maintain the quality and reliability of your Docker projects. Are you here because of the error message above when building the Dockerfile? You're not alone. This is a common issue that many developers encounter. Understanding the Issue Understanding the Issue Dockerfile keywords can be written in uppercase or lowercase, but it’s best not to mix them for clarity and consistency. The error message "WARN: FromAsCasing: 'as' and 'FROM' keywords' casing do not match" specifically points to a mismatch in the casing of the 'FROM' and 'AS' keywords. This rule highlights instances where mixed case is used in a FROM instruction that includes the AS keyword to specify a stage name. For example, FROM ubuntu as a builder would trigger this warning because 'FROM' is in uppercase while 'as' is in lowercase. FROM ubuntu as a builder Why Casing Matters Why Casing Matters Consistency in casing improves readability and reduces the potential for errors. When working in teams or on open-source projects, maintaining a standard practice for casing can prevent misunderstandings and ensure that scripts and Dockerfiles are more predictable and easier to manage. In Docker, the 'FROM' keyword is used to specify the base image, and the 'AS' keyword is used to name the build stage in multi-stage builds. The inconsistency in casing can lead to warnings and, in some cases, errors that can disrupt your build process. Recent Changes in Docker Some recent versions of Docker enforce stricter guidelines for keyword casing. This means that Docker now expects the 'FROM' and 'AS' keywords to match in casing, typically both being in uppercase. This enforcement helps maintain a standard across Dockerfiles and aids in avoiding potential issues that arise from mixed casing. Quick Fix To resolve this issue, you need to ensure that both 'FROM' and 'AS' keywords are in the same case. The recommended approach is to use uppercase for both, as shown below: # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder # Use the Ubuntu image and name the stage "builder"
FROM ubuntu:latest AS builder This will prevent the warning from appearing during your build process. Additional Tips Consistency Across the Dockerfile: Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN, CMD, COPY, and ENV. Linting Tools: Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early. Documentation and Team Standards: Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards. Updating Existing Dockerfiles: If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Consistency Across the Dockerfile : Ensure that all Dockerfile instructions follow the same casing convention. This includes other keywords like RUN , CMD , COPY , and ENV . Consistency Across the Dockerfile RUN CMD COPY ENV Linting Tools : Use Dockerfile linting tools to automatically check for issues related to casing and other best practices. These tools can be integrated into your CI/CD pipeline to catch errors early. Linting Tools CI/CD Documentation and Team Standards : Document the casing standards in your project's guidelines and ensure that all team members are aware of and follow these standards. Documentation and Team Standards Updating Existing Dockerfiles : If you have existing Dockerfiles with mixed casing, take the time to update them. This can prevent future warnings and ensure that your Dockerfiles are compliant with the latest Docker versions. Updating Existing Dockerfiles Consistency is key, and following these guidelines will help maintain the quality and reliability of your Docker projects.

Fix Dockerfile Error: WARN: FromAsCasing: 'as' and 'FROM' Keywords' Casing Do Not match

There is a requirement to use Spark Operator in a K8s cluster to run a spark job. The official image contains many vulnerabilities, including those due to Hadoop libraries. Let's build our own Spark Operator image. To build our image, we'll need a spark image as a base image and a Golang image to build Spark Operator itself. Spark image Building a Spark image without Hadoop using a specific version of Spark RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz Spark-operator image We build the Spark Operator image, we will need several Hadoop libraries to run submit commands. For example, the FIPS version build is given, the differences in the build and run commands. For building on Go, the parameter GOEXPERIMENT=boringcrypto is used For running spark-submit, the java parameter for Bouncy Castle is used Djavax.net.ssl.trustStorePassword=password You can build an image without FIPS changes. To run spark-submit, we will add Hadoop libraries during the build process: hadoop-client-runtime
hadoop-client-api
slf4j-api entrypoint.sh is used from the official Kubeflow repository https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh Example Dockerfile for building Spark Operator ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] Conclusion After the build, we still have several vulnerabilities in the Hadoop library hadoop-client-runtime: org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410
org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 Since without this library we'll not be able to run spark-submit, but the rest of the huge part of the vulnerabilities is removed along with the main Hadoop libraries. There is a requirement to use Spark Operator in a K8s cluster to run a spark job. The official image contains many vulnerabilities, including those due to Hadoop libraries. Let's build our own Spark Operator image. To build our image, we'll need a spark image as a base image and a Golang image to build Spark Operator itself. Spark image Building a Spark image without Hadoop using a specific version of Spark RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz RUN curl -L https://dlcdn.apache.org/spark/spark-3.5.1/spark-3.5.1-bin-without-hadoop.tgz -o spark-3.5.1-bin-without-hadoop.tgz \
    && tar -xvzf spark-3.5.1-bin-without-hadoop.tgz \
    && mv spark-3.5.1-bin-without-hadoop /opt/spark \
    && rm spark-3.5.1-bin-without-hadoop.tgz Spark-operator image We build the Spark Operator image, we will need several Hadoop libraries to run submit commands. For example, the FIPS version build is given, the differences in the build and run commands. For building on Go, the parameter GOEXPERIMENT=boringcrypto is used GOEXPERIMENT=boringcrypto For running spark-submit, the java parameter for Bouncy Castle is used Djavax.net.ssl.trustStorePassword=password Djavax.net.ssl.trustStorePassword=password You can build an image without FIPS changes. To run spark-submit, we will add Hadoop libraries during the build process: hadoop-client-runtime hadoop-client-api slf4j-api hadoop-client-runtime hadoop-client-runtime hadoop-client-api hadoop-client-api slf4j-api slf4j-api entrypoint.sh is used from the official Kubeflow repository https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh entrypoint.sh https://github.com/kubeflow/spark-operator/blob/master/entrypoint.sh Example Dockerfile for building Spark Operator ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] ARG SPARK_IMAGE=spark-3.5.1-bin-without-hadoop
ARG GOLANG_IMAGE=golang-1.21
ARG SPARK_OPERATOR_VERSION=1.3.1
ARG HADOOP_VERSION_DEFAULT=3.4.0
ARG HADOOP_TMP_HOME="/opt/hadoop"
ARG TARGETARCH=amd64

# Prepare spark-operator build
FROM ${GOLANG_IMAGE} as builder
WORKDIR /app/spark-operator

ARG SPARK_OPERATOR_VERSION
RUN curl -Ls https://github.com/kubeflow/spark-operator/archive/refs/tags/spark-operator-chart-${SPARK_OPERATOR_VERSION}.tar.gz | tar -xz --strip-components 1 -C /app/spark-operator

RUN GOTOOLCHAIN=go1.22.3 go mod download

# Build
ARG TARGETARCH
RUN CGO_ENABLED=0 GOOS=linux GOARCH=${TARGETARCH} GO111MODULE=on GOTOOLCHAIN=go1.22.3 GOEXPERIMENT=boringcrypto go build -a -o /app/spark-operator/spark-operator main.go

#Install Hadoop jars
ARG HADOOP_VERSION_DEFAULT
ARG HADOOP_TMP_HOME
RUN mkdir -p ${HADOOP_TMP_HOME}
RUN curl -Ls https://archive.apache.org/dist/hadoop/common/hadoop-${HADOOP_VERSION_DEFAULT}/hadoop-${HADOOP_VERSION_DEFAULT}.tar.gz | tar -xz --strip-components 1 -C ${HADOOP_TMP_HOME}

# Prepare spark-operator image
FROM ${ECR_URL}:${SPARK_IMAGE}
WORKDIR /opt/spark-operator
USER root

ENV PATH $JAVA_HOME/bin:$PATH
ENV SPARK_HOME="/opt/spark"
ENV JAVA_HOME="/opt/jdk-11.0.21"
ENV SPARK_SUBMIT_OPTS="${SPARK_SUBMIT_OPTS} -Djavax.net.ssl.trustStorePassword=password"
ENV PATH=${PATH}:${SPARK_HOME}/bin:${JAVA_HOME}/bin:

RUN yum update -y && \
    yum install --setopt=tsflags=nodocs -y openssl && \
    yum clean all

ARG HADOOP_TMP_HOME
COPY --from=builder ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-runtime-*.jar ${HADOOP_TMP_HOME}/share/hadoop/client/hadoop-client-api-*.jar ${HADOOP_TMP_HOME}/share/hadoop/common/lib/slf4j-api-*.jar /opt/spark/jars/

COPY --from=builder /app/spark-operator/spark-operator /opt/spark-operator/
COPY --from=builder /app/spark-operator/hack/gencerts.sh /usr/bin/

COPY entrypoint.sh /opt/spark-operator/
RUN chmod a+x /opt/spark-operator/entrypoint.sh
ENTRYPOINT ["/opt/spark-operator/entrypoint.sh"] Conclusion After the build, we still have several vulnerabilities in the Hadoop library hadoop-client-runtime: org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410 org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 org.apache.avro:avro (hadoop-client-runtime-3.4.0.jar) – CVE-2023-39410 org.apache.commons:commons-compress – CVE-2024-25710, CVE-2024-26308 Since without this library we'll not be able to run spark-submit, but the rest of the huge part of the vulnerabilities is removed along with the main Hadoop libraries.

Building a Custom Docker Image for K8s Spark Operator to Fix Vulnerabilities

Stories, News & Mentions about DOCKER

Categories

Trending Topics

Stories, News & Mentions about DOCKER

Docker’s Stories on HackerNoon

Docker's News & Mentions

Categories

Trending Topics