Zuckerberg said Meta's Llama 4 models were training on an H100 cluster "bigger than anything that I've seen reported for what others are doing."

Mark Zuckerberg Flexes Meta's Nvidia Chip Cluster - Business Insider

Cluster

cluster

Hello friends, The Oracle DBA occasionally needs to update some of the Oracle Clusterware network configurations. Today let's check how to update the private network part. The Environment Oracle RAC with 2 nodes: ol8-19-rac1 and ol8-19-rac2

Oracle Linux Server release 8.8

Grid Version
34318175;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34318175)
34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)
34139601;ACFS RELEASE UPDATE 19.16.0.0.0 (34139601)
34133642;Database Release Update : 19.16.0.0.220719 (34133642)
33575402;DBWLM RELEASE UPDATE 19.0.0.0.0 (33575402)

DB Version
34086870;OJVM RELEASE UPDATE: 19.16.0.0.220719 (34086870)
34160635;OCW RELEASE UPDATE 19.16.0.0.0 (34160635)
34133642;Database Release Update : 19.16.0.0.220719 (34133642)

When you see <dbenv>, load the DB HOME variables.
When you see <gridenv>, load the GRID HOME variables.

IPs
# Public
192.168.56.161  ol8-19-rac1.localdomain  ol8-19-rac1
192.168.56.162  ol8-19-rac2.localdomain  ol8-19-rac2

# Virtual
192.168.56.163  ol8-19-rac1-vip.localdomain    ol8-19-rac1-vip
192.168.56.164  ol8-19-rac2-vip.localdomain    ol8-19-rac2-vip

# Private
10.1.2.161  ol8-19-rac1-priv.localdomain   ol8-19-rac1-priv
10.1.2.162  ol8-19-rac2-priv.localdomain   ol8-19-rac2-priv As the grid user, take a backup of profile.xml on all cluster nodes before proceeding. [oracle@ol8-19-rac1 ~]$ cp -p $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml.bkp

[oracle@ol8-19-rac1 ~]$ ll $ORACLE_HOME/gpnp/`hostname`/profiles/peer/profile.xml.bkp

-rw-r--r--. 1 oracle oinstall 2171 May 27 20:09 /u01/app/19.0.0/grid/gpnp/ol8-19-rac1/profiles/peer/profile.xml.bkp Now, let’s check the Oracle CRS networks and the Linux networks. It’s possible to identify that Oracle is using the network as his private one. “eth2 10.1.2.0” [oracle@ol8-19-rac1 ~]$ oifcfg getif
eth1  192.168.56.0  global  public
eth2  10.1.2.0  global  cluster_interconnect,asm

[oracle@ol8-19-rac1 ~]$ ip -c -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             10.0.2.15/24 fe80::a00:27ff:fe63:d73f/64
eth1             UP             192.168.56.161/24 192.168.56.163/24 192.168.56.166/24 192.168.56.165/24 fe80::a00:27ff:fe4f:b8aa/64
eth2             UP             10.1.2.161/24 169.254.1.211/19 fe80::a00:27ff:fe01:a7de/64
eth3             UP             10.1.3.161/24 fe80::a00:27ff:fef3:172b/64

[oracle@ol8-19-rac2 ~]$ ip -c -br a
lo               UNKNOWN        127.0.0.1/8 ::1/128
eth0             UP             10.0.2.15/24 fe80::a00:27ff:fe63:d73f/64
eth1             UP             192.168.56.162/24 192.168.56.164/24 192.168.56.167/24 fe80::a00:27ff:fe76:b954/64
eth2             UP             10.1.2.162/24 169.254.26.118/19 fe80::a00:27ff:fe7f:8663/64
eth3             UP             10.1.3.162/24 fe80::a00:27ff:fea8:d7c3/64 Let’s also check the ASM Listener configuration and take note. [oracle@ol8-19-rac1 ~]$ srvctl config listener -asmlistener
Name: ASMNET1LSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 10.1.2.0
Home: <CRS home>
End points: TCP:1525
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes:

[oracle@ol8-19-rac1 ~]$ srvctl status listener -l ASMNET1LSNR_ASM
Listener ASMNET1LSNR_ASM is enabled
Listener ASMNET1LSNR_ASM is running on node(s): ol8-19-rac2,ol8-19-rac1

[oracle@ol8-19-rac1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNET1LSNR_ASM Firstly, let’s check if both nodes are able to reach the network we intend to use. [oracle@ol8-19-rac1 tmp]$ ping -c 3 10.1.3.161
PING 10.1.3.161 (10.1.3.161) 56(84) bytes of data.
64 bytes from 10.1.3.161: icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 10.1.3.161: icmp_seq=2 ttl=64 time=0.014 ms
64 bytes from 10.1.3.161: icmp_seq=3 ttl=64 time=0.018 ms

--- 10.1.3.161 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2052ms
rtt min/avg/max/mdev = 0.010/0.014/0.018/0.003 ms

[oracle@ol8-19-rac1 tmp]$ ping -c 3 10.1.3.162
PING 10.1.3.162 (10.1.3.162) 56(84) bytes of data.
64 bytes from 10.1.3.162: icmp_seq=1 ttl=64 time=0.168 ms
64 bytes from 10.1.3.162: icmp_seq=2 ttl=64 time=0.151 ms
64 bytes from 10.1.3.162: icmp_seq=3 ttl=64 time=0.171 ms

--- 10.1.3.162 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2068ms
rtt min/avg/max/mdev = 0.151/0.163/0.171/0.013 ms

[oracle@ol8-19-rac2 tmp]$ ping -c 3 10.1.3.161
PING 10.1.3.161 (10.1.3.161) 56(84) bytes of data.
64 bytes from 10.1.3.161: icmp_seq=1 ttl=64 time=0.153 ms
64 bytes from 10.1.3.161: icmp_seq=2 ttl=64 time=0.156 ms
64 bytes from 10.1.3.161: icmp_seq=3 ttl=64 time=0.176 ms

--- 10.1.3.161 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2081ms
rtt min/avg/max/mdev = 0.153/0.161/0.176/0.017 ms

[oracle@ol8-19-rac2 tmp]$ ping -c 3 10.1.3.162
PING 10.1.3.162 (10.1.3.162) 56(84) bytes of data.
64 bytes from 10.1.3.162: icmp_seq=1 ttl=64 time=0.010 ms
64 bytes from 10.1.3.162: icmp_seq=2 ttl=64 time=0.025 ms
64 bytes from 10.1.3.162: icmp_seq=3 ttl=64 time=0.015 ms

--- 10.1.3.162 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2037ms
rtt min/avg/max/mdev = 0.010/0.016/0.025/0.007 ms Since everything seems OK, now we will start changing the private network. We will change it from the subnet to . As the grid user, proceed as below in only one of the nodes. eth2:10.1.2.0 eth3:10.1.3.0 [oracle@ol8-19-rac1 ~]$ oifcfg setif -global eth3/10.1.3.0:cluster_interconnect,asm
[oracle@ol8-19-rac1 ~]$ srvctl add listener -asmlistener -l ASMNEWLSNR_ASM -subnet 10.1.3.0
[oracle@ol8-19-rac1 ~]$ srvctl update listener -listener ASMNET1LSNR_ASM -asm -remove -force

--The error below is expected, as we are only testing if the previous listener still exists.
[oracle@ol8-19-rac1 ~]$ lsnrctl stop ASMNET1LSNR_ASM

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 28-MAY-2023 07:55:09

Copyright (c) 1991, 2023, Oracle.  All rights reserved.

TNS-01101: Could not find listener name or service name ASMNET1LSNR_ASM Check if the ASM Listener was created correctly. [oracle@ol8-19-rac1 ~]$ srvctl config asm
ASM home: <CRS home>
Password file: +CRS/orapwASM
Backup of Password file: +CRS/orapwASM_backup
ASM listener: LISTENER
ASM instance count: 2
Cluster ASM listener: ASMNEWLSNR_ASM

[oracle@ol8-19-rac1 ~]$ srvctl config listener -asmlistener
Name: ASMNEWLSNR_ASM
Type: ASM Listener
Owner: oracle
Subnet: 10.1.3.0
Home: <CRS home>
End points: TCP:1526
Listener is enabled.
Listener is individually enabled on nodes:
Listener is individually disabled on nodes: In the 19c version, we also have one more resource to handle, the ASM Network. [oracle@ol8-19-rac1 ~]$ srvctl config asmnetwork
ASM network 1 exists
Subnet IPv4: 10.1.2.0//
Subnet IPv6:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:

[oracle@ol8-19-rac1 ~]$ srvctl status asmnetwork
ASM network is running on ol8-19-rac2,ol8-19-rac1 As the root user, proceed as below. [root@ol8-19-rac1 scripts]# srvctl remove asmnetwork -netnum 1 -force
[root@ol8-19-rac1 scripts]# srvctl add asmnetwork -netnum 1 -subnet 10.1.3.0 Now let’s restart the entire cluster. [root@ol8-19-rac1 ~]# crsctl stop cluster -all -f
CRS-2673: Attempting to stop 'ora.crsd' on 'ol8-19-rac2'
CRS-2673: Attempting to stop 'ora.crsd' on 'ol8-19-rac1'
...
CRS-2673: Attempting to stop 'ora.cssd' on 'ol8-19-rac1'
CRS-2677: Stop of 'ora.cssd' on 'ol8-19-rac1' succeeded

[root@ol8-19-rac1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac1'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@ol8-19-rac2 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac2'
...
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'ol8-19-rac2' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@ol8-19-rac1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@ol8-19-rac2 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started. Check if everything is OK. [oracle@ol8-19-rac1 tmp]$ crsctl status res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.chad
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.net1.network
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.ons
               ONLINE  ONLINE       ol8-19-rac1              STABLE
               ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.proxy_advm
               OFFLINE OFFLINE      ol8-19-rac1              STABLE
               OFFLINE OFFLINE      ol8-19-rac2              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNEWLSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.CRS.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              Started,STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              Started,STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
      2        OFFLINE OFFLINE                               STABLE
ora.cdbrac.db
      1        ONLINE  ONLINE       ol8-19-rac1              Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /dbhome_1,STABLE
      2        ONLINE  ONLINE       ol8-19-rac2              Open,HOME=/u01/app/o
                                                             racle/product/19.0.0
                                                             /dbhome_1,STABLE
ora.cvu
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.ol8-19-rac1.vip
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.ol8-19-rac2.vip
      1        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.qosmserver
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       ol8-19-rac2              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       ol8-19-rac1              STABLE
-------------------------------------------------------------------------------- In my case, the ASM Network was not running on node 2, so I started it manually, and I will check in future reboots if I face the same issue again. [oracle@ol8-19-rac2 tmp]$ srvctl start asmnetwork

[oracle@ol8-19-rac2 tmp]$ srvctl status asmnetwork
ASM network is running on ol8-19-rac2,ol8-19-rac1 Since everything is running smoothly, let’s remove the previous network from the Oracle CRS. As the grid user, proceed as below. [oracle@ol8-19-rac1 tmp]$ oifcfg delif -global eth2/10.1.2.0

[oracle@ol8-19-rac1 tmp]$ oifcfg getif
eth1  192.168.56.0  global  public
eth3  10.1.3.0  global  cluster_interconnect,asm In the next article, I will explain how to update the IP from the other layers of the Oracle CRS. That’s it!

How to Modify Private Network Information in Oracle Clusterware 19c

A Raspberry Pi is a mini-computer board to which you can connect a monitor, mouse, and keyboard, and install a Linux-based operating system with a GUI. Or you can use it in “headless” mode with no GUI and run, for example, a database server. There are you can give a Raspberry Pi—from building a Minecraft server to smart mirrors, the possibilities are endless. many usages Since I started to discover MariaDB and learned about database clusters, , and , the idea of building a Raspberry Pi cluster has been in the back of my head. A cluster like this is a great way to experiment with distributed systems. Distributed SQL Xpand In this article, I’ll show you how to build a Raspberry Pi cluster with: 8 nodes 32 cores 64 GB of RAM 2TB of storage What You Need If you want to build your own cluster, you don’t have to buy 8 Raspberry Pi devices. Even one device is enough to learn stuff. The instructions in this article are useful even if you plan to configure only one My advice, however, is to have at least three devices to build your cluster if you truly want to boost your skills in administration, , , , … you name it! Raspberry Pi. Linux Ansible Docker Kubernetes database clusters With that in mind, please replace 8 with whatever is your number. Do the same with the specifications of the Raspberry Pi devices if you have different models with less or more RAM or storage. Here’s what I used: 8 x (8 GB RAM) Raspberry Pi 4 Model B 1 x (10-Ports) Anker 60W Desktop Charger 8 x (5A, 40W, 23cm) Short USB Type C cable 8 x (256 GB) MicroSD card 2 x (4 layers with cooling fan) Raspberry Pi cluster case Adapt the quantities to your setup. You can either get one or more if you have very few devices (say 1 or 2) or a multi-USB charger. If you go with a multi-USB charger, make sure that each port can deliver at least 2.4Am (5V). Pick microSD cards that fit your goals and budget as well. I recommend going with at least 32 GB SD cards. Raspberry Pi power supplies On the cables and case side, pick anything that accommodates your setup. You don’t necessarily need cooling fans, but I recommend them if you plan to leave your devices on for prolonged periods of time. I also recommend getting all the ingredients before you start “cooking," especially if you want to use multiple Raspberry Pi devices. Installing Raspberry Pi OS (Headless) The operating system (OS) we’ll use is , a Debian-based OS optimized for Raspberry Pi boards. Raspberry Pi OS is installed on the microSD cards using your computer. You later connect these microSD cards to your Raspberry Pi devices and boot them. Raspberry Pi OS Get all the microSD cards and Raspberry Pi boxes on your desk. You’ll need an SD card adapter (they usually come with microSD cards) or an SD card USB reader if your working computer doesn’t have a slot for SD cards. Bring a Sharpie as well. Download the application, and install it on your computer. Take a microSD card and connect it to your computer. Open the Raspberry Pi Imager application, and click on . From the list, click on and select if your Raspberry Pi devices are 64-bit or otherwise: Raspberry Pi Imager CHOOSE OS Raspberry Pi OS (other), Raspberry Pi OS Lite (64-bit) Raspberry Pi OS Lite (32-bit) Click on and select the microSD card. Double-check that you selected the correct drive, and click on the “gear” icon (advanced options). For hostname use , or something similar. You’ll be naming the devices as , , , etc. CHOOSE STORAGE, rpi01 rpi01 rp02 rp03 Check the and options. Set a username (I’ll leave the default ), and set a secure password. Enable SSH, Use password authentication pi Check the option, and enter the name and password of your WiFi connection. Configure your locale settings as well. Configure wireless LAN Since we have to do this once with each microSD card, make sure to set the options field to: . This way the settings will be saved, and the process will be easier for the next cards: Image customization to always use Now, you can click and then to start the process. Once completed, eject the card and connect it to one of the Raspberry Pi devices, put it back into its box, and mark the box with the number . Repeat this process for all of the devices. SAVE WRITE 01 Assembling the Cluster Depending on what kind of cluster case you have, the assembling process might vary. Follow the instructions that come with your case, or look for photographs online to get a clear idea of how the devices should be placed in the layers. I recommend looking ahead to decide which parts to assemble first. For example, with the case I picked, I had to install the cooling fans before mounting the Raspberry Pi devices on the layers. Once you mount a Raspberry Pi, use a Sharpie to write its number on the layer. This will help you in the future if you have to take a microSD card out for reconfiguration or any other administrative tasks that involve touching the hardware. Once you have all the devices mounted, connect the USB cables to the Raspberry Pi devices and the multi-USB charger. You can optionally use and a instead of the USB charger if you prefer, but I’ll leave that for you to explore; it’s definitely worth checking. Power over Ethernet network switch You can either have the charger (or network switch if you go for it) next to your Raspberry Pi rack or attached to it. I used rubber bands to keep the charger attached to the side of the case. It worked fine. With the USB cables plugged in, you are ready to start all those mini-computers! Connect the charger, and switch it on. Depending on the Raspberry Pi models that you use, they might take some time to boot, so be patient. Meanwhile, enjoy the LEDs flashing and fans starting. Connecting Through SSH If everything went well, you should be able to reach the Raspberry Pi devices through SSH. Give it a try by running the following from your computer: ssh pi@rpi01.local Introduce your password, and answer to add the device to the list of known hosts. Congrats! Your cluster is live now. yes You may want to run commands to check the hardware information. For example: : Information about the CPU architecture lscpu : File system disk space usage df -H : Partition information sudo fdisk -l : Amount of used, free and total amount of RAM free -m : Linux kernel information cat /proc/version Manual Configuration With raspi-config If you have one Raspberry Pi, you can connect to it through SSH as described above, and use the utility program to configure various settings. You can modify the WiFi connection ( ), change the hostname ( ) and user password ( ), and expand the filesystem ( ), among many other things. raspi-config S1 S4 S3 A1 Automated Configuration With Ansible If you have more than one Raspberry Pi in your cluster, you might want to automate the configuration process using a tool like . With Ansible, you can run a command on multiple machines without having to perform repetitive tasks. For example, let’s say we want to expand the filesystem on each Raspberry Pi. This can be done from the command line as follows: Ansible sudo raspi-config --expand-rootfs Without automation, you’d have to SSH to , run the command above, and end the SSH session. You’d have to repeat all these steps for , , , and so forth. Instead, you can simply tell Ansible to run the command for you on all the machines. Let’s see how to do this. rpi01.local rpi02.local pr03.local rp04.local You have to install Ansible on a that is connected to your local network. It can be one of the Raspberry Pi devices, your working laptop, or any other machine as long as it runs a Unix-based operating system like Linux or macOS. control node I happen to have an old laptop that I connected to my network, so I used it as the control node (I’ll show how to install a database with replication in the Raspberry Pi devices in a future article). repurposed as a dedicated database I won’t go through the instructions on installing Ansible since the process is different on different operating systems. Check the and install it on your laptop or any device that you want to assign as a control node. official documentation Before you start, I recommend generating an example configuration file that you can tweak later as you wish: sudo su
ansible-config init --disabled -t all > /etc/ansible/ansible.cfg
exit In this file, disable host key checking to simplify the process. Don’t do that in production environments! Change the following line: ;host_key_checking=True to this: host_key_checking=False Next, you need to define an inventory. This is a list of machines that you want to control with Ansible. The inventory is defined in the file. For example, you could add the following machines to the inventory using their IP addresses or hostnames: /etc/ansible/hosts [mymachines]
foo.example.com
bar.example.com
192.0.2.50
192.0.2.51 If the machines have some kind of pattern in their IP addresses or hostnames, you can alternatively use ranges. This is what you can do with your Raspberry Pi devices. Edit the file by adding the following to the end: /etc/ansible/hosts [rpis]
rpi[01-08].local Change the pattern to match your hostnames and numbers. This is equivalent to: [rpis]
rpi01.local
rpi02.local
rpi03.local
rpi04.local
rpi05.local
rpi06.local
rpi07.local
rpi08.local In this inventory, is an arbitrary name that you can use to refer to all the Raspberry Pi devices when running commands on them using Ansible. rpis You need to configure the SSH username that Ansible will use when connecting to the machines. Add the following to the file: /etc/ansible/hosts [rpis:vars]
ansible_user=pi Time to control de machines. A good start is to them: ping ansible rpis -m ping --ask-pass Type the SSH password. You should see in the output how the “ping” is answered by a “pong” for each machine (I’m showing only one here): rpi01.local | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
} Let’s back to the filesystem expansion (although, I think nowadays this is not required anymore, let me know in the comments if you can confirm this). To perform this action on all the machines, run: ansible rpis -m shell -a "raspi-config --expand-rootfs" --become --ask-pass And as simple as that, all your Raspberry Pi devices have an expanded filesystem. This is a good time to update the system on all the machines: ansible rpis -m shell -a "apt update -y" --become --ask-pass
ansible rpis -m shell -a "apt upgrade -y" --become --ask-pass You can reboot all the devices as follows: ansible rpis -m shell -a "reboot" --become --ask-pass And once you are done and want to safely turn all the machines off, just run: ansible rpis -m shell -a "poweroff" --become --ask-pass What's Next? This is only the beginning. You now have a cluster of small computers ready to be controlled by Ansible. You can try setting up any kind of server software, including web servers or databases ( ). There also is much more about Ansible. see this example I merely scratched the surface here using , but you can for example create that contain the state in which you want your cluster to be. Explore the , and have fun! ad-hoc commands playbooks documentation Also published here

How to Build a 32-Core Raspberry Pi Cluster From Scratch

Cómo construir un clúster Raspberry Pi de 32 núcleos desde cero

Comment construire un cluster Raspberry Pi à 32 cœurs à partir de zéro

स्क्रैच से 32-कोर रास्पबेरी पाई क्लस्टर कैसे बनाएं

32 コアの Raspberry Pi クラスターをゼロから構築する方法

Como construir um cluster Raspberry Pi de 32 núcleos a partir do zero

Cách xây dựng cụm Raspberry Pi 32 lõi từ Scratch

如何从零开始构建 32 核 Raspberry Pi 集群

is an open source system for automatically deploying, scaling, and managing containerized applications. Containerization is a lightweight alternative to full machine virtualization that involves encapsulating an application in a container with its own operating environment. Containerization has matured over the past few years and has led to greater adoption of Kubernetes across the tech industry. Kubernetes Kubernetes greatly increases a user’s ability to run testing and simulations. At a high level, it allows users to expand their technical resources outside of their traditional limits; for example, users can simulate a 101-node blockchain cluster without needing to own 101 physical machines. This is ideal for quickly standing up staging environments for a blockchain cluster. It is also particularly useful for simulating blockchain networks with proof of stake (PoS) or delegated proof of stake (DPoS) consensus mechanisms, which involve a finite amount (e.g., 101) of geographically distributed block producing / verifying nodes. Starting with our , IoTeX has used Kubernetes to deploy and optimize our Testnet infrastructure. Overall, Kubernetes has greatly improved our operations — in this blog, we share our experience and some tips to run your own containerized applications and environments on Kubernetes. Testnet Beta “Epik” release Setting up the IoTeX cluster on Kubernetes We begin by defining two type of Kubernetes : services used to boot up the whole IoTeX cluster. It is the entry point of any new iotex nodes and helps new nodes join the P2P network. All nodes identify each other through their internal Pod’s IP address. bootnode service: exposes the JSON-RPC API from all iotex nodes. The JSON-RPC interface is used by our and our action injection tool. iotex service: open source IoTeX Explorer All node configurations are deployed through the Kubernetes , including the genesis block. This allows us to redeploy a new cluster without waiting for a new docker image to be built. ConfigMap Applications running in Kubernetes Pods are stateless by default, which means saved data from any applications will not be available after a redeployment. To counteract this, we use the following setup process. In our scenario each IoTeX node will utilize the same copy of data (i.e., distributed ledger), so it is not necessary to persist every nodes’ data. Therefore, we mount the Kubernetes to only one iotex node — in our case, we mounted it to our boot node. We also back up our persisted data to object storage spaces like S3 and Digital Ocean. Since all nodes except the boot node will not have saved data after a redeployment, they will need to first download the backup before startup. We use init container to achieve this. This process enables our cluster to restart from a crash our failure without worrying about losing previous data. Persistent Volume Cluster Monitoring It is not trivial to track the status of 21 nodes (let alone 101 nodes down the road), and it becomes more complex with a larger amount of nodes. To have better observability within our IoTeX cluster, we set up monitoring capabilities in Kubernetes, which consists of a logging stack, metrics stack, and alert manager. For the logging stack, we use + + + . Fluentd sends logs from each node to the Elastic Search client, and the Elastic Search master server indexes the logs which are query-able through Kibana. Finally, we use the Elastic Search Curator to clean up the outdated logs. Fluentd Elastic Search Kibana Elastic Search Curator Refer from Log aggregation with ElasticSearch, Fluentd and Kibana stack on ARM64 Kubernetes cluster For the metrics stack, we use + + . provides an easy way to configure Prometheus running in a Kubernetes cluster. We can hook up the IoTeX node’s Prometheus client with the Prometheus server by simply exposing a metrics service. Prometheus Prometheus Operator Grafana CoreOS Prometheus Operator Refer from CoreOS: Prometheus Operator We are also working on setting up alerts with Prometheus Alert Manager to send out alerts to our on-call engineer when we have abnormal metrics. Using Helm to Manage Multiple Configurations Using yaml configuration to deploy different types of applications on Kubernetes is fairly straightforward for a single environment. However, when you have multiple environments (e.g., testing, staging, production) and different node twists / configurations (e.g., 21 vs. 101 delegates), users encounter higher overhead and redundancy to manage various Kubernetes yaml configurations for the same application. To solve this issue, we use Helm. is a package manager for Kubernetes applications. It allows us to create our IoTeX cluster as a chart package with version and default values. With Helm, we don’t need to manage duplicate Kubernetes yaml files; rather, we only need to manage a smaller subset of configurations for different environments and twists which overwrite the default value. It also simplifies the command we need to run to start a fresh IoTeX cluster into one single command: Helm Helm also allows us to publish a chart easily. In the future, we will publish our IoTeX Helm chart to the community, so you can setup an IoTeX cluster within minutes. More to come! With the power of Kubernetes and other operational tools, IoTeX’s Development team not only saves time on cluster operation, but also digests issues quicker and iterates even faster than before. We look forward to sharing more technical perspectives in the future — feel free to reach out with any questions to support@iotex.io. About IoTeX IoTeX is the world’s first privacy-centric blockchain platform that is fast, flexible, and Internet of Things (IoT) friendly. IoTeX’s global team is comprised of Ph.Ds in Cryptography, Distributed Systems, and Machine Learning, top tier engineers, and experienced ecosystem builders. Designed and optimized for IoT, IoTeX uses state-of-the-art privacy, consensus, and subchain innovations to capture the full potential of IoT. By enabling trusted data, interoperability, and M2M automation, IoTeX connects the physical and digital worlds and brings trusted machine economies to the masses. Stay connected with us! Website: Twitter: Telegram Announcement Channel: Telegram Group: Medium: Reddit: Join us: https://iotex.io/ https://twitter.com/iotex_io https://t.me/iotexchannel https://t.me/IoTeXGroup https://medium.com/@iotex https://www.reddit.com/r/IoTeX/ https://iotex.io/careers

Deploy, operate, and monitor IoTeX cluster with Kubernetes

This post is part of “ ” series explaining how to use concepts with . In this part, I will show you how to setup an cluster on using & as shown in the diagram below : IaC Infrastracture as Code Terraform etcd AWS CoreOS Terraform All the templates used in this demo can be found on my . Github So let’s start with “ ” file which contains the global variables such as AWS region, cluster instances type … variables.tf Note: As of writing this article, the latest stable version is . CoreOS 1465.6.0 So make sure to find an that is as close to the latest version as possible. AMI Next, we need to define a for our cluster. For simplicity, Im going to make this open to the world. Even though security is important, this tutorial serves an educational purposes and you should never have all ports open in production. security group security group And finally, we will define our which consists of : cluster 3 Nodes In order to bring up an , I used a file that I passed as a parameter to attribut: etcd cluster cloud config user_data Note: Make sure to grab the and place it into the discovery parameter: discovery token, $ curl https://discovery.etcd.io/new?size=3 https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de Once you defined all templates required, just type the following command to bring up the : etcd cluster terraform apply Note: Don’t forget to set the as an before: AWS credentials envrionment variables export AWS_ACCESS_KEY_ID=”YOUR ACCESS KEY ID” export AWS_SECRET_ACCESS_KEY=”YOUR SECRET ACCESS KEY” Setting up an in action is shown below: etcd cluster Once done, go to your then navigate to your : AWS Management Console EC2 Dashboard Congratulations ! You have your . CoreOS cluster To verify the c , you can either point your browser to the discovery url you generated earlier: luster health or to one of your cluster nodes using the command: SSH ssh core@<NODE-IP> Then, use the to fetch the : etcd command line cluster status Now we have an ready to use. Let’s see what we can do with it: etcd cluster Through etcdctl: etcdctl set nginx/port 80 etcdctl get nginx/port 80 etcdctl ls nginx etcdctl rm nginx/port Through HTTP API: curl -sS -X PUT -d value=”80" | jq ‘.’ # Create http://localhost:2379/v2/keys/nginx/port curl -sS | jq ‘.’ # Get http://localhost:2379/v2/keys/nginx/port curl -sS -X DELETE | jq ‘.’ # Delete http://localhost:2379/v2/keys/nginx/port

Setting up an etcd cluster on AWS using CoreOS & Terraform

In this post, I'm going to show you how to potentially triple your application's performance by managing multiple threads. This is an important tutorial, where the methods and the examples shown will give you what you need to set up production-ready thread management. Node Watch The Video on YouTube Child Processes, Clustering, and Worker Threads For the longest time, Node's had the ability to be multi-threaded, by using either , , or the more recent preferred method of a module called . Child Processes Clustering Worker Threads Child processes were the initial means of creating multiple threads for your application and have been available since version 0.10. This was achieved by spawning a node process for every additional thread you wanted to be created. Clustering, which has been a stable release since around version 4, allows us to simplify the creation and management of Child Processes. It works brilliantly when combined with . PM2 Now before we get into multithreading our app, there are a few points that you need to fully understand: 1. Multithreading already exists for I/O tasks There is a layer of Node that's already multithreaded and that is the thread-pool. I/O tasks such as files and folder management, TCP/UDP transactions, compression, and encryption are handed off to libuv, and if not asynchronous by nature, get handled in the libuv's thread-pool. libuv 2. Child Processes/Worker Threads only work for synchronous JavaScript logic Implementing multithreading using Child Processes or Worker Threads will only be effective for your synchronous JavaScript code that's performing heavy-duty operations, such as looping, calculations, etc. If you try to offload I/O tasks to Worker Threads as an example, you will not see a performance improvement. 3. Creating one thread is easy. Managing multiple threads dynamically is hard Creating one additional thread in your app is easy enough, as there are tons of tutorials on how to do so. However, creating threads equivalent to the number of logical cores your machine or VM is running and managing the distribution of work to these threads is way more advanced, and to code this, logic is above most of our pay grades 😎. Thank goodness we are in a world of open source and brilliant contributions from the Node community. Meaning, there is already a module that will give us the full capability of dynamically creating and managing threads based on the CPU availability of our machine or VM. Worker Pool The module we will work with today is called . Created by , Worker Pool offers an easy way to create a pool of workers for both dynamically offloading computations as well as managing a pool of dedicated workers. It's basically a thread-pool manager for Node JS, supporting Worker Threads, Child Processes, and Web Workers for browser-based implementations. Worker Pool Jos de Jong To make use of the Worker Pool module in our application, the following tasks will need to be performed: Install Worker Pool First, we need to install the Worker Pool module - npm install workerpool Init Worker Pool Next, we'll need to initialize the Worker Pool on the launch of our App Create Middleware Layer We'll then need to create a middleware layer between our heavy-duty JavaScript logic and the Worker Pool that will manage it Update Existing Logic Finally, we need to update our App to hand off heavy-duty tasks to the Worker Pool when required Managing Multiple Threads Using Worker Pool At this point, you have 2 options: Use your own NodeJS app (and install and modules), or download the from GitHub for this tutorial and my video series. workerpool bcryptjs source code NodeJS Performance Optimization If going for the latter, the files for this tutorial will exist inside the folder . Once downloaded, enter into the root project folder and run npm install. After that, enter into the folder to follow along. 06-multithreading 06-multithreading In the folder, we have 2 files: one is the controller logic for the Worker Pool (controller.js). The other holds the functions that will be triggered by the threads…aka the middleware layer I mentioned earlier (thread-functions.js). worker-pool worker-pool/controller.js WorkerPool = ( ) Path = ( ) poolProxy = init = (options) => { pool = WorkerPool.pool(Path.join(__dirname, ), options)
  poolProxy = pool.proxy() .log( )
} get = { poolProxy
} exports.init = init
exports.get = get 'use strict' const require 'workerpool' const require 'path' let null // FUNCTIONS const async const './thread-functions.js' await console `Worker Threads Enabled - Min Workers: - Max Workers: - Worker Type: ` ${pool.minWorkers} ${pool.maxWorkers} ${pool.workerType} const => () return // EXPORTS The controller.js is where we require the module. We also have 2 functions that we export, called and . The function will be executed once during a load of our application. It instantiates the Worker Pool with options we'll provide and a reference to the . It also creates a proxy that will be held in memory for as long as our application is running. The function simply returns the in-memory proxy. workerpool init get init thread-functions.js get worker-pool/thread-functions.js WorkerPool = ( ) Utilities = ( ) bcryptHash = { Utilities.bcryptHash(password)
} WorkerPool.worker({
  bcryptHash
}) 'use strict' const require 'workerpool' const require '../2-utilities' // MIDDLEWARE FUNCTIONS const ( ) => password return // CREATE WORKERS In the file, we create worker functions that will be managed by the Worker Pool. For our example, we're going to be using to hash passwords. This usually takes around 10 milliseconds to run, depending on the speed of one's machine, and makes for a good use case when it comes to heavy duty tasks. Inside the file is the function and logic that hashes the password. All we are doing in the thread-functions is executing this via the workerpool function. This allows us to keep code centralized and avoid duplication or confusion of where certain operations exist. thread-functions.js BcryptJS utilities.js bcryptHash 2-utilities.js BCrypt = ( ) bcryptHash = (password) => { BCrypt.hash(password, )
}

exports.bcryptHash = bcryptHash 'use strict' const require 'bcryptjs' const async return await 8 .env NODE_ENV= PORT= WORKER_POOL_ENABLED= "production" 6000 "1" The .env file holds the port number and sets the variable to "production". It's also where we specify if we want to enable or disable the Worker Pool, by setting the to "1" or "0". NODE_ENV WORKER_POOL_ENABLED 1-app.js ( ).config() Express = ( ) App = Express() HTTP = ( ) Utilities = ( ) WorkerCon = ( ) App.get( , (req, res) => { password = result = workerPool = (process.env.WORKER_POOL_ENABLED === ) {
    workerPool = WorkerCon.get()
    result = workerPool.bcryptHash(password)
  } {
    result = Utilities.bcryptHash(password)
  }

  res.send(result)
}) port = process.env.PORT server = HTTP.createServer(App)

; { (process.env.WORKER_POOL_ENABLED === ) { options = { : } WorkerCon.init(options)
  } server.listen(port, () => { .log( , port)
  })
})() 'use strict' require 'dotenv' const require 'express' const const require 'http' const require './2-utilities' const require './worker-pool/controller' // Router Setup '/bcrypt' async const 'This is a long password' let null let null if '1' await else await // Server Setup const const ( ) => ( async // Init Worker Pool if '1' const minWorkers 'max' await // Start Server console 'NodeJS Performance Optimizations listening on: ' Finally, our holds the code that will be executed on the launch of our App. First, we initialize the variables in the file. We then set up an server and create a route called . When this route is triggered, we will check to see if the Worker Pool is enabled. If yes, we get a handle on the Worker Pool proxy and execute the function that we declared in the file. This will in turn execute the function in and return us the result. If the Worker Pool is disabled, we simply execute the function directly in . 1-app.js .env Express /bcrypt bcryptHash thread-functions.js bcryptHash Utilities bcryptHash Utilities At the bottom of our , you'll see we have a self-calling function. We're doing this to support async/await, which we are using when interacting with the Worker Pool. Here is where we initialize the Worker Pool if it's enabled. The only config we want to override is setting the to "max". This will ensure that the Worker Pool will spawn as many threads as there are logical cores on our machine, with the exception of 1 logical core, which is used for our main thread. In my case, I have 6 physical cores with hyperthreading, meaning I have 12 logical cores. So with set to "max", the Worker Pool will create and manage 11 threads. Finally, the last piece of code is where we start our server and listen on port 6000. 1-app.js minWorkers minWorkers Testing the Worker Pool Testing the Worker Pool is as simple as starting the application and while it's running, preforming a get request to . If you have a load testing tool like , you can have some fun seeing the difference in performance when the Worker Pool is enabled/disabled. AutoCannon is very easy to use. http://localhost:6000/bcrypt AutoCannon Conclusion I hope this tutorial has provided insight into managing multiple threads in your Node application. The embedded video at the top of this article provides a live demo of testing the Node App. Till next time, cheers :) Previously published at http://bleedingcode.com/managing-multiple-threads-nodejs/

How to Manage Multithreaded Node JS Applications for Better Performance

Cómo administrar aplicaciones JS de nodo multiproceso para un mejor rendimiento

Introduction With ever-increasing demands from other business units, IT departments have to be constantly looking for service improvements and cost-saving opportunities. This article showcases several concrete use-cases for companies that are investigating or already using Kafka, in particular, Kafka Connect. Kafka Connect is an enterprise-grade solution for integrating a plethora of applications, ranging from traditional databases to business applications like Salesforce and SAP. Possible integration scenarios range from continuously streaming events and data between applications to large-scale, configurable batch jobs that can be used to replace manual data transfers. Why Kafka and Heroku started as an in-house project at LinkedIn intended to replace the various queuing and message-passing systems that had evolved over the years. The goal was to create a horizontally scalable system, able to reliably store and handle many millions of messages per second. Today, the Kafka community provides a whole ecosystem of tools and features ranging from simple log-message passing to performing basic on-the-fly ETL jobs to complex data-pipeline processing. Its versatility and robustness in addition to its scalability make Kafka a great choice for the enterprise. Apache Kafka The power of Kafka comes at a price: While it's easy to use Kafka from a client perspective, the setup and operation of Kafka is a difficult task. Setting up a reliable Kafka cluster is a challenge that requires experience. If your company does not have the experience, it can take weeks, even months to set up and fine-tune a production-ready cluster. That's where Kafka on Heroku shines: , which makes it feasible for IT departments to focus on the services they want to provide instead of operating a Kafka cluster. Setting up a private Kafka cluster literally takes minutes Adding Kafka Connect is the logical next step, once you are using Kafka. But is a huge time-saver, even if you are not planning to use Kafka directly from your application. With little operational overhead, Kafka Connect provides an integration point for any number of applications. Instead of running and maintaining different solutions for recurring data transfer, on-demand data transfer, and continuous updates between systems, Kafka Connect provides a unified solution to those problems. Kafka Connect on Heroku In the next section you will learn about some typical use cases, and also find out which applications Kafka Connect is able to integrate. Typical Business Use Cases for Kafka Connect IT departments constantly struggle with requirements from business units that must connect and integrate all kinds of custom and off-the-shelf applications. These integrations often use less-reliable mechanisms, like recurring batch jobs, instead of continuously "streaming" updates across systems. Kafka Connect provides a scalable, reliable solution that can often replace slow batch jobs by instead "streaming" updates. Common scenarios include: Your sales department regularly needs to update an internal SQL database with data from Salesforce. Use Kafka Connect to continuously stream the changes from Salesforce to your on-premise database. Your compliance department needs an audit trail for updates to certain databases or applications. Use Kafka Connect to stream the relevant changes to an on-premise file server, to S3 for offsite storage, or even to Elasticsearch for quick retrieval. Replace your unreliable, slow-running ETL batch jobs with a robust solution. Use Kafka Connect to: update data across multiple applications while making them faster and more reliable; continuously stream changes from structured files (CSV, JSON, logs, etc.) into databases and applications; reliably and continuously stream updates between heterogeneous applications and databases IT departments could even go as far as creating new self-service opportunities, where manual or labor-intensive requests for data transfer or extraction are replaced by self-service processes that reduce the IT department's workload while improving response time for business units. Consider a system in which your company's marketing department creates a ticket in your service-desk software, which automatically creates and triggers a data-transfer that otherwise the IT service desk would have had to handle manually. An (Incomplete) List of Systems That Kafka Connect Can Integrate Kafka Connect can an ever-increasing number of applications and services, including enterprise applications, like SAP and Salesforce: read from and write to any SQL database: Oracle, Microsoft SQL Server, IBM Db2, MySQL, PostgreSQL, MariaDB, and others NoSQL databases like MongoDB, Redis, Cassandra, InfluxDB, and others message queues like ActiveMQ, RabbitMQ, IBM MQ, Amazon SQS, Azure Event Hub, Google Pub/Sub, and others IT service-desk applications like Jira, ServiceNow, and Zendesk logging and monitoring services, applications, and protocols like SNMP, Syslog, Elasticsearch, Amazon CloudWatch, Appdynamics, Metrics, Splunk, and others generic "low-level" data sources and protocols, including HTTP, SFTP, MQTT, JMS, the file system, and others robust custom connectors can be easily written using Java, taking full advantage of the reliable Kafka Connect framework and the underlying infrastructure since Kafka Connect uses Kafka as the underlying infrastructure, any application that uses Kafka as a message bus is automatically available for integration Setting Up a Kafka Connect Integration Is Fast and Easy Let's take a concrete example. A common integration scenario is this: You have two SQL databases and you need to update one database with information from the other database. For example, when the status of order changes in your fulfillment system, you might need to update the internal CRM database to reflect this change. Here is the concrete scenario: whenever the STATUS table changes in the fulfillment database write the changes to the ORDER_STATUS table in the CRM database For simplicity, we'll assume both are MySQL databases. The steps for other connectors would be the same, only with different configurations. It takes two steps to set up this integration, assuming you have a working Kafka Cluster with Kafka Connect: create the "source-connection" to read data from the STATUS table of fulfillment database create the "sink-connection" to write data to the ORDER_STATUS table of the CRM database Here are the corresponding cURL calls, and an explanation of what each of them does. They use the to create the source and sink. Note that these calls are not specific to Heroku. They will work with any Kafka Connect installation: Kafka Connect REST API 1. Creating the source-connection. curl -X POST : , : { : , : , : , : , : , : , : , : , : }   
} http: //localhost:8083/connectors -H "Content-Type: application/json" -d '{ "name" "jdbc_source_mysql_01" "config" "connector.class" "io.confluent.connect.jdbc.JdbcSourceConnector" "connection.url" "jdbc:mysql://fulfillmentdbhost:3306/fulfillmentdb" "connection.user" "fullfilment_user" "connection.password" "<password>" "topic.prefix" "order-status-update-" "mode" "timestamp" "table.whitelist" "fulfullmentdb.status" "timestamp.column.name" "LAST_UPDATED" "validate.non.null" false ' 2. Creating the sink-connection. curl -X POST : , : { : , : , : , : , : , : }
} http: //localhost:8083/connectors -H "Content-Type: application/json" -d '{ "name" "jdbc_sink_mysql_01" "config" "connector.class" "io.confluent.connect.jdbc.JdbcSinkConnector" "connection.url" "jdbc:mysql://crmdbhost:3306/crmdb" "connection.user" "crm_user" "connection.password" "<password>" "topics" "order-status-update-status" "table.name.format" "crmdb.order_status" ' And you're done! From this point on, any new entries in the STATUS table of the fulfillment database will automatically be written to the ORDER_STATUS table of the CRM database. Let's dive a little into what each call is doing. cURL call no. 1: Connecting the Source The first cURL command tells Kafka Connect to use a specific type of source connector, namely JdbcSourceConnector, to connect to the MySQL database at fulfillmentdbhost:3306/fulfillmentdb using the supplied credentials. It also configures this connector to monitor the table called "status" in that database for new entries based on the timestamp in the column "LAST_UPDATED." As a result, whenever a new row is written to the table, Kafka Connect will automatically write a new message into the Kafka topic "order-status-update-status" for each new row. Next, we need a sink connector that monitors this Kafka topic for changes. That's what the second cURL call does. cURL call no. 2: Connecting the Sink The second cURL command creates a sink connector (JdbcSinkConnector) that connects to a different database, namely crmdbhost:3306/crmdb. This connector will monitor the topic "order-status-update-status" for new messages and will write any new messages to the table "order_status" in the crmdb-database. This example uses the JdbcSourceConnector and JdbcSinkConnector to connect two relational databases. Other types of connectors can be installed and then used in the same way. Each connector comes with its own documentation and set of options. An unofficial list of available connectors with links to their documentation can be found on the . Confluent website Taking It to the Next Level You have seen how to set up a continuously streaming integration between two SQL databases with just two commands. In a similar fashion, you could integrate any of the applications that Kafka Connect provides connectors for -- or even write your own. Just set up a source and a sink connector and you are ready to stream your updates. But this is not all. Once you have set up a source connector, you can use the same source to update multiple destinations ("sinks"). In our example, you could run a third cURL command to write all order-status updates to another database as well. Conclusion If your team faces any of the problems described in this article, you should give Kafka Connect a try. Kafka Connect adds a whole new set of capabilities to an existing Kafka cluster, that will make your team's life easier in the long run. And if you are currently not running a Kafka cluster, this article has shown you how to set it up within minutes. Once Kafka Connect is up and running, your team can focus on solving actual business problems. Once your team has mastered the basic workflows described in this article, you might want also want to check out Kafka Connect's ability to data on the fly, which can be used to solve a whole new set of problems. transform

Tips About Kafka Connect On Heroku You Can't Afford To Miss

If you've spent days (or even weeks?) trying to spin up a Kubernetes cluster for learning purposes or to test your application, then your worries are over. Spawned from a Kubernetes Special Interest Group, KIND is a tool that provisions a Kubernetes cluster running IN Docker. From the docs: kind is a tool for running local Kubernetes clusters using Docker container "nodes". kind is primarily designed for testing Kubernetes 1.11+, initially targeting the . conformance tests Installing KIND As it is built using go, you will need to make sure you have the latest version of installed on your machine. golang According to the k8s , is preferred. To install kind, run these commands (it takes a while): docs golang -v 1.11.5 go get -u sigs.k8s.io/kind
kind create cluster Then confirm kind cluster is available: kind get clusters Setting up kubectl Also, install the latest using or . The latest Docker has Kubernetes feature but it may come with older . Check its version by running this command: kubernetes-cli Homebrew Chocolatey kubectl kubectl version Make sure it shows or above. If you find you are running from Docker, try or reorder path environment variable. GitVersion: "v1.14.1" kubectl brew link Once and kind are ready, open bash console and run these commands: kubectl KUBECONFIG=”$(kind get kubeconfig-path)”
kubectl cluster-info export If is properly set up, some information will be shown. Now you are ready to proceed. Yay! kind Deploying first application What should we deploy on the cluster? We are going to attempt deploying Cassandra since the docs have a pretty decent walk-through on it. First of all, download and for later. Then create kustomization.yaml by running two cat commands. cassandra-service.yaml cassandra-statefulset.yaml Once those files are prepared, layout them as following: yaml k8s-wp/
  kustomization.yaml
  mysql-deployment.yaml
  wordpress-deployment.yaml Then apply them to your cluster: k8s-wp
kubectl apply -k ./ cd Validating (optional) Get the Cassandra Service. kubectl get svc cassandra The response is: NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   None         <none>        9042/TCP   45s Note that Service creation might have failed if anything else is returned. Read for common issues. Debug Services Finishing up That's really all you need to know to get started with KIND, I hope this makes your life a little easier and lets you play with Kubernetes a little bit more :) About the author - Sudip is a Solution Architect with more than 15 years of working experience, and is the founder of Javelynn . He likes sharing his knowledge by regularly writing for Hackernoon , DZone , Appfleet and many more. And while he is not doing that, he must be fishing or playing chess. Previously posted at https://appfleet.com/ .

How To Manage Local Kubernetes Testing with KIND Tool

Customise your EC2 to your deployment needs Picture the scene. You’ve got a few small, simple microservices. They all run beautifully on 200mb of ram and scale instantly. Engineering bliss. But wait… in this blue sky, a scar in the scenery, lies an omen. A storm cloud in the shape of a monolith. You need to get this beast running in your Kubernetes cluster but it’s dramatically larger than your other applications. In the words of the internet, “wut do?” Is your Microservice EC2 Instance Type enough? Your Microservices can be packed into small boxes, but your monolith needs some juice. There simply isn’t enough horsepower in a single instance to run your application as a pod. The purists are going to tell you “Well then, refactor it into microservices”. Reality isn’t often so black and white. So do you deploy everything onto big boxes? Your first port of call might be to reassess everything — drain the existing nodes and deploy onto bigger boxes. That way you don’t need to run multiple node pools. This will work! But it comes at a cost. There is a greater chance of wasted resources in your cluster. Bigger boxes mean you’re more likely to have very underutilized instances, costing you money. Okay, so can I have multiple node types? Yes you can, you lucky devil. It’s straightforward and requires only a few steps. For the sake of this tutorial, I’m going to assume you’re familiar with Kubernetes concepts like or . If you’re not, you should read through the . Additionally, I’ll assume you’re comfortable with AWS terminology like Autoscaling Group (ASG) and EC2. If you’re not, . For the sake of brevity and clarity, I won’t be including changes to IAM roles and security groups. The includes these details. pod node Kubernetes documentation have a read through the AWS documentation first official documentation Okay! The Cluster Autoscaler The first thing you’ll need to hook up is the . This has an AWS specific installation mode. It will detect when there is no free space in your EC2 instances to deploy pods, at which point it will increase the number of EC2 instances in your ASG. It works by adjusting the “desired” field in the ASG, up to a predefined maximum that you’ve set. I won’t go over the installation in detail here because . When you’re following these instructions, be sure to set it up for “Autodiscovery”. That way, when you add new ASGs, it will detect them and you won’t need to reconfigure your cluster. cluster autoscaler the documentation is pretty good Next, your Autoscaling Group Spinning up an ASG is straight forward. For the sake of example, we’ll be using Terraform. In the following code snippets, we will create an auto scaling group and a run configuration. The run configuration is the template that your ASG should use when it’s spinning up new EC2 instances. There are some key things to note here. We’re ignoring changes to . This is because the Cluster autoscaler will be using this value to scale the cluster up and down. We don’t want to reset it back to 2 every time we do a . desired_capacity terraform apply There are two highlighted tag keys. is to ensure that the cluster autoscaler can detect your Auto scaling groups. The second, isn’t mandatory but is a good idea if you’re running multiple clusters. This will prevent two clusters picking up the same ASG. k8s.io/cluster-autoscaler/enabled mandatory k8s.io/cluster-autoscaler/cluster_name resource "aws_autoscaling_group" "worker_node_asg" {desired_capacity = 2max_size         = 10min_size         = 1name_prefix      = "worker-node-"lifecycle {ignore_changes = ["desired_capacity"]}tag {key = "Name"value = "worker-node"propagate_at_launch = true}tag {key = "k8s.io/cluster-autoscaler/enabled"value = "true"propagate_at_launch = true}tag {key = "k8s.io/cluster-autoscaler/cluster_name"value = "true"propagate_at_launch = true}} And the launch configuration (template) for your ASG. Note the field highlighted in bold. instance_type resource "aws_launch_configuration" "worker_launch_configuration" {image_id             = "${data.aws_ami.worker_ami.id}"instance_type        = "t2.medium"name_prefix          = "worker-node-"security_groups      = ["${some_security_group_ID}"]create_before_destroy = true} Create this infrastructure for each of the EC2 instance types you would like to create. We’ll assume, from here on out, you’ve done it twice for two different instance types. Let’s make the Pods more picky Your pods are pretty laid back at the moment — they sigh and say “oh just put me anywhere” as they’re flung at the k8s api. We need to give them a bit of choice and we can do that with . This will allow the pod to specify the type of node it would like to run on. node selectors Find your label If you run you’ll see some EC2 instances. Look for the label, that’ll let you know what the EC2 instance type is. You can use this in your resource yaml to pick which box you want your applications to deploy to. There are other ways to go about this that don’t pollute your yaml with AWS specific stuff —as always, be mindful. A simple example yaml might look something like this: kubectl get node -o yaml beta.kubernetes.io/instance-type apiVersion: v1kind: Podmetadata:  name: nginx  labels:    env: testspec:  containers:  - name: nginx    image: nginx  nodeSelector:    beta.kubernetes.io/instance-type: t2.medium Using the flag has given this pod a preference — it’s picky. Once you run and push it to the k8s API, kubernetes will find a box with the correct instance type and deploy it. If it doesn’t exist, it will trigger the autoscaling group to create an instance for you. nodeSelector kubectl apply -f myfile.yaml And there you have it! You’ve got a cluster that supports multiple EC2 instance types. You can add as many ASGs as you like. Go and take over the world you little scamp! If you enjoyed this tutorial, I’m regularly writing and throwing other articles . about on twitter

Configuring Multiple Node Pools

Community detection is a topic that can be applied to datasets that have an inherent order among themselves. For example, redwood, birch and oak are types of trees. Fish, algae and octopus are types of underwater creatures. These entities can be grouped under a common category, and the task of doing that is called as community detection. Sometimes we have a dataset that is unstructured and we want to get some value out of it, in those cases community detection can be useful. In this article I will be talking about how to detect communities using LPA, label propagation algorithm in GraphFrames package of Spark. The code for the article can be found . here Label Propagation Algorithm Label Propagation Algorithm is a fast algorithm that detects communities in a graph. It does not require prior information about the communities, and it uses network structure to detect them. We label some initial communities and it takes it from there. LPA was proposed by Raghavan in , and it works by labelling the nodes, and propagating these labels throughout the network and forming communities based on the process of label propagation. this paper The idea is that a label will become dominant in a densely connected group of nodes, but will have trouble propagating in a sparsely connected region. When the algorithm finishes, the nodes with the same label are considered as part of the community. The algorithm works as follows: a) Every node is initialized with a unique label. b) Label are propagated through the network c) At every iteration of the propagation, each node updates its label to the one that the maximum number of its neighbours belongs to. d) LPA reaches convergence when each node has the majority label of its neighbours. e) LPA stops if either convergence or the user defined maximum number of iterations are achieved. Implementation Dataset and Processing I am using the dataset provided by , it is a dataset that contains a word, and a reference to the wikipedia url about that word. There are 10 files, 600 MB each, and i am working with one file. It contains 18 million rows. UMASS It is in the following format: URL: ftp://217.219.170.14/Computer%20Group/Faani/vaset%20fani/second/sattari/word/2007/source/s%20crt.docx MENTION vacuum tube 421 http://en.wikipedia.org/wiki/Vacuum_tube MENTION vacuum tubes 10838 http://en.wikipedia.org/wiki/Vacuum_tube MENTION electron gun 598 http://en.wikipedia.org/wiki/Electron_gun MENTION fluorescent 790 http://en.wikipedia.org/wiki/Fluorescent MENTION oscilloscope 1307 http://en.wikipedia.org/wiki/Oscilloscope MENTION computer monitor 1503 http://en.wikipedia.org/wiki/Computer_monitor MENTION computer monitors 3066 http://en.wikipedia.org/wiki/Computer_monitor MENTION radar 1657 http://en.wikipedia.org/wiki/Radar MENTION plasma screens 2162 http://en.wikipedia.org/wiki/Plasma_screen URL is the link to a particular topic, and mention contains the word that was mentioned in the article along with the Wikipedia link (also mentioned in the article). I used the following code to preprocess the data to arrive at a meaningful format: After preprocessing, i have the data in the following format: ftp://ftp.pcbsd.org/pub/handbook 9.0/handbook_en_ver9.0.html,Zfs ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,BSD_UNIX ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Acpi ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Softupdates ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Compiz ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Pcx ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Journaling_file_system ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Nextstep ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Freebsd_jail ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,Mac_address ftp://ftp.pcbsd.org/pub/handbook/9.0/handbook_en_ver9.0.html,IPv6_address Basically, in this format, I have a URL followed by each word that was mentioned in the original file. Applying LPA Now that we have the data in the required format, we can apply LPA. I use the following code to do that. The largest community is has 8500 nodes. Let us look at it’s contents: ... ... ... ... ... ... ... ... http://commons.wikimedia.org/wiki/File:Vestv%C... ... http://commons.wikimedia.org/wiki/File:Aston_M http://www.free-photos.biz/photographs/industr http://commons.wikimedia.org/wiki/File:Mercede http://commons.wikimedia.org/wiki/File:Odiorne http://commons.wikimedia.org/wiki/File:Hannove http://commons.wikimedia.org/wiki/File:Autosta http://www.free-photos.biz/photographs/food/ve http://commons.wikimedia.org/wiki/File:Smeaton http://commons.wikimedia.org/wiki/File:Mitsubi As we can see, the largest community is about free photos. Let us look at another: Philip_Neame Monsoon#Asia Arab_Jew Cave_of_Elijah Shi’a_Islam Culture_of_the_Song_Dynasty Safavid_dynasty John_Romer_(Egyptologist) Bamyan_Province Henry_Phillips_(horticulturist) Christ_Church,_Nazareth The second largest community is about dynasties or provinces, and people related to them. It has about 6000 nodes. There are some false positives as well, which means more iterations in LPA can probably give good results. Third largest community is: Hansi Vadodara_district Cox-Forbes_theory Puppis_(constellation) University_of_Alexandria historical_scholarship Villuercas Center_of_the_universe genre Velayuthampalayam and it is mostly about science. It has 4000 nodes, and by the looks of it, it can probably use more number of iterations as well. Conclusion In this post we went through what communities are, and how do we detect them. We used the algorithm Label Propagation algorithm to determine that. It works by initially labelling nodes, then iterating and propagating labels and determining the final label at convergence. We then used a dataset that contains a URL and references to wikipedia text for several mentions of words in that url, and determined communities from them. Most popular community is about stock photos, then about dynasties and then science. I hope you liked my post.

Stories, News & Mentions about CLUSTER

Categories

Trending Topics

Stories, News & Mentions about CLUSTER

Cluster’s Stories on HackerNoon

Cluster's News & Mentions

Categories

Trending Topics