EMERALDWHALE operation compromises over 15,000 cloud credentials, exploiting exposed Git and Laravel files.

EMERALDWHALE Steals 15,000+ Cloud Credentials, Stores Data in S3 Bucket

Bucket

bucket

Amazon Simple Storage Service (Amazon S3) is a scalable object storage service that allows you to store and retrieve any amount of data from anywhere on the web. Creating an S3 bucket in AWS (Amazon Web Services) is a straightforward process that involves a few simple steps. This guide will walk you through the process of creating an S3 bucket using the AWS Management Console. Step-by-Step Guide: 1. Sign in to the AWS Management Console: Navigate to AWS Management Console, and sign in using your credentials. 2. Open the S3 Console: Once logged in, find and select "S3" from the list of AWS services. Alternatively, you can click here to go directly to the S3 console. 3. Create a New Bucket: In the S3 console, click on the "Create bucket" button. This will start the process of creating a new S3 bucket. 4. Configure Bucket Properties: Bucket Name: Choose a unique name for your bucket. Bucket names must be globally unique across all of AWS. Naming Conventions: Follow AWS naming conventions for S3 bucket names. They should be DNS-compliant and unique; avoid the use of capital letters. Region: Select the AWS region where you want your bucket to be located. This choice can affect latency and compliance regulations. Configure options: You can set options like versioning, logging, and tagging for your bucket. These configurations are optional but can enhance your management and security. 5. Set Permissions: Choose who can access your bucket and the objects within it. By default, only the bucket owner has access, but you can configure more granular permissions using Access Control Lists (ACLs) and Bucket Policies. 6. Review and Create: Review all the configurations you have chosen for your bucket. Ensure that the settings align with your requirements. Click "Create bucket" to finalize and create your new S3 bucket. 7. Access Your Bucket: Once the bucket is created, you can start uploading objects to it using the AWS Management Console, AWS SDKs, or AWS CLI (Command Line Interface). ‘Object’ tab: In Amazon S3 (Simple Storage Service), the "object tab" typically refers to the list of objects (files or data) stored within a specific S3 bucket. When you navigate to a bucket in the AWS Management Console or interact with it programmatically, you can view or manage the objects stored within that bucket. This tab or section displays details such as object names, sizes, timestamps, and other metadata associated with each object. It's a fundamental part of managing and accessing data stored in S3 buckets. ‘Properties’ tab: Accessing and managing these properties from the Properties tab in the AWS Management Console allows you to configure your S3 bucket according to your specific requirements, ensuring security, compliance, performance, and cost-effectiveness for your storage needs. ‘Permissions’ tab: Managing permissions effectively ensures that your S3 buckets and objects are secure and accessible only to authorized users and applications. It’s essential to configure permissions based on the principle of least privilege, granting only the minimum permissions necessary for users and applications to perform their required tasks. To manage media files in an S3 bucket using your program while ensuring that the "Block all public access" setting remains unchecked in development but enforced in production, you would typically handle this through AWS IAM policies and possibly S3 bucket policies. In your development environment, you may want to allow broader access to facilitate testing and development. You can achieve this by defining an IAM policy that grants necessary permissions to your IAM users or roles. Here’s an example policy that grants full access to an S3 bucket and its objects. Cross-Origin Resource Sharing (CORS) is a security feature implemented by web browsers that allows a web application running at one origin (domain) to access resources from a different origin. An origin is defined by the combination of scheme (protocol), host, and port of a URL. ‘Metrics’ tab: You can configure S3 to collect and report metrics on the bucket and object-level activity, such as the number of requests, data transfer size, and storage metrics. These metrics help you monitor the performance and usage of your S3 resources. ‘Management’ tab: The Management tab in Amazon S3 consolidates these features and settings in one place, making it easier for you to optimize storage costs, manage data lifecycle, ensure compliance, and monitor usage and performance metrics for your S3 buckets and objects. ‘Access Points’ tab: Access points provide a simplified and secure way to manage access to your S3 data, especially in scenarios where you need to segregate access based on different applications, teams, or security requirements. They help simplify access management by encapsulating permissions within the access point configuration, reducing the complexity of managing access controls at the bucket level. You can start managing your media files in AWS cloud storage by uploading directly or through your program. Every file contains an object URL that can be used on your site for end users. https://guideinhackernoon.s3.amazonaws.com/sample.png Security: Always apply the principle of least privilege when configuring bucket permissions. Monitoring and Management: Utilize AWS CloudWatch to monitor your S3 bucket metrics, and set up alerts for any unusual activity. By following these steps, you can create an S3 bucket in AWS quickly and securely, ready to store and manage your data in the cloud. Amazon Simple Storage Service (Amazon S3) is a scalable object storage service that allows you to store and retrieve any amount of data from anywhere on the web. Creating an S3 bucket in AWS (Amazon Web Services) is a straightforward process that involves a few simple steps. This guide will walk you through the process of creating an S3 bucket using the AWS Management Console. Step-by-Step Guide: Step-by-Step Guide: Step-by-Step Guide: 1. Sign in to the AWS Management Console: 1. Sign in to the AWS Management Console: Navigate to AWS Management Console, and sign in using your credentials. 2. Open the S3 Console: 2. Open the S3 Console: Once logged in, find and select "S3" from the list of AWS services. Alternatively, you can click here to go directly to the S3 console. 3. Create a New Bucket: 3. Create a New Bucket: In the S3 console, click on the "Create bucket" button. This will start the process of creating a new S3 bucket. 4. Configure Bucket Properties: 4. Configure Bucket Properties: Bucket Name : Choose a unique name for your bucket. Bucket names must be globally unique across all of AWS. Bucket Name Bucket Name Naming Conventions : Follow AWS naming conventions for S3 bucket names. They should be DNS-compliant and unique; avoid the use of capital letters. Naming Conventions Region : Select the AWS region where you want your bucket to be located. This choice can affect latency and compliance regulations. Region Region Configure options: You can set options like versioning, logging, and tagging for your bucket. These configurations are optional but can enhance your management and security. 5. Set Permissions: 5. Set Permissions: Choose who can access your bucket and the objects within it. By default, only the bucket owner has access, but you can configure more granular permissions using Access Control Lists (ACLs) and Bucket Policies. 6. Review and Create: 6. Review and Create: Review all the configurations you have chosen for your bucket. Ensure that the settings align with your requirements. Click "Create bucket" to finalize and create your new S3 bucket. 7. Access Your Bucket: 7. Access Your Bucket: Once the bucket is created, you can start uploading objects to it using the AWS Management Console, AWS SDKs, or AWS CLI (Command Line Interface). ‘Object’ tab : ‘Object’ tab ‘Object’ tab In Amazon S3 (Simple Storage Service), the "object tab" typically refers to the list of objects (files or data) stored within a specific S3 bucket. When you navigate to a bucket in the AWS Management Console or interact with it programmatically, you can view or manage the objects stored within that bucket. This tab or section displays details such as object names, sizes, timestamps, and other metadata associated with each object. It's a fundamental part of managing and accessing data stored in S3 buckets. ‘Properties’ tab : Accessing and managing these properties from the Properties tab in the AWS Management Console allows you to configure your S3 bucket according to your specific requirements, ensuring security, compliance, performance, and cost-effectiveness for your storage needs. ‘Properties’ tab ‘Properties’ tab ‘Permissions’ tab : Managing permissions effectively ensures that your S3 buckets and objects are secure and accessible only to authorized users and applications. It’s essential to configure permissions based on the principle of least privilege, granting only the minimum permissions necessary for users and applications to perform their required tasks. ‘Permissions’ tab ‘Permissions’ tab To manage media files in an S3 bucket using your program while ensuring that the "Block all public access" setting remains unchecked in development but enforced in production, you would typically handle this through AWS IAM policies and possibly S3 bucket policies. In your development environment, you may want to allow broader access to facilitate testing and development. You can achieve this by defining an IAM policy that grants necessary permissions to your IAM users or roles. Here’s an example policy that grants full access to an S3 bucket and its objects. Cross-Origin Resource Sharing (CORS) is a security feature implemented by web browsers that allows a web application running at one origin (domain) to access resources from a different origin. An origin is defined by the combination of scheme (protocol), host, and port of a URL. ‘Metrics’ tab : You can configure S3 to collect and report metrics on the bucket and object-level activity, such as the number of requests, data transfer size, and storage metrics. These metrics help you monitor the performance and usage of your S3 resources. ‘Metrics’ tab ‘Metrics’ tab ‘Management’ tab : The Management tab in Amazon S3 consolidates these features and settings in one place, making it easier for you to optimize storage costs, manage data lifecycle, ensure compliance, and monitor usage and performance metrics for your S3 buckets and objects. ‘Management’ tab ‘Management’ tab ‘Access Points’ tab : Access points provide a simplified and secure way to manage access to your S3 data, especially in scenarios where you need to segregate access based on different applications, teams, or security requirements. They help simplify access management by encapsulating permissions within the access point configuration, reducing the complexity of managing access controls at the bucket level. ‘Access Points’ tab ‘Access Points’ tab You can start managing your media files in AWS cloud storage by uploading directly or through your program. Every file contains an object URL that can be used on your site for end users. https://guideinhackernoon.s3.amazonaws.com/sample.png https://guideinhackernoon.s3.amazonaws.com/sample.png Security : Always apply the principle of least privilege when configuring bucket permissions. Security Security Monitoring and Management: Utilize AWS CloudWatch to monitor your S3 bucket metrics, and set up alerts for any unusual activity. By following these steps, you can create an S3 bucket in AWS quickly and securely, ready to store and manage your data in the cloud.

A Step-by-Step Guide to Creating an S3 Bucket in AWS Cloud

The Token Bucket Algorithm for API Rate Limiting in NodeJS: A Simple Guide

The TechBeat: Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management (4/13/2024)

The TechBeat: Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management (4/11/2024)

Backing Up Weaviate with MinIO S3 Buckets to Achieve Strategic Enhancement to Data Management

ডাটা ম্যানেজমেন্টে কৌশলগত বর্ধন অর্জনের জন্য MinIO S3 বাকেটের সাহায্যে ওয়েভিয়েট ব্যাক আপ করা হচ্ছে

Sichern von Weaviate mit MinIO S3-Buckets zur strategischen Verbesserung des Datenmanagements

Realizar una copia de seguridad de Weaviate con MinIO S3 Buckets para lograr una mejora estratégica en la gestión de datos

Sauvegarde de Weaviate avec les buckets MinIO S3 pour obtenir une amélioration stratégique de la gestion des données

डेटा प्रबंधन में रणनीतिक वृद्धि हासिल करने के लिए मिनियो एस3 बकेट के साथ वेविएट का बैकअप लेना

MinIO S3 バケットを使用して Weaviate をバックアップし、データ管理の戦略的な強化を実現

데이터 관리의 전략적 향상을 위해 MinIO S3 버킷으로 Weaviate 백업

Fazendo backup do Weaviate com buckets MinIO S3 para obter aprimoramento estratégico no gerenciamento de dados

Резервное копирование Weaviate с помощью сегментов MinIO S3 для достижения стратегического улучшения управления данными

Veri Yönetiminde Stratejik İyileştirme Sağlamak için Weaviate'i MinIO S3 Kovalarıyla Yedekleme

Sao lưu Weaviate bằng nhóm MinIO S3 để đạt được cải tiến chiến lược trong quản lý dữ liệu

使用 MinIO S3 Buckets 备份 Weaviate，实现数据管理的战略增强

Rate-limiting, or limiting the speed of request processing, is an important component in modern web applications and APIs. It allows you to manage traffic and protects the system from overloading due to an excessive number of requests. One of the most effective tools for implementing rating in Java applications is the Bucket4j library. Introduction to Bucket4j Bucket4j is a Java language library that provides an API for implementing rate limiting using token buckets. The token bucket method is based on an analogy with a bucket into which tokens drip at a certain rate. Each request to the service extracts a certain number of tokens from this bucket. If there are not enough tokens in the bucket to process the request, the request is postponed or rejected. The Main Features of Bucket4j Bucket4j offers a flexible configuration of rating policies: Multiple restriction strategies: Bucket4j supports various rate limiting strategies, including a limit based on the number of requests per time interval and a bandwidth limit. Flexible configuration: Developers can customize the bucket size, token replenishment rate, and maximum request queue size. Distributed limitation: Bucket4j integrates with distributed repositories such as Hazelcast, Ignite, and Redis, which allows the use of rate-limiting in scalable distributed systems. High performance: The library is optimized to provide high performance and low latency when processing requests. An Example of Using Bucket4j Let's look at a simple example of using Bucket4j to limit the number of requests to the REST API. Suppose we need to limit the number of requests to a certain API to no more than 100 requests per minute. Adding a dependency: First, let's add Bucket4j to the project as a Maven dependency: <dependency>
    <groupId>io.github.bucket4j</groupId>
    <artifactId>bucket4j-core</artifactId>
    <version>последняя версия</version>
</dependency> Creating and configuring a bucket: Bucket bucket = Bucket4j.builder()
    .addLimit(Bandwidth.classic(100, Refill.greedy(100, Duration.ofMinutes(1))))
    .build(); Here, we create a bucket with a limit ofa 100 tokens and a replenishment rate of 100 tokens per minute. Applying the Restriction Before processing each API request, we check whether there are enough tokens in the bucket: if (bucket.tryConsume(1)) {
  // Processing the request
} else {
  // We are sending a response with an error about exceeding the request limit
} Advantages and Limitations of Bucket4j Advantages: Bucket4j allows you to fine-tune the rating policy according to the requirements of the application. Flexibility: Integration with distributed storage makes it an ideal choice for scalable applications. Scalability: Bucket4j integrates seamlessly with various Java frameworks. Easy integration: Limitations: : Bucket4j is designed exclusively for Java applications. Java Dependency In distributed systems, it is necessary to manage the state of buckets to ensure consistency. The need for state management: Conclusion Bucket4j is a powerful tool for implementing rate-limiting in Java applications. Its flexibility, performance, and support for distributed systems make it an ideal choice for modern web applications and APIs that require efficient traffic management and overload protection.

How to Use Bucket4j for Ratelimiters

Moby-Dick; or The Whale, by Herman Melville is part of HackerNoon’s Book Blog Post series. You can jump to any chapter in this book here: . Chapter 78: Cistern and Buckets [LINK TO TABLE OF LINK] CHAPTER 78. Cistern and Buckets. Nimble as a cat, Tashtego mounts aloft; and without altering his erect posture, runs straight out upon the overhanging mainyard-arm, to the part where it exactly projects over the hoisted Tun. He has carried with him a light tackle called a whip, consisting of only two parts, travelling through a single-sheaved block. Securing this block, so that it hangs down from the yard-arm, he swings one end of the rope, till it is caught and firmly held by a hand on deck. Then, hand-over-hand, down the other part, the Indian drops through the air, till dexterously he lands on the summit of the head. There—still high elevated above the rest of the company, to whom he vivaciously cries—he seems some Turkish Muezzin calling the good people to prayers from the top of a tower. A short-handled sharp spade being sent up to him, he diligently searches for the proper place to begin breaking into the Tun. In this business he proceeds very heedfully, like a treasure-hunter in some old house, sounding the walls to find where the gold is masoned in. By the time this cautious search is over, a stout iron-bound bucket, precisely like a well-bucket, has been attached to one end of the whip; while the other end, being stretched across the deck, is there held by two or three alert hands. These last now hoist the bucket within grasp of the Indian, to whom another person has reached up a very long pole. Inserting this pole into the bucket, Tashtego downward guides the bucket into the Tun, till it entirely disappears; then giving the word to the seamen at the whip, up comes the bucket again, all bubbling like a dairy-maid’s pail of new milk. Carefully lowered from its height, the full-freighted vessel is caught by an appointed hand, and quickly emptied into a large tub. Then remounting aloft, it again goes through the same round until the deep cistern will yield no more. Towards the end, Tashtego has to ram his long pole harder and harder, and deeper and deeper into the Tun, until some twenty feet of the pole have gone down. Now, the people of the Pequod had been baling some time in this way; several tubs had been filled with the fragrant sperm; when all at once a queer accident happened. Whether it was that Tashtego, that wild Indian, was so heedless and reckless as to let go for a moment his one-handed hold on the great cabled tackles suspending the head; or whether the place where he stood was so treacherous and oozy; or whether the Evil One himself would have it to fall out so, without stating his particular reasons; how it was exactly, there is no telling now; but, on a sudden, as the eightieth or ninetieth bucket came suckingly up—my God! poor Tashtego—like the twin reciprocating bucket in a veritable well, dropped head-foremost down into this great Tun of Heidelburgh, and with a horrible oily gurgling, went clean out of sight! “Man overboard!” cried Daggoo, who amid the general consternation first came to his senses. “Swing the bucket this way!” and putting one foot into it, so as the better to secure his slippery hand-hold on the whip itself, the hoisters ran him high up to the top of the head, almost before Tashtego could have reached its interior bottom. Meantime, there was a terrible tumult. Looking over the side, they saw the before lifeless head throbbing and heaving just below the surface of the sea, as if that moment seized with some momentous idea; whereas it was only the poor Indian unconsciously revealing by those struggles the perilous depth to which he had sunk. At this instant, while Daggoo, on the summit of the head, was clearing the whip—which had somehow got foul of the great cutting tackles—a sharp cracking noise was heard; and to the unspeakable horror of all, one of the two enormous hooks suspending the head tore out, and with a vast vibration the enormous mass sideways swung, till the drunk ship reeled and shook as if smitten by an iceberg. The one remaining hook, upon which the entire strain now depended, seemed every instant to be on the point of giving way; an event still more likely from the violent motions of the head. “Come down, come down!” yelled the seamen to Daggoo, but with one hand holding on to the heavy tackles, so that if the head should drop, he would still remain suspended; the negro having cleared the foul line, rammed down the bucket into the now collapsed well, meaning that the buried harpooneer should grasp it, and so be hoisted out. “In heaven’s name, man,” cried Stubb, “are you ramming home a cartridge there?—Avast! How will that help him; jamming that iron-bound bucket on top of his head? Avast, will ye!” “Stand clear of the tackle!” cried a voice like the bursting of a rocket. Almost in the same instant, with a thunder-boom, the enormous mass dropped into the sea, like Niagara’s Table-Rock into the whirlpool; the suddenly relieved hull rolled away from it, to far down her glittering copper; and all caught their breath, as half swinging—now over the sailors’ heads, and now over the water—Daggoo, through a thick mist of spray, was dimly beheld clinging to the pendulous tackles, while poor, buried-alive Tashtego was sinking utterly down to the bottom of the sea! But hardly had the blinding vapor cleared away, when a naked figure with a boarding-sword in his hand, was for one swift moment seen hovering over the bulwarks. The next, a loud splash announced that my brave Queequeg had dived to the rescue. One packed rush was made to the side, and every eye counted every ripple, as moment followed moment, and no sign of either the sinker or the diver could be seen. Some hands now jumped into a boat alongside, and pushed a little off from the ship. “Ha! ha!” cried Daggoo, all at once, from his now quiet, swinging perch overhead; and looking further off from the side, we saw an arm thrust upright from the blue waves; a sight strange to see, as an arm thrust forth from the grass over a grave. “Both! both!—it is both!”—cried Daggoo again with a joyful shout; and soon after, Queequeg was seen boldly striking out with one hand, and with the other clutching the long hair of the Indian. Drawn into the waiting boat, they were quickly brought to the deck; but Tashtego was long in coming to, and Queequeg did not look very brisk. Now, how had this noble rescue been accomplished? Why, diving after the slowly descending head, Queequeg with his keen sword had made side lunges near its bottom, so as to scuttle a large hole there; then dropping his sword, had thrust his long arm far inwards and upwards, and so hauled out poor Tash by the head. He averred, that upon first thrusting in for him, a leg was presented; but well knowing that that was not as it ought to be, and might occasion great trouble;—he had thrust back the leg, and by a dexterous heave and toss, had wrought a somerset upon the Indian; so that with the next trial, he came forth in the good old way—head foremost. As for the great head itself, that was doing as well as could be expected. And thus, through the courage and great skill in obstetrics of Queequeg, the deliverance, or rather, delivery of Tashtego, was successfully accomplished, in the teeth, too, of the most untoward and apparently hopeless impediments; which is a lesson by no means to be forgotten. Midwifery should be taught in the same course with fencing and boxing, riding and rowing. I know that this queer adventure of the Gay-Header’s will be sure to seem incredible to some landsmen, though they themselves may have either seen or heard of some one’s falling into a cistern ashore; an accident which not seldom happens, and with much less reason too than the Indian’s, considering the exceeding slipperiness of the curb of the Sperm Whale’s well. But, peradventure, it may be sagaciously urged, how is this? We thought the tissued, infiltrated head of the Sperm Whale, was the lightest and most corky part about him; and yet thou makest it sink in an element of a far greater specific gravity than itself. We have thee there. Not at all, but I have ye; for at the time poor Tash fell in, the case had been nearly emptied of its lighter contents, leaving little but the dense tendinous wall of the well—a double welded, hammered substance, as I have before said, much heavier than the sea water, and a lump of which sinks in it like lead almost. But the tendency to rapid sinking in this substance was in the present instance materially counteracted by the other parts of the head remaining undetached from it, so that it sank very slowly and deliberately indeed, affording Queequeg a fair chance for performing his agile obstetrics on the run, as you may say. Yes, it was a running delivery, so it was. Now, had Tashtego perished in that head, it had been a very precious perishing; smothered in the very whitest and daintiest of fragrant spermaceti; coffined, hearsed, and tombed in the secret inner chamber and sanctum sanctorum of the whale. Only one sweeter end can readily be recalled—the delicious death of an Ohio honey-hunter, who seeking honey in the crotch of a hollow tree, found such exceeding store of it, that leaning too far over, it sucked him in, so that he died embalmed. How many, think ye, have likewise fallen into Plato’s honey head, and sweetly perished there? About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Urbana, Illinois: Project Gutenberg. Retrieved April 2022 from Melville, Herman, 2001. Moby-Dick: or The Whale. https://www.gutenberg.org/files/2701/2701-h/2701-h.htm#link2HCH0078 This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

Moby-Dick; or The Whale: Chapter 78 - Cistern and Buckets

There has been a significant rise in the use of Decentralized storage over the past few years. One of the largest decentralized storage networks, Filecoin and Arweave, have brought about significant innovation of robust frameworks that use decentralized infrastructure to store and manage data. One of these frameworks is the 4EVERLAND bucket which allows uploads and pinning of Files to IPFS and Arweave. How does 4EVERLAND Bucket work? 4EVERLAND bucket is a decentralized object storage infrastructure that allows users to upload images, videos, and other supported files to decentralized storage networks like IPFS. By uploading a file to 4EVERLAND bucket, you get an IPFS hash which can be used to verify the data is stored in a decentralized manner. You also have an option to upload files through Arweave which offers permanent storage. 4EVERLAND Bucket can be leveraged in two ways: The dashboard is an interactive user interface accessible via a browser. The interface is simplified enough to allow a non-technical person to upload files to the bucket. A dashboard is also a great option if you find limitations in using the Buckets API. 1.  Dashboard 4EVERLAND Buckets API is compatible with Amazon S3 which is an Object storage service provided by Amazon. The set-up process for 4EVERLAND Buckets API is quite similar to that of Amazon S3 but the endpoints will be pointing to 4EVERLAND’s API. 2.  API Integration Once you upload a file, it can then be visible on 4EVERLAND’s dashboard. Some of the things you can do with 4EVERLANDs S3 API are: Fetching buckets File Upload Listing Files in the bucket Getting IPFS hash How to use 4EVERLAND’s Bucket API? 4EVERLAND's bucket API is simple to use and it can be used on Frameworks like React to deploy a single page Application or to upload images or files on the front-end. To get started. you will need to install the dependencies. For this, you need to have npm on your Dev environment. Below are the commands to install the AWS SDK: npm install @aws-sdk/client-s3 After installation, we will proceed to identify the Endpoint for our API.  4EVERLAND uses as the endpoint. We will also need to have our Auth tokens which can be obtained from your 4EVERLAND dashboard as seen in the image below. https://endpoint.4everland.co Once you have your Token, we will proceed to do an initial config of your s3 storage system. const AWS= require('aws-sdk');
const As3 = new AWS.s3({
    apiVersion: '2022-04-25',
    accessKeyID: 'SpecifyAPIKeyhere',
    secretAccessKey: 'specifySecretHere',
    endpoint: 'https://endpoint.4everland.co',
    region: 'us-north-1',
    s3ForcePathStyle: true
 }); After configuration, you can then proceed to call commands like to play with the S3 Client. ListObjects If you wish to upload a File to your bucket, you can use the putObject command and the configuration will look like this: Uploading a file to the 4EVERLAND bucket const FileUpload = {
    Bucket: 'testing',
    key: 'Home/hackathon',
    ContentType: 'image',
    Body: hackathonFile,
    ACL: 'Public',
    };
const request = s3.putObject(FileUpload);
request.on('header'), (statusCode, headers) => {
const ipfsHash = headers['4everland-hash'];
}).send(); In the code snippet above, we created a request to upload a file to my 4EVERLAND bucket which is . We also used the command putObject to upload the file to the Bucket from my local environment. testing The snipped also includes a request to get the IPFS hash of the Object. When you upload a file to 4EVERLAND bucket, 4EVERLAND inserts the IPFS hash in to the http header of the file. Once the files are uploaded, you have the option to sync the content to Arweave (AR) through 4EVERLNAD bucket. This option is not available via CLI but it can be done through the dashboard.  All you need to do is go to Buckets > Click on the bucket you wish to sync files to AR > Syc the content to AR. Once the content is synced, you will get access to the AR hash to verify the info is stored on Arweave. Fetching files from 4EVERLAND bucket Through the S3 SDK, you can also fetch buckets in which the files are stored. const listFiles = {
    Bucket: "testing",
};
s3.listObjectsV2(testing, function (err, list) {
    if (err) {
	    consolte.log("Error, check again!", err);
	 } else {
		console.log("Objects displayed correctly", list);
	}
	}); Aside from fetching files, you can also list out the Buckets in which your files are stored. Below is a simple example on how you can do it by using the command: listBuckets s3.listBuckets(function (err, list) {
    if (err) {
	    console.log("error when listing buckets", err);
	} else {
		console.log("data listed correctly", list);
	}
	}); Another alternative for listing Objects is through the . In your CLI, run to define your Access Keys and secret Key. From there you can list the objects by running: AWS CLI aws configure aws --endpoint-url https://endpoint.4everland.co s3 ls s3://testing Limitations There are a few limitations on the use of S3 which have been highlighted on the .  Some of the limitations like the length for bucket names can be mitigated by using the Buckets Dashboard which offers better flexibility. 4EVERLAND documentation There are many other use cases for 4EVERLAND buckets based on the s3 commands which are listed in the . Documentation Conclusion 4EVERLAND buckets API is a great alternative for users used to the Amazon AS3 bucket. By using the API, your Objects get stored on decentralized storage that is censorship-resistant and highly reliable.  4EVERLAND also allows you to sync your Files to Arweave which offers permanent storage of the files.

4EVERLAND's Bucket API for Decentralized Object Storage

Our team recently implemented an internal static website that allows employees to download technical reports. Since we're heavy AWS users, we naturally decided to host it on AWS S3, which provides a dedicated feature to build static websites ( ). S3 static website hosting Very quickly, however, we ran into an issue: AWS S3 does not provide any native, out-of-the-box authentication/authorization process. Because it was an internal-only website, we needed some kind of authorization mechanism to prevent non-authorized users from accessing our website and reports. We needed to find a solution to secure our static website on AWS S3. Discovering the Solution With CloudFront and Lambda@Edge We use for all Identity and User Management, so whatever solution we found had to plug-in with Okta. Okta Okta has several authentication/authorization flows, all of which require the application to perform a back-end check, such as verifying that the response/token returned by Okta is legit. So we needed to find a way to carry these checks/actions on a static website which uses a back end that we don't control. That's when we learned about , which lets you run at different stages of a request and response to and from CloudFront: AWS Lambda@Edge Lambda Functions We can trigger a Lambda Function at four different stages: When the request enters CloudFront ( ) viewer-request When the request goes out to the origin ( ) origin-request When the response is returned from the origin ( ) origin-response When the response is returned from CloudFront ( ) viewer-response We saw a solution to our original issue: that would check if the user is authorized, under two conditions: trigger a Lambda at the viewer-request stage If the user is authorized, let the request continue and return the restricted content If the user is not authorized, send an HTTP response to redirect them to a login page Implementing the Lambda@Edge function We'll cover here the key elements and main issues we faced. The complete code is available . Feel free to use it in your project! here Lambda@Edge restrictions and caveats During the development of the solution, we ran into several restrictions and caveats of Lambda@Edge. 1 – Environment variables Lambda@Edge Functions . That meant that we needed to find another way to pass data to our function. We opted for and in the Node.js code (we use Terraform to render the template when deploying the Lambda Function). cannot use environment variables SSM parameters templated parameter names 2 – Lambda package size limit For viewer events (reminder: we use the event), the Lambda package can be . One MB is pretty small considering that it includes (except of course the runtime/standard library) of your Lambda Function. viewer-request 1 MB at most all dependencies That's why instead of the original Python, because the Python package with its dependencies exceeded the 1 MB limit. we had to rewrite our Lambda in Node.js 3 – Lambda region Lambda@Edge functions can region. It's not a big issue but it means you'll need to: only be created in the us-east-1 Provision your AWS resources in that region to make things easier In Terraform, you'll need to have a separate AWS to access the bucket you want to protect if it's not in provider us-east-1 4 – Lambda role permission The IAM execution role associated with the Lambda@Edge functions in addition to the usual . See . must allow the principal service edgelambda.amazonaws.com lambda.amazonaws.com AWS - Setting IAM permissions and roles for Lambda@Edge Authorization mechanism with Okta Once we managed the above restrictions and caveats, we focused on the authorization/authorization. Okta offers several ways to authenticate and authorize users. We decided to go with , the industry-standard protocol for authorization. OAuth2 Okta implements the which adds a thin authentication layer on top of OAuth2 (that's the purpose of the ID token mentioned hereafter). Our solution would also work with pure OAuth2 with minimal modifications (removal of the ID token use in the code). Note: OpenID Connect (OIDC) standard OAuth2 itself offers several depending on the kind of application using it. In our case, we need the . authorization flows Authorization Code flow Here is the complete diagram of the Authorization Code flow taken from that shows how it works: developer.okta.com To summarize the flow: Our Lambda Function redirects the user to Okta where they will be prompted to login Okta redirects the user to our website/Lambda Function with a code Our Lambda Function checks if the code is legit and exchanges it for access and ID by sending a request to Okta tokens Depending on the result returned by Okta, we: Allow or deny access to the restricted content If access is allowed, save the access and ID tokens in a cookie to avoid having to re-authorize the user on every page Using JSON Web Tokens to store authorization result So far we have a working authorization process; however, we need to check the access/ID token on (a malicious user could forge an invalid cookie/tokens). Checking the tokens means sending a request to Okta and waiting for the response on the user visits, which and is clearly sub-optimal. every request every page slows down the loading times significantly While local verification of the Okta token is , as of this writing uses a (in-memory) cache when fetching the keys used to check the tokens. Because we're using AWS Lambda, and memory/state of the program isn't kept between invocations, the SDK is useless to us: it would still send one HTTP request to Okta for every user request, to retrieve the JWKs (JSON Web Keys). Worse, there is a limitation of 10 JWK requests per minute, which would make our solution stop working if there is more than 10 requests per minute. Note: theoretically possible the SDK provided by Okta LRU We decided to use to work around this. The initial authorization process is the same except that, instead of saving the access/ID tokens into a cookie, we create a JWT containing these tokens, and then save the JWT into a cookie. JSON Web Tokens Since the JWT is cryptographically signed: A malicious actor cannot forge one (they would need the private key used to sign them) The checking step required on every request is fast: we traded a long and I/O expensive HTTP request for a quick cryptographic check. Note on JWT expiration and renewal The JWT has a , to avoid having a valid JWT containing expired or revoked access/ID tokens. Another option would be to check the access/ID tokens regularly and revoke the associated JWT if needed, but then we would need a revocation mechanism, which would makes things more complex. pre-defined expiration time which should be reasonably short Finally, as suggested above, the tokens provided by Okta have an expiration time. It is possible (so the user doesn't have to re-login when the tokens expire) but we didn't implement that. to transparently renew them using a refresh token Conclusion While adding OAuth2 authentication to an S3 static bucket with Okta (or any other OAuth2 provider) is possible in an AWS-integrated and secure manner, it's certainly not straightforward. It requires writing a middleware between AWS and the OAuth2 provider (Okta in our case) using Lambda@Edge. We had to do the following ourselves: Validate the user authentication Remember the user authentication Refresh the user authentication (not implemented in our solution) Revoke the user authentication (TTL is implemented, but revocation before the end of the TTL is not) Finally, a bunch of AWS resources must be created to glue everything together and make it work. All this is worth the effort, because it works and our website is now more secure. You can find the code of the Lambda@Edge as well as the infrastructure (Terraform) here: https://github.com/GuiTeK/aws-s3-oauth2-okta Author: Guillaume Truchot, Site Reliability Engineer at Algolia, GitHub

How to Add OAuth2 Authentication to an S3 Static Bucket With Okta

Managing Django Media & Static Files on Heroku with Bucketeer

Stories, News & Mentions about BUCKET

Categories

Trending Topics

Stories, News & Mentions about BUCKET

Bucket’s Stories on HackerNoon

Bucket's News & Mentions

Categories

Trending Topics