We recently compiled a list of the Worst Cloud Stocks To Buy According to Short Sellers. In this article, we will look at where MongoDB, Inc. (NASDAQ:MDB) stands against the worst cloud stocks to buy according to short sellers. Cloud computing refers to the use of remote servers, typically ...

Is MongoDB, Inc. (MDB) the Worst Cloud Stock To Buy According to Short Sellers?

MongoDB occupies a unique niche in database market, offering document-oriented approach suited for handling vast amounts of data. Find out why MDB stock is a Buy.

MongoDB: This Growing Company Tends To Surprise The Market ...

MongoDB, Inc. is an American software company that develops and provides commercial support for the source-available database MongoDB, a NoSQL database that stores data in JSON-like documents with flexible schemas.

MongoDB Inc. - Wikipedia

The American software company 10gen began developing MongoDB in 2007 as a component of a planned platform-as-a-service product. In 2009, the company shifted to an open-source development model and began offering commercial support and other services. In 2013, 10gen changed its name to MongoDB Inc.

MongoDB - Wikipedia

Needham &amp; Company LLC reaffirmed a &quot;buy&quot; rating and set a $495.00 price objective on shares of MongoDB in a research report on Wednesday.

Needham & Company LLC Reaffirms "Buy" Rating for MongoDB (NASDAQ:MDB) ...

Anyone from MongoDB company here to refer me to a VP customer success role based in London or Germany ? ... Digite, Inc. I am Data Scientist with 5+ years of experience and I am from India. Currently I am working with the Equity research team of a global Bank. I am looking for remote opportunities from Europe, US, APAC and UK. Skills: Machine Learning, Problem formulation, Statical Analysis, NLP, Deep Learning Python, Pytorch, Pyspark, Keras, SQL, MongoDB ...

Working at MongoDB | Glassdoor

Fair Value is the appropriate price ... you might expect after purchasing shares in the company and holding them over the default time horizon of 5 years, based on the EPS growth rate that we have projected. ... MongoDB, Inc....

MongoDB, Inc. (MDB) Stock Price, News, Quote & History - Yahoo Finance

MongoDB | 703,916 followers on LinkedIn. A developer data platform for developers to do their best work. #LoveYourDevelopers | Headquartered in New York, MongoDB&#x27;s mission is to empower innovators to create, transform, and disrupt industries by unleashing the power of software and data.

MongoDB | LinkedIn

M&amp;A solutions provider Datasite leverages MongoDB Atlas to deliver a faster, more seamless experience to its users.Read customer story → · How Finnish delivery company uses MongoDB to scale and support new and unique customer requirements.Read customer story →

Our Customers | MongoDB

One of our core company values at MongoDB is “Embrace the Power of Differences” which means that we commit to creating a culture of inclusivity with employees from different backgrounds and experiences. We value a diverse workforce as a way to broaden our perspectives, foster innovation, ...

Life at MongoDB | MongoDB Blog

MongoDB

mongodb

MongoDB is a document DB that comes with great flexibility and scaling. It is built using a scale out architecture where more nodes can be added to meet increasing demand. Having said that, setting it on multiple regions comes with a challenge to keep it available and consistent when regions go down. As MongoDB requires majority votes to pick Primary, a bad setup can leave it in read only mode. When I first worked on a project requiring a MongoDB deployment across multiple active-active regions, I thought it would be straightforward. But when I started going over the available options, I realized that knowledge of MongoDB’s election process and quorum requirements is important before choosing the best approach. In this article, I’m sharing my discoveries and the methods I investigated. The Quorum Problem in Multi-Region MongoDB Deployments MongoDB’s replica sets keep things running smoothly by using an election process that requires more than half of the voting members to pick a primary. If a primary cannot collect a majority of votes, it steps down, and no new primary can be elected, leaving the set read-only (all members become secondaries). This mechanism prevents split-brain scenarios with multiple primaries. For example, in a 4-member set split 2–2 by a network partition, neither side reaches three votes (the majority of 4), resulting in no primary until the partition heals. Even-numbered clusters are not ideal because a perfect split can block write availability. The election protocol will re-run elections if a tie occurs, but repeated ties delay failover. The best practice is to use an odd number of voting nodes to eliminate this risk. Consider a Deployment with: 3 nodes in East (voting members)


3 nodes in West (voting members)


1 additional node in North (tie-breaker node) 3 nodes in East (voting members) 3 nodes in East (voting members) 3 nodes in West (voting members) 3 nodes in West (voting members) 1 additional node in North (tie-breaker node) 1 additional node in North (tie-breaker node) If an entire region goes down, the remaining 3 nodes alone do not have a majority (they need 4 votes out of 7). A 7th node in a neutral region (North) act as a tie-breaker to maintain quorum. This node can either be an arbiter (vote-only) or a data-bearing node. Let’s examine the trade-offs. Approach 1: Setup with an Arbiter When setting up a MongoDB replica set, it is important to know how write concerns work. Especially when using an arbiter. In a three node configuration, there is one primary and one secondary, along with an arbiter. In this setup, the majority is equal to the number of data-bearing voting members. If you set the write concern to majority, it can lead to issues. If a data-bearing member goes down, writes might time out or never get acknowledged. To avoid this, it's generally better to use another data-bearing voting member instead of an arbiter. Arbiters don’t store any data, so they don’t have an oplog. This means they can’t track the latest operations and compare them like other nodes. During elections, this can sometimes cause arbiter voting for a secondary with lower priority, which might not be ideal. In larger setups, like three nodes in one, three in another, and a secondary in a third location. This kind of setup helps with redundancy and failover but also introduces considerations like latency and data consistency. 3 nodes in West (voting)
3 nodes in East (voting)
1 arbiter in North (voting, no data) 3 nodes in West (voting) 3 nodes in East (voting) 1 arbiter in North (voting, no data) Why It Works If West fails, the 3 East nodes and arbiter can elect a new primary.
If East fails, the 3 West nodes and arbiter can elect a new primary.
Requires no extra storage since the arbiter doesn’t hold data. If West fails, the 3 East nodes and arbiter can elect a new primary. If East fails, the 3 West nodes and arbiter can elect a new primary. Requires no extra storage since the arbiter doesn’t hold data. Downsides Arbiters don’t acknowledge writes, meaning that if a data-bearing secondary is also down, writes requiring majority can fail.


MongoDB discourages arbiters in production, as they add risk without redundancy. Arbiters don’t acknowledge writes, meaning that if a data-bearing secondary is also down, writes requiring majority can fail. Arbiters don’t acknowledge writes, meaning that if a data-bearing secondary is also down, writes requiring majority can fail. MongoDB discourages arbiters in production, as they add risk without redundancy. MongoDB discourages arbiters in production, as they add risk without redundancy. Best for: When cost constraints prevent adding a full data-bearing node, but a third vote is needed for quorum. Best for: Approach 2: 3-3-1 Setup with a Data-Bearing Tie-Breaker Node 3 nodes in West (voting)
3 nodes in East (voting)
1 data-bearing secondary in North 3 nodes in West (voting) 3 nodes in East (voting) 1 data-bearing secondary in North Why It Works If an entire region fails, the tie-breaker in third region can help elect majority.
The extra node stores data, reducing the risk of data loss.
Unlike an arbiter, this node can be queried.
MongoDB recommends this setup over using an arbiter. If an entire region fails, the tie-breaker in third region can help elect majority. The extra node stores data, reducing the risk of data loss. Unlike an arbiter, this node can be queried. MongoDB recommends this setup over using an arbiter. Downsides Higher storage and replication cost than an arbiter. Higher storage and replication cost than an arbiter. Higher storage and replication cost than an arbiter. Best for: High availability where an extra backup and read scaling are required. Best for: While arbiters can help with elections, they don’t contribute to data durability. Having odd number of data-bearing voting members is safer for maintaining a healthy and reliable replica set. Alternative Approaches Hidden Node Instead of a Tie-Breaker A hidden node is a data-bearing secondary that doesn’t vote or get queried but provides a full backup. It acts as a recovery option and can be manually promoted if needed but it does not participate in automatic elections. Sharded Cluster with Region-Aware Primaries For true active-active writes, each region can be a primary for different subsets of data using sharded clusters with zone sharding. This eliminates cross-region replication lag. This approach requires application-level changes and increased complexity. Clarifying Active-Active in MongoDB MongoDB does not support multi-primary replication in a replica set. Only one primary exists at a time, and secondaries replicate from it. If a region wants its own primary, it must be configured via sharded clusters. What to Remember: MongoDB replica sets require a majority for elections.
Only one primary exists at a time.
Arbiters should only be used as a last resort.
A two-region deployment must include a third vote in a separate location.
For true active-active, use sharding. MongoDB replica sets require a majority for elections. Only one primary exists at a time. Arbiters should only be used as a last resort. A two-region deployment must include a third vote in a separate location. For true active-active, use sharding. Final Thoughts If you can afford it, use a data-bearing node instead of an arbiter to maintain high availability and prevent write inconsistencies. Understanding above options helps in designing a resilient multi-region MongoDB deployment. References: https://www.mongodb.com/docs/manual/replication/ https://www.mongodb.com/docs/manual/replication/ https://www.mongodb.com/docs/manual/replication/ https://www.mongodb.com/docs/manual/sharding/ https://www.mongodb.com/docs/manual/sharding/ https://www.mongodb.com/docs/manual/sharding/ https://www.mongodb.com/docs/manual/core/replica-set-elections/ https://www.mongodb.com/docs/manual/core/replica-set-elections/ https://www.mongodb.com/docs/manual/core/replica-set-elections/ https://www.mongodb.com/docs/manual/core/replica-set-secondary/ https://www.mongodb.com/docs/manual/core/replica-set-secondary/ https://www.mongodb.com/docs/manual/core/replica-set-secondary/ https://www.mongodb.com/docs/manual/reference/write-concern/#calculating-majority-for-write-concern https://www.mongodb.com/docs/manual/reference/write-concern/#calculating-majority-for-write-concern https://www.mongodb.com/docs/manual/reference/write-concern/#calculating-majority-for-write-concern

How to Stop Your MongoDB Cluster from Going Read-Only at the Worst Possible Time

This text is a practical guide to performing basic operations in MongoDB, one of the most popular NoSQL databases for storing and managing data. It focuses on essential topics like CRUD operations (Create, Read, Update, Delete), customizing queries with sorting, pagination, and projections, and more advanced techniques such as the Aggregation Framework. In addition to these, this guide touches on: Using query operators for filtering data (e.g., $eq, $gt, $in), working with nested fields, utilizing logical operators ($and, $or, $not), managing arrays with operators like $push and $unwind, and even performing joins between collections with $lookup. Whether you’re a beginner looking to get started or an experienced user aiming to brush up on key concepts, this guide provides a clear and concise overview of MongoDB's most practical and widely used features. 1. CRUD Operations (Create, Read, Update, Delete) 1.1 Create - Inserting Data insertOne(): Insert a single document.
db.collection.insertOne({ name: "John", age: 25 });



insertMany(): Insert multiple documents.
db.collection.insertMany([{ name: "Maria" }, { name: "Peter" }]); 1.2 Read - Querying Data find(): Retrieve multiple documents.
db.collection.find({ age: { $gte: 18 } });



findOne(): Retrieve a single document.
db.collection.findOne({ name: "John" }); Query Operators $eq: Equal to.
$ne: Not equal to.
$gt, $lt, $gte, $lte: Numeric comparisons.
$in, $nin: Check if a value is inside/outside a list.
$exists: Check if a field exists. 1.3 Update - Updating Data updateOne(): Update a single document.
db.collection.updateOne({ name: "John" }, { $set: { age: 30 } });



updateMany(): Update multiple documents.
db.collection.updateMany({ age: { $lt: 18 } }, { $inc: { age: 1 } }); Update Operators $set: Modify fields.
$inc: Increment numeric values.
$push, $pull: Add/remove elements from arrays. 1.4 Delete - Deleting Data deleteOne(): Delete a single document.
db.collection.deleteOne({ name: "Peter" });



deleteMany(): Delete multiple documents.
db.collection.deleteMany({ age: { $lt: 18 } }); 2. Customizing Queries 2.1 Field Projection Return specific fields:
db.collection.find({}, { name: 1, _id: 0 }); 2.2 Sorting and Pagination Sorting:
db.collection.find().sort({ age: 1 });



Pagination:
db.collection.find().skip(5).limit(10); 2.3 Regular Expressions Partial search:
db.collection.find({ name: /john/i }); 2.4 Querying Nested Fields Query within subfields:
db.collection.find({ "address.city": "New York" }); 2.5 Logical Operators $and:
db.collection.find({ $and: [{ age: { $gte: 18 } }, { city: "Los Angeles" }] });



$or:
db.collection.find({ $or: [{ city: "New York" }, { city: "San Francisco" }] }); 3. Aggregation Framework 3.1 Aggregation Pipeline A sequence of stages to transform data:
db.collection.aggregate([
  { $match: { age: { $gte: 18 } } },
  { $group: { _id: "$city", total: { $sum: 1 } } }
]); 3.2 Key Stages 3.2.1 $match (Filtering Data) Filter documents in the pipeline:
{ $match: { age: { $gte: 18 } } } 3.2.2 $project (Data Transformation) Create or modify fields:
{ $project: { fullName: { $concat: ["$firstName", " ", "$lastName"] } } } 3.2.3 $group (Data Grouping) Group by a field and perform calculations:
{ $group: { _id: "$city", total: { $sum: 1 } } } 3.2.4 Sorting and Limiting Sorting:
{ $sort: { total: -1 } }



Limiting:
{ $limit: 10 } 3.2.5 Array Manipulation $unwind: Expand arrays into individual documents:
{ $unwind: "$hobbies" } 3.2.6 $lookup (Relationships) Perform "joins" between collections:
{
  $lookup: {
    from: "orders",
    localField: "userId",
    foreignField: "userId",
    as: "userOrders"
  }
} 3.2.7 $out (Export Results) Export results to a new collection:
{ $out: "newCollection" } This text is a practical guide to performing basic operations in MongoDB, one of the most popular NoSQL databases for storing and managing data. It focuses on essential topics like CRUD operations (Create, Read, Update, Delete), customizing queries with sorting, pagination, and projections, and more advanced techniques such as the Aggregation Framework. In addition to these, this guide touches on: Using query operators for filtering data (e.g., $eq, $gt, $in), working with nested fields, utilizing logical operators ($and, $or, $not), managing arrays with operators like $push and $unwind, and even performing joins between collections with $lookup. Whether you’re a beginner looking to get started or an experienced user aiming to brush up on key concepts, this guide provides a clear and concise overview of MongoDB's most practical and widely used features. 1. CRUD Operations (Create, Read, Update, Delete) 1. CRUD Operations (Create, Read, Update, Delete) 1.1 Create - Inserting Data 1.1 Create - Inserting Data insertOne(): Insert a single document.
db.collection.insertOne({ name: "John", age: 25 }); insertMany(): Insert multiple documents.
db.collection.insertMany([{ name: "Maria" }, { name: "Peter" }]); insertOne(): Insert a single document. db.collection.insertOne({ name: "John", age: 25 }); insertOne() : Insert a single document. insertOne() db.collection.insertOne({ name: "John", age: 25 }); db.collection.insertOne({ name: "John", age: 25 }); insertMany(): Insert multiple documents. db.collection.insertMany([{ name: "Maria" }, { name: "Peter" }]); insertMany() : Insert multiple documents. insertMany() db.collection.insertMany([{ name: "Maria" }, { name: "Peter" }]); db.collection.insertMany([{ name: "Maria" }, { name: "Peter" }]); 1.2 Read - Querying Data 1.2 Read - Querying Data find(): Retrieve multiple documents.
db.collection.find({ age: { $gte: 18 } }); findOne(): Retrieve a single document.
db.collection.findOne({ name: "John" }); find(): Retrieve multiple documents. db.collection.find({ age: { $gte: 18 } }); find() : Retrieve multiple documents. find() db.collection.find({ age: { $gte: 18 } }); db.collection.find({ age: { $gte: 18 } }); findOne(): Retrieve a single document. db.collection.findOne({ name: "John" }); findOne() : Retrieve a single document. findOne() db.collection.findOne({ name: "John" }); db.collection.findOne({ name: "John" }); Query Operators Query Operators $eq: Equal to. $ne: Not equal to. $gt, $lt, $gte, $lte: Numeric comparisons. $in, $nin: Check if a value is inside/outside a list. $exists: Check if a field exists. $eq : Equal to. $eq $ne : Not equal to. $ne $gt , $lt , $gte , $lte : Numeric comparisons. $gt $lt $gte $lte $in , $nin : Check if a value is inside/outside a list. $in $nin $exists : Check if a field exists. $exists 1.3 Update - Updating Data 1.3 Update - Updating Data updateOne(): Update a single document.
db.collection.updateOne({ name: "John" }, { $set: { age: 30 } }); updateMany(): Update multiple documents.
db.collection.updateMany({ age: { $lt: 18 } }, { $inc: { age: 1 } }); updateOne(): Update a single document. db.collection.updateOne({ name: "John" }, { $set: { age: 30 } }); updateOne() : Update a single document. updateOne() db.collection.updateOne({ name: "John" }, { $set: { age: 30 } }); db.collection.updateOne({ name: "John" }, { $set: { age: 30 } }); updateMany(): Update multiple documents. db.collection.updateMany({ age: { $lt: 18 } }, { $inc: { age: 1 } }); updateMany() : Update multiple documents. updateMany() db.collection.updateMany({ age: { $lt: 18 } }, { $inc: { age: 1 } }); db.collection.updateMany({ age: { $lt: 18 } }, { $inc: { age: 1 } }); Update Operators Update Operators $set: Modify fields. $inc: Increment numeric values. $push, $pull: Add/remove elements from arrays. $set : Modify fields. $set $inc : Increment numeric values. $inc $push , $pull : Add/remove elements from arrays. $push $pull 1.4 Delete - Deleting Data 1.4 Delete - Deleting Data deleteOne(): Delete a single document.
db.collection.deleteOne({ name: "Peter" }); deleteMany(): Delete multiple documents.
db.collection.deleteMany({ age: { $lt: 18 } }); deleteOne(): Delete a single document. db.collection.deleteOne({ name: "Peter" }); deleteOne() : Delete a single document. deleteOne() db.collection.deleteOne({ name: "Peter" }); db.collection.deleteOne({ name: "Peter" }); deleteMany(): Delete multiple documents. db.collection.deleteMany({ age: { $lt: 18 } }); deleteMany() : Delete multiple documents. deleteMany() db.collection.deleteMany({ age: { $lt: 18 } }); db.collection.deleteMany({ age: { $lt: 18 } }); 2. Customizing Queries 2. Customizing Queries 2.1 Field Projection 2.1 Field Projection Return specific fields:
db.collection.find({}, { name: 1, _id: 0 }); Return specific fields: db.collection.find({}, { name: 1, _id: 0 }); Return specific fields: db.collection.find({}, { name: 1, _id: 0 }); db.collection.find({}, { name: 1, _id: 0 }); 2.2 Sorting and Pagination 2.2 Sorting and Pagination Sorting:
db.collection.find().sort({ age: 1 }); Pagination:
db.collection.find().skip(5).limit(10); Sorting: db.collection.find().sort({ age: 1 }); Sorting : Sorting db.collection.find().sort({ age: 1 }); db.collection.find().sort({ age: 1 }); Pagination: db.collection.find().skip(5).limit(10); Pagination : Pagination db.collection.find().skip(5).limit(10); db.collection.find().skip(5).limit(10); 2.3 Regular Expressions 2.3 Regular Expressions Partial search:
db.collection.find({ name: /john/i }); Partial search: db.collection.find({ name: /john/i }); Partial search: db.collection.find({ name: /john/i }); db.collection.find({ name: /john/i }); 2.4 Querying Nested Fields 2.4 Querying Nested Fields Query within subfields:
db.collection.find({ "address.city": "New York" }); Query within subfields: db.collection.find({ "address.city": "New York" }); Query within subfields: db.collection.find({ "address.city": "New York" }); db.collection.find({ "address.city": "New York" }); 2.5 Logical Operators 2.5 Logical Operators $and:
db.collection.find({ $and: [{ age: { $gte: 18 } }, { city: "Los Angeles" }] }); $or:
db.collection.find({ $or: [{ city: "New York" }, { city: "San Francisco" }] }); $and: db.collection.find({ $and: [{ age: { $gte: 18 } }, { city: "Los Angeles" }] }); $and : $and db.collection.find({ $and: [{ age: { $gte: 18 } }, { city: "Los Angeles" }] }); db.collection.find({ $and: [{ age: { $gte: 18 } }, { city: "Los Angeles" }] }); $or: db.collection.find({ $or: [{ city: "New York" }, { city: "San Francisco" }] }); $or : $or db.collection.find({ $or: [{ city: "New York" }, { city: "San Francisco" }] }); db.collection.find({ $or: [{ city: "New York" }, { city: "San Francisco" }] }); 3. Aggregation Framework 3. Aggregation Framework 3.1 Aggregation Pipeline 3.1 Aggregation Pipeline A sequence of stages to transform data:
db.collection.aggregate([
  { $match: { age: { $gte: 18 } } },
  { $group: { _id: "$city", total: { $sum: 1 } } }
]); A sequence of stages to transform data: db.collection.aggregate([
  { $match: { age: { $gte: 18 } } },
  { $group: { _id: "$city", total: { $sum: 1 } } }
]); A sequence of stages to transform data: db.collection.aggregate([
  { $match: { age: { $gte: 18 } } },
  { $group: { _id: "$city", total: { $sum: 1 } } }
]); db.collection.aggregate([
  { $match: { age: { $gte: 18 } } },
  { $group: { _id: "$city", total: { $sum: 1 } } }
]); 3.2 Key Stages 3.2 Key Stages 3.2.1 $match (Filtering Data) 3.2.1 $match (Filtering Data) Filter documents in the pipeline:
{ $match: { age: { $gte: 18 } } } Filter documents in the pipeline: { $match: { age: { $gte: 18 } } } Filter documents in the pipeline: { $match: { age: { $gte: 18 } } } { $match: { age: { $gte: 18 } } } 3.2.2 $project (Data Transformation) 3.2.2 $project (Data Transformation) Create or modify fields:
{ $project: { fullName: { $concat: ["$firstName", " ", "$lastName"] } } } Create or modify fields: { $project: { fullName: { $concat: ["$firstName", " ", "$lastName"] } } } Create or modify fields: { $project: { fullName: { $concat: ["$firstName", " ", "$lastName"] } } } { $project: { fullName: { $concat: ["$firstName", " ", "$lastName"] } } } 3.2.3 $group (Data Grouping) 3.2.3 $group (Data Grouping) Group by a field and perform calculations:
{ $group: { _id: "$city", total: { $sum: 1 } } } Group by a field and perform calculations: { $group: { _id: "$city", total: { $sum: 1 } } } Group by a field and perform calculations: { $group: { _id: "$city", total: { $sum: 1 } } } { $group: { _id: "$city", total: { $sum: 1 } } } 3.2.4 Sorting and Limiting 3.2.4 Sorting and Limiting Sorting:
{ $sort: { total: -1 } } Limiting:
{ $limit: 10 } Sorting: { $sort: { total: -1 } } Sorting : Sorting { $sort: { total: -1 } } { $sort: { total: -1 } } Limiting: { $limit: 10 } Limiting : Limiting { $limit: 10 } { $limit: 10 } 3.2.5 Array Manipulation 3.2.5 Array Manipulation $unwind: Expand arrays into individual documents:
{ $unwind: "$hobbies" } $unwind: Expand arrays into individual documents: { $unwind: "$hobbies" } $unwind : Expand arrays into individual documents: $unwind { $unwind: "$hobbies" } { $unwind: "$hobbies" } 3.2.6 $lookup (Relationships) 3.2.6 $lookup (Relationships) Perform "joins" between collections:
{
  $lookup: {
    from: "orders",
    localField: "userId",
    foreignField: "userId",
    as: "userOrders"
  }
} Perform "joins" between collections: {
  $lookup: {
    from: "orders",
    localField: "userId",
    foreignField: "userId",
    as: "userOrders"
  }
} Perform "joins" between collections: {
  $lookup: {
    from: "orders",
    localField: "userId",
    foreignField: "userId",
    as: "userOrders"
  }
} {
  $lookup: {
    from: "orders",
    localField: "userId",
    foreignField: "userId",
    as: "userOrders"
  }
} 3.2.7 $out (Export Results) 3.2.7 $out (Export Results) Export results to a new collection:
{ $out: "newCollection" } Export results to a new collection: { $out: "newCollection" } Export results to a new collection: { $out: "newCollection" } { $out: "newCollection" }

MongoDB - A Practical Guide for Beginners and Experts Alike

We are back with another Company of the Week feature! Every week, we share an awesome tech brand from our tech company database, making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. This week, we are proud to present MongoDB! You may have caught a mention of MongoDB during the pilot of the hit TV show “Silicon Valley,” but the NoSQL database product is not only popular with budding startups. In fact, you’d be amazed to learn that MongoDB powers some of the largest enterprises in the world, including prior HackerNoon company of the week Bosch which uses MongoDB Atlas when dealing with a vast amount of data in its workflows. Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon! MongoDB <> HackerNoon Targeted Ads For all you programmers out there, you may have seen MongoDB appear across our website recently, especially if you were browsing any coding or programming-related tag on HackerNoon. Well, that’s because MongoDB and HackerNoon partnered together to highlight some amazing features from MongoDB to help you develop your own RAG or cloud document database for artificial intelligence. How’s that for cool!? In effect, you could use the same technology that is being utilized by the likes of Bosch, Novo Nordisk, Wells Fargo, and even Toyota, to create or innovate on your products! Learn How You Can Advertise to Your Specific Niche on HackerNoon Meet MongoDB: #FunFacts The founders of MongoDB — Dwight Merriman, Eliot Horowitz, and Kevin Ryan — previously worked with the online advertising company Doubleclick which was acquired by Google in the late 2000s for a whopping $3.1 billion. Using their knowledge of the limitations of relational databases, all three got together the same year Doubleclick was sold to found MongoDB whose mission is to empower innovators to create, transform, and disrupt industries by unleashing the power of software and data. Today, MongoDB’s Atlas is one of the most loved vector databases in the world, and the company, which is listed on Nasdaq, consistently ranks as one of the best places to work in. MongoDB is also a leader in cloud database management systems, as ranked by Gartner. How’s that for interesting 😏 Share Your Company's Story via HackerNoon That's all this week, folks! Stay Creative, Stay Iconic. The HackerNoon Team We are back with another Company of the Week feature! Every week, we share an awesome tech brand from our tech company database , making their evergreen mark on the internet. This unique HackerNoon database ranks S&P 500 companies and top startups of the year alike. Company of the Week Company of the Week tech company database tech company database This week, we are proud to present MongoDB ! You may have caught a mention of MongoDB during the pilot of the hit TV show “Silicon Valley,” but the NoSQL database product is not only popular with budding startups. In fact, you’d be amazed to learn that MongoDB powers some of the largest enterprises in the world, including prior HackerNoon company of the week Bosch which uses MongoDB Atlas when dealing with a vast amount of data in its workflows. MongoDB powers prior Bosch Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon! Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon ! Want to be featured on HackerNoon’s Company of the Week? Request Your Tech Company Page on HackerNoon MongoDB <> HackerNoon Targeted Ads For all you programmers out there, you may have seen MongoDB appear across our website recently, especially if you were browsing any coding or programming-related tag on HackerNoon. Well, that’s because MongoDB and HackerNoon partnered together to highlight some amazing features from MongoDB to help you develop your own RAG or cloud document database for artificial intelligence. How’s that for cool!? RAG cloud document database cool!? cool!? In effect, you could use the same technology that is being utilized by the likes of Bosch, Novo Nordisk, Wells Fargo, and even Toyota, to create or innovate on your products! Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Learn How You Can Advertise to Your Specific Niche on HackerNoon Meet MongoDB: #FunFacts The founders of MongoDB — Dwight Merriman, Eliot Horowitz, and Kevin Ryan — previously worked with the online advertising company Doubleclick which was acquired by Google in the late 2000s for a whopping $3.1 billion. Using their knowledge of the limitations of relational databases, all three got together the same year Doubleclick was sold to found MongoDB whose mission is to empower innovators to create, transform, and disrupt industries by unleashing the power of software and data. mission Today, MongoDB’s Atlas is one of the most loved vector databases in the world, and the company, which is listed on Nasdaq, consistently ranks as one of the best places to work in . MongoDB is also a leader in cloud database management systems, as ranked by Gartner. most loved vector databases best places to work in leader How’s that for interesting 😏 Share Your Company's Story via HackerNoon Share Your Company's Story via HackerNoon Share Your Company's Story via HackerNoon Share Your Company's Story via HackerNoon That's all this week, folks! Stay Creative, Stay Iconic. The HackerNoon Team

Meet MongoDB: HackerNoon Company of the Week

Ontmoet MongoDB: HackerNoon Company of the Week

Знаёмцеся, MongoDB: Кампанія тыдня HackerNoon

Conozca MongoDB: la empresa de la semana de HackerNoon

MongoDB से मिलिए: HackerNoon कंपनी ऑफ द वीक

Rankontre MongoDB: HackerNoon Konpayi Semèn nan

Ismerje meg a MongoDB-t: A hét HackerNoon cége

MongoDB のご紹介: HackerNoon の今週の企業

გაიცანით MongoDB: კვირის HackerNoon კომპანია

ជួបជាមួយ MongoDB: ក្រុមហ៊ុន HackerNoon ប្រចាំសប្តាហ៍

Susipažinkite su MongoDB: „HackerNoon“ savaitės kompanija

Hihaona amin'ny MongoDB: HackerNoon Company of the Week

Kopana le MongoDB: Khamphani ya HackerNoon ya Beke

Hello JavaScript Enthusiasts!Welcome to this week’s edition of "This Week in JavaScript"!This week, we have exciting updates that will make your development workflow smoother and faster. From VoidZero’s unified JavaScript toolchain to ESLint’s new features, and even MongoDB’s performance boost, there's a lot to unpack! VoidZero Inc. Raises $4.6M to Unify JavaScript Tooling VoidZero has raised $4.6M in seed funding, led by Accel, to develop a unified, high-performance toolchain for the JavaScript ecosystem. Why is this Important? JavaScript tooling is currently fragmented, with developers relying on various third-party dependencies. VoidZero aims to streamline the development process by creating a single toolchain that handles everything from bundling to testing. Key Highlights: Unified Framework: Same AST and resolver for all tasks, reducing redundant parsing.
Performance-Focused: Built in a compile-to-native language with support for parallelization and optimized JS plugins.
Modular Components: Easily customizable building blocks for advanced use cases.
Cross-Environment Support: Works across all JavaScript environments without runtime limitations. VoidZero’s new bundler, Rolldown, will be in alpha later this year and set to integrate with Vite soon. ESLint now Lints JSON and Markdown JSON and Markdown files can now be linted directly! This is part of ESLint’s push to become a more general-purpose linter, expanding beyond JavaScript. What’s New? JSON Support: With the new @eslint/json plugin, you can lint JSON, JSONC, and JSON5 files. To get started, just install the plugin and update your ESLint config to include the JSON rules. Markdown Support: The @eslint/markdown plugin introduces linting for CommonMark and GitHub-Flavored Markdown. Just install and configure it in your project to start catching issues in your markdown files. This update opens up more possibilities for linting different file types in web projects, bringing everything under the same ESLint umbrella. Tauri 2.0 is Here Tauri 2.0’s stable release is now live, bringing key improvements and mobile support for iOS and Android. This Rust-based framework lets you build lightweight, cross-platform desktop and mobile applications simply by using HTML, JavaScript, and CSS. It does all this and makes the file size smaller and makes performance faster. With Tauri 2.0, you get: Unified Codebase: One UI codebase for desktop (Windows, macOS, Linux) and mobile (iOS, Android).
Hot-Module Replacement (HMR): Seamlessly update your app's frontend without rebuilding.
Mobile Support: Use Swift or Kotlin to build mobile-specific functionalities while sharing Rust logic.
Advanced Plugins: Extend your app's capabilities with a more refined plugin system, simplifying development. Tauri continues to evolve, offering developers a highly efficient and flexible way to build for all platforms. New Tools and Releases Tabulator 6.3: Simplify managing complex data tables in web apps with powerful editing, filtering, and sorting features.
MongoDB 8.0: Enjoy a 30% performance boost and enhanced security with Queryable Encryption and better scaling.
Eleventy 3.0: The latest release of this static site generator makes creating fast, efficient websites easier than ever.
µExpress: Speed up your Node.js apps with this optimized version of Express, offering up to 4x faster requests. That’s a wrap for this week! If you found these JavaScript updates useful, give this post an upvote to help others discover it. Don’t forget to share it with your fellow developers and keep an eye out for more insights in the JavaScript world. Until next time, keep coding and stay curious! References VoidZero Raises $4.6M ESLint Now Lints JSON and Markdown Tauri 2.0 Release Tabulator 6.3 MongoDB 8.0Eleventy 3.0 µExpress Hello JavaScript Enthusiasts! Welcome to this week’s edition of " This Week in JavaScript "! This week, we have exciting updates that will make your development workflow smoother and faster. From VoidZero’s unified JavaScript toolchain to ESLint’s new features, and even MongoDB’s performance boost, there's a lot to unpack! This Week in JavaScript VoidZero Inc. Raises $4.6M to Unify JavaScript Tooling VoidZero Inc. Raises $4.6M to Unify JavaScript Tooling VoidZero has raised $4.6M in seed funding, led by Accel, to develop a unified, high-performance toolchain for the JavaScript ecosystem. VoidZero Why is this Important? Why is this Important? JavaScript tooling is currently fragmented, with developers relying on various third-party dependencies. VoidZero aims to streamline the development process by creating a single toolchain that handles everything from bundling to testing. Key Highlights: Key Highlights: Unified Framework: Same AST and resolver for all tasks, reducing redundant parsing. Performance-Focused: Built in a compile-to-native language with support for parallelization and optimized JS plugins. Modular Components: Easily customizable building blocks for advanced use cases. Cross-Environment Support: Works across all JavaScript environments without runtime limitations. Unified Framework: Same AST and resolver for all tasks, reducing redundant parsing. Unified Framework: Performance-Focused: Built in a compile-to-native language with support for parallelization and optimized JS plugins. Performance-Focused: Modular Components: Easily customizable building blocks for advanced use cases. Modular Components: Cross-Environment Support: Works across all JavaScript environments without runtime limitations. Cross-Environment Support: VoidZero’s new bundler, Rolldown , will be in alpha later this year and set to integrate with Vite soon. Rolldown ESLint now Lints JSON and Markdown ESLint now Lints JSON and Markdown JSON and Markdown files can now be linted directly! This is part of ESLint ’s push to become a more general-purpose linter, expanding beyond JavaScript. ESLint What’s New? JSON Support: With the new @eslint/json plugin, you can lint JSON, JSONC, and JSON5 files. To get started, just install the plugin and update your ESLint config to include the JSON rules. JSON Support: With the new @eslint/json plugin, you can lint JSON, JSONC, and JSON5 files. To get started, just install the plugin and update your ESLint config to include the JSON rules. JSON Support: Markdown Support: The @eslint/markdown plugin introduces linting for CommonMark and GitHub-Flavored Markdown. Just install and configure it in your project to start catching issues in your markdown files. Markdown Support: The @eslint/markdown plugin introduces linting for CommonMark and GitHub-Flavored Markdown. Just install and configure it in your project to start catching issues in your markdown files. Markdown Support: This update opens up more possibilities for linting different file types in web projects, bringing everything under the same ESLint umbrella. ESLint Tauri 2.0 is Here Tauri 2.0 is Here Tauri 2.0’s stable release is now live, bringing key improvements and mobile support for iOS and Android. This Rust-based framework lets you build lightweight, cross-platform desktop and mobile applications simply by using HTML, JavaScript, and CSS. It does all this and makes the file size smaller and makes performance faster. Tauri 2.0’s With Tauri 2.0, you get: With Tauri 2.0, you get: Unified Codebase: One UI codebase for desktop (Windows, macOS, Linux) and mobile (iOS, Android). Hot-Module Replacement (HMR): Seamlessly update your app's frontend without rebuilding. Mobile Support: Use Swift or Kotlin to build mobile-specific functionalities while sharing Rust logic. Advanced Plugins: Extend your app's capabilities with a more refined plugin system, simplifying development. Unified Codebase: One UI codebase for desktop (Windows, macOS, Linux) and mobile (iOS, Android). Unified Codebase: Hot-Module Replacement (HMR): Seamlessly update your app's frontend without rebuilding. Hot-Module Replacement (HMR): Mobile Support: Use Swift or Kotlin to build mobile-specific functionalities while sharing Rust logic. Mobile Support: Advanced Plugins: Extend your app's capabilities with a more refined plugin system, simplifying development. Advanced Plugins: Tauri continues to evolve, offering developers a highly efficient and flexible way to build for all platforms. Tauri New Tools and Releases New Tools and Releases Tabulator 6.3: Simplify managing complex data tables in web apps with powerful editing, filtering, and sorting features. MongoDB 8.0: Enjoy a 30% performance boost and enhanced security with Queryable Encryption and better scaling. Eleventy 3.0: The latest release of this static site generator makes creating fast, efficient websites easier than ever. µExpress: Speed up your Node.js apps with this optimized version of Express, offering up to 4x faster requests. Tabulator 6.3: Simplify managing complex data tables in web apps with powerful editing, filtering, and sorting features. Tabulator 6.3: MongoDB 8.0: Enjoy a 30% performance boost and enhanced security with Queryable Encryption and better scaling. MongoDB 8.0: Eleventy 3.0: The latest release of this static site generator makes creating fast, efficient websites easier than ever. Eleventy 3.0: µExpress: Speed up your Node.js apps with this optimized version of Express, offering up to 4x faster requests. µExpress: That’s a wrap for this week! If you found these JavaScript updates useful, give this post an upvote to help others discover it. Don’t forget to share it with your fellow developers and keep an eye out for more insights in the JavaScript world. Until next time, keep coding and stay curious! References VoidZero Raises $4.6M VoidZero Raises $4.6M ESLint Now Lints JSON and Markdown ESLint Now Lints JSON and Markdown Tauri 2.0 Release Tauri 2.0 Release Tabulator 6.3 Tabulator 6.3 MongoDB 8.0 Eleventy 3.0 MongoDB 8.0 Eleventy 3.0 µExpress µExpress

VoidZero, The New ESLint, MongoDB 8.0, and More - This Week in JS

VoidZero, The New ESLint, MongoDB 8.0 a další – tento týden v JS

VoidZero, The New ESLint, MongoDB 8.0 og mere - denne uge i JS

VoidZero, das neue ESLint, MongoDB 8.0 und mehr – Diese Woche in JS

VoidZero, el nuevo ESLint, MongoDB 8.0 y más: esta semana en JS

VoidZero、新しい ESLint、MongoDB 8.0 など - 今週の JS

VoidZero, The New ESLint, MongoDB 8.0, ແລະອື່ນໆອີກ - ໃນອາທິດນີ້ໃນ JS

„VoidZero“, „The New ESLint“, „MongoDB 8.0“ ir dar daugiau – šią savaitę JS

VoidZero, The New ESLint, MongoDB 8.0 болон бусад зүйлс - Энэ долоо хоногт JS дээр

VoidZero, ESLint Baharu, MongoDB 8.0 dan Banyak Lagi - Minggu Ini dalam JS

VoidZero, ESLint i ri, MongoDB 8.0 dhe më shumë - Këtë javë në JS

VoidZero, The New ESLint, MongoDB 8.0 va boshqalar - bu hafta JSda

VoidZero、新 ESLint、MongoDB 8.0 等 - 本周 JS 资讯

File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance. Problem Statement Handling file uploads efficiently involves several challenges: Scalability: Managing large files without overwhelming server memory.
Security: Preventing unauthorized access and ensuring data integrity.
Performance: Uploading and retrieving files quickly.
Storage Management: Efficiently storing and retrieving files from the database. Solution We will use the following stack to address these challenges: Node.js: For the server-side environment.
Express.js: As the web framework.
Multer: Middleware for handling multipart/form-data, which is used for uploading files.
MongoDB: For storing metadata and files.
GridFS: A specification for storing and retrieving large files in MongoDB. Step-by-Step Implementation 1. Setting Up the Project First, let’s create a new Node.js project and install the necessary dependencies. mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream 2. Creating the Server Next, we will set up a basic Express server. // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); 3. Testing the Upload Feature We can test our implementation using tools like Postman. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file.
Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files.
Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details.
Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser.
Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file. Conclusion In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications. File uploads are a common feature in many web applications, but handling them efficiently and securely can be challenging. In this article, we will explore how to implement efficient file uploads using Node.js, Express.js, and MongoDB. We will cover the entire process, from setting up the server to storing files in MongoDB GridFS, ensuring that your application can handle large files without sacrificing performance. Problem Statement Handling file uploads efficiently involves several challenges: Scalability: Managing large files without overwhelming server memory. Security: Preventing unauthorized access and ensuring data integrity. Performance: Uploading and retrieving files quickly. Storage Management: Efficiently storing and retrieving files from the database. Scalability : Managing large files without overwhelming server memory. Scalability Security : Preventing unauthorized access and ensuring data integrity. Security Performance : Uploading and retrieving files quickly. Performance Storage Management : Efficiently storing and retrieving files from the database. Storage Management Solution We will use the following stack to address these challenges: Node.js: For the server-side environment. Express.js: As the web framework. Multer: Middleware for handling multipart/form-data, which is used for uploading files. MongoDB: For storing metadata and files. GridFS: A specification for storing and retrieving large files in MongoDB. Node.js : For the server-side environment. Node.js Express.js : As the web framework. Express.js Multer : Middleware for handling multipart/form-data , which is used for uploading files. Multer multipart/form-data MongoDB : For storing metadata and files. MongoDB GridFS : A specification for storing and retrieving large files in MongoDB. GridFS Step-by-Step Implementation 1. Setting Up the Project First, let’s create a new Node.js project and install the necessary dependencies. mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream mkdir file-upload-app
cd file-upload-app
npm init -y
npm install express multer mongoose gridfs-stream 2. Creating the Server Next, we will set up a basic Express server. // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); // server.js
const express = require('express');
const mongoose = require('mongoose');
const multer = require('multer');
const { GridFsStorage } = require('multer-gridfs-storage');
const Grid = require('gridfs-stream');
const methodOverride = require('method-override');
const bodyParser = require('body-parser');
const crypto = require('crypto');
const path = require('path');

const app = express();

// Middleware
app.use(bodyParser.json());
app.use(methodOverride('_method'));
app.set('view engine', 'ejs');

// Mongo URI
const mongoURI = 'mongodb://localhost:27017/fileupload';

// Create mongo connection
const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true });

// Init gfs
let gfs;
conn.once('open', () => {
  // Init stream
  gfs = Grid(conn.db, mongoose.mongo);
  gfs.collection('uploads');
});

// Create storage engine
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString('hex') + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: 'uploads'
        };
        resolve(fileInfo);
      });
    });
  }
});
const upload = multer({ storage });

// @route POST /upload
// @desc  Uploads file to DB
app.post('/upload', upload.single('file'), (req, res) => {
  res.json({ file: req.file });
});

// @route GET /files
// @desc  Display all files in JSON
app.get('/files', (req, res) => {
  gfs.files.find().toArray((err, files) => {
    // Check if files
    if (!files || files.length === 0) {
      return res.status(404).json({
        err: 'No files exist'
      });
    }

    // Files exist
    return res.json(files);
  });
});

// @route GET /files/:filename
// @desc  Display single file object
app.get('/files/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }
    return res.json(file);
  });
});

// @route GET /image/:filename
// @desc Display Image
app.get('/image/:filename', (req, res) => {
  gfs.files.findOne({ filename: req.params.filename }, (err, file) => {
    if (!file || file.length === 0) {
      return res.status(404).json({
        err: 'No file exists'
      });
    }

    // Check if image
    if (file.contentType === 'image/jpeg' || file.contentType === 'image/png') {
      // Read output to browser
      const readstream = gfs.createReadStream(file.filename);
      readstream.pipe(res);
    } else {
      res.status(404).json({
        err: 'Not an image'
      });
    }
  });
});

// @route DELETE /files/:id
// @desc  Delete file
app.delete('/files/:id', (req, res) => {
  gfs.remove({ _id: req.params.id, root: 'uploads' }, (err, gridStore) => {
    if (err) {
      return res.status(404).json({ err: err });
    }
    res.redirect('/');
  });
});

const port = 5000;

app.listen(port, () => console.log(`Server started on port ${port}`)); 3. Testing the Upload Feature We can test our implementation using tools like Postman. Upload a File: Use a POST request to http://localhost:5000/upload with form data containing the file. Retrieve Files: Use a GET request to http://localhost:5000/files to get the list of all files. Retrieve a Single File: Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details. Display Image: Use a GET request to http://localhost:5000/image/:filename to view the image in the browser. Delete a File: Use a DELETE request to http://localhost:5000/files/:id to delete a file. Upload a File : Use a POST request to http://localhost:5000/upload with form data containing the file. Upload a File http://localhost:5000/upload Retrieve Files : Use a GET request to http://localhost:5000/files to get the list of all files. Retrieve Files http://localhost:5000/files Retrieve a Single File : Use a GET request to http://localhost:5000/files/:filename to get a specific file’s details. Retrieve a Single File http://localhost:5000/files/:filename Display Image : Use a GET request to http://localhost:5000/image/:filename to view the image in the browser. Display Image http://localhost:5000/image/:filename Delete a File : Use a DELETE request to http://localhost:5000/files/:id to delete a file. Delete a File http://localhost:5000/files/:id Conclusion In this article, I demonstrated how to handle file uploads efficiently using Node.js, Express.js, and MongoDB. By leveraging GridFS, we can store and manage large files seamlessly, ensuring scalability, security, and performance. This approach can be further customized and expanded based on specific application requirements, making it a versatile solution for file management in modern web applications.

Efficient File Uploads in Node.js: Using Express, MongoDB, and GridFS for Scalable Storage

How to Build Your Own TODO-list Service With Golang and MongoDB

How to Integrate MongoDB in an AWS EC2 Instance

In this post, I will share real experience that I gained while working with hundreds of millions of pieces of data in MongoDB. Don't Store All Data in a Single Mongo Collection This was the worst mistake that we made, which became the root cause of all issues. We used to store 500 million complex structured data in a single collection, which caused the following side effects: Indexes creation takes a very long time.
If collection is dropped accidentally, all data is gone.
Querying data takes longer than usual, even though with indexes.
Counting with filters is timing out because of large document scanning. Probably there will be even more issues as the data increases on a daily basis. Bad Document Structure Invest some time to define the document structure precisely. This is a critical part that never gets attention when starting the project. Our mistake was excluding the field entirely from some of the documents instead of assigning a default value for the field. That ended up with slow query scanning because of existence checking. In MongoDB, it's possible to give partial indexes, but it's really not working that well where field existence is checked. The example below shows null value assigned for phone instead of excluding it from the document completely. {
  "_id": "507f191e810c19729de860ea",
  "name": "John Doe",
  "age": 30,
  "email": "john.doe@example.com",
  "phone": null, //assign default value instead of removing
  "isActive": true,
  "tags": ["user", "active", "new"]
  "registeredOn": "2024-04-19T15:00:00Z"
} {$exists: true} filter is slow so consider adding a default value and indexing the fields for fast query results. Use Bulk Operations Bulk operations are used to execute multiple write operations (inserts, updates, deletes) efficiently in a single command. This method can significantly reduce the number of round trips between your application and the MongoDB server, leading to performance improvements, especially when dealing with large volumes of data. Since we were processing millions of data from RabbitMQ queues, it required handling messages in batches and making bulk DB operations for each batch. Here's the documentation for pymongo on how to use bulk operations for better write performance. Use Indexes Indexing plays a crucial part in the DB performance. Create required indexes in the collection based on the filters you are using in the codebase. That also refers to the part about creating a valid document structure. Instead of removing the field completely, assign a default value to benefit from indexing power. Instead of Count, Use Aggregation Pipeline If you need to get count of data based on a specific filter, I wouldn't recommend using count() , instead, use a combination of $match and $count aggregation for fast performance. $match filters the documents first, which means only the relevant documents that meet the specified conditions are passed down to the $count stage. This is more efficient than counting all documents and then filtering them, as count() might initially do without any conditions. Wrap Up That's all that I have experienced so far with MongoDB. I will update the article once I find more hints and best practices to help optimize the performance. In this post, I will share real experience that I gained while working with hundreds of millions of pieces of data in MongoDB. Don't Store All Data in a Single Mongo Collection This was the worst mistake that we made, which became the root cause of all issues. We used to store 500 million complex structured data in a single collection, which caused the following side effects: Indexes creation takes a very long time. If collection is dropped accidentally, all data is gone. Querying data takes longer than usual, even though with indexes. Counting with filters is timing out because of large document scanning. Indexes creation takes a very long time. If collection is dropped accidentally, all data is gone. Querying data takes longer than usual, even though with indexes. Counting with filters is timing out because of large document scanning. Probably there will be even more issues as the data increases on a daily basis. Bad Document Structure Invest some time to define the document structure precisely. This is a critical part that never gets attention when starting the project. Our mistake was excluding the field entirely from some of the documents instead of assigning a default value for the field. That ended up with slow query scanning because of existence checking. In MongoDB, it's possible to give partial indexes, but it's really not working that well where field existence is checked. The example below shows null value assigned for phone instead of excluding it from the document completely. null phone {
  "_id": "507f191e810c19729de860ea",
  "name": "John Doe",
  "age": 30,
  "email": "john.doe@example.com",
  "phone": null, //assign default value instead of removing
  "isActive": true,
  "tags": ["user", "active", "new"]
  "registeredOn": "2024-04-19T15:00:00Z"
} {
  "_id": "507f191e810c19729de860ea",
  "name": "John Doe",
  "age": 30,
  "email": "john.doe@example.com",
  "phone": null, //assign default value instead of removing
  "isActive": true,
  "tags": ["user", "active", "new"]
  "registeredOn": "2024-04-19T15:00:00Z"
} {$exists: true} filter is slow so consider adding a default value and indexing the fields for fast query results. {$exists: true} Use Bulk Operations Bulk operations are used to execute multiple write operations (inserts, updates, deletes) efficiently in a single command. This method can significantly reduce the number of round trips between your application and the MongoDB server, leading to performance improvements, especially when dealing with large volumes of data. Since we were processing millions of data from RabbitMQ queues, it required handling messages in batches and making bulk DB operations for each batch. Here's the documentation for pymongo on how to use bulk operations for better write performance. documentation documentation pymongo Use Indexes Indexing plays a crucial part in the DB performance. Create required indexes in the collection based on the filters you are using in the codebase. That also refers to the part about creating a valid document structure. Instead of removing the field completely, assign a default value to benefit from indexing power. Instead of Count, Use Aggregation Pipeline If you need to get count of data based on a specific filter, I wouldn't recommend using count() , instead, use a combination of $match and $count aggregation for fast performance. count() $match $count $match filters the documents first, which means only the relevant documents that meet the specified conditions are passed down to the $count stage. This is more efficient than counting all documents and then filtering them, as count() might initially do without any conditions. $match $count count() Wrap Up That's all that I have experienced so far with MongoDB. I will update the article once I find more hints and best practices to help optimize the performance.

Lessons I Learned From Managing Hundreds of Millions of Data in MongoDB

Once, I was tasked with improving the database and general app operations of backend services which were using MongoDB as their main database. These services were part of a huge infrastructure where millions of messages came through the queues and needed to be processed based on the message actions. That means tons of DB ops each second and other additional checks while processing. Real Case Scenario The processing layer of the service was using Pymongo to interact with MongoDB and the service itself was running in a synchronous environment. Even though the database operations were handled in bulk, the performance was still not capable of handling incoming data. Synchronous code was making things even worse. The code execution waits for the result from the current operation to move forward. That's a serious bottleneck in scalable systems. This was causing queue overflows and potential data loss every time. Asynchronous Environment The solution I implemented was a combination of: Motor
Asyncio
Uvloop Let's quickly go through the definitions of these items. PyMongo is the official MongoDB driver for Python, providing a simple and intuitive way to interact with MongoDB databases. It's synchronous, meaning each database operation blocks the execution of your program until it completes it, which can be a bottleneck in I/O-bound tasks. Motor is the asynchronous driver for MongoDB, built on top of PyMongo and designed to take advantage of Python's asyncio library. Motor allows you to perform non-blocking database operations, making it suitable for high-performance applications that require concurrency. To illustrate the performance differences, I prepared a stress test using two scripts: one using Motor (asynchronous) and the other using PyMongo (synchronous). Both scripts performed the same task of reading and writing documents to MongoDB in batches. Both scripts read 300k documents from the source collection and migrated them to the new target collection. Asynchronous Script (Motor) import logging
import asyncio
import time
from bson import ObjectId
from motor.motor_asyncio import AsyncIOMotorClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'

client = AsyncIOMotorClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db["new_collection"]

async def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = await cursor.to_list(length=batch_size)
    return products

async def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = await target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

async def process_batches(batch_size, concurrency_limit):
    tasks = []
    last_id = None
    while True:
        products = await fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        tasks.append(bulk_write_to_mongo(products))
        if len(tasks) >= concurrency_limit:
            await asyncio.gather(*tasks)
            tasks = []
    # Process remaining tasks if any
    if tasks:
        await asyncio.gather(*tasks)

async def main():
    batch_size = 1000
    concurrency_limit = 10

    start_time = time.time()
    await process_batches(batch_size, concurrency_limit)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    asyncio.run(main()) Synchronous Script (Pymongo) import logging
import time
from bson import ObjectId
from pymongo import MongoClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'
TARGET_COLLECTION_NAME = 'new_collection'

client = MongoClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db[TARGET_COLLECTION_NAME]

def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = list(cursor)
    return products

def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

def process_batches(batch_size):
    last_id = None
    while True:
        products = fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        bulk_write_to_mongo(products)

def main():
    batch_size = 1000

    start_time = time.time()
    process_batches(batch_size)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    main() Results and Analysis Execution Time of Migrating 300k documents: Asynchronous script: 17.15 seconds
Synchronous script: 23.26 seconds The asynchronous script completed the task 6.11 seconds faster than the synchronous script. While this might not seem like a significant difference for a single run, it becomes more pronounced in high-load scenarios or when processing large datasets continuously. Benefits of Using Motor and Asynchronous Environment Improved Throughput: Asynchronous operations can handle more tasks concurrently, increasing overall throughput. This is especially beneficial in applications with high I/O operations, such as web servers handling multiple database queries simultaneously. Non-Blocking I/O: Asynchronous operations do not block the main thread, allowing other tasks to continue running. This results in better CPU utilization and smoother application performance, particularly under load. Scalability: Asynchronous code scales better with the number of concurrent operations. For example, a web application using Motor can handle more simultaneous requests compared to one using PyMongo. Resource Efficiency: Asynchronous operations can lead to more efficient use of system resources. For instance, the event loop in asyncio allows the application to switch between tasks, reducing idle times and improving overall efficiency. You can find the source code on the GitHub repository below: https://github.com/PylotStuff/motor-asyncio-performance?embedable=true Conclusion The choice between Motor and PyMongo depends on the specific needs of your application. For applications that require high concurrency and efficient I/O handling, Motor and the asynchronous approach offer significant advantages. However, for simpler applications or scripts where ease of implementation is a priority, PyMongo's synchronous approach might be sufficient. By leveraging asynchronous operations with Motor, you can build more scalable and performant applications, making it a worthwhile consideration for modern web development. Once, I was tasked with improving the database and general app operations of backend services which were using MongoDB as their main database. These services were part of a huge infrastructure where millions of messages came through the queues and needed to be processed based on the message actions. That means tons of DB ops each second and other additional checks while processing. Real Case Scenario The processing layer of the service was using Pymongo to interact with MongoDB and the service itself was running in a synchronous environment. Even though the database operations were handled in bulk, the performance was still not capable of handling incoming data. Synchronous code was making things even worse. The code execution waits for the result from the current operation to move forward. That's a serious bottleneck in scalable systems. This was causing queue overflows and potential data loss every time. Asynchronous Environment The solution I implemented was a combination of: Motor Asyncio Uvloop Motor Asyncio Uvloop Let's quickly go through the definitions of these items. PyMongo is the official MongoDB driver for Python, providing a simple and intuitive way to interact with MongoDB databases. It's synchronous, meaning each database operation blocks the execution of your program until it completes it, which can be a bottleneck in I/O-bound tasks. PyMongo Motor is the asynchronous driver for MongoDB, built on top of PyMongo and designed to take advantage of Python's asyncio library. Motor allows you to perform non-blocking database operations, making it suitable for high-performance applications that require concurrency. Motor To illustrate the performance differences, I prepared a stress test using two scripts: one using Motor (asynchronous) and the other using PyMongo (synchronous). Both scripts performed the same task of reading and writing documents to MongoDB in batches. Both scripts read 300k documents from the source collection and migrated them to the new target collection. Asynchronous Script (Motor) import logging
import asyncio
import time
from bson import ObjectId
from motor.motor_asyncio import AsyncIOMotorClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'

client = AsyncIOMotorClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db["new_collection"]

async def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = await cursor.to_list(length=batch_size)
    return products

async def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = await target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

async def process_batches(batch_size, concurrency_limit):
    tasks = []
    last_id = None
    while True:
        products = await fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        tasks.append(bulk_write_to_mongo(products))
        if len(tasks) >= concurrency_limit:
            await asyncio.gather(*tasks)
            tasks = []
    # Process remaining tasks if any
    if tasks:
        await asyncio.gather(*tasks)

async def main():
    batch_size = 1000
    concurrency_limit = 10

    start_time = time.time()
    await process_batches(batch_size, concurrency_limit)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    asyncio.run(main()) import logging
import asyncio
import time
from bson import ObjectId
from motor.motor_asyncio import AsyncIOMotorClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'

client = AsyncIOMotorClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db["new_collection"]

async def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = await cursor.to_list(length=batch_size)
    return products

async def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = await target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

async def process_batches(batch_size, concurrency_limit):
    tasks = []
    last_id = None
    while True:
        products = await fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        tasks.append(bulk_write_to_mongo(products))
        if len(tasks) >= concurrency_limit:
            await asyncio.gather(*tasks)
            tasks = []
    # Process remaining tasks if any
    if tasks:
        await asyncio.gather(*tasks)

async def main():
    batch_size = 1000
    concurrency_limit = 10

    start_time = time.time()
    await process_batches(batch_size, concurrency_limit)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    asyncio.run(main()) Synchronous Script (Pymongo) import logging
import time
from bson import ObjectId
from pymongo import MongoClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'
TARGET_COLLECTION_NAME = 'new_collection'

client = MongoClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db[TARGET_COLLECTION_NAME]

def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = list(cursor)
    return products

def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

def process_batches(batch_size):
    last_id = None
    while True:
        products = fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        bulk_write_to_mongo(products)

def main():
    batch_size = 1000

    start_time = time.time()
    process_batches(batch_size)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    main() import logging
import time
from bson import ObjectId
from pymongo import MongoClient

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# MongoDB setup
MONGO_URI = 'mongodb://root:root@localhost:27019'
DB_NAME = 'products'
COLLECTION_NAME = 'gmc_products'
TARGET_COLLECTION_NAME = 'new_collection'

client = MongoClient(MONGO_URI)
db = client[DB_NAME]
collection = db[COLLECTION_NAME]
target_collection = db[TARGET_COLLECTION_NAME]

def fetch_products(batch_size, last_id=None):
    query = {'_id': {'$gt': last_id}} if last_id else {}
    cursor = collection.find(query).sort('_id').limit(batch_size)
    products = list(cursor)
    return products

def bulk_write_to_mongo(products):
    for product in products:
        product['_id'] = ObjectId()  # Generate a new ObjectId for each product

    try:
        result = target_collection.insert_many(products, ordered=False)
        logger.info(f'Inserted {len(result.inserted_ids)} products into MongoDB.')
    except Exception as e:
        logger.error(f'Error inserting products into MongoDB: {e}')

def process_batches(batch_size):
    last_id = None
    while True:
        products = fetch_products(batch_size, last_id)
        if not products:
            break
        last_id = products[-1]['_id']
        bulk_write_to_mongo(products)

def main():
    batch_size = 1000

    start_time = time.time()
    process_batches(batch_size)
    end_time = time.time()
    logger.info(f'Total time: {end_time - start_time:.2f} seconds.')

if __name__ == '__main__':
    main() Results and Analysis Execution Time of Migrating 300k documents: Asynchronous script: 17.15 seconds Synchronous script: 23.26 seconds Asynchronous script: 17.15 seconds Synchronous script: 23.26 seconds The asynchronous script completed the task 6.11 seconds faster than the synchronous script. 6.11 seconds While this might not seem like a significant difference for a single run, it becomes more pronounced in high-load scenarios or when processing large datasets continuously. Benefits of Using Motor and Asynchronous Environment Improved Throughput : Asynchronous operations can handle more tasks concurrently, increasing overall throughput. This is especially beneficial in applications with high I/O operations, such as web servers handling multiple database queries simultaneously. Improved Throughput Non-Blocking I/O : Asynchronous operations do not block the main thread, allowing other tasks to continue running. This results in better CPU utilization and smoother application performance, particularly under load. Non-Blocking I/O Scalability : Asynchronous code scales better with the number of concurrent operations. For example, a web application using Motor can handle more simultaneous requests compared to one using PyMongo. Scalability Resource Efficiency : Asynchronous operations can lead to more efficient use of system resources. For instance, the event loop in asyncio allows the application to switch between tasks, reducing idle times and improving overall efficiency. Resource Efficiency You can find the source code on the GitHub repository below: https://github.com/PylotStuff/motor-asyncio-performance?embedable=true https://github.com/PylotStuff/motor-asyncio-performance?embedable=true Conclusion The choice between Motor and PyMongo depends on the specific needs of your application. For applications that require high concurrency and efficient I/O handling, Motor and the asynchronous approach offer significant advantages. However, for simpler applications or scripts where ease of implementation is a priority, PyMongo's synchronous approach might be sufficient. By leveraging asynchronous operations with Motor, you can build more scalable and performant applications, making it a worthwhile consideration for modern web development.

Boost MongoDB Performance: Motor Client vs PyMongo - Which is Faster?

Working with Mongoose models can sometimes feel like keeping track of a complex web. Unlike many SQL databases, finding free and easy tools to visualize your Mongoose schemas can be tough. This can make it difficult to understand how your data is connected and collaborate with other developers. To solve this problem, I built an open-source tool that visualizes your Mongoose schemas. Now you can easily see how your data fits together and keep your projects organized! URL: http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Repo: https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool NPM Package: https://www.npmjs.com/package/mongoose-parser My Approach While it might be tempting to directly query the database using the connection URI to extract model relationships, this approach has a few drawbacks: Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials.
Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool. To address these concerns, I opted to obtain the schema by parsing the JavaScript files. This approach has some key advantages: Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe.


Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Parsing the JavaScript files does require some additional logic, but the benefits in terms of security and efficiency make it a worthwhile trade-off. It’s important to note that this entire operation happens within your browser. Since it’s a static site, there are no API calls involved. You simply upload your JavaScript files, and the tool does its magic to visualize your Mongoose schema relationships, all without ever touching your database directly. Parsing the Schema Files Extracting model relationships from JavaScript files can be done using regular expressions. However, this approach has some limitations: Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity.


Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. To address these limitations, I opted for a more robust approach: parsing the Abstract Syntax Tree of the JavaScript files. What’s an AST? An AST, or Abstract Syntax Tree, is a tree-like representation of the code’s structure. Imagine it like a family tree but for your code! It breaks down the code into its core elements, like variables, functions, and expressions, making it easier to understand the relationships between different parts of the code. Where are ASTs Used? ASTs are a powerful tool used in various programming contexts beyond this schema visualization tool. Here are a few examples: Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output.


Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Benefits of Using an AST for Schema Visualisation By converting the JavaScript files to ASTs, the tool gains several advantages: Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code.


Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles.


Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. While parsing with an AST requires a bit more initial setup compared to regular expressions, the benefits in terms of accuracy, flexibility, and maintainability make it a worthwhile investment for this tool. It allows for robust and reliable extraction of model relationships from your JavaScript schema files. Since mongoose schemas are javascript files, I made use of acorn (https://www.npmjs.com/package/acorn) a javascript parser written in javascript 🙃 to build the AST. Acron creates the AST for the given javascript files and from that i look for mongoose methods and extract the associated model definitions. I converted this set of functions into an NPM package of its own (https://www.npmjs.com/package/mongoose-parser). The tool uses this package to extract the schemas. Once the schema is extracted, the frontend react app converts this schema into a set of nodes and edges and creates the visual representation with the help of react-flow (https://www.npmjs.com/package/reactflow). Usage Just drag and drop the JavaScript files containing your Mongoose schema definitions here, and watch your schemas come to life! http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Thanks for reading :) Socials Linkedin: https://www.linkedin.com/in/abdul-mohammad-a567b2183/ Github: https://github.com/iam-abdul Twitter: https://x.com/iam_abd038 Working with Mongoose models can sometimes feel like keeping track of a complex web. Unlike many SQL databases, finding free and easy tools to visualize your Mongoose schemas can be tough. This can make it difficult to understand how your data is connected and collaborate with other developers. To solve this problem, I built an open-source tool that visualizes your Mongoose schemas. Now you can easily see how your data fits together and keep your projects organized! URL: http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Repo: https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool https://github.com/iam-abdul/Mongoose-Schema-Visualization-Tool NPM Package: https://www.npmjs.com/package/mongoose-parser https://www.npmjs.com/package/mongoose-parser My Approach My Approach While it might be tempting to directly query the database using the connection URI to extract model relationships, this approach has a few drawbacks: Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials. Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool. Security Concerns: Sharing connection URIs can be a security risk, as it exposes sensitive database credentials. Security Concerns: Unnecessary Complexity: For a simple task like schema visualization, querying the database adds unnecessary complexity to the tool. Unnecessary Complexity: To address these concerns, I opted to obtain the schema by parsing the JavaScript files. This approach has some key advantages: Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe. Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe. Security: By parsing the code, there’s no need to handle connection URIs, keeping those credentials safe. Security: Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Efficiency: Parsing JavaScript files is generally faster and lighter on resources compared to querying the database. Efficiency: Parsing the JavaScript files does require some additional logic, but the benefits in terms of security and efficiency make it a worthwhile trade-off. It’s important to note that this entire operation happens within your browser. Since it’s a static site, there are no API calls involved. You simply upload your JavaScript files, and the tool does its magic to visualize your Mongoose schema relationships, all without ever touching your database directly. Parsing the Schema Files Extracting model relationships from JavaScript files can be done using regular expressions. However, this approach has some limitations: Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity. Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity. Complexity: Mongoose schema definitions can involve complex nesting and property structures. Regular expressions can quickly become cumbersome and error-prone when dealing with such complexity. Complexity: Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. Maintainability: As your schemas evolve, regex patterns might need frequent updates to keep up, making the code harder to maintain in the long run. Maintainability: To address these limitations, I opted for a more robust approach: parsing the Abstract Syntax Tree of the JavaScript files. What’s an AST? What’s an AST? An AST, or Abstract Syntax Tree, is a tree-like representation of the code’s structure. Imagine it like a family tree but for your code! It breaks down the code into its core elements, like variables, functions, and expressions, making it easier to understand the relationships between different parts of the code. Where are ASTs Used? Where are ASTs Used? ASTs are a powerful tool used in various programming contexts beyond this schema visualization tool. Here are a few examples: Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output. Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output. Compilers: Compilers, which translate code from one language to another, often use ASTs as an intermediate representation to analyze and optimize the code before generating the final output. Compilers: Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Linters and Static Code Analysers: These tools use ASTs to identify potential errors, stylistic inconsistencies, or inefficiencies in the code without actually running it. Linters and Static Code Analysers: Benefits of Using an AST for Schema Visualisation Benefits of Using an AST for Schema Visualisation By converting the JavaScript files to ASTs, the tool gains several advantages: Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code. Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles. Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code. Accuracy: ASTs provide a more accurate and reliable way to identify schema definitions within your code. Accuracy: Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles. Flexibility: ASTs can handle complex code structures more effectively than regular expressions, making the tool adaptable to different schema definition styles. Flexibility: Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. Maintainability: Using ASTs makes the parsing logic cleaner and easier to maintain as your schemas evolve. Maintainability: While parsing with an AST requires a bit more initial setup compared to regular expressions, the benefits in terms of accuracy, flexibility, and maintainability make it a worthwhile investment for this tool. It allows for robust and reliable extraction of model relationships from your JavaScript schema files. Since mongoose schemas are javascript files, I made use of acorn ( https://www.npmjs.com/package/acorn ) a javascript parser written in javascript 🙃 to build the AST. https://www.npmjs.com/package/acorn Acron creates the AST for the given javascript files and from that i look for mongoose methods and extract the associated model definitions. I converted this set of functions into an NPM package of its own ( https://www.npmjs.com/package/mongoose-parser ). The tool uses this package to extract the schemas. https://www.npmjs.com/package/mongoose-parser Once the schema is extracted, the frontend react app converts this schema into a set of nodes and edges and creates the visual representation with the help of react-flow ( https://www.npmjs.com/package/reactflow ). https://www.npmjs.com/package/reactflow Usage Just drag and drop the JavaScript files containing your Mongoose schema definitions here, and watch your schemas come to life! http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ http://mongoose-schema-visualize.s3-website.ap-south-1.amazonaws.com/ Thanks for reading :) Socials Socials Linkedin: https://www.linkedin.com/in/abdul-mohammad-a567b2183/ https://www.linkedin.com/in/abdul-mohammad-a567b2183/ Github: https://github.com/iam-abdul https://github.com/iam-abdul Twitter: https://x.com/iam_abd038

Stories, News & Mentions about MONGODB

Categories

Trending Topics

Stories, News & Mentions about MONGODB

Mongodb’s Stories on HackerNoon

Mongodb's News & Mentions

Categories

Trending Topics