Capture Facebook-Compatible 4K 360 Videos of 3D Scenes in Your Browser

A capture from inside of a Three.js example scene.

This article describes how to generate buzz-making 360 videos and photos in 4K that will play right on your Facebook timeline or YouTube page, using the popular Three.js graphics library for browser-based 3D scenes.

The code is available on Github.

Overview

1. Setup a 3D environment and include the modified capture libraries.
2. Export a 360 photo sequence. Write metadata for any 360 photos.
3. Stitch all the 360 photos together into a 360 video and add metadata.
4. Post to social media and see the magic!

An example of a 4K 360 Video Embed from YouTube

Set the scene in 3D in Three.js

I started with the boilerplate from the Three.js GitHub page, which puts a rotating cube in front of the camera. Since we’re in 360, we want to put geometry at the following locations in our test scene so that there’s something to look in each of the major directions: front, behind, above, below, left, and right. I changed the geometry to a sphere and then added a simple test UV grid image texture material.

Then, I made some modifications to the CCapture.js library, where I added a CC360Encoder class that calls into an cubemap to equirectangular image capture library from the same author. I made modifications to that library also, where I prepare the cube camera data for the encoder with the preBlob class.

You can include my modified versions of the libraries and enable them with a few lines of code. First, instantiate a capturer360 when you initialize your scene.

capturer360 = new CCapture({format: 'threesixty'});

Add a managed CubemapToEquirectangular camera when you setup your scene. Here we use “4K” but you can also use “2K” or “1K” as resolutions.

equiManaged = new CubemapToEquirectangular(renderer, true,"4K");

And add this at the end of your render loop.

capturer360.capture(canvas);

I added two simple buttons in the HTML and wired them up to functions that start and stop the capture.

function startCapture360(event) {capturer360.start();}

function saveCapture360(event) {capturer360.stop();}

Make a sequence of 360 photos

First, click the “Start Capture” button.

You will notice that it feels like things are moving more slowly as the processing load increases. This is typical.

Use the frame counter to determine how long the capture should be (FFMPEG defaults to 25fps). To avoid “Out of Memory” errors and crashes due to the size of the equirectangular images in the browser memory, I modified the CCapture library to download a batch of images every 3 seconds or 75 frames. This allows you to capture really long movies. The browser will automatically trigger a download every batch.

Move the camera during capture at your discretion; you should begin by leaving it alone and allowing the user to be the only one to control the view. This is an area to experiment, but be wary of user comfort.

When you’re done click “Stop Capture”.

Now you’ve got a bunch of .tar files containing your captures.

Extract all of the .tar archives to a /captures directory somewhere on your computer using a tool like WinRAR or Z-Zip on Windows or UnRarX on Mac.

Starts at 1.66gb of images for a 30-second clip at 4K resolution (4096px by 2048px)

A sequence of 360 Photos

Add Metadata For a Single 360 Photo

If you want to convert a single image for social media, you need to add the Projection Type metadata tag that the platforms need to recognize the image as a 360 photo.

Download exiftool for your platform and add it to your PATH or drop its binary into the /captures folder alongside the images.

Enter the folder from your terminal and then run this command:

exiftool -ProjectionType="equirectangular" 0000000.jpg;

That’s it. Now you can post that image to Facebook as a 360 Photo. You can do this for any of the images in your archive. These will post to Facebook as 360 photos.

Test photo is trippy!

Stitching a video together using FFMPEG

After you have FFMPEG installed on your machine, back at the terminal you can run a one line command to stitch all of the images in the current directory together:

ffmpeg -i %07d.jpg video.mp4

The “%07d” tells FFMPEG that there are 7 decimals before the “.jpg” extension in each filename. Run the command from the /captures directory and let the video process until it is complete. On Windows you can just drop the FFMPEG executable into the /captures folder if it isn’t in your PATH.

I’ve found that its about 1 megabyte per second of converted 4k 360 video with default settings.

I had to double check, but after stitching one of my test 4K 360 videos is only 3.12 mb (500x smaller!)

Adding Metadata For a 360 Video

Download Google’s Spatial Media Metadata Injector tool for your platform. Run the program and open your video. You have a spherical video. Select “Inject Metadata” and choose where to save the injected file.

Running the tool is very simple.

Let it out into the world!

Open up your Facebook feed and upload the final product.

A 360 Video post on Facebook before the player starts

If its a video, you will get a notification telling you that Facebook will let you know when its done. You will also be able to set the initial view orientation into the photo or video, an important step in making sure that users have a great experience. Facebook will automatically loop 360 videos on your timeline and go fullscreen when you turn your phone sideways (but not loop full screen).

With YouTube, upload the video and edit your description—the 360 effect will be applied after you post. YouTube can play 360 videos in ridiculous 4K, and will give you the option to watch in VR. I haven’t found an option to set the initial orientation yet.

Another test render, with labels for each of the locations around you.

Why 360?

I’ve enjoyed seeing the rise of 360 video content on the internet, most notably on Facebook and YouTube. I still remember the first time I watched the Blue Angels on my feed, looking around from inside the cockpit. What a thrill!

Making 360 video content can be hard, whether you are capturing the real or virtual world: if you’re doing real-world capture the cameras can be expensive or difficult to use and the stitching and editing process laborious; virtual-world capture means finding a specialist to work with plugins inside of a complicated game engine like Unity or Unreal or going to an expensive piece of Motion Graphics software like Cinema4d or Maya. Iterating on the web is fast and approachable, and the results are pretty incredible.

After you export your first 360 photo or video, you will think a lot about space. Particularly about how much of it there is. You’ll encounter the stultifying artistic problem of how to actually fill that space with interesting content. In 360, there is no proscenium — the stage is the whole world.

As artists and entrepreneurs exploring a new medium, we need to be able to prototype quickly and to share those prototypes to get feedback. Too often, that means building a whole application, asking our friends and family to put on a cumbersome piece of equipment, or visit some weird link. It’s important that we be able to put these experiences in front of people where they are already hanging out. Ergo, social media.

360 is an exciting new creative space, one that blurs the lines between traditional screen-based content, full immersive virtual reality, and sensor-mediated augmented reality. It’s recognizable, but new. It’s the easiest way for the most people to dip their feet into the shallow end of the pool. The more quickly we can iterate and receive criticism of those iterations, the sooner we’ll be able to reach depth of meaning.

What To Make

There are so many directions to go in here in types of content!!

Arrange some animated 3D characters. Procedurally generate a whole field of stars or hills dotted with flowers. Imagine a magical forest full of floating particles.

Try less game-like pieces, since you’ve got the power of the HTML5 canvas. Embed images and text for a news piece. What about moving graphs for a financial markets data visualization?

Sound sources can also be arranged in 360: use ambisonics to create 3D soundscapes and make unique music videos and powerful campaign messaging. Spatial sound is a technique that deserves as much attention as visual immersion.

Get in touch with me about custom 360 content for your brand or company, or to hear more about more versatile versions of this capture product that can run as a standalone app for Windows/Mac or in the cloud.

Awesome libraries and programs used

• Three.js for 3D using webGL inside of an HTML5 page
• Ccapture.js (use my modified version) for capturing frames smoothly
• Cubemap to Equirectangular Camera (use my modified version) for taking 360 Photos
• WinRAR or UnRarx for extracting the image sequence .tar file
• Exiftool for writing 360 photo metadata
• Ffmpeg for converting stills to a movie
• Spatial Media metadata editor for making 360 Videos Facebook / Youtube compatible

Related Links

Photo Sphere XMP Metadata | Street View | Google Developers_The panorama namespace described here contains properties that provide information regarding the creation and rendering…_developers.google.com

Editing 360 Photos & Injecting Metadata_Facebook recognizes and processes 360 photos by looking for camera-specific metadata found in photos taken using 360…_facebook360.fb.com