Automatic 3D Movies Using AI | The Curious Refuge Podcast
Automatic 3D Movies Using AI | Episode 02
The Curious Refuge Podcast | 3D Movies in 5 Seconds
Welcome to episode 2 of The Curious Refuge Podcast! In this episode we talk with two phenomenal people from the team at Leia Inc, Marlon Fuentes and Nima Zeighami. Marlon is Leia Inc's Social Media and Community Manager, and Nima is their Director of Product.
In the episode, we talk about a handful of different topics from the history of Leia Inc, to 3D tools, and even how to make your short film into a 3D film in just 5 seconds.
Join Our Newsletter for the Latest AI Filmmaking News
Our AI Filmmaking Course Starts Soon
Check out Leia Inc
Leia Inc. is a leading provider of eyewear-free 3D display hardware and software solutions. Our breakthrough technology, born from pioneering research at HP Labs and Philips, leverages advanced optics and AI to transform ordinary displays into naturally immersive experiences. We envision a future where the three-dimensional reality of our physical world is seamlessly reflected in the digital space to transform the way we work, play, and connect — and strive to make 3D accessible to anyone, anywhere, on any device. The company is headquartered in Menlo Park, CA.
Link from the show:
The Curious Refuge Podcast | Episode 02
Here is a transcript of the podcast:
We're really excited about this episode because we sat down with the team at Laya Inc to chat about the future of 3D Filmmaker. So typically when you work on a 3D film, you either have to shoot it in 3D or you have to convert it in post-production.
And it's a very tedious process. In turn, 3D films really are only created by high end studios. But what if it was possible for you as an independent creator to create your own 3D films within seconds? Well, that's what we're going to talk about in this episode of the podcast. So now I'm going to kick it over to Shelby to get started.
Layering is an organization that has been utilizing A.I. technology for their entire existence. They make entertainment more captivating by utilizing 3D and AI technology in conjunction with one another. Layer has over ten 3D apps from 3D streaming 3D gaming 3D charts. Not to mention they also have a tablet called the Loom Pad two that provides immersive 3D viewing without glasses in real time.
In short, they have a tablet that converts any video in to 3D, which is incredible. So here today with us is Marlon Fuentes, who is the social media and community manager over at Lia, and Nima Zanganeh, who is the director of product. Thank you for joining us, Marlon, Nima, and welcome to the podcast. Thanks. Good to be here.
So I know I've slyly introduced you both, but I'd love to hear because Leo is so unique in what you do, and we're pretty new to you guys, but you've been around for a very long time. How do you explain baby to your parents or to someone who's not familiar with Lia, what it is that your company does and what you do there?
Yeah, I think seeing really is believing. So I try not to even explain it because when I whenever I do, people are like, I don't know what that means. Like, what is that? So usually what I'll do is I'll take the loop ad two and I'll just flip the screen over. I'll be like, Look at this. And then they immediately they're like, Well, do either immediately get to go, my God, this amazing.
Or they go, How is this possible? What what is this wizardry? What kind of magic is this if I don't have one of our devices around? I, of course, use Olympics and I go, Hey, check this out. What's your favorite celebrity? Pull it up on Google. Download an image, Be like Boom. Check this out. Check out this incredible depth animation.
Or I say, Hey, let me take a picture of you and your friend real quick. Take the photo, turn around, be like, check this out. And once people see it, they understand, they get it. They go, How do I do this? How does this accessible? When does this come out? And I get to go, Hey, it's been out for a while.
You can access it right now. It's you don't have to wait. That's hilarious. That's amazing. That's right. Yeah. 3D anywhere. Any time. On any device. That's. That's the vision and the idea there. That is so cool. I mean, the the image speaks for itself. It sounds like I feel like the first time that I used live pics for the first time, it was absolutely mind blowing, especially in the context of using artificial intelligence, because to those of you who have never used it literally, you just upload an image and immediately it creates this kind of 3D parallax effect that is maybe somewhat similar to if you've worked on an after Effects project file in the
past with like images and you've cut out the different planes and used cameras that fly in 3D space, but it does it automatically so you literally don't have to adjust anything. You just drop in the image and it automatically does the movements. It's really incredible. And it was very much a magical experience getting to interface with it for the very first time.
I'm curious with that, like how long have you guys had Layer Picks as one of the tools? So the original app is actually just called Layer fixing. It was an app on our on the smartphone that we made about five years ago. I was a launch app for that. We had a little bit earlier for beta testers too, and that's basically a social network for image sharing.
And we saw lots of people were creating incredible things, but most people were using just regular 3D photos, so they had to have a 3D camera and then they were going and capturing that. But we saw some creators were going and doing some really crazy stuff. They were, you know, either rendering things to get the photos to share with people or they were, you know, trying to use like a third party tool on desktop and spending, you know, like minutes to generate a depth map for a single image and then using that to to create something.
And so that sort of evolved to the point where we're like, well, this is really cool. How do we make it more accessible? And so one of the first things we did was we go, okay, We're like, one of the problems is that people can only see it on a 3D device. So we made it so that if you send a link from an image that you uploaded to MPIX, if you send that to someone, it would have this automatic left and right horizontal animation to it so people could take their 3D photos and they would get this this image where they send the link to to their friend.
And people love that because they're like, Wow, well now I don't have to only share with other people three displays. I can send the links to the things I post to like my friends, my family, and they can they can get this sense of the depth on any device. And so once that happened, we ended up seeing a need in the market for something that was a little bit more accessible.
So you didn't need a layered device to access it. And we saw some open source projects out there that were pretty interesting. And then I was evolving at a breakneck pace. And so we basically started working on laser fix converter about three years ago now. And then it was on hiatus off and on until two years ago when I finally me and a couple of other our engineers, we we finally got the app out.
And so it's been out for two years. And then about a year ago, as I'm sure you all know, that I explored Asian happened and suddenly we saw ourselves rocketing to the top of, you know, top ten air tools lists all over the place. And that really completely changed the change the world for that. But, you know, the original idea was, hey, how do we make an accessible application that everyone can use and access 3D, which will hopefully bring more people into our platform?
And it evolved into like, Wow, we have this incredible tool. How do we make this a part of every single 2D and 3D workflow, regardless of what platform or device you're on, whether you're on 2D, whether you're on our 3D displays, whether you're on a VR headset, our products useful. And so it's been evolving to be as useful as possible to as many people as possible.
And how do you guys envision that working with creatives? So obviously Layer pegs, which is the tool I'm most familiar with, because that's kind of what introduced me to what you guys are doing. It's an online application and very easy to use, but from what you're saying, it sounds like you actually want to involve it at a different part of the creative process and maybe more in line with creatives as they are actually putting these things together beyond just a simple web page application that you drag and drop images.
You know, it's really exciting what's happening right now because creators are beginning to use the depth map output from apex converter to do incredible things. You know, just yesterday, you know, we were following a creator that is using them for mastery construction, using unity and creating all sorts of really amazing demonstration is another creator is doing this and mixing it with Adobe Aero to put, you know, create like a Jurassic scene and put dinosaurs into the scene.
And you know, so I think creators are really beginning to understand the possibilities when using, you know, depth and it's really exciting to be kind of a part of that, that creative process for them, not to mention just even storytellers that, you know, want to keep the integrity of the art, to generative art that they're creating, but add depth and feel to, to these, you know, short films, we call them vignettes.
It's almost like a graphic novel, but we're talking about introducing new genres of storytelling completely, you know, with some of these tools. So it's pretty exciting. Absolutely. That is so cool. And I'm curious, you were and you were mentioning about the explosion of AI in this last year. You guys started in 2014, is that right? Yeah. Leah was found was spun out of HP in 2014.
Yeah. Okay. And so that's wild. So 2014 to now 2023 in this last year of AI, how has the company and the use of AI just changed in that time? And I'm curious, how has this last year impacted your company and your team just with the growth So I think what's pretty interesting is that it's it's like completely revolutionized like what the company is, right?
So we are primarily a display company. We make the best displays in the world. They're 3D displays that you can look at and see depth without needing any glasses or anything like that. They're a flat display you can put in any device, smartphone, tablet, laptop, etc. And just to see the depth instantaneously. But it does require you to purchase a new next generation product to be able to see that, hey, I was kind of a means to an end for us, right?
We realized like, okay, we're capturing all this content, but like, for example, our first smartphone, it had a 3D camera on it, but we're like, man, Like the depth limited. It's really hard to make a 3D camera on a phone that's so tiny and have it have tons of depth. So we're going to use AI to kind of boost that depth.
We started doing that. We were like, Well, when you're boosting the depth, you're going to end up like causing some like stretching and some some smearing so that we could do some painting there. And then once you do the painting, you're like, Well, this is pretty cool. But like, what if you could like had an animation? And then we're like, okay, well, you know what if what are the unique things you can do with 3D?
It's like, Well, what if you could relight the image so you can just change the lighting and use that 3D depth to do that? And all these things kind of just kept evolving from like, okay, the camera needs the 3D to do this. And then it got to the point where we're like, okay, well what if we can make photos turn 3D automatically?
And then, you know, now we're even doing some really crazy stuff. Like what if you could turn your soccer live stream into 3D in real time and view it view like the same soccer livestream in 3D in a VR headset and our on our devices and you know, that's stuff that's ready and is going to be coming out in the next next few months, next year.
But I think the really cool thing about it now is that instead of AI being a kind of a means to an end, like a tool that just makes our existing stuff better, we're just like solves a problem for us. AI has become a medium within itself and people are engaging with it just because of the creativity and the freedom and the possibilities that it adds, right?
It becomes part of the process to playfully just try things and see, see how things look in different ways and just bring that joy in. Just engaging with the tool itself as opposed to the tool just being a solution to a problem. And I think that's really been the massive difference we've seen in the last year versus before.
I have so many questions based on what you just said. So first off, you just said that Leah has developed technology, the Luma iPad two, that essentially allows the audience to watch a 3D film or video without glasses. So I guess my first question here is how how how is that even possible? Yeah, it seems like that's just some sort of wizardry.
Yeah, I mean, the hand-waving answer is nanotech. That's like usually what I do because you don't want to get super deep into it. But we have a few different display technologies and ultimately, like I'm just gonna explain at a high level what you're able to do is if you have the right structures, you can bend light in different directions, like bend photons, like a very small scale.
And if we are able to bend the photons and send different light to each of your eyes, you're going to think that what you're seeing is 3D, because that's how the real world works. Each of your eyes gets light from every object from a slightly different angle and so we're basically just trying to send the right photons to the right eye at the right time.
And if we do that, then you're going to see 3D, even though the display itself is flat. I'm very curious because I actually have strabismus. Do you know what this is? So I have a lazy eye and I, I have never been able to see a 3D movie. So I go to the theaters. I can't like I put on the glasses.
It's just like the the screen's flat. So I'm very curious if you guys are looking at how to can folks with like conditions like mine see 3D with these pap or. That's a great question. Yeah. So effectively there's actually if people don't talk about this, but there are a variety of what we call depth cues in human perception.
And so the main one that everyone talks about stereoscopic, So stereoscopic is as a disparity in what your left eye and right I see. And a lot of people are stereo blind or have various eye health issues that makes one eye weaker than the other or one eye is offset from the other. And all these issues can cause issues with the perception of stereoscopic.
But so you ask me is not the only depth you there's actually a few other ones, there's accommodation and then there's crucially parallax. Parallax is like, like the easiest way to kind of see it is when you look into a far distance and you like move your head back and forth. You'll notice that like objects in your vision move at different speeds based on how far away they are from you and in human like human perception, what we've learned is like even when people think that they're like not moving, they're like, I'm sitting somewhere and I'm looking at something straight.
We're our whole like perception of the world is based on micro movements. We're always just slightly moving our head, even if it's imperceptibly. And we're doing that for both the ability to get these parallax cues and for our ears to be able to hear things from slightly different angles, which is how spatial audio works. So to to kind of answer that question, we are one of the first 3D platforms that supports both stereoscopic and parallax, which means even if you have issues perceiving stereoscopic depth as you move your head around, you're going to be seeing new angles on the experience, which is basically means anyone can can experience 3D.
That's amazing. All right, I need it now. That's amazing. So exciting. Yeah. I'd love to hear what you think of it once you get a chance to. To put eyes on it. Yeah, we definitely want to play around with one. I can. I can tell you that much. So you said something else earlier that I feel like might have flown under the radar a little bit but feels incredibly revolutionary in terms of the way in which we consume content.
You said that you guys are putting together essentially the software and technological structures in place to live stream what would be a 2D video image, and that would then get converted into 3D that you can consume in VR or in 3D using a device like the one you guys are creating. That seems like an incredible leap forward in terms of like the way in which we experience things that are live because, you know, we've all watched things that are live, whether it's twitch livestreams or even like concerts or sports or things like that on a TV screen.
It's fun and obviously, like it's a great communal experience, but it doesn't feel like you're actually at that event. From what you're saying, you're kind of beginning to simulate what it would be like to to be there. Have you guys played around with that live streaming and how does that change it and how do you, I guess, even emotionally feel consuming this 3D experience versus a traditional 2D experience in livestreaming?
Yeah, I mean, I think the one of the core benefits of having having a be part of your media decoding process is that you as a viewer can kind of transform it to your own desire. Right now we're, we're focused on just giving, adding that depth to things so that, you know, when you're when you're looking at a screen that has a soccer match, it's suddenly more looks like you're in the bleachers, in the stadium, looking down at the soccer players rather than like looking through a little rectangle at like a window, you know.
But, you know, in the future, that can be crazy things like if you're if you want to watch a sports game with your kid, but your kid doesn't care about cricket, you could be like, Hey, what if I turn every one of the players into a Teenage Mutant Ninja Turtle and then your kids can be watching with you and be like, yeah.
Like Rafael went and he just scored right there and, you know, get them really excited about it that way. But I think are going to enable all that. I think that the interesting thing about our technology is that it's really a bridge between the past and the future. You can take movies from 100 years ago like classic black and white movies and see them with full 3D depth and just see them in a completely new and exciting and interesting way.
But you can also bring those old films into new mediums like VR and AR, and watch them in a way where you really wouldn't do that normally. Most VR usage today is people playing VR games because those are made for VR. They're completely new novel experiences. If you want to watch a movie, you're going to take the headset off, go sit at your TV and watch the movie.
Right? But our technology kind of gives you a reason to keep the headset on. It lets you say, Well, what if I could watch that video of my best friend's birthday a month ago and see what that looks like from from this angle? Or what if I could see my favorite movie in a whole new way? Or what if I could, you know, instead of having, you know, a 60 inch TV that's in my living room that I sit away from?
What if I've got this, like, massive, like 100 foot theater display that's fully 3D with things coming out of the screen. And that's what our technology enables. And we have it working in real time. We have different models of it. So if you're on a mobile device, we have real time for photos and videos. It runs at 30 apps on the on mobile devices and it looks really good.
But the stuff that you're seeing on, for example, EPIX converter, it is running in the cloud. We're spending a little bit more time. We're not doing in real time because photos don't need to be 100% real time. But we give you really high quality depth maps, really high quality processing, but that's using the exact same tech just, you know, tuned for that use case.
And then we've got the real time stuff for like the sports live streaming. So that runs in real time 60 apps in our cloud, which means that if sports teams want to work with us or news stations or really anyone with good livestream content they want, they'll be able to hit our API, send all their frames to us, and we'll just return 3D frames and they can send those 3D frames anywhere they can send to YouTube.
They can set them to VR headset, they can send it to a loom pad to they can send them anywhere, use that same feed and also have the 2D there. So it's backwards compatible. So it wants to turn the 3D off for a second. And seeing 2D, they can't. There's no downside to encoding the 3D data using our formats.
So yeah, lots of options. You know, something that's really, really exciting for creators that have shot 360 video monosyllabic 36 in the past. You know, I was one of those people that was using like 3D printed, you know, through 60 camera holders and, you know, bought a bunch of GoPros, you know, basically dug some together and, you know, without shooting content.
Right. The idea of shooting stereo 360 at that time was like such a hurdle, right? I mean, not to mention the editing process was was was was very, very daunting, you know, and and still it's very challenging to be able to playback 360 footage instead so you just from YouTube is is absolutely like mind blowing to anybody that had that experience because you know of course you know you just it's a heightened sense of realism and you know as you mentioned, you know, putting on a headset, doing microscopic content like was fine and it was great.
And, you know, very fulfilling for people that went out and shot amazing films. But since stereo now is quite a treat, you know, and that's all, you know, think Sky. So you're saying that when you're consuming, let's say a 360 video on YouTube and looking around that that this technology actually converts it to where it feels not only immersive in terms of being all around you, but also it has the depth at the same time.
Yeah, you got it. So if you and there are like 3D 360 cameras but they costs like, you know, Insta360 SX model that does 3-D, The cheapest one I got is like $5,000. And then the high end, what they have is $15,000. And so if you're like a professional, you could have bought one of those and then worked with that.
And that's really hard. But with our technology, you can use a $400 camera and then using our air, you're getting up there and almost 1 to 1 matching the quality of like the $5,000 camera, which I think that's really one of the really cool things about a five, is once you have the models, the models are your software, right?
Like you can distribute it wherever you want, whenever you want, which helps kind of bridge the gap between people's hardware. Right? A lot of people don't have games that can sorry, they don't have PCs that can run 4K 60 apps. But if you have an A.I. that's doing upscaling from 1080 to 4K, suddenly you do have a PC that can run.
4K does the same thing with our technology. If you have a a cheap 360 camera that only cost you 200, three or 400 bucks and you're pretty sad. You're like, Well, everything's flat. It's not 3D. Suddenly I goes, okay, you're your cheap camera. It does the same things as the super expensive cinema quality camera. And I think that's that's kind of where the magic comes from.
It does seem like across the board. It's funny that you're using language that I've heard almost every single AI enthusiast and company bring up like words like bridge and amplify like, like it seems like A.I., what it's doing is taking these rudimentary tools and then it's putting them in a new league, a new category, like it's elevating what you've created and kind of putting it in these the the same quality where professionals used to live versus everyday creators.
And so it seems like it's very much bridging that gap and kind of shrinking that the divide and the difference between what an everyday creator can put together and what some of the biggest studios are capable of. And and I would say on that note, so you just said that a $400 360 camera can produce, you know, the same quality of a multi-thousand dollar camera, which is an incredible innovation, even if that's all you guys did like.
That's amazing, right? But it seems like from an audience's perspective and really, as you know, a filmmaker or an independent filmmaker is watching this, it seems like your technology then can be used or maybe one day will be able to be used to convert someone's film that was not shot in 3D at all into a 3D film that looks, you know, to your point pretty good.
Like where are we at with that technology as it relates to your software? Is that available to filmmakers? I'm really curious. Yeah, that's a great question. So we launched the real time conversion for video about I'd say like doing it locally is really the only way that I would do it, which we launched that about a year ago.
So you can have any movie, any 2D movie, any video clip, and if you had an honor device, our device would in real time convert into 3D for you. So that works with any movie that you've got, any clip that you took, videos of birthday parties and favorite memories of your friends, etc. it works, right? But what we're moving towards is within a few weeks we're going to be adding video to our Cloud API.
And once video is in the Cloud API, what we want to see happen is right now a big part of the reason why a lot of movies are not 3D, only the most expensive movies, right? Like the biggest Hollywood blockbusters, the the Marvel films, Disney movies, Pixar, etc. those all get 3D because they're massive movies. They have huge budgets.
It's worth it for them to do 3D. But for a lot of people it's not worth it. Do 3D because they're just too expensive to do so expensive to hire studio to do it. You don't know if you're going to get your ROI on that. But once our our API is video ready, the real beauty there is the ability to hit that API and get back an 80% production ready depth map offer every single frame of the movie.
And so instead of doing and going from 0 to 100, you only have to go from 80 to 100. So we just cut your labor costs in like, you know, 80% of the labor cost just disappear into thin air for like a really cheap price. Like our per frame cost right now for photos is like $0.04 a frame.
Right. This is, this is incredibly cheap. Normally people are paying anywhere from like, you know, 60,000 for like a short film up to like a quarter million, even like one or two or three or 4 million to do a 3D conversion of a movie. And, you know, we're cutting that down. And not only does it save you money, saves you time, if you have a movie and the turnaround the movie needs going on two months, lot of time, you're just like, Yeah, we can't do a 3D conversion.
There's not enough time to do it. Well, we just cut out 80% of the work. So suddenly you're like, Well, two months is. Yeah, I think we can do that. That sounds great. So it's pretty much ready to go. The tech is here. We just haven't put it on the server. We want access. Yeah. So can you talk real quick about technologically then what is the difference between the current stereoscopic conversion workflow versus how you imagine once you release these new tools, what that process would be like?
Yeah, so the way that a lot of people do it, assuming that you don't have a depth map from the movie originally or you're not doing stereo rendering, like for example, like historically for 3D animation movies, they would just do a stereoscopic rendering of the movie, which is pretty complicated. But they even stopped doing that. Or if you had like a marvel movie, they every single scene is recreated in full 3D anyway because they need to do that to do all the VFX and stuff like that.
So for those, those, the process is sometimes different, but the way that it works over 90% of the time today it has for the last like seven or eight years is that you start by getting just a 2D movie and then people go through and they, they use special tools and they kind of like rotoscope out each of the characters and sets and objects that matter.
And then you have to like, map them between the different scenes so that it like interpolate between them as they move. And that that even uses AI to kind of do edge detection and to like automate like, okay, that's a a person and that person's moving here and then you have to like set the levels of depth manually and draw that and then use the tool to interpolate that over and over again for it to all be set up.
And then, you know, once you once you've got it to a good spot, then you'll render it, then you'll like check it with your glasses based 3D display and you'll be like, that has an error. So you have to take it off and you have to try again and hope that you got it right. Sometimes you can do it live, sometimes you can't.
The way that it's going to work from now on is that you're going to hit our API. It's going to give you back to things, going to give you a depth map for every single frame, and it's going to give you a full 3D version of them. So you'll have a full 3D version of the movie right at the get go.
You watch the 3D version and you just take notes and like, okay, I want to change that. I want to change that. I want to change that. So you got like timestamps everything, and then you just go to those depth maps and you just like clean them up and then hit re render and then you get the full movie.
So maybe you only need to do like two revisions and you'll be good to go because you're starting out by being like I see in 3D here the things I want to change. You go change them, get the render. You're like, Okay, maybe I'll change this part and then you're done. So that that goes from all the parts of like having to manually find each object and track them between the frames you have to.
I just does it for you. It just gives you the full depth that well, I mean, it sounds like really what's happened is the the labor intensive and really tedious work of essentially rotoscoping, which is widely considered to be one of the worst roles in the process of putting together a film. It sounds like you guys are really elevating those artist who work on those projects to more be folks who are coming in and editing and polishing and really using their taste to refine as opposed to the hyper labor intensive projects that can be a little mind numbing, for lack of a better word, 100%.
And I think A.I. just does things that it doesn't even make sense to have someone do. Like I remember there's there's this one scene, and obviously I was just really very particular about stuff. There's one scene in Spider-Man far from home, where they're like, They're in the mountains and they're like in a bus. And I saw the camera panned over and I see some trees and the depth on the trees is just like completely wrong because they basically they didn't it doesn't make sense to do this.
They would have had to go through and like manually, like have every single leaf and set the depth for every leaf and all the things through it. So they were like, it's just kind of like a short scene. What if we just kind of fuzz it? But then the problem is, is that if you look through the leaves instead of the background of being far back like it's supposed to be, the background is just at the exact same place the trees are, which doesn't make any sense.
Right. But I would just do that for you. So and even if A.I. is like, let's say, like a little wrong, like, so that is 20% wrong. It's way easier to just be like, I'm going to go clean up a few of these rather than go, I'm going to go tag every single leaf and go tag the background and try to do some color correction and set the segment based on on the color values and all that kind of stuff.
And, you know, I think people could still do that if they, you know, hate their lives and want to do a bunch of needless work. But I'd rather just make that easy for them. I want them to focus on the parts that matter. How do you make the 3D look really cool? How do you make it so that when Spider-Man's shooting his web, it comes out of the screen and just looks beautiful and feels like you're going to get hit with the web if you don't move out of the way.
Those are the things they should be spending their time on. Not like little details about why that tree looks bad when when people see it in the theater. I love that. And I feel like that really hits on kind of the ethos of what we talk about here at Curious Refuge. And I'm sure the folks at Layla feel the same way.
You know, A.I. really is a it's a tool to help elevate everyone and to help elevate their not only creative potential, but to allow them to introduce taste into the process much sooner. So you're not having to worry about the utility. You know, sit down either, like creating the thing 100% from scratch. You totally have that right and you're able to do that.
But now you're actually able to interface with these tools. And it's like working with an assistant, a creative assistant that can can do the the parts that maybe you aren't entirely passionate about, which is super cool. Yeah. I mean, I think one way to look at that is that people always say that, you know, the best camera is the one that you have with you.
And I think the reason for that is that, you know, even as creatives, you know, people who are super passionate and want to build things in the world, the more friction there is, the less of the thing that you can do and the less you want to do that thing. And I think that what our tools do is they they make they free from all that tedium.
They free you from from the difficulty of that process and let you just create. I think that one of the reasons we really developed A.I. is because we realized that, like if we had a smartphone with a three display original and was not going to have a 3D camera, it was just going to have a 3D display. And it was like, Hey, you can watch three movies on it.
If you have a 3D camera, go take photos and you could bring it on to the device and view them, but you can't take 3D photos with it. And the realization was like, That doesn't work. Like if if people have to go do that, they're just not going to take three photos. There's not going to share three photos.
So we should put the camera on the device. But once you put the camera on the device, you're like, Well, a smartphone is really small. We can't make something where the depth is really far. And so the AI enabled us to put a small baseline 3D camera into a smartphone and let people take three pictures, which then has turned into one of the biggest open 3D image datasets in the world that's based on based on our social network effects and has allowed literally hundreds of thousands actually at this point, millions and millions of images to be uploaded through the limbic system and hundreds of thousands of active users.
We have, you know, millions of total accounts, but 100,000 active users. And none of that would have been possible without this A.I.. And I think reducing that friction was really critical to that. I think Marlon was going to say something about that, too. Yeah, I think it's just such a wonderful time to be a general, to be a creative generalist and to, you know, be inspired and know that you can take action on those inspirations using all of these tools.
You know, I think, you know, this is such an incredible time to be a creative person. And I think while there's still a lot of merit in like specializing in different skill sets and different tools and techniques, one thing that's really inspiring for me is just seeing people that come from so many diverse backgrounds. You know, maybe they didn't have like a super technical background, but they're inspired to tell a story and they're able to just go ahead and do that.
I know for myself, you know, I didn't come from a tech background per se. I started my career in theater and, you know, as immersive media, you know, started to bubble up. You know, I just started going to meet ups and I started to get whatever equipment I could get my hands on, you know, and this is going back to like Google cardboard days, right?
So where we're at now, where any content, you know, we're all, you know, clearly 3D and not to mention all the other, you know, experiments and really, really exciting projects that you can do with depth just just makes it just an incredible time to be a creator. That's so cool And I love how you both have talked about reducing friction because I feel like in many ways the film industry is an industry of friction, right?
Like, like the whole thing. There's, there's this, whether it's technical barriers, political barriers, social barriers, there's just financial barriers. Like, like there's just so many spots and opportunities for a creative project to not happen. Like it really is frankly, a miracle that that projects even can make it to the theater because of the amount of work and the challenges that creative projects have to go through to get there.
It really seems like technologies, like the ones you're developing and like we've seen over the last few months and years, are reducing that friction. And I would imagine that that a lot. To your point, Marlon, general creatives who now will be able to get higher level stories and maybe even like feature films out there in using technologies like 3-D for the very first time, it sounds like it's a very exciting time to be a general creative.
Yeah, not to mention, I think one of the things that got me started in the 3D display space was just the fact that a lot of my content was only viewable in a headset. And as we all know, you know, not everyone for different reasons, either wants or can just put on a headset, right? And so I was looking for other modalities to begin to showcase, you know, my immersive work.
And so what's really exciting for me is just a world where 3-D is just so accessible and, you know, available in your pocket, in your backpack, on your desk, you know, in your living room. And it's it's an option. I mean, I think it's just just a really, really exciting time, you know, to be able to showcase, you know, immersive content in ways, you know, outside the headset.
I'm curious, is there a point in time where you see that folks will be able to gather around a TV in the same room and watch something maybe using your product from the same screen, or will they always need to have their own headset or their own little iPad? Yeah, I mean, I think it's it's inevitable that that's going to happen.
I think that the big every every thing that you do, whether it's a creative work or whether it's a hardware product or whether it's a software product or whatever, is all just a set of tradeoffs, right? Like what can the market currently bear? And there are other products on the market right now where you can have multiple people view it in 3D at the same time.
And our opinion is and it seems like the market's opinion too, is that it's the quality is just not good enough. So you're going to end up by doing that. You end up getting a much reduced resolution. And there's a lot of other complexity too, because, for example, there's not really any way to have it work that way and also have it be 2D, which means you basically be swapping TV out for a TV that's only 3D and you can't do 2D on it, and then it's going to be really expensive and it's not even going to be very sharp.
So if you want if you're willing to make all those trade offs, there's a product on the market for you right now today. But instead what we want to do is we want to focus on products that people usually use by themselves anyway. That's going to be their smartphone, that's going to be their tablet, that's going to be their laptop, that's going to be their computer monitor and all these things.
99 of the time you're using it completely by yourself. You're not showing someone else. And even with like a phone or tablet, if you want to show someone, you usually just give it to them. You're not going to be like crowding around and seeing what they're doing. You're like, I set this up and I just want you to see it.
And I think as long as you're willing to deal with that one trade off, which is that it's one person at a time, there's no other trade off with our system. It is beautiful. There's you get full 2D quality, you get super high resolution, you get an experience you don't get from any other display, which is that it's has this incredible 3D depth to it and it comes with tons and tons of content and tools that let you have unlimited content, any photo or any video that you've ever taken or you can ever find on the Internet just works in 3D on our device.
So yeah, I think we're focused on that. But to answer your question, yeah, in the future, if we can get to the point where I think probably you get to either 16 K and if not 16 K like 32 K displays, yeah, it'll be much, much easier for us to be able to do something like that. I have a very quick question.
So let's say that I'm a filmmaker and I have a short film that I've created and I want to watch that film in 3D. Can you give me a timeline of how long it would take to convert it to 3D and to watch it in 3D using? Your technology? Yeah. So so just for the sake of this, where is your movie?
Like, is it on is like on a flash drive, is it on a computer? Is it on YouTube? And just answer that and I'll tell you exactly how you can do it. I have a film on YouTube. How long till I can watch it in 3D? Yeah, you open the YouTube app on the launch pad too. You hit the share button, you hit the layer two button, and then 5 seconds later, you're watching your movie in 3D.
So you're saying that as a filmmaker, I can convert anything I've ever worked on into 3D in 5 seconds, in real time. In real time. It just takes the reason it takes 5 seconds is because that's a buffer, just like 2D. So that's the buffer. Your your your video from YouTube. So instantaneously, actually, yes. Anything to be converted in real time.
So faster basically than me walking to my front door and welcoming someone in my house. I can convert a film that I've worked on into 3D. That's incredible and it looks great. And our data shows that this is the primary way that people use our device, their they love watching all the YouTube content that they normally watch and watching in 3D.
They prefer it to 2D. It's that sticky. Now, I mean, I'm going to add my caveat, which is that if you're an artist and a creator, the real time mobile conversion that we have, it's going to look really good. But it might not be the perfect quality that you want. So you might want to use our API and get in video using the cloud to do the conversion instead, which is going to be even higher quality.
But both work. It's just like what tier of quality do you want? And in both cases you can either have it be completely instant or you can have it spend more time to get you even higher quality if you like that. But the answer is, is in the in the best case, you don't have to wait. You can convert right now real time as much video as you want, live streams, videos that you take in 4K, it all just works.
Where do I sign up? Where do I get one layer income? And you can get yourself a lump out to or if you want to build an app or a system using our APIs, you can go to cloud layer, pick WSJ.com and that will give you full access to our APIs. Right now we're focused on the photos and the animations like I said, but within a few weeks video is going to going to go there as well.
And if people want to work with us more closely or they have a special thing that they want to work on, they can just hit us up business at Intercom and we'd love to work with them on that. And you know, we give people access to it early. If someone is like, Hey, I have a movie and I want it to be in 3D in one week, email me and I'll make it happen.
But if you wait a few weeks, we'll be on the API and anyone around the world can access it and convert whatever they want at any time really easily. Well, I mean, that sounds like a really great way to begin to wrap up this conversation. I have one quick question and then I think Shelby had some rapid fire questions for you guys.
So my last question really is, I guess if we could all just use our imagination a little over the next 12 months, I, of course, is going to continue to progress not only at Leah, but just for the entire creative world. What do you think are some dramatic changes that we're going to see in the creative process? What are some innovations that haven't become consumer facing or creator facing at this point that will begin to be unveiled that you guys are excited about or you think might really change the landscape of creativity?
Molly All starts off well. You know, from a creators perspective. I think there's going to be an intentionality. I think, you know, we've kind of we are experiencing a time right now where, you know, you were kind of poking at what's possible and just amazed at the ripples. Right. And right now there's a saturation of content. You know, that's that's generative.
That is, you know, I hired, let's call it. And I'm really excited for for creators to really think deeply about what they can do with these new capabilities or what stories they want to tell, you know, from the heart, you know, with a lot of intention and you know how these tools are really going to unlock those visions.
You know, and I think back of my favorite directors, Akira Kurosawa, you know, Alejandro Jodorowsky. Wow. They had this technology. Can you imagine, you know, what Holy Mountain would have been like, You know, with 3-D in mind? You know, what movie Like dreams, you know, Martin Scorsese and Cassava together, you know what I've done. And so I'm really excited for for, you know, filmmakers, creators, designers to to now really think intentionally about, you know, how they're going to use these tools for storytelling.
You know, what that's going to unlock for them and for us to maybe even see like new genres, new formats for for, for media that we haven't seen before. Yeah. I will say, you know, I'm a former Grammy Awards manager, so I'm very familiar with the institutional space and how it what it takes to change in institutions. You know, rules in terms of entering, you know, media.
And I can tell you right now there are serious debates happening. You know, Academy of Motion Pictures, the Recording Academy about the use of AI for creative products that are being submitted for the highest awards. And there is going to be a very, very, very cool future, you know, for this type of media, you know, as it as it rises, you know, you know, to these institutions and, you know, ultimately out in public mass, I mean, I want to have a whole nother conversation with you, Marlon, about managing Grammy Awards.
That's wild. What a job. Absolutely. Yeah. You know, institutions in many ways, you know, they are a like a, you know, kind of serve as like a yardstick for social and societal change, you know, and AI is disrupting, you know, it's it's really kind of bending the rules, changing, you know, what it means to be a creator, what it means to make a film, what it means to me, you know, any recording, you know, what is photography and all these sorts of things, you know, we're starting to see incredible breakthroughs.
So, you know, coming from from from that and and seeing how fast the technology is moving and how creators are, you know, just gripping it and running with it, I think is going to really, you know, shake up a lot of these institutions. And yeah, we're going to see like incredible things come to light. I love that you brought up Kurosawa because I'm so curious.
I've had this nagging suspicion that the very best filmmakers in the world, because of that creative friction that we talked about earlier, we've never actually seen their films before because they haven't had the creative tools. They haven't had the opportunity to get their story out there, whether it's because of finances, life situations, things like that. I'm curious if you think that what's going to happen moving forward is just the great democratization of storytelling and how, you know, we truly will now begin to see the very best stories coming from the very best storytellers and do you believe that those storytellers are going to come from places that you wouldn't have expected?
Like they're not all out here in Hollywood, right? Like, where are we going to see these stories? And like, do you have any sort of idea of like some under represented stories or some some like artists trends that you've seen potential and absolutely, you know, and it's it's so, so cool that you mentioned this because there's a really great story about Akira Kurosawa in the production of Ron.
I'm not sure if you're familiar, but the production was delayed due to finance and crew saw began to illustrate the scenes. And there is if you look this up, it's incredible. There's a whole collection of these like, beautiful storyboards for the movie. Ron You know, my favorite movie of all time. And the way that he was able to, you know, actually manifest these characters in these scenes, you know, from his visions in doing these illustrations is absolutely wild.
And today we can see how someone would pick up, you know, for example, you know, we have an out called Dream on Loon Pad to where you can just, you know, begin to prompt, you know, different scenes and ideas. You know, it's powered by steel diffusion. Wow. What, what, what, what what a concept of knowing that there's nothing that's going to stop you from making your film.
That's nothing is going to stop you from getting your idea. Right. And, you know, I've been part of different, you know, creator cohorts of, you know, you know, working with, you know, communities that that are, you know, not the typical not the usual suspects. Right. That that are able to, you know, produce these things. And I think we're going to start to see a lot more of that, you know, as production costs go down.
As you know, the accessibility of technology, you know, is this more abundant? You know, we're going to begin to see many more voices, you know, into the space. And so where these things are going to, where the where these, you know, works are going to come from, it's going to be every corner of the world, you know, And that's what's really exciting, you know?
So, you know, I can't wait to continue seeing this way, you know? And yeah, I'd love to know what Nima thinks about all of this, too. Yeah, I think kind of the things that we're going to really see in the short term is I think we're going to I think they're going to be like two kinds of big changes.
So we're already seeing Mid Journey is making 3D a core part of their pipeline. So depth maps are just becoming a part of how you prompt images. So I think that we're going start seeing the evolution of literally all media having 3D. And it doesn't mean that you're going to be seeing it in 3D. What it means is that we're going to be upgrading all of our existing files to have depth maps built into it, just because that gives you significantly more optionality.
And that's just the direction we're moving into the future. So we're going to see that process in one year is almost everything. All the new media that's being generated will have some sort of depth map, whether it's from iPhone 15 pros, you know, spatial video and spatial or whether it's going to be captured on a headset or whether it's existing, you know, family photos and videos from like the seventies, eighties, nineties that you're now just upgrading that to have 3D.
We're going to see the media do that. And then then the next one is I think that we're going to start seeing AI video co processors and these are going to be there's some device already have them, but going to start seeing them become standard, they might become part of everyone's TV box or HD receiver or even part of your TV or part of your phones and other displays where this entire chip, the whole focus of it is doing various A.I. upscaling on your content, whether that means you're going to be able to take black and white movies and make them in color in real time, whether you're going to be able to take low
res standard def and make it 4K in real time, whether it means you're going to be able to turn it, take 2D and make a 3D in real time like what we do, whether you're want to be able to do style transfer on it in real time, meaning like you are watching a show and you're like, Hey, I want this in the style of Akira Toriyama.
I want to see this look like Dragon Ball Z, where I want to make it look like it was shot film, you know, like all this type of style transfer relighting, things like that. I think you are going to able to change that. Just like today. They can change the brightness and contrast on their on their TV. You're going to have all these new AI enabled options built into your displays that will let you transform whatever media you want to fit your desires and wants and needs at that exact point in time.
And I think that's that's where moving in. Can I ask you, where are we technologically? How far are we from these chips, these devices being able to do that? Because right now, if I want to, let's say, upgrade something from 1080 to 4K, I have to use a third party application. It takes forever. My computer hates me, crashes.
Half the time. It feels almost impossible. But are there other developments that have been coming out and just like like where are we? Is this like 12 months away that we will be able to adjust the resolution like we adjust our volume? Yeah, I mean, we're already seeing a lot of solutions that, for example, like Samsung TVs and LG have been releasing some upscaling tech on their televisions for for quite a while now.
I think one of the big differences that's kind of hard to communicate is that a lot of our computing devices are general purpose computers and because of that, the CPUs and GPUs and even even the ML cores that are built into your ASIC like those are not meant for specific tasks. They're meant to be able to be manipulated and programed to do a million different things.
Right? I think one of the reasons people have lots of issues with doing conversion on current devices like I do too on on all my devices is it might have extra general purpose. They're not meant for conversion. I'm forcing my GPU to convert something which is not what it was originally designed to do. Same thing with with our technology, like none of the computers that we use and the chips we use were designed to turn 2D and 3D.
We're programing it to do that. But if you made custom hardware that was designed for video upscaling, which we've seen some people do we see something to do with FPGA, especially in the retro gaming scene? There's like the open source scan converter, there's the retro tank 4K that's coming out and all these are FPGA based solutions that are being programed to be their own chip.
And the chip is hyper focused on just doing video transformations and different types of scaling, whether it's upscaling the interlacing, whatever, it's going to be. Right. And so I think to answer your question, it's like the technology is here, who is going to start shipping it? Who is going to commit to building, you know, a million units of an upscaling chip that has, you know, X, Y and Z feature and say do TV manufacturer, you monitor manufacturer, you phone manufacturer.
You need this chip inside your device. So it needs to pull the trigger. Feel like they have to know it's the future and make it happen. But I think that technologically we're there. We just need someone to commit and have some conviction. You know? That's amazing. I, I don't there's one more question. I feel like I really just want to hear from you guys on and it's just working with AI.
How does it different doing work with AI versus traditional applications? I feel like I'm just so curious about that experience. I'd say the biggest difference for me is how hard to predict what you're going to get. Like we're so used to computers being very digital and it's like, I want this output. The computer is going to give me this output.
Even when it fails, give me that output. It fails in a consistent way. It fails in a way that's deterministic. I'm able to I'm able to understand like why I did that. But the thing about AI is it's like non deterministic by design, right? You want to give it different seeds and get different outputs and you just don't really know how to even even the people like creating the model.
They don't always know why things happen the way they do. You have to like research it and go back to each of the layers and try to understand why it's making these changes. And I think that non deterministic aspect of it, it's a lot more like working with people than it is like working with computers. Because with computers I'm I am doing X because I want Y and I have to learn how to do X the right way so I can get y.
But with AI it's like, Hey, I want this thing. And you know, people sometimes don't don't understand what that means, and neither do they. I mean, like one of my favorite examples of like this keeps happening in like people meme about it, but it actually happened in my school where like a teacher said, like we were going to we had to do reports on euthanasia.
Like it's a very serious concept, but people keep making reports that are about youth in Asia, like children in Asia, and they like completely misunderstand what the assignment is. And I feel like AI has that exact same problem where like you think you're telling it exactly what you want, you're like, I'm being clear and they just completely misunderstand and go, go do the wrong thing.
So it's a lot more like working with people in that way. That is so funny. I don't know where we heard this, but I don't know, Caleb, if you just heard seeing this or heard of Sarah, but it's like, who needs throw a salmon into a tornado and expect sushi. This is like working with A.I.. Yeah, that was Pixar talking about their work on Elemental.
They used some of that. Yeah. That's so funny. I mean, this is such a fine conversation. Thank you both for your time. And, of course, thank you. It's been amazing. So, Caleb are you okay if I move on to rapid fire questions, let's do it. All right. So I have two, three, four, five, six questions for you. And it's kind of yes or no.
You just answer yes or no. You each can answer. Okay. So Rapidfire questions. The first one is, is watching 3D on a loop iPad smoother than watching a 3D film in the theater? Yes. Should artists be scared of A.I.? No. Is the media overly critical of artificial intelligence? No, because I think whenever something is, you know, revolutionary, people should not criticize it, but they should be skeptical.
And that's why I think it's okay. I mean, I don't think they're like people who are like haters. I don't think they're going to succeed. They're going to fail. A.I. is the best, but I don't know. I think it's unreasonable to think that people aren't going to be scared about the possibilities. I think that's that's reasonable and it's fine.
And, you know, it's kind of interesting to hear, hear both sides share this. 1972 Alvin Toffler, my favorite futurists, wrote the book Future Shock, and he basically said, future is either something you're going to acclimate to or something that's going to happen to you. Right. And there's no theory of, you know, transformational change. Economic change is not a theory of conflict.
Right. And so right now we're entering a period where there's going to be radical transformation in every industry. And of course, there's going to be things to iron out. Right. There's there's all sorts of really, you know, thoughtful considerations that people are bringing up right now. And that's just a part of the process. Yeah, That's so good. I want to read that book now.
Sounds amazing. It's aged very well. We'll put it in the show notes for everyone. There will be like, okay, are your developers actually magicians? Not all of them. Only a few of them practice magic regularly, but if they do that, I'm sure they use 311 cards because they're the coolest. Do you believe aliens are real? Yes, I think I think it's statistically unlikely that they don't.
We're the aliens. Last question. Do you guys believe that we're living in a simulation? There's really compelling evidence. I guess I'd say I'm agnostic on this. I don't I, I don't know. One, I don't really care. But I'd say there's really compelling evidence for there's not a lot of compelling evidence against. But it's hard to hard to believe when I don't have tons and tons of evidence.
Yeah. I like to think where the director of our movie. So is that. No, not a simulation, right? You know, it's it's it's, you know, it's certain things are just so hard to comprehend. But I focus on the things that I have control over, you know, And I think whether whether we're in whether or whether in, you know, video game or something, I think ultimately I do believe in and freedom and yeah, our ability to change our circumstances.
Yeah, I align with you, Marlon. Caleb, I'm curious your answer. I've never heard this. Do you believe we're living in 100% simulation? Yeah, no doubt. You see glitches in your video. Yeah.
Well, thank you so much for hopping on the podcast. This was really a fascinating conversation, and I know that our filmmaking audience and the creators that are part of our community are absolutely going to be interested in testing out your tools and in all of the developments that you guys have coming down the pipeline. This is really interesting stuff.
And I mean, I really like as someone who, you know, consumes media, I just like, can't thank you guys enough for the work that you're doing to push just creative entertainment forward. It's really awesome. Thanks so much for having us. It was really fun conversation. Got me thinking about a lot of possibilities. Absolutely. And thanks for teaching. You know, so many people, you know, all these amazing techniques.
Yeah, no problem. I mean, it's. It's in my DNA for sure. Yeah. To be a bridge. That's our hope, right? So from that past and future. So I love that. Well, thank you both. Caleb, would you like to wrap up? Thank you so much for listening to this episode of the Curious Refuge Podcast. If you want to learn how to do AI filmmaking for yourself, you can of course check out our boot camp over on Curious Reputation.com, where we cover all of the tools available to you as an aspiring A.I. filmmaker, including Leah Pics, and I'm sure some of their new technology that will be coming out in just a few months.
Thank you so much for listening. We'll see you in the next one. Thank you. See you soon.
