It would be nice to have some idea going in for what the requirements are.
- I have to admit, I don't understand why I can't preview the 3D experiences right on my computer.
- Likewise, when I try the QR code on my iPhone 6 ... well, I'm hardly surprised to not see anything in 3D, but it wouldn't hurt to have a little more detail than "Scan this QR code on your iPhone or iPad."
I suppose I could mention that I've done some game development and a little bit of VR/AR, but have never heard of the USDZ file format before.
USDZ is the format that Apple is backing for VR/AR delivery (developed by Pixar)[1]. We're betting that it is what is going to be used in any Apple headset.
Right now you can't easily view USDZs in the browser (the format is a moving target), however Apple is making strides to make it available with the proposed Model element[2].
We were doing some USDZ --> GLTF converstion (GLTF being a format that will work in the browser), but it is jankey (technical term).
I have not tried an iPhone 6 yet. I will have to pry it from my children's hands and remove a layer of cheese and excretions before I do.
I've made a video on how to do this via the browser here[3].
TBH, the experience a heap better in the iOS App (currently in testflight) [4].
Hey very cool stuff, one piece of feedback it would be good to be able to delete models on the beta iPhone app, or if we can it’s not particularly obvious how to do so.
Anyway to take a model that was created and get it into CAD / modeling software? I don't know much about VR/AR but I know it is a pain in the bum to replicate an object / create a model for 3d printing.
If it's not a secret, I'd love to read some design document about the backend behind this app. What's the blackbox that takes set of images as an input and produces a 3D model as an output?
He asked for the “blackbox that takes set of images as an input and produces a 3D model as an output.”
That black box is Object Capture API, which takes a folder of images and outputs an USDZ (or OBJ) file. Model I/O isn’t necessary.
Kudos to Apple for releasing a photogrammetry pipeline that allows web and app developers to build tools like this, with zero knowledge of 3D reconstruction required.
I’ve just been trying out the photogrammetry session, it’s super simple to use. I think Apple’s photogrammetry software isn’t as advanced as other’s, but it is free and works well small scale.
I work in the same field, making a capture app for photogrammetry software
How are you handling spam or other kinds of content moderation? Is it easier because users don’t upload a regular text/image and need to create something custom?
At early stage you are not going to get spam anyway. Once it grows (and i hope it does) you will get fare share of spam. As you said at that point you could check solutions like Akismet, OOPSpam.
If you don't have an iphone, you can view he model by downloading them in the "options" drop-down menu to obtain the usdz file, which you can then use either with Pixar open-source suite https://github.com/PixarAnimationStudios/USD or other software that support the format.
Very cool, reminds me a bit of the photogrammetry software Autodesk has. Always thought it could be more "consumerized" so that a series of photos can easily be made into 3D models.
Very Interesting project! esp. when compared to the swathe of junky or overly complex photogrammetry tools out there from a few years back (but mostly still polluting the app store)
I've only recently been trying out poly.cam which is probably the best example of a user friendly workflow ... but it's knowingly priced. Have you guys got a pricing model in mind? Also do you have any documentation around best practices/results, etc?
Anyway sounds like things are going to pick up again quickly with apple's new APIs. Good luck - exciting times ahead!
Hey Nobbis, I've just requested an invite to your SDK competitor to Apple and others. What you've done looks great and I have a few questions:
- Does your solution have the same photogrammetry requirements as the Apple/others?
- Is there a reason you're only supporting iOS at this time? Is that because you're using an iOS framework, or because you're relying on a file format (say HEIC)?
- How long until you guys open up the API?
- Avoid thin objects
- Avoid reflective objects
- Avoid transparent objects
- Avoid objects that deform
- Hardshadows are not good, use soft lighting
- Your photos should have a crisp focus and be well lit (across the object)
- Things work best when the background has a solid colour
- Make the object take up the full frame (more data for the algorithm)
- Make sure to get shots from all sides
Is there something like neutral lighting where a texture can be photographed and then the engine's artificial lighting makes it look realistic post lighting?
Generally for photogrammetry you want to aim for more diffuse lighting. Avoid direct light sources as they will cast shadows onto your subject that get baked into the texture. Cloudy days or light diffuser umbrellas are helpful.
There are methods for de-lighting, but it’s better to just capture without if possible.
Or if you want to get fancy you can build a capture rig to create intentional shadows and process them into texture normals with something like Substance.
This looks great, congrats! I’ve tried a number of photogrammetry software and they typically had challenges. One question: are you able to determine the size of the object?
Nice! I have been "borrowing" the kid's toys too.
With enough photos you can get some great shots (such as this one)[1].
Looks better as a USDZ, than a GLTF.
This looks great. Unfortunately the app is not working for me on an iPhone SE 2020. I am getting an ‘Error accessing your camera. Please check your permissions’.
The permission to allow Hologrid to access my camera is correct in settings.
Testing it… I’m not sure. I tap on the 3d cube logo in the picture of an object, and then “camera screen” pops up, but it’s not the camera app, it’s safari, I see that in the apps list (by double clicking the home button). Nothing in downloads either. Very strange.
It will. We've been building the mobile app using a cross platform framework (Capacitor) with the view of making it Android compatible. Capacitor does the job, but we've had to do some native code to fill holes.
You could probably whip up a quick script that runs on Android. My first go of doing mobile (for enlisting my co-founder) was to use Apple Shortcuts w/ the API.
I can downvote comments but not stories. Is there a karma threshold for downvoting stories?
EDIT:Ugh, sorry about this comment. Looks like I somehow posted it under the wrong story. No clue how this happened. Looks like I'm unable to delete it as the deletion time window has expired.
We are "very alpha" right now.
Edit: We have an API over at: https://usdz.app/api