Bibliography
https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
https://github.com/aras-p/UnityGaussianSplatting
https://www.youtube.com/watch?v=KFOy354zf9E
Hardware
- You need an Nvidia GPU with at least 24GB VRAM.
Software
Git https://git-scm.com/downloads
Once installed, open a command line and type git --version to check if it's working.
Anaconda https://www.anaconda.com/download
It will install all the packages and wrappers that you need.
CUDA toolkit 11.8 https://developer.nvidia.com/cuda-toolkit-archive
Once installed open a command line and type nvcc --version to check if it's working.
Visual Studio with C++ https://visualstudio.microsoft.com/vs/older-downloads/
Once installed open Visual Studio Installer and install Desktop development C++.
Colmap https://github.com/colmap/colmap/releases
This tool is for creating camera positions.
Add it to environment variables.
Edit environment variables, doble click on "path" variable and add a new one and paste the path where Colmap is stored.
ImageMagik https://imagemagick.org/script/download.php
This tool is for resizing images.
Test it by typing these lines one by one in the command line.
magick logo: logo.gif
magick identify logo.gif
magick logo.gif win:
FFMPEG https://ffmpeg.org/download.html
Add it to environment variables.
Open a command line and type ffmpeg to check if it's working.
To convert a video to photos, go to the folder where ffmpeg is downloaded.
Type ffmpeg.exe -i pathToVideo.mov -vf fps=2 out%04d.jpg
Finally restart your computer.
How to capture gaussian splats?
Same rules as photogrammetry but less images are needed.
Do not move too fast, we don't want blurry frames.
Take between 200 - 1000 photos.
Fixed exposure, otherwise it will create flickering in the final model.
Processing
Create a folder called "dataset".
Inside create another folder called "input" and place all the photos.
Now we need to use Colmap to obtain the camera poses. You could use RealityCapture or Metashape to do the same thing.
We can do this from the command line, but for simplicity let's use the gui.
Open Colmap, file - new. Set the database to your "dataset" folder and call it database.db Set the images to the "input folder". Save.
Processing - feature extraction. Enable "shared for all images if there is no changing in zoom in your photos". Click on extract. This will take a few minutes.
Processing - feature matching. Sequential is faster and exhaustive more precisse. This will take a few minutes.
Save the Colmap scene in "dataset" - "colmap". (create the folder).
Reconstruction - Reconstruction options. Uncheck multiple_models as we are reconstructing a single scene.
Reconstruction - Start reconstruction. This will take the longer, potentially hours, depending on the amount of photos.
Once Colmap has finished you will see the camera poses and the sparse pointcloud.
File - Export model and save it in "dataset" - "distorted" - "sparse" - "0". Create directories.
Train the 3D gaussian splatting model
Open a command line and type git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
This will be downloaded in your users folder. gaussian-splatting
Open an anaconda prompt and go to the directory where the gaussian-splatting was downloaded.
Type these line one at a time.
SET DISTUTILS_USE_SDK=1
conda env create --file environment.yml
conda activate gaussian_splatting
Cd to the folder where gaussian splatting was downloaded.
Type these lines one at a time.
pip install plyfile tqdm
pip install submodules/diff-gaussian-rasterization
pip install submodules/simple-knn
Before training the model we need to undistor the images.
Type python convert.py -s $FOLDER_PATH --skip_matching
This is going to create a folder called sparse and another one called stereo, and also a couple of files.
Train the model.
python train.py -s $FOLDER_PATH -m $FOLDER_PATH/output
This will train the model and export two pointclouds, one at 7000 iterations and another one at 30000 iterations.
Visualizing the model
Download the viewer here: https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/binaries/viewers.zip
From a terminal: SIBR_gaussianViewer_app -m $FOLDER_PATH/output
Unity 2022.3.9f1
Load the project. https://github.com/aras-p/UnityGaussianSplatting
Tools - Gaussian splats - Create.
Select the pointcloud, create.
Select the gaussian splats game object and attach the pointcloud.
Do your thing!