Poly Keyboard makes it possible in XR, to switch prints on physical keyboard.
If you uses more than two languages on a daily basis, this must have happened to you: You are used to where certain keys are in one language but can not find them on the keyboard in the other language.
​
Using English/German switch as an example, because the "Y" and "Z" are swapped, I am either typing it wrong because the physical keyboard has German printing yet it is set to type in English, or both are in German but my fingers still remember the English key positions. On a normal day, this typo can happen over 20 times.
​
​
Part Two
Environment AI Model Training
1.
Experiments with untrained AI
An untrained AI doesn't understand what an HDRI image is and how it should look.
An HDRI is a 360-degree image that captures all environmental data, including lighting. However, Midjourney mistakes the texture sphere for the meaning of HDRI, making the generated images unusable.
​
Even using a real HDRI image as an embedding did not improve the result.
Generated with Midjourney text prompt:
an hdri texture from hdri haven, urban street --ar 2:1 --v 5.1
Generated with Midjourney image embedding and text prompt:
an hdri texture from hdri haven, urban street --ar 2:1 --v 5.1
Using ControlNet can produce usable HDRI images, but always too similar to the ControlNet embedding, even when in reference only mode.
Generated with Stable Diffusion txt to img + ControlNet:
an hdri texture from hdri haven, urban street
ControlNet embedding
2.
Train Stable Diffusion Model for generating HDRIs
After the initial tryouts with Midjourney and Stable Diffusion, it became clear that training an HDRI AI model is necessary for generating high-quality environment in XR.
​
The first model was trained with 70 HDRIs downloaded from HDRI Haven using LoRA.
​
Using the first model, more HDRIs were generated and put into the dataset for the second model.
​
The resulting second model worked quite well for generating outdoor HDRIs.
The trained model OutdoorHDRI
Part Three
Image to Music
1.
Train Stable Diffusion Model for generating HDRIs
Turn on audio and watch the volume
Environment and music mapping test in Unreal Engine 5, Meta Quest Pro
For testing, music were generated from two AI-generated HDRIs using the "Image to Music" API by fffiloni on huggingface.
​
It uses CLIP Interrogator to caption the image, and then run the caption text through Mubert to generate music.
The resulted music loop was satisfying, but it took quite long which has to be improved in the future.