One of the things that I was really impressed with during the hands on was the ability to resize and position windows in a very natural and intuitive way. This is in contrast to how it works in the simulator which is a very clunky and frustrating experience trying to position things, particularly at specific depths or orientations. This is one area that the simulator fails to simulate the on-device experience.

I have been exploring having different independent windows showing information, as well as content better adapting to the resizing of windows. For example, if a user wants to resize the main game window, which usually shows the run worm and ball-by-ball information, it adapts to being short and wide by showing a simple game banner.

I’m also experimenting with having the ability to pop-out this content to have both views visible at once. Whether this will be a useful approach or too confusing in practice remains to be seen. This approach takes more work, but is a better experience than either artificially restricting the size of the window, or trying to compress a lot of information into a small space.

Practically the way this is achieved is liberal use of ViewThatFits, to provide alternative views for different sized windows. When using the pop-out window button, I set a initial display size that’s appropriate for the header view, which currently is actually the exact same view as the game window, so it’s possible for the user to make this bigger, and it will revert to the full game detail as appropriate.

An extension of this approach is what happens if the window gets really large, should it add even more information, that’s something else I want to play around with going forwards. Perhaps a single window with everything self contained will actually work better in practice, but my feeling in that case is that users will want the ability to customise the layout, perhaps hide and resize views within a window, which seems like a lot of overhead to manage. It would effectively be a window manager within the App itself.

In addition to content adapting to the window size, there are two other screens of information which can all be open simultaneously for a game. This way users can open the information they are interested in, as well as position and resize it as they wish. This is in contrast to iOS where views navigate in and out full screen, meaning there can be some jumping around to get to different information. My feeling is the extra real estate that the Vision Pro affords lends itself to this approach, rather than being limited to a single window. It also means I can rely on the in-built window management.

Now I’m not expecting everyone, or even most people will want this kind of detail, but I’m hoping those that do appreciate the flexibility the app affords.

Apple’s Human Interface Guidelines actually warn against this kind of behaviour stating:

I’m hoping with the specific use case of this app, that best practice can be stretched somewhat. It’s still possible to get the full experience of the app with only one open window, but for “power users” it’s possible to get more from the app. I’ll be interesting to see how my approach works in practice as I feel these windows aren’t intrusive and all provide value.

Hopefully this aspect of the User Experience is something that will standardise over time, once developers show their approaches and the platform matures beyond the initial release.

I’ve been pondering an idea for the last few months, since WWDC, which is whether we have passed Peak App? Has the idea of what we think an App is reached a crest and are we currently experiencing the beginning of a new era for the humbleApp? What I mean by Peak App, isn’t that Apps are doomed, but more what we think of as an App is changing for ever, and that we’re seeing in a new phase of their evolution with the explosion of platforms and widgets.

The way in which we experience Apps is no-longer hidden behind an icon, a simple full screen, pixel perfect, single process view of their data. For the last few years, focus has been on Apps supporting more platforms, iPhone, iPad, Apple Watch, Mac, Vision Pro, mostly powered by SwiftUI’s rise.  Additionally Apps can support Widgets, which are now interactive, Siri Intents, Live Activities, Share Clips, Shortcuts, Notifications and with iPhone 15 the Action Button. A myriad of ways for users to interact with an App without actually opening it. Jason Snell had a similar insight on the Upgrade podcast, episode 477  where he said:

“we’re potentially entering an era where Apple is sort of saying, there are lots of interactions you do with apps that you don’t have to do in the app…. That’s an interesting shift in philosophy to say opening the apps is not the way you have to interact with them, in fact some apps I would imagine, it be very rare that you need to go in”. Jason Snell

More and more we’ll see the core functionality of an app accessed through system interactions, offered by the app, but presented by processes outside of the Apps control. In the future, I can see apps being more like APIs, exposing their data through endpoints (intents), allowing users to indirectly interact with the app in even more places. It will be possible to combine functionality between different apps, joining the output of one intent with another or presenting them together in new ways. We see this already with Shortcuts, and somewhat with Interactive Widgets, but this ability will only get more powerful and easy for the user to control going forwards.

Apple often foreshadows their plans with features that may look underwhelming in themselves, but form part of a grander vision. There’s been a push for Intents for years, first donating them to the system, then with Shortcut integration and now with Interactive widgets. Widgets are everywhere, Apple has been slowly making them more powerful and more useful across home screens, lock screens, StandBy and watch faces. With the inevitability of an LLM based Siri coming, how else will that work for App integration, if not for an Intent based system with results visually displayed on the best device to you, the phone in your hand or on it’s stand, the TV on the wall, or directly into your world through the Vision Pro.

I can see Apps no longer being delivered via a coherent interface, but a set of features that can be mixed and matched together, the experience would be widget like, just a lot more powerful and not restricted to the Home Screen. Drop a weather widget into your sports app, combine a time tracking widget next to your event in the calendar app. The control will be with the user in how they interact with your features, not the developer. Why limit the use of a widget to the confines of a screen designed for App Icons? It’s clear the current Home Screen is too limiting for Widgets as they mature, but will Apple go so far as to remove icons themselves from the Home Screen altogether? It doesn’t seem out of the question, a small widget offering live information is way more powerful than a static icon. Could it lead to a Widget Store? A future where functionality and quick interactions supersedes complexity and breadth of features?

In Federico Viticci’s excellent iOS and iPadOS 17 review on MacStories he concludes with the same sentiment;

A modern app for the Apple ecosystem is more than an icon on the Home Screen: it’s a container for a diverse set of experiences on a range of different hardware products…. What does it even mean to be an app anymore? - Federico Viticci

What does this mean for developers? I would recommend leaning into App Intents, integrating interactive widgets, App Shortcuts and developing your apps to be as modular as possible. Plan for all features to be accessed independently and if you haven’t already then lean into SwiftUI and native development. If you’re reliant on cross platform tools, you’ll be more and more ostracised from this journey.

If the last 15 years is anything to go by, then just as you think things have reach their peak and will settle down, there’s another paradigm waiting just over the horizon. It seems like we are currently experiencing the beginnings of this again, and once more things will change forever.

The App is dead. Long live the App.

The Cricket App I’ve been building has been a slow burn, initially built at a time long before Swift Charts, and given that I’d been coding it exclusively on Swift Playgrounds, support for Charts was even longer coming. So by the time it was available, I had a chart I was happy with and couldn’t see a good reason to change. Until now.

With the introduction of the Vision Pro, I thought now was a good time (excuse) to transition across to benefit from better native support for things like interactions and view resizing. Having allocated an evening to get things up and running, I was pleased to discover that with only a few lines of code, I was able to get a chart up and running. With a bit more playing with styling, particularly with axis, I could a near replica of the original chart (left) with only 10% of the original code. There’s a few visual changes, mainly to accommodate the chart on the Vision Pro at this point.

Going forwards this new approach will allow much quicker iterations on how the chart looks and feels and hopefully lets me explore some new options with other data that can be charted in other areas of the app without having to code a completely new custom chart.

Getting it running was one thing, but being a good platform citizen and fitting in seamlessly is another.

The first thing to tackle was the app had no windowing support, so creating an WindowGroup and allowing an individual game’s content to be opened separately was the first step. Getting this working was straightforward, but extracting out some assumptions of the data model to accomodate this new concept took a little more time.

Once the practical steps were out the way, I could turn my attention to making it a bit more compelling of an experience. Adopting ornaments was the first step, adding in a Tab Bar in order to benefit from the rather handy navigation it enables was a natural addition. Whilst playing with ornaments I also tried adding the window title as a top ornament, which I liked the look of more than what I’ve seen as standard, which is a large title in the top left of a window. I’m not completely sold and could drop the idea, but I feel it frees up the content and is more obvious than the subtle tab bar icons to differentiate the current selection. This change also forced me to adopt a grid view rather than a list view for the upcoming games. On visionOS the windows are a default size, so long lists don’t work as well as a grid of content.

I did fall foul of not providing the ornament to the correct view. It only worked when the ornament was added to the main Tab Bar View, not each individual Tab’s View. I was caught here as if added to the Tab’s View it does show, but not if you change tabs and not always. I’m unsure if this is a bug, or user error, but either way it’s something to look out for.

Next was to clean up some of the backgrounds and colours used. One of the first things I noticed was that coloured text just doesn’t work on visionOS. The glass background makes the addition of coloured text very hard to read. There might be exceptions, I would assume this is something that all apps will need to look out for. It is something called out in an Apple Design Video, with their suggesting to add solid colours as a background, but in my situation where the content isn’t a button, this wasn’t an option.

There were also a few places, where I’d used fixed colours, such as the chart background as well as .secondary and .tertiary as background fills as well as layered different blur effects in places. These didn’t work on the Vision Pro, where care needs to be taken to alternate light and dark background views, or at minimum not have lighter on light views. I’m still not happy with where this ended up, but it was already a vast improvement over the initial version.

One area I’ve found hard to understand is sizing and spacing of content. Everything just looks so small and compact on the simulator. I’m aware that windows viewed on device, will be larger and clearer than trying to view on a window on my Mac, but it’s hard to fight the temptation just to double the size of everything to accommodate. Which reminds me, I haven’t yet optimised the layout for accessibility settings, which there are more of than ever on visionOS. I’m confident that adopting the layout to a variety of options such as dynamic type and reduced transparency will help refine the usability across the board.

This is how I anticipate the app being used, in conjunction with live content, giving the viewer additional context about the ongoing game.

All in all, for only a few hours work, I’m really happy with how much progress I was able to make. There’s lots still to improve and iterate on, but considering this is a whole new platform, the transition isn’t as dramatic as it could have been. If you have any suggestions how I might further improve things please let me know.

First thing’s first, if you’re extending an existing App, then you may need to Enable Multiple Windows support in the info.plist of your app. Adding the `UIApplicationSupportsMultipleScenes` with a value of true will unlock support (required on visionOS and iPadOS. Then it’s as simple as calling openWindow by specifying the id of your window group using the openWindow environment action.

Once you’ve done that, there’s seemingly no limit to the number of windows your app can open or where your users can put them.

struct MainWindowView: View {
    @Environment(\.openWindow) private var openWindow
    var body: some View {
        VStack {
            Text("Main Window").font(.title)
            Button("Open") {
                openWindow(id: "childWindow")
            }
        }
    }
}

Apple changed the way windowing works on Beta 2, allowing more control of window sizing. The below has been updated to reflect that.

In Apple’s attempt to control the user experience, and ensure the user isn’t disorientated, by default every window launched is the same size and front and centre of the user’s view. Initially you couldn’t override this functionality, but as of Beta 2 it is possible to add a `defaultSize` modifier to set the initial size of the window, but still not its position. It also doesn’t seem possible to adjust this programatically after the fact (without some very low level, and clearly unintended, trickery)

.defaultSize(width: 0.5, height: 0.75, depth: 0.0, in: .meters)

If you do need a specific and fixed size for your content, then one trick is to use a volumetric window, which does allow you to set a size. Volumetric windows are still opened in the middle of the user’s eyeliner, but can be moved around and placed, but not ever resized by the user.

Points in visionOS are calculated differently to other platforms, and don’t represent a fixed number of pixels. Instead a point on visionOS is an angle, which means content can dynamically resize as it moves closer or further away from you, in order to maintain legible content. The recommendation is to only fix the size of an object if it represents a real world object, of a specific size, otherwise it’s best to let the user control how they interact with your content.

The following code allows you to create a window with a fixed size, which can’t be controlled by the user. Despite this being a Volumetric window, it can still contain only ‘UIKit’ elements, the only difference seemingly is no-matter what the depth of the view is set to, it’s possible to move the window around from the ‘side’ which also re-orientates the view perpendicular to the user.

WindowGroup(id: "main") {
    MainWindow()
}
.windowStyle(.volumetric)
.defaultSize(width: 1, height: 1, depth: 0.1, in: .meters)

There’s no concept of a Window in visionOS, only WindowGroup. That means a user can open multiple copies of a window by default. This may be intentional, as I imagine it could get easy to lose windows in your environment, so having the ability to create them at will means the user can control this and even leave multiple copies of windows around their environment.

If you want to ensure only one copy of your window group, then the easiest way to manage that, is to dismiss the WindowGroup right before you open it. This will mean that it re-opens front and centre where the user is looking. This could of course be a benefit, in case they’ve misplaced the window somewhere.

Button("open") {
    dismissWindow(id: "targetView")
    openWindow(id: "targetView")
}

If you don’t want windows jumping around, then you’ll need to keep track of which views are open and dismissed using a combination of Scene Phases and/or `onAppear`, `onDisappear` modifiers to manually control the ability for the user to open a new instance of that WindowGroup. Scene Phase doesn’t report when a window is dismissed, either using dismissWindow or using the visionOS close button that’s attached to every window. It also doesn’t support the `controlActiveState` environment variable, which is a a macOS solution to get this information.

WindowGroup("Target View", id: "targetView") {
    TargetView()
        .onAppear {
            print("target window appeared")
        }
        .onDisappear {
            print("target window disappeared")
        }
}

struct TargetView: View {
    @Environment(\.scenePhase) private var scenePhase
    var body: some View {
        Text("Target View")
            .onChange(of: scenePhase, { _, phase in
                print("scene changed: \(phase)")
            })
    }
    
}

Immersion

If you need more than just windows then the Immersive space is for you. Here you get ultimate control of where objects are placed and aren’t restricted to a window at all (although windows still work here too). Even if you do have an app that suits an immersive experience, it’s still recommended to launch into a regular window to start with, using say a button within that window to open an immersive space under the control of the user. Perhaps if you have a true immersive experience or game, then after the first time, subsequent launches can go straight to the action if appropriate. To open directly into an immersive space requires setting another plist value, this time `UIApplicationPreferredDefaultSceneSessionRole` needs a value of `UISceneSessionRoleImmersiveSpaceApplication`.

Note: It doesn’t currently seem possible to open an immersive space upon app launch in simulator. You do need to specify a key in the info.plist to allow this in the first place, but in my experience, the immersive view isn’t rendered. In Beta 2 it seems this key is ignored and the app refuses to launch.

It’s worth knowing, that currently the original window group remains open when opening an immersive space. This may not be desirable for your experience, it certainly feels a bit redundant in some circumstances, even if the user can always manually close this window. This does reveal a weird UX issue though, as if the user closes the initial window, then also closes the immersive space, then your app may have no windows left, so it will be fully closed.

If you don’t want this behaviour then you can resort to managing the opening and closing window groups manually. Apple handles this weirdly in the Earth sample app, by using opacity to hide the main view, replacing it with a smaller, space specific modal. I’m not a fan of this approach as if you move that view out the way, the main view is confusingly (to the user) also moved. From a design perspective, if the new view looks and acts like a different window, it should be treated as such. My preference is to open a separate window group, or use the attachments option of the RealityView to position the modal view within the scene itself.

In the main window, open the immersive space as normal:
await openImmersiveSpace(id: "ImmersiveSpace")

When the immersive space opens, dismiss the original window:
.onAppear {
    dismissWindow(id: "mainView")
}

When you want to exit out of the immersive space:

//re-open the main window
openWindow(id: "mainView")

//if we don't wait here, the app doesn't seem to have time to open the window
DispatchQueue.main.asyncAfter(deadline: .now() + 0.1) {
    Task {
        //then dismiss the immersive space
        await dismissImmersiveSpace()
    }
}

A window into the future

As developers explore the options with windows, then I expect certain paradigms to become the standard, perhaps with some changes to visionOS as the beta period continues. One such limitation with visionOS Beta1, is that there is no access to the Digital Crown, which means there’s no way to test the progressive immersive space, in which the user controls the amount of immersion of the space, even going from .mixed through to the .full display options as they desire. This has now been resolved as of visionOS 1.0 Beta 2, it’s now possible to set the immersion values to 30, 50 or 100% via the I/O menu.

I also haven’t touched upon opening windows onto surfaces, re-orientating windows so they’re flat on a floor/table, or portals. I feel like a part 2 is coming…

Updated for visionOS 1.0 beta 2

Hand Tracking is a key feature of the Vision Pro, enabling the main user interaction with the device. It allows the device to keep track of a user’s hand position, rotation and size (transform) in the scene as well as each joint within the hand. With permission, it’s possible for an app to access this data in order to power their own types of features, which is what we’ll dive into in this article.

The data we as App developers have access to is provided under an HandAnchor entity. This is a special type of Anchor that contains a number of important princes of information. Chirality tells us whether this is the left or right hand. It contains a hand skeleton, revealing details about the joints in the hand. It also consists of the base transform of the hand in world space.

In the WWDC session, “Meet ARKit for spatial computing” we get (the only?) insight into what this skeleton is:

note: in visionOS beta 2, the “hand” prefix has been removed from these references

Each joint is attached to a parent joint, with the .wrist being the root. The `wrist` being the root is the only gotcha with this hierarchy. All of the fingers, the thumb, but also the forearm flow back here. This means the parent of `forearmArm` is `forearmWrist`, whose parent is `wrist`, which is a bit counterintuitive

For those anatomists among us, you may be confused with the splitting of the metacarpal and the knuckle, as each refers to the same thing. Here it seems Apple is referring to the metacarpal bone and the knuckle joint. It’s a bit different to the references used for the Vision (2D) version of Hand Tracking which uses the actual joint names to reference the positions. These can be mapped as below:

VisionOS                                                   Vision
(SkeletonDefinition.JointName)	            (VNHumanHandPoseObservation.JointName)
.indexFingerTip	                            .indexTip
.indexFingerIntermediateTip	    .indexDIP
.indexFingerIntermediateBase       .indexPIP
.indexFingerKnuckle	                    .indexMCP
.indexFingerMetacarpal	            No equivalent
.thumbTip	                                    .thumbTip
.thumbIntermediateTip	            .thumbIP
.thumbIntermediateBase	            .thumbMP
.thumbKnuckle	                            .thumbCMC
.wrist	                                            .wrist
.forearmWrist	                            No equivalent
.forearmArm	                                    No equivalent

Each of the fingers follow the same pattern as above, with ‘index’ replaced by, ‘middle’, ‘ring’ and ‘little’ as appropriate. It’s certainly a bit confusing, to go from anatomically correct names, to made up landmarks, with some mixed up reference to anatomy.

Tracking a Hand

How do we go about actually tracking a hand? Well first caveat, is without a device, you can’t! Hand tracking isn’t available in the simulator and there’s no other device that exposes the HandAnchor for us to play with, so this is all theoretical for now. Regardless, if you want to prepare your app for when it is possible, there are a few things to be aware of. First off, you need an ARKitSession, then you need the user’s permission to access that session. This comes in the form of a .handTracking request via ARKit. Note, unlike permission prompts on other platforms, this is an option request, as ARKit will automatically prompt the user if you leave this out, rather than crash outright. However, without type of check, you won’t know if someone has denied your request, and be able to handle it appropriately.

let session = ARKitSession()
func requestAuth() async {
        let request = await session.requestAuthorization(for: [.handTracking])
        for (authorizationType, authorizationStatus) in request {
            //handle auth here
        }
    }

Once we have permission, we can add hand tracking to the session, using a HandTrackingProvider, then retrieve any updates asynchronously to the hand anchors as they are made available.

class HandTracking {
    let session = ARKitSession()
    let handTracking = HandTrackingProvider()
    
    func runSession() async {
        do {
            try await session.run([handTracking])
        } catch {
            print("error starting session: \(error)")
        }
    }
    func processHandUpdates() async {
        for await update in handTracking.anchorUpdates {
            let handAnchor = update.anchor
            
            guard handAnchor.isTracked else { continue }
            
            //do something with hand/joints
        }
    }
}

Limitations

We’ve already come across one limitation of hand tracking, in that it’s not available in the simulator at present. The second limitation is your app has to be running in an independent full space, not in the shared space with other apps. This means your app needs to be the only one in focus in order to access any hand tracking data (as well as other types of ARKit scene data). If you had ideas about a fully gesture based Todo app, or say an always open music player that you could control with gestures, whilst working in other apps, then that won’t be possible. It makes sense, as you could easily have multiple apps with conflicting gestures, as well it being a real drain on system resources.

What is Possible?

So what can you do with hand data? Well custom gestures is one, as shown in the Happy Beam sample code provided by Apple. It’s a bit convoluted right now, to extract each joint position, calculate distances between them and use that to determine when a gesture has been made, but it is possible. An extension of this is direct interaction with objects in the spatial world. Picking up objects or applying forces are possible by attaching physics objects directly to your hands.

Finally attaching 3D models or 2D overlays to the hand opens up other possibilities, particularly for games. Who wouldn’t want a Tony Stark style Power Gauntlet to control?

It will be interesting to see what Apple has planned for developers to access these features, without have a Vision Pro to test with, as it seems like it will be difficult otherwise. In the meantime, if you want to simulate what it could be like, then see my other article about faking hand tracking in the VisionOS Simulator…

The first stumbling block was having no cylinder mesh resource in RealityKit, like there is in SceneKit. For “simple” things I prefer to generate meshes and materials in code where possible. It keeps app sizes down, and keeps the asset workflow simpler.

However in this case, it was necessary to rely on Reality Composer Pro, which does have a Cylinder geometry and opened up the potential to explore the shader graph feature for the first time.

With a bit of research and some trial and error I was able to create a nice pixel shader effect that uses the height of the pixel to determine the transparency, along with an added remap node to ensure the bottom of the cylinder (< 0.2 in y) is also transparent. With a bit of experimenting and reversing the gradient with a -1 constant multiplication for the y position, local to the mesh I had the gradient “glow” I was looking for.

This was the effect I was after, but I couldn’t uncover a way to make the material double sided. I couldn’t see any options for the unlit surface I thought was appropriate, nor, in my desperation figure out a vertex shader that would achieve the effect.

Almost simultaneously there was a discussion on Mastodon about the lack of an out-the-box skybox for VisionOS. In Apple’s sample code, they use a trick of negative scaling the entity to create a skybox. This is certainly one solution, but isn’t possible in Reality Composer Pro as negative scales aren’t possible. This means you can only visualise the result within Xcode (either the preview or Simulator).

//scale a mesh entity to reverse the surface rendering
.scale *= .init(x: -1, y: 1, z: 1)

It’s also not a solution for my situation as I needed a double sided material, not just an inverted one. One way around this is to duplicate the entity and only invert one, again within code. This does solve the problem, but I wasn’t satisfied with it as a solution, as it just feels wasteful and relies on a runtime fix. Online I found a few other options, one being to duplicate just the mesh, rewriting the vertices in reverse order. This certainly inverts the face rendering, but without testing, my gut feel this would be more expensive than duplicating the entity itself.

Going back to experimenting, I discovered that my original choice of material, ‘unlit’ may have been hampering my options. Two other materials, ‘Custom’ and ‘PhysicallyBasedMaterial’ have a ‘faceCulling’ option, which is described by Apple as:

To improve performance, RealityKit culls polygons, or faces, that it determines won’t be visible. Discarding faces that aren’t part of the final render eliminates the need to do any calculations for those faces.

The three options for culling are ‘back’ (default), ‘front’, or ‘none’. Bingo, this is exactly what we needed. This serves the purpose for both what I needed (.none) and the requirement for a skybox (.front), without needing to scale the entity.

There are two issues with this approach. There’s added complexity (and potential performance implications) of these shaders over the ‘unlit’ I’d been using. My assumption would be the simplicity of unlit, allows for some performance gains under the hood over a custom or physics based shader. This could certainly do with some additional investigation, but is a small problem compared to the additional issue of CustomMaterial’s weirdly not yet being available for VisionOS, so aren’t even an option here if shaders are part of the rendering. It’s still a great solution for a Skybox situation, using a PhysicallyBasedMaterial, but isn’t possible yet using a Reality Composer Pro workflow (no access to any parameters for a `ShaderGraphMaterial`).

One thing to note, is there’s an issue (FB12605220) with RealityKit when using blending and no face culling, that causes some erroneous culling still as shown below in the left image.

A fix is to specify a value for ‘opacityThreshold’ on the material, even if it’s just the default of 0.0. This seems to kick the rendering into gear and it then correctly renders the transparency with no-culling of “hidden” faces.

There’s currently (as of Xcode 15 beta3 VisionOS SDK 1.0) no way of testing or simulating hand tracking in the Vision Pro Simulator. In fact, many of the Recognizers, the bridges between the real world and our apps, that will unlock many of the engaging Spatial experiences don’t (yet?) exist in the Simulator.

To combat this I have been experimenting with using Playgrounds on the iPad to get a feel of how scene, plane, hand and body tracking would work. This works really well for prototyping and is a super responsive workflow. Changes can been seen “instantly” in the preview and the ability to take the iPad anywhere (and make changes from anywhere) is important for AR. This workflow falls short with replicating the experience of running an app on the Vision Pro as Playgrounds doesn’t have the Vision SDK (understandable, it’s unlikely to), or even the latest iPadOS 17 features. (I’m hoping for an updated Playgrounds app soon).

Experimenting

I was experimenting with running a customML object detection model on a video playing within the Vision Pro. It happened that code snippet I found for extracting the frame buffer included a CIFilter step. That got me thinking about hand tracking, and if I could bring a form of hand tracking in the Simulator.

Thanks to same Apple sample code and a the work I’d already done on the iPad with VNHumanHandPoseObservation, along with some careful camera placement I was able to fake hand tracking, in 2D, on the simulator.

So how does this magic work? Here’s how it looks from another angle, without the green screen effect:

There are three parts to this, one the video, two the video filter and three the hand detection.

Video

First I recorded a clip of just my hand, on a “green screen” (my practice putting mat) and imported that into the Xcode Project (After first resaving it to remove the HDR element as that seemed to mess up playback).

Using AVFoundation and an AVPlayer in a UIViewControllerRepresentable I was able to get video playback, but in a way I could also add a hook to the AVPlayerItemVideoOutput necessary to access the frame buffer for the video.

struct PlayerView: UIViewControllerRepresentable {
    
    @Binding var url: URL?
    
    func updateUIViewController(_ uiView: UIViewController, context: UIViewControllerRepresentableContext<PlayerView>) {
    }

    func makeUIViewController(context: Context) -> UIViewController {
        let view = PlayerUIView(frame: .zero,
                                url: $url)
        let controller = PlayerController()
        controller.view = view
        
        return controller
    }
}

class PlayerUIView: UIView {
    private let playerLayer = AVPlayerLayer()
    private let _myVideoOutputQueue = DispatchQueue(label: "VideoFrames", qos: .background, attributes: .concurrent, autoreleaseFrequency: .workItem, target: nil)
    
    lazy var displayLink: CADisplayLink = CADisplayLink(target: self, selector: #selector(displayLinkDidRefresh(link:)))
    var player: AVPlayer
    var videoOutput: AVPlayerItemVideoOutput
    
    @Binding var url: URL?
    
    required init?(coder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    init(frame: CGRect,
         url: Binding<URL?>,
         handObservable: HandObservable) {
        
        _url = url
        // Setup the player
        player = AVPlayer(url: url.wrappedValue!)
        
        let settings = [ String(kCVPixelBufferPixelFormatTypeKey): kCVPixelFormatType_32BGRA ]
        let output = AVPlayerItemVideoOutput(pixelBufferAttributes: settings)
        self.videoOutput = output
        super.init(frame: frame)
        
        playerLayer.player = player
        playerLayer.videoGravity = .resizeAspect
        layer.addSublayer(playerLayer)
        
        attachVideoComposition()
        
        player.currentItem?.add(output)
        displayLink.add(to: .main, forMode: .common)

        // Start the movie
        player.play()
    }
}

This is mostly boilerplate, but the important pieces of code here are hidden in ‘attachVideoComposition’ for adding the alpha along with the displayLink setup for processing the individual frames with Vision.

Filter

The attachVideoComposition function creates a CIColorCube filter, which converts anything of a certain colour hue (in our case green) to alpha.

private func attachVideoComposition() {
        //remove green hues
        let chromaCIFilter = self.chromaKeyFilter(fromHue: 0.2, toHue: 0.5)

        if let playerItem = player.currentItem {
            let asset = playerItem.asset
            
            AVMutableVideoComposition.videoComposition(with: asset) { filteringRequest in
                let source = filteringRequest.sourceImage
                chromaCIFilter?.setValue(source, forKey: kCIInputImageKey)

                // Apply CoreImage provessing here
                filteringRequest.finish(with: chromaCIFilter?.outputImage ?? source, context: nil)
            } completionHandler: { composition, error in
                playerItem.videoComposition = composition
            }
        }
    }

The filter itself is directly taken from Apple’s sample code here, which couldn’t have been more perfect for what I needed.

That gave me a nice transparent version of the video, once I’d tweaked the `fromHue` and `toHue` values to something that matched my specific green.

The player view is loaded in a simple SwiftUI View within the project, that loads the local video file on the press of a button.

struct VideoViewDemo: View {

    @State private var url: URL?
    
    var body: some View {
                ZStack {
                    if url != nil {
                        PlayerViewDemo(url: $url)
                    } else {
                        Rectangle()
                            .stroke(.green)
                            .frame(width: 326)
                        Button(action: {
                            url = Bundle.main.url(forResource: "IMG_2067", withExtension: "mov")
                            print("url:\(url?.absoluteString ?? "no file")")
                            }) {
                            Text("Play")
                                .font(.headline)
                                
                        }
                    }
                }.edgesIgnoringSafeArea(.all)
    }
}

I’ve since come across a much more robust video playback example from Apple themselves and is a better starting point for video control than shown here.

The only other thing I needed to change was adding the .windowStyle(.plain) modifier to the WindowGroup, otherwise I still had the glass effect on the app’s window.

Here’s the result of the transparency side by side with the raw video:

VisionKit

That was pretty cool as it was, but I also wanted to see if I could run any of the confusingly named Vision detection algorithms on the video as well. This is an existing framework, dedicated to extracting clever things from images and video, unrelated to VisionOS, Vision Pro. That’s where the DisplayLink code comes in, which gives us notification every time a new video frame has been shown, and means we can extract a pixel buffer if it’s available. The code to do this is:

@objc func displayLinkDidRefresh(link: CADisplayLink) {
        let itemTime = player.currentTime()
        if videoOutput.hasNewPixelBuffer(forItemTime: itemTime) {
            var presentationItemTime: CMTime = .zero
            if let pixelBuffer = videoOutput.copyPixelBuffer(forItemTime: itemTime, itemTimeForDisplay: &presentationItemTime) {
                // process the pixelbuffer here
                processImage(pixelBuffer: pixelBuffer)
            }
        }
    }

With a pixel buffer available, which is one input that Vision can process, we are able to spin up a VNDetectHandPoseRequest, retrieve the observations, filter for those that have a high confidence, and finally store those in a location we can get to from the SwiftUI view. There’s a lot going on here, and I’ll dig more into the details of Vision and how the different detectors work in future posts, but it doesn’t take much code to get this working.

private var currentBuffer: CVPixelBuffer?
    
    func processImage(pixelBuffer: CVPixelBuffer) {
//the request can take longer than frames are sent to it, so only process one at a time
        guard currentBuffer == nil else {
            return
        }
        currentBuffer = pixelBuffer
        
        let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
        DispatchQueue.global(qos: .userInteractive).async {
            do {
                let handRequest = VNDetectHumanHandPoseRequest()
                handRequest.maximumHandCount = 1
                defer { self.currentBuffer = nil }
                try handler.perform([handRequest])
                
                if let observation = handRequest.results?.first as? VNHumanHandPoseObservation {
                    let allPoints = try observation.recognizedPoints(.all)
// filter for high confidence observations
                    let filtered = allPoints.filter { observation in
                        observation.value.confidence > 0.5
                    }
                    
//                    print("got hand: \(filtered.count)")
                    
                    DispatchQueue.main.async {
//send to an observable class on the main thread
                        self.handObservable.handPoints = Array(filtered.values)
                    }
                }
            } catch {
                print("error: \(error.localizedDescription)")
            }
        }
    }

Lastly we take these observations and display them back to the user. One thing to note, is these observations are only in 2D, so of limited use for a full Vision application. The code here isn’t great, with some hardcoded values for the size of the window and video frame offsets. In reality you should convert the points from the Vision step (x: 0.0...1.0, y: 1.0...0.0) onto the video frame accurately, more on that in another post. But for this demo these value work well enough to get across the idea.

Canvas { context, size in
                                for point in handObservable.handPoints {
                                    let xWidth = ((size.width - 326)/2)
                                    let path = Path(ellipseIn:
                                                        CGRect(origin: CGPoint(x: (point.x * 326) + xWidth, y: 620 - (580 * point.y)), size: CGSize(width: 8, height: 8)))
                                    
                                    context.fill(path, with: .color(.green))
                                }
                            }

And that’s it, a pretty convincing effect once you carefully move the viewing position of the window to make sure it’s in the bottom left of the screen.

The Vision performance isn’t great (but is also completely unoptimised, running on a 4k video feed) so there’s a bit of a lag but for a POC it works surprisingly well.

So what?

Okay, so what use is this anyway? Well, it should be possible to convert these 2D points into a estimated 3D skeleton that can act like the actual hands detected natively in Vision. If I can get that to work, then it will enable a workflow of pre-recording some hand videos, and using them for testing actual 3D hand tracked gestures and experiences in the Simulator. That same workflow will also be useful for testing experiences using VisionOS detected hands, which already come with 3D data. The ability to “record” hand positions and re-play them back over and over will make testing Vision Pro apps and games easier in the long run.

However if like me you’ll also quickly become frustrated with how cumbersome it can be to navigate the simulated world, particularly if you have to move around your content regularly to validate the effect your trying to create. There are a number of built in cameras to the preview that you can jump between, but what if you need a specific angle not covered by these?

That’s where preview cameras come in, which were introduced in Xcode 15.0 beta 3. With the new #Preview macro, you are able to define your own PreviewCamera’s with options for setting a direction, or more usefully setting a camera location, looking towards another location.

To use these camera previews, you will ned to add them to the #Preview macro as follows:

BEFORE:
#Preview {
    MyModelView()
}

AFTER:
#Preview(body: {
    MyModelView()
}, cameras: {
    PreviewCamera(from: .topTrailingFront,
                  zoom: 0.5,
                  name: "cam0")
    PreviewCamera(lookingAt: Point3D(x: 0, y: -1, z: 2),
                  from: Point3D(x: 1, y: 0, z: 2),
                  name: "cam1")

})

Here we’re creating two Preview Cameras. The first is using a UnitPoint3D, to specify a position to look down on the content from, whilst also zooming out from the content. The second is a more flexible way to define a camera. It’s looking at a point that is near the ground and a 2m away from the back wall of the room (which seems to be the 0 position), with the camera’s position (“from”) being 1m to the side, and on the ceiling (defined as 0m). With the second option, you don’t have the ability to set the zoom level specifically.

When you start the preview, it will now use the first camera you set as the PreviewCamera, but you can switch between them easily using the camera button in the lower right of the Preview Canvas.

You can add as many camera’s as you like, just be sure to give them a unique name so that you can differentiate them in the UI.

Happy Viewing!

I wish Swift Playgrounds had a better way of running tests, I really don’t want to fall back to Xcode just for this, having written the whole app on the iPad so far…

Subscribe now