Thursday, August 11, 2022
HomeiOS DevelopmentWorking with Picture Monitoring in ARKit

Working with Picture Monitoring in ARKit

Ranging from iOS 12, builders can use the back-facing digital camera to detect and monitor recognized pictures by utilizing an API referred to as ARImageTrackingConfiguration. What does it imply to you? For those who’ve watched the Harry Potter film earlier than, you in all probability keep in mind the wizarding newspaper with animated pictures in print. With the picture monitoring configuration, we are able to flip this right into a actuality by way of an ARKit app.

On this chapter, you’ll learn to use ARImageTrackingConfiguration to carry out picture monitoring and construct an AR newspaper app much like that appeared within the Harry Potter film. It’s not straightforward to explain how the app works in phrases. So, please head over to this hyperlink and see the demo app in motion. When the app acknowledges a particular photograph, it overlays the picture with a video. As you possibly can see within the video, what’s superb is that ARKit tracks the place change of the picture and preserve anchoring the video content material.

Trying attention-grabbing, proper? Let’s get began.

Making a New Mission Utilizing ARKit template

We’ll construct this mission from scratch. Assuming you’ve opened Xcode, you can begin by creating a brand new mission. Select the Augmented Actuality template and click on Subsequent. For the mission info, identify the mission ARNewspaper and be sure to set the Content material Know-how to SceneKit.


When you saved the mission, Xcode will generate the ARKit mission for you. There are a few issues we don’t want within the generated code. By default, the template generates a 3D spaceship, which is loaded when the app launches. Clearly, we don’t want this mannequin. Within the ViewController.swift file, change the next line of code in viewDidLoad() from:


For the viewWillAppear technique, it’s loaded with ARWorldTrackingConfiguration():

We don’t want it too as a result of we’ll use one other monitoring configuration to deal with the picture monitoring. In the meantime, empty the strategy like this:

Getting ready the Assets

Now that we’ve got created the mission skeleton, let’s transfer onto the subsequent step and import the required assets. First, obtain the useful resource pack from Since this app goes to play a video when a specific picture is acknowledged, it’s important to put together the next assets:

  1. A picture to be acknowledged. It may be any picture (JPG/PNG) in your photograph library. Or you should use the one (i.e. snowboarding.jpg) which comes with our useful resource pack.
  2. A video to be performed after the picture is detected. Once more, it may be any video in mp4 format. Within the useful resource pack, it is best to discover a video file named snowboarding.mp4.
  3. A doc containing the picture. That is fully elective. However to imitate the Harry Potter newspaper, the useful resource pack features a PDF file named arkit-newspaper.pdf, as proven in determine 45.2. You may print it out in A4 measurement paper for later testing. For those who don’t have a printer, that is okay. You may open the file on an iPad or a Mac. We’ll talk about extra about how one can run the take a look at within the later part.

Subsequent, we’ll import the video file (i.e. snowboarding.mp4) into Xcode. Within the mission navigator of Xcode, proper click on ARNewspaper and create a brand new group named Assets. Drag the video file from Finder into Xcode and place it within the Assets folder. When Xcode prompts you to decide on choices for including the file, please be sure to tick the checkbox of ARNewspaper within the Add to targets possibility. For those who’ve efficiently imported the video file, it is best to be capable of preview it in Xcode.

Setting the Reference Picture for Picture Monitoring

We haven’t imported the picture but. As you realize, we used so as to add the pictures to the asset catalog. However this time, it wants a bit of additional work for picture monitoring. Open Property and right-click to decide on AR and Textures > New AR Useful resource Group. This creates a brand new useful resource group to specify the reference pictures for picture monitoring. You may drag the snowboarding.jpg file (or your personal picture) to the useful resource group.

Whereas we solely have one picture for this demo, you possibly can really put a number of pictures for picture monitoring on this group. As you possibly can see, there’s a large exclamation mark on the picture, which suggests there’s something lacking. Now choose the picture within the group and open the Attributes inspector.


To detect the precise picture, ARKit requires you to specify the real-world measurement of the picture. Assuming you print the newspaper file on a A4-size paper, the picture ought to have a width of 9cm and a peak of 16cm. So, change the Items to Centimeters and set the width to 9. When you set the width, Xcode robotically calculates the peak worth.

Please notice that the real-world measurement of the picture varies. For those who print it on a A3-size paper or show the PDF file on a 42-inch display screen, the bodily measurement of the picture turns into bigger. You’ll then want to regulate these values accordingly.

Establishing the AR Picture Monitoring Configuration

Within the earlier chapters, we used ARWorldTrackingConfiguration to trace the system’s place and blend the digital content material with the actual world. Apart from the world monitoring configuration, ARKit offers different varieties of configuration. To carry out picture monitoring, we use a configuration referred to as ARImageTrackingConfiguration. You employ this configuration to detect and monitor the movement of recognized 2D pictures.

To arrange this picture monitoring configuration, replace the viewWillAppear technique like this:

To precisely detect the place and orientation of a 2D picture in the actual world, ARKit requires preprocessed picture information and information of the picture’s real-world dimensions. The ARReferenceImage class encapsulates this info.


Within the code, we specify to make use of ARImageTrackingConfiguration and inform ARKit what pictures to trace. We name the referenceImages technique of ARReferenceImage to load all referenced pictures from the asset catalog. The inGroupNamed parameter takes within the group identify of the AR useful resource group. Recalled that we named it AR Assets, because of this we go it to the strategy.

As soon as we loaded the reference pictures, we assigned them to the trackingImages property of the configuration. Lastly, we name the run technique to start the picture monitoring operation utilizing the again digital camera.

Overlaying a Video Participant to the Detected Picture

Now that we’ve got created the picture monitoring configuration, the query is how can we overlay a video on high of the detected picture?

When ARKit begins the picture monitoring, it robotically provides an ARImageAnchor object when the reference picture is acknowledged. In a Scened-based ARKit app, the renderer(_:didAdd:for:) technique of ARSCNViewDelegate shall be referred to as when the anchor is added to the scene. Thus, we are able to implement the strategy to overlay the video on high of the detected picture.

First, let’s declare the video play for the video playback in ViewController like this:

We load the snowboarding.mp4 file and assign it to AVPlayer for playback. We simply put together the participant for the video playback. The video received’t begin enjoying till we name its play technique.

Now implement the strategy within the ViewController class like this:

As stated, each time a brand new anchor is added to the AR session. The tactic above shall be referred to as. Subsequently, we first have to verify if the newly added anchor is an ARImageAnchor. If it’s a picture anchor, we create a aircraft by utilizing SCNPlane for holding the video participant. The scale of the oblong aircraft is about to its real-world measurement that you just supplied earlier when configuring the reference picture.

Subsequent, we instantiate a SCNNode with the aircraft we simply created and add the aircraft to the scene by calling addChildNode. Lastly, we name the play() technique of the participant to begin the video playback.

It’s possible you’ll surprise why we have to modify the eulerAngles to rotate the aircraft. By default, SCNPlane is vertically oriented. Nevertheless, the detected picture is meant to be horizontal. To match the detected picture’s orientation, we’ve got to rotate the aircraft. The determine under exhibits you the default orientation of the aircraft.

For those who can’t wait to check out the picture monitoring, you possibly can deploy the app to your iPhone. Assuming you’ve printed out the PDF doc, you possibly can level the system digital camera on the newspaper. When the app detects the “Snowboarding” picture, it robotically overlay the video participant on high of the picture and begins the video playback.

In case that you just don’t have a printer, chances are you’ll load the PDF doc on iPad, which has an identical measurement of the A4 paper. Alternatively, you possibly can open the file in your Mac for testing. However please attempt to resize it to A4 measurement.

Replaying the Video

The picture monitoring ought to work now. Nevertheless, the video playback solely happens as soon as. When the video finishes enjoying, it is not going to play once more. To repair this problem, insert the next code within the viewDidLoad technique:

We added an observer to hearken to the AVPlayerItemDidPlayToEndTime notification, which is posted when the video playback ends. On this case, we set the playback time again to zero.

To play the video once more, insert the next technique in ViewController:

This technique is robotically referred to as by ARKit when the anchor is up to date. We first verify the reference picture is seen on display screen after which begin the replay if the result’s true.

Disabling Statistics

You need to discover a statistics bar appeared on the backside of the display screen. If you don’t want it, you possibly can disable it by altering the next line of code in viewDidLoad and set its worth to false:

Develop the View to Full Display screen

By default, the Scene View generated by Xcode has constraints that have been configured respect to the secure space. That is why you see an empty white bar on the backside of the display screen. If you wish to make the scene view full display screen, you turn over to Fundamental.storyboard and alter the scene view’s backside constraint. Within the Attributes inspector, change the Second Merchandise to Superview.Backside and set its fixed worth to 0.


On this chapter, we discover one other configuration supplied by ARKit that allows you to carry out picture monitoring. With ARImageTrackingConfiguration, it empowers builders to construct some attention-grabbing apps that merges digital content material with the actual world atmosphere. The Harry Potter newspaper is simply one of many many examples. It’s possible you’ll apply the method to complement the person expertise and engagement of some real-world objects like postcard and restaurant menus.

It is a pattern chapter of our Intermediate iOS Programming with Swift guide. To obtain the complete supply code and entry all its content material, you possibly can test it out right here.



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments