2016 version was implemented so that the player gets a selfie with all the AR props (implemented with just a screen capture), which is sent to a website. Then, the game shows a QR code which the player can scan with the smartphone and land on a website which enables selection between a 100, 500 or 1000 yen coupon for different shops and download the photo. The most valuable coupon was available only after reaching a certain score... something no player ever managed to get the first time. Almost everyone tried twice since the game has a well-designed learning curve and the prize follows the score.

In 2015 we did not count the number of games played, only the actual coupons redeemed. This was about 10-15 each day, worth about 200 000 yen in total. Not bad! In 2016 our game produced exactly 7345 coupons, meaning about 200 every day! This was a total surprise to us and definitely made us think of the real possibilities of this kind of marketing. It also means the total value of all the coupons is somewhere around 3-4 million yen. (Yet, we cannot disclose the exact number and the total value of the coupons that were actually redeemed ... that’s classified.) Anyway, we can conclude that this sort of pure advertisement game can make real money circle around the neighborhood.

Technical Considerations to Game Design

1) Skeletal tracking distractors

For the 2015 version, we spent days and days testing the gameplay and adjusting parameters that regulated the fish capture. This was because Kinect cannot track the exact specifics of each different player's hand movement from 3 meters away all the time perfectly. Especially, when the hand movement is fast.

There were some hilarious moments when the AR glove appeared in this lady's handbag right below the actual hand. Yet, the game worked as intended! That was not luck, just the results of countless hours of testing.

Whatever is designed to be used in this kind of shopping mall environment has to be robust enough to handle all the variety of additional props and disturbances that in a normal Xbox setting at home would be considered as something on your troubleshooting list:

All of this will be there and things should still just work. What I did is I went through the game several times with different clothes (here, especially shirts), holding bags and even pillows, and tried to find practical limitations of what props would ruin the gameplay. Eventually, I ended up just limiting a minimum distance to recognize a bimanual catch gesture:

That gave us wide enough space for recognizing the gesture. Yet, false positives were minimal since standing normally wouldn't trigger the gesture. The most important improvement was the latter item. A conventional programmer would probably pick the obvious choice: both gloves should collide with the fish object. But this is the part where a lot of testing needs to be done. My first implementation with this approach completely failed even my own tests, since there was too much variability in tracking hands in rapid movements: tracking state can become Not Tracked and the trajectory can vary a lot when hands are close together at the end of the catching movement. Additionally, without any limitations on the distance from the body to the hands, catching the fish would become far too easy. One could just hold the hand in the middle of the fish’s jumping trajectory which would kill all the fun. Especially when the bystanders would see this to be the winning strategy which would ruin the entire experience and marketing impact altogether! The difficulty level must be set exactly right even for a simple game like this.

The third and final implementation was then actually simpler than the previous two, yet more robust. Usually, I try to accommodate different body sizes in all distance calculations, but the third implementation seemed to work well with kids too so I could keep the gesture rules simple. Eventually, once a good balance between different parameters was found, we decided to let the rest go and we realized that much of the real-world disturbances can actually be used as a randomness factor in the game. Why not! As long as every player could catch at least a couple of fishes the first time, to try the game a second time was almost mandatory since we managed to induce the ”I almost got it!” feeling. Brilliant!

Even in cases where the person’s hair and rapid hand movement around the head caused skeletal tracking to mistake the hand to the hair didn’t seem to have much negative impact on the experience. As soon as Kinect resumed to track the person normally, which was a second or two later, the game went on as usual.