North Kingdom

(Sorry, this is the longest Behind The Scenes post ever!) Despite the short timeline on this project, it had many moving parts – we worked across four countries and many more cities, with technology we hadn’t used before, and with 27 team members on the project at its peak.  Looking back it is quite amazing how far we traveled in such a short time.  We had never done an interactive billboard before, but if you’re going to do one you might as well do one in Times Square right?

Y&R New York came to us to help them build a “good news”  billboard in Times Square, with a digital character to help spread that good news.  Inspired by the idea of an “alive” character, our design director remembered a creature called Gorg, created for a Swedish snowboard company several years ago (that’s Gorg in the sketch below).  We started to think about how a textured creature like that might work in Times Square, and how he might interact with people rather than just broadcast news.

…And so Gil (that’s Life is Good backwards) was born.

The idea was to keep the space very clean to distinguish our little guy from the clutter of Times Square.  And it seemed that all the other boards in Times Square limited themselves to loops – even the famous Forever 21 billboard does the same thing over and over again.  We wanted Gil to be different, and to feel like he was adapting and responding to the pace of life in New York.  We wanted him to have a level of artificial intelligence so he could be responsive.

TAKIN’ IT TO THE STREETS

Having a more physical style to Gil (as opposed to something too slick and robotic) meant we could also envision him actually taking a break from the screen and hanging out with New Yorkers in the “real” world.

And then we thought about really making this guy alive, and setting up his whole move to New York part of an ongoing story, so building a backstory about Gil moving to New York and then finding his “apartment” in Times Square, where he would set himself up as the world’s first Good News Ambassador.  We wanted to plaster the area around Times Square with posters, inviting people to come to Times Square for the launch night, but all within the “Gil’s apartment” storyline.  We decided to create a costume which we could use for Gil to attend Yankees games or grab a burger at McDonald’s.

CHARACTER DEVELOPMENT

Our first big challenge was defining the character (for a more detailed look at this process, check out our Design Director’s blog The Making of Gil over at Design Chapel).  Working with our favourite illustrators Therese Larsson and Anton Stenvall and our friends at Zoink Games in Gothenburg, as well as North Kingdom’s own talented designers, we created literally hundreds of potential characters.  You can see we went all over the place with these guys,  we had to find something lovable, something LG, something happy but still cool.  No easy task.  Here are a few sketches, but it’s hard to explain how much we worked on this – we have a discussion thread 262 messages long!

Working with the Y&R creatives and several illustrators, eventually LG chose the sketch in the top left here by a company called Bigshot Toyworks. From there, Rob (Design Director) and Klaus (Animation Director) developed the character to make him a bit more sophisticated.  It was important to keep the designers, 3D team and tech guys all involved in the character development as so much depended on the 3D model.

GIL’S WORLD

We wanted a low-polygon kind of feel to Gil’s “apartment”, partly to keep it clean and uncluttered to contrast with all the other stuff in Times Square.

Here’s a test video of Gil “reading the news” (and doing his best Larry King impersonation):

Art Director Carolina Cwiklinska and designer Riccardo Tagliabue put together a clean but colorful style guide for Gil’s world.  Elements were kept very tactile and three-dimensional, usually hanging on strings, and arranged on the z-axis to give a sense of space.  The curve on the board and the view from street level created a lot of challenges with perspective, and we had to keep Gil from lingering in the curve of the board or placing objects on the “floor”.

Regarding props, we decided to go for a “Stanley Kubrick meets LG” sort of feel, so that everything was kept simple but still quirky.

A big challenge though is that what looks good on a nice pretty Mac monitor doesn’t always look so good on an 80 foot wide LED billboard!  When we got to New York, a big setback was that our tests showed that light colors didn’t work well on the board, so we had to quickly rethink our color palette to be a little darker and yet not turn the overall feeling too sombre.  We also had to test textures that would help conceal the LED patterns in the background, but discovered it was easy for details to fade into oblivion.  Another challenge was the fact that the colors looked so different on the board to what we could see on a monitor.

We also had to adjust our handcrafted fonts to be more legible, and then sneak over to Times Square in the middle of the night to test them on the real screen. But in the end the brighter palette, realistic textures and darker block-colored backgrounds looked different to anything else in Times Square.

We also created a custom 3D font and combined it with DIN (also imported into 3D) to have enough variation.

TECHNICAL DEVELOPMENT: UNITY 3D and REALTIME INTERACTION

Technically, the first thing we needed was some R&D. This was the first time we worked with creating content for a billboard and this was no ordinary billboard, It was a 50 ft. high x 80 ft. wide free form LED screen with a 90° bend placed in the middle of Times Square.

Since we wanted our character to be dynamic and able to respond to user interaction in real time we chose to work with Unity3D. This made it possible for us to work with a 3D character with high enough resolution and with a nice furshader he looked awesome! From the start we also had an idea of that we wanted to be able to control the character live, as a puppet or a videogame character. This was another reason to choose a real-time rendering approach. The live puppetering tool we built was later used at the launch day to answer questions from people on the square.

As a proof of concept we took our initial character and  created a demo Unity App where we could control our little character and make him do whatever we wanted (including our favourite, the “sexy tiger-striped cowboy”), with all the animation being rendered dynamically.

Another part of the R&D phase was to figure out how to make it as stable as possible, the project was designed to run for a long time. Since we didn’t expect the Unity client to run forever without crashing we decided to use two identical PCs for the Unity client and move all logic to a Mono application (called The Brain) running on a Linux server. This in combination with a custom built observer system connected to all three machines, a KVM switch and a couple of iBoots made it possible for us to handle crashes and daily reboots without it showing on the screen and without the character forgetting what he was doing before the crash. This worked surprisingly well after a few tweaks, but it also meant we had to ship identical KVM switches and test machines to our teams in Gothenburg and Slovenia to make sure we were building on the same system as would be live in New York.

The third part of the R&D was to investigate the different sources of Good News and how they could be fed into the system. We wanted the Unity app not only to be manually controlled but also integrate various feeds and interactions like weather, RSS news feeds, SMS, Twitter stats and tweets, product videos and other predefined messages based on time and day.

For SMS messaging we used Tropo and were really pleased with the ease of use of their service. We also looked into ways of using video image analysis of the square and mobile phone games like MegaPhone or Quotia to make some fun interaction with that but decided to leave that to the next phase because of the tight deadline.

We also decided that the character was to present all these Good News based on a schedule split into 5 minute blocks introduced by video “bumpers”. That way he could follow the rhythm of the city and present his content in different ways depending on the time of day.

We built a web app on an Amazon cloud server (called The Mothership) that took care of all the sources of Good news and then built a CMS that controlled all that data, statistics, settings and the schedule. Of course the News, SMS and Tweets would need to be moderated by a human (and for SMS that moderation had to be in realtime) so the CMS also had to have an easy to use tool for the moderation team.

The Unity client with Gil himself had a large library of animations and props that could be controlled from The Brain app. To have some control of Gil’s actions we defined a large set of behaviors that Gil could draw from for each part of his schedule and sent them in XML format from The Brain to the Unity client. The Brain pulled the content from The Mothership web app and told the Unity client what to do with it.

The whole setup was housed together with all the equipment needed to get the image up on the billboard in a a tiny corner of a tiny room that we affectionately called “The Meatlocker”, a small metal box behind the LG sign on the roof of a building on 7th Ave and 45th St.

We also created a “Puppetmaster” mode where we could manually control Gil.  Thanks to the realtime rendering allowed by Unity, we could have Gil respond dynamically to input and also write responses to messages received from passersby through SMS or Twitter.

MOTION DESIGN

Motion design was an important element of the overall design, as this board was going in an environment where there is so much movement and clutter, we wanted something very clean, striking and simple.

Once we had decided on the different ways Gil would share “good news” with the world, motion designer Félix Hill created beautiful video “bumpers” for each section, and an animated GOOD NEWS logo.  We had to be careful not to feel too much like a one-way broadcast tv channel, but also to make sure it was clear what the purpose of the board was. (A whole lot more of Félix’s process is described, with plenty of video, over on his site at work.felixhill.com).

INTERACTION DESIGN

Our research at the beginning of the project told us that if you get 6 seconds of attention towards a billboard in Times Square, then you have done well.  But we wanted more than 6 seconds, we actually wanted people to send in texts and tweets, take photos of Gil and start to feel like he was a New Yorker as well.  Designing interaction for a billboard was a completely new ballgame for us – getting people’s attention enough to urge them to interact, and also working around a corner: the big curve in the middle meant mostly you could only see one side as a spectator.  As a result, Gil had to be balanced with other elements such as calls to action or message boards.

ANIMATION

Klaus Lyngeled at Zoink Games was our animation director, coordinating work between Paris, Gothenburg and Skellefteå and working closely with our Producer and Art Director in Stockholm.  The team in Stockholm didn’t have time to storyboard out the hundreds of animations we needed, so we got our intrepid designer Riccardo to act them out and then sent movie clips to the animation team to run with (and probably laugh at).

The animators had to get used to working with the Unity animation rig, which only uses bones and doesn’t support morphing.  This made it a learning curve for some of the team, and the animations were done with what Klaus called “naked Gil”, so minus the shaders — which sometimes produced furry surprises when the shaders were added later in Unity!

All the tiny little animations were then put into Unity to be controlled by the logic of “the brain”. It took a lot of work to get the shaders, fur textures and colors and even the feeling of his eyes just right.

Some things that seemed like they should have been easy weren’t.  For example when Gil throws a paint ball against the wall and it smashes to display an SMS, the paint ball splashes required us to call our friends at Redrum Post in Stockholm to help us model them.  It took us weeks to get them the way we wanted them. Also communicating the subtleties of expression had to be done carefully so that Gil’s personality was consistent (we eventually used pictures of Laura Dern to illustrate what Gil’s mouth would NOT do!)

THE COSTUME

Meanwhile, we also had to work on developing the costume that we would use when Gil appeared on the streets of New York – and also at the actual launch, where “physical” Gil would accept the keys to his new “apartment” from an LG exec and then wander across Times Square to the billboard where he would suddenly appear in digital form.

We worked with the creator of the original Gorg, Tobias Allanson, to build us a Gil costume which was flexible enough to fit various actors without losing Gil’s distinctive shape.

The costume version of Gil could blink using controllers in the hands, but it was so hot that we had to install battery-operated computer fans!  Despite some confusion thanks to the Christmas season (“is he a reindeer?”) New Yorkers loved him, and we even got fan mail begging for more Gil.

NORTH KINGDOM NEW YORK

For the final two weeks before launch, a team from Stockholm, Gothenburg and Slovenia relocated to a hotel room in New York, which we barely left the entire time (our Art Director’s boyfriend referred to us as Room 919 NK Special Ops.  Which wasn’t so far from the truth- the bedroom was full of walkie-talkies and we suspect the cleaning lady thought she was working for Jason Bourne).  Testing proved to be very difficult, as building The Brain on two PCs shoved into a bathroom, testing against a Unity Client previewed on a Mac in the bedroom, and then trying to get results consistent with an 80 foot billboard located several miles away with no quality internet connection wasn’t exactly an ideal setup.

We worked closely with D3, a bunch of fantastic guys in New York who have built many of the boards in Times Square, and Innotek, a division of LG, which flew technicians in from Korea.

Some of the challenges we had to overcome included snow chaos and missed flights, an SVN system which liked to overwrite files, missing computer hardware which we discovered had been sitting in our hotel storage room for two days but no-one bothered to tell us, time differences, food poisoning, an issue with the LG hardware which meant the board resolution was changed the night before launch, hotel staff interrupting us literally five times a day to bring us slippers, and a mysterious silent man sleeping in the “meat locker”.  Needless to say it was a very intense time, survived largely through a constant supply of coffee from the Swedish cafe across the street, and a growing fascination with The Hoff.

LIVE PRESS CONFERENCE AND THE BIG LAUNCH

On December 14th 2010 the project was launched with what was probably the world’s first live press conference with a billboard.  People could SMS their questions up to Gil, who would then write his response on a piece of board and display it.  This is when some enterprising chap decided to propose to his girlfriend via Gil – apparently she said yes!

Here’s Gil receiving SMSes on launch day:

And here’s a little iPhone video of the mad panic press conference in the meatlocker (as you can see, it was a bit of a squeeze!):

GIL – THE FUTURE

For 2011 the plan was to greatly enhance Gil’s Artificial Intelligence logic so that he could be really “alive”, add in personalized greetings using geographically-based check-ins on Foursquare or Facebook Places, allow Gil and his environment to visibly respond to time of day, mood and level of interaction, and introduce a real sense of daily life by having him sleep, wake up, shower, eat and participate in life on the street below.  We also considered ideas like “hot words” so if you texted a message with the word “kiss” in it, Gil would react by blowing a kiss. And we had plans to install a high-res camera feed up to the control room so that Gil could surprise his audience by mimicking them or writing to them unprovoked.

Unfortunately LG has now cancelled the project, so we can’t build on our ideas, but we are proud of what we managed to do in just 6 weeks of production, and we hope that we at least made someone in Times Square smile…

(You can see more behind the scenes photos on Carolina’s blog, Félix’s site, Riccardo’s blog, Adele’s blog and Rob Lindström’s Design Chapel blog.)

CLIENT: Y&R NEW YORK
TEAM:

NORTH KINGDOM

Carolina Cwiklinska-Lead Art Director
Adele Major-Producer
Alfredo Aponte-User Experience Designer
Félix Hill-Motion Designer
Riccardo Tagliabue – Designer
Daniel Isaksson – Technical Director
Robert Lindström – Design Director
Lucian Trofin – Animator
David Eriksson/Daniel Ilic – Creative Director
James Robinson -Account Director

and our partners

UNITY DEVELOPMENT

Mikael Emtinger – Lead Unity Developer
Edvin Besic – Developer
Lasse Järvensivu – Unity Developer
Henrik Andersson – XML Developer

3D MODELLING AND ANIMATION

Klaus Lyngeled – Animation Director
Alain Maindron – Lead Animator
Martin Rasmusson – Texturing and animation
Peter, Peter and Jonas at Zoink Games

BACKEND AND CMS
MOTIVITI

Stephen Donnelly – Program Manager
Tadej Gregorcic – Lead Developer & Project Manager
Saso Horvat  - System Development
Aljosa Hancman – Web Development
David Delibasic – System Development & Webadmin
Ajda Gregorcic – Design
Jernej Kocjancic – Project Management

COSTUME

Costume created by Tobias Allanson

With big thanks to our friends at:

D3 LED (New York)
Redrum Post (Stockholm)

and an extra thanks and hug to the boys from Public Class (Stockholm) who turned up at North Kingdom New York armed with laptops volunteering to help out!