Mobile crap - ads

This series of articles is a rant about mobile game design. It may extend to console and PC gaming, but I only talk about what I experienced through most games featured on the iPad app store in the past year or so.

A game does not need in-game ads to be profitable. Real-money transactions can do the job if the game is polished and the business model solid. See Clash of Clans for example.

But even if we ignore the games with traditional ads, there are still many with some sort of advertising. For example, Puzzle and Dragons has no in-game ads per se, but they do frequently collaborate with third-parties to advertise their games. Among these collabs, PAD introduced a dungeon themed after Angry Birds, and Angry Birds placed a banner or introduced a temporary content themed after PAD. These collaborations may have been profitable for the marketing of both PAD and Angry Birds, but 1) the theme of PAD has now become an inconsistent mess: I would much rather have new content themed after dragons or gods than dragons plus Angry Birds plus Batman plus ... 2) in a collecting game like PAD, the players who start after a collab has ended will never be able to collect the monsters from that collab, and 3) it reminds the player that if she did not pay to play, then she is the product. As a player, I dislike collabs.

There are also many Chinese, Japanese, and Korean games on the app store. One of them is Mother of Myth. In-between levels, they frequently display an in-game ad for ... their own paid content! These ads may increase the number of payers, but 1) they disrupt the play experience, 2) why use an old-fashioned ad when a well-designed game mechanic could do it? If you have to use an ad, then either your game design sucks, or you are not selling the right thing.

And then, of course, there are tons of ad-like messages that my brain ignores automatically. One such message is login through Facebook to play with your friends. PAD remains discreet about it - you can play decently without friends. But Birzzle was much heavier and splashed a bring-your-friends ad after every level.

Mobile crap - daily login

This series of articles is a rant about mobile game design. It may extend to console and PC gaming, but I only talk about what I experienced through most games featured on the iPad app store in the past year or so.

Designers could invent new mechanics to improve the core gameplay, and thereby improve retention metrics in the long run. But they cheat: they invent new mechanics to immediately improve retention metrics, thereby ignoring or forgetting to improve the core gameplay.

Mobile games try to maximize various metrics: their number of daily and monthly active users (DAU and MAU), the average session length, and so on. To do so, they introduce game mechanics that encourage players to come back every day or play longer. In games like Clash of Clans or Farmville, the come-back mechanic is the core game loop, so it is not a problem. Waiting and coming back is what these games are about. But in traditional fighting, racing, or match-three games, the only reason why players would come back or play longer is because they like the fighting, racing, or match-three mechanics, not because they like waiting.

More and more games have been adding come-back mechanics on top of their core gameplay. In 2004, World of Warcraft started granting a rest XP bonus to players who do not login for a long time. Blizzard designers originally implemented the rest mechanic to prevent hardcore players to level up too quickly. Beta-testers hated it, so Blizzard made it a reward for casual players instead of a punishment for hardcore players. Nothing changed mathematically, but players prefer mechanics if they are framed as a reward.

More recently in mobile games, Plundernauts has been giving out a welcome-back reward for logging in. They also give out bounties which must be completed within 24 hours - an incentive to play at least a few battles during the day, most likely in one session. 24-hour bounties are great for metrics such as the average session length, but they do not make the game more fun in the long run.

Puzzle and Dragons gives two login bonuses: 1) a consecutive login bonus, with rewards increasing every consecutive day, and reset the day the player does not login, and 2) a cumulative login bonus, increasing every day the player logs in, and never reset. Also, there are 5 exclamation marks for 7 lines of text in the login bonus message. By abusing the exclamation mark they over-emphasize my login, and it looks very amateur.

And finally, mobile games such as Birzzle or Mother of Myths show a daily login calendar when the player logs in. Some calendars focus on the week (4x7 cells), others on the month (5x6 cells). The calendar resembles an Advent Calendar, except 1) nothing happens at the end, and 2) the calendar mechanic is not properly tied to the core gameplay (receive more and more gold, or increasingly better gifts). The only point of the calendar is to keep people logging in for the rewards, not to have more fun playing the game.

A lot of mobile games have mediocre and dull core gameplay. These games are often sugarcoated in happy-shiny graphics and habit-forming mechanics. None of those improve their core gameplay. These games remain shallow.

Mobile crap - clones

This series of articles is a rant about mobile game design. It may extend to console and PC gaming, but I only talk about what I experienced through most games featured on the iPad app store in the past year or so.

Ideally, designers should keep a working recipe and incorporate new mechanics into it, or at least aim for a fresh take on an old recipe. These days, entire genres should be avoided entirely: designs involving a match-three mechanic have nearly all been done, one way or another. I covered Clash of Clans clones before, Raph Koster and others came up with a small family tree of shmups, and somebody probably covered the lineage of match-three games. I won't talk much about cloning in itself - everybody knows it is omnipresent in the game industry. So I focus on the imperfect features that cloners should upgrade or replace, but actually keep by laziness.

UI-wise, some developers are really lazy: instead of trying to improve an existing one, they just copy-paste it blindly. For example, Dungeon Keeper's UI HUD is a near-exact copy of Clash of Clans. Yet CoC had, and still has, several UI issues raised on the CoC forums by the players. For example, the default battle zoom-in forced players to always zoom-out before starting a fight. The team behind CoC fixed it, but the Dungeon Keeper team could have thought about it too. Another CoC example is the spells that remain selected when players try to deploy troops, thereby wasting a spell. One solution would be to deselect the spell after it has been cast, but this issue has not been fixed yet, and it's easy to spot with playtesters. A last example of UI laziness can be found in Plundernauts. Plundernauts adapted the item game made famous by WoW and Diablo to spaceship battles. They changed nothing to the UI when displaying equipment stats. In short, if you are cloning a mechanic, at least try to question and improve the interface.

Finally, the name. I understand that developers target a particular audience. Sometimes, this audience needs to know that the game is a puzzle game. So they make the name of their game rhyme with puzzle: Birzzle, Chuzzle, and so on. It's nice, but it looks like even the names are clones!

Blizzard's Hadoop platform

Talk given by Brian Griffith and Amanda Gerdes at the OC Hadoop user group meeting in October 2013.

Blizzard uses the same Hadoop platform for Diablo 3, Starcraft 2, WoW, and Hearthstone. This platform went live in March 2013. Before this platform, game developers would log game events in log files, and use custom scripts to ETL these log files into relational databases for analysis. Problem: cumbersome, hard to maintain, low performance.

Solution: game developers, on their own, decide what to track in their game, and send that data to the platform. Instead of a log file, the game developers send protobuf objects. The 20 nodes in the Hadoop cluster receive and deserialize around a billion objects per day. The message's headers determine where to store each protobuf object within Hadoop. Blizzard also uses Hadoop as an operational data store. The cluster runs map-reduce jobs to filter and aggregate the stored protobuf objects. Currently, the 20 nodes store 60TB, but now that Blizzard realizes what they can do with Hadoop, they plan a 100-node cluster storing 1PB. Current bottleneck: CPU for deserialization. They rarely hit the disk for data so no IO waiting bottleneck.

For the messaging, they use a federation of machines running RabbitMQ. 50 producers worldwide (China, Europe, US, etc.) and 8 consumers (most likely in the Blizzard headquarters in California). When an Internet cable got cut with China, some messages were queued for 40 hours.

The analysts were using Greenplum for processing game data in parallel and ETL. Now that Hadoop is in place, they could start using Pig for ETL. But sales and customer data still come from relational databases, and the Greenplum is great at ETL. So there is no reason to force analysts into Hadoop. Solution: ETL jobs pull data from Hadoop and store it into Greenplum for warehousing. Greenplum is still in charge of its own ETL jobs.

Some data (such as a character's level or class in WoW) stay interesting across patches, but others (such as transmogrification usage) are only interesting when the feature launches. So the platform allows to build KPIs daily (e.g. activity aggregated per character level) and dive deep on demand.

Use case: WoW economy. Every gold transaction is sent to Hadoop. Reduce step aggregates by NPC id, item id, or player id. Makes it possible to:

• Check if the gold sinks follow designers' expectations.
Detect networks of gold farmers.
• Keep an eye on auction house prices.
• More ...

Game Architecture and Design - part 2 and 3: Management and Architecture

Game Architecture and Design - parts 2: Management and 3: Architecture, by Rollings and Morris, 2004

Ch9 - Current methods of team management

Several developer stereotypes pose problems.

• Mavericks are skilled and trust no one else.
• Prima donnas know they are the best and consider others as threats.
• Shy guys are ... shy, so they don't always say everything, which reduces the project visibility.
• Sleepers appear nice to their boss, but actually attack the management in their back.
• Jacks of all trades are overconfident and sell themselves too well. They can get overwhelmed.

Ch16 - Current development methods

Two quotes in this chapter exemplify the two main concerns I have with this book:

• Games are less original that they used to be. The authors explain further down that in a dozen years, graphics have improved a lot, but gameplay not as much. I disagree: with the blooming indie scene, and more and more games being made every year, this sentence sounds more like nostalgic bitterness than a constructive remark.
• C++ was considered too slow to be useful for game programming. The authors refer to Michael Abrash and his Assembly skills with so much awe that it gets a bit awkward. I think mentioning Assembly optimizations is worthless, and maybe even detrimental, to a 21st-century introductory book about video games. We have engines and high-level languages now!

Ch17 - Initial design

Tokens are elements of the game directly or indirectly manipulated by the player. Tokenization is the process between game design and implementation. Start with the token interactions and basic state machines, then the token-property interactions, and finally the property-property interactions. Example for Pacman:

Token interactions

	X	X	X
	Pacman death	X	X
	Ghost eaten	X	X

And the state machine for the ghosts.

Token-property interactions

Token	Properties	Hungry	Strong	Eddible	Weak
	Hungry	X	Pacman death	Score++	Ghost eaten
	Strong	Pacman death	X	X	X
	Weak	Ghost eaten	X	X	X
	Eddible	X	Score++	X	X

Ch18 - Use of technology

Game reviewers do not have time, so they give good reviews to the shallowest aspects of games: the graphics, not the mechanics.

Research and development is risky. Why not teaming up with a local university?

Ch19 - Building blocks

A bunch of design patterns useful for game programming. Here's chain of responsibility.

Observer:

Below is State:

Strategy:

And finally, Template:

Ch21 - Development

• Plan for reuse
• Document
• Design then develop
• Schedule ad communicate
• Catch mistakes as you go
• Limit R&D
• Know when it is good enough
• Team ownership/"invisible" management
• No feature creep
• Team solidarity

Agile game dev (5/5) - Keith 2010

Agile game development with Scrum, Clinton Keith, 2010

Part V - Getting started

Read the series: 1/5, 2/5, 3/5, 4/5, and 5/5.

Myths and challenges of Scrum

To prevent endless development without focus, the product owner should hold his vision.

Scrum is a template/framework, not a formal development process. It fosters continual improvement and transparent practices. It also promotes change, but any method introducing change should be reversible and incremental, and its impact should be measurable.

Agile has a lot of meetings because communication must happen. Plus, sprint review, planning, and retrospective meetings only take at most a whole day every 2-4 weeks, and the daily scrum should be timeboxed to 15 mins. So precious time is not wasted in excessively too many meetings.

Developers get tired and less motivated after prolonged periods of overtime; this becomes obvious when velocity is being tracked.

Working with a publisher

Problems: traditionally, publishers tie a detailed plan and schedule to their contract with a studio. The publisher is relieved: the responsibility to generate profits has now passed from their hands into the hands of the studio. Several problems arise:

• When they have problems, studios won't tell publishers by fear that publishers stop the contract. The game is playable only in alpha, and by then it is too late to fix it. Millions of dollars are wasted, and studios and publishers don't trust each other anymore.
• Publishers ask for minimum required feature sets such as "3 playable levels, AI attacking players, and online gameplay". Quality can not be defined in a contract, so developers rush for features ignoring quality.
• For business, marketing, or portfolio reasons, publishers can impose decisions on their first-party studios. These studios, feeling like they have lost sovereignty, do not want to collaborate with their publisher anymore.

Solution: transparency and collaboration to build trust.

Transparency: Following the Cerny method, studios and publishers must agree on conditions for entering prod. Pre-prod should result in production plans detailing metrics that measure cost, such as the number of people-days required to produce a level. Prod forecasts and metrics should be part of every release deliverable. Studios should also list all known risks and how much control they have on them, so that the publisher is not surprised when an expected risk happens, and eventually helps solving the risk. To that extent, the publisher should attend release planning meetings.

Collaboration happens by having a product owner on both sides. Both owners measure ROI based on the velocity of the team and the value and cost of features. They adjust the scope by prioritizing the product backlog, and can decide to extend the release date.

The publisher-side product owner is the bridge between the studio and the publisher's execs, marketing, and sales departments. He also reviews and plays each sprint build and attends release meetings. He takes care of the release goals (ie epic user stories) while the studio's product owner is in charge of the release plan (ie which user stories make it for the release).

Milestones define when the publisher funds the team, and when the team delivers a game. Funding and deliverable milestones can match, but the deliverable milestones should not be too detailed. Both parties should agree on a date range for entering production rather than an actual date: the range width will decrease as uncertainty decreases.

The publisher can opt for a kill-gate model: the studio builds 4 game concepts for a first milestone. The publisher funds 2 for pre-prod, and finally funds only 1 for production. Everyone wins: publishers reduce risk and get a game of higher quality, and teams gain development knowledge.

Launching Scrum

Progressing in Scrum is like progressing in martial arts: people start apprentice, become journeyman, and finally master.

The apprentice stage takes 3 to 12 months. Developers learn Scrum's basics: adjusting to sprint pacing, understanding that "done" includes polishing, decreasing the time spent maintaining a working build (= better tools and practices), reporting to the team rather than to the scrum master, and focusing on value rather than on tasks.

The journeyman stage takes a couple years. Journeyman developers:

• are collocated,
• use TDD,
• collaborate tightly with QA,
• create 1 polished level rather than 3 incomplete levels in 1 sprint,
• measure their velocity,
• use story points to estimate release plans,
• belong to well-established communities of practice,
• avoid hardening sprints,
• learn to adapt the basics of Scrum to the situation.

For instance, a team is facing the following problem: when a character walks, sound should be played, but animators submit their work late, and sound designers have to rush in to make the character fit in the sprint. This results in poor sound effects. To solve this problem, a team of apprentices would place a cut-off date 5 days before the sprint end for animators to submit their work. This would give enough time for sound artists to keep the quality high. However, the animators do nothing for the last 5 days, and their velocity is reduced. A journeyman team would have the animators deliver animation mockups right from the start of the sprint, so that sound artists can add their art quickly too. Thanks to better tools, animators iterate and update their animations throughout the sprint seamlessly for sound artists.

The master stage is never-ending. Only teams with good chemistry, in control of their methods, with strong emergent leaders, experts, and visionaries, all collaborating. Some master teams abandon sprints for continuous Kanban.

Company-wide adoption can happen progressively through a beach-head team of volunteers who decide to try out Scrum. They should be given an easy set of tasks to complete because a lot of management issues will be raised. After a couple releases, Scrum can spread to the company. The beach-head team can be split into 2 or 3 teams, filled until full with developers to teach Scrum to. Pro: The non-beach-head team members learn Scrum quickly. Con: the beach-head team has just been broken, and they may have liked to stay together. Alternative: keep the beach-head team intact, but have some of its members work 50% as Scrum coaches for other teams.

Agile game dev (4/5) - Keith 2010

Agile game development with Scrum, Clinton Keith, 2010

Part IV - Agile disciplines

Read the series: 1/5, 2/5, 3/5, 4/5, and 5/5.

Programming

Flexibility is essential to not waste time during development. Over-architecting can not take into account all the possibilities from the start.

XP includes TDD and pair-programming. TDD maintains stability when refactoring happens. It works best when accompanied by a continuous integration server. Pair-programming helps junior coders learn the ropes and decreases the time spent surfing the web or checking emails: when someone else sits next to you, you don't dare.

Debugging and QA happen throughout development. Bugs transform into stories, and make their way in the product backlog. Their priority is decided by the owner based on the magnitude of the consequences.

Optimization should also happen throughout development to be able to reach the shippable definition of "done". Example of a definition of "done" not for shippable but to be able to see the fun in the game: 50% of frames run at 60+ FPS, 95% at 30+ FPS, and the level loading time should not exceed 1 min. Spikes can be used to explore optimization techniques if need be.

Art and audio

Teamwork in a cross-disciplinary team turns artists who deliver art into art-specialized developers who deliver an experience. Artists generate assets ranging from textures, and models to sounds and music. To make their assets, artists rely on tools. If artists work in purely artistic teams, they observe an unavoidable tradeoff between the cost of making assets and their quality. In a cross-discipline team, tools and practices are continuously being improved, hence quality may increase while cost decreases.

Art QA is performed all along by the whole team. To test their assets in-game, artists need a build that is relatively stable. The quality of assets can be verified manually in-game and/or automatically by tools. Often, assets have to be approved by the art director to be considered "done". To save time to everyone, an evident "art director approval" column can be added to the Kanban board.

Pre-prod is really important because it is when production costs are learned. 50% of production costs come from level production. These costs can be estimated by building a vocabulary (ie patterns) of simple levels and their mechanics. Artists should also know, before entering production, the constraints and budget: how many bones per character, how many polygons on the screen, which character behaviors and motions are necessary, etc.

Game design

The roles assumed by designers are varied. Since designers represent players, senior designers can assume the role of product owner, provided they take into account cost and ROI. In teams, designers should go in teams where they know the domain: designers with UI/UX skills go to the HUD team, and senior designers to the core gameplay team.

The design document should not be too detailed. It should be updated daily with what becomes known about the game. It should not try to answer the unknown, because the only way to answer unknown features is to try them in-game. The design document also holds the vision for the game. Designers in cross-disciplinary teams relay this vision to their team and filter and adapt ideas generated by the team.

Designing from "parts on the garage floor" happens when parts are developed separately and integrated 2 weeks before shipping. The resulting game is very different from the vision, but it is too late to fix or polish.

Set-based development is preferred to point-based development. Point-based development consists of iterating each discipline in isolation from the others. For instance, the designers want the feeling of a seamless world. They think that contiguous locations could be streamed from the disc, so they go ahead with their design without asking the programmer if it is actually possible. The next build shows it is not possible with the current engine. Designers also learn that artists did not have the appropriate tools to bake streamable assets. They abandon the idea.
In set-based development, disciplines share their possibilities and constraints. In our previous example, designers would ask around whether it is possible to stream scenes to provide a seamless world experience. They get feedback from programmers, who decide to update the engine to support streamable scenes, and from artists, who request tools to bake streamable assets.

QA and producer

QA in traditional development processes happens in post-alpha, when minor tweaks and polishing are possible, but major defects can not be fixed. A studio culture that considers bugs to be part of making games does not help. Agile changes this.

In production, white-box testing can be performed in-house every day. QA takes care of regression testing on the build, developing testing tools, and testing pipeline changes. Each agile team already does QA to improve its practices, especially with TDD. Yet at the beginning of prod, teams can contain one tester to increase the quality of testing. In a team, testers help define "done" and the CoS for the stories. They also help the team keep a consumer eye on gameplay. If smoke-tests can take care of what can be automated, testers can still conduct walkthroughs to assess usability, playability, challenge, pacing, and the level of polishing.

In post-prod, or even at the end of production, testing and tuning ramp up. Pools of testers can be shared between teams. Bug-tracking repositories can replace sprint boards. QA can also organize playtest sessions using local university students or local game shop customers.

Producers in scrum can take the role of scrum master, but should never micro-manage. They can also be (co-)product owner, taking care of licensing, outsourcing, and publishing issues. They also plan resources.

Read more:

• McConnell 2004, Code Complete
• Beck 2004, XP explained
• McGuire 2006, Game design with agile methodologies (online)
• Poppendieck 2003, Set-based development (online, ch2 p26)
• Irish 2005, Game producer's handbook

Agile game dev (3/5) - Keith 2010

Agile game development with Scrum, Clinton Keith, 2010

Part III - Agile game dev

Read the series: 1/5, 2/5, 3/5, 4/5, and 5/5.

Video game project planning

The minimum required feature set are the must-have stories. For an FPS, they may be: 8-12 hours of story, shipping for Christmas, advanced NPC AI, or online gameplay.

Publishers require early concepts.

Any game development has the following 4 phases. During the concept phase, ideas are generated iteratively until the green light of the publisher. In pre-prod, typical levels and assets are created iteratively to estimate the value of the minimum feature set (which mechanics are fun, which artistic themes are interesting) and cost (hours, license fees, skills, or technology). Production relies on the knowledge acquired during pre-prod to build, at the lowest cost, the 8-12 hours of content. Production is pipelined, ie mass-produced like in a Ford factory. Post-production deals with polishing.

The production debt is the amount of work that needs to be done in production right after going out of pre-prod.

Production is assembly-line work/pipeline, there should be no starvation. This pipeline model does not fit well with the sprint model: imagine a sprint centered on a particular theme such as a dragon boss. Modelers in the team would spend the first week making the model, then animators would spend the next week animating it, but modelers would have no work to do. Each specialty end up working only a fraction of the sprint - this is not good.

Solution: lean practices, and more particularly a Kanban board. In a Kanban board, tasks flow from the left-most column to the right-most column as they progress throughout the pipeline. To pipeline the creation of characters, each discipline (e.g. modeling) should pass a task to the next discipline (e.g. animation) on its right on the Kanban board every week. Much like any assembly line, when being part of a prod pipeline, the developer can only spend a limited time on a task, otherwise he will slow everyone down.

But some tasks take more time than others. For instance, it has been estimated that it takes 2 weeks to complete the modeling of a character, 3 weeks for the animation, and 1 week to add the character's audio effects. So the character asset ideally spends a total of 6 weeks in the pipeline. This is called the cycle time, although definitions differ. But once the pipeline has been filled, a character is done every week. This is called the takt time, or production rate. In our example, to reach a takt time of 1 week, the team should have 2 modelers, 3 animators, and 1 sound effects engineer. To prevent starvation, task buffers between columns can be put in place, but they increase the cycle time (because tasks end up spending a couple extra weeks in the buffer columns).

Teams

Teams are usually cross-disciplinary (e.g. 2 designers, 3 programmers, 2 graphics artists, and 1 sound artist), self-managed, and self-organized. Yet the lead developers manage planning and resource allocation. They also mentor and review the team members so that they can improve their practices.

Working 100% on a few features is better than working 25% on many because it gives a sense of accomplishment. But it is not always possible (e.g. when 4 teams share an audio effects engineer).

Shared infrastructure teams provide low-level support that multiple games rely on. They are not always cross-disciplinary. For instance, the team working on the foundation of the engine may be entirely composed of engineers. Such teams should keep in mind their users: most of the time, they are other developers. These teams have their own product backlog, sometimes longer sprints taking into account in-house support and maintenance, and their product owner should be an exec (e.g. the CTO for the engine team).

Meeting-wise, scaling Scrum happens during the Scrum of Scrums. This meeting is attended by 1-3 members from each team, and is moderated by one of the scrum masters. Mike Cohn gives advice on the agenda and frequency of this meeting, but it's usually weekly or bi-weekly. The goal is to discuss a backlog of shared impediments, and eventually trade specialists such as a mo-cap technician between teams. Specialist-swapping requires some planning to avoid overlapping demands between teams. Another way to approach specialists consists of having them stay in their team, but dedicate only one half of their task points to their team, and the other half to support/advise other teams.

Vision-wise, if there are too many teams for 1 owner, then there should be a lead owner and a few owners, each in charge of a domain such as UI or AI. The owners should meet frequently to keep their vision in sync.

If the sprint dates of 5 or more teams coincide, then a single owner can not attend the sprint review meeting of all the teams. Pros: teams can swap members and gain a more holistic view of the build. Con: there must be an additional owner, risk of diluted vision.

Communities of practice gather developers from the same discipline, such as graphics or AI. Those meetings happen weekly or bi-weekly. In those meetings, developers share progress and information to avoid duplication of effort. For instance, no 2 teams should have their AI engineers develop their own engine on their side: they should first talk about it in their community of practice to see if other teams have coded the engine already.

Read online the chapter about teams.

Faster iterations

Development iterations suffer from overheads such as compilation, delays to propagate the tuning of a variable (need to recompile the whole game to change the HP of 1 monster), asset baking (transforming objects from the format they were created in to a format that the platform can import directly into the game), bug fixing, or administrative issues. Agile aims at reducing this overhead, as measured by automated baking/compiling tools, and plot it every day to see if changes are effective. Solutions to reduce iteration time include fixing bugs, improving asset creation tools, upgrading development machines, or being able to hot-load assets or edit parameters in game.

Tests: With small commits, the team can find which commit introduced bugs by running tests. Much like the increasingly demanding definitions of "done", there are increasingly demanding test strategies. Continuous testing happens at the order of the minute, and includes: the build compiles, passes unit tests, and passes asset validations. Hourly tests include passing platform smoke-tests (loads and starts running) or loading a level without crash. Daily tests include running a scripted bot play-through, or a QA play-through. QA can play through the daily build twice a day for 30 min. Everyone understands the stability and of a build by its file name and date: "MyGame_20120727_smoke".

Assets should be compressed when transferred between SVN repos. If bandwidth becomes limiting, either pre-transfer builds and assets overnight to those who need them, or buy more bandwidth!

Read more:

• Poppendieck 2006, Implementing lean software development
• Ladas 2009, Scrumban
• Larman 2009, Scaling lean and agile
• Crispin 2009, Agile testing
• Lakos 1996, Large-scale C++ software design

Agile game dev (2/5) - Keith 2010

Agile game development with Scrum, Clinton Keith, 2010

Part II - Scrum and agile planning

Read the series: 1/5, 2/5, 3/5, 4/5, and 5/5.

This part should be dealing with Scrum and agile in general, but in the book, a lot of the examples are actually drawn from game development.

Actors

The team is often collocated, and members organize themselves as they want. Scrum teams are made of 6-10 people. There should be designers, programmers, and artists in the team.

The scrum master is not a manager, but rather a sheep dog. She makes the bridge between the stakeholders, who talk about ROI or budgets, and the developers, who talk about gameplay, art, or engineering. She educates the team to scrum by reminding them that they own the product, and that debugging/polishing must happen throughout the sprint. She also facilitates planning by preparing meetings and monitoring progress. And she also puts problems to the front stage so that anyone in the team can help in solving them: bugs, dev pipeline stuck, buying a tool, or administrative issues. The scrum master is usually not a developer, and handles 2-4 teams at a time. In the game industry, the role of scrum master is often taken by producers or senior/lead developers.

The stakeholders are the customers. In the game industry, they are the publisher(s), marketing, sales, IP owners, the studio management, but also the players.

The product owner is the voice of the customers to the team. He knows the market, so his vision for the game is trusted, and he has the final word on the priority of the features to be developed. Increasing the priority of features with most value and low cost is how he maximizes the ROI for the customers. He plans releases (date and features in them) and contributes to sprint planning and sprint reviews. He regularly asks the top execs for advice (e.g. the CTO for technical feasibility). Examples of illustrious product owners include Miyamoto and Sid Meier.

User stories

A design document does not indicate which features are of highest priority. Moreover, updating it quickly is difficult. And finally, it does not communicate value to everyone. Instead, agile uses a product backlog, which is basically a list of features (called PBIs) prioritized by the product owner, with input from the team, domain experts, and stakeholders, before each sprint. The product backlog of a console game contains 300-500 stories, while an iPhone game 100-200.

PBIs are also called user stories. They take this form: "As a [role] I want [goal] so that [benefit]". Role can be "player", but can also be an artist using a tool, or a programmer using a continuous integration server. The more detailed the role, the better: "FPS expert player", "Healer", or "proficient 3DSmax designer". Everyone should be able to understand the story and see its value. For instance, "As an audio programmer, I want a checkbox to control the boolean bLooping" should instead be "As an audio programmer, I want to control the looping background sound effects." Stories are collected and estimated during a workshop attended by the team and moderated by the owner. They can also come from marketing and/or focus groups at the start of the project. Good stories show the qualities listed in INVEST:

Quality	Example
Independent	"As a player, shooting a door makes it explode with wooden particles" and "As a player, shooting a window makes it explode with glass particles" are dependent because they require the same tech. Combine them: "As a player, shooting props makes them explode with shards".
Negotiable	"As a driver, I want to see water spray when driving over water" does not harness the team's creativity. Who knows, maybe mud effects could be more appropriate. Replacement: "As a driver, I want to see effects when driving over various surfaces", and eventually add the CoS "Over water, there should be a water spray."
Valuable	"As a designer, I want all the rigid bodies that are near each other to be grouped into the same cluster" does not show value. Add "so that the game can run at 30 FPS" and everyone gets it.
Estimable	The owner should be able to estimate risk and whether the value of a story is worth its cost. When he does not know, he asks for a particular sprint called a spike to gain knowledge. Example of story in a spike: "As an owner, I want to see a video of the fighting mechanics on iPhone".
Small	Break down large tasks into small ones. Eventually group tasks worth less than an hour into a single one.
Testable	Use CoS, or a lead developer can approve stories by saying "OK, level 1 is fun", but that's subjective. A definition of completeness also helps: "the character model has been mocapped, rigged, textured, and tested in game".

Epics are large stories that require more than a dozen hours to implement. They are broken down in smaller stories as their priority increases and they are moved up the product backlog. Using "hours of work" to estimate a story size is not appropriate: teams work at different speeds, so "10 hours of work" for an experienced team may actually mean 30 hours for a team of beginners. Moreover, a story has to be broken down into developer tasks. In the end, tasks are the ones being estimated using points. The number of points completed per sprint by the team is called velocity, and it is tracked by the scrum master. Velocity is useful to gauge if a project is on time, and if changes to practices are effective.

Velocity and points are used to track progress on a burndown chart, where a flat line shows stagnation and a steep decline shows progress. The team can see their efficiency at processing tasks over the sprint, and determine if they are going to make it by the end of the sprint.

The burndown chart is usually hung in the war room, a room where the team holds its daily scrum meeting. It does not have chairs, otherwise people would be tempted to sit on them and not talk - hence the name stand-up meeting. In the war room, there is also a task board on which 3-by-5 cards are hung to show tasks and their progress. Cards are good because they can be customized, color-coded, hung on a side-wall, etc.

The release plan is a subset of the product backlog, detailing the release date and the release goals (in terms of stories). Like in the product backlog, the stories at the bottom of the release plan have low priority and have not been broken down. Stories are grouped together in goals and themes that can be tackled by the team in a single sprint.

The conditions of satisfaction specify the tests that a story should pass, ie the definition of "done" for that sprint. For instance, "As a player, I want to see enemies react when shot" can have CoS written at the back of the story's 3-by-5 card like "when shot in the head, knock-back; when shot from the right, shift left". The definition of "done" can take multiple predefined levels, such as "Runs on dev machine", "Runs on all platforms", or "No memory leak". The magazine demo usually has a demanding "done": fun and challenging, no large memory leak, no missing asset, and a clean UI. It may require a sprint dedicated only to polishing, called a "hardening sprint". The debt induced by a sprint is the amount of work remaining to reach the shippable definition of "done" after the sprint is over. When definitions of "done" are stricter, ie closer to the shippable "done", there is less risk of debt. Hardening sprints reduce debt at the expense of time.

Meetings

Meeting	When	How long	Who	Why	Details
Sprint prioritization meeting	Beginning of each sprint	Until the sprint is filled	Team, owner, scrum master, and eventually a domain expert (e.g. network programmer or mo-cap technician)	Prepare the product backlog and identify sprint goals. Establish 2 themed sprints worth of PBI. It is the opportunity for the team to ask questions to the owner and understand PBIs and the vision.	Sprint planning: The PBI "player can jump" can be broken down into the sprint backlog tasks "make jump animations: 10h", "program jumping controls: 5h", "program jump physics: 15h", and "make the sounds for jumping: 2h". PBIs at the top of the product backlog should require less than a dozen hours, and fit in a single sprint. If not, they should be broken down into smaller ones. All the tasks coming from a PBI should fit in one sprint. If not, break that PBI in smaller PBIs. PBIs at the end of the product backlog are vague and need to be broken down and detailed when their time comes. For instance, "online play" may be broken down into "lobby", "capture the flag gameplay", and "multiplayer analytics". If the team overestimated the work to be done, and has finished a few days before the end, call a mini sprint planning meeting and add a couple small PBIs to work on until the end of the sprint.
Sprint planning meeting				Add tasks to the sprint backlog from the product backlog. The scrum master first identifies constraints against the goal such as holidays or engine updates. The team and the owner then discuss the design and implementation of the highest-priority PBIs, break them in tasks, estimate the amount of work for each task, and add those tasks to the sprint backlog until there is work for the whole sprint. The owner has the last word on which PBIs the team should work on for the sprint.
Daily scrum	Daily	15 min, timebox	Team, scrum master	Share progress and impediments. Each team member explains what they've done, what they're going to do, and what is slowing them down.	"I spent 6 hours on that bug and still have not figured it out" calls for more people to look at it.
Sprint review	End of the sprint	~ 1 day	Team, owner, scrum master, stakeholders	Review of the sprint by the owner, who plays the game or watches a demo, and updates the product backlog accordingly. If not satisfied, he decides which features go back to the product backlog (to be worked on again) or are deleted. Not all stakeholders need to attend, but the publisher should be present.	Large projects with many teams should have a single big meeting with everyone. Pros: It saves time for the owner and gives a unifying vision to everyone. Con: the owner may take it as a ceremony since it's rare and there are lots of people. Alternative: the owner can review each team separately. Pros: more casual, detailed, and direct. Cons: more time consuming. When the sprint review is over, productivity won't be 100%, so the team can spend the rest of the day celebrating or playing games.
Retrospective meeting	Just after the sprint review	30min to 3h, timebox	Team, scrum master	Talk about which practices work, should stop, or be tried. The scrum master keeps track of answers and whether things are fixed or not.	"The build server should send an email when the build is broken" gives an action item for the sysadmin/build team. If no sysadmin is in the team, the scrum master should go ask the sysadmin.
Release plan meeting	End of the sprint	4-8h	Team, owner, experts, stakeholders	Discuss epics, story themes, and release goals. Prioritize and estimate stories, set the release date, and select the length and goals of the next 2 sprints.	For each story, everyone raises a card showing their point estimation. Estimations can only take values in {0 (trivial), 1, 2, 3, 5, 8, 13, 20, 40, 100}. People debate and re-vote until everyone shows the same points. Different estimates should not be averaged: this would hide matters that need to be discussed.
Planning poker				Team, owner, and experts determine the workload of a story based on similar stories already completed. Points are used, not hours, because teams have different velocities. Spend a couple minutes per user story.

Miscellanea

The sprint goals/PBIs should not change throughout the sprint. In a way, the team is "locked" on those PBIs for the duration of the sprint, unless the owner asks for a sprint reset. In a sprint reset, the team stops the tasks it was currently working on and calls a new planning meeting. A reset can happen when big goals change, such as when the CEO wants a new feature in 2 weeks to demo the game and raise funds. The team and owner should assess if it's doable, then reset. Resets can also happen when the sprint is running out of time, working overtime is not possible, and the owner does not want to remove lower-priority features from the sprint.

Sprint length: 2-4 weeks, but the owner has the last word. The length should not change too often, otherwise teams can't get used to it. Shorter sprints are appropriate in uncertain contexts such as pre-prod (when customer feedback is frequently required) or for teams new to agile or to working together. In general, 10-20% of sprints should run out of time before they end. If less, it looks like the team is fearing commitments. If more, it means they keep miscalculating, and may not take their commitments or estimations seriously.

Read more:

• Cohn 2006, Agile estimating and planning
• Cohn 2004, User stories applied

MVC and game developers

MVC in software engineering

The model stores data, and runs domain-specific logic. An example of domain-specific logic in the banking domain is if customer's account is below zero, refuse withdrawals. The view gets data to display from the model. Some MVC have the view pull data from the model itself, others have the view notified by the model, and others have the controller tell the view to update itself. The controller receives and converts user inputs in actions understandable by the model. In web development, such as ASP.net or Java servlets, the controller picks which view is going to be displayed, and then the selected view is actually in charge of displaying information on the screen. If this description is not clear enough, the wikipedia page about MVC has a scenario that helps explain how the model, view and controller fit together.

MVC in games

Many amateur game developers say ideas from MVC can be applied to games but in general it is just a waste of time. For many, there is no such thing as game architecture: The only difference with classic MVC is that in a game, the graphics data, logic and game algorithm is fused together. Others argue that in a rapid prototyping context, where iterating is key, one should not waste time conceiving a modular/maintainable system, but rather focus on testing the game as fast as possible.

Many non-indie games are made with MVC in mind. Splitting data from presentation induces extra work on the code and managing slightly more objects in memory. However, this is exactly what allows an easier testing and debugging of mechanics, UI, and network components. In fact, MVC has been applied to game programming since at least 1998. A blogger at TigSource states that MVC can be applied in any part of a game. Taking the example of a network connection: The input view in this case is a network receiver running in it's own thread and listening for data from a socket. When the view receives data, it's translated into an internal class and sent to a controller for processing.

Middle ground

I find that explicitly implementing an MVC pattern for the network connection is a bit of an exaggeration. Heads-up displays and other UI input perfectly fit as controllers, and are naturally decoupled from the model. On the other hand, there is a strong tendency by many game developers to ignore software engineering practices. Some even put forward the Extreme Programming side of rapid prototyping to justify their ignorance of any architecture. Agile methods do not contradict the use of any architectural pattern.
That may explain why so many of pygame's projects contain a single file with thousands of lines of code ...

Player-developer communication in EVE Online

CCP has an interesting take on community management. The developer's blog of EVE contains interesting behind-the-scene technical information. What is even more interesting is that these articles do not seem to be filtered or censored by any community manager. Articles are direct from the development team. I guess that the management team originally thought this communication policy was a good idea, and the developers were more than happy to post about their work. In any case, this communication policy has advantages and drawbacks.

Drawbacks

On the one hand, revealing to your players that a major database update requires a 14-hour downtime and may result in more bugs in the game (October 2010) is kind of risky. Another developer article (February 2011) started with:

Testing may show that the changes made are not safe and we won't be able to use them. I thought this would be interesting enough to share, but please do keep this disclaimer in mind as you read on.

Players start complaining on the forum boards, and it gets worse when the down time actually takes longer than the announced 14 hours. Moreover, when developers and players get close together, developers may get involved in the game and start giving unfair advantages to the players they like. This kind of cheating can get really messy (also look at the Wikipedia summary of the events).

Advantages, and why it works for EVE

On the other hand, EVE benefits from a huge support of its player base. A bunch of volunteer programs, composed of hardcore and loyal players, add content into the game, administrate a wiki, and provide player support at no cost for CCP. EVE designers targeted the game's steep learning curve for hardcore players. It's not because you are a hardcore player that you never need information or help. In fact, I think it's the opposite: hardcore players bookmark any possible source of information for their game. Therefore, EVE needs its active player community, and needs to keep a tight relationship with its player base. To receive fast and broad feedback, CCP opened nine player positions in the Council of Stellar Management in March 2008. Members of this council were elected for six months by all EVE players and invited to the development studio. They gave feedback to the developers in person. This council may have also been a marketing coup from CCP, as the studio also invited various journalists to cover the event.

So far, tight player-developer communication has been a winning strategy on many fronts for CCP. Yet, EVE does not put forward any charismatic developer. Maybe is it better like this?

Indie game developement and polishing

Video games are among the most complex softwares to design and implement. They deal with graphics, database, UI and so on. Yet programming them is very accessible: there are books, forums and frameworks for game programming in nearly any language. All recommend using rapid prototyping, and player-centric game design requires a design -> code -> test -> redesign loop as short as possible. After a few iterations in the loop, game development can get very messy if the code has not been organized into components. Software components are embodiments of the software engineering principles of encapsulation, abstraction and modularity (Taylor et al. p69). In other words, building a game from early prototype to end-product requires modularity (and therefore components). More generally, modularity is the key to adding and editing features in rapid prototyping and agile software development processes.

But looking at game repositories such as pygame, it seems the myriad of small projects it contains seem to pay no attention at all to modularity. I think they may have started all right, but as development progressed, the code got tangled without them noticing it. Locating themselves in the development process became more and more difficult. In the end, their project does not have any distinct component, they have three 500-line files named "main", "game" and "classes". Each of these files contains more or less the whole game and does not have any comment. The game is certainly working, but it is ugly and buggy and does not give a good experience to the player.

Braid is the opposite. It has been polished to provide a unique experience to the player. Images taken from the Gamasutra article about The Art Of Braid: Creating A Visual Identity For An Unusual Game.

Reknown game developers such as Chris Hecker ranted about unfinished games. Hecker gave the example of games (or rather game design proofs-of-concept?) developed in less than 20 hours. Cactus, the main indie developer targeted by Hecker, explained that his mentality was to make games available for others to be inspired by. Other indie developers of growing fame such as Mathew Galant disagreed cautiously: nobody can create a masterpiece without first making a hundred crude sketches, and It's better to get those ideas out there, even if they're flawed and incomplete!

I think there are very distinct categories of indie game developers: veterans, explorers and beginners. For veterans such as Hecker, agile processes promote sustainable development (taken from the Agile Manifesto). For explorers such as Cactus, agile is actually cowboy coding (not in a derogatory sense). For beginners, agile is a way to learn. Voluntarily or not, explorers and beginners do not pay attention to modularity. In my opinion, this is what sets apart veteran masterpieces from beginner and explorer trials.

Comparing video games to films - 3/4

You can read the first, second, third and fourth parts of this article.

Findings

During my research on the respective histories of early Gothic horror movies and early video games, I have noted several similarities and differences between the two media.

The earliest film of Human history was Roundhay Garden Scene in 1888. The earliest video game was the Cathode Ray Tube Amusement Device in 1947. Both have the same nanoscopic size in today's cultural literacy: aside from field experts or historians, no one knows about them. In 1902, ie 14 years after the first movie, Le Voyage dans la Lune was realized by Méliès. The movie stays notable and even praised in our modern society. More than a century after its release, Ebert even wrote the movie had artistry and imagination. Surprisingly, the video game released fourteen year after the first video game was ... Spacewar!. Some mention modern video games owe a lot to Spacewar!.

However, Steve Russell is much less famous than Georges Méliès. Ebert considers games are not art because Video games by their nature require player choices, which is the opposite of the strategy of serious film and literature, which requires authorial control. Could video games be more artistic if their authorial intentionality was stronger and more obvious? The Civilization series have not (yet) been considered artistic by Ebert, although the name of their designer, Sid Meier, is clearly associated to the title. The same way James Cameron (and not the actors) is mentioned on Avatar's poster.

The early video game industry seems to have been much more competitive and business-oriented than the film industry. For example, Atari and other companies were "inspired" by the Odissey to make their own home console. Hence in 1975, Magnavox started filling lawsuits against everyone. I do not think such a hard competition happened between early film makers.

Moreover, while the first movies were not much more than recorded theater plays, the first video games were geeky electronic hacks. Tennis for two had been created to attract people to the exposition. It seems many console companies jumped at the chance of copying Odissey just to make money. Moreover, computer game developers of the 1970's such as Don Daglow were just writing games for each other for fun. These approaches differed a lot from the clearly artistic approaches taken by the early movies of the 10's or 20's.

Video games are seeking legitimacy

In September 2008, Steve Gaynor, a game designer, formulated what legitimacy was: a broad cultural relevance to the lives of the general population. In other words, it is not exactly whether video games as a whole have an impact on society at large (financially and so forth) but whether content of the medium itself is relevant to, say, your grandmother. Did Battlezone or Centipede speak to her personal experience?. He gave another illustration when he mentioned that a senator villifying [sic] video games to get his name on people's lips means that the medium is divisive. It also means that the works themselves are completely irrelevant to the senator as pieces of entertainment or expression, or else he'd be enjoying and defending them.

Leigh Alexander asked in April 2009: aren't the cultural and practical differences between film and games so broad that it's useless to analogize? Later in November, she detailed her point: repeatedly raising Kane is amateurish and useless. It's self-defeating shorthand for what Bogost and Wasteland correctly identify as the real desire: legitimacy for games. Matthew Wasteland (a gamer/game critique) formulated the real issue behind the Citizen Kane effigy: people want games that will artistically legitimize them to everyone who doesn’t play them. Eric Swain paraphrased that the "Citizen Kane of games" buzz is about gamer's insecurities and wanting a title to point to that everyone [society] will recognize, though may not have played, as art like Citizen Kane. They [gamers] want that so they wont feel insecure when they talk about thier [sic] hobby. The whole question has nothing to do with intellectual stimulation.

Bogost argued that the artistry of a media can not be established as it used to be half a century ago: Legitimacy has become distributed, a mesh. Indeed, video games are not movies, and the social context has changed a lot between the establishment of film legitimacy and video games birth. With the recent social changes in mind, Lafarge gave in the second paragraph of WINSIDE OUT several areas of change that she thought made games metamorphosing into a richly expressive medium:

• the convergence of games with fiction and art
• shifts in representation and the deployment of information in games
• the assimilation of a filmic first-person point of view
• the growth of a culture of cheating and hacking
• rethinking of the win-lose dichotomy
• the development of immersive role-playing and emergence of cooperative relationships as central to game play

Read the fourth part.

History of RO emulators

This post follows this one about private server emulators. To my mind, understanding the story behind RO emulators can help explain how some servers easily reach 140,000 accounts and 1000 PCU as of March 2010.

This cladogram took me roughly 20 hours. I relied on a very stuffed tree with no dates (made by Tsuyuki before 2008). Other sources are mentioned in the SVG cladogram here. A (google) translation of the Chinese website I gave for AEGIS can be found here. My cladogram ends in 2008 because nothing has changed since then, except maybe a recent ASB battle cry. Many ending dates could not be found, so I sometimes ended them arbitrarily.

I could not find when AEGIS had been stolen from cRO. Given that cRO opened in May 2003, and UniAEGIS and ASB, based on AEGIS, started in early 2003, there seems to be a time problem. Unless cRO was in beta before 2003.

I could not make any link between Athena and AEGIS. Both seems to come from Japanese developers/hackers, but I do not know if Athena was reverse-engineered from scratch (in the pure sense of an emulator) or based on AEGIS.

Cladogram made with gnuclad and converted from SVG to PNG on fileformat.

Edited March 22 to add Cronus, eAPP and other AEGIS leaks and ASB updates. Split of Weiss (delphi/pascal) and Athena (C).