Hello.
I'm Naoki Sakaue. I work as a QA engineer in "Ryu Ga Gotoku Studio" of SEGA Games. I have been creating the video games of "Ryu ga Gotoku/Yakuza" series about 10 years.
In the following article, I will explain the role of a QA engineer and playback the game development history of "Ryu Ga Gotoku Studio".
- What Do We Do to Test the Games?
- Works of a QA Engineer
- (2009-) Automated Playtesting
- (2013-) The Birth of a QA Engineer and the Acceleration of Automated Testing
- (2015-) Analysis of Testing Result
- (2018-) Thought of Test Pyramid and QA Engineering in the Future
- Conclusion
- Reference
What Do We Do to Test the Games?
Before we go on to the main topic, let me explain what a QA in the game development is. QA stands for Quality Assurance. When we use a word "QA" in the game development, it usually means the work itself but also the departments or the project teams, depending on the organization. The basic purpose of QA in the game development is to find bugs and make sure if the games are working correctly in required quality.
Then, what do we do when we use a word "Test" in the game development? In most cases, it means to play the games manually to find bugs and fix them. (Unit testing and smoke testing which are well-known in the software development are also held in the game development, and I will explain them later in this article.)
The games under development are checked by people called a "QA Tester" to see if there is no bugs left in the games, the products are well followed by the specification documents, the games meet the product standard, and the games are come up fun as the final products. Their works cover a wide range, such as a simple testing of crashing a wall to find collision bugs, finding failures triggered by a complex procedure or specific timing, and a physical testing of plugging or unplugging cables.
It is likely that people think of game testing as just a simple process of finding bugs, but it is only a step to QA and not a goal. The main purpose of game QA is to assure that the games are working correctly in required quality.
So, this process is very important that the games cannot be made into the final products without it!
Works of a QA Engineer
Recently, as the game development is getting bigger and bigger in size, it’s becoming more and more difficult to test and develop the games manually. To solve such problems by engineering technology, a new profession called a "QA Engineer" (like me) was born. What is required for a QA engineer might differ by organization. At our office "Ryu Ga Gotoku Studio", the QA engineers make efforts to automate the environment of the game development and QA and to increase its productivity every day.
At our studio, to improve the productivity of the game developing environment, we automate build and convert of game assets, and improve efficiency for the workflow of issue tracker (Trac/Redmine/JIRA etc.). In some offices, people called a "Build Engineer" do these jobs. But at our studio, the QA engineers usually take these roles, and sometimes ask for help from other engineers of our team or different departments, depending on the situation.
In this article, I will mainly explain the game testing on my work and can give you only a brief explanation of the automation technology for the game developing environment. So, if you are interested in the automation technology and want to know further information, please check out the following link of Japan Symposium on Software Testing in Tokyo 2016 (JaSST'16 Tokyo).
(2009-) Automated Playtesting
From here, I will move on to the next topic "Automated Testing", which is one of the main works of a QA engineer. I will explain how we came up with a new profession “QA Engineer” at “Ryu Ga Gotoku Studio” through a history of “Ryu ga Gotoku/Yakuza” series.
We have been releasing "Ryu ga Gotoku/Yakuza" series every single year in Japan, and the latest title which was released this year is the 14th. To release the mass scale games in such a high speed, at “Ryu Ga Gotoku Studio”, each one of engineers and a team as a whole always have been thinking what we can do to automate the game developing environment and to enhance its productivity. This spirit gave a birth to automated playtesting!
When we had started developing games for PS3 generation, we had developed our original automated playtesting system “AutoTest” and started operating it. This is a system which enabled PC and development machines to run through automatically throughout the midnight, so that they can detect errors even after the engineers left the office.
In the following figure (Figure 1), "AutoTest" gets the latest build and starts to play the game automatically, and Player AI (like bot) can emulate pushing buttons and play the game automatically.
Figure 1: The Architecture of "AutoTest" System
At the beginning, we tried to make the player in the game move randomly. We made them crash into a wall to find collision bugs. And then, we gradually expanded the area of automated testing, and now that we can clear the main story of the games with the replay data on the basis of playing the games in manual testing! Therefore "AutoTest" can expand the area of automated testing.
We kept brushing up the functions of "AutoTest" even further. In the game of “Fist of the North Star: Lost Paradise”, "AutoTest" cleared the main story all by itself. For “Fist of the North Star: Lost Paradise”, we made additional functions to automated testing which can cope with widely varying situation, and now that it enables us to find various types of bugs. For example, "AutoTest" can select the bot AI that mashes the decide-button during dialogue scenes, and changes to the battle’s bot AI when it attacks enemies (Figure 2). It can select bot AI that meets the circumstance of the game.
Figure 2: "AutoTest" Demo (Fist of the North Star: Lost Paradise)
Furthermore, an effective cooperation system is built up between manual testing and automated testing, so that "AutoTest" can take charge of testing numerous combination steps that can be hardly handled in manual tests.
You may think that the more the automated playtesting finds various patterns of bugs efficiently, the better works we accomplish as a QA engineer. But it is preferable that AI technology concentrates on removing only easily-find-bugs out, while we take our time to find more complex bugs, because we always want to focus on improving quality of our games and create the best and the most fun games ever before. The automated playtesting does support and help the manual testing, but not all the process of game testing can be easily replaced by the automated testing.
(2013-) The Birth of a QA Engineer and the Acceleration of Automated Testing
We have been improving and adding new functions to the automated environment for the game development and QA every time we release "Ryu ga Gotoku/Yakuza" series at our studio “Ryu Ga Gotoku Studio”. But on the one hand, it contributes to speed up the game development and improve the game quality, on the other hand, maintaining the automated environment increased the burden on developers. Therefore, I stepped forward and started to take in charge of the QA engineering since the development of "Ryu ga gotoku Ishin!" which was released in 2014 in Japan.
It was a wise decision to assign an exclusive QA engineer in a developing team that it carried out a great evolution to QA engineering technology at unbelievable speed on a daily basis. We reorganized the original developing environment and started integrating the systems in order to manage and maintain them easier. We also expanded the functions of “AutoTest”, and moreover we implemented the new functions for log analysis which I will explain in the next paragraph. QA engineering technology of “Ryu Ga Gotoku Studio” is still continuing evolving and it accelerates its speed even today.
(2015-) Analysis of Testing Result
At our studio, we assigned an exclusive engineer and we can now handle and manage the automated testing much easier. But there, we faced another new problem. Other than "AutoTest", we use Jenkins for automated testing. (Of course we use it as a continuous integration tool.) But it can only tell us either the result of the test is a success or a failure. We wanted the results in much more detail. so we started to analyze the logs of testing since "Ryu ga Gotoku KIWAMI" (Released in 2016 in Japan/The English version is "Yakuza Kiwami") and "Ryu ga Gotoku 6: Inochi no Uta" (Released in 2016 in Japan/The English version is "Yakuza 6: The Song of Life").
The following graphs (Figure 3 and 4) show the visualization of heap maps which are created by collecting usage of VRAM (Video RAM/Graphics Memory) (Figure 3) and collision errors in "AutoTest" (Figure 4).
Figure 3: Heat Map for Usage of VRAM (Fist of the North Star: Lost Paradise)
The heat map below visualizes the usage of VRAM in the city of "Eden".
Red is high usage, and X is NG (over 100% usage).
Figure 4: Collision Error Map in Very Huge "Wasteland" (Fist of the North Star: Lost Paradise)
As a result of making the “AutoTest” play the games, defects other than easily-found-bugs, such as warnings, were found, just like when we play the games by ourselves. And also it can check the game performance for us. By collecting the logs and visualizing them, we can now save a lot of time and fix bugs much faster with high accuracy.
Moreover, we analyzed logs of manual playtests and could get feedback of game level design. It enabled us to create more fun games. The following graphs (Figure 5 and 6) are examples of feedback of game level design to analyze logs of manual tests (Figure 5) and "AutoTest" (Figure 6).
Figure 5: The Game Over Points of Manual Playtesting (Yakuza 6: The Song of Life)
We visualized points where player’s HP is zero in manual tests. We could make use of it for game level design.
Figure 6: Item Drop Odds of Automated Playtesting (Fist of the North Star: Lost Paradise)
We measured odds of items that the enemies dropped in automated playtests. We confirmed the item odds are correct and the enemies can drop the rare items for game level design.
If you want to know about our automated playtesting and playlog analysis in more detail, please take a look at the following link of Japan Symposium on Software Testing in Tokyo 2018 (JaSST'18 Tokyo)
(2018-) Thought of Test Pyramid and QA Engineering in the Future
No matter how efficiently we became to analyze the result of automated testing, we can never stop improving the QA environment. Nowadays, AI technology is a big trend everywhere around, and the efficiency of the automation technology and the log analysis in the game development which have been based on our experience are expected to make much more progress with an effect of AI technology.
In Computer Entertainment Developers Conference 2018 (CEDEC 2018) in Japan, I joined a session "On the Future of QA and AI: How to Implement AI into Game Development Correctly" as a panelist. In the article below, I will follow up on the test pyramid which I had explained in CEDEC 2018.
The test pyramid is often used to show the execution of automation efficiency. In system software or web development, the test pyramid is desired to be like the figure below (Figure 7).
Figure 7: Test Pyramid of System and Web Development
| UI Testing | The product testing. It is high cost and difficult to automate. |
| Integration Testing | Testing if the combination of individual functions and modules work correctly. |
| Unit Testing | Testing of individual functions and modules. It is the lowest-cost testing in this test pyramid. |
By increasing the ratio of lower-cost unit testing and automating it, it is possible to cut cost and improve efficiency. Then what about the test pyramid in the game development? In most cases, it is likely to be a pyramid below (Figure 8).
Figure 8: Test Pyramid of Game Development (Real)
| Manual Playtesting | Testing to be played by hand. |
| Smoke Testing |
Simple testing to check if the games launch and the individual modules work correctly. The shorter the time, the better testing. (about a few minutes) |
|
Engine and Library Testing |
Testing if samples of the engine and the library work correctly when they are developed in the company. |
| Assets Testing |
Individual testing if the game assets (the models, the textures and the motions etc.) work correctly. |
As you can see from the figure above (Figure 8), the pyramid is upside down and very unbalanced. The ratio of the manual testing at the last stage of the game development is very high, which means it is inefficient and high costs. In the game development, these testing relies on human resources that it is difficult to increase personnels in a short period of time, especially under such a current circumstance of labor shortage.
And because of its data-driven architecture in the game development, it always seems to be very buggy by data(assets). Sometimes the number of these bugs is even greater than that of programming bugs.
So, we incorporated unit testing before final testing. You may think “Why do we need unit testing in the game development?”, but once you know that it can detect errors and check consistency of individual assets before the final tests, you may easily imagine how extra works and retake-costs we can eliminate by having unit testing.
We have been trying to make further efforts to fix the inverted triangle to a well-balanced triangle. For example, except for “AutoTest” and log analysis which I have already explained, we strengthen detecting errors in assets-converting and increasing smoke testing.
Figure 9: Test Pyramid of Game Development (Ideal)
With the ideal pyramid above (Figure 9), you may wonder why we had started to automate playtesting which is highest-cost testing first.
There are two reasons for it. Because the results of unit testing in the games can be observed only in the graphics output, it is difficult to determine if the result is a success or a failure. And the other reason is that the members of the development teams accept errors of automated testing as errors we must fix.
The second reason is especially important. In a testing process, once an error is found, the error must be fixed right away. But if the errors which automated testing found are not trusted by the developers, they would have a question such as “Do we really encounter the same errors in the product?”, and that these bugs would be possibly ignored and thus left over in the games. In this case, we will end up in just wasting time and labor. We should build the test environment exactly like the product which surely ensures the reliability of automated testing. It is important to raise the organization culture that every time the automated testing finds errors, we must fix them immediately.
At “Ryu Ga Gotoku Studio”, every team member recognizes the need of the “Automated Playtesting” and trusts the system, thus we are able to keep increasing the ratio of low-cost automated testing. If you are willing to try the automated testing, my advice for you is to build automated playtests which enable a player in the game move randomly by mashing buttons, and then gradually share them with your teammates.
Conclusion
As you can see from this article, QA engineering is very essential technology in the game development now, and it is desired and expected to evolve greater in the future.
Talking about my career, I first started working as a game programmer, and then experienced game development and QA as a developer. That’s why I can easily tell what is exactly desired for QA and what should be automated in the game development. In addition, I can suggest the concrete solutions for automation of the game development and seek for better productivity. In this sense, I believe that I’m making the most of my career. If you are interested in QA, a QA engineer would be a good choice for whom already have experienced game creating as a programmer.
We have been trying to make a comfortable environment also for developers. If you are interested in our jobs, please knock the door and contact us any time. We are always waiting for you here at SEGA.
Reference
- How Do We Create the High-End Video Games within 10 Months?: The Foundation of Technology for "Ryu ga Gotoku/Yakuza" Series (CEDEC 2010) (Japanese)
- What Do We Do to Debug Faster for "Ryu ga Gotoku/Yakuza" Series?: High-Speed Workflow to Fix Enormous Bugs (JaSST’16 Tokyo) (Japanese)
- Automated Playtesting and Playlog Analysis to Create More Fun Games for "Ryu ga Gotoku/Yakuza" Series (JaSST’18 Tokyo) (Japanese)
- On the Future of QA and AI: How to Implement AI into Game Development Correctly (CEDEC 2018) (Japanese)
* The game screenshots in this article were taken under development. It is different from the products.
©SEGA
