Performance Testing for Game Development: Process Automation

Mobile games require performance testing before they hit the markets, 否则,针对高端设备制作的游戏可能会遭到成千上万用户的抱怨,并导致财务和声誉损失. 幸运的是,大多数主要的市场参与者都意识到了这一点并选择了逃跑 性能和负载利记手机app on a regular basis, during development, before every next release, etc.,因此,利记手机app过程成为必要的例行程序.

We are happy to share our expertise in performance testing, 因为利记手机app的性能利记手机app利记手机在许多不同规模的移动开发者中很受欢迎.

Table of Contents

寻找性能和负载利记手机app提供商?

Drop us a line. 很可能,在之前的300个项目中,利记手机app已经处理过类似的问题.

游戏公司的性能利记手机app面临的挑战

性能利记手机app程序通常包括一个标准的7步程序:系统分析, 利记手机app方法开发, scripts development, 利记手机app场景的执行, results analysis, 在出现负面结果时进行调优, and a release. 这些阶段确实需要人类工程,而且最有趣的是脚本开发和调优. 他们值得利记手机app投入时间. However, the testing process itself and result analysis take hours and manual labour, and they are boring, too. In this article we suggest one of the ways to automate them, 这样,您的利记手机app工程师的注意力就会转移到真正需要的地方:调整系统.

另一个问题是,许多游戏开发者必须在一个共享的利记手机app环境中同时利记手机app数款不同的游戏, and then compare the obtained results to the results of previous tests. Using Apache Jmeter to test game performance is quite common, often accompanied by Jenkins 用于持续集成(CI).  但是当每个项目中的利记手机app数量增加时, 要分析的文件有千兆字节, 涉众可能对无需外部QA帮助就可以使用的利记手机app自动化机会更加感兴趣.

Performance testing in a gaming company- process automation

解决方案:Jmeter负载利记手机app中心

Jmeter负载利记手机app中心 is an online web app/dashboard for CI of load testing with JMeter, built with Django. LTC (Load Testing Center)使用PostgreSQL存储数据. It also uses pandas 模块分析文件与数据. Pandas is an open source data analysis and manipulation tool, 能够从Jmeter分析csv文件, 即使它们的重量是gb.

LTC’s main modules are:

1. Dashboard – front page with general information about the last tests;
2. Analyzer – to build reports, analyze results and compare results with another;
3. 在线——用于运行利记手机app的在线监视;
4. 控制器——配置和运行利记手机app;
5. 管理员-配置不同的参数.

To run a test, just open Jenkins, choose a project and click Start.

Performance testing in a gaming company- process automation

After that, Jenkins launches the main instance of Jmeter on the main server, 詹金斯和LTC所在的地方, 以及一个或多个所需数量的远程虚拟机上的Jmeter利记手机器. 利记手机app过程开始了.

During the test, 一个包含JMeter结果的CSV文件被创建,并使用项目$WORKSPACE文件夹中的数据进行更新, as well as another CSV file with remote host monitoring data.

Then you can open LTC and watch your test running online. At this time, 应用程序将解析上述CSV文件,并将它们放在数据库中的临时表中, based on which, Online draws graphs:

Performance testing in a gaming company- process automation

Otherwise, 你可以等到利记手机app结束, 当一个特殊的脚本将在Analyzer中收集一个表中的所有数据并与其他利记手机app的结果进行比较时. Voila!

Performance testing in a gaming company- process automation

同时利记手机app多个游戏项目

There is only one last problem: if you are testing several projects at a time, and they require different capacities to carry out the load, i.e., 来模拟不同数量的虚拟用户, 如何分配可用的资源, say, 10个虚拟机生成器? 您可以将每个项目分配给特定的生成器, make a schedule, 或使用一个阻塞插件的Jenkins. 但也有另一种选择. 

As we mentioned above, the backend is written on the Django framework. 在前端的开发中,使用了所有的标准库:jquery和bootstrap. For graphs, c3.js  is a solution that easily draws the data received in JSON format. 

数据库中的表通常有一对键和一个数据类型为JSONField()的字段. 使用JSONField是因为稍后您可以轻松地向该表添加新指标,而无需更改其结构.

Thus, 在响应时间上存储数据的典型模型, the number of errors and other things during one test looks very simple:

class TestData(models.Model):

   test = models.ForeignKey(Test)

   data = JSONField()

   class Meta:

      db_table = ‘test_data’

In the data field, there are JSONs that store data aggregated in 1 minute.

Performance testing in a gaming company- process automation

To extract data from the table, there is one endpoint in urls.py,它调用一个函数来处理这些数据,并返回一个方便阅读的JSON:

Endpoint:

url(r’^test/(?P\d+)/rtot/$, views.test_rtot),

Function:

def test_rtot(request, test_id):

获取利记手机app开始时间戳的代码:

   min_timestamp = TestData.objects. \

       filter(test_id=test_id). \

       values(“test_id”).\

       总(min_timestamp = Min (

           RawSQL(“((data->>%s)::timestamp)”, (‘timestamp’,))))[‘min_timestamp’]  

   min_timestamp = TestData.objects. \

       filter(test_id=test_id). \

       values(“test_id”).\

       总(min_timestamp = Min (

           RawSQL(“((data->>%s)::timestamp)”, (‘timestamp’,))))[‘min_timestamp’]  

从数据库中提取数据, subtract the min_timestamp obtained above to get the absolute test time, sort by timestamp, 将其转换为JSON并返回:

 d = TestData.objects. \

       filter(test_id=test_id). \

       注释(时间戳= (RawSQL (“((data->>%s)::timestamp)”, (‘timestamp’,)) – min_timestamp)). \

       注释(平均= RawSQL (“((data->>%s)::numeric)”, (‘avg’,))). \

       annotate(median=RawSQL(“((data->>%s)::numeric)”, (‘median’,))). \

       annotate(rps=(RawSQL(“((data->>%s)::numeric)”, (‘count’,))) / 60). \

       values(‘timestamp’, “average”, “median”, “rps”). \

       order_by(‘timestamp’)

   data = json.loads(

       json.dumps(list(d), indent=4, sort_keys=True, default=str))

   return JsonResponse(数据、安全=False)

 d = TestData.objects. \

       filter(test_id=test_id). \

       注释(时间戳= (RawSQL (“((data->>%s)::timestamp)”, (‘timestamp’,)) – min_timestamp)). \

       注释(平均= RawSQL (“((data->>%s)::numeric)”, (‘avg’,))). \

       注释(值= RawSQL (“((data->>%s)::numeric)”, (‘median’,))). \

       annotate(rps=(RawSQL(“((data->>%s)::numeric)”, (‘count’,))) / 60). \

       values(‘timestamp’, “average”, “median”, “rps”). \

       order_by(‘timestamp’)

   data = json.loads(

       json.dumps(list(d), indent=4, sort_keys=True, default=str))

   return JsonResponse(数据、安全=False)

在前面,得到c3.引用这个端点的Js图形:

var test_rtot_graph = c3.generate({

       data: {

           url: ‘/analyzer/test/’ + test_id_1 + ‘/rtot/’,

           mimeType: ‘json’,

           type: ‘line’,

           keys: {

               x: ‘timestamp’,

               value: [‘average’, ‘median’, ‘rps’],

           },

           xFormat: ‘%H:%M:%S’,

           axes: {

               rps: ‘y2’

           },

       },

       zoom: {

           enabled: true

       },

       axis: {

           x: {

               type: ‘timeseries’,

               tick: {

                   format: ‘%H:%M:%S’

               }

           },

           y: {

               padding: {

                   top: 0,

                   bottom: 0

               },

               label: ‘response times (ms)’,

           },

           y2: {

               min: 0,

               show: true,

               padding: {

                   top: 0,

                   bottom: 0

               },

               label: ‘Requests/s’,

           }

 

       },

       bindto: ‘#test_rtot_graph’

   });

 

And this is the result:

Performance testing in a gaming company- process automation

Actually, 整个应用程序由如下图所示的图形组成,这些图形从后端相应的端点绘制数据.

You can see how the whole system for analyzing tests works from the source codes, 所以接下来利记手机app想分享游戏负载利记手机app的案例模板,因为这可能是由你们自己的QA团队完成的.

我可不想自己做利记手机app?

There is no better place for a QA solution than Performance Lab.
给利记手机app写信,看看利记手机app团队能为你做什么.

加载利记手机app游戏:案例模板

负载利记手机app环境

The entire load testing environment should consist of one main server admin.loadtest, and several generatorN.loadtest servers. admin.loadtest 对于这种情况可以是一个Debian Linux 9虚拟机, with 16 cores/16 gigs, running Jenkins, LTC和其他可忽略的软件. generatorN.loadtest servers are bare virtual machines Debian Linux 8, with Java 8 installed. 他们的实际力量可能会有所不同. On admin.loadtest, Jmeter(包含最基本插件的最新版本)作为预组装的deb发行版安装在/var/lib/apache-jmeter文件夹中.

GIT

每个项目的利记手机app计划应该位于GitLab的一个单独的项目中, and developers or QA from each team can suggest their own corrections. 每个项目都应该配置为使用Git. 每个项目包括 ./jmeter_ext_libs/src/, ./test-plan.jmx, and ./prepareAccouts.sh.

    • jmeter_ext_libs是一个文件夹,里面有使用Gradle收集的额外插件的源代码,在每次利记手机app前放在/var/lib/apache-jmeter/lib/ext中;
    • test-plan.jmx is a test plan;
    • *.sh stores additional scripts for preparing user accounts, etc.

Test plan

Each test plan uses a Stepping Thread Group with three variables: thread_count, ramp_up, and duration.

The values for these variables come from Jenkins when the test is run, 但首先,它们必须放在利记手机app计划的主元素用户定义变量中, 与所有其他参数化变量相似. 利记手机app称它为最重要的一个 pool, is the one where a serial number for each running Jmeter server is sent to, in order to subsequently differentiate the data pools used (for example, user logins). In ${__P(THREAD_COUNT,1)}, THREAD_COUNT is the name of the variable that will come from Jenkins, 1是默认值, if it does not come.

Performance testing in a gaming company- process automation

Also, 在每个利记手机app计划中,都有一个SimpleDataWriter,它将抽样的结果存储在CSV文件中. 以下选项被激活:

 <time>truetime>

            <latency>truelatency>

            <timestamp>truetimestamp>

            <success>truesuccess>

            <label>truelabel>

            <code>truecode>

            <fieldNames>truefieldNames>

            <bytes>truebytes>

            <threadCounts> truethreadCounts>

<time>truetime>

<latency>truelatency>

<timestamp>truetimestamp>

<success>truesuccess>

<label>truelabel>

<code>truecode>

<fieldNames>truefieldNames>

<bytes>truebytes>

<threadCounts> truethreadCounts>

Jenkins

Before running the test, 每个用户可以设置传递给上述Jmeter利记手机app计划变量的一些参数:利记手机器名, thread count, duration and ramp up.

Running the test

现在让利记手机app进入脚本. To begin with, prepare the Jmeter distribution in the pre-build script:

    • 创建临时目录/tmp/jmeter-xvgenq/;
    • copy into it the main distribution package from /var/lib/apache-jmeter/;
    • collect additional plugins from the jmeter_ext_libs folder (if any);
    • copy the collected *.在/ tmp / jmeter-xvgenq /瓶;
    • extend the ready-made temporary distribution package of Jmeter to load generators.

#!/bin/bash

export PATH=$PATH:/opt/gradle/gradle-4.2.1/bin

 

echo “JMeter home: $JMETER_HOME

JMETER_INDEX=$(cat /dev/urandom | tr -dc ‘a-zA-Z0-9’ |折叠-w 8 |头-n 1)

Generate a random name:

JMETER_DIR=“/tmp/jmeter$JMETER_INDEX

 

echo “JMeter directory: $JMETER_DIR

echo $JMETER_DIR > “/tmp/jmeter_dir$JOB_NAME

 

mkdir $JMETER_DIR

cp -rp $JMETER_HOME* $JMETER_DIR

 

if [ -d $WORKSPACE/jmeter_ext_libs” ]; then  

    echo “建立额外的JMeter库”

    cd $WORKSPACE/jmeter_ext_libs”

    gradle jar

    cp ./build/libs/* $JMETER_DIR/lib/ext/

    ls $JMETER_DIR/lib/ext/

fi

cd $WORKSPACE

回到上面讨论的挑战, 如果您有多个设计用于生成负载的虚拟机(并且您不知道当前是否正在运行来自其他项目的利记手机app), based on the required total number of emulated THREAD_COUNT threads, 假设您需要运行一定数量的Jmeter利记手机器,以在这些虚拟机上模拟所需的负载.

In our script, this is done using three lines in the Jenkins bash script, which naturally leads to more serious operations in the LTC scripts:

python /var/lib/jltc/manage REMOTE_HOSTS_DATA = '.py shell -c “import controller.views as views; print(views.prepare_load_generators(’”$JOB_NAME“‘,'”$WORKSPACE“‘,'”$JMETER_DIR“‘, ‘$THREAD_COUNT‘, ‘$duration));”`

THREADS_PER_HOST = ' python - c ' json,导入系统;数据= dict(’”$REMOTE_HOSTS_DATA”);打印数据(“threads_per_host”)`

REMOTE_HOSTS_STRING = ' python - c ' json,导入系统;数据= dict(’”$REMOTE_HOSTS_DATA”);打印数据(“remote_hosts_string”)`

python /var/lib/jltc/manage REMOTE_HOSTS_DATA = '.py shell -c “import controller.views as views; print (views. prepare_load_generators(’”$JOB_NAME“‘, ‘”$WORKSPACE“‘, ‘”$JMETER_DIR“‘, ‘$THREAD_COUNT‘, ‘$duration));”`

THREADS_PER_HOST = ' python - c ' json,导入系统;数据= dict(’”$REMOTE_HOSTS_DATA“‘); print data[“threads_per_host”]’`

REMOTE_HOSTS_STRING = ' python - c ' json,导入系统;数据= dict(’”$REMOTE_HOSTS_DATA“‘); print data[“remote_hosts_string”]’`

Thus, 在第一行中,调用prepare_load_generators函数并将各种数据放入其中, namely, the project name, 项目工作区目录的路径, the path to the temporary Jmeter distribution (/tmp/jmeter-xvgenq/) created above, 利记手机app持续时间, and, most importantly, the desired total number of emulated threads $THREAD_COUNT. 接下来发生的事情在库中是可见的. To summarize it:

    • 所需Jmeter利记手机器数量的计算基于给定的THREAD_COUNT. Let it be X.
    • Then, based on the current load and available memory on the generatorN.loadtest machines, 在每台机器上启动Jmeter利记手机器的数量,直到它们的总数达到X.
    • 接下来,通过rsync将相同的临时Jmeter分发加载到每个选定的generatorN中.loadtest machines.
    • On each generator, a number of Jmeter servers is started (obtained in the previous step), and you should pass a sequential pool number to each running process, 用于数据池的分布.
    • 利记手机app之后,所有关于运行Jmeter实例的数据都存储在数据库中, 根据这些数据,他们都会被销毁.
    • 最后,函数返回这样的JSON:

 {

       “remote_hosts_string”: “generator1.generator2 loadtest: 10000.loadtest:10000, generator2.loadtest:10001”,

       “threads_per_host”: 100

   }

 {

       “remote_hosts_string”: “generator1.generator2 loadtest: 10000.loadtest:10000, generator2.loadtest:10001”,

       “threads_per_host”: 100

   }

As you can see, at that moment all Jmeter servers are running and waiting to be connected; then, 利记手机app本身就会开始. Thus, 在第二行和第三行中,利记手机app从这个JSON中获取值,并将它们传递给Jmeter的主实例. THREAD_COUNT的初始值被分配给远程Jmeter利记手机器和threads_per_host的每个帐户(注意,对于THREAD_COUNT,利记手机app传递threads_per_host的值,如上所述):

java -jar -server $JAVA_ARGS $JMETER_DIR/bin/ApacheJmeter.jar -n -t $WORKSPACE/test-plan.jmx -R $REMOTE_HOSTS_STRING -GTHREAD_COUNT=$threads_per_host -GDURATION=$DURATION -GRAMPUP=$RAMPUP

java -jar -server $JAVA_ARGS $JMETER_DIR/bin/ ApacheJmeter.jar -n -t $WORKSPACE/test-plan.jmx -R $REMOTE_HOSTS_STRING -GTHREAD_COUNT=  $threads_per_host -GDURATION=$DURATION -GRAMPUP=$RAMPUP

There’s a different model for running remote Jmeter servers. Save data about tests running, the virtual machines, ports, ids of this process, etc. 当需要销毁相同的jmeter利记手机器时,有必要进一步停止利记手机app.

class JmeterInstance(models.Model):

    test_running = models.ForeignKey (TestRunning on_delete =模型.CASCADE)

    load_generator = models.ForeignKey (LoadGenerator)

    pid = models.IntegerField(default=0)

    port = models.IntegerField(default=0)

    jmeter_dir = models.CharField(max_length=300, default=“”)

    project = models.ForeignKey(项目,on_delete =模型.CASCADE)

    threads_number = models.IntegerField(default=0)

    class Meta:

        db_table = ‘jmeter_instance’

class JmeterInstance(models.Model):

    test_running = models.ForeignKey (TestRunning on_delete =模型.CASCADE)

    load_generator = models.ForeignKey (LoadGenerator)

    pid = models.IntegerField(default=0)

    port = models.IntegerField(default=0)

    jmeter_dir = models.CharField (max_length=300, default=“”)

    project = models.ForeignKey(项目,on_delete =模型.CASCADE)

    threads_number = models.IntegerField(default=0)

    class Meta:

        db_table = ‘jmeter_instance’

Meanwhile, 首页显示了很好的视觉效果,总结了一些运行利记手机app的数据和负载发电机的状态.

Performance testing in a gaming company- process automation

Stopping the test

After the test, 有必要销毁所有正在运行的Jmeter利记手机器, delete temporary Jmeter distributions and collect the results. 构建后脚本应该是这样的:

JMETER_DIR = $ (cat / tmp / JMETER_DIR$JOB_NAME)

echo "从admin中移除Jmeter目录: $JMETER_DIR

rm -rf $JMETER_DIR


python /var/lib/jltc/manage.py shell -c “import controller.views as views; print(views.stop_test_for_project(’”$JOB_NAME“‘))”

JMETER_DIR = $ (cat / tmp / JMETER_DIR$JOB_NAME)

echo "从admin中移除Jmeter目录: $JMETER_DIR

rm -rf $JMETER_DIR

python /var/lib/jltc/manage.py shell -c “import controller.views as views; print (views.stop_test_for_project(’”$JOB_NAME“‘))”

Start with deleting the temporary distributive from the main server, 然后调用stop_test_for_project函数, 将项目名称传递给它. The function will go through a special table in the database, 哪些存储关于运行Jmeter实例的信息, and stop them. 

For the final step, 收集您的利记手机app结果, 你将不得不使用两种可能的方法之一. 第一个是运行这个脚本:

python /var/lib/jltc/datagenerator_linux.py

python /var/lib/jltc/ datagenerator_linux.py

否则,你可以在本地调用web利记手机:

curl –data “results_dir=$JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_NUMBER/”http://localhost:8888/controller/parse_results

curl –data “results_dir=$JENKINS_HOME/jobs/$JOB_NAME/ builds/$BUILD_NUMBER/” http://localhost:8888/controller/parse_results

curl –data “results_dir=$JENKINS_HOME /jobs/$JOB_NAME/builds/ $BUILD_NUMBER/” http://localhost: 8888 /控制器/ parse_results

That’s it!

Performance testing in a gaming company- process automation

The results

Of course, 使用这个游戏性能利记手机app用例模板所得到的结果并不是通用的. You can also use InfluxDB to store metrics and Grafana dashboard to visualize them. 但要分析不同利记手机app的结果或比较不同项目的利记手机app结果, you will either have to automate your reports with InfluxDB and Grafana, (你可以读到它 here), or use this solution. 如果您的团队在这些情况下需要帮助, please, 让利记手机app知道:利记手机app有400多名经过认证的利记手机app人员,他们在世界各地处理了300多个性能或负载利记手机app项目. To get a skilled team of testers to work on your project, contact us!

Have a project in mind?

There is no better place for a QA solution than Performance Lab.
给利记手机app写信,看看利记手机app团队能为你做什么.

Share this post!

Latest posts from us

API微焦点ALM和性能中心
API微焦点ALM和性能中心: Basics for Performance Testing Engineers
How to avoid your financial application crash under load
How to Avoid Your Financial Application Crash Under Load
在线游戏和游戏利记手机器的性能利记手机app
在线游戏和游戏利记手机器的性能利记手机app
Performance Lab以独家移动应用开发利记手机赢得GoodFirms的认可
Performance Lab以独家移动应用开发利记手机赢得GoodFirms的认可
User manual for k6
Performance and Load Testing with K6 - open-source tool and cloud service