You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 11:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Fri Aug 28, 2026 00:44:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
373994722550,9sleep3411rcuc/319:05:113
139099920,53rtkit-daemon0-21swapper/119:08:221
374125726444,14sleep20-21swapper/219:08:162
374124326344,13sleep00-21swapper/019:08:060
38302602470,3sleep13056-21PK-Backend23:20:011
37540712460,3sleep03754062-21switchtime19:40:220
3741535993412,11cyclictest34-21ksoftirqd/220:35:012
3741535993012,9cyclictest34-21ksoftirqd/220:40:012
3741535992817,9cyclictest883-21dbus-daemon22:04:522
3741535992711,4cyclictest34-21ksoftirqd/220:45:022
374153599250,23cyclictest978-21snmpd20:50:162
38348772240,4chrt3056-21PK-Backend23:30:230
374154199240,22cyclictest3795151-21taskset21:38:453
3741535992420,3cyclictest331rcuc/222:30:572
3741535992415,6cyclictest3831915-21cron23:25:002
3741529992420,3cyclictest3810583-21taskset22:21:300
3741529992413,9cyclictest3844367-40updatedb.mlocat00:00:010
139099240,17rtkit-daemon3056-21PK-Backend22:35:331
139099240,16rtkit-daemon3056-21PK-Backend22:55:571
374153599232,16cyclictest3799003-21runrttasks21:50:112
3741535992321,1cyclictest0-21swapper/219:52:432
3741535992320,2cyclictest331rcuc/222:45:562
3741535992320,2cyclictest331rcuc/222:45:562
3741535992320,2cyclictest331rcuc/222:39:352
3741535992320,2cyclictest331rcuc/220:49:112
3741535992320,2cyclictest331rcuc/219:17:452
3741535992320,2cyclictest331rcuc/200:13:342
3741535992319,3cyclictest331rcuc/223:43:112
3741535992319,3cyclictest331rcuc/221:59:012
3741535992319,3cyclictest331rcuc/220:27:252
3741535992318,3cyclictest3843261-21hddtemp_smartct23:55:172
374153399230,21cyclictest37655172chrt20:15:071
139099230,17rtkit-daemon3056-21PK-Backend00:03:471
139099230,16rtkit-daemon3056-21PK-Backend21:36:091
139099230,15rtkit-daemon3056-21PK-Backend23:51:291
38178372220,3chrt3056-21PK-Backend22:45:001
3741541992220,1cyclictest0-21swapper/321:58:063
3741541992220,1cyclictest0-21swapper/321:18:383
3741541992219,2cyclictest0-21swapper/300:15:343
3741541992218,3cyclictest0-21swapper/320:30:123
3741541992210,11cyclictest42-21ksoftirqd/300:35:013
374154199220,20cyclictest3855062-21cat00:30:013
3741535992220,1cyclictest0-21swapper/221:03:072
3741535992219,2cyclictest331rcuc/223:54:212
3741535992219,2cyclictest331rcuc/222:44:122
3741535992219,2cyclictest331rcuc/222:28:482
3741535992219,2cyclictest331rcuc/222:13:412
3741535992219,2cyclictest331rcuc/200:33:232
3741535992217,4cyclictest34-21ksoftirqd/223:34:562
374153599221,5cyclictest3820805-21ntp_states22:50:192
3741535992214,5cyclictest3749145-21cron19:29:592
374153599220,7cyclictest3827735-21ntp_states23:10:182
3741533992220,1cyclictest3056-21PK-Backend23:01:021
3741533992219,2cyclictest3056-21PK-Backend23:21:301
3741533992219,2cyclictest3056-21PK-Backend19:53:581
3741533992219,2cyclictest3056-21PK-Backend19:35:321
374153399220,20cyclictest3056-21PK-Backend21:59:371
374153399220,20cyclictest3056-21PK-Backend21:47:341
374153399220,20cyclictest3056-21PK-Backend00:29:091
139099220,16rtkit-daemon3056-21PK-Backend19:59:101
139099220,16rtkit-daemon3056-21PK-Backend19:59:101
139099220,15rtkit-daemon3056-21PK-Backend19:30:201
374154199212,16cyclictest3840002-21python323:45:203
3741541992119,1cyclictest0-21swapper/323:22:493
3741541992119,1cyclictest0-21swapper/323:17:353
3741541992119,1cyclictest0-21swapper/322:57:273
3741541992119,1cyclictest0-21swapper/322:11:443
3741541992119,1cyclictest0-21swapper/321:13:173
3741541992119,1cyclictest0-21swapper/320:21:203
3741541992119,1cyclictest0-21swapper/319:53:473
3741541992119,1cyclictest0-21swapper/319:41:383
374154199211,7cyclictest3816343-21apt22:40:103
374154199211,2cyclictest141rcu_preempt20:45:003
3741541992112,7cyclictest3778941-21ntp_states20:50:183
374154199211,18cyclictest3795532-21apt21:40:103
374154199210,20cyclictest0-21swapper/320:10:253
374153599212,5cyclictest34-21ksoftirqd/221:45:172
3741535992119,1cyclictest0-21swapper/223:45:562
3741535992119,1cyclictest0-21swapper/223:36:382
3741535992119,1cyclictest0-21swapper/223:29:142
3741535992119,1cyclictest0-21swapper/223:16:022
3741535992119,1cyclictest0-21swapper/223:05:352
3741535992119,1cyclictest0-21swapper/222:58:252
3741535992119,1cyclictest0-21swapper/221:41:472
3741535992119,1cyclictest0-21swapper/221:32:002
3741535992119,1cyclictest0-21swapper/220:22:442
3741535992119,1cyclictest0-21swapper/219:49:122
3741535992119,1cyclictest0-21swapper/219:43:512
3741535992119,1cyclictest0-21swapper/219:30:062
3741535992119,1cyclictest0-21swapper/219:11:132
3741535992119,1cyclictest0-21swapper/200:39:512
3741535992119,1cyclictest0-21swapper/200:17:332
3741535992119,1cyclictest0-21swapper/200:06:252
3741535992118,2cyclictest331rcuc/222:07:272
3741535992118,2cyclictest331rcuc/220:56:292
3741535992118,2cyclictest331rcuc/220:16:282
3741535992118,2cyclictest331rcuc/220:09:142
3741535992118,2cyclictest331rcuc/220:01:322
3741535992118,2cyclictest331rcuc/200:27:502
3741535992117,3cyclictest331rcuc/221:23:202
3741535992113,6cyclictest3790329-21apt21:25:112
374153599210,20cyclictest0-21swapper/220:14:402
3741533992119,1cyclictest3056-21PK-Backend23:07:241
3741533992119,1cyclictest3056-21PK-Backend22:02:081
3741533992119,1cyclictest3056-21PK-Backend21:33:151
3741533992119,1cyclictest3056-21PK-Backend00:36:471
3741533992119,1cyclictest3056-21PK-Backend00:16:191
3741533992119,1cyclictest0-21swapper/121:51:131
3741533992119,1cyclictest0-21swapper/120:42:181
3741533992118,2cyclictest3056-21PK-Backend21:09:221
3741533992118,2cyclictest3056-21PK-Backend20:31:491
3741533992113,6cyclictest3777332-21python320:45:211
374153399210,20cyclictest3056-21PK-Backend23:48:271
374153399210,19cyclictest3056-21PK-Backend23:59:421
374153399210,19cyclictest3056-21PK-Backend22:45:481
374153399210,19cyclictest3056-21PK-Backend22:45:471
374153399210,19cyclictest3056-21PK-Backend21:25:571
374153399210,19cyclictest3056-21PK-Backend21:16:591
374153399210,19cyclictest3056-21PK-Backend19:29:521
374153399210,19cyclictest3056-21PK-Backend00:33:141
374152999210,8cyclictest38114002cat22:25:170
139099210,4rtkit-daemon0-21swapper/023:04:160
139099210,15rtkit-daemon3056-21PK-Backend19:11:511
139099210,14rtkit-daemon3056-21PK-Backend22:06:031
139099210,14rtkit-daemon3056-21PK-Backend00:10:041
139099210,14rtkit-daemon0-21swapper/122:20:061
139099210,14rtkit-daemon0-21swapper/000:18:250
139099210,13rtkit-daemon0-21swapper/020:11:250
3741541992018,1cyclictest3056-21PK-Backend20:36:343
3741541992018,1cyclictest0-21swapper/323:55:273
3741541992018,1cyclictest0-21swapper/322:06:093
3741541992018,1cyclictest0-21swapper/322:03:403
3741541992018,1cyclictest0-21swapper/321:47:113
3741541992018,1cyclictest0-21swapper/321:25:213
3741541992018,1cyclictest0-21swapper/320:48:173
3741541992018,1cyclictest0-21swapper/320:02:003
3741541992018,1cyclictest0-21swapper/319:16:023
3741541992018,1cyclictest0-21swapper/319:10:363
3741541992012,6cyclictest3855460-21cut00:30:123
3741541992012,6cyclictest3847001-21awk00:05:143
3741541992012,6cyclictest3750181-21ntp_states19:30:263
374154199200,19cyclictest0-21swapper/323:10:243
374154199200,19cyclictest0-21swapper/322:51:403
374154199200,19cyclictest0-21swapper/322:47:483
374154199200,19cyclictest0-21swapper/322:47:483
374154199200,19cyclictest0-21swapper/322:17:193
374154199200,19cyclictest0-21swapper/320:26:203
374154199200,19cyclictest0-21swapper/320:18:483
374154199200,19cyclictest0-21swapper/320:06:463
374154199200,19cyclictest0-21swapper/319:45:223
374153599209,8cyclictest436-20systemd-journal19:24:502
374153599203,3cyclictest34-21ksoftirqd/200:21:082
3741535992018,1cyclictest0-21swapper/222:22:282
3741535992018,1cyclictest0-21swapper/200:01:322
3741535992012,6cyclictest3785117-21find21:10:112
374153599200,6cyclictest34-21ksoftirqd/221:05:122
374153599200,1cyclictest0-21swapper/223:02:172
374153599200,1cyclictest0-21swapper/222:15:322
374153599200,19cyclictest0-21swapper/221:37:342
3741533992018,1cyclictest3056-21PK-Backend23:41:171
3741533992018,1cyclictest3056-21PK-Backend21:21:451
3741533992018,1cyclictest3056-21PK-Backend20:58:331
3741533992018,1cyclictest3056-21PK-Backend20:27:241
3741533992018,1cyclictest3056-21PK-Backend20:21:501
3741533992018,1cyclictest3056-21PK-Backend20:09:471
3741533992018,1cyclictest3056-21PK-Backend19:21:541
3741533992018,1cyclictest3056-21PK-Backend00:20:241
3741533992017,2cyclictest3056-21PK-Backend23:36:031
3741533992017,2cyclictest3056-21PK-Backend21:40:311
374153399200,19cyclictest3056-21PK-Backend22:16:551
374153399200,19cyclictest3056-21PK-Backend22:11:201
3741529992018,1cyclictest0-21swapper/020:01:580
3741529992018,1cyclictest0-21swapper/000:01:410
3741529992013,6cyclictest3775121-21needreboot20:40:180
3741529992012,6cyclictest3768734-21cron20:25:000
374152999200,1cyclictest0-21swapper/023:53:290
374152999200,19cyclictest3056-21PK-Backend21:07:270
374152999200,19cyclictest0-21swapper/023:15:300
374152999200,19cyclictest0-21swapper/023:05:500
374152999200,19cyclictest0-21swapper/021:48:310
374152999200,19cyclictest0-21swapper/021:34:100
374152999200,19cyclictest0-21swapper/000:24:530
139099200,14rtkit-daemon3056-21PK-Backend21:10:511
139099200,14rtkit-daemon0-21swapper/020:18:250
3741541991917,1cyclictest0-21swapper/322:22:193
3741541991917,1cyclictest0-21swapper/321:32:033
3741541991917,1cyclictest0-21swapper/319:36:313
3741541991914,4cyclictest3852439-21cat00:20:203
3741541991914,3cyclictest3833057-21python323:25:203
3741541991914,3cyclictest3784266-21python321:05:213
374154199191,3cyclictest3781668-21apt21:00:113
3741541991912,5cyclictest436-20systemd-journal00:15:003
3741541991912,5cyclictest3811700-21sed22:25:213
374154199190,1cyclictest0-21swapper/323:09:023
374154199190,1cyclictest0-21swapper/320:55:263
374154199190,18cyclictest0-21swapper/323:50:473
3741535991917,1cyclictest0-21swapper/219:35:472
3741535991911,6cyclictest3787616-21ntpq21:15:202
374153599190,6cyclictest34-21ksoftirqd/219:59:552
374153599190,6cyclictest34-21ksoftirqd/219:59:542
374153399196,11cyclictest3773949-21sh20:40:011
3741533991917,1cyclictest3056-21PK-Backend23:34:411
3741533991917,1cyclictest3056-21PK-Backend23:28:051
3741533991917,1cyclictest3056-21PK-Backend22:34:241
374153399190,1cyclictest3056-21PK-Backend19:47:571
374153399190,1cyclictest3056-21PK-Backend00:05:581
3741529991917,1cyclictest0-21swapper/023:46:200
3741529991917,1cyclictest0-21swapper/023:39:430
3741529991917,1cyclictest0-21swapper/023:25:580
3741529991917,1cyclictest0-21swapper/022:55:020
3741529991917,1cyclictest0-21swapper/022:49:140
3741529991917,1cyclictest0-21swapper/022:49:140
3741529991917,1cyclictest0-21swapper/022:06:420
3741529991917,1cyclictest0-21swapper/021:56:560
3741529991917,1cyclictest0-21swapper/021:52:430
3741529991917,1cyclictest0-21swapper/021:43:100
3741529991917,1cyclictest0-21swapper/021:36:400
3741529991917,1cyclictest0-21swapper/021:25:460
3741529991917,1cyclictest0-21swapper/021:17:210
3741529991917,1cyclictest0-21swapper/020:47:270
3741529991917,1cyclictest0-21swapper/020:30:300
3741529991917,1cyclictest0-21swapper/019:27:440
3741529991917,1cyclictest0-21swapper/019:13:510
3741529991917,1cyclictest0-21swapper/000:37:560
3741529991917,1cyclictest0-21swapper/000:28:580
3741529991915,3cyclictest13-21ksoftirqd/019:20:000
3741529991912,5cyclictest3762798-21ntpq20:05:190
374152999190,1cyclictest3056-21PK-Backend00:07:070
374152999190,1cyclictest0-21swapper/023:21:180
374152999190,1cyclictest0-21swapper/022:15:210
374152999190,1cyclictest0-21swapper/022:12:510
374152999190,1cyclictest0-21swapper/022:00:330
374152999190,1cyclictest0-21swapper/021:22:420
374152999190,1cyclictest0-21swapper/020:36:510
374152999190,18cyclictest0-21swapper/022:42:520
374152999190,18cyclictest0-21swapper/022:31:290
374152999190,18cyclictest0-21swapper/020:57:140
374152999190,18cyclictest0-21swapper/019:30:280
3741541991813,4cyclictest2156-21gnome-shell23:34:563
3741541991813,3cyclictest3789339-21ntp_states21:20:203
374154199181,15cyclictest3815477-21python322:35:223
374154199181,15cyclictest3759355-21ntp_states19:55:183
374154199181,15cyclictest3759355-21ntp_states19:55:183
3741533991816,1cyclictest3056-21PK-Backend23:11:161
3741533991816,1cyclictest3056-21PK-Backend19:42:021
3741533991816,1cyclictest3056-21PK-Backend19:15:051
3741533991815,2cyclictest3056-21PK-Backend20:04:591
3741529991816,1cyclictest0-21swapper/020:51:460
3741529991812,5cyclictest3757484-21python319:50:200
374152999180,1cyclictest0-21swapper/022:38:250
374152999180,1cyclictest0-21swapper/019:47:440
374152999180,1cyclictest0-21swapper/019:35:210
139099180,3rtkit-daemon3056-21PK-Backend22:28:201
374154199179,6cyclictest3813615-21ntp_states22:30:203
3741541991715,1cyclictest0-21swapper/323:38:093
3741541991715,1cyclictest0-21swapper/321:50:153
3741541991715,1cyclictest0-21swapper/300:04:263
3741541991713,2cyclictest3746696-21ntpq19:20:183
374154199170,15cyclictest3838395-21cat23:40:193
374154199170,15cyclictest3823928-21hddtemp_smartct23:00:153
374152999179,6cyclictest436-20systemd-journal19:24:500
3741529991715,1cyclictest0-21swapper/000:13:570
374152999170,1cyclictest0-21swapper/000:34:250
374154199160,15cyclictest3747694-21apt19:25:113
3741533991614,1cyclictest3056-21PK-Backend22:54:391
3741533991614,1cyclictest0-21swapper/120:51:581
3741529991614,1cyclictest0-21swapper/023:43:010
3741529991612,3cyclictest3781642-21/usr/sbin/munin21:00:170
3741533991512,2cyclictest3056-21PK-Backend21:00:161
374152999159,4cyclictest3821047-21grep22:50:220
3741529991513,1cyclictest0-21swapper/020:25:300
3741529991510,4cyclictest3759492-21wc19:55:200
3741529991510,4cyclictest3759492-21wc19:55:200
374152999150,3cyclictest3786603-21awk21:15:000
3741533991412,1cyclictest3056-21PK-Backend20:14:331
374152999141,10cyclictest3826997-21apt23:10:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional