You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 05:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Sep 20, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1717421350,5sleep00-21swapper/022:00:190
4986821190,5sleep342-21ksoftirqd/323:32:333
416159921060,3sleep00-21swapper/019:40:160
139099890,53rtkit-daemon0-21swapper/319:06:303
414949928140,13sleep10-21swapper/119:07:131
414963926343,14sleep00-21swapper/019:09:020
414955226343,14sleep20-21swapper/219:07:552
350732590,2sleep30-21swapper/322:50:143
235972520,3sleep03056-21PK-Backend22:20:010
523862460,3sleep00-21swapper/023:40:120
41671252390,1chrt4166380-21/usr/sbin/munin19:55:200
372752390,4sleep03056-21PK-Backend22:55:220
41604042360,3sleep33056-21PK-Backend19:35:233
41743892340,2sleep33056-21PK-Backend20:15:213
139099310,4rtkit-daemon1788643-21eadedCompositor20:21:200
414986099268,3cyclictest141rcu_preempt21:50:203
414986099259,14cyclictest0-21swapper/323:05:223
414986099236,3cyclictest42-21ksoftirqd/323:35:333
414985899239,5cyclictest34-21ksoftirqd/220:58:092
4149858992316,5cyclictest4153746-21apt-get19:20:012
4149858992315,7cyclictest30363-21users22:35:232
4149858992315,6cyclictest4158602-21python319:30:212
4149858992314,7cyclictest61836-21users00:05:242
414985499234,16cyclictest4173322-21dump-pmu-power20:15:001
4149854992320,2cyclictest3056-21PK-Backend22:34:441
4149854992313,7cyclictest67017-21sed00:20:221
4149853992321,1cyclictest0-21swapper/022:53:360
4149853992320,2cyclictest0-21swapper/021:05:070
414986099222,18cyclictest0-21swapper/321:35:133
4149860992219,2cyclictest3056-21PK-Backend21:41:063
4149860992219,2cyclictest3056-21PK-Backend19:23:233
4149860992214,6cyclictest4179375-21memory20:30:183
4149860992210,4cyclictest42-21ksoftirqd/320:50:083
414986099220,21cyclictest0-21swapper/321:00:233
414986099220,10cyclictest240532cstates22:20:133
414985899229,4cyclictest34-21ksoftirqd/223:41:582
4149858992213,7cyclictest20749-21/usr/sbin/munin22:10:182
4149858992211,9cyclictest4162988-21cpu19:45:152
414985899220,1cyclictest4161877-21strings19:40:192
4149854992220,1cyclictest0-21swapper/123:46:571
4149854992220,1cyclictest0-21swapper/122:06:251
4149854992220,1cyclictest0-21swapper/121:15:011
4149854992219,2cyclictest3056-21PK-Backend00:02:331
4149854992211,8cyclictest436-20systemd-journal20:00:011
414985499220,20cyclictest3056-21PK-Backend20:43:241
414985499220,20cyclictest3056-21PK-Backend20:36:011
414985499220,20cyclictest3056-21PK-Backend00:16:311
414985499220,20cyclictest3056-21PK-Backend00:16:311
414985499220,20cyclictest0-21swapper/120:20:131
4149853992220,1cyclictest0-21swapper/023:06:330
4149853992220,1cyclictest0-21swapper/022:11:450
4149853992220,1cyclictest0-21swapper/021:13:460
4149853992220,1cyclictest0-21swapper/020:43:450
4149853992220,1cyclictest0-21swapper/020:35:200
4149853992220,1cyclictest0-21swapper/020:19:370
4149853992220,1cyclictest0-21swapper/000:36:310
4149853992220,1cyclictest0-21swapper/000:32:040
4149853992220,1cyclictest0-21swapper/000:25:010
414985399220,21cyclictest0-21swapper/022:23:490
67872210,5sleep30-21swapper/321:30:243
4149860992119,1cyclictest0-21swapper/323:29:553
4149860992119,1cyclictest0-21swapper/320:46:183
4149860992119,1cyclictest0-21swapper/319:12:573
4149860992119,1cyclictest0-21swapper/300:36:473
414986099211,7cyclictest5047-21python321:25:223
4149860992113,5cyclictest4163804-21python319:45:233
4149860992112,7cyclictest28769-21apt-get22:35:003
414986099210,20cyclictest0-21swapper/323:22:333
414986099210,20cyclictest0-21swapper/322:58:133
414986099210,20cyclictest0-21swapper/322:43:523
4149858992119,1cyclictest0-21swapper/221:15:242
4149858992114,5cyclictest17806-21http22:05:002
4149858992113,6cyclictest436-20systemd-journal00:40:002
4149858992113,6cyclictest4190685-21systemd-run21:05:002
4149858992113,6cyclictest4166380-21/usr/sbin/munin19:55:212
414985899210,20cyclictest0-21swapper/221:43:022
414985899210,20cyclictest0-21swapper/219:37:422
4149854992119,1cyclictest3056-21PK-Backend22:44:251
4149854992119,1cyclictest3056-21PK-Backend22:35:321
4149854992119,1cyclictest3056-21PK-Backend21:05:151
4149854992119,1cyclictest0-21swapper/122:01:181
4149854992119,1cyclictest0-21swapper/121:53:481
4149854992119,1cyclictest0-21swapper/121:12:181
4149854992119,1cyclictest0-21swapper/120:01:411
4149854992119,1cyclictest0-21swapper/119:24:231
4149854992119,1cyclictest0-21swapper/100:30:371
414985499210,20cyclictest3056-21PK-Backend20:56:161
414985499210,20cyclictest0-21swapper/120:15:401
414985499210,20cyclictest0-21swapper/119:39:101
414985499210,19cyclictest3056-21PK-Backend23:23:381
414985499210,19cyclictest3056-21PK-Backend23:12:441
414985499210,19cyclictest3056-21PK-Backend22:46:011
414985499210,19cyclictest3056-21PK-Backend22:15:521
414985499210,19cyclictest3056-21PK-Backend21:29:011
4149853992119,1cyclictest0-21swapper/023:35:060
4149853992119,1cyclictest0-21swapper/021:50:010
4149853992119,1cyclictest0-21swapper/020:49:590
4149853992119,1cyclictest0-21swapper/020:07:010
4149853992113,6cyclictest9102-21cron21:40:000
414985399210,2cyclictest0-21swapper/000:12:180
414985399210,20cyclictest0-21swapper/023:20:190
414985399210,20cyclictest0-21swapper/022:29:560
414985399210,20cyclictest0-21swapper/020:27:210
414985399210,20cyclictest0-21swapper/019:54:360
414985399210,20cyclictest0-21swapper/000:07:030
139099210,4rtkit-daemon1389-21rtkit-daemon19:50:151
139099210,3rtkit-daemon1389-21rtkit-daemon23:57:111
139099210,3rtkit-daemon1389-21rtkit-daemon22:05:250
4149860992018,1cyclictest3056-21PK-Backend20:03:113
4149860992018,1cyclictest0-21swapper/322:07:153
4149860992018,1cyclictest0-21swapper/321:09:513
4149860992018,1cyclictest0-21swapper/319:52:573
4149860992018,1cyclictest0-21swapper/300:09:233
414986099201,6cyclictest22220-21grep22:15:123
414986099200,1cyclictest3056-21PK-Backend20:00:003
414986099200,1cyclictest3056-21PK-Backend00:01:123
414986099200,1cyclictest0-21swapper/320:41:253
414986099200,19cyclictest3056-21PK-Backend23:54:503
414986099200,19cyclictest0-21swapper/300:32:213
414986099200,19cyclictest0-21swapper/300:26:203
414986099200,18cyclictest3056-21PK-Backend21:21:193
414986099200,18cyclictest3056-21PK-Backend21:21:193
414985899202,16cyclictest46853-21find23:25:122
4149858992018,1cyclictest0-21swapper/223:50:432
4149858992018,1cyclictest0-21swapper/223:46:312
4149858992018,1cyclictest0-21swapper/223:38:542
4149858992018,1cyclictest0-21swapper/223:20:422
4149858992018,1cyclictest0-21swapper/222:30:532
4149858992018,1cyclictest0-21swapper/221:59:372
4149858992018,1cyclictest0-21swapper/221:46:462
4149858992018,1cyclictest0-21swapper/221:34:092
4149858992018,1cyclictest0-21swapper/221:22:132
4149858992018,1cyclictest0-21swapper/221:22:122
4149858992018,1cyclictest0-21swapper/221:06:442
4149858992018,1cyclictest0-21swapper/220:27:242
4149858992018,1cyclictest0-21swapper/220:15:082
4149858992018,1cyclictest0-21swapper/200:18:022
4149858992018,1cyclictest0-21swapper/200:18:012
4149858992018,1cyclictest0-21swapper/200:12:332
4149858992018,1cyclictest0-21swapper/200:00:442
4149858992014,5cyclictest19028-21ntp_kernel_pll_22:05:202
4149858992013,6cyclictest23340-21sshd22:17:282
4149858992013,5cyclictest436-20systemd-journal22:24:192
4149858992013,5cyclictest42322-21needreboot23:10:192
4149858992012,6cyclictest68087-21fw_packets00:25:162
4149858992012,6cyclictest40311-21df_inode23:05:152
414985899201,1cyclictest0-21swapper/221:37:072
414985899200,19cyclictest0-21swapper/223:55:442
414985899200,19cyclictest0-21swapper/223:01:562
414985899200,19cyclictest0-21swapper/222:48:102
414985899200,19cyclictest0-21swapper/222:28:302
414985899200,19cyclictest0-21swapper/220:50:502
414985899200,19cyclictest0-21swapper/220:48:412
414985899200,19cyclictest0-21swapper/220:01:422
414985899200,19cyclictest0-21swapper/200:31:332
414985899200,19cyclictest0-21swapper/200:21:392
4149854992018,1cyclictest3056-21PK-Backend23:15:061
4149854992018,1cyclictest3056-21PK-Backend23:15:061
4149854992018,1cyclictest3056-21PK-Backend22:29:371
4149854992018,1cyclictest3056-21PK-Backend00:05:441
4149854992018,1cyclictest0-21swapper/121:57:331
4149854992018,1cyclictest0-21swapper/119:40:591
4149854992015,3cyclictest10850-21idleruntime-cro21:45:001
4149854992013,6cyclictest49373-21awk23:30:211
4149854992012,6cyclictest36531-21awk22:55:131
4149854992012,5cyclictest25288-21munin-plugin-st22:25:001
414985499200,1cyclictest0-21swapper/123:41:091
414985499200,19cyclictest3056-21PK-Backend22:51:201
414985499200,19cyclictest3056-21PK-Backend20:25:341
414985499200,19cyclictest3056-21PK-Backend00:39:021
414985499200,19cyclictest0-21swapper/119:27:351
414985499200,19cyclictest0-21swapper/119:16:391
414985499200,18cyclictest3056-21PK-Backend23:26:011
414985499200,18cyclictest3056-21PK-Backend23:01:141
4149853992018,1cyclictest0-21swapper/021:41:090
4149853992018,1cyclictest0-21swapper/020:52:080
4149853992018,1cyclictest0-21swapper/020:12:560
4149853992018,1cyclictest0-21swapper/000:19:340
4149853992018,1cyclictest0-21swapper/000:19:340
4149853992013,5cyclictest4163644-21awk19:45:210
414985399200,19cyclictest0-21swapper/022:36:520
414985399200,19cyclictest0-21swapper/021:59:420
414985399200,19cyclictest0-21swapper/021:23:520
414985399200,19cyclictest0-21swapper/021:23:520
414985399200,19cyclictest0-21swapper/021:02:100
414985399200,19cyclictest0-21swapper/019:17:450
414985399200,19cyclictest0-21swapper/019:10:160
41794992190,3chrt0-21swapper/220:30:192
414986099192,4cyclictest42-21ksoftirqd/321:12:143
4149860991917,1cyclictest3056-21PK-Backend21:59:463
4149860991917,1cyclictest3056-21PK-Backend00:22:553
4149860991917,1cyclictest0-21swapper/322:11:483
4149860991917,1cyclictest0-21swapper/320:27:113
414986099191,6cyclictest33378-21/usr/sbin/munin22:45:133
4149860991914,3cyclictest4187442-21basename20:55:113
4149860991914,3cyclictest30143-21gzip22:35:203
4149860991913,5cyclictest53313-21sshd23:42:193
4149860991911,6cyclictest42-21ksoftirqd/321:45:303
4149860991911,6cyclictest42-21ksoftirqd/320:20:233
4149860991911,6cyclictest42-21ksoftirqd/320:06:463
4149860991911,6cyclictest42-21ksoftirqd/319:31:283
414986099190,1cyclictest3056-21PK-Backend23:58:483
414986099190,1cyclictest3056-21PK-Backend21:19:573
414986099190,1cyclictest0-21swapper/322:27:373
414986099190,18cyclictest3056-21PK-Backend19:16:413
4149858991917,1cyclictest0-21swapper/220:43:462
4149858991917,1cyclictest0-21swapper/219:27:422
4149858991917,1cyclictest0-21swapper/219:20:332
4149858991917,1cyclictest0-21swapper/219:12:482
414985899191,3cyclictest141rcu_preempt20:10:002
4149858991911,6cyclictest43394-21find23:15:112
4149858991911,6cyclictest43394-21find23:15:112
414985899190,1cyclictest0-21swapper/220:10:342
4149854991917,1cyclictest3056-21PK-Backend19:13:411
4149854991917,1cyclictest3056-21PK-Backend00:13:541
4149854991917,1cyclictest0-21swapper/121:36:511
414985499190,1cyclictest3056-21PK-Backend21:31:101
414985499190,1cyclictest3056-21PK-Backend20:52:101
414985499190,18cyclictest0-21swapper/119:34:581
4149853991917,1cyclictest0-21swapper/019:20:090
4149853991912,6cyclictest58347-21awk23:55:230
4149853991911,6cyclictest66470-21cut00:20:170
414985399190,1cyclictest0-21swapper/019:30:300
414985399190,18cyclictest0-21swapper/021:19:460
4149860991813,3cyclictest62468-21/usr/sbin/munin00:10:143
4149860991812,5cyclictest38932-21ntp_kernel_pll_23:00:193
4149860991810,6cyclictest42-21ksoftirqd/322:00:163
4149860991810,6cyclictest42-21ksoftirqd/320:10:223
4149858991814,3cyclictest4192865-21cstates21:10:132
4149858991812,4cyclictest36686-21cat22:55:132
4149858991810,6cyclictest5131-21wc21:25:232
414985899180,1cyclictest0-21swapper/220:24:492
414985499180,17cyclictest3056-21PK-Backend23:06:351
414985499180,17cyclictest3056-21PK-Backend21:22:261
414985499180,17cyclictest3056-21PK-Backend21:22:251
414985499180,17cyclictest0-21swapper/100:26:121
4149853991816,1cyclictest0-21swapper/020:59:590
4149853991816,1cyclictest0-21swapper/020:31:200
4149853991813,3cyclictest7388-21munin-run21:35:010
4149853991811,5cyclictest11303-21expr21:45:120
4149853991810,7cyclictest59959-21sed00:00:230
4149860991715,1cyclictest0-21swapper/300:15:533
4149860991715,1cyclictest0-21swapper/300:15:523
4149860991713,3cyclictest42-21ksoftirqd/320:40:003
414986099171,14cyclictest4162801-21latency_hist19:45:013
4149860991710,6cyclictest42-21ksoftirqd/323:11:183
4149860991710,2cyclictest42-21ksoftirqd/323:46:053
4149860991710,2cyclictest42-21ksoftirqd/319:27:083
414986099170,15cyclictest43409-21apt23:15:123
414986099170,15cyclictest43409-21apt23:15:123
414985899179,6cyclictest34746-21awk22:50:132
4149858991715,1cyclictest0-21swapper/221:52:072
4149858991711,5cyclictest4165013-21ls19:50:132
4149858991710,5cyclictest48663-21awk23:30:132
4149858991710,5cyclictest31890-21swap22:40:222
414985499173,9cyclictest55542-21apt23:50:111
4149854991715,1cyclictest3056-21PK-Backend22:11:331
4149854991715,1cyclictest3056-21PK-Backend20:32:171
4149854991711,5cyclictest4171153-21sed20:05:251
4149853991715,1cyclictest0-21swapper/023:49:540
4149853991715,1cyclictest0-21swapper/023:29:180
4149853991715,1cyclictest0-21swapper/022:47:130
4149853991715,1cyclictest0-21swapper/019:39:430
4149853991710,5cyclictest56078-21sed23:50:160
4149853991710,5cyclictest4169549-21seq20:01:130
4149858991612,2cyclictest4181990-21latency_hist20:40:002
414985499169,5cyclictest11543-21cat21:45:161
4149854991611,3cyclictest50399-21cpuspeed23:35:121
414985399169,5cyclictest4156420-21ls19:25:170
414985399169,5cyclictest27460-21tr22:30:120
139099160,13rtkit-daemon0-21swapper/022:41:580
41640982150,4sleep13056-21PK-Backend19:46:191
4149854991513,1cyclictest0-21swapper/120:47:301
4149853991513,1cyclictest0-21swapper/021:26:030
4149853991511,3cyclictest39013-21processes23:00:200
4149853991510,3cyclictest43846-21if_enp2s023:15:170
4149853991510,3cyclictest43846-21if_enp2s023:15:160
414985399149,3cyclictest1095-21runrttasks23:30:130
414985399138,3cyclictest43309-21latency_hist23:15:000
414985499128,3cyclictest4189383-21tr21:00:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional