You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-01 - 11:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Sat Nov 01, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991110,27rtkit-daemon2287-21gmain20:53:322
265866128964,16sleep30-21swapper/319:05:293
162399870,8rtkit-daemon1622-21rtkit-daemon00:27:383
162399830,50rtkit-daemon0-21swapper/219:06:292
27566852710,5sleep02197014-21PK-Backend23:45:030
27145012700,4sleep2331rcuc/221:45:142
265871027038,9sleep00-21swapper/019:06:060
162399670,9rtkit-daemon1622-21rtkit-daemon00:12:083
162399670,8rtkit-daemon1622-21rtkit-daemon23:48:462
162399670,11rtkit-daemon1622-21rtkit-daemon21:16:022
162399650,10rtkit-daemon1622-21rtkit-daemon22:53:432
265916399629,16cyclictest1753-21gnome-shell00:00:043
265916399629,16cyclictest1753-21gnome-shell00:00:043
162399620,9rtkit-daemon1622-21rtkit-daemon00:13:452
2659163996111,22cyclictest1753-21gnome-shell21:45:013
265869626141,14sleep10-21swapper/119:05:561
27533512600,19sleep342-21ksoftirqd/323:35:153
162399600,9rtkit-daemon1622-21rtkit-daemon23:01:583
162399600,9rtkit-daemon1622-21rtkit-daemon22:49:103
2659163995910,17cyclictest1753-21gnome-shell21:50:023
162399590,8rtkit-daemon1622-21rtkit-daemon23:15:433
162399590,8rtkit-daemon1622-21rtkit-daemon23:15:433
162399580,9rtkit-daemon1622-21rtkit-daemon23:48:143
162399580,9rtkit-daemon1622-21rtkit-daemon22:27:123
162399580,8rtkit-daemon1622-21rtkit-daemon23:24:293
162399580,8rtkit-daemon1622-21rtkit-daemon22:02:423
2659163995710,28cyclictest1753-21gnome-shell21:25:253
2659163995710,28cyclictest1753-21gnome-shell21:25:243
162399570,9rtkit-daemon1622-21rtkit-daemon23:28:563
162399570,8rtkit-daemon1622-21rtkit-daemon22:59:563
162399570,7rtkit-daemon1622-21rtkit-daemon00:00:292
162399570,7rtkit-daemon1622-21rtkit-daemon00:00:292
265916399568,19cyclictest2737634-21uname22:50:223
162399560,9rtkit-daemon1622-21rtkit-daemon21:20:483
265916399558,29cyclictest1753-21gnome-shell20:13:333
2659163995511,21cyclictest1753-21gnome-shell19:36:513
2659163995511,18cyclictest1753-21gnome-shell20:40:013
162399550,9rtkit-daemon1622-21rtkit-daemon21:14:393
162399550,8rtkit-daemon1622-21rtkit-daemon22:19:383
26631722540,14sleep0161rcuc/019:20:020
265916399545,17cyclictest2677909-21diskstats20:00:173
2659163995411,22cyclictest1753-21gnome-shell19:20:323
2659163995411,22cyclictest1753-21gnome-shell19:20:323
162399540,9rtkit-daemon1622-21rtkit-daemon21:39:583
162399540,9rtkit-daemon1622-21rtkit-daemon00:06:033
26763072530,7sleep00-21swapper/019:55:190
265916399539,17cyclictest1753-21gnome-shell22:12:133
265916399538,20cyclictest2671598-21users19:40:323
2659163995310,17cyclictest2726866-21cpuspeed_turbos22:20:153
162399530,8rtkit-daemon1622-21rtkit-daemon23:40:173
162399530,8rtkit-daemon1622-21rtkit-daemon21:06:383
265916399529,16cyclictest1753-21gnome-shell23:05:223
265916399528,17cyclictest1753-21gnome-shell00:30:163
265916399526,25cyclictest1753-21gnome-shell22:05:093
265916399526,25cyclictest1753-21gnome-shell22:05:093
2659163995210,26cyclictest1753-21gnome-shell20:20:253
265916399519,26cyclictest1753-21gnome-shell22:34:573
265916399519,26cyclictest1753-21gnome-shell19:33:253
265916399519,24cyclictest1753-21gnome-shell20:57:493
265916399519,24cyclictest1753-21gnome-shell20:33:003
265916399518,26cyclictest1753-21gnome-shell21:31:403
265916399518,22cyclictest1753-21gnome-shell00:15:173
265916399514,22cyclictest2693443-21apt20:45:093
2659163995111,19cyclictest1753-21gnome-shell00:35:173
2659163995111,15cyclictest1753-21gnome-shell20:00:013
2659163995110,24cyclictest1753-21gnome-shell21:56:413
2659163995110,15cyclictest1753-21gnome-shell20:50:153
162399510,6rtkit-daemon0-21swapper/220:12:212
162399510,36rtkit-daemon2693834-40lsb_release20:45:202
265916399509,23cyclictest2683476-21ntp_states20:15:243
265916399509,23cyclictest2683476-21ntp_states20:15:243
265916399508,9cyclictest2715097-21ps21:45:283
265916399508,25cyclictest1753-21gnome-shell23:11:263
265916399508,24cyclictest1753-21gnome-shell23:30:053
265916399508,19cyclictest1753-21gnome-shell22:40:023
265916399504,17cyclictest1753-21gnome-shell21:15:213
265916399502,13cyclictest3242253-21nautilus22:40:013
2659163995013,21cyclictest1753-21gnome-shell20:07:543
2659163995013,17cyclictest1753-21gnome-shell20:40:233
162399500,6rtkit-daemon1622-21rtkit-daemon23:24:502
162399500,4rtkit-daemon0-21swapper/223:58:512
265916399499,23cyclictest2758933-21df_inode23:50:183
265916399499,23cyclictest1753-21gnome-shell23:55:173
265916399498,16cyclictest1753-21gnome-shell00:22:503
265916399497,26cyclictest2665052-21apt19:25:123
2659152994917,13cyclictest1753-21gnome-shell23:10:250
265916399489,15cyclictest1753-21gnome-shell21:02:433
265916399488,22cyclictest1753-21gnome-shell19:46:493
2659161994831,6cyclictest34-21ksoftirqd/222:50:012
162399480,5rtkit-daemon0-21swapper/219:33:492
162399480,41rtkit-daemon2660866-21munin-node23:44:162
26991402470,16sleep013-21ksoftirqd/021:00:190
265916399473,19cyclictest2659946-21memory19:10:253
265916399470,30cyclictest854-21avahi-daemon19:50:593
265916199472,10cyclictest2764184-21munin-run00:10:012
2659157994711,20cyclictest2711851-21timerwakeupswit21:35:311
265915299479,17cyclictest27381412cat22:50:320
27272402460,6sleep02197014-21PK-Backend22:20:220
27162132460,6sleep013-21ksoftirqd/021:50:140
265916399463,19cyclictest2662648-21sed19:15:313
265916399462,26cyclictest1753-21gnome-shell20:25:043
2659161994629,13cyclictest855-21dbus-daemon20:01:552
2659161994622,14cyclictest34-21ksoftirqd/219:40:282
2659161994614,8cyclictest17531gnome-shell19:25:282
2659157994613,14cyclictest141rcu_preempt23:46:461
2659161994530,10cyclictest2689087-21systemd-udevd20:31:082
2659161994526,14cyclictest34-21ksoftirqd/222:40:022
2659161994525,15cyclictest34-21ksoftirqd/222:30:202
2659161994522,14cyclictest34-21ksoftirqd/219:20:182
2659161994522,14cyclictest34-21ksoftirqd/219:20:182
2659161994518,23cyclictest34-21ksoftirqd/220:30:012
265915799452,13cyclictest1753-21gnome-shell22:20:231
2659152994535,6cyclictest0-21swapper/019:45:130
27745692440,7sleep02197014-21PK-Backend00:40:000
27500782440,7chrt2197014-21PK-Backend23:25:282
2659161994426,14cyclictest34-21ksoftirqd/221:45:012
2659161994426,14cyclictest34-21ksoftirqd/200:27:482
2659161994425,14cyclictest34-21ksoftirqd/222:55:242
2659161994425,14cyclictest34-21ksoftirqd/221:50:192
2659161994425,14cyclictest34-21ksoftirqd/220:40:272
2659161994425,14cyclictest34-21ksoftirqd/220:20:242
2659161994424,14cyclictest34-21ksoftirqd/222:25:202
2659157994427,13cyclictest26-21ksoftirqd/122:15:311
265915799441,15cyclictest2719692-21cstates22:00:131
2659152994410,12cyclictest141rcu_preempt19:50:500
2659161994326,13cyclictest2701897-21/usr/sbin/munin21:10:012
2659161994325,14cyclictest34-21ksoftirqd/221:55:322
2659161994325,14cyclictest34-21ksoftirqd/220:20:022
2659161994325,14cyclictest34-21ksoftirqd/220:20:022
2659161994325,14cyclictest34-21ksoftirqd/220:10:012
2659161994325,14cyclictest34-21ksoftirqd/219:14:362
2659161994325,13cyclictest34-21ksoftirqd/222:10:112
2659161994324,7cyclictest34-21ksoftirqd/221:04:132
2659161994324,15cyclictest34-21ksoftirqd/219:20:012
2659161994324,14cyclictest34-21ksoftirqd/223:50:182
2659161994324,14cyclictest34-21ksoftirqd/223:35:032
2659161994324,14cyclictest34-21ksoftirqd/223:35:022
2659161994324,14cyclictest34-21ksoftirqd/222:20:172
2659161994324,14cyclictest34-21ksoftirqd/222:00:212
2659161994321,14cyclictest34-21ksoftirqd/222:15:142
2659161994321,13cyclictest34-21ksoftirqd/200:30:312
2659161994320,14cyclictest34-21ksoftirqd/223:00:282
265916199431,4cyclictest141rcu_preempt22:10:032
265916199431,4cyclictest141rcu_preempt22:10:022
2659157994325,14cyclictest26-21ksoftirqd/123:34:551
2659157994325,14cyclictest26-21ksoftirqd/119:35:251
2659157994324,7cyclictest26-21ksoftirqd/120:00:021
2659157994324,14cyclictest26-21ksoftirqd/122:50:111
2659157994324,14cyclictest26-21ksoftirqd/120:45:131
2659157994321,14cyclictest26-21ksoftirqd/122:10:081
265915799431,9cyclictest2685532-21sshd20:20:341
2659152994323,15cyclictest855-21dbus-daemon19:45:020
27084302420,6sleep10-21swapper/121:25:291
27084302420,6sleep10-21swapper/121:25:291
26779982420,5sleep1251rcuc/120:00:201
2659161994232,6cyclictest34-21ksoftirqd/221:12:432
2659161994232,6cyclictest34-21ksoftirqd/220:35:122
2659161994224,6cyclictest34-21ksoftirqd/223:15:302
2659161994224,6cyclictest34-21ksoftirqd/223:15:292
2659161994224,14cyclictest34-21ksoftirqd/221:35:022
2659161994224,14cyclictest34-21ksoftirqd/219:47:322
265916199422,35cyclictest3242253-21nautilus00:40:012
2659161994223,14cyclictest34-21ksoftirqd/222:44:592
2659161994223,14cyclictest34-21ksoftirqd/221:40:012
2659161994223,14cyclictest34-21ksoftirqd/219:40:022
265915799429,9cyclictest141rcu_preempt23:15:311
265915799429,9cyclictest141rcu_preempt23:15:311
2659157994225,13cyclictest26-21ksoftirqd/119:10:181
2659157994224,14cyclictest26-21ksoftirqd/123:22:041
2659157994224,14cyclictest26-21ksoftirqd/123:14:461
2659157994224,14cyclictest26-21ksoftirqd/121:18:341
2659157994224,14cyclictest26-21ksoftirqd/121:00:321
2659157994224,14cyclictest26-21ksoftirqd/120:50:211
2659157994224,14cyclictest26-21ksoftirqd/100:15:231
2659157994223,14cyclictest26-21ksoftirqd/120:30:331
2659157994223,14cyclictest26-21ksoftirqd/100:20:271
2659157994222,15cyclictest26-21ksoftirqd/119:42:281
2659157994222,13cyclictest26-21ksoftirqd/120:45:021
2659157994220,17cyclictest26-21ksoftirqd/119:50:081
2659157994214,6cyclictest26-21ksoftirqd/120:10:291
2659157994211,6cyclictest141rcu_preempt23:35:041
2659152994224,14cyclictest13-21ksoftirqd/020:45:240
2659152994224,14cyclictest13-21ksoftirqd/020:18:100
2659152994224,14cyclictest13-21ksoftirqd/020:18:100
2659152994224,13cyclictest855-21dbus-daemon00:05:020
2659152994224,13cyclictest855-21dbus-daemon00:05:020
2659152994215,7cyclictest141rcu_preempt22:03:030
2659152994212,13cyclictest141rcu_preempt00:06:210
2659152994212,10cyclictest141rcu_preempt21:47:240
26813772410,6sleep013-21ksoftirqd/020:10:170
2659161994124,14cyclictest34-21ksoftirqd/221:00:002
2659161994124,13cyclictest34-21ksoftirqd/221:30:012
2659161994124,13cyclictest34-21ksoftirqd/221:30:012
2659161994123,6cyclictest34-21ksoftirqd/200:20:012
2659161994123,14cyclictest34-21ksoftirqd/200:20:202
2659161994122,14cyclictest34-21ksoftirqd/223:05:222
2659161994120,16cyclictest34-21ksoftirqd/219:51:332
265915799414,11cyclictest26-21ksoftirqd/121:48:591
2659157994129,8cyclictest1753-21gnome-shell19:46:291
2659157994124,6cyclictest26-21ksoftirqd/120:25:311
265915799412,35cyclictest1753-21gnome-shell22:55:101
2659157994123,14cyclictest26-21ksoftirqd/123:42:211
2659157994123,14cyclictest26-21ksoftirqd/123:28:361
2659157994123,14cyclictest26-21ksoftirqd/121:55:021
2659157994123,13cyclictest26-21ksoftirqd/122:30:121
2659157994123,13cyclictest26-21ksoftirqd/100:07:111
2659157994122,7cyclictest26-21ksoftirqd/100:05:031
2659157994122,7cyclictest26-21ksoftirqd/100:05:021
2659157994120,17cyclictest26-21ksoftirqd/122:09:451
2659157994120,17cyclictest26-21ksoftirqd/122:09:451
2659157994119,14cyclictest26-21ksoftirqd/119:19:011
265915299418,15cyclictest141rcu_preempt20:51:340
2659152994124,13cyclictest13-21ksoftirqd/022:31:090
2659152994123,14cyclictest13-21ksoftirqd/021:30:120
2659152994122,14cyclictest13-21ksoftirqd/021:15:140
2659152994121,15cyclictest3242253-21nautilus22:10:030
2659161994018,14cyclictest34-21ksoftirqd/223:10:162
265915799402,9cyclictest1753-21gnome-shell20:05:231
2659157994028,7cyclictest1753-21gnome-shell19:30:141
2659157994023,13cyclictest26-21ksoftirqd/122:44:531
2659157994022,6cyclictest26-21ksoftirqd/100:35:001
2659157994022,14cyclictest26-21ksoftirqd/123:08:491
2659157994022,14cyclictest26-21ksoftirqd/121:30:251
2659157994022,14cyclictest26-21ksoftirqd/100:25:461
265915799401,13cyclictest1753-21gnome-shell23:50:161
265915299404,20cyclictest1753-21gnome-shell00:20:000
2659152994023,13cyclictest2670005-21awk19:40:010
2659152994022,6cyclictest13-21ksoftirqd/000:30:200
2659152994022,13cyclictest438-20systemd-journal00:00:000
2659152994022,13cyclictest13-21ksoftirqd/023:40:010
2659152994021,14cyclictest13-21ksoftirqd/020:45:010
265915299401,11cyclictest13-21ksoftirqd/022:15:130
2659157993924,11cyclictest2697669-21ntp_kernel_err20:55:241
2659157993922,13cyclictest26-21ksoftirqd/119:27:001
2659157993920,6cyclictest26-21ksoftirqd/100:00:031
265915299396,6cyclictest141rcu_preempt21:11:230
265915299396,5cyclictest141rcu_preempt22:55:090
265915299392,33cyclictest1753-21gnome-shell22:07:300
2659152993922,13cyclictest13-21ksoftirqd/023:25:150
2659152993922,13cyclictest13-21ksoftirqd/021:35:440
2659152993920,14cyclictest13-21ksoftirqd/022:12:180
2659152993920,14cyclictest13-21ksoftirqd/020:20:190
2659152993917,13cyclictest13-21ksoftirqd/021:05:210
162399390,26rtkit-daemon2197014-21PK-Backend19:55:392
26800462380,7sleep00-21swapper/020:05:260
2659161993820,13cyclictest34-21ksoftirqd/221:20:152
2659157993824,10cyclictest2765144-21ntp_kernel_pll_00:10:201
2659157993822,13cyclictest26-21ksoftirqd/123:00:261
2659157993821,5cyclictest26-21ksoftirqd/121:55:281
2659157993821,13cyclictest26-21ksoftirqd/120:35:211
2659157993821,12cyclictest26-21ksoftirqd/122:40:021
2659157993820,6cyclictest26-21ksoftirqd/121:10:291
2659157993820,6cyclictest26-21ksoftirqd/100:35:141
2659157993820,14cyclictest26-21ksoftirqd/122:45:221
2659152993828,6cyclictest2691176-21sh20:40:000
265915299382,7cyclictest141rcu_preempt19:15:020
2659152993822,12cyclictest13-21ksoftirqd/022:40:020
2659152993821,13cyclictest13-21ksoftirqd/023:00:320
265915299381,9cyclictest2197014-21PK-Backend20:31:080
2659152993814,10cyclictest141rcu_preempt00:21:050
162399380,26rtkit-daemon0-21swapper/121:42:371
2659157993720,6cyclictest26-21ksoftirqd/122:27:331
2659157993720,13cyclictest26-21ksoftirqd/120:15:111
2659157993720,13cyclictest26-21ksoftirqd/120:15:111
2659157993718,6cyclictest26-21ksoftirqd/121:20:301
265915299377,9cyclictest181irq_work/000:10:090
265915299375,10cyclictest2197014-21PK-Backend23:05:230
265915299372,6cyclictest2197014-21PK-Backend21:58:030
2659152993726,7cyclictest1753-21gnome-shell21:41:000
2659152993722,11cyclictest2706097-21interrupts21:20:210
2659152993720,13cyclictest13-21ksoftirqd/020:29:590
2659152993718,14cyclictest13-21ksoftirqd/019:25:210
265915299371,32cyclictest2707500-21grep21:25:110
265915299371,32cyclictest2707500-21grep21:25:110
265915299371,14cyclictest26785692sed20:00:300
265915299366,13cyclictest141rcu_preempt22:40:310
2659152993623,9cyclictest2758295-21munin-run23:50:000
265915299362,28cyclictest2666784-21apt19:30:120
2659152993621,11cyclictest2697849-21switchtime20:55:310
2659152993613,10cyclictest141rcu_preempt00:28:260
2659157993524,7cyclictest1753-21gnome-shell19:22:471
2659157993524,7cyclictest1753-21gnome-shell19:22:471
2659157993520,10cyclictest438-20systemd-journal21:08:341
265915299354,4cyclictest141rcu_preempt23:50:340
265915299353,4cyclictest141rcu_preempt22:45:340
265915299351,9cyclictest0-21swapper/023:18:360
265915299351,9cyclictest0-21swapper/023:18:360
2659152993519,6cyclictest13-21ksoftirqd/019:25:010
2659152993519,6cyclictest13-21ksoftirqd/019:25:010
2659152993517,14cyclictest13-21ksoftirqd/023:25:000
2659152993515,15cyclictest13-21ksoftirqd/023:32:160
265915299351,10cyclictest2197014-21PK-Backend22:25:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional