You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 03:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Wed May 06, 2026 00:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13185992724,2cyclictest6854-21awk21:10:012
1319299250,24cyclictest6815-21diskmemload21:27:463
1319299250,1cyclictest0-21swapper/322:22:143
13185992523,1cyclictest0-21swapper/223:26:052
1318599252,22cyclictest0-21swapper/200:26:272
1318599250,25cyclictest0-21swapper/200:30:562
1318299250,24cyclictest0-21swapper/100:02:001
1318299250,23cyclictest0-21swapper/121:34:251
1318299250,0cyclictest0-21swapper/100:28:411
13179992524,1cyclictest62550irq/127-eno122:21:250
1317999250,24cyclictest6815-21diskmemload23:33:230
1317999250,24cyclictest0-21swapper/022:42:160
1317999250,0cyclictest0-21swapper/023:03:160
13192992423,1cyclictest0-21swapper/321:39:343
13192992418,1cyclictest807-21diskmemload21:32:133
1319299240,23cyclictest30824-21ssh23:24:433
1319299240,1cyclictest6815-21diskmemload22:03:433
1319299240,1cyclictest34-21ksoftirqd/322:36:363
1319299240,1cyclictest0-21swapper/321:11:193
1319299240,0cyclictest0-21swapper/323:34:583
13185992423,0cyclictest16375-21ssh23:40:352
13185992423,0cyclictest0-21swapper/200:01:392
1318599242,22cyclictest6815-21diskmemload22:57:532
1318599242,22cyclictest6815-21diskmemload21:56:202
13185992422,2cyclictest26337-21/usr/sbin/munin00:16:342
13185992422,1cyclictest6815-21diskmemload00:39:522
13185992422,1cyclictest2807-21ssh21:34:592
13185992422,1cyclictest0-21swapper/223:06:342
1318599242,1cyclictest8213-21ssh21:39:422
1318599242,1cyclictest16043-21ssh22:42:532
13185992421,2cyclictest19734-21sh22:17:312
13185992419,4cyclictest0-21swapper/223:45:512
1318599241,22cyclictest0-21swapper/221:26:062
1318599240,23cyclictest29115-21latency_hist22:55:012
1318599240,23cyclictest0-21swapper/223:22:082
1318599240,23cyclictest0-21swapper/222:06:142
1318599240,23cyclictest0-21swapper/200:11:212
13182992423,0cyclictest6815-21diskmemload23:30:041
1318299242,21cyclictest30726-21sh22:56:261
13182992422,1cyclictest18276-21cron21:20:011
1318299240,24cyclictest0-21swapper/100:13:021
1318299240,23cyclictest6815-21diskmemload00:05:521
1318299240,23cyclictest10041-21ssh21:11:401
1318299240,1cyclictest6815-21diskmemload00:31:481
1318299240,1cyclictest0-21swapper/122:30:021
1318299240,0cyclictest6815-21diskmemload23:40:031
13179992421,2cyclictest0-21swapper/022:17:140
1317999240,2cyclictest19230-21ssh23:14:310
1317999240,24cyclictest0-21swapper/023:51:360
1317999240,24cyclictest0-21swapper/021:45:280
1317999240,24cyclictest0-21swapper/021:32:250
1317999240,23cyclictest6815-21diskmemload22:35:510
1317999240,23cyclictest0-21swapper/000:15:000
13192992323,0cyclictest0-21swapper/323:43:283
13192992323,0cyclictest0-21swapper/321:20:563
13192992323,0cyclictest0-21swapper/300:15:283
1319299232,1cyclictest34-21ksoftirqd/321:53:543
1319299232,1cyclictest34-21ksoftirqd/321:53:543
13192992321,1cyclictest0-21swapper/300:26:273
1319299230,23cyclictest0-21swapper/323:45:433
1319299230,23cyclictest0-21swapper/322:40:233
1319299230,22cyclictest9674-21grep23:06:263
1319299230,22cyclictest3498-21ssh23:01:193
1319299230,22cyclictest17646-21ssh00:09:423
1319299230,22cyclictest12027-21fschecks_time22:11:273
1319299230,22cyclictest0-21swapper/321:45:463
1319299230,22cyclictest0-21swapper/300:10:033
1319299230,1cyclictest17933-21perf19:20:013
1319299230,1cyclictest0-21swapper/322:35:003
1319299230,0cyclictest6815-21diskmemload23:59:513
1319299230,0cyclictest6815-21diskmemload23:27:533
1319299230,0cyclictest0-21swapper/322:28:203
1319299230,0cyclictest0-21swapper/321:43:253
1319299230,0cyclictest0-21swapper/321:16:003
13185992323,0cyclictest0-21swapper/220:01:252
13185992322,1cyclictest6815-21diskmemload23:15:472
13185992322,0cyclictest0-21swapper/223:36:442
13185992321,1cyclictest0-21swapper/221:50:012
1318599231,21cyclictest0-21swapper/221:53:422
1318599231,21cyclictest0-21swapper/221:53:422
1318599231,1cyclictest0-21swapper/222:35:012
1318599230,2cyclictest12235-21rm21:42:222
1318599230,23cyclictest0-21swapper/223:51:542
1318599230,23cyclictest0-21swapper/222:30:122
1318599230,23cyclictest0-21swapper/200:20:302
1318599230,22cyclictest9992-21munin-run20:10:012
1318599230,22cyclictest14795-21grep20:20:012
1318599230,22cyclictest0-21swapper/223:04:162
1318599230,22cyclictest0-21swapper/222:46:092
1318599230,1cyclictest6815-21diskmemload23:31:252
1318599230,0cyclictest6815-21diskmemload22:00:462
13182992323,0cyclictest6815-21diskmemload23:49:561
13182992323,0cyclictest6815-21diskmemload22:15:031
13182992323,0cyclictest0-21swapper/123:15:451
13182992323,0cyclictest0-21swapper/122:50:141
13182992321,1cyclictest622-21gdbus20:50:001
13182992321,1cyclictest15620-21ssh21:46:251
13182992321,1cyclictest0-21swapper/123:51:291
13182992320,2cyclictest5506-21sh22:06:011
1318299230,23cyclictest0-21swapper/123:36:491
1318299230,22cyclictest841-21sh00:22:511
1318299230,22cyclictest6815-21diskmemload23:25:511
1318299230,22cyclictest3136-21apt-get19:56:231
1318299230,22cyclictest20360-21sh21:50:091
1318299230,22cyclictest20360-21sh21:50:091
1318299230,22cyclictest19732-21cpuspeed_turbos21:21:241
1318299230,22cyclictest18551-21sh22:45:581
1318299230,22cyclictest15773-21sh22:42:251
1318299230,0cyclictest6815-21diskmemload00:35:381
1318299230,0cyclictest0-21swapper/123:24:431
1318299230,0cyclictest0-21swapper/123:00:181
1318299230,0cyclictest0-21swapper/122:02:251
13179992323,0cyclictest6815-21diskmemload23:17:210
13179992323,0cyclictest0-21swapper/023:46:060
13179992323,0cyclictest0-21swapper/021:51:300
13179992323,0cyclictest0-21swapper/021:51:300
13179992323,0cyclictest0-21swapper/021:21:000
13179992323,0cyclictest0-21swapper/000:01:010
13179992322,1cyclictest6815-21diskmemload22:31:010
13179992322,1cyclictest62550irq/127-eno121:37:060
13179992320,2cyclictest27116-21sh21:26:340
13179992320,2cyclictest2223-21sendmail23:00:000
1317999230,22cyclictest6815-21diskmemload00:33:050
1317999230,22cyclictest5850-21ssh23:59:090
1317999230,22cyclictest27218-21apt-get20:46:240
1317999230,22cyclictest26152-21ssh21:55:200
1317999230,22cyclictest2269-21sh22:01:500
1317999230,22cyclictest14935-21sh00:36:110
1317999230,22cyclictest0-21swapper/023:37:260
1317999230,22cyclictest0-21swapper/000:05:060
1317999230,1cyclictest9208-21apt-get23:06:240
1317999230,1cyclictest0-21swapper/020:10:010
1317999230,0cyclictest6815-21diskmemload21:17:210
1317999230,0cyclictest0-21swapper/023:26:140
1317999230,0cyclictest0-21swapper/022:25:200
1319299228,14cyclictest0-21swapper/320:23:473
13192992222,0cyclictest6815-21diskmemload00:35:423
13192992222,0cyclictest0-21swapper/323:15:093
13192992222,0cyclictest0-21swapper/322:46:583
13192992222,0cyclictest0-21swapper/300:30:563
13192992220,1cyclictest14798-21latency_hist20:20:013
13192992220,1cyclictest0-21swapper/322:53:353
13192992220,1cyclictest0-21swapper/322:16:403
1319299221,21cyclictest34-21ksoftirqd/321:55:053
1319299221,20cyclictest0-21swapper/320:41:253
1319299220,2cyclictest28149-21ssh23:50:353
1319299220,22cyclictest0-21swapper/300:00:213
1319299220,21cyclictest6815-21diskmemload23:10:023
1319299220,21cyclictest619-21polkitd21:10:013
1319299220,21cyclictest331rcuc/322:55:143
1319299220,21cyclictest1650-21cut21:00:013
1319299220,21cyclictest15012-21sh23:39:043
1319299220,21cyclictest0-21swapper/322:05:293
13185992222,0cyclictest0-21swapper/220:56:242
13185992221,1cyclictest0-21swapper/221:01:252
13185992220,2cyclictest12802-21ssh00:05:342
13185992220,1cyclictest0-21swapper/221:15:322
13185992220,0cyclictest0-21swapper/222:25:012
1318599221,21cyclictest6815-21diskmemload23:12:092
1318599220,2cyclictest21903-21ssh21:22:172
1318599220,21cyclictest3955-21ssh23:56:382
1318599220,21cyclictest22144-21ssh22:20:232
1318599220,21cyclictest0-21swapper/219:35:012
1318599220,21cyclictest0-21swapper/219:26:232
1318599220,1cyclictest7272-21sh21:10:162
1318599220,1cyclictest0-21swapper/222:10:562
13182992221,1cyclictest0-21swapper/121:40:141
13182992220,1cyclictest0-21swapper/122:36:291
1318299221,21cyclictest0-21swapper/122:27:521
1318299220,21cyclictest0-21swapper/121:26:261
1318299220,21cyclictest0-21swapper/120:05:001
1318299220,1cyclictest8374-21kworker/u8:023:07:511
1318299220,1cyclictest29862-21ssh21:58:041
1318299220,1cyclictest23968-21sh00:15:221
1318299220,1cyclictest22600-21ssh22:20:571
1318299220,1cyclictest15677-21ssh22:14:471
1318299220,1cyclictest0-21swapper/121:36:301
1318299220,1cyclictest0-21swapper/120:15:011
1318299220,1cyclictest0-21swapper/119:11:241
13179992222,0cyclictest6815-21diskmemload22:50:040
13179992222,0cyclictest0-21swapper/023:40:260
13179992221,1cyclictest6815-21diskmemload00:25:080
13179992220,1cyclictest0-21swapper/022:46:260
1317999220,22cyclictest0-21swapper/021:40:260
1317999220,22cyclictest0-21swapper/021:13:580
1317999220,21cyclictest15162-21apt-get20:21:230
1317999220,21cyclictest0-21swapper/023:20:100
1317999220,21cyclictest0-21swapper/000:18:000
1317999220,1cyclictest5331-21ssh22:05:480
1317999220,1cyclictest0-21swapper/022:11:260
1317999220,1cyclictest0-21swapper/000:20:580
13192992121,0cyclictest0-21swapper/320:01:253
13192992119,1cyclictest0-21swapper/320:26:233
13192992119,1cyclictest0-21swapper/320:10:013
13192992119,1cyclictest0-21swapper/319:31:233
1319299210,20cyclictest0-21swapper/320:11:233
1318599210,20cyclictest0-21swapper/220:26:242
13182992120,0cyclictest29613-21apt-get20:51:231
13182992119,1cyclictest20011-21apt-get20:31:241
1318299210,21cyclictest0-21swapper/119:41:241
1318299210,0cyclictest0-21swapper/119:26:231
13179992121,0cyclictest25946-21apt-get19:36:210
1317999210,0cyclictest0-21swapper/020:36:250
13192992020,0cyclictest0-21swapper/319:26:233
1319299200,6cyclictest0-21swapper/319:45:113
1319299200,20cyclictest0-21swapper/320:34:333
1319299200,0cyclictest0-21swapper/319:36:213
1319299200,0cyclictest0-21swapper/319:11:243
13185992019,0cyclictest0-21swapper/219:56:232
13185992019,0cyclictest0-21swapper/219:11:242
13185992017,2cyclictest0-21swapper/220:51:242
1318599200,20cyclictest0-21swapper/220:31:242
13182992020,0cyclictest0-21swapper/120:36:241
13179992020,0cyclictest20760-21apt-get19:26:230
13179992020,0cyclictest0-21swapper/019:46:240
13179992018,1cyclictest0-21swapper/019:25:010
13179992018,1cyclictest0-21swapper/019:25:010
1317999200,20cyclictest0-21swapper/020:41:240
1317999200,20cyclictest0-21swapper/020:31:230
1317999200,0cyclictest0-21swapper/020:11:230
13192991918,1cyclictest0-21swapper/321:01:253
13185991919,0cyclictest0-21swapper/219:21:232
13185991919,0cyclictest0-21swapper/219:21:232
13185991917,1cyclictest25858-21ntpq20:41:302
13185991915,3cyclictest28322-21phc2sys20:46:302
13182991919,0cyclictest0-21swapper/120:56:231
13182991919,0cyclictest0-21swapper/120:21:231
13182991918,1cyclictest4801-21apt-get21:06:231
13179991919,0cyclictest0-21swapper/021:01:250
13179991919,0cyclictest0-21swapper/019:41:240
13179991918,1cyclictest13498-21apt-get19:11:240
13192991818,0cyclictest0-21swapper/319:51:233
13192991817,1cyclictest18310-21apt-get19:21:233
13192991817,1cyclictest18310-21apt-get19:21:233
13192991817,1cyclictest0-21swapper/320:51:233
13185991818,0cyclictest0-21swapper/219:36:212
13185991817,1cyclictest22408-21apt-get20:36:252
1318599181,16cyclictest0-21swapper/219:16:232
13182991818,0cyclictest0-21swapper/120:41:241
13179991818,0cyclictest0-21swapper/019:31:240
13179991816,1cyclictest7587-21perf20:05:000
1317999180,0cyclictest0-21swapper/019:16:230
13185991717,0cyclictest0-21swapper/220:21:232
13185991717,0cyclictest0-21swapper/219:51:242
13185991715,1cyclictest32433-21grep19:46:332
13182991717,0cyclictest0-21swapper/119:21:231
13182991717,0cyclictest0-21swapper/119:21:231
13182991715,1cyclictest14796-21perf20:20:011
13179991717,0cyclictest0-21swapper/021:06:230
13179991717,0cyclictest0-21swapper/020:51:230
13179991716,1cyclictest12736-21apt-get20:16:240
13179991715,1cyclictest0-21swapper/019:51:230
13192991616,0cyclictest0-21swapper/319:56:233
13192991613,2cyclictest24457-21awk20:40:003
13182991614,1cyclictest18461-21memory20:26:281
1318299160,15cyclictest2016-21apt-get21:01:251
1319299151,4cyclictest151rcu_preempt20:46:263
13182991515,0cyclictest0-21swapper/120:06:241
13182991515,0cyclictest0-21swapper/119:30:021
1319299140,0cyclictest0-21swapper/319:40:023
1318599140,14cyclictest0-21swapper/220:10:162
1318599140,14cyclictest0-21swapper/219:40:062
13182991414,0cyclictest0-21swapper/119:46:041
13182991414,0cyclictest0-21swapper/119:15:201
1318299140,14cyclictest0-21swapper/119:36:171
1318299140,0cyclictest0-21swapper/119:50:041
1317999140,14cyclictest0-21swapper/020:25:090
1317999140,14cyclictest0-21swapper/019:55:140
1317999140,0cyclictest0-21swapper/020:55:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional