You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-09 - 10:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot5s.osadl.org (updated Tue Jun 09, 2026 00:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18507992620,1cyclictest0-21swapper/221:44:182
18501992623,2cyclictest15635-21nvmesmart_nvme022:44:201
1849799260,25cyclictest20845-21idleruntime-cro19:15:000
1850999250,24cyclictest11752-21diskmemload23:48:193
1850799250,24cyclictest11752-21diskmemload23:20:022
1850799250,24cyclictest1047-21apt-get19:44:152
1850799250,24cyclictest0-21swapper/221:21:112
18501992522,2cyclictest2154-21sh00:02:511
1850199250,24cyclictest11752-21diskmemload21:42:191
1850199250,24cyclictest11752-21diskmemload00:19:051
1850199250,24cyclictest0-21swapper/123:54:141
1849799252,22cyclictest6453-21ssh00:06:160
1849799250,24cyclictest8100-21sh22:07:180
1849799250,24cyclictest27481-21sh21:24:320
18509992422,1cyclictest0-21swapper/322:44:053
1850999240,23cyclictest8744-21sh22:38:393
1850999240,23cyclictest11752-21diskmemload21:18:053
1850999240,23cyclictest11752-21diskmemload00:08:093
1850999240,23cyclictest0-21swapper/323:18:473
1850999240,23cyclictest0-21swapper/322:00:553
1850999240,23cyclictest0-21swapper/321:23:323
1850999240,23cyclictest0-21swapper/300:34:193
1850999240,1cyclictest0-21swapper/323:53:343
18507992421,2cyclictest14514-21sh22:44:142
1850799240,23cyclictest5996-21grep22:05:012
1850799240,23cyclictest11752-21diskmemload21:27:162
1850799240,23cyclictest11752-21diskmemload00:33:562
1850799240,23cyclictest11752-21diskmemload00:16:562
1850799240,23cyclictest0-21swapper/200:00:582
18501992423,1cyclictest13575-21ssh21:12:251
18501992423,1cyclictest11752-21diskmemload23:59:381
1850199240,23cyclictest25228-21apt-get22:54:141
1850199240,23cyclictest22585-21ssh22:20:011
1850199240,23cyclictest11752-21diskmemload21:30:201
1850199240,23cyclictest0-21swapper/123:10:441
1850199240,0cyclictest11752-21diskmemload23:19:071
18497992422,1cyclictest8739-21sh21:38:330
18497992420,3cyclictest17152-21sed22:15:000
18497992420,1cyclictest151rcu_preempt22:51:000
1849799240,2cyclictest32110-21/usr/sbin/munin23:29:220
1849799240,24cyclictest13-21ksoftirqd/021:52:240
1849799240,24cyclictest0-21swapper/021:43:170
1849799240,23cyclictest0-21swapper/021:10:100
1849799240,1cyclictest0-21swapper/021:26:070
1849799240,0cyclictest31894-21ntpq22:29:190
18509992323,0cyclictest11752-21diskmemload22:48:563
18509992323,0cyclictest11752-21diskmemload22:10:093
18509992321,1cyclictest9353-21cron21:05:013
18509992321,1cyclictest126442sleep321:11:003
18509992321,1cyclictest12612-21sh23:11:493
18509992321,1cyclictest0-21swapper/322:31:123
18509992321,1cyclictest0-21swapper/322:25:523
18509992320,2cyclictest774-21runrttasks21:52:513
18509992320,2cyclictest30644-21fschecks_time23:59:153
18509992320,2cyclictest22550-21basename20:25:013
18509992320,2cyclictest14139-21ssh00:14:153
1850999230,2cyclictest313-21systemd-journal00:18:193
1850999230,2cyclictest23452-21ssh22:21:113
1850999230,23cyclictest0-21swapper/323:23:503
1850999230,23cyclictest0-21swapper/321:38:443
1850999230,22cyclictest24461-21sh00:23:283
1850999230,22cyclictest13500-21apt-get20:09:153
1850999230,22cyclictest0-21swapper/323:00:523
1850999230,22cyclictest0-21swapper/300:37:023
1850999230,22cyclictest0-21swapper/300:04:263
1850999230,1cyclictest12889-21cron20:05:003
1850999230,0cyclictest0-21swapper/323:44:543
1850999230,0cyclictest0-21swapper/323:33:003
1850999230,0cyclictest0-21swapper/323:05:113
18507992323,0cyclictest0-21swapper/223:40:582
18507992323,0cyclictest0-21swapper/200:37:132
18507992323,0cyclictest0-21swapper/200:12:142
18507992321,1cyclictest0-21swapper/222:36:352
18507992321,1cyclictest0-21swapper/222:16:562
1850799230,23cyclictest0-21swapper/223:36:522
1850799230,23cyclictest0-21swapper/222:51:012
1850799230,23cyclictest0-21swapper/221:30:022
1850799230,22cyclictest20879-21munin-run19:15:002
1850799230,22cyclictest1928-21ssh22:31:292
1850799230,22cyclictest17440-21rm23:46:182
1850799230,22cyclictest11752-21diskmemload23:50:472
1850799230,22cyclictest0-21swapper/223:04:142
1850799230,22cyclictest0-21swapper/222:14:262
1850799230,22cyclictest0-21swapper/221:11:592
1850799230,22cyclictest0-21swapper/200:29:142
1850799230,1cyclictest606-21dbus-daemon20:40:002
1850799230,0cyclictest11752-21diskmemload23:05:072
1850799230,0cyclictest0-21swapper/222:47:362
1850799230,0cyclictest0-21swapper/222:23:342
1850799230,0cyclictest0-21swapper/221:54:452
18501992323,0cyclictest11752-21diskmemload23:28:331
18501992323,0cyclictest11752-21diskmemload00:37:001
18501992323,0cyclictest0-21swapper/123:00:421
18501992323,0cyclictest0-21swapper/122:33:241
18501992323,0cyclictest0-21swapper/121:47:311
18501992323,0cyclictest0-21swapper/121:29:041
18501992323,0cyclictest0-21swapper/121:20:281
18501992321,1cyclictest29876-21ssh22:58:131
18501992321,1cyclictest0-21swapper/123:46:501
1850199230,22cyclictest23343-21ssh21:51:201
1850199230,22cyclictest12534-21rm22:10:561
1850199230,22cyclictest11752-21diskmemload23:38:321
1850199230,22cyclictest0-21swapper/121:35:361
1850199230,1cyclictest13138-21ssh23:42:591
1850199230,0cyclictest11752-21diskmemload23:09:081
1850199230,0cyclictest0-21swapper/122:05:131
18497992323,0cyclictest11752-21diskmemload21:56:430
18497992323,0cyclictest0-21swapper/023:56:300
18497992323,0cyclictest0-21swapper/023:42:270
18497992323,0cyclictest0-21swapper/023:23:180
18497992323,0cyclictest0-21swapper/022:38:140
18497992323,0cyclictest0-21swapper/022:03:280
18497992323,0cyclictest0-21swapper/000:26:200
18497992322,1cyclictest62550irq/127-eno122:19:380
18497992322,1cyclictest62550irq/127-eno100:36:580
18497992322,1cyclictest0-21swapper/022:32:200
18497992321,2cyclictest17318-21ssh23:15:450
18497992321,1cyclictest11752-21diskmemload21:20:010
1849799231,21cyclictest0-21swapper/023:06:460
1849799230,2cyclictest2719-21ssh23:33:230
1849799230,23cyclictest29350-21sh22:57:310
1849799230,22cyclictest18082-21ssh21:46:240
1849799230,22cyclictest13931-21sh23:13:440
1849799230,22cyclictest11792-21basename21:10:010
1849799230,22cyclictest11792-21basename21:10:010
1849799230,22cyclictest0-21swapper/023:46:100
1849799230,22cyclictest0-21swapper/000:23:410
1849799230,0cyclictest0-21swapper/023:37:060
1849799230,0cyclictest0-21swapper/021:30:370
1849799230,0cyclictest0-21swapper/019:34:140
1849799230,0cyclictest0-21swapper/000:00:560
18509992222,0cyclictest0-21swapper/322:55:023
18509992222,0cyclictest0-21swapper/322:16:133
18509992222,0cyclictest0-21swapper/321:58:243
18509992221,1cyclictest0-21swapper/322:54:143
18509992221,1cyclictest0-21swapper/321:40:393
18509992220,1cyclictest0-21swapper/323:35:343
18509992220,1cyclictest0-21swapper/321:45:533
1850999220,2cyclictest23271-21latency_hist19:20:003
1850999220,22cyclictest0-21swapper/323:26:273
1850999220,21cyclictest8293-21apt-get19:59:133
1850999220,21cyclictest5982-21apt-get22:05:023
1850999220,21cyclictest32718-21apt-get20:49:143
1850999220,21cyclictest0-21swapper/321:32:003
1850999220,21cyclictest0-21swapper/321:09:133
1850999220,21cyclictest0-21swapper/321:09:133
1850999220,1cyclictest5151-21apt-get20:59:133
1850999220,1cyclictest0-21swapper/300:29:143
18507992222,0cyclictest11752-21diskmemload21:45:102
18507992222,0cyclictest0-21swapper/223:58:112
18507992222,0cyclictest0-21swapper/223:15:552
1850799222,19cyclictest20148-21munin-run20:20:012
1850799220,2cyclictest19825-21sh21:18:482
1850799220,21cyclictest7567-21apt-get21:04:132
1850799220,21cyclictest11752-21diskmemload21:37:302
1850799220,21cyclictest0-21swapper/223:31:062
1850799220,21cyclictest0-21swapper/219:19:122
1850799220,1cyclictest0-21swapper/223:10:122
1850799220,1cyclictest0-21swapper/220:00:022
1850799220,1cyclictest0-21swapper/200:08:292
1850799220,0cyclictest11752-21diskmemload22:28:462
1850799220,0cyclictest0-21swapper/222:55:492
1850799220,0cyclictest0-21swapper/222:07:202
18501992222,0cyclictest0-21swapper/122:15:381
18501992222,0cyclictest0-21swapper/122:00:131
18501992222,0cyclictest0-21swapper/100:20:251
18501992220,1cyclictest29157-21sh00:27:391
18501992219,2cyclictest2862-21turbostat19:45:011
1850199220,21cyclictest28146-21sh22:25:151
1850199220,21cyclictest23036-21sh23:21:191
1850199220,21cyclictest11752-21diskmemload00:05:121
1850199220,21cyclictest0-21swapper/123:30:061
1850199220,21cyclictest0-21swapper/122:45:231
1850199220,21cyclictest0-21swapper/120:34:141
1850199220,1cyclictest450-21ssh00:30:031
1850199220,1cyclictest28260-21sh21:55:461
1850199220,1cyclictest0-21swapper/100:10:291
1850199220,0cyclictest0-21swapper/122:36:241
18497992221,1cyclictest0-21swapper/022:22:550
18497992220,1cyclictest25515-21apt-get20:34:140
18497992220,1cyclictest17110-21sh22:45:110
1849799220,22cyclictest0-21swapper/000:16:210
1849799220,21cyclictest11752-21diskmemload23:50:460
1849799220,21cyclictest11719-21sh00:10:350
1849799220,1cyclictest739-21ssh00:30:330
1849799220,1cyclictest5313-21turbostat19:50:000
1849799220,1cyclictest12066-21ssh22:40:330
18509992120,0cyclictest0-21swapper/320:39:153
18509992120,0cyclictest0-21swapper/320:14:143
1850999210,21cyclictest0-21swapper/321:25:133
1850999210,20cyclictest0-21swapper/319:24:143
1850799210,21cyclictest0-21swapper/223:25:022
1850799210,20cyclictest0-21swapper/220:34:142
1850799210,1cyclictest7728-21munin-run19:55:002
18501992121,0cyclictest0-21swapper/119:14:141
18501992120,0cyclictest31053-21apt-get19:39:141
18501992119,1cyclictest5903-21apt-get19:54:141
18497992118,2cyclictest25682-21latency_hist19:25:010
18509992020,0cyclictest26260-21apt-get19:29:133
18509992020,0cyclictest0-21swapper/320:54:133
18507992020,0cyclictest0-21swapper/219:34:142
18507992018,1cyclictest0-21swapper/220:14:142
1850799200,20cyclictest0-21swapper/219:39:142
18501992020,0cyclictest3449-21apt-get19:49:131
18501992020,0cyclictest0-21swapper/120:00:021
18501992017,2cyclictest15305-21latency_hist20:10:001
18497992020,0cyclictest0-21swapper/021:04:140
18497992020,0cyclictest0-21swapper/020:29:130
18497992020,0cyclictest0-21swapper/019:29:140
18497992019,0cyclictest10120-21apt-get20:00:020
18497992018,1cyclictest411-21sendmail19:40:010
1849799200,0cyclictest21448-21apt-get19:19:120
18509991919,0cyclictest0-21swapper/320:44:143
18509991919,0cyclictest0-21swapper/319:34:143
18501991919,0cyclictest30325-21apt-get20:44:141
18501991919,0cyclictest0-21swapper/121:04:141
18501991919,0cyclictest0-21swapper/120:29:131
18501991918,0cyclictest28659-21apt-get19:34:141
18501991918,0cyclictest23867-21apt-get19:24:141
18501991917,1cyclictest6993-21perf21:00:001
18497991919,0cyclictest18290-21apt-get20:19:120
18497991919,0cyclictest0-21swapper/019:59:130
18497991919,0cyclictest0-21swapper/019:54:140
18509991818,0cyclictest0-21swapper/319:44:153
18507991817,1cyclictest23112-21apt-get20:29:132
18501991818,0cyclictest0-21swapper/120:54:131
18501991818,0cyclictest0-21swapper/120:19:121
18501991816,1cyclictest29754-21sendmail-msp20:40:001
18497991818,0cyclictest0-21swapper/020:09:150
18509991717,0cyclictest0-21swapper/319:49:133
18507991717,0cyclictest0-21swapper/219:49:132
18501991717,0cyclictest0-21swapper/120:49:141
18497991717,0cyclictest0-21swapper/020:59:130
18497991715,1cyclictest29746-21munin-run20:40:000
18497991715,1cyclictest2702-21apt-get20:54:130
1850999160,15cyclictest31247-21tr19:39:153
18507991614,1cyclictest15329-21perf20:10:002
18507991614,1cyclictest0-21swapper/220:54:132
18507991612,3cyclictest6988-21munin-run21:00:002
18501991616,0cyclictest0-21swapper/120:24:131
18507991513,1cyclictest606-21dbus-daemon20:25:012
18497991515,0cyclictest20721-21apt-get20:24:130
1850999140,14cyclictest0-21swapper/320:30:253
1850999140,14cyclictest0-21swapper/320:15:183
1850999140,0cyclictest0-21swapper/320:26:263
1850999140,0cyclictest0-21swapper/319:51:373
1850999140,0cyclictest0-21swapper/319:11:233
18507991414,0cyclictest0-21swapper/220:40:542
1850799140,14cyclictest0-21swapper/221:05:062
1850799140,14cyclictest0-21swapper/221:05:062
1850799140,14cyclictest0-21swapper/220:46:272
1850799140,14cyclictest0-21swapper/219:55:372
1850799140,14cyclictest0-21swapper/219:25:052
1850799140,0cyclictest0-21swapper/219:21:242
1850199145,8cyclictest793-21lldpd20:11:081
18501991414,0cyclictest0-21swapper/119:26:211
1850199140,14cyclictest0-21swapper/119:55:141
1850199140,14cyclictest0-21swapper/119:15:121
18497991414,0cyclictest0-21swapper/020:40:290
1849799140,14cyclictest0-21swapper/020:10:190
1849799140,0cyclictest0-21swapper/019:40:290
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional