You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 13:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot5s.osadl.org (updated Mon May 25, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1601992623,2cyclictest20514-21grep22:30:012
160199261,1cyclictest28-21ksoftirqd/222:02:302
1591992626,0cyclictest27266-21diskmemload21:40:460
1603992523,1cyclictest27266-21diskmemload21:53:353
1603992523,1cyclictest0-21swapper/322:30:193
160399250,24cyclictest6715-21ssh23:14:383
1601992523,1cyclictest0-21swapper/221:34:572
160199252,22cyclictest31599-21ssh22:39:402
160199250,24cyclictest27266-21diskmemload00:03:492
160199250,23cyclictest313-21systemd-journal19:20:012
160199250,23cyclictest313-21systemd-journal19:20:012
159799250,24cyclictest24150-21taskset00:27:471
159799250,0cyclictest0-21swapper/121:37:301
159199250,24cyclictest4069-21rm22:43:440
159199250,24cyclictest27266-21diskmemload23:56:170
160399242,21cyclictest17682-21ssh21:30:183
160399240,23cyclictest27266-21diskmemload23:06:283
160399240,23cyclictest27266-21diskmemload23:01:403
160399240,23cyclictest14254-21ssh23:49:463
160399240,23cyclictest0-21swapper/300:17:553
160399240,23cyclictest0-21swapper/300:10:363
160399240,1cyclictest27266-21diskmemload22:37:513
1601992423,1cyclictest0-21swapper/223:02:212
1601992422,2cyclictest0-21swapper/223:50:002
1601992421,2cyclictest18870-21diskmemload23:53:182
1601992421,2cyclictest13360-21ssh23:20:012
1601992421,2cyclictest0-21swapper/223:18:002
160199241,0cyclictest28-21ksoftirqd/223:33:322
160199240,23cyclictest27266-21diskmemload22:40:532
160199240,23cyclictest27266-21diskmemload21:26:092
160199240,23cyclictest0-21swapper/223:40:192
160199240,23cyclictest0-21swapper/221:54:342
160199240,23cyclictest0-21swapper/221:38:482
160199240,23cyclictest0-21swapper/200:26:542
1597992421,2cyclictest7658-21ssh23:43:411
159799240,24cyclictest27266-21diskmemload23:26:561
159799240,24cyclictest0-21swapper/100:39:561
159799240,23cyclictest27266-21diskmemload22:51:371
159799240,1cyclictest0-21swapper/121:14:591
159799240,0cyclictest0-21swapper/121:31:491
1591992423,1cyclictest27266-21diskmemload21:34:490
1591992422,0cyclictest151rcu_preempt00:21:410
159199241,23cyclictest27266-21diskmemload23:50:360
159199240,24cyclictest0-21swapper/022:53:220
159199240,24cyclictest0-21swapper/021:37:390
159199240,24cyclictest0-21swapper/000:00:090
159199240,23cyclictest13-21ksoftirqd/022:22:170
159199240,23cyclictest0-21swapper/022:47:170
159199240,0cyclictest0-21swapper/023:21:320
1603992323,0cyclictest27266-21diskmemload22:55:463
1603992323,0cyclictest27266-21diskmemload21:15:023
1603992323,0cyclictest0-21swapper/323:55:093
1603992323,0cyclictest0-21swapper/323:24:413
1603992323,0cyclictest0-21swapper/300:21:063
1603992321,1cyclictest0-21swapper/322:27:233
1603992321,1cyclictest0-21swapper/322:20:343
1603992321,1cyclictest0-21swapper/322:06:373
1603992321,1cyclictest0-21swapper/321:35:353
1603992320,2cyclictest32051-21ssh22:12:153
1603992320,2cyclictest20129-21diskmemload23:26:383
1603992320,2cyclictest16129-21ssh21:28:303
1603992320,2cyclictest13999-21sh21:55:333
160399230,23cyclictest27266-21diskmemload00:29:423
160399230,22cyclictest27266-21diskmemload23:30:323
160399230,1cyclictest27266-21diskmemload23:35:303
160399230,0cyclictest16888-21apt-get23:52:593
160399230,0cyclictest0-21swapper/321:24:503
160399230,0cyclictest0-21swapper/321:13:453
1601992323,0cyclictest0-21swapper/223:14:352
1601992323,0cyclictest0-21swapper/223:06:092
1601992323,0cyclictest0-21swapper/222:45:322
1601992323,0cyclictest0-21swapper/221:56:252
1601992322,1cyclictest0-21swapper/223:25:252
1601992321,1cyclictest0-21swapper/222:21:082
1601992321,1cyclictest0-21swapper/221:12:162
1601992321,1cyclictest0-21swapper/200:07:332
160199231,2cyclictest0-21swapper/200:21:072
160199230,2cyclictest5601-21sh22:17:452
160199230,2cyclictest29938-21sh00:32:542
160199230,22cyclictest5238-21sh00:38:172
160199230,22cyclictest24640-21sh23:58:272
160199230,22cyclictest23222-21awk19:55:002
160199230,22cyclictest19488-21ssh22:58:002
160199230,22cyclictest13298-21rm22:52:452
160199230,22cyclictest0-21swapper/200:10:082
160199230,0cyclictest0-21swapper/223:37:202
1597992323,0cyclictest27266-21diskmemload22:59:541
1597992323,0cyclictest0-21swapper/122:37:461
1597992323,0cyclictest0-21swapper/121:25:101
1597992323,0cyclictest0-21swapper/100:23:271
1597992323,0cyclictest0-21swapper/100:16:021
1597992323,0cyclictest0-21swapper/100:06:311
1597992321,1cyclictest0-21swapper/121:00:011
159799231,1cyclictest25106-21sh22:33:351
159799230,2cyclictest7504-21ssh21:21:531
159799230,23cyclictest27266-21diskmemload22:49:071
159799230,23cyclictest0-21swapper/123:15:401
159799230,23cyclictest0-21swapper/122:15:211
159799230,22cyclictest901-21ssh21:15:211
159799230,22cyclictest29346-21ssh23:06:031
159799230,22cyclictest23053-21ssh23:00:081
159799230,22cyclictest13655-21sh21:55:071
159799230,1cyclictest0-21swapper/123:54:021
159799230,1cyclictest0-21swapper/123:34:371
159799230,1cyclictest0-21swapper/123:10:151
159799230,1cyclictest0-21swapper/122:02:211
159799230,0cyclictest27266-21diskmemload00:00:371
159799230,0cyclictest0-21swapper/123:39:241
159799230,0cyclictest0-21swapper/121:46:451
159799230,0cyclictest0-21swapper/100:31:181
1591992323,0cyclictest1975-21ssh23:10:230
1591992323,0cyclictest141rcuc/000:33:430
1591992323,0cyclictest0-21swapper/023:25:020
1591992323,0cyclictest0-21swapper/021:50:430
1591992322,1cyclictest141rcuc/022:56:030
1591992322,1cyclictest141rcuc/021:47:080
1591992321,2cyclictest13-21ksoftirqd/021:13:000
1591992321,2cyclictest0-21swapper/023:00:160
1591992320,2cyclictest25624-21wc20:00:000
159199230,2cyclictest23213-21ssh00:26:340
159199230,23cyclictest27266-21diskmemload23:15:370
159199230,23cyclictest0-21swapper/023:45:070
159199230,23cyclictest0-21swapper/022:35:230
159199230,23cyclictest0-21swapper/022:07:250
159199230,22cyclictest7648-21ssh23:43:400
159199230,22cyclictest62550irq/127-eno121:21:560
159199230,22cyclictest30713-21sh23:07:550
159199230,22cyclictest27266-21diskmemload23:33:490
159199230,22cyclictest27266-21diskmemload00:35:030
159199230,22cyclictest27266-21diskmemload00:05:460
159199230,22cyclictest0-21swapper/022:00:240
159199230,1cyclictest62550irq/127-eno100:18:080
159199230,0cyclictest27266-21diskmemload23:36:340
159199230,0cyclictest0-21swapper/022:30:470
1603992222,0cyclictest0-21swapper/300:00:073
1603992220,1cyclictest0-21swapper/323:15:043
1603992219,2cyclictest12499-21basename20:40:003
160399220,22cyclictest0-21swapper/322:47:533
160399220,22cyclictest0-21swapper/322:42:333
160399220,22cyclictest0-21swapper/300:30:363
160399220,22cyclictest0-21swapper/300:06:503
160399220,21cyclictest8824-21latency_hist19:25:013
160399220,21cyclictest5124-21sh23:42:333
160399220,21cyclictest4124-21ssh22:15:403
160399220,21cyclictest10774-21apt-get22:50:023
160399220,21cyclictest1032-21ssh00:35:053
160399220,21cyclictest0-21swapper/321:45:033
160399220,0cyclictest5720-21apt-get20:27:593
1601992222,0cyclictest0-21swapper/222:10:332
1601992221,1cyclictest18925-21apt-get19:48:002
1601992221,1cyclictest0-21swapper/222:05:412
1601992221,0cyclictest0-21swapper/220:43:002
1601992219,2cyclictest313-21systemd-journal21:05:012
160199221,21cyclictest28-21ksoftirqd/222:30:462
160199221,20cyclictest28-21ksoftirqd/221:45:332
160199220,22cyclictest28-21ksoftirqd/220:49:422
160199220,21cyclictest27266-21diskmemload21:15:502
160199220,21cyclictest27266-21diskmemload00:15:382
160199220,1cyclictest25625-21turbostat.cron20:00:002
160199220,1cyclictest0-21swapper/221:44:122
1597992222,0cyclictest0-21swapper/122:42:131
1597992222,0cyclictest0-21swapper/122:09:281
1597992220,1cyclictest28520-21apt-get20:07:581
1597992220,1cyclictest2848-21munin-run20:20:011
1597992219,2cyclictest28009-21cron20:05:011
159799220,22cyclictest8188-21ssh21:50:201
159799220,21cyclictest8846-21munin-run19:25:011
159799220,21cyclictest5932-21rm00:11:111
159799220,21cyclictest27266-21diskmemload23:45:581
159799220,21cyclictest0-21swapper/121:03:001
159799220,1cyclictest27266-21diskmemload23:58:001
159799220,1cyclictest18705-21/usr/sbin/munin22:28:071
159799220,1cyclictest0-21swapper/123:20:261
159799220,1cyclictest0-21swapper/122:10:271
1591992222,0cyclictest0-21swapper/022:15:450
1591992222,0cyclictest0-21swapper/021:27:100
1591992222,0cyclictest0-21swapper/000:11:320
159199220,22cyclictest27266-21diskmemload22:11:210
159199220,22cyclictest0-21swapper/021:15:420
159199220,21cyclictest3325-21apt-get20:22:580
159199220,21cyclictest0-21swapper/020:33:000
159199220,21cyclictest0-21swapper/020:03:000
159199220,21cyclictest0-21swapper/019:37:580
159199220,1cyclictest15435-21sh21:57:330
1603992120,0cyclictest0-21swapper/320:07:593
1603992119,1cyclictest3970-21apt19:15:013
1603992119,1cyclictest27266-21diskmemload22:00:153
1603992119,1cyclictest15514-21apt-get20:45:023
1601992120,1cyclictest27266-21diskmemload21:20:072
1601992119,1cyclictest16527-21apt-get19:42:582
160199210,20cyclictest0-21swapper/219:12:592
1597992120,1cyclictest5720-21apt-get20:27:591
1597992119,1cyclictest15775-21apt-get20:47:591
159799210,1cyclictest30917-21apt-get20:13:001
159799210,0cyclictest4451-21apt-get19:17:581
159799210,0cyclictest4451-21apt-get19:17:581
1591992120,0cyclictest0-21swapper/020:37:590
1591992120,0cyclictest0-21swapper/019:12:580
159199210,20cyclictest21318-21apt-get19:53:000
160399200,0cyclictest889-21apt-get20:17:593
1601992019,1cyclictest0-21swapper/220:07:582
1601992019,0cyclictest0-21swapper/221:08:022
1601992017,2cyclictest2944-21grep20:20:012
1601992017,2cyclictest0-21swapper/220:58:082
160199200,20cyclictest0-21swapper/220:27:592
1597992020,0cyclictest0-21swapper/120:52:591
1597992020,0cyclictest0-21swapper/120:22:591
1597992020,0cyclictest0-21swapper/119:57:581
159799200,0cyclictest0-21swapper/119:48:001
159799200,0cyclictest0-21swapper/119:37:581
1591992020,0cyclictest0-21swapper/020:43:000
1591992020,0cyclictest0-21swapper/020:13:000
159199200,20cyclictest0-21swapper/021:08:020
1603991919,0cyclictest0-21swapper/319:57:583
1601991917,2cyclictest10057-21cron20:35:002
1601991916,2cyclictest4390-21grep20:23:042
1601991916,2cyclictest19492-21grep20:53:062
1591991919,0cyclictest0-21swapper/019:42:580
1591991918,1cyclictest13-21ksoftirqd/020:49:420
1603991818,0cyclictest0-21swapper/319:33:003
160399180,1cyclictest0-21swapper/321:00:013
1601991818,0cyclictest0-21swapper/220:37:592
1597991818,0cyclictest0-21swapper/119:42:581
1597991816,1cyclictest2055-21apt-get19:12:581
1591991818,0cyclictest0-21swapper/020:52:590
1591991818,0cyclictest0-21swapper/020:07:580
1603991717,0cyclictest0-21swapper/319:48:003
1603991716,1cyclictest12986-21apt-get20:43:003
1603991715,1cyclictest606-21dbus-daemon19:55:003
1603991715,1cyclictest16005-21cron19:40:003
1603991715,1cyclictest10089-21perf20:35:003
1601991717,0cyclictest0-21swapper/220:13:002
1601991716,1cyclictest14108-21apt-get19:37:582
1601991714,2cyclictest0-21swapper/219:23:002
160199171,12cyclictest0-21swapper/219:28:082
1591991717,0cyclictest0-21swapper/021:03:000
1591991717,0cyclictest0-21swapper/020:17:580
1591991715,1cyclictest7642-21munin-run20:30:000
159199170,16cyclictest13-21ksoftirqd/019:22:590
1603991616,0cyclictest0-21swapper/320:12:593
1603991615,1cyclictest18174-21apt-get20:52:593
1601991616,0cyclictest26129-21apt-get20:03:002
1597991616,0cyclictest0-21swapper/120:33:001
1597991615,0cyclictest25380-21apt-get21:08:021
1591991616,0cyclictest0-21swapper/019:48:000
1591991614,2cyclictest22036-21ssh20:58:080
1591991613,2cyclictest10855-21wc19:28:080
1597991515,0cyclictest10570-21apt-get20:37:591
1603991414,0cyclictest0-21swapper/320:22:333
1603991414,0cyclictest0-21swapper/319:26:273
160399140,14cyclictest0-21swapper/320:01:103
160399140,14cyclictest0-21swapper/319:15:553
160399140,14cyclictest0-21swapper/319:15:553
160399140,0cyclictest0-21swapper/321:05:163
160399140,0cyclictest0-21swapper/321:01:213
160399140,0cyclictest0-21swapper/319:40:523
1597991414,0cyclictest0-21swapper/120:40:121
1597991414,0cyclictest0-21swapper/119:50:041
159799140,14cyclictest0-21swapper/119:25:061
1591991414,0cyclictest0-21swapper/019:30:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional