You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 10:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat Sep 05, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1655321620,1sleep00-21swapper/000:25:170
241992158136,15sleep10-21swapper/119:06:251
348921500,3sleep10-21swapper/122:39:551
244482141105,25sleep00-21swapper/019:09:360
244512140103,25sleep30-21swapper/319:09:393
241332136100,25sleep20-21swapper/219:05:322
83022640,1sleep30-21swapper/323:30:253
311272640,2sleep00-21swapper/000:35:240
12662640,1sleep20-21swapper/219:30:112
292882630,2chrt0-21swapper/022:35:010
144712560,1sleep10-21swapper/100:23:571
52562550,2sleep20-21swapper/223:05:122
43502550,2sleep10-21swapper/122:15:241
187592550,4sleep118733-21apt-get21:15:121
18232550,2sleep30-21swapper/323:25:403
147692520,1sleep10-21swapper/123:11:221
80052480,1sleep00-21swapper/023:30:220
242512480,1sleep20-21swapper/200:06:242
215502480,2sleep30-21swapper/322:52:503
24755994611,21cyclictest12000-21sh21:57:303
273992430,1sleep00-21swapper/023:20:310
236882430,1sleep30-21swapper/321:42:063
197602430,2sleep00-21swapper/022:51:020
268812420,1sleep00-21swapper/019:15:010
2475599372,10cyclictest6842-21ssh21:30:123
2475599372,10cyclictest6842-21ssh21:30:113
106232370,1sleep10-21swapper/123:56:231
69132360,2chrt0-21swapper/223:30:132
24755993612,21cyclictest7376-21seq00:18:343
138172360,2chrt0-21swapper/023:59:360
45602330,2chrt0-21swapper/321:28:093
24755993311,19cyclictest1312-21sh23:50:133
2475599329,7cyclictest4776-21awk20:50:253
2475599329,5cyclictest22688-21cat22:30:013
2475599323,3cyclictest0-21swapper/323:15:123
24755993212,18cyclictest17194-21sh23:13:473
24755993211,19cyclictest20947-21turbostat.cron00:09:563
24755993211,18cyclictest1371-21sh21:50:053
2475599319,4cyclictest10456-21sh22:20:193
2475599313,6cyclictest26710-21apt-get23:45:113
2475599311,8cyclictest75-21kswapd000:35:053
24755993112,17cyclictest26487-21sh22:32:123
24755993111,18cyclictest2798-21sh22:15:103
24755993110,19cyclictest19394-21sh21:39:333
2475599309,6cyclictest8983-21cat19:45:303
2475599308,6cyclictest30330-21idleruntime22:35:163
2475599308,5cyclictest25635-21sed22:55:243
2475599308,5cyclictest25635-21sed22:55:233
2475599308,5cyclictest10926-21expr22:45:123
24755993011,16cyclictest17556-21sh23:37:593
2475599299,4cyclictest9238-1kworker/u9:100:00:133
2475599299,3cyclictest18872-21sh21:15:123
2475599298,4cyclictest1576-21uname20:45:183
2475599296,6cyclictest21415-21date23:40:243
2475599296,6cyclictest21415-21date23:40:233
2475599296,5cyclictest4128-21grep22:40:113
2475599295,6cyclictest24905-21expr19:10:113
2475599295,5cyclictest5827-21grep19:40:153
2475599291,9cyclictest23501-21ntpq20:20:203
2475599291,8cyclictest27025-21ssh21:45:123
24755992911,4cyclictest32740-21cat19:25:233
24755992911,15cyclictest11609-21diskmemload00:27:483
2475599291,10cyclictest9283-21ssh00:20:143
2475599291,10cyclictest25697-21ssh21:20:123
2475599291,10cyclictest1605-21ssh22:14:113
24755992910,3cyclictest6700-21timerwakeupswit23:05:243
2475599290,9cyclictest14844-21apt-key22:00:003
122302290,2sleep00-21swapper/021:57:450
2475599289,4cyclictest1490-21expr19:30:133
2475599287,2cyclictest0-21swapper/300:30:183
2475599285,4cyclictest0-21swapper/320:00:163
2475599281,9cyclictest22489-21/usr/sbin/munin22:05:173
2475599281,8cyclictest31489-21ssh23:00:153
2475599281,8cyclictest28765-21fschecks_time00:10:173
24755992812,9cyclictest25583-21ssh23:20:123
2475599280,5cyclictest7822-21seq23:55:063
2475299281,25cyclictest0-21swapper/019:15:490
237752280,1chrt0-21swapper/120:20:221
70842270,2chrt0-21swapper/222:17:562
268632270,2sleep00-21swapper/020:30:000
248282270,1sleep00-21swapper/021:19:470
2475599278,3cyclictest7309-21sendmail21:00:013
2475599278,3cyclictest12473-21idleruntime19:55:163
2475599277,3cyclictest20961-21gltestperf20:15:143
2475599276,4cyclictest25200-21grep20:25:153
2475599275,6cyclictest8131-21sh21:00:143
2475599275,5cyclictest9794-21sort21:05:123
2475599275,5cyclictest0-21swapper/319:50:113
2475599271,9cyclictest31005-21basename19:25:013
2475599271,8cyclictest19087-21cat20:10:213
2475599271,7cyclictest5190-21cron19:39:593
2475599271,7cyclictest29930-21ntpq20:35:193
2475599271,7cyclictest17483-21sed20:05:233
2475599271,7cyclictest12634-21grep21:10:163
24753992719,6cyclictest21085-21ssh23:15:571
253722260,14sleep228-21ksoftirqd/223:20:002
2475599261,8cyclictest31333-21cut20:40:133
2475599261,8cyclictest31333-21cut20:40:123
2475599261,8cyclictest27436-21cut20:30:153
2475599261,23cyclictest0-21swapper/319:15:043
2475399261,24cyclictest0-21swapper/119:13:591
24752992613,9cyclictest24158-21ssh22:55:120
24752992613,9cyclictest24158-21ssh22:55:110
227452260,3sleep222742-21latency_hist22:30:002
24754992522,2cyclictest0-21swapper/219:14:052
2475499251,22cyclictest0-21swapper/219:15:042
190212250,2sleep00-21swapper/021:39:110
2475299249,13cyclictest735-21cut23:25:200
263572230,9sleep228-21ksoftirqd/222:32:062
24753992313,7cyclictest16226-21pluginstate22:00:211
24753992312,8cyclictest22221-21df_inode22:05:151
2475399230,14cyclictest11609-21diskmemload21:21:301
24754992217,4cyclictest19379-21ssh22:50:392
24754992212,6cyclictest52750irq/139-enp3s0-21:20:112
2475399229,11cyclictest7451-21cat23:55:011
2475399222,12cyclictest11609-21diskmemload22:33:501
2475399221,15cyclictest29468-21sh00:35:041
24753992211,3cyclictest266602diskmemload23:45:091
2475399221,11cyclictest20906-21hddtemp_smartct21:40:151
2475299223,13cyclictest13300-21ssh23:35:110
24754992115,4cyclictest3469-21sendmail00:40:002
24754992113,6cyclictest10496-21ls19:50:172
24754992112,7cyclictest13130-21seq00:22:352
24754992112,6cyclictest23761-21pluginstate00:30:202
24753992119,1cyclictest0-21swapper/119:15:041
24753992113,6cyclictest7915-21forks21:00:131
24753992113,6cyclictest24679-21fw_packets22:55:151
24753992113,6cyclictest24679-21fw_packets22:55:141
24753992113,6cyclictest11963-21seq21:10:081
24753992111,7cyclictest21335-21ntp_kernel_err20:15:181
2475399211,12cyclictest11609-21diskmemload23:30:431
2475299218,9cyclictest29101-21apt-get19:20:130
2475299213,12cyclictest12078-21/usr/sbin/munin23:10:260
24752992112,7cyclictest17094-21sed22:25:170
2475299211,17cyclictest11609-21diskmemload21:54:330
24752992110,8cyclictest0-21swapper/021:10:200
169472210,2sleep20-21swapper/200:00:582
57502200,3chrt0-21swapper/220:55:142
263622200,1sleep00-21swapper/021:20:170
2475499209,3cyclictest118622ssh22:20:592
2475499203,15cyclictest31078-21kworker/2:223:00:152
2475499201,7cyclictest165552gltestperf00:25:172
24754992014,4cyclictest28674-21sh23:21:492
24754992014,4cyclictest28-21ksoftirqd/220:55:012
24754992013,5cyclictest14813-21gzip21:35:212
24754992012,6cyclictest23223-21if_enp3s020:20:162
24754992011,6cyclictest278-21systemd-journal22:05:002
24754992011,6cyclictest278-21systemd-journal20:30:002
24754992011,6cyclictest11029-21seq21:56:312
24754992010,7cyclictest671-21sh22:37:052
24754992010,6cyclictest2902-21grep21:50:232
2475399202,12cyclictest8224-21ssh21:55:111
2475399202,12cyclictest30892-21apt-get23:00:111
24753992014,4cyclictest22-21ksoftirqd/121:40:001
24753992013,5cyclictest6368-21awk19:40:191
24753992013,5cyclictest30603-21seq20:35:581
24753992013,5cyclictest1447-21seq22:14:011
24753992012,6cyclictest692-21seq19:27:211
24753992012,6cyclictest560-21dbus-daemon00:35:011
2475399201,11cyclictest11609-21diskmemload21:28:511
24753992010,7cyclictest6121-21cat20:55:181
2475299202,12cyclictest31448-21apt-get19:25:120
24752992012,5cyclictest30933-21sendmail20:39:590
24752992011,6cyclictest6413-21which20:55:220
24752992011,6cyclictest52850irq/140-enp3s0-22:10:110
24752992010,7cyclictest10282-21basename22:20:180
2475299200,6cyclictest21656-21ssh23:16:320
2475499192,14cyclictest11948-21apt-get23:10:042
2475499191,5cyclictest0-21swapper/221:30:132
2475499191,5cyclictest0-21swapper/221:30:122
24754991913,4cyclictest27089-21sh22:08:572
24754991913,4cyclictest2294-21sh23:26:072
24754991913,4cyclictest12862-21sh23:58:392
24754991912,5cyclictest8226-21sh22:42:552
24754991912,4cyclictest30196-21ssh22:10:252
24754991911,6cyclictest8497-21ntpq21:00:202
24754991911,6cyclictest31661-21tune2fs20:40:152
24754991911,6cyclictest31661-21tune2fs20:40:142
24754991911,6cyclictest30304-21wc19:20:242
24754991911,6cyclictest28520-21seq20:31:402
24754991911,6cyclictest28328-21unixbench_multi23:45:252
24754991911,6cyclictest21155-21ls21:40:182
24754991911,6cyclictest12286-21cat21:10:132
24754991911,5cyclictest8378-21missed_timers19:45:172
24754991910,6cyclictest24827-21seq21:19:472
24754991910,6cyclictest1272-21expr23:50:112
2475399199,7cyclictest52950irq/141-enp3s0-23:25:131
2475399198,3cyclictest123192sh22:21:271
2475399192,12cyclictest10721-21apt-get22:45:131
2475399191,7cyclictest11609-21diskmemload23:07:081
2475399191,7cyclictest11609-21diskmemload22:40:361
2475399191,7cyclictest11609-21diskmemload00:27:051
24753991914,3cyclictest4684-21unixbench_multi20:50:241
24753991914,3cyclictest31963-21missed_timers20:40:191
24753991914,3cyclictest31963-21missed_timers20:40:181
24753991914,3cyclictest30112-21sh00:10:341
24753991913,4cyclictest16648-21gltestperf20:05:161
24753991912,5cyclictest2505-21ntpq21:50:191
24753991911,6cyclictest11849-21seq19:55:021
2475399191,15cyclictest28329-21python321:45:211
2475399191,11cyclictest11609-21diskmemload22:25:401
24753991910,7cyclictest29654-21expr19:20:151
24753991910,7cyclictest23012-21cat00:05:231
2475299192,3cyclictest141rcu_preempt00:10:000
2475299191,8cyclictest20948-21turbostat00:05:000
24752991915,3cyclictest52850irq/140-enp3s0-00:11:420
2475299191,4cyclictest0-21swapper/021:31:210
2475299191,4cyclictest0-21swapper/021:31:200
24752991913,4cyclictest27203-21cstates21:45:130
24752991912,5cyclictest31153-21sed22:35:230
24752991912,5cyclictest28132-21cron22:10:000
24752991912,5cyclictest1597-21cut20:45:180
24752991911,6cyclictest23400-21seq23:42:070
24752991911,6cyclictest23400-21seq23:42:060
24752991911,6cyclictest15560-21tune2fs22:00:150
24752991911,6cyclictest13997-21sed20:00:000
24752991911,6cyclictest12355-21awk22:45:240
24752991911,4cyclictest11609-21diskmemload22:17:560
24752991910,7cyclictest1-21systemd21:10:000
24752991910,6cyclictest52850irq/140-enp3s0-21:25:120
24752991910,6cyclictest52850irq/140-enp3s0-00:30:110
24752991910,6cyclictest27497-21latency23:45:170
2475499187,8cyclictest5473-21apt-get19:40:122
2475499183,10cyclictest0-21swapper/219:55:122
2475499181,6cyclictest1073-21needreboot21:25:192
24754991813,3cyclictest22351-21sh20:20:012
24754991812,4cyclictest26526-21sh22:56:042
24754991812,4cyclictest26526-21sh22:56:032
24754991812,4cyclictest22132-21sh23:40:492
24754991812,4cyclictest22132-21sh23:40:482
24754991811,5cyclictest27927-21expr21:45:172
24754991811,5cyclictest14301-21ntpq23:35:182
24754991811,4cyclictest11609-21diskmemload22:47:572
2475499181,11cyclictest14045-21apt-get20:00:112
24754991810,6cyclictest624-21gdbus20:40:002
24754991810,6cyclictest2697-21seq20:48:422
24754991810,5cyclictest3510-21grep00:15:212
2475399182,5cyclictest11609-21diskmemload22:54:101
2475399182,4cyclictest11609-21diskmemload00:19:011
2475399182,11cyclictest9748-21apt-get19:50:111
2475399182,11cyclictest13282-21apt-get23:35:111
24753991813,3cyclictest3820-21gltestperf19:35:151
24753991813,3cyclictest10342-21if_err_enp3s021:05:171
24753991811,5cyclictest7094-21expr21:30:131
24753991811,5cyclictest7094-21expr21:30:121
24753991811,4cyclictest20911-21missed_timers23:40:191
24753991811,4cyclictest20911-21missed_timers23:40:181
24753991810,6cyclictest19049-21cat20:10:191
24753991810,5cyclictest25830-21cat23:20:141
2475299189,7cyclictest21310-21awk21:40:180
2475299189,6cyclictest8826-21switchtime19:45:290
2475299182,12cyclictest4019-21ssh22:40:120
2475299182,11cyclictest981-21apt-get23:50:110
24752991812,4cyclictest12-21ksoftirqd/023:10:010
24752991811,5cyclictest15927-21cron20:05:010
24752991811,4cyclictest6404-21sh00:17:360
24752991810,6cyclictest31691-21sed23:00:170
24752991810,6cyclictest10078-21sed00:20:190
24752991810,5cyclictest11656-21basename19:55:010
145482180,2sleep10-21swapper/100:00:051
2475499178,6cyclictest16375-21cat20:05:132
2475499176,9cyclictest18781-1kworker/u9:000:10:112
24754991712,3cyclictest9685-21/usr/sbin/munin21:05:162
24754991711,3cyclictest31324-21apt-config19:25:012
24754991710,5cyclictest3440-21expr19:35:122
2475399179,6cyclictest28713-21seq20:34:101
2475399179,6cyclictest1338-21cat19:30:111
2475399172,11cyclictest7611-21apt-get19:45:121
24753991712,3cyclictest24840-21cpuspeed_turbos20:25:131
2475299179,6cyclictest5240-21basename19:39:590
2475299172,9cyclictest16184-21apt-get20:05:110
2475299172,2cyclictest141rcu_preempt20:20:150
2475299171,11cyclictest18322-21apt-get20:10:150
24752991710,5cyclictest9638-21wc21:05:000
24752991710,5cyclictest4718-21awk20:50:240
24754991610,3cyclictest19303-21head20:10:232
2475399169,5cyclictest2529-21seq20:46:341
2475399169,5cyclictest14615-21hostname20:00:151
2475299167,6cyclictest28328-21unixbench_multi20:30:230
2475299166,7cyclictest0-21swapper/020:40:240
2475299166,7cyclictest0-21swapper/020:40:240
24752991610,4cyclictest21139-21latency20:15:160
24752991610,4cyclictest0-21swapper/019:30:130
24752991510,3cyclictest5448-21/usr/sbin/munin19:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional