You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 09:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Sun Apr 19, 2026 00:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
224899270,22cyclictest28501-21diskmemload00:35:013
2248992622,3cyclictest28501-21diskmemload21:22:073
2248992620,5cyclictest3989-21turbostat23:55:013
2248992620,5cyclictest26606-21sendmail-msp20:00:003
2248992522,3cyclictest0-21swapper/321:26:443
2248992520,4cyclictest611-21irqbalance22:51:503
2248992520,4cyclictest16322-21ssh00:05:533
2248992519,5cyclictest23583-21grep21:00:013
224899250,24cyclictest0-21swapper/323:42:593
224899250,24cyclictest0-21swapper/323:12:483
224899250,24cyclictest0-21swapper/300:35:023
224899250,20cyclictest0-21swapper/322:14:593
224899250,1cyclictest0-21swapper/322:02:303
224899250,1cyclictest0-21swapper/300:21:153
224899250,0cyclictest0-21swapper/321:40:393
2247992523,1cyclictest9421-21perf00:00:012
2247992523,1cyclictest28501-21diskmemload23:48:212
2247992523,1cyclictest10642-21latency_hist23:30:002
2246992523,1cyclictest0-21swapper/100:27:411
2248992422,1cyclictest606-21dbus-daemon20:20:013
2248992422,1cyclictest0-21swapper/300:04:123
2248992421,3cyclictest28501-21diskmemload00:11:313
2248992421,3cyclictest0-21swapper/320:14:563
2248992421,2cyclictest29285-21processes21:10:053
2248992420,3cyclictest28501-21diskmemload21:56:223
2248992420,3cyclictest28501-21diskmemload00:27:183
224899240,24cyclictest0-21swapper/322:30:173
224899240,24cyclictest0-21swapper/322:26:583
224899240,23cyclictest28501-21diskmemload00:16:073
224899240,23cyclictest2766-21idleruntime22:20:013
224899240,23cyclictest0-21swapper/322:08:283
224899240,0cyclictest1026-21ssh21:45:373
224799240,3cyclictest26169-21latency_hist23:45:012
224799240,23cyclictest22903-21ssh22:07:592
224799240,23cyclictest0-21swapper/223:07:452
224799240,23cyclictest0-21swapper/222:59:362
224799240,1cyclictest9646-21ssh22:25:522
2246992424,0cyclictest28501-21diskmemload21:53:291
2246992423,1cyclictest11222-21sh00:00:501
2246992422,1cyclictest968-21kworker/u8:000:05:041
224699242,21cyclictest606-21dbus-daemon23:00:001
2246992421,2cyclictest12721-21dump-pmu-power22:30:011
224699240,24cyclictest0-21swapper/123:00:031
224699240,23cyclictest24763-21sh21:38:081
224699240,23cyclictest14268-21perf22:00:001
224699240,1cyclictest1796-21ssh00:25:001
224699240,0cyclictest0-21swapper/122:30:441
224699240,0cyclictest0-21swapper/121:33:231
2245992423,1cyclictest0-21swapper/023:52:080
2245992422,1cyclictest9391-21cron00:00:010
2245992422,1cyclictest28501-21diskmemload21:45:010
224599240,24cyclictest0-21swapper/023:35:000
224599240,23cyclictest29127-21sh23:47:180
224599240,23cyclictest0-21swapper/022:43:550
224599240,0cyclictest0-21swapper/022:06:030
2248992323,0cyclictest0-21swapper/323:19:253
2248992320,3cyclictest28501-21diskmemload23:01:023
2248992320,3cyclictest0-21swapper/323:34:553
2248992320,3cyclictest0-21swapper/321:16:343
2248992320,2cyclictest0-21swapper/322:56:093
2248992318,4cyclictest28501-21diskmemload23:36:163
224899230,23cyclictest0-21swapper/321:53:153
224899230,22cyclictest7802-21ssh23:58:443
224899230,22cyclictest0-21swapper/323:21:513
224899230,22cyclictest0-21swapper/322:47:413
224899230,0cyclictest0-21swapper/323:47:393
224899230,0cyclictest0-21swapper/323:26:173
224899230,0cyclictest0-21swapper/322:40:143
224899230,0cyclictest0-21swapper/322:22:353
2247992323,0cyclictest0-21swapper/222:49:532
2247992323,0cyclictest0-21swapper/222:13:282
2247992321,1cyclictest5231-21awk23:25:012
2247992321,1cyclictest28501-21diskmemload21:19:332
2247992321,1cyclictest14115-21apt-get19:34:582
2247992321,1cyclictest0-21swapper/221:55:282
2247992320,2cyclictest5046-21ssh22:52:492
2247992320,2cyclictest16292-21ssh23:04:352
224799231,21cyclictest2100-21ssh23:52:452
224799230,2cyclictest28939-21sh21:41:382
224799230,22cyclictest18484-21ssh23:37:082
224799230,22cyclictest12629-21apt-get00:35:022
224799230,22cyclictest0-21swapper/222:41:452
224799230,22cyclictest0-21swapper/221:25:412
224799230,0cyclictest28501-21diskmemload23:32:362
224799230,0cyclictest28501-21diskmemload00:12:032
224799230,0cyclictest0-21swapper/200:16:052
2246992323,0cyclictest28501-21diskmemload21:17:451
2246992323,0cyclictest0-21swapper/123:58:271
2246992323,0cyclictest0-21swapper/121:26:071
2246992321,1cyclictest26587-21ssh22:42:551
2246992321,1cyclictest19389-21ssh22:35:131
2246992321,1cyclictest0-21swapper/100:20:011
2246992320,2cyclictest19777-21ssh23:07:001
224699230,23cyclictest0-21swapper/123:24:191
224699230,23cyclictest0-21swapper/123:14:091
224699230,23cyclictest0-21swapper/122:03:061
224699230,22cyclictest32562-21diskmemload22:17:041
224699230,22cyclictest31864-21ssh22:47:411
224699230,22cyclictest30822-21rm21:44:391
224699230,22cyclictest29181-21sh21:10:041
224699230,22cyclictest15282-21ssh00:36:271
224699230,22cyclictest12465-21perf19:30:011
224699230,22cyclictest11332-21apt20:35:001
224699230,1cyclictest313-21systemd-journal23:25:031
224699230,1cyclictest2637-21sh23:53:331
2245992323,0cyclictest28501-21diskmemload23:15:150
2245992323,0cyclictest0-21swapper/023:41:040
2245992323,0cyclictest0-21swapper/022:21:370
2245992323,0cyclictest0-21swapper/021:27:320
2245992323,0cyclictest0-21swapper/000:00:470
2245992321,1cyclictest29143-21apt-get00:20:010
2245992321,1cyclictest28501-21diskmemload21:20:030
2245992321,1cyclictest0-21swapper/023:36:510
2245992321,1cyclictest0-21swapper/000:27:150
2245992321,1cyclictest0-21swapper/000:15:060
224599230,23cyclictest0-21swapper/022:17:020
224599230,22cyclictest9562-21ssh22:56:540
224599230,22cyclictest8471-21rm21:54:190
224599230,22cyclictest29338-21ls20:05:000
224599230,22cyclictest28501-21diskmemload00:10:120
224599230,22cyclictest15905-21ssh00:37:230
224599230,22cyclictest15006-21sh23:02:280
224599230,22cyclictest1469-21ssh21:46:170
224599230,22cyclictest0-21swapper/023:25:080
224599230,22cyclictest0-21swapper/021:39:180
224599230,0cyclictest28501-21diskmemload22:33:590
224599230,0cyclictest28501-21diskmemload21:30:110
224599230,0cyclictest0-21swapper/023:05:050
2248992220,1cyclictest12123-21memory20:35:013
2248992220,1cyclictest0-21swapper/320:25:013
2248992219,3cyclictest0-21swapper/322:36:233
224899220,21cyclictest8251-21latency_hist20:30:003
224899220,21cyclictest21874-21sed20:55:013
224899220,21cyclictest16745-21date20:45:013
224899220,21cyclictest11314-21cron20:35:003
224899220,21cyclictest0-21swapper/319:15:023
224899220,1cyclictest25383-21ssh21:39:043
224899220,1cyclictest0-21swapper/323:06:473
224899220,0cyclictest28501-21diskmemload21:30:143
2247992222,0cyclictest0-21swapper/221:51:172
2247992221,1cyclictest28501-21diskmemload22:15:382
2247992221,1cyclictest22327-21apt-get22:39:592
2247992220,1cyclictest25443-21sh23:12:522
2247992220,1cyclictest0-21swapper/222:00:482
2247992220,1cyclictest0-21swapper/221:30:212
2247992220,1cyclictest0-21swapper/219:20:012
2247992220,1cyclictest0-21swapper/200:20:522
2247992219,2cyclictest29366-21awk20:05:012
224799220,22cyclictest28501-21diskmemload23:18:502
224799220,22cyclictest0-21swapper/222:20:332
224799220,21cyclictest313-21systemd-journal19:15:002
224799220,21cyclictest22247-21wc19:50:002
224799220,21cyclictest0-21swapper/222:31:172
224799220,21cyclictest0-21swapper/221:39:142
224799220,21cyclictest0-21swapper/221:11:252
224799220,21cyclictest0-21swapper/200:05:502
224799220,21cyclictest0-21swapper/200:01:112
224799220,1cyclictest6923-21apt-get00:29:582
224799220,1cyclictest20568-21apt-get20:54:552
224799220,1cyclictest0-21swapper/221:20:012
224799220,0cyclictest0-21swapper/200:30:532
2246992222,0cyclictest28501-21diskmemload22:11:071
2246992222,0cyclictest0-21swapper/123:40:241
2246992222,0cyclictest0-21swapper/123:17:071
2246992222,0cyclictest0-21swapper/122:09:161
2246992221,1cyclictest9689-21sh21:23:211
2246992221,1cyclictest4218-21ssh22:20:181
2246992220,1cyclictest0-21swapper/120:30:001
224699220,21cyclictest943-21apt-get20:14:561
224699220,21cyclictest25385-21apt-get21:04:591
224699220,21cyclictest12241-21ssh23:30:151
224699220,21cyclictest0-21swapper/120:04:571
224699220,1cyclictest28772-21ssh23:46:381
224699220,1cyclictest2717-21sh21:48:161
224699220,1cyclictest23155-21ssh00:13:411
224699220,1cyclictest0-21swapper/123:36:121
2245992222,0cyclictest0-21swapper/023:20:150
2245992222,0cyclictest0-21swapper/022:13:580
2245992222,0cyclictest0-21swapper/000:05:140
2245992221,1cyclictest11957-21ssh21:56:470
2245992220,1cyclictest1966-21munin-run20:15:010
2245992220,1cyclictest0-21swapper/022:50:010
224599220,21cyclictest9843-21ssh22:26:110
224599220,21cyclictest28501-21diskmemload22:46:410
224599220,21cyclictest28501-21diskmemload21:11:250
224599220,21cyclictest27777-21apt-get21:09:580
224599220,21cyclictest0-21swapper/022:02:200
224599220,21cyclictest0-21swapper/020:45:000
224599220,21cyclictest0-21swapper/000:33:390
224599220,1cyclictest0-21swapper/021:19:530
224599220,1cyclictest0-21swapper/020:40:000
224599220,1cyclictest0-21swapper/019:15:020
224899210,1cyclictest28495-21cron21:10:013
2247992120,1cyclictest26114-21apt-get19:59:592
2247992120,0cyclictest0-21swapper/220:24:562
2247992119,1cyclictest2150-21sh21:47:262
2247992119,1cyclictest0-21swapper/220:10:012
224799210,0cyclictest0-21swapper/220:49:592
2246992121,0cyclictest0-21swapper/119:44:571
2246992121,0cyclictest0-21swapper/100:30:251
2246992119,1cyclictest28530-21perf21:10:011
224699210,21cyclictest0-21swapper/119:49:571
224699210,1cyclictest4709-21munin-plugin-st19:15:001
224699210,0cyclictest0-21swapper/122:51:011
2245992119,1cyclictest4695-21apt19:15:000
2245992119,1cyclictest1-21systemd20:20:010
2248992018,1cyclictest23854-21sh19:55:003
2248992017,2cyclictest313-21systemd-journal19:35:013
2247992020,0cyclictest0-21swapper/220:30:002
2247992019,0cyclictest23719-21apt-get19:55:002
2247992018,1cyclictest11332-21apt-get20:35:022
2247992017,2cyclictest26072-21munin-plugin-st21:05:012
224799200,0cyclictest0-21swapper/219:24:582
2246992020,0cyclictest0-21swapper/120:44:581
2246992020,0cyclictest0-21swapper/120:09:571
2246992020,0cyclictest0-21swapper/119:39:581
2246992020,0cyclictest0-21swapper/119:19:581
2246992019,1cyclictest12214-21strings20:35:021
2245992020,0cyclictest8175-21apt-get20:30:000
224599200,0cyclictest0-21swapper/019:35:010
2248991918,1cyclictest18926-21apt-get19:44:573
2247991919,0cyclictest0-21swapper/220:34:572
2247991919,0cyclictest0-21swapper/219:44:572
2247991919,0cyclictest0-21swapper/219:39:582
2247991918,0cyclictest15780-21apt-get20:44:582
2246991919,0cyclictest0-21swapper/120:54:561
2245991919,0cyclictest0-21swapper/021:05:010
2245991919,0cyclictest0-21swapper/020:49:590
2245991919,0cyclictest0-21swapper/019:29:580
2245991917,1cyclictest17337-21sendmail-msp19:40:010
224599190,0cyclictest5775-21apt-get20:24:560
2246991818,0cyclictest0-21swapper/120:59:591
224599180,4cyclictest0-21swapper/020:34:570
2248991715,1cyclictest22137-21perf19:50:003
2247991714,2cyclictest1936-21cron20:15:012
2246991717,0cyclictest0-21swapper/120:20:011
2245991717,0cyclictest20568-21apt-get20:54:550
2245991717,0cyclictest0-21swapper/019:44:570
2248991615,0cyclictest0-21swapper/320:04:573
2248991613,2cyclictest11715-21apt-get19:29:583
2246991614,2cyclictest18177-21apt-get20:49:591
2245991616,0cyclictest0-21swapper/019:55:000
2245991616,0cyclictest0-21swapper/019:24:580
2245991615,0cyclictest0-21swapper/019:59:590
224899150,14cyclictest0-21swapper/319:35:073
2247991513,1cyclictest4260-21munin-run20:20:012
224799150,0cyclictest12470-21/usr/sbin/munin19:30:012
2246991513,1cyclictest0-21swapper/119:30:061
224599158,5cyclictest19982-21irqstats19:45:010
224899140,14cyclictest0-21swapper/321:00:033
224899140,14cyclictest0-21swapper/320:40:243
224899140,14cyclictest0-21swapper/319:20:333
224899140,14cyclictest0-21swapper/319:10:083
224899140,0cyclictest0-21swapper/320:05:563
2247991414,0cyclictest20534-21/usr/sbin/munin20:55:052
2247991414,0cyclictest0-21swapper/221:06:512
2246991414,0cyclictest0-21swapper/120:20:021
2246991414,0cyclictest0-21swapper/119:20:271
224699140,14cyclictest0-21swapper/119:55:211
224699140,14cyclictest0-21swapper/119:50:411
2245991412,1cyclictest29822-21ntp_states20:05:020
224599140,0cyclictest0-21swapper/020:55:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional