You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-19 - 17:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Thu Mar 19, 2026 12:44:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
956499270,26cyclictest18718-21ssh09:56:061
957399260,22cyclictest0-21swapper/310:39:373
957399260,22cyclictest0-21swapper/309:48:443
957399260,1cyclictest0-21swapper/309:54:113
9573992522,3cyclictest0-21swapper/311:02:413
9573992520,1cyclictest24751-21rm11:35:123
9573992520,1cyclictest0-21swapper/311:54:063
957399250,24cyclictest0-21swapper/311:21:433
957399250,21cyclictest0-21swapper/309:26:213
956899251,23cyclictest14149-21ssh09:20:212
9564992523,1cyclictest0-21swapper/109:47:291
956299253,1cyclictest6515-21awk12:20:010
956299253,1cyclictest6515-21awk12:20:010
956299251,23cyclictest62550irq/127-eno109:39:540
956299250,2cyclictest470-21idleruntime-cro09:05:010
956299250,24cyclictest3303-21diskmemload11:41:250
956299250,24cyclictest31375-21sh11:10:490
957399240,23cyclictest3303-21diskmemload10:25:213
957399240,23cyclictest0-21swapper/312:26:063
957399240,0cyclictest0-21swapper/312:20:153
956899240,24cyclictest14116-21ssh10:53:382
956899240,23cyclictest3303-21diskmemload12:39:592
956899240,23cyclictest1614-21ssh11:43:562
956899240,23cyclictest0-21swapper/212:16:232
956899240,23cyclictest0-21swapper/212:16:232
956899240,0cyclictest3303-21diskmemload09:38:102
9564992424,0cyclictest0-21swapper/112:38:251
956499240,24cyclictest0-21swapper/110:03:501
956499240,23cyclictest605-21avahi-daemon08:30:011
956499240,23cyclictest24994-21sh11:35:341
956499240,23cyclictest19894-21kworker/u8:211:32:221
9562992424,0cyclictest0-21swapper/009:49:010
9562992423,1cyclictest13827-21ssh09:51:310
9562992423,0cyclictest3303-21diskmemload10:01:050
9562992422,1cyclictest0-21swapper/008:17:240
956299240,23cyclictest3303-21diskmemload12:27:310
956299240,23cyclictest0-21swapper/012:32:270
956299240,0cyclictest0-21swapper/012:38:260
956299240,0cyclictest0-21swapper/010:20:090
9573992323,0cyclictest0-21swapper/311:13:233
9573992323,0cyclictest0-21swapper/310:55:213
9573992323,0cyclictest0-21swapper/310:00:103
9573992322,0cyclictest0-21swapper/312:06:303
9573992321,1cyclictest0-21swapper/310:54:353
9573992320,0cyclictest3303-21diskmemload09:20:163
957399230,2cyclictest26335-21rm11:05:593
957399230,22cyclictest3303-21diskmemload12:37:303
957399230,22cyclictest3303-21diskmemload11:31:123
957399230,22cyclictest2552-21ssh09:40:343
957399230,22cyclictest18571-21ssh12:00:113
957399230,22cyclictest17648-21ssh11:59:133
957399230,22cyclictest11928-21awk07:15:013
957399230,1cyclictest22129-21ssh10:30:243
957399230,1cyclictest0-21swapper/312:30:143
957399230,0cyclictest3303-21diskmemload11:48:163
957399230,0cyclictest3303-21diskmemload11:25:123
957399230,0cyclictest3303-21diskmemload10:44:343
957399230,0cyclictest3303-21diskmemload09:31:103
957399230,0cyclictest0-21swapper/312:11:363
957399230,0cyclictest0-21swapper/310:16:133
957399230,0cyclictest0-21swapper/309:57:063
957399230,0cyclictest0-21swapper/309:37:033
9568992323,0cyclictest0-21swapper/212:12:252
9568992323,0cyclictest0-21swapper/211:01:032
9568992323,0cyclictest0-21swapper/209:25:342
9568992323,0cyclictest0-21swapper/209:11:472
9568992321,0cyclictest0-21swapper/212:20:072
9568992320,2cyclictest15342-21sh10:55:062
9568992320,2cyclictest14525-21switchtime10:22:322
9568992318,4cyclictest0-21swapper/209:40:592
956899230,2cyclictest25290-21ssh09:31:312
956899230,2cyclictest20863-21perf08:40:002
956899230,2cyclictest18317-21ssh09:55:212
956899230,23cyclictest3303-21diskmemload09:49:192
956899230,23cyclictest0-21swapper/212:03:232
956899230,23cyclictest0-21swapper/211:38:442
956899230,22cyclictest9393-21ssh11:20:232
956899230,22cyclictest7251-21ssh11:18:032
956899230,22cyclictest622-21gdbus10:40:012
956899230,22cyclictest26670-21/usr/sbin/munin12:07:322
956899230,22cyclictest19911-21sh11:30:442
956899230,22cyclictest1913-21rm10:11:062
956899230,22cyclictest0-21swapper/211:06:462
956899230,0cyclictest3303-21diskmemload11:56:452
956899230,0cyclictest3303-21diskmemload11:51:352
956899230,0cyclictest0-21swapper/210:30:072
9564992323,0cyclictest0-21swapper/111:40:291
9564992323,0cyclictest0-21swapper/110:28:421
9564992323,0cyclictest0-21swapper/110:08:071
9564992321,1cyclictest3303-21diskmemload09:39:161
9564992321,1cyclictest27929-21ssh10:36:221
9564992321,1cyclictest0-21swapper/111:52:501
9564992321,1cyclictest0-21swapper/109:40:341
9564992320,2cyclictest6663-21grep10:47:241
9564992320,2cyclictest5949-21ssh11:47:421
9564992320,2cyclictest31106-21rm11:10:251
9564992320,2cyclictest15909-21ntp_states11:57:281
956499230,22cyclictest23833-21rm09:29:341
956499230,22cyclictest16517-21apt-get08:32:241
956499230,22cyclictest12325-21apt-get09:50:021
956499230,22cyclictest0-21swapper/111:05:071
956499230,22cyclictest0-21swapper/110:20:421
956499230,22cyclictest0-21swapper/110:10:491
956499230,22cyclictest0-21swapper/109:11:261
956499230,1cyclictest4253-21grep12:17:301
956499230,1cyclictest4253-21grep12:17:301
956499230,1cyclictest3303-21diskmemload12:13:081
956499230,1cyclictest25245-21ssh09:31:271
956499230,1cyclictest0-21swapper/110:18:521
956499230,0cyclictest0-21swapper/112:30:081
956499230,0cyclictest0-21swapper/111:28:101
956499230,0cyclictest0-21swapper/111:16:461
9562992323,0cyclictest3303-21diskmemload11:36:500
9562992323,0cyclictest0-21swapper/011:25:090
9562992323,0cyclictest0-21swapper/011:15:330
9562992323,0cyclictest0-21swapper/010:28:220
9562992323,0cyclictest0-21swapper/009:56:030
9562992322,1cyclictest62550irq/127-eno111:20:250
9562992322,1cyclictest3303-21diskmemload11:48:460
9562992322,1cyclictest0-21swapper/012:05:280
9562992321,1cyclictest0-21swapper/009:31:490
956299230,23cyclictest0-21swapper/012:21:500
956299230,23cyclictest0-21swapper/012:11:450
956299230,23cyclictest0-21swapper/012:02:470
956299230,23cyclictest0-21swapper/011:58:380
956299230,23cyclictest0-21swapper/011:33:300
956299230,23cyclictest0-21swapper/010:54:470
956299230,23cyclictest0-21swapper/010:12:250
956299230,22cyclictest30544-21perf09:00:010
956299230,22cyclictest27855-21ssh10:36:150
956299230,22cyclictest0-21swapper/010:30:360
956299230,22cyclictest0-21swapper/010:17:280
956299230,0cyclictest3303-21diskmemload10:55:130
956299230,0cyclictest0-21swapper/011:51:190
956299230,0cyclictest0-21swapper/009:26:080
9573992222,0cyclictest0-21swapper/311:40:113
9573992219,2cyclictest1579-21sh12:15:023
9573992219,2cyclictest1579-21sh12:15:023
957399220,2cyclictest8817-21sh09:15:093
957399220,22cyclictest0-21swapper/309:11:213
957399220,21cyclictest28623-21rm10:05:173
957399220,21cyclictest0-21swapper/310:45:463
957399220,21cyclictest0-21swapper/310:11:503
957399220,0cyclictest0-21swapper/310:20:503
9568992222,0cyclictest15804-21apt-get11:27:232
9568992222,0cyclictest0-21swapper/210:41:232
9568992221,1cyclictest8894-21ssh09:15:162
9568992221,1cyclictest23586-21sh10:00:252
9568992220,1cyclictest12959-21dpkg09:50:022
956899220,2cyclictest6532-21sh10:15:192
956899220,22cyclictest0-21swapper/210:26:422
956899220,22cyclictest0-21swapper/210:06:592
956899220,21cyclictest5649-21ssh10:46:082
956899220,21cyclictest2792-21ssh11:45:262
956899220,21cyclictest0-21swapper/211:10:052
956899220,1cyclictest3316-21ssh09:10:012
956899220,0cyclictest0-21swapper/212:31:132
956899220,0cyclictest0-21swapper/212:25:222
9564992222,0cyclictest0-21swapper/112:22:161
9564992222,0cyclictest0-21swapper/112:00:181
9564992222,0cyclictest0-21swapper/110:56:571
9564992221,1cyclictest0-21swapper/108:15:011
9564992220,2cyclictest12196-21fschecks_time10:52:251
9564992220,1cyclictest30984-21apt-get09:02:231
956499220,22cyclictest0-21swapper/112:26:201
956499220,22cyclictest0-21swapper/112:06:101
956499220,21cyclictest3303-21diskmemload09:20:461
956499220,21cyclictest25732-21awk08:50:001
956499220,21cyclictest1351-21apt-get09:07:241
956499220,21cyclictest0-21swapper/107:57:231
956499220,1cyclictest24097-21idleruntime-cro07:40:011
956499220,1cyclictest11208-21ssh09:17:281
956499220,0cyclictest3303-21diskmemload11:01:341
956499220,0cyclictest3303-21diskmemload10:40:551
9562992222,0cyclictest0-21swapper/010:40:110
9562992222,0cyclictest0-21swapper/010:05:340
9562992221,1cyclictest0-21swapper/008:42:230
9562992220,2cyclictest2378-21sh09:40:090
9562992220,1cyclictest3647-21cron08:05:000
9562992220,1cyclictest14347-21sendmail-msp07:20:010
956299220,21cyclictest3303-21diskmemload11:05:180
956299220,21cyclictest3303-21diskmemload09:20:280
956299220,21cyclictest0-21swapper/009:10:050
956299220,1cyclictest18979-21kworker/u8:211:00:270
956299220,0cyclictest3303-21diskmemload10:46:310
9573992120,1cyclictest29318-21apt-get07:52:233
9573992120,1cyclictest14768-21apt-get07:22:223
9573992120,1cyclictest14768-21apt-get07:22:223
957399210,20cyclictest0-21swapper/311:15:513
957399210,20cyclictest0-21swapper/308:17:233
957399210,20cyclictest0-21swapper/308:07:233
957399210,1cyclictest30556-21awk09:00:013
957399210,0cyclictest0-21swapper/308:27:243
9568992121,0cyclictest0-21swapper/209:02:232
9568992121,0cyclictest0-21swapper/208:07:232
956899210,20cyclictest9302-21apt-get08:17:242
956899210,20cyclictest0-21swapper/207:17:242
956899210,0cyclictest9955-21apt-get07:12:222
956899210,0cyclictest23779-21apt-get08:47:262
956899210,0cyclictest21382-21apt-get08:42:232
956899210,0cyclictest0-21swapper/208:32:242
9564992121,0cyclictest0-21swapper/107:52:231
9564992119,1cyclictest8498-21apt-get11:20:011
956499210,20cyclictest0-21swapper/108:07:231
9562992121,0cyclictest3303-21diskmemload09:15:070
9562992121,0cyclictest0-21swapper/008:22:230
9562992121,0cyclictest0-21swapper/007:37:250
9562992120,0cyclictest0-21swapper/007:22:220
9562992120,0cyclictest0-21swapper/007:22:220
956299211,18cyclictest0-21swapper/007:57:230
956299210,0cyclictest0-21swapper/008:27:240
9573992020,0cyclictest0-21swapper/307:37:253
9573992020,0cyclictest0-21swapper/307:27:243
9573992019,0cyclictest30984-21apt-get09:02:233
9573992018,1cyclictest6904-21apt-get08:12:233
957399200,20cyclictest0-21swapper/308:42:233
957399200,20cyclictest0-21swapper/308:22:243
9568992020,0cyclictest0-21swapper/208:52:242
9568992020,0cyclictest0-21swapper/208:02:242
9568992019,1cyclictest30536-21ls09:00:012
956899200,20cyclictest0-21swapper/207:27:242
9564992020,0cyclictest0-21swapper/107:32:231
9564992020,0cyclictest0-21swapper/107:27:241
956499200,0cyclictest0-21swapper/108:42:231
9562992020,0cyclictest0-21swapper/008:10:020
9562992019,1cyclictest19559-21apt-get07:32:230
9562992019,0cyclictest9955-21apt-get07:12:230
9573991919,0cyclictest0-21swapper/307:17:253
9568991919,0cyclictest31715-21apt-get07:57:232
9564991917,1cyclictest28896-21munin-plugin-st07:50:001
9564991917,1cyclictest24531-21apt-get07:42:231
9562991919,0cyclictest0-21swapper/008:47:260
9562991919,0cyclictest0-21swapper/008:37:240
9562991919,0cyclictest0-21swapper/008:32:240
9573991818,0cyclictest0-21swapper/308:52:243
9573991818,0cyclictest0-21swapper/308:47:263
9573991818,0cyclictest0-21swapper/307:57:243
9573991817,0cyclictest0-21swapper/307:42:233
9568991818,0cyclictest0-21swapper/207:52:232
9564991818,0cyclictest0-21swapper/108:02:241
9562991817,0cyclictest26173-21apt-get08:52:240
9573991717,0cyclictest0-21swapper/308:32:243
9573991717,0cyclictest0-21swapper/307:47:233
9568991717,0cyclictest11725-21apt-get08:22:232
9573991616,0cyclictest0-21swapper/309:07:243
9573991616,0cyclictest0-21swapper/307:32:233
9564991614,1cyclictest28125-21dump-pmu-power08:55:001
9562991614,1cyclictest4112-21apt-get08:07:230
9562991614,1cyclictest3310-21grep09:10:010
9568991515,0cyclictest26925-21apt-get07:47:232
956899150,14cyclictest0-21swapper/207:37:282
9564991515,0cyclictest0-21swapper/107:17:251
9564991515,0cyclictest0-21swapper/107:12:231
9564991514,0cyclictest0-21swapper/108:22:231
9573991414,0cyclictest1716-21apt-get08:02:243
957399140,14cyclictest0-21swapper/308:36:543
956899140,14cyclictest0-21swapper/208:25:502
956899140,14cyclictest0-21swapper/208:11:052
956899140,14cyclictest0-21swapper/207:30:132
956899140,14cyclictest0-21swapper/207:20:212
956899140,14cyclictest0-21swapper/207:20:212
956499140,14cyclictest0-21swapper/108:55:521
956499140,0cyclictest0-21swapper/108:35:061
956499140,0cyclictest0-21swapper/107:20:191
956499140,0cyclictest0-21swapper/107:20:191
9562991414,0cyclictest0-21swapper/007:50:160
9562991414,0cyclictest0-21swapper/007:25:180
956299140,14cyclictest0-21swapper/007:41:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional