You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 02:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Aug 09, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
273702186108,28sleep10-21swapper/107:09:151
273912179108,64sleep20-21swapper/207:09:322
271612175102,27sleep00-21swapper/007:06:330
273422165107,30sleep30-21swapper/307:08:533
34662620,2sleep30-21swapper/310:35:543
34662620,2sleep30-21swapper/310:35:543
239802580,2sleep30-21swapper/310:51:073
64952560,2sleep1211rcuc/111:02:341
271712500,8sleep012-21ksoftirqd/012:05:240
165282490,4sleep016523-21ssh10:45:270
224422470,2sleep30-21swapper/311:38:463
27702994410,20cyclictest29070-21sh09:44:071
19512440,2sleep20-21swapper/211:47:122
65592420,1sleep10-21swapper/108:50:141
116072420,1sleep30-21swapper/310:42:163
2770499379,7cyclictest4564-21tune2fs07:30:183
260022370,4chrt0-21swapper/010:53:090
27704993613,3cyclictest14350-21sh09:56:193
27704993612,4cyclictest14654-21diskmemload09:30:133
27704993611,6cyclictest11992-21tr11:30:243
27704993611,5cyclictest26863-21date12:30:003
27704993610,5cyclictest28185-21grep07:10:173
27702993611,22cyclictest1763-21find07:25:101
27702993611,22cyclictest1763-21find07:25:101
27704993513,4cyclictest7954-21sh12:39:503
27704993510,5cyclictest26502-21cat08:20:173
2770299350,8cyclictest15754-21ssh12:20:531
2770499347,7cyclictest278-21systemd-journal07:20:003
27704993413,3cyclictest31719-21sh09:21:423
27704993411,4cyclictest10018-21sh10:17:093
2770499339,4cyclictest16294-21processes09:10:223
2770499338,6cyclictest31430-21sed08:30:253
2770499331,30cyclictest0-21swapper/307:58:383
2770499331,23cyclictest0-21swapper/307:39:503
27704993311,3cyclictest22463-21latency_hist11:15:003
2770499325,9cyclictest0-21swapper/312:09:533
2770499323,7cyclictest20099-21ssh10:25:113
2770499321,8cyclictest18930-21ssh12:24:063
27704993211,5cyclictest24078-21uniq08:15:153
27704993211,5cyclictest24078-21uniq08:15:153
27704993211,3cyclictest23154-21sh12:02:553
27704993211,3cyclictest14654-21diskmemload11:05:143
27702993224,6cyclictest0-21swapper/108:02:291
2770299321,1cyclictest0-21swapper/107:34:011
2770299320,5cyclictest14654-21diskmemload09:32:011
2770299320,1cyclictest0-21swapper/107:24:421
2770499319,6cyclictest34-21ksoftirqd/309:05:133
2770499319,6cyclictest34-21ksoftirqd/309:05:133
2770499319,5cyclictest1862-21date08:40:023
27704993110,6cyclictest34-21ksoftirqd/308:50:153
27704993110,5cyclictest34-21ksoftirqd/308:00:203
2770499310,1cyclictest0-21swapper/308:14:003
2770299319,19cyclictest5155-21seq09:50:141
2770299318,5cyclictest9673-21awk10:40:251
2770299318,20cyclictest3200-21seq10:35:391
2770299318,20cyclictest3200-21seq10:35:391
2770299317,6cyclictest5298-21readlink11:25:241
27702993113,15cyclictest19327-21diskstats12:00:151
27702993112,16cyclictest14278-21sh11:32:181
2770299310,7cyclictest1641-21ssh09:24:011
2770299310,4cyclictest14654-21diskmemload11:23:211
266592310,4chrt0-21swapper/011:17:290
77572300,2sleep00-21swapper/009:28:210
2770499309,6cyclictest34-21ksoftirqd/311:25:133
27704993022,6cyclictest0-21swapper/307:53:083
2770299309,18cyclictest28987-21rm10:07:141
2770299309,18cyclictest28244-21sh09:20:001
2770299304,3cyclictest6573-21cstates07:35:131
27702993024,5cyclictest0-21swapper/108:08:101
2770299302,3cyclictest0-21swapper/112:05:121
2770299302,3cyclictest0-21swapper/111:10:141
27702993023,5cyclictest0-21swapper/107:41:451
27702993022,6cyclictest0-21swapper/108:13:531
27702993012,15cyclictest32140-21sh10:10:081
27702993010,17cyclictest26836-21/usr/sbin/munin10:30:161
45052290,2chrt0-21swapper/309:49:593
280642290,7sleep122-21ksoftirqd/111:42:381
2770499295,5cyclictest0-21swapper/310:05:153
27704992922,5cyclictest0-21swapper/307:47:243
27704992922,5cyclictest0-21swapper/307:43:383
2770299299,17cyclictest4739-21gltestperf08:45:171
2770299299,17cyclictest16171-21rm10:21:311
2770299297,5cyclictest8535-21cat08:55:131
2770299297,5cyclictest28354-21df08:25:151
2770299295,5cyclictest30640-21expr07:15:191
2770299291,4cyclictest13894-21/usr/sbin/munin07:50:211
27702992913,13cyclictest15452-21sh09:57:251
2770299290,5cyclictest18850-21ssh09:12:251
2770299290,5cyclictest14654-21diskmemload10:58:051
2770199291,25cyclictest1007-21ls07:20:240
27704992819,7cyclictest0-21swapper/308:08:203
2770499281,10cyclictest32453-21ssh12:10:123
2770499281,10cyclictest32453-21ssh12:10:123
2770299289,17cyclictest3979-21ssh12:35:541
2770299287,4cyclictest10987-21expr07:45:131
2770299282,4cyclictest0-21swapper/112:30:121
2770299282,23cyclictest4951-21cat11:50:011
2770299281,8cyclictest25207-21ssh11:16:011
2770299281,25cyclictest0-21swapper/108:18:001
2770299281,25cyclictest0-21swapper/108:18:001
27702992811,14cyclictest21680-21sh10:01:411
27702992810,16cyclictest15321-21cstates07:55:131
27702992810,15cyclictest20576-21fschecks_time10:25:141
2770299280,6cyclictest14654-21diskmemload12:29:571
27701992824,2cyclictest0-21swapper/007:37:490
2770199281,25cyclictest0-21swapper/007:56:120
2770199281,25cyclictest0-21swapper/007:43:420
121442280,2chrt0-21swapper/011:30:250
27704992710,3cyclictest9604-21sh12:16:323
27704992710,3cyclictest14654-21diskmemload09:38:113
2770399270,25cyclictest1393-21gpm07:23:552
2770299279,16cyclictest8313-21kworker/u8:010:16:011
2770299278,17cyclictest12435-21rm11:06:441
2770299277,4cyclictest12885-21tr09:05:141
2770299277,4cyclictest12885-21tr09:05:141
2770299274,5cyclictest27907-21seq07:10:141
2770299271,9cyclictest5377-21ssh09:25:591
2770299271,9cyclictest20716-21ssh11:37:021
2770299271,9cyclictest1259-21/usr/sbin/munin12:10:221
2770299271,9cyclictest1259-21/usr/sbin/munin12:10:221
2770299271,8cyclictest5174-21gpgconf11:50:011
2770299271,8cyclictest11598-21/usr/sbin/munin09:00:231
2770299271,7cyclictest0-21swapper/110:45:241
27702992711,14cyclictest12079-21/usr/sbin/munin11:55:241
2770199271,24cyclictest0-21swapper/008:15:350
2770199271,24cyclictest0-21swapper/008:15:350
2770199271,24cyclictest0-21swapper/008:11:000
2770199271,24cyclictest0-21swapper/008:07:030
2770199271,24cyclictest0-21swapper/007:52:150
2770199270,25cyclictest0-21swapper/008:04:100
2770499261,23cyclictest0-21swapper/307:28:193
2770499261,23cyclictest0-21swapper/307:28:193
27703992624,1cyclictest0-21swapper/207:34:122
2770399261,24cyclictest0-21swapper/208:04:532
2770399261,24cyclictest0-21swapper/207:52:222
2770399261,24cyclictest0-21swapper/207:35:412
2770399261,23cyclictest0-21swapper/207:25:032
2770399261,23cyclictest0-21swapper/207:25:032
2770299265,4cyclictest7072-21expr12:15:141
2770299265,4cyclictest30728-21grep09:45:141
2770299265,4cyclictest1183-21cat08:35:261
2770299261,7cyclictest16533-21apt-config09:35:011
2770299261,6cyclictest2408-21grep08:40:151
2770299260,8cyclictest22914-21awk10:50:241
27701992624,1cyclictest0-21swapper/008:55:340
27701992624,1cyclictest0-21swapper/007:25:530
27701992624,1cyclictest0-21swapper/007:25:520
27701992614,8cyclictest31486-21memory09:45:210
2770199260,24cyclictest0-21swapper/007:45:340
2770199260,24cyclictest0-21swapper/007:30:130
92682250,2sleep012-21ksoftirqd/012:16:120
2770499253,15cyclictest24125-21ssh11:40:133
2770499251,6cyclictest14663-21python310:20:243
2770499251,23cyclictest0-21swapper/307:23:123
27703992523,1cyclictest0-21swapper/208:06:092
27703992523,1cyclictest0-21swapper/207:45:082
27703992516,6cyclictest13777-21unixbench_multi11:55:232
2770399251,23cyclictest0-21swapper/208:26:552
2770399251,23cyclictest0-21swapper/208:14:002
2770399251,23cyclictest0-21swapper/207:55:452
2770399251,23cyclictest0-21swapper/207:40:372
2770399251,1cyclictest0-21swapper/208:15:312
2770399251,1cyclictest0-21swapper/208:15:312
2770399250,23cyclictest0-21swapper/208:21:472
2770299256,3cyclictest276392chrt08:23:291
27701992514,8cyclictest16525-21df11:10:150
236592250,2sleep20-21swapper/211:15:192
132632250,1sleep00-21swapper/011:07:330
39302240,2sleep20-21swapper/210:12:492
2770499240,22cyclictest0-21swapper/308:26:553
2770299241,6cyclictest30275-21tr08:30:141
27704992321,1cyclictest0-21swapper/308:42:413
27703992314,7cyclictest2826-21ntp_states12:35:212
27703992313,7cyclictest28208-21fw_packets07:10:172
2770399230,1cyclictest0-21swapper/208:40:432
23052230,1sleep30-21swapper/311:47:333
2770399229,11cyclictest13119-21sendmail12:20:012
2770399221,5cyclictest0-21swapper/209:30:592
27701992220,1cyclictest0-21swapper/008:24:300
27701992213,7cyclictest18718-21python311:35:240
27701992212,7cyclictest25839-21df_inode10:05:140
27701992212,6cyclictest6079-21cut09:50:220
2770199220,10cyclictest66050lldpd11:21:350
238712220,1sleep30-21swapper/309:15:403
131122220,2sleep012-21ksoftirqd/010:43:460
320712210,2sleep00-21swapper/010:33:540
291832210,1sleep20-21swapper/210:55:192
2770499211,7cyclictest3408-21chrt10:12:193
27704992113,6cyclictest444-21seq10:58:183
2770399218,10cyclictest28-21ksoftirqd/211:50:012
2770399213,13cyclictest26885-21ssh10:30:132
27703992113,6cyclictest23628-21seq10:03:362
27703992111,8cyclictest15630-21tune2fs09:10:172
27703992111,7cyclictest2989-21cat11:00:132
2770199212,14cyclictest20494-21apt-get12:25:130
27701992119,1cyclictest0-21swapper/008:37:430
27701992112,6cyclictest5804-21entropy11:50:170
2770199211,18cyclictest22558-21seq12:02:190
27701992111,6cyclictest11549-21awk09:30:260
112852210,2sleep20-21swapper/211:05:372
92102200,1sleep20-21swapper/209:29:472
70132200,2sleep20-21swapper/208:50:202
2770499206,5cyclictest29820-21cstates11:20:133
27704992013,4cyclictest18487-21sendmail12:00:003
27704992012,6cyclictest15689-21grep10:45:193
27704992012,5cyclictest6489-21timerwakeupswit09:50:253
27704992011,6cyclictest5460-21cat08:45:253
2770499201,10cyclictest86392sh11:51:573
2770399208,10cyclictest28-21ksoftirqd/212:30:002
2770399208,10cyclictest28-21ksoftirqd/209:50:002
2770399207,11cyclictest0-21swapper/209:15:172
2770399202,13cyclictest23672-21ssh09:40:152
27703992012,6cyclictest5874-21ls09:50:192
27703992011,6cyclictest2796-21cat09:25:012
27703992011,6cyclictest14783-21cut12:20:212
2770199202,13cyclictest27199-21apt-get12:30:140
2770199202,13cyclictest13342-21apt-get10:20:110
27701992015,3cyclictest31785-21munin-run07:20:000
27701992014,4cyclictest31690-21vmstat08:30:270
27701992013,4cyclictest3394-21grep12:35:250
27701992012,6cyclictest28305-21seq07:10:180
27701992012,6cyclictest27745-21seq09:19:300
27701992012,5cyclictest14654-21diskmemload09:38:110
27701992011,7cyclictest29879-21ls10:55:260
27701992011,6cyclictest12728-21seq09:05:060
27701992011,6cyclictest12728-21seq09:05:060
56812191,15sleep00-21swapper/011:01:440
2770499195,4cyclictest141rcu_preempt09:40:153
27704991912,5cyclictest4223-21grep09:25:243
27704991910,6cyclictest2627-21apt11:00:103
2770399197,10cyclictest28-21ksoftirqd/212:10:002
2770399197,10cyclictest0-21swapper/210:05:162
27703991914,3cyclictest19699-21sh12:00:182
27703991914,3cyclictest11224-21sh10:41:532
27703991913,4cyclictest6376-21sh11:26:072
27703991912,5cyclictest7517-21seq10:39:562
27703991912,5cyclictest7517-21seq10:39:562
27703991912,5cyclictest28029-21sed12:30:182
27703991912,5cyclictest21142-21awk10:50:002
27703991911,6cyclictest9051-21basename08:55:162
27703991911,6cyclictest30330-21perl07:15:162
27703991911,6cyclictest21360-21seq09:38:122
27703991911,6cyclictest12567-21cat09:55:182
27703991911,5cyclictest28-21ksoftirqd/210:20:232
27703991910,7cyclictest12205-21cat11:30:252
27703991910,6cyclictest22675-21cat11:15:002
2770199199,7cyclictest6673-21cat08:50:150
2770199199,7cyclictest15054-21cat12:20:240
2770199191,9cyclictest1258-21ssh10:35:130
2770199191,9cyclictest1258-21ssh10:35:120
27701991917,1cyclictest0-21swapper/008:26:210
27701991913,3cyclictest14654-21diskmemload10:26:240
27701991912,5cyclictest5071-21open_files11:25:230
27701991912,4cyclictest2307-21sh10:11:120
27701991911,6cyclictest7162-21sed10:15:170
27701991911,6cyclictest24762-21pluginstate09:40:240
27701991911,6cyclictest13428-21awk09:55:260
27701991910,7cyclictest1351-21sensors_temp12:10:230
27701991910,7cyclictest1351-21sensors_temp12:10:230
101852190,2sleep20-21swapper/210:17:202
2770499189,6cyclictest19398-21hwlatdetect10:00:193
27704991813,3cyclictest31164-21sh12:32:393
27704991812,4cyclictest26867-21apt-get10:30:133
27704991811,5cyclictest23658-21cut11:15:183
27703991813,3cyclictest23407-21sh10:27:012
27703991812,4cyclictest508-21sh11:22:012
27703991812,4cyclictest20219-21sh11:36:342
27703991811,5cyclictest1653-21grep12:10:262
27703991811,5cyclictest1653-21grep12:10:252
27703991811,5cyclictest12756-21basename09:05:122
27703991811,5cyclictest12756-21basename09:05:122
27703991811,4cyclictest14654-21diskmemload10:50:132
2770399180,16cyclictest14654-21diskmemload11:43:412
2770199181,8cyclictest18696-21ssh10:00:120
2770199181,4cyclictest0-21swapper/009:23:100
2770199181,15cyclictest3938-21cron08:45:000
27701991811,5cyclictest12383-21expr11:55:110
27701991811,4cyclictest14654-21diskmemload11:45:020
27701991810,6cyclictest30596-21wc11:45:020
2770199180,15cyclictest4959-21grep08:45:190
2770499179,6cyclictest8759-21expr08:55:133
27704991710,5cyclictest10908-21expr09:00:153
2770399179,6cyclictest5635-21awk08:45:262
2770399179,6cyclictest32105-21cut08:35:002
27703991712,3cyclictest30439-21munin-node08:35:122
27701991711,4cyclictest16117-21ntp_kernel_pll_09:10:200
27703991611,3cyclictest10700-21cpuspeed_turbos09:00:142
2770199168,5cyclictest11205-21idleruntime09:00:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional