You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-26 - 09:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Apr 26, 2026 00:45:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
278232236102,101sleep00-21swapper/019:06:380
261812213104,28sleep10-21swapper/119:05:071
280462176104,27sleep20-21swapper/219:09:332
277442163107,30sleep30-21swapper/319:05:373
160692640,2sleep00-21swapper/021:57:520
135122570,1sleep30-21swapper/321:31:543
155092550,2sleep30-21swapper/321:57:183
185692510,2chrt0-21swapper/021:35:240
253452500,1sleep225344-21sensors23:40:232
217222490,1sleep30-21swapper/300:01:473
159772480,1sleep20-21swapper/223:34:102
14152460,11sleep334-21ksoftirqd/323:46:503
71922450,1sleep00-21swapper/000:15:160
277402420,1sleep20-21swapper/222:55:042
161902400,2chrt0-21swapper/300:21:383
173402390,1sleep30-21swapper/323:59:223
320312320,2sleep00-21swapper/022:57:510
2835899311,28cyclictest19932-21ssh23:12:473
62772300,2chrt0-21swapper/223:02:352
2835899304,15cyclictest30135-21ssh21:44:533
2835899303,9cyclictest27119-21rm21:18:173
2835899303,9cyclictest27119-21rm21:18:163
15302300,4sleep01535-21fschecks_time22:35:170
52072280,4sleep1211rcuc/123:50:131
2835899283,8cyclictest5228-21ssh22:37:563
2835799281,3cyclictest22624-21grep23:15:132
28356992818,7cyclictest801-21chrt00:35:041
2835699281,2cyclictest24051-21ls23:15:241
2835599281,25cyclictest0-21swapper/000:35:340
210742280,1sleep20-21swapper/222:50:072
2835899275,6cyclictest15416-21diskmemload23:34:223
2835899274,20cyclictest7744-21/usr/sbin/munin00:15:223
2835899273,12cyclictest0-21swapper/323:25:053
2835899273,12cyclictest0-21swapper/323:25:053
28357992722,3cyclictest785-21ssh00:35:032
2835899264,7cyclictest34-21ksoftirqd/322:10:183
2835899264,7cyclictest16948-21ssh22:46:123
2835899262,6cyclictest29094-21processes22:55:233
2835899261,9cyclictest3685-21rm00:12:333
2835899261,9cyclictest3685-21rm00:12:333
2835899261,7cyclictest0-21swapper/322:05:193
2835699265,12cyclictest10555-21ssh23:30:121
28355992624,1cyclictest0-21swapper/023:13:090
28355992622,2cyclictest22517-21/usr/sbin/munin23:15:210
2835899259,2cyclictest141rcu_preempt21:51:463
2835899253,6cyclictest471-21cpuspeed_turbos20:35:123
2835899252,8cyclictest25765-21ssh22:53:183
2835899252,6cyclictest34-21ksoftirqd/323:18:063
2835899252,14cyclictest31170-21ssh21:20:343
28358992521,3cyclictest331rcuc/322:23:333
2835899252,11cyclictest8372-21sh21:28:283
2835899252,10cyclictest23460-21ssh00:27:123
2835899251,8cyclictest13541-21sh23:08:063
2835899251,22cyclictest0-21swapper/300:35:353
2835699259,4cyclictest22-21ksoftirqd/122:31:291
2835699252,16cyclictest11710-21ssh23:55:121
28356992519,4cyclictest474-21ssh00:35:001
28356992518,5cyclictest19548-21cron00:25:001
2835699250,24cyclictest0-21swapper/123:12:511
28355992517,6cyclictest27825-21ssh22:06:100
312712240,1sleep30-21swapper/321:45:143
2835899245,7cyclictest34-21ksoftirqd/323:50:243
2835899243,6cyclictest20274-21sh21:13:083
2835899243,6cyclictest15539-21sed21:10:003
2835899243,6cyclictest15416-21diskmemload22:42:473
2835899243,6cyclictest15416-21diskmemload22:42:473
2835899242,6cyclictest34-21ksoftirqd/322:01:313
2835899242,6cyclictest34-21ksoftirqd/322:01:313
28358992420,3cyclictest331rcuc/323:39:383
2835899241,6cyclictest28191-21ls00:30:253
28355992416,6cyclictest5007-21switchtime21:25:240
155092240,0sleep0131rcuc/021:10:000
91372230,3chrt0-21swapper/121:29:151
2835899236,10cyclictest34-21ksoftirqd/322:16:403
2835899233,6cyclictest20495-21ssh21:37:103
28358992320,2cyclictest331rcuc/319:50:193
28358992316,5cyclictest30372-21seq20:28:593
28358992316,5cyclictest30372-21seq20:28:593
28358992314,7cyclictest3786-21python323:00:253
28358992313,7cyclictest30866-21df19:15:153
28358992311,9cyclictest17930-21cat20:00:163
2835799232,14cyclictest26725-21ssh22:30:132
2835699236,4cyclictest22-21ksoftirqd/100:16:041
2835699232,14cyclictest29302-21ssh23:20:121
2835699232,14cyclictest15416-21diskmemload00:05:121
28356992313,8cyclictest1881-21seq19:21:261
28356992313,7cyclictest560-21dbus-daemon20:10:001
2835599239,12cyclictest508-21basename00:35:010
2835599231,9cyclictest66050lldpd22:24:350
177542230,3sleep00-21swapper/021:10:360
315992221,18sleep231705-21latency21:45:182
290522220,2chrt0-21swapper/300:07:123
2835899222,8cyclictest8395-21taskset19:37:383
2835899222,8cyclictest8395-21taskset19:37:383
2835899222,8cyclictest28323-21ssh23:43:013
28358992218,3cyclictest331rcuc/322:26:353
28358992213,7cyclictest24541-21expr20:15:133
28358992213,6cyclictest20056-21ls20:05:143
28358992212,8cyclictest16588-21run-parts19:55:213
2835899220,9cyclictest32738-21sh23:22:083
28357992212,6cyclictest15416-21diskmemload22:45:122
28356992214,6cyclictest12866-21latency_hist22:20:011
28356992212,6cyclictest52950irq/141-enp3s0-22:05:141
2835699221,14cyclictest24013-21ssh23:40:111
28356992210,4cyclictest210762sh22:50:061
28356992210,3cyclictest22-21ksoftirqd/122:24:281
2835599223,13cyclictest7727-21ssh22:40:120
2835599223,13cyclictest7727-21ssh22:40:120
28355992213,7cyclictest7763-21ntpq20:50:210
28355992211,3cyclictest100072missed_timers20:55:180
2835899213,6cyclictest34-21ksoftirqd/320:30:383
2835899212,6cyclictest13030-21taskset21:02:043
28358992113,6cyclictest28859-21fw_packets19:10:153
28358992112,6cyclictest278-21systemd-journal20:50:003
28358992112,6cyclictest26432-21apt20:20:113
2835899210,7cyclictest34-21ksoftirqd/322:33:333
2835799218,8cyclictest53050irq/142-enp3s0-00:25:112
2835799216,10cyclictest6589-21sh22:15:122
28357992114,4cyclictest15416-21diskmemload21:10:002
28357992113,6cyclictest30879-21diskstats19:15:162
28357992113,6cyclictest12029-21sed19:45:192
28357992113,6cyclictest10982-21sed21:30:182
28357992112,7cyclictest25839-21cut22:05:162
28357992112,6cyclictest6068-21users19:30:262
2835799210,9cyclictest160882sh21:57:532
2835699212,7cyclictest0-21swapper/121:13:371
2835699212,13cyclictest515-21irqbalance22:25:121
28356992111,8cyclictest15627-21which22:45:221
2835599212,13cyclictest17284-21ssh23:35:120
28355992115,4cyclictest12-21ksoftirqd/000:10:000
28355992113,6cyclictest11398-21ntp_states21:30:230
28355992112,7cyclictest12121-21sed21:00:190
28355992112,6cyclictest32592-21cat22:10:130
28355992111,6cyclictest52750irq/139-enp3s0-23:40:120
108922210,2sleep00-21swapper/023:54:370
2835899209,7cyclictest11213-21apt-get19:45:123
28358992012,6cyclictest7815-21awk20:50:213
28358992011,6cyclictest23174-21grep20:10:243
2835899200,7cyclictest2069-21taskset19:23:503
2835799208,10cyclictest28-21ksoftirqd/221:35:132
2835799205,13cyclictest560-21dbus-daemon22:15:012
28357992012,6cyclictest3857-21cat20:40:232
28357992012,6cyclictest25549-21cat00:05:122
28357992011,7cyclictest9558-21diskstats20:55:142
28357992010,5cyclictest3657-21apt-get23:25:042
28357992010,5cyclictest3657-21apt-get23:25:032
2835699206,10cyclictest17633-21df_inode23:35:131
2835699204,3cyclictest141rcu_preempt21:48:281
2835699201,3cyclictest22-21ksoftirqd/122:10:121
28356992011,6cyclictest7519-21sed19:35:201
28356992011,6cyclictest7519-21sed19:35:201
28356992011,6cyclictest278-21systemd-journal23:00:011
2835699201,14cyclictest18490-21ssh00:00:031
2835699201,12cyclictest15416-21diskmemload21:30:141
28356992010,8cyclictest13977-21ssh21:55:461
2835599202,15cyclictest20127-21tr22:25:130
2835599201,6cyclictest30906-21apt-get21:45:120
28355992014,4cyclictest1376-21sendmail_mailst19:20:220
28355992014,3cyclictest636-21runrttasks22:53:430
28355992013,4cyclictest2676-21cpuspeed_turbos20:40:120
28355992012,6cyclictest560-21dbus-daemon21:55:000
28355992012,6cyclictest30141-21cat21:20:180
28355992012,6cyclictest29908-21seq21:44:390
28355992012,6cyclictest16008-21seq00:21:260
28355992011,7cyclictest3929-21grep23:25:130
28355992011,7cyclictest3929-21grep23:25:130
28355992010,7cyclictest765-21df_inode20:35:130
23302200,8sleep228-21ksoftirqd/223:47:452
2835899191,7cyclictest34-21ksoftirqd/319:34:283
2835899191,7cyclictest34-21ksoftirqd/319:26:493
2835899191,6cyclictest10543-21taskset19:42:303
2835899190,8cyclictest10950-21runrttasks20:58:073
2835799196,5cyclictest6699-21timerandwakeup21:50:252
2835799195,12cyclictest13756-21kworker/2:220:00:002
2835799193,12cyclictest23967-21apt-get21:40:112
28357991913,4cyclictest6169-21sh22:38:532
28357991913,3cyclictest16954-21sh23:10:212
28357991912,5cyclictest8882-21awk22:40:232
28357991912,5cyclictest8882-21awk22:40:222
28357991912,5cyclictest32705-21seq00:34:462
28357991912,5cyclictest11951-21cat23:55:132
28357991911,6cyclictest7025-21expr19:35:142
28357991911,6cyclictest7025-21expr19:35:142
28357991911,6cyclictest28935-21sed21:20:002
28357991911,6cyclictest28935-21sed21:20:002
28357991910,6cyclictest20177-21cat22:00:262
28357991910,6cyclictest20177-21cat22:00:262
2835699199,2cyclictest22-21ksoftirqd/122:40:261
2835699199,2cyclictest22-21ksoftirqd/122:40:251
2835699198,2cyclictest22-21ksoftirqd/121:50:171
2835699198,2cyclictest22-21ksoftirqd/121:24:211
2835699195,10cyclictest23810-21ssh21:40:021
2835699192,15cyclictest1768-21cron23:00:001
2835699192,13cyclictest22584-21apt-get21:15:121
2835699192,13cyclictest22584-21apt-get21:15:121
28356991912,4cyclictest1195-21sh00:10:261
28356991912,4cyclictest1195-21sh00:10:251
28356991911,6cyclictest5072-21df_inode20:45:151
28356991911,5cyclictest16035-21expr19:55:151
28356991910,7cyclictest17848-21expr20:00:151
28356991910,6cyclictest10835-21sed23:05:261
28356991910,6cyclictest10430-21ls20:55:241
28355991914,3cyclictest27459-21processes20:20:220
28355991913,4cyclictest3300-21sh00:12:110
28355991913,4cyclictest3300-21sh00:12:100
28355991913,4cyclictest12-21ksoftirqd/023:59:590
28355991913,3cyclictest9120-21/usr/sbin/munin23:05:140
28355991912,5cyclictest8506-21seq22:15:370
28355991912,5cyclictest30833-21cat23:45:150
28355991912,5cyclictest28754-21expr20:25:130
28355991912,5cyclictest28754-21expr20:25:130
28355991912,5cyclictest16893-21unixbench_multi19:55:240
28355991911,6cyclictest31514-21seq23:20:540
28355991911,6cyclictest28833-21perl19:10:150
28355991911,6cyclictest27051-21expr22:30:140
28355991911,5cyclictest8894-21cat23:05:010
2835599191,15cyclictest19153-21fschecks_time00:00:170
28355991910,7cyclictest6818-21expr19:35:140
28355991910,7cyclictest6818-21expr19:35:140
241052190,10sleep122-21ksoftirqd/122:04:111
241052190,10sleep122-21ksoftirqd/122:04:111
28358991811,5cyclictest3381-21cat20:40:193
2835799189,7cyclictest30554-21date20:30:002
2835799189,7cyclictest30554-21date20:30:002
2835799189,6cyclictest19874-21tr20:05:142
2835799184,11cyclictest30164-21grep23:20:182
2835799183,9cyclictest32282-21ssh00:10:132
2835799183,9cyclictest32282-21ssh00:10:132
2835799182,11cyclictest9150-21apt-get23:05:132
2835799181,6cyclictest5368-21diskstats23:50:142
28357991812,4cyclictest18882-21unixbench_multi23:35:242
28357991811,5cyclictest8778-21basename20:55:002
28357991811,5cyclictest6850-21expr00:15:142
28357991811,5cyclictest639-21expr20:35:122
28357991811,5cyclictest31901-21ssh21:21:162
28357991811,5cyclictest27532-21run-parts20:20:222
28357991811,5cyclictest21409-21cat22:25:242
28357991811,5cyclictest12704-21expr21:00:252
28357991811,4cyclictest13393-21cpuspeed_turbos00:20:142
2835799181,12cyclictest13328-21ssh22:20:152
28357991810,6cyclictest5237-21awk21:25:262
2835699189,6cyclictest30650-21grep20:30:001
2835699189,6cyclictest30650-21grep20:30:001
2835699189,6cyclictest278-21systemd-journal19:45:031
2835699188,8cyclictest22-21ksoftirqd/120:40:101
2835699187,8cyclictest13526-21apt-get21:05:131
2835699186,9cyclictest30282-21cron19:15:001
2835699184,3cyclictest22-21ksoftirqd/100:25:141
2835699183,5cyclictest22-21ksoftirqd/123:47:281
2835699182,2cyclictest22-21ksoftirqd/122:36:041
2835699182,10cyclictest4566-21apt-get19:30:121
28356991813,3cyclictest14269-21ntp_states19:50:221
28356991812,4cyclictest17213-21cstates21:35:141
28356991811,5cyclictest8348-21grep20:50:251
28356991811,5cyclictest22-21ksoftirqd/123:25:011
28356991811,5cyclictest22-21ksoftirqd/123:25:011
28356991810,6cyclictest560-21dbus-daemon20:40:001
28356991810,6cyclictest4448-21date19:30:001
28356991810,6cyclictest11498-21expr21:00:141
2835599182,11cyclictest17653-21apt-get20:00:150
28355991813,3cyclictest8838-21/usr/sbin/munin19:40:220
28355991813,3cyclictest24982-21latency20:15:180
28355991813,3cyclictest21899-21acpi20:10:110
28355991813,3cyclictest18645-21/usr/sbin/munin22:00:190
28355991813,3cyclictest18645-21/usr/sbin/munin22:00:190
28355991812,4cyclictest22814-21sh00:26:330
28355991812,3cyclictest15032-21sh22:45:160
28355991811,5cyclictest32572-21cut19:20:000
28355991811,5cyclictest20759-21cut20:05:220
28355991811,5cyclictest14237-21ntpq19:50:200
28355991811,4cyclictest26552-21sh21:17:420
28355991811,4cyclictest26552-21sh21:17:420
28355991810,6cyclictest6613-21cron20:50:000
28355991810,6cyclictest5503-21awk19:30:210
28355991810,6cyclictest30971-21ls20:30:130
28355991810,6cyclictest12090-21sed23:30:260
2835799179,6cyclictest16702-21cat21:10:202
2835799179,5cyclictest29889-21grep19:10:252
2835799172,10cyclictest30721-21apt-get20:30:132
28357991711,4cyclictest24665-21cut20:15:152
28357991710,5cyclictest28-21ksoftirqd/200:00:392
28357991710,5cyclictest13533-21expr19:50:132
28357991710,5cyclictest10280-21awk19:40:252
28356991712,3cyclictest22307-21fschecks_count20:10:151
28356991710,5cyclictest30774-21expr19:15:151
28356991710,5cyclictest27564-21systemctl20:20:221
2835599179,6cyclictest11950-21cat19:45:190
28355991710,5cyclictest2690-21expr19:25:140
2835799169,5cyclictest4793-21awk20:45:142
28357991611,3cyclictest3834-21unixbench_multi19:25:262
28357991611,3cyclictest22899-21pmu-power20:10:222
28357991611,3cyclictest2172-21latency_hist19:25:002
28357991611,3cyclictest19532-21latency_hist20:05:012
28356991611,3cyclictest30775-21cpuspeed_turbos20:30:131
28356991611,3cyclictest24245-21/usr/sbin/munin20:15:151
28356991611,3cyclictest10174-21timerwakeupswit19:40:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional