You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-05 - 16:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Sun Apr 05, 2026 12:43:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
234975995745,5cyclictest0-21swapper/210:00:252
234975995240,9cyclictest345240-21idleruntime-cro10:40:012
234973995243,6cyclictest291494-21systemd-run09:35:010
234973995134,10cyclictest12350irq/32-eth010:25:360
23497599503,41cyclictest387316-21ssh11:30:362
23497599503,41cyclictest387316-21ssh11:30:362
23497399481,44cyclictest300866-21ssh09:45:320
234975994741,3cyclictest0-21swapper/209:45:372
234974994739,5cyclictest434909-21latency_hist12:40:021
234974994739,5cyclictest434909-21latency_hist12:40:021
234975994637,6cyclictest0-21swapper/209:40:012
234975994637,6cyclictest0-21swapper/209:40:012
234975994635,7cyclictest0-21swapper/209:41:202
234974994637,6cyclictest0-21swapper/110:40:011
234974994636,7cyclictest409649-21unin-run12:00:011
23497399464,36cyclictest399212-21ssh11:45:340
234973994627,14cyclictest12350irq/32-eth007:10:280
234975994525,13cyclictest421-21dbus-daemon08:45:322
234975994525,13cyclictest421-21dbus-daemon08:45:322
234975994432,6cyclictest0-21swapper/212:00:332
23497599441,7cyclictest270956-21diskmemload09:10:312
234974994337,3cyclictest0-21swapper/110:40:341
234975994234,5cyclictest1-21systemd11:20:022
23497599423,29cyclictest383269-21ssh11:25:352
234974994235,4cyclictest377984-21munin-plugin-st11:20:011
234973994236,4cyclictest391694-21grep11:35:460
234973994234,5cyclictest266469-21systemd-run08:55:010
234973994234,5cyclictest266469-21systemd-run08:55:010
234975994133,5cyclictest279197-21readlink09:20:012
234975994133,5cyclictest269484-21systemd-run09:05:012
23497599412,29cyclictest395300-21sh11:40:362
234974994133,5cyclictest288338-21missed_timers09:30:281
23497499413,32cyclictest399121-21ssh11:45:311
234974994131,6cyclictest249986-21systemd-run08:00:011
234974994129,9cyclictest296663-21needreboot09:40:301
234975994031,5cyclictest405794-21idleruntime-cro11:55:012
234975994031,5cyclictest358767-21ntp_kernel_pll_10:55:332
234974994032,5cyclictest413341-21sh12:05:011
234974994032,5cyclictest324576-21cron10:15:011
234974994031,5cyclictest292267-21if_eth009:35:251
234974994031,5cyclictest292267-21if_eth009:35:251
234973994034,4cyclictest1056-21mta-sts-daemon11:15:360
234973994023,11cyclictest12350irq/32-eth012:20:330
234973994023,11cyclictest12350irq/32-eth012:20:330
234975993932,4cyclictest0-21swapper/212:00:012
234975993931,4cyclictest0-21swapper/210:00:012
234975993930,5cyclictest1-21systemd08:20:012
234975993927,9cyclictest280976-21unixbench_singl09:20:432
234974993933,3cyclictest0-21swapper/112:20:011
234974993929,6cyclictest252979-21systemd-run08:10:011
23497399395,28cyclictest414408-21ssh12:05:320
234973993934,2cyclictest0-21swapper/009:20:310
234973993934,2cyclictest0-21swapper/009:20:010
234973993932,5cyclictest3552651-21cups-browsed11:20:330
234973993931,6cyclictest421-21dbus-daemon08:15:010
234973993927,9cyclictest332862-21grep10:25:000
234975993830,5cyclictest333455-21aten_r2power_vo10:25:242
234975993830,5cyclictest245490-21systemd-run07:45:022
234975993828,6cyclictest421-21dbus-daemon10:21:082
234974993830,5cyclictest369891-21munin-plugin-st11:10:001
234974993830,5cyclictest1056-21mta-sts-daemon11:03:531
234974993830,3cyclictest0-21swapper/110:00:331
234974993826,9cyclictest394422-21acpi11:40:191
234973993832,3cyclictest0-21swapper/009:10:330
234973993831,4cyclictest347083-21vmstat10:40:450
234973993830,5cyclictest406821-21needreboot11:55:280
23497399382,29cyclictest325666-21sshd10:15:280
23497399382,29cyclictest325666-21sshd10:15:280
234975993728,6cyclictest420-21avahi-daemon09:30:182
234975993728,6cyclictest357581-21munin-plugin-st10:55:012
234975993726,7cyclictest284332-21/usr/sbin/munin09:25:282
234975993725,9cyclictest390951-21latency11:35:302
23497599372,28cyclictest338230-21ssh10:30:362
23497599372,27cyclictest317477-21ssh10:05:332
23497499373,28cyclictest317548-21ssh10:05:361
234974993730,5cyclictest0-21swapper/107:20:001
234974993729,5cyclictest394007-21munin-run11:40:001
234974993729,5cyclictest275629-21aten_r2power_vo09:15:201
234973993731,3cyclictest12450irq/33-eth009:55:330
234973993730,4cyclictest363461-21sed11:00:420
234973993728,6cyclictest0-21swapper/012:05:010
234975993628,3cyclictest0-21swapper/212:25:322
234975993624,9cyclictest346384-21ntp_kernel_pll_10:40:332
234975993620,4cyclictest171rcu_preempt07:55:322
234974993629,4cyclictest1056-21mta-sts-daemon11:20:031
234974993629,3cyclictest0-21swapper/112:25:311
234974993628,4cyclictest1056-21mta-sts-daemon09:25:161
234974993625,5cyclictest0-21swapper/110:20:311
234974993624,9cyclictest424111-21cpu12:20:221
234974993624,9cyclictest424111-21cpu12:20:221
234974993624,9cyclictest416885-21sh12:10:011
23497499361,31cyclictest350584-21ssh10:45:361
234973993631,3cyclictest0-21swapper/009:25:330
234973993630,4cyclictest420859-21fschecks_count12:15:220
234973993630,4cyclictest239475-21/usr/sbin/munin07:25:020
234973993626,7cyclictest383593-21timerandwakeup11:25:410
234973993620,10cyclictest12350irq/32-eth010:55:300
234975993528,4cyclictest243886-21unixbench_singl07:35:422
234975993527,5cyclictest351127-21unixbench_multi10:45:452
234975993527,5cyclictest257464-21cron08:25:012
234975993526,6cyclictest421500-21mailstats12:15:362
234975993525,8cyclictest365874-21sh11:05:012
234974993527,5cyclictest431417-21ssh12:35:011
234974993527,5cyclictest431417-21ssh12:35:011
234974993517,13cyclictest235805-21sshd07:10:281
234973993529,4cyclictest262491-21fschecks_time08:40:240
234973993528,4cyclictest252913-21unixbench_singl08:05:440
234973993526,7cyclictest353467-21awk10:50:010
234973993512,14cyclictest151ktimers/012:25:370
234975993427,4cyclictest399686-21unixbench-2d11:45:422
234975993427,3cyclictest0-21swapper/211:15:012
234975993426,6cyclictest234972-21cyclictest11:05:442
234975993425,5cyclictest305203-21proc_pri09:50:382
234974993428,4cyclictest243998-21kworker/u8:13-rpciod08:05:011
234974993428,3cyclictest0-21swapper/110:25:321
234974993427,4cyclictest0-21swapper/109:45:331
234974993425,6cyclictest269479-21true09:05:011
234973993426,5cyclictest3552651-21cups-browsed11:05:310
23497399342,25cyclictest371061-21ssh11:10:330
234973993413,16cyclictest151ktimers/009:05:350
234973993413,16cyclictest151ktimers/009:05:350
23497399341,28cyclictest342346-21ssh10:35:340
234975993327,3cyclictest0-21swapper/212:35:192
234975993327,3cyclictest0-21swapper/212:35:192
234975993326,3cyclictest0-21swapper/212:20:392
234975993326,3cyclictest0-21swapper/212:20:392
234974993327,3cyclictest0-21swapper/109:00:001
234974993327,3cyclictest0-21swapper/109:00:001
234973993327,4cyclictest255971-21awk08:20:010
234973993325,4cyclictest0-21swapper/012:10:250
23497399331,30cyclictest312870-21diskstats10:00:240
234975993222,6cyclictest3552651-21cups-browsed10:15:492
234975993222,6cyclictest3552651-21cups-browsed10:15:492
234975993220,3cyclictest171rcu_preempt12:10:302
234974993226,4cyclictest234972-21cyclictest07:30:241
234974993226,4cyclictest0-21swapper/108:15:281
234974993225,4cyclictest247419-21cpuspeed07:50:221
234974993224,6cyclictest0-21swapper/110:00:011
234974993222,6cyclictest271409-21aten2.4-expect09:10:211
23497399324,24cyclictest321769-21rm10:10:380
234973993217,13cyclictest394997-21missed_timers11:40:270
23497399321,21cyclictest292737-21ssh09:35:360
23497399321,21cyclictest292737-21ssh09:35:360
23497599314,19cyclictest430226-21ssh12:30:362
23497599314,19cyclictest430226-21ssh12:30:362
234975993125,3cyclictest0-21swapper/212:05:302
234975993123,4cyclictest229729-21kworker/u8:20-xprtiod07:24:232
234975993115,3cyclictest171rcu_preempt08:10:392
234975993115,12cyclictest351ktimers/208:25:302
234974993123,6cyclictest270956-21diskmemload11:30:391
234974993123,6cyclictest270956-21diskmemload11:30:391
234974993123,6cyclictest0-21swapper/110:30:291
234974993123,5cyclictest258958-21sh08:30:011
234974993122,5cyclictest358116-21aten2.4-expect10:55:211
23497399312,25cyclictest3552651-21cups-browsed08:25:310
234973993121,3cyclictest151ktimers/009:40:360
234973993120,8cyclictest386883-21kernelversion11:30:260
234973993120,8cyclictest386883-21kernelversion11:30:260
234973993119,10cyclictest151ktimers/007:50:360
234973993113,13cyclictest12350irq/32-eth010:05:340
234975993025,2cyclictest0-21swapper/208:40:312
234975993023,4cyclictest378975-21missed_timers11:20:262
23497599301,5cyclictest0-21swapper/208:30:312
23497499304,18cyclictest428-21mta-sts-daemon08:20:371
23497499303,20cyclictest418004-21ssh12:10:331
234974993025,2cyclictest0-21swapper/110:55:011
234974993024,3cyclictest0-21swapper/110:15:281
234974993024,3cyclictest0-21swapper/110:15:281
234974993023,5cyclictest243946-21cron07:40:001
234974993023,4cyclictest239126-21switchtime07:20:411
234974993021,7cyclictest306632-21sh09:52:301
23497399303,22cyclictest304969-21ssh09:50:300
234973993023,5cyclictest354536-21/usr/sbin/munin10:50:280
234973993021,6cyclictest3552651-21cups-browsed10:32:520
234975992922,3cyclictest0-21swapper/210:10:312
234975992920,6cyclictest234995-21latency_hist07:10:022
234975992920,6cyclictest234995-21latency_hist07:10:022
23497499295,20cyclictest3552651-21cups-browsed08:39:291
234974992923,3cyclictest0-21swapper/111:25:431
234974992922,4cyclictest402954-21needreboot11:50:291
234974992921,5cyclictest0-21swapper/108:35:011
23497399292,23cyclictest430144-21ssh12:30:320
23497399292,23cyclictest430144-21ssh12:30:320
234975992823,2cyclictest0-21swapper/207:20:002
234975992820,5cyclictest0-21swapper/207:10:342
234975992820,4cyclictest0-21swapper/209:05:312
234975992820,4cyclictest0-21swapper/209:05:312
234974992821,3cyclictest0-21swapper/109:20:421
234973992820,5cyclictest403108-21ssh11:50:340
234973992815,9cyclictest12350irq/32-eth007:40:360
234975992721,3cyclictest0-21swapper/207:35:002
234975992719,5cyclictest266463-21systemd-run08:55:012
234975992719,5cyclictest266463-21systemd-run08:55:012
234975992713,3cyclictest171rcu_preempt07:50:322
234974992721,3cyclictest0-21swapper/111:10:211
234974992720,4cyclictest0-21swapper/108:50:331
234974992720,4cyclictest0-21swapper/108:50:331
234974992720,3cyclictest0-21swapper/108:40:371
234974992719,5cyclictest270638-21switchtime09:05:411
234974992719,5cyclictest270638-21switchtime09:05:411
23497399272,19cyclictest432467-21ssh12:35:310
23497399272,19cyclictest432467-21ssh12:35:310
234973992719,5cyclictest229729-21kworker/u8:20+xprtiod08:05:010
234975992615,6cyclictest3552651-21cups-browsed07:25:322
234975992615,6cyclictest3552651-21cups-browsed07:25:322
234975992613,6cyclictest171rcu_preempt08:05:312
234974992618,5cyclictest240900-21unixbench_singl07:25:431
234974992618,5cyclictest240900-21unixbench_singl07:25:431
234973992619,5cyclictest3552651-21cups-browsed08:21:400
234973992617,6cyclictest241115-21latency_hist07:30:020
234973992617,6cyclictest241115-21latency_hist07:30:020
234973992613,9cyclictest0-21swapper/007:10:020
234973992613,9cyclictest0-21swapper/007:10:020
23497599252,18cyclictest250814-21modprobe08:00:292
234975992519,3cyclictest0-21swapper/208:40:012
234975992515,3cyclictest171rcu_preempt07:50:012
234973992520,3cyclictest0-21swapper/007:59:290
234973992518,4cyclictest242475-21systemd-run07:35:010
234974992416,6cyclictest245760-21aten_r2power_en07:45:201
234973992418,4cyclictest3552651-21cups-browsed08:39:180
234973992418,4cyclictest151ktimers/009:00:330
234973992416,5cyclictest0-21swapper/008:45:370
234973992416,5cyclictest0-21swapper/008:45:370
234974992317,3cyclictest0-21swapper/107:45:021
234973992317,3cyclictest151ktimers/008:30:340
234973992317,3cyclictest12450irq/33-eth007:35:320
234973992317,3cyclictest0-21swapper/007:20:000
23497399231,17cyclictest753-21ntpd07:45:250
234974992217,2cyclictest0-21swapper/108:10:291
234975992115,3cyclictest0-21swapper/209:00:002
234975992115,3cyclictest0-21swapper/209:00:002
234974992115,3cyclictest0-21swapper/108:45:371
234974992115,3cyclictest0-21swapper/108:45:371
234973992115,3cyclictest249993-21kworker/u8:6-xprtiod09:00:010
234973992115,3cyclictest249993-21kworker/u8:6-xprtiod09:00:010
23497699194,9cyclictest0-21swapper/310:15:353
23497699194,9cyclictest0-21swapper/310:15:353
23497699192,12cyclictest0-21swapper/309:25:303
23497699192,11cyclictest0-21swapper/310:20:343
23497699183,9cyclictest0-21swapper/310:00:233
23497699183,8cyclictest0-21swapper/310:05:343
23497699182,9cyclictest0-21swapper/311:20:363
23497699182,9cyclictest0-21swapper/309:20:203
23497699181,10cyclictest0-21swapper/310:50:303
234974991811,4cyclictest234928-21/usr/sbin/munin07:10:011
234974991811,4cyclictest234928-21/usr/sbin/munin07:10:011
23497699174,7cyclictest0-21swapper/309:15:283
23497699173,9cyclictest0-21swapper/312:00:333
23497699173,9cyclictest0-21swapper/310:40:323
23497699172,9cyclictest0-21swapper/312:15:273
23497699172,8cyclictest0-21swapper/311:30:293
23497699172,8cyclictest0-21swapper/311:30:293
23497699172,8cyclictest0-21swapper/309:10:393
23497699171,8cyclictest0-21swapper/310:55:273
23497699171,10cyclictest0-21swapper/309:40:363
23497699163,8cyclictest0-21swapper/310:35:293
23497699163,6cyclictest0-21swapper/311:05:213
23497699162,9cyclictest0-21swapper/311:25:233
23497699162,9cyclictest0-21swapper/310:25:283
23497699162,9cyclictest0-21swapper/309:30:313
23497699162,8cyclictest0-21swapper/312:35:313
23497699162,8cyclictest0-21swapper/312:35:313
23497699162,8cyclictest0-21swapper/311:50:343
23497699162,8cyclictest0-21swapper/311:35:363
23497699162,8cyclictest0-21swapper/311:15:303
23497699162,8cyclictest0-21swapper/310:45:283
23497699162,8cyclictest0-21swapper/308:30:283
23497699162,6cyclictest0-21swapper/307:55:283
23497699161,9cyclictest0-21swapper/311:00:343
23497699161,8cyclictest0-21swapper/311:40:293
23497699153,9cyclictest0-21swapper/309:55:253
23497699152,8cyclictest0-21swapper/312:35:013
23497699152,8cyclictest0-21swapper/312:35:013
23497699152,8cyclictest0-21swapper/312:10:263
23497699152,8cyclictest0-21swapper/311:10:313
23497699152,6cyclictest0-21swapper/308:35:293
23497699152,5cyclictest0-21swapper/309:00:293
23497699151,9cyclictest0-21swapper/309:50:363
23497699151,9cyclictest0-21swapper/309:45:283
23497699151,9cyclictest0-21swapper/309:40:023
23497699151,9cyclictest0-21swapper/309:40:023
23497699151,8cyclictest0-21swapper/312:25:273
23497699151,8cyclictest0-21swapper/311:45:303
23497699151,8cyclictest0-21swapper/309:10:023
23497699151,8cyclictest0-21swapper/309:10:023
23497699151,7cyclictest0-21swapper/307:50:303
23497699151,6cyclictest0-21swapper/308:00:293
23497699150,7cyclictest0-21swapper/307:30:293
23497699142,8cyclictest0-21swapper/312:05:313
23497699142,8cyclictest0-21swapper/310:30:363
23497699142,5cyclictest0-21swapper/308:05:353
23497699141,8cyclictest0-21swapper/310:10:313
23497699141,7cyclictest0-21swapper/308:15:333
23497699133,6cyclictest0-21swapper/308:55:373
23497699133,6cyclictest0-21swapper/308:55:373
23497699132,5cyclictest0-21swapper/307:10:293
23497699131,8cyclictest0-21swapper/311:55:203
23497699131,7cyclictest0-21swapper/308:25:293
23497699131,6cyclictest0-21swapper/308:40:343
23497699122,5cyclictest0-21swapper/308:50:293
23497699122,5cyclictest0-21swapper/308:50:293
23497699122,5cyclictest0-21swapper/307:40:363
23497699121,6cyclictest0-21swapper/307:25:253
23497699121,6cyclictest0-21swapper/307:25:253
23497699121,5cyclictest0-21swapper/307:20:293
23497699120,6cyclictest0-21swapper/308:10:403
23497699114,4cyclictest0-21swapper/308:45:323
23497699114,4cyclictest0-21swapper/308:45:323
23497699112,5cyclictest0-21swapper/307:15:313
23497699111,6cyclictest0-21swapper/312:20:213
23497699111,6cyclictest0-21swapper/312:20:213
23497699111,6cyclictest0-21swapper/307:35:243
23497699102,5cyclictest0-21swapper/307:45:273
23497699100,7cyclictest431ktimers/307:10:013
23497699100,7cyclictest431ktimers/307:10:013
2349769991,4cyclictest0-21swapper/308:25:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional