You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 21:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Tue Dec 02, 2025 12:39:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2486680996341,18cyclictest0-21swapper/211:09:212
2486678995944,5cyclictest2512177-21systemd-run08:35:020
2486680995544,6cyclictest2605445-21kworker/u8:3-xprtiod11:55:522
2486679995239,7cyclictest0-21swapper/112:34:191
2486679995239,7cyclictest0-21swapper/112:34:191
2486680995126,6cyclictest657-21snmpd12:39:282
2486679995042,5cyclictest2661947-21/usr/sbin/munin12:39:151
2486680994937,7cyclictest2533087-21cron09:25:012
2486678994739,5cyclictest422-21dbus-daemon09:25:010
2486680994535,6cyclictest2647919-21needreboot12:19:162
2486680994535,6cyclictest2647919-21needreboot12:19:162
2486679994535,7cyclictest266-21systemd-journal10:44:141
2486678994436,5cyclictest1-21systemd10:10:020
2486679994336,4cyclictest2565413-21kworker/u8:5-xprtiod11:00:001
2486679994334,6cyclictest2503186-21systemd-run08:05:011
2486679994330,10cyclictest2641620-21ntp_offset12:09:251
2486679994235,3cyclictest0-21swapper/111:14:211
2486679994234,5cyclictest2612141-21cut11:25:011
2486679994231,8cyclictest2542440-21ssh09:40:021
2486678994223,14cyclictest12450irq/32-eth011:59:210
248667899421,35cyclictest151ktimers/009:09:170
248668099413,30cyclictest2574424-21ssh10:29:222
2486679994134,4cyclictest0-21swapper/108:49:261
2486679994133,5cyclictest2519773-21latency_hist09:00:021
2486679994133,4cyclictest0-21swapper/109:49:221
2486678994135,4cyclictest2542390-21sendmail09:40:010
248667899412,35cyclictest2522661-21diskmemload10:04:180
248667899412,33cyclictest2538763-21ssh09:34:390
2486680994032,5cyclictest2558712-21systemd-run10:05:012
2486680994029,8cyclictest2555272-21ssh10:00:012
2486679994033,4cyclictest955-21mta-sts-daemon08:29:231
2486679994031,6cyclictest2632947-40mandb11:55:511
248667999401,31cyclictest2621373-21ssh11:39:181
2486678994033,5cyclictest2622256-21grep11:40:010
2486678994033,4cyclictest2663354-21sh12:40:010
2486678994032,6cyclictest2598793-21systemd-run11:05:010
2486678994031,6cyclictest2547489-21fschecks_count09:49:130
2486678994031,5cyclictest2605445-21kworker/u8:3-xprtiod11:30:020
2486678994026,8cyclictest12450irq/32-eth012:29:220
2486678994026,8cyclictest12450irq/32-eth012:29:220
2486680993930,6cyclictest2638003-21fschecks_time12:04:102
248668099391,31cyclictest2621425-21sh11:39:202
2486679993932,4cyclictest69-21kswapd009:54:231
2486679993931,5cyclictest2544254-21aten_r2power_vo09:44:091
2486679993930,6cyclictest422-21dbus-daemon10:34:231
2486679993928,8cyclictest2574367-21ntp_kernel_pll_10:29:221
248667999392,29cyclictest2561329-21grep10:09:261
248667999391,31cyclictest2557853-21ssh10:04:191
2486678993926,11cyclictest2570857-21sshd10:24:150
2486678993923,13cyclictest422-21dbus-daemon10:40:020
2486680993830,5cyclictest1-21systemd11:45:022
248667999386,25cyclictest2597933-21ssh11:04:201
2486679993830,5cyclictest2526243-21unixbench_singl09:14:331
2486679993830,5cyclictest2522661-21diskmemload12:29:291
2486679993830,5cyclictest2522661-21diskmemload12:29:291
2486679993830,5cyclictest2507692-21systemd-run08:20:021
2486679993829,6cyclictest1662287-21cups-browsed10:19:211
2486679993829,4cyclictest0-21swapper/110:39:211
2486679993827,8cyclictest1662287-21cups-browsed09:21:531
2486678993832,4cyclictest2565384-21latency_hist10:15:010
2486678993830,5cyclictest1-21systemd07:10:010
2486678993822,12cyclictest151ktimers/007:24:240
2486680993729,5cyclictest2495669-21idleruntime-cro07:40:012
2486680993727,6cyclictest2489545-21/usr/sbin/munin07:19:292
2486679993730,3cyclictest0-21swapper/111:49:211
2486679993729,4cyclictest0-21swapper/111:29:111
2486678993731,3cyclictest2645576-21cron12:15:010
2486678993729,3cyclictest0-21swapper/011:14:210
2486678993725,4cyclictest2555228-21cron10:00:010
2486680993630,3cyclictest0-21swapper/209:40:012
2486680993629,4cyclictest2642413-21latency_hist12:10:022
2486680993628,5cyclictest2545548-21munin-plugin-st09:45:012
2486680993627,6cyclictest474-21in:imuxsock12:20:152
2486680993627,6cyclictest474-21in:imuxsock12:20:152
2486680993625,9cyclictest2570938-21ssh10:24:162
2486679993629,4cyclictest2625678-21cron11:45:011
248667899363,26cyclictest2611361-21ssh11:24:210
2486678993628,5cyclictest421-21avahi-daemon07:44:290
2486678993627,5cyclictest2489667-21munin-run07:20:010
2486678993627,5cyclictest0-21swapper/009:19:320
2486678993626,8cyclictest2506986-21needreboot08:19:170
2486678993623,8cyclictest12450irq/32-eth010:49:190
2486678993622,12cyclictest2601801-21ssh11:09:310
2486680993527,5cyclictest2588681-21cron10:50:012
2486680993527,5cyclictest2562058-21latency_hist10:10:022
2486680993527,5cyclictest2501567-21tail07:59:352
2486680993527,5cyclictest2501567-21tail07:59:352
2486680993524,6cyclictest2645576-21cron12:15:012
2486680993523,10cyclictest266-21systemd-journal11:04:532
2486680993518,12cyclictest351ktimers/209:49:292
2486679993527,5cyclictest2570756-21kernelversion10:24:141
2486679993527,5cyclictest2535009-21fschecks_count09:29:111
2486679993527,4cyclictest2565114-21unixbench_multi10:14:311
2486679993527,4cyclictest0-21swapper/109:59:211
2486679993526,6cyclictest2488176-21systemd-run07:15:011
2486678993527,5cyclictest1662287-21cups-browsed09:04:160
2486678993521,9cyclictest12450irq/32-eth012:19:170
2486678993521,9cyclictest12450irq/32-eth012:19:170
2486680993427,5cyclictest2486677-21cyclictest10:54:172
2486680993427,5cyclictest2486677-21cyclictest10:54:172
2486680993426,5cyclictest2529691-21systemd-run09:20:012
2486680993426,5cyclictest2503060-21tail08:04:292
2486680993425,5cyclictest421-21avahi-daemon11:35:012
2486679993428,3cyclictest0-21swapper/107:40:011
2486679993427,4cyclictest2632101-21sed11:54:331
2486679993426,5cyclictest955-21mta-sts-daemon07:50:011
2486679993426,5cyclictest2522661-21diskmemload12:24:281
2486679993426,5cyclictest2522661-21diskmemload12:24:281
248667999342,27cyclictest2528829-21ssh09:19:271
2486678993429,3cyclictest2536319-21latency_hist09:30:020
2486678993428,4cyclictest2652070-21tail12:24:320
2486678993428,4cyclictest2652070-21tail12:24:320
2486678993426,6cyclictest956-21mta-sts-daemon10:40:210
2486678993426,6cyclictest2515590-21kworker/u8:16-nfsiod09:45:010
2486678993421,9cyclictest12450irq/32-eth012:09:290
2486678993421,3cyclictest0-21swapper/007:49:140
2486678993420,6cyclictest12450irq/32-eth010:34:190
2486680993326,4cyclictest2566724-21sh10:17:532
2486680993325,5cyclictest2517183-21fschecks_count08:54:112
2486680993325,5cyclictest2513701-21grep08:40:022
2486680993324,6cyclictest2564893-21switchtime10:14:302
2486680993321,9cyclictest2611337-21ntp_kernel_pll_11:24:212
248668099331,27cyclictest1662271-21cupsd12:29:252
248668099331,27cyclictest1662271-21cupsd12:29:252
2486679993327,3cyclictest0-21swapper/107:35:021
2486679993326,4cyclictest2515167-21munin-plugin-st08:45:011
2486679993325,5cyclictest2500188-21latency_hist07:55:021
2486679993323,7cyclictest2617941-21missed_timers11:34:151
2486678993327,4cyclictest2506149-21cron08:15:010
2486678993326,4cyclictest2659679-21cron12:35:000
2486678993326,4cyclictest2659679-21cron12:35:000
2486678993325,5cyclictest2503132-21head08:04:300
2486680993223,5cyclictest2497174-21munin-plugin-st07:45:012
2486680993222,8cyclictest2628391-21mailstats11:49:272
2486680993221,8cyclictest2516023-21ntp_kernel_err08:49:222
2486679993226,3cyclictest0-21swapper/108:35:011
2486679993224,5cyclictest2588384-21unixbench_multi10:49:311
2486679993224,5cyclictest2492691-21systemd-run07:30:021
2486679993223,6cyclictest1662287-21cups-browsed12:04:281
2486679993218,8cyclictest2538755-21sh09:34:391
2486678993211,18cyclictest2525563-21ssh09:14:190
2486680993125,3cyclictest0-21swapper/209:55:012
2486680993125,3cyclictest0-21swapper/209:15:012
2486680993125,3cyclictest0-21swapper/207:15:012
2486680993124,4cyclictest0-21swapper/208:20:012
2486680993123,5cyclictest2659332-21unixbench-2d12:34:302
2486680993123,5cyclictest2659332-21unixbench-2d12:34:302
2486680993122,5cyclictest2514707-21proc_pri08:44:252
248667999314,21cyclictest2601370-21ssh11:09:201
2486679993125,3cyclictest0-21swapper/110:55:021
2486679993125,3cyclictest0-21swapper/110:55:021
2486679993125,3cyclictest0-21swapper/107:44:141
2486679993124,4cyclictest2513223-21processes08:39:311
2486679993123,5cyclictest422-21dbus-daemon12:20:011
2486679993123,5cyclictest422-21dbus-daemon12:20:011
2486678993125,4cyclictest2515232-21latency_hist08:45:020
2486678993124,4cyclictest0-21swapper/011:20:010
2486678993123,6cyclictest2518057-21sed08:54:310
2486678993120,8cyclictest2631904-21switchtime11:54:320
2486680993025,2cyclictest0-21swapper/210:44:322
2486680993024,3cyclictest0-21swapper/211:55:012
2486680993023,4cyclictest0-21swapper/210:35:012
248668099302,24cyclictest2594511-21ssh10:59:212
2486679993025,2cyclictest0-21swapper/112:15:021
2486679993023,4cyclictest2503708-21/usr/sbin/munin08:09:151
2486679993022,5cyclictest2491170-21cron07:25:011
2486678993019,8cyclictest12450irq/32-eth010:59:220
2486678993016,11cyclictest151ktimers/008:24:230
2486678993016,11cyclictest151ktimers/008:24:230
2486680992921,5cyclictest2491199-21systemd-run07:25:022
2486680992920,5cyclictest2519695-21latency_hist09:00:022
248668099291,20cyclictest2535419-21ssh09:29:192
2486679992922,3cyclictest0-21swapper/108:00:011
2486679992922,3cyclictest0-21swapper/108:00:011
2486679992921,5cyclictest2505214-21gltestperf08:14:121
2486678992925,2cyclictest0-21swapper/007:59:140
2486678992925,2cyclictest0-21swapper/007:59:140
2486678992924,3cyclictest0-21swapper/010:29:140
2486678992923,4cyclictest1662287-21cups-browsed07:31:370
2486678992923,3cyclictest0-21swapper/010:19:180
2486678992922,4cyclictest956-21mta-sts-daemon07:14:220
2486678992922,4cyclictest2616273-21ssh11:31:180
2486678992921,5cyclictest2491709-21fschecks_count07:29:100
2486678992916,6cyclictest2628142-21ssh11:49:200
2486680992821,5cyclictest2509978-21missed_timers08:29:142
2486680992821,4cyclictest2520735-21processes09:04:242
2486680992820,4cyclictest2565413-21kworker/u8:5-events_unbound11:14:112
2486680992820,4cyclictest0-21swapper/208:25:012
2486680992820,4cyclictest0-21swapper/208:25:012
248667999283,17cyclictest437-21mta-sts-daemon07:19:221
2486679992823,2cyclictest0-21swapper/111:20:011
2486679992822,3cyclictest0-21swapper/109:10:011
248667899289,14cyclictest422-21dbus-daemon08:39:250
248667899282,20cyclictest2550931-21ssh09:54:190
2486678992814,7cyclictest2510847-21latency_hist08:30:020
2486678992812,11cyclictest12450irq/32-eth011:44:240
2486680992721,3cyclictest0-21swapper/211:29:102
2486680992721,3cyclictest0-21swapper/210:39:202
2486680992721,3cyclictest0-21swapper/208:35:012
2486680992721,3cyclictest0-21swapper/207:50:012
2486680992720,5cyclictest0-21swapper/209:10:012
2486680992720,4cyclictest2485334-21kworker/u8:1-rpciod07:30:022
248668099271,20cyclictest2538659-21rm09:34:352
2486679992715,8cyclictest957-21mta-sts-daemon09:04:231
2486678992722,3cyclictest0-21swapper/012:04:160
2486678992720,5cyclictest0-21swapper/008:59:210
2486680992620,4cyclictest2486677-21cyclictest11:19:282
2486680992617,6cyclictest959-21mta-sts-daemon07:34:142
2486680992617,6cyclictest1662287-21cups-browsed08:08:512
248667999264,15cyclictest437-21mta-sts-daemon08:24:161
248667999264,15cyclictest437-21mta-sts-daemon08:24:161
2486678992619,3cyclictest0-21swapper/007:39:210
2486680992516,6cyclictest1662287-21cups-browsed08:14:042
2486679992514,4cyclictest171rcu_preempt08:54:201
2486678992520,2cyclictest0-21swapper/010:54:170
2486678992520,2cyclictest0-21swapper/010:54:170
2486678992516,6cyclictest2500079-21unixbench_multi07:54:280
2486678992514,7cyclictest12450irq/32-eth008:49:260
248668099242,17cyclictest2499538-21sshd07:54:142
2486679992416,5cyclictest2486856-21latency_hist07:10:021
248667899222,15cyclictest2504042-21ntpq08:09:200
2486680991910,6cyclictest491-21gdbus07:10:012
248668199183,8cyclictest0-21swapper/309:44:303
248668199182,8cyclictest0-21swapper/311:55:483
248668199181,7cyclictest0-21swapper/310:39:243
248668199173,6cyclictest0-21swapper/312:39:253
248668199172,7cyclictest0-21swapper/311:04:213
248668199171,6cyclictest0-21swapper/311:19:183
248668199162,7cyclictest0-21swapper/309:59:223
248668199162,6cyclictest0-21swapper/309:04:203
248668199161,8cyclictest0-21swapper/312:29:253
248668199161,8cyclictest0-21swapper/312:29:253
248668199154,8cyclictest0-21swapper/309:29:213
248668199153,6cyclictest0-21swapper/312:09:243
248668199152,8cyclictest0-21swapper/311:39:193
248668199152,8cyclictest0-21swapper/311:14:213
248668199152,6cyclictest0-21swapper/308:09:223
248668199151,9cyclictest0-21swapper/312:24:203
248668199151,9cyclictest0-21swapper/312:24:203
248668199151,8cyclictest0-21swapper/310:54:183
248668199151,8cyclictest0-21swapper/310:54:183
248668199151,8cyclictest0-21swapper/310:04:193
248668199151,8cyclictest0-21swapper/309:49:223
248668199151,8cyclictest0-21swapper/307:14:223
248668199151,6cyclictest0-21swapper/308:59:213
248668199151,6cyclictest0-21swapper/308:19:213
248668199142,7cyclictest0-21swapper/311:34:153
248668199142,7cyclictest0-21swapper/311:29:233
248668199142,7cyclictest0-21swapper/310:14:173
248668199142,7cyclictest0-21swapper/310:09:183
248668199142,7cyclictest0-21swapper/309:14:253
248668199142,6cyclictest0-21swapper/311:09:183
248668199142,5cyclictest0-21swapper/307:49:213
248668199141,8cyclictest0-21swapper/310:49:193
248668199141,8cyclictest0-21swapper/309:24:173
248668199141,7cyclictest0-21swapper/311:49:183
248668199141,7cyclictest0-21swapper/308:44:193
248668199141,7cyclictest0-21swapper/308:04:193
248668199141,7cyclictest0-21swapper/307:24:183
248668199141,6cyclictest0-21swapper/308:49:243
248668199141,6cyclictest0-21swapper/308:34:213
248668199141,6cyclictest0-21swapper/307:39:183
248668199132,7cyclictest0-21swapper/312:34:193
248668199132,7cyclictest0-21swapper/312:34:193
248668199132,7cyclictest0-21swapper/311:24:213
248668199132,6cyclictest0-21swapper/312:18:153
248668199132,6cyclictest0-21swapper/312:18:153
248668199132,6cyclictest0-21swapper/312:04:203
248668199132,6cyclictest0-21swapper/310:44:193
248668199132,6cyclictest0-21swapper/309:39:193
248668199132,6cyclictest0-21swapper/309:34:333
248668199132,6cyclictest0-21swapper/308:39:253
248668199131,7cyclictest0-21swapper/310:59:193
248668199131,7cyclictest0-21swapper/310:34:133
248668199131,7cyclictest0-21swapper/310:24:223
248668199131,7cyclictest0-21swapper/309:19:303
248668199131,6cyclictest0-21swapper/308:24:163
248668199131,6cyclictest0-21swapper/308:24:163
248668199131,6cyclictest0-21swapper/307:59:223
248668199131,6cyclictest0-21swapper/307:59:223
248668199130,7cyclictest0-21swapper/309:54:183
248668199122,6cyclictest0-21swapper/311:54:183
248668199122,6cyclictest0-21swapper/309:09:193
248668199122,5cyclictest0-21swapper/307:19:223
248668199122,4cyclictest0-21swapper/307:54:133
248668199121,6cyclictest0-21swapper/308:54:203
248668199121,6cyclictest0-21swapper/308:14:253
248668199121,6cyclictest0-21swapper/307:34:403
248668199121,5cyclictest0-21swapper/307:29:163
248668199112,6cyclictest0-21swapper/310:18:263
248668199112,5cyclictest0-21swapper/312:14:103
248668199111,6cyclictest0-21swapper/311:44:133
248668199103,4cyclictest0-21swapper/307:44:253
248668199102,4cyclictest0-21swapper/310:25:153
248668199102,4cyclictest0-21swapper/308:29:183
24866819981,4cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional