You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-05 - 07:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack2slot7s.osadl.org (updated Fri Dec 05, 2025 00:39:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3337630995330,16cyclictest12450irq/32-eth020:49:240
3337632995244,5cyclictest3476782-21grep23:29:302
3337632995137,11cyclictest3403685-21timerandwakeup21:49:352
3337632994842,3cyclictest0-21swapper/222:14:242
333763299482,41cyclictest3376713-21ssh21:14:192
3337632994736,7cyclictest3337635-21systemd-run19:10:012
3337630994737,7cyclictest0-21swapper/000:00:020
3337632994637,6cyclictest422-21dbus-daemon20:40:032
3337630994536,6cyclictest422-21dbus-daemon19:49:230
3337632994436,5cyclictest3472600-21fschecks_count23:24:132
3337632994433,8cyclictest3348118-21idleruntime-cro19:45:012
3337632994433,8cyclictest3348118-21idleruntime-cro19:45:012
333763199445,7cyclictest171rcu_preempt20:24:231
3337631994436,5cyclictest3503252-21latency_hist00:05:031
3337631994435,6cyclictest3499418-21systemctl00:00:011
3337631994435,6cyclictest3440212-21needreboot22:39:181
3337630994436,5cyclictest3499535-21cups-browsed00:39:200
3337631994335,5cyclictest955-21mta-sts-daemon23:19:261
3337631994335,5cyclictest3506582-21cron00:10:011
3337631994335,5cyclictest3340649-21systemd-run19:20:011
3337631994334,6cyclictest3351143-21latency_hist19:55:021
3337631994330,9cyclictest3525782-21rm00:39:311
3337632994236,3cyclictest0-21swapper/200:30:002
3337632994236,3cyclictest0-21swapper/200:30:002
3337632994232,6cyclictest421-21avahi-daemon23:59:152
3337632994231,7cyclictest421-21avahi-daemon22:25:012
3337632994231,7cyclictest421-21avahi-daemon22:25:012
3337632994230,9cyclictest3361631-21sh20:30:012
3337632994133,5cyclictest3358648-21systemd-run20:20:012
3337632994131,6cyclictest3433213-21ld22:29:232
3337632994131,6cyclictest3433213-21ld22:29:232
3337632994130,7cyclictest421-21avahi-daemon21:22:512
333763299412,30cyclictest3462273-21ssh23:09:232
3337631994133,5cyclictest3459217-21unixbench_singl23:04:341
3337631994133,5cyclictest3415659-21munin-plugin-st22:05:011
3337631994133,4cyclictest0-21swapper/121:14:231
3337630994125,9cyclictest151ktimers/020:19:200
3337632994031,6cyclictest3499535-21cups-browsed00:12:572
3337632994030,6cyclictest3448362-21head22:49:352
3337632994029,7cyclictest422-21dbus-daemon23:52:142
3337632994029,7cyclictest422-21dbus-daemon23:52:142
3337631994033,4cyclictest0-21swapper/123:14:241
3337631994031,6cyclictest422-21dbus-daemon22:45:011
3337630994031,6cyclictest3385048-21systemd-run21:25:010
3337630994026,11cyclictest3421840-21aten_r2power_po22:14:110
3337630994021,4cyclictest171rcu_preempt22:49:210
3337632993933,2cyclictest0-21swapper/222:40:012
333763299392,25cyclictest3466039-21ssh23:14:282
3337631993932,4cyclictest3433644-21threads22:29:351
3337631993932,4cyclictest3433644-21threads22:29:351
3337631993927,9cyclictest3488330-21cron23:45:011
333763199391,6cyclictest171rcu_preempt20:09:241
3337630993930,7cyclictest1-21systemd20:30:020
3337630993930,6cyclictest3441281-21latency_hist22:40:020
3337632993830,5cyclictest3398957-21fschecks_time21:44:122
3337632993829,6cyclictest1-21systemd22:35:012
3337632993829,6cyclictest1-21systemd22:35:012
3337632993829,5cyclictest3369098-21cron20:55:012
3337632993829,5cyclictest3366671-21fschecks_time20:49:132
333763199386,26cyclictest3391754-21ssh21:34:271
3337631993832,4cyclictest3337629-21cyclictest00:14:161
3337631993831,3cyclictest0-21swapper/123:24:191
3337631993830,5cyclictest3512901-21sh00:19:241
3337631993830,5cyclictest3512901-21sh00:19:241
3337631993830,5cyclictest3400272-21cron21:45:011
3337630993829,6cyclictest3466821-21systemd-run23:15:020
333763299374,26cyclictest3487584-21ssh23:44:272
3337632993731,3cyclictest0-21swapper/222:59:342
3337632993730,3cyclictest0-21swapper/220:45:012
3337632993728,6cyclictest266-21systemd-journal19:40:012
3337632993728,5cyclictest3348667-21fschecks_time19:49:132
3337632993727,6cyclictest422-21dbus-daemon00:05:012
3337632993727,6cyclictest0-21swapper/221:25:252
3337632993724,9cyclictest421-21avahi-daemon22:17:122
3337632993724,7cyclictest956-21mta-sts-daemon00:19:302
3337632993724,7cyclictest956-21mta-sts-daemon00:19:302
3337631993729,5cyclictest422-21dbus-daemon23:40:021
3337631993729,5cyclictest3522601-21rm00:34:311
3337631993729,5cyclictest3522601-21rm00:34:311
333763099378,21cyclictest3487412-21ssh23:44:200
3337630993727,8cyclictest3519575-21mailstats00:29:390
3337630993727,8cyclictest3519575-21mailstats00:29:390
3337630993725,9cyclictest3473879-21cron23:25:010
333763299365,26cyclictest3418656-21ssh22:09:242
3337632993629,4cyclictest698-21cron22:05:012
3337632993627,6cyclictest3373613-21diskmemload00:39:382
3337632993627,5cyclictest0-21swapper/223:49:152
3337632993627,5cyclictest0-21swapper/223:49:152
3337632993627,5cyclictest0-21swapper/221:54:142
333763199363,7cyclictest171rcu_preempt20:14:251
3337631993628,5cyclictest3481066-21grep23:35:011
3337631993628,5cyclictest3381277-21idleruntime-cro21:20:011
3337631993627,7cyclictest3352610-21idleruntime-cro20:00:011
3337630993630,4cyclictest3462671-21switchtime23:09:360
3337630993629,5cyclictest957-21mta-sts-daemon23:49:250
3337630993629,5cyclictest957-21mta-sts-daemon23:49:250
3337632993528,4cyclictest0-21swapper/221:05:012
3337632993527,5cyclictest3370554-21unixbench_singl20:59:322
3337632993526,5cyclictest3451469-21ntpq22:54:252
3337632993522,8cyclictest351ktimers/200:34:212
3337632993522,8cyclictest351ktimers/200:34:212
3337631993527,3cyclictest0-21swapper/123:54:221
3337631993527,3cyclictest0-21swapper/123:54:221
3337631993524,8cyclictest3338445-21needreboot19:14:171
3337630993529,4cyclictest3368144-21fschecks_count20:54:160
3337630993528,5cyclictest3432817-21fschecks_time22:29:160
3337630993528,5cyclictest3432817-21fschecks_time22:29:160
3337630993520,11cyclictest12450irq/32-eth021:54:210
333763299345,20cyclictest3410842-21ssh21:59:222
3337632993427,3cyclictest0-21swapper/222:45:012
3337632993425,4cyclictest0-21swapper/223:15:032
3337632993424,7cyclictest0-21swapper/200:10:022
3337632993421,6cyclictest171rcu_preempt23:04:162
3337631993429,2cyclictest0-21swapper/123:49:151
3337631993429,2cyclictest0-21swapper/123:49:151
3337631993427,4cyclictest3418567-21ntp_kernel_pll_22:09:211
3337631993427,3cyclictest0-21swapper/122:49:121
3337631993426,5cyclictest955-21mta-sts-daemon21:25:101
3337631993424,7cyclictest3347012-21aten_r2power_po19:44:091
3337631993424,7cyclictest3347012-21aten_r2power_po19:44:091
3337631993422,8cyclictest3454886-21cut22:59:191
3337630993428,4cyclictest3495671-21cron23:55:000
3337630993428,4cyclictest3495671-21cron23:55:000
3337630993427,4cyclictest3404131-21systemd-run21:50:020
3337630993425,6cyclictest3346661-21latency_hist19:40:020
3337630993422,9cyclictest0-21swapper/022:54:200
3337630993420,9cyclictest151ktimers/022:09:210
333763099341,27cyclictest657-21snmpd22:44:300
3337630993411,19cyclictest3399407-21ssh21:44:210
3337632993326,4cyclictest956-21mta-sts-daemon21:37:442
3337632993325,5cyclictest3339664-21fschecks_time19:19:112
3337632993325,5cyclictest3339104-21cron19:15:012
3337632993322,7cyclictest421-21avahi-daemon21:19:422
3337632993322,7cyclictest0-21swapper/200:24:222
3337632993322,7cyclictest0-21swapper/200:24:222
3337632993315,13cyclictest266-21systemd-journal21:31:392
333763199333,24cyclictest3422626-21ssh22:14:281
3337631993324,6cyclictest956-21mta-sts-daemon00:29:371
3337631993324,6cyclictest956-21mta-sts-daemon00:29:371
3337631993324,6cyclictest3404508-21sshd21:50:241
3337631993317,13cyclictest3429714-21sshd22:24:241
3337631993317,13cyclictest3429714-21sshd22:24:241
333763099339,5cyclictest14-21ksoftirqd/020:24:230
3337630993326,4cyclictest3415651-21cron22:05:010
3337630993320,10cyclictest0-21swapper/019:29:220
3337630993319,10cyclictest1-21systemd21:39:240
333763299322,25cyclictest3373054-21sshd21:09:242
3337632993221,8cyclictest3357113-21/usr/sbin/munin20:15:022
3337632993220,3cyclictest171rcu_preempt20:24:192
3337632993218,4cyclictest3441479-21kworker/u8:6-writeback23:32:572
3337632993216,11cyclictest351ktimers/219:29:202
3337631993225,4cyclictest957-21mta-sts-daemon21:36:041
3337631993225,4cyclictest955-21mta-sts-daemon23:07:241
3337631993225,4cyclictest3411774-21munin-plugin-st22:00:001
3337631993224,5cyclictest422-21dbus-daemon20:50:011
3337631993224,5cyclictest3499535-21cups-browsed00:21:471
3337631993224,5cyclictest3499535-21cups-browsed00:21:471
3337631993224,5cyclictest3369097-21cron20:55:011
3337631993223,6cyclictest3366138-21true20:45:011
3337631993221,4cyclictest0-21swapper/120:19:181
3337630993226,4cyclictest3436997-21mailstats22:34:340
3337630993226,4cyclictest3436997-21mailstats22:34:340
3337630993226,4cyclictest3376496-21latency21:14:140
3337630993222,5cyclictest3429663-21ssh22:24:220
3337630993222,5cyclictest3429663-21ssh22:24:220
3337630993213,14cyclictest3373613-21diskmemload23:34:290
3337632993123,5cyclictest3484444-21head23:39:332
3337632993115,3cyclictest171rcu_preempt19:54:252
3337631993125,3cyclictest0-21swapper/119:50:001
3337631993124,4cyclictest3384715-21unixbench_multi21:24:321
3337631993123,6cyclictest271ktimers/119:39:201
3337631993123,5cyclictest3362153-21fschecks_count20:34:181
3337631993123,5cyclictest1484156-21cups-browsed22:19:361
333763199312,22cyclictest3403298-21ssh21:49:261
3337630993125,4cyclictest3339036-21unixbench_multi19:14:300
3337630993124,4cyclictest3373605-21cron21:10:010
3337630993123,6cyclictest12450irq/32-eth000:09:210
333763099312,23cyclictest3469543-21ssh23:19:210
3337630993120,8cyclictest3388284-21switchtime21:29:330
3337630993119,9cyclictest421-21avahi-daemon21:19:320
3337630993119,7cyclictest12450irq/32-eth000:34:240
3337630993119,7cyclictest12450irq/32-eth000:34:240
3337630993117,9cyclictest151ktimers/019:44:210
3337630993117,9cyclictest151ktimers/019:44:210
3337630993115,8cyclictest957-21mta-sts-daemon20:34:280
3337632993017,3cyclictest171rcu_preempt20:09:242
3337632993016,3cyclictest171rcu_preempt20:05:022
3337631993022,5cyclictest0-21swapper/119:34:301
333763099307,17cyclictest12450irq/32-eth000:14:230
333763099306,7cyclictest14-21ksoftirqd/020:14:250
3337630993025,3cyclictest0-21swapper/023:04:130
3337630993024,4cyclictest3364055-21ntp_offset20:39:230
3337630993022,5cyclictest3351045-21unixbench_singl19:54:310
3337630993013,12cyclictest266-21systemd-journal21:59:240
3337631992923,3cyclictest0-21swapper/122:55:011
3337631992922,3cyclictest0-21swapper/120:59:241
3337631992921,5cyclictest3437305-21unixbench_multi22:34:381
3337631992921,5cyclictest3437305-21unixbench_multi22:34:381
3337631992920,6cyclictest453-21systemd-logind23:30:011
3337630992923,4cyclictest422-21dbus-daemon00:20:020
3337630992923,4cyclictest422-21dbus-daemon00:20:020
3337630992919,6cyclictest0-21swapper/023:29:230
333763299282,9cyclictest36-21ksoftirqd/219:59:262
3337632992817,3cyclictest171rcu_preempt20:34:152
3337632992814,2cyclictest171rcu_preempt19:34:212
3337631992822,3cyclictest0-21swapper/120:39:231
3337631992819,5cyclictest955-21mta-sts-daemon21:00:141
3337630992821,5cyclictest0-21swapper/021:34:280
3337630992817,8cyclictest0-21swapper/023:40:010
3337630992816,10cyclictest3337635-21systemd-run19:10:010
3337631992719,4cyclictest3360675-21fschecks_time20:29:131
3337631992718,6cyclictest422-21dbus-daemon21:10:011
3337630992722,2cyclictest0-21swapper/020:05:010
333763099272,19cyclictest3516351-21ntpq00:24:250
333763099272,19cyclictest3516351-21ntpq00:24:250
3337630992715,9cyclictest421-21avahi-daemon22:17:120
3337630992712,12cyclictest3455076-21ssh22:59:260
3337632992613,2cyclictest171rcu_preempt19:24:172
3337631992617,6cyclictest453-21systemd-logind19:25:011
3337630992620,3cyclictest959-21mta-sts-daemon00:03:270
3337630992620,3cyclictest955-21mta-sts-daemon20:59:240
3337630992619,5cyclictest0-21swapper/020:00:010
3337630992617,7cyclictest3371658-21proc_pri21:04:310
3337630992617,6cyclictest3344171-21fschecks_time19:34:120
3337631992515,2cyclictest171rcu_preempt20:04:201
333763099256,11cyclictest684-21munin-node19:19:190
3337631992416,5cyclictest3342447-21aten_r2power_en19:29:081
3337631992318,2cyclictest0-21swapper/119:10:031
333763099232,17cyclictest591-21sshd20:44:230
333763099212,14cyclictest3341594-21ntpq19:24:200
3337630992110,3cyclictest14-21ksoftirqd/020:09:240
333763399193,8cyclictest0-21swapper/322:49:263
333763399191,9cyclictest0-21swapper/323:29:253
333763399182,9cyclictest0-21swapper/321:54:193
333763399182,6cyclictest0-21swapper/320:24:223
333763399173,8cyclictest0-21swapper/321:24:223
333763399173,6cyclictest0-21swapper/321:29:203
333763399172,9cyclictest0-21swapper/321:44:183
333763399172,9cyclictest0-21swapper/319:49:233
333763399172,7cyclictest0-21swapper/322:29:253
333763399172,7cyclictest0-21swapper/322:29:253
333763399172,7cyclictest0-21swapper/319:14:223
333763399171,9cyclictest0-21swapper/322:24:243
333763399171,9cyclictest0-21swapper/322:24:243
333763399163,7cyclictest0-21swapper/323:49:263
333763399163,7cyclictest0-21swapper/323:49:263
333763399162,8cyclictest0-21swapper/319:34:223
333763399162,6cyclictest0-21swapper/322:59:223
333763399162,6cyclictest0-21swapper/321:19:273
333763399161,7cyclictest0-21swapper/321:04:273
333763399161,6cyclictest0-21swapper/320:34:233
333763399152,7cyclictest0-21swapper/323:19:213
333763399152,7cyclictest0-21swapper/322:44:233
333763399152,7cyclictest0-21swapper/321:45:233
333763399152,7cyclictest0-21swapper/321:14:173
333763399152,7cyclictest0-21swapper/300:34:243
333763399152,7cyclictest0-21swapper/300:34:243
333763399152,7cyclictest0-21swapper/300:09:193
333763399151,8cyclictest0-21swapper/322:09:283
333763399151,7cyclictest0-21swapper/323:54:193
333763399151,7cyclictest0-21swapper/323:54:193
333763399151,7cyclictest0-21swapper/320:49:243
333763399151,7cyclictest0-21swapper/319:59:233
333763399151,7cyclictest0-21swapper/319:24:203
333763399151,6cyclictest0-21swapper/321:09:223
333763399150,8cyclictest0-21swapper/323:09:283
333763399142,7cyclictest0-21swapper/323:39:193
333763399142,7cyclictest0-21swapper/323:34:263
333763399142,6cyclictest0-21swapper/323:04:213
333763399142,6cyclictest0-21swapper/300:24:203
333763399142,6cyclictest0-21swapper/300:24:203
333763399141,8cyclictest0-21swapper/323:14:213
333763399141,8cyclictest0-21swapper/300:39:283
333763399141,7cyclictest0-21swapper/322:34:253
333763399141,7cyclictest0-21swapper/322:34:253
333763399141,6cyclictest0-21swapper/320:29:243
333763399141,6cyclictest0-21swapper/320:04:203
333763399134,6cyclictest0-21swapper/322:19:263
333763399132,6cyclictest0-21swapper/323:59:213
333763399132,6cyclictest0-21swapper/322:04:183
333763399132,5cyclictest0-21swapper/320:39:193
333763399132,5cyclictest0-21swapper/319:54:203
333763399131,7cyclictest0-21swapper/323:24:203
333763399131,7cyclictest0-21swapper/322:39:273
333763399131,7cyclictest0-21swapper/320:54:233
333763399131,6cyclictest0-21swapper/322:54:193
333763399131,6cyclictest0-21swapper/320:09:243
333763399131,6cyclictest0-21swapper/319:29:233
333763399131,5cyclictest0-21swapper/300:03:413
333763399122,6cyclictest0-21swapper/321:59:243
333763399121,7cyclictest0-21swapper/323:44:213
333763399121,6cyclictest0-21swapper/321:34:273
333763399121,6cyclictest0-21swapper/320:44:233
333763399121,6cyclictest0-21swapper/319:40:013
333763399112,6cyclictest0-21swapper/322:14:193
333763399112,6cyclictest0-21swapper/300:19:173
333763399112,6cyclictest0-21swapper/300:19:173
333763399112,5cyclictest0-21swapper/320:19:183
333763399111,7cyclictest0-21swapper/300:14:183
333763399111,6cyclictest0-21swapper/300:29:163
333763399111,6cyclictest0-21swapper/300:29:163
333763399111,5cyclictest0-21swapper/320:59:193
333763399111,5cyclictest0-21swapper/320:14:193
333763399104,4cyclictest0-21swapper/321:39:303
333763399102,6cyclictest0-21swapper/319:41:213
333763399102,6cyclictest0-21swapper/319:41:213
33376339991,4cyclictest0-21swapper/319:20:023
33376339991,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional