You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 00:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack2slot7s.osadl.org (updated Wed Dec 03, 2025 12:39:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
295605995647,5cyclictest418858-21/usr/sbin/munin11:24:322
295603995636,13cyclictest12550irq/33-eth011:09:250
295604995142,6cyclictest422-21dbus-daemon12:20:021
295603995127,20cyclictest0-21swapper/009:14:320
29560399491,23cyclictest421678-21ssh11:29:280
295604994832,11cyclictest0-21swapper/107:14:191
29560399486,21cyclictest405126-21ssh11:04:200
295603994824,22cyclictest447976-21kworker/u8:3+xprtiod12:14:140
295603994822,23cyclictest441681-21cut12:00:010
295603994821,4cyclictest454533-21cron12:20:010
295605994636,6cyclictest338707-21idleruntime-cro09:20:012
295603994634,8cyclictest370179-21systemd-run10:10:010
295603994634,8cyclictest370179-21systemd-run10:10:010
295603994618,7cyclictest466614-21latency12:39:160
295605994536,6cyclictest298401-21latency_hist07:20:022
29560599453,36cyclictest425091-21ssh11:34:262
29560399441,24cyclictest421-21avahi-daemon11:49:230
295605994335,4cyclictest959-21mta-sts-daemon12:14:232
295604994336,3cyclictest0-21swapper/111:14:261
295603994313,9cyclictest12450irq/32-eth012:29:260
295605994234,5cyclictest461089-21systemd-run12:30:022
295605994234,3cyclictest0-21swapper/209:29:232
295605994233,6cyclictest1-21systemd09:15:012
295605994230,9cyclictest401283-21aten_r2power_vo10:59:092
295604994234,5cyclictest451128-21cron12:15:011
295605994134,3cyclictest0-21swapper/211:54:192
295604994133,5cyclictest328575-21munin-plugin-st09:00:011
295603994127,10cyclictest957-21mta-sts-daemon11:20:440
295605994034,3cyclictest0-21swapper/207:40:012
29560499403,24cyclictest434950-21ssh11:49:251
295604994031,6cyclictest402717-21systemd-run11:00:011
295604994030,6cyclictest1-21systemd10:55:011
29560399403,31cyclictest411824-21ssh11:14:220
295603994031,6cyclictest322614-21systemd-run08:40:020
295603994030,7cyclictest382732-21/usr/sbin/munin10:30:010
295605993931,5cyclictest457420-21unixbench_multi12:24:302
295605993931,5cyclictest353371-21fschecks_count09:44:152
295605993931,5cyclictest353371-21fschecks_count09:44:152
295605993930,6cyclictest435723-21ssh11:50:022
295605993930,6cyclictest330112-21systemd-run09:05:012
295604993932,4cyclictest955-21mta-sts-daemon11:44:241
295604993931,5cyclictest316600-21awk08:20:011
295603993932,5cyclictest324081-21cron08:45:010
295603993930,6cyclictest3675387-21cupsd09:54:160
295603993927,8cyclictest353669-21missed_timers09:44:180
295603993927,8cyclictest353669-21missed_timers09:44:180
295603993927,4cyclictest0-21swapper/009:59:250
295603993920,2cyclictest0-21swapper/010:54:190
295605993831,3cyclictest0-21swapper/211:39:172
295605993830,5cyclictest295624-21latency_hist07:10:022
295604993832,3cyclictest0-21swapper/111:20:011
295604993830,5cyclictest386149-21kworker/u8:17-rpciod10:50:011
295604993827,8cyclictest322614-21systemd-run08:40:021
295603993829,7cyclictest298381-21/usr/sbin/munin07:20:020
295605993730,3cyclictest0-21swapper/212:40:022
295605993729,5cyclictest440810-21/usr/sbin/munin11:59:182
295605993728,6cyclictest957-21mta-sts-daemon10:20:022
295605993726,7cyclictest422-21dbus-daemon08:44:162
295604993728,6cyclictest266-21systemd-journal09:47:201
295604993728,6cyclictest266-21systemd-journal09:47:201
295604993724,10cyclictest421-21avahi-daemon10:45:011
29560399373,30cyclictest391780-21ssh10:44:210
295603993731,3cyclictest0-21swapper/009:20:010
295603993729,5cyclictest438318-21unixbench_multi11:54:320
295603993726,8cyclictest376557-21munin-run10:20:010
29560599363,26cyclictest431756-21ssh11:44:262
295605993628,5cyclictest331583-21diskmemload09:55:012
295605993625,8cyclictest391679-21missed_timers10:44:182
295604993629,4cyclictest0-21swapper/110:05:011
295604993629,4cyclictest0-21swapper/109:29:241
295604993628,5cyclictest387983-21/usr/sbin/munin10:39:101
295604993627,6cyclictest375134-21aten_r2power_po10:19:141
295604993627,5cyclictest381848-21ntp_kernel_err10:29:181
295604993626,8cyclictest467669-21sendmail12:40:011
295603993630,4cyclictest402684-21systemctl11:00:010
295603993625,8cyclictest0-21swapper/011:45:020
29560599353,27cyclictest372687-21ssh10:14:282
29560599353,27cyclictest372687-21ssh10:14:282
295605993529,4cyclictest389031-21unixbench_multi10:39:342
295605993526,6cyclictest369087-21fschecks_count10:09:122
295605993526,6cyclictest369087-21fschecks_count10:09:122
295605993526,6cyclictest0-21swapper/209:39:202
295605993525,8cyclictest313595-21cut08:10:012
295605993516,15cyclictest351ktimers/207:54:212
295604993526,6cyclictest350380-21fschecks_count09:39:151
295604993525,6cyclictest421-21avahi-daemon10:23:351
295604993525,6cyclictest421-21avahi-daemon10:23:351
295604993524,7cyclictest3675390-21cups-browsed12:22:541
29560399354,25cyclictest463436-21sh12:34:180
295603993526,5cyclictest151ktimers/008:04:250
295603993524,8cyclictest0-21swapper/007:44:460
295603993514,17cyclictest415104-21ssh11:19:300
295605993427,4cyclictest393353-21kworker/u8:0-rpciod10:55:012
295605993424,7cyclictest328076-21sh08:59:372
295605993424,6cyclictest3675390-21cups-browsed12:19:312
295604993427,3cyclictest0-21swapper/112:34:291
295604993426,5cyclictest360487-21idleruntime-cro09:55:011
295604993426,4cyclictest0-21swapper/111:29:201
295604993425,6cyclictest327025-21head08:54:321
295604993424,7cyclictest421-21avahi-daemon11:30:181
295604993424,4cyclictest0-21swapper/109:34:201
29560399343,24cyclictest350920-21ssh09:39:280
295603993422,9cyclictest297101-21systemd-run07:15:010
295603993417,14cyclictest313622-21latency_hist08:10:020
29560599333,23cyclictest447285-21rm12:09:292
295605993326,3cyclictest0-21swapper/210:29:202
295605993325,5cyclictest326120-21fschecks_count08:54:132
29560599332,25cyclictest463442-21ssh12:34:182
295604993326,4cyclictest342070-21ssh09:25:011
295604993324,6cyclictest474-21in:imuxsock07:34:191
295604993324,4cyclictest0-21swapper/111:39:241
29560499332,25cyclictest447258-21ssh12:09:271
295603993327,3cyclictest330240-21latency_hist09:05:020
295603993321,7cyclictest12450irq/32-eth012:04:260
295603993320,7cyclictest331583-21diskmemload10:49:250
295605993224,5cyclictest385938-21systemd-run10:35:012
295605993224,5cyclictest385938-21systemd-run10:35:012
295605993224,4cyclictest0-21swapper/209:34:212
295605993222,6cyclictest0-21swapper/210:25:012
295605993222,6cyclictest0-21swapper/210:25:012
295604993225,4cyclictest955-21mta-sts-daemon07:24:201
295604993221,8cyclictest955-21mta-sts-daemon07:44:071
295604993221,7cyclictest421-21avahi-daemon11:55:251
295604993221,7cyclictest3675387-21cupsd11:01:471
295604993219,8cyclictest369466-21sh10:09:181
295604993219,8cyclictest369466-21sh10:09:181
295603993225,4cyclictest955-21mta-sts-daemon10:36:310
295603993225,4cyclictest309104-21systemd-run07:55:020
295603993225,3cyclictest0-21swapper/011:39:200
295603993213,13cyclictest341329-21rm09:24:240
295603993213,13cyclictest12450irq/32-eth010:24:160
295603993213,13cyclictest12450irq/32-eth010:24:160
29560399321,26cyclictest366358-21ssh10:04:190
295605993125,4cyclictest295602-21cyclictest11:29:462
295605993124,3cyclictest0-21swapper/211:09:262
295604993125,3cyclictest0-21swapper/109:20:011
295604993125,3cyclictest0-21swapper/107:40:011
295604993124,4cyclictest957-21mta-sts-daemon11:09:191
295604993123,5cyclictest422-21dbus-daemon09:44:291
295604993123,5cyclictest422-21dbus-daemon09:44:291
295604993123,5cyclictest325590-21munin-plugin-st08:50:011
295604993123,4cyclictest0-21swapper/110:14:321
295604993123,4cyclictest0-21swapper/110:14:321
295604993123,4cyclictest0-21swapper/108:44:201
295604993114,12cyclictest261rcuc/109:59:211
295603993123,5cyclictest491-21gdbus11:31:030
295603993121,7cyclictest345276-21latency_hist09:30:020
29560399311,25cyclictest356659-21ssh09:49:230
29560399311,25cyclictest356659-21ssh09:49:230
295605993023,4cyclictest959-21mta-sts-daemon11:14:172
295605993023,4cyclictest959-21mta-sts-daemon08:50:022
295605993023,4cyclictest957-21mta-sts-daemon09:55:472
295605993022,5cyclictest959-21mta-sts-daemon09:05:572
295605993021,7cyclictest316593-21idleruntime-cro08:20:012
295604993024,3cyclictest0-21swapper/107:20:011
295604993023,3cyclictest0-21swapper/109:05:011
295604993022,5cyclictest335559-21latency_hist09:15:021
295604993021,5cyclictest421-21avahi-daemon11:22:171
295603993025,3cyclictest0-21swapper/008:20:010
295603993024,4cyclictest348465-21cron09:35:010
295603993024,4cyclictest326270-21latency08:54:150
295603993023,4cyclictest0-21swapper/007:25:010
29560599299,16cyclictest422-21dbus-daemon07:14:192
295605992922,4cyclictest959-21mta-sts-daemon11:01:422
295605992922,3cyclictest0-21swapper/212:04:182
295605992921,5cyclictest319609-21systemd-run08:30:012
295605992921,5cyclictest319609-21systemd-run08:30:012
295605992921,5cyclictest315022-21unixbench_singl08:14:292
295605992921,5cyclictest302987-21tail07:34:312
295605992914,12cyclictest366263-21sed10:04:172
295604992923,3cyclictest0-21swapper/110:34:201
295604992923,3cyclictest0-21swapper/110:34:201
295604992923,3cyclictest0-21swapper/108:10:011
295604992923,3cyclictest0-21swapper/108:05:011
295604992922,4cyclictest0-21swapper/112:05:011
295604992922,3cyclictest0-21swapper/112:30:011
295604992921,5cyclictest321009-21unixbench_multi08:34:381
295604992921,5cyclictest307481-21tail07:49:301
295604992919,4cyclictest0-21swapper/111:54:521
29560499291,6cyclictest171rcu_preempt07:29:521
29560499291,6cyclictest171rcu_preempt07:29:521
295603992922,4cyclictest373371-21latency_hist10:15:010
295603992922,4cyclictest373371-21latency_hist10:15:010
295605992822,3cyclictest0-21swapper/209:50:012
295605992822,3cyclictest0-21swapper/209:50:012
29560499284,13cyclictest957-21mta-sts-daemon08:14:231
295603992821,5cyclictest302481-21ntp_kernel_pll_07:34:200
295603992820,5cyclictest0-21swapper/012:09:180
295603992818,7cyclictest331573-21cron09:10:010
29560599273,18cyclictest341151-21ssh09:24:172
295605992721,3cyclictest0-21swapper/210:49:092
295605992721,3cyclictest0-21swapper/208:40:012
295605992718,7cyclictest298669-21timerwakeupswit07:20:062
295604992721,3cyclictest0-21swapper/108:00:011
295604992718,6cyclictest957-21mta-sts-daemon08:28:081
295604992718,6cyclictest957-21mta-sts-daemon08:28:081
295603992717,7cyclictest0-21swapper/010:34:250
295603992717,7cyclictest0-21swapper/010:34:250
295603992716,5cyclictest955-21mta-sts-daemon12:22:370
295605992622,2cyclictest0-21swapper/207:50:022
29560599262,17cyclictest415068-21ssh11:19:292
295605992620,4cyclictest956-21mta-sts-daemon07:44:352
295605992620,3cyclictest0-21swapper/207:59:232
295605992613,5cyclictest955-21mta-sts-daemon08:24:262
295605992519,3cyclictest0-21swapper/208:05:012
295603992521,2cyclictest0-21swapper/009:00:010
295603992519,3cyclictest0-21swapper/007:40:010
29560399251,19cyclictest657-21snmpd08:49:190
29560599246,5cyclictest59950lldpd08:30:392
295605992419,2cyclictest0-21swapper/207:29:402
295605992419,2cyclictest0-21swapper/207:29:402
295603992418,4cyclictest955-21mta-sts-daemon07:29:490
295603992418,4cyclictest955-21mta-sts-daemon07:29:490
295603992418,3cyclictest0-21swapper/008:14:200
29560499232,16cyclictest308514-21sshd07:54:191
295604992315,6cyclictest331127-21grep09:09:231
295604992315,5cyclictest295614-21systemd-run07:10:011
29560399231,18cyclictest319049-21sshd08:29:200
29560399231,18cyclictest319049-21sshd08:29:200
295603992310,10cyclictest0-21swapper/008:24:260
295604992216,3cyclictest0-21swapper/108:24:151
295603992216,4cyclictest306646-21fschecks_time07:49:120
29560399221,16cyclictest310082-21ntpq07:59:270
295603991812,3cyclictest0-21swapper/008:33:490
29560699176,6cyclictest0-21swapper/310:59:203
29560699172,8cyclictest0-21swapper/312:24:253
29560699172,7cyclictest0-21swapper/311:44:143
29560699162,7cyclictest0-21swapper/311:59:183
29560699156,5cyclictest0-21swapper/311:29:343
29560699156,5cyclictest0-21swapper/311:19:283
29560699152,8cyclictest0-21swapper/310:54:283
29560699152,7cyclictest0-21swapper/312:04:273
29560699152,7cyclictest0-21swapper/310:04:203
29560699152,6cyclictest0-21swapper/308:39:203
29560699151,7cyclictest0-21swapper/309:34:243
29560399150,12cyclictest295649-21latency_hist07:10:020
29560699147,4cyclictest0-21swapper/311:35:013
29560699147,4cyclictest0-21swapper/309:30:013
29560699146,4cyclictest0-21swapper/310:20:013
29560699141,8cyclictest0-21swapper/311:39:203
29560699141,7cyclictest0-21swapper/312:34:143
29560699141,7cyclictest0-21swapper/307:59:253
29560699141,6cyclictest0-21swapper/310:14:193
29560699141,6cyclictest0-21swapper/310:14:193
29560699141,6cyclictest0-21swapper/307:19:333
295606991410,2cyclictest0-21swapper/307:10:013
29560699137,4cyclictest0-21swapper/311:54:273
29560699137,4cyclictest0-21swapper/311:09:203
29560699137,4cyclictest0-21swapper/310:40:243
29560699136,4cyclictest0-21swapper/312:27:293
29560699136,4cyclictest0-21swapper/312:10:013
29560699136,4cyclictest0-21swapper/311:45:563
29560699136,4cyclictest0-21swapper/311:15:033
29560699136,4cyclictest0-21swapper/311:04:053
29560699136,4cyclictest0-21swapper/310:49:133
29560699136,4cyclictest0-21swapper/310:31:153
29560699136,4cyclictest0-21swapper/310:31:153
29560699136,4cyclictest0-21swapper/310:24:133
29560699136,4cyclictest0-21swapper/310:24:133
29560699136,4cyclictest0-21swapper/309:57:033
29560699136,4cyclictest0-21swapper/309:54:253
29560699136,4cyclictest0-21swapper/309:22:313
29560699136,4cyclictest0-21swapper/309:19:133
29560699136,4cyclictest0-21swapper/309:00:013
29560699136,4cyclictest0-21swapper/308:54:313
29560699136,4cyclictest0-21swapper/308:49:313
29560699136,4cyclictest0-21swapper/308:34:383
29560699136,4cyclictest0-21swapper/308:19:193
29560699132,7cyclictest0-21swapper/311:24:203
29560699132,6cyclictest0-21swapper/310:09:193
29560699132,6cyclictest0-21swapper/310:09:193
29560699132,6cyclictest0-21swapper/308:24:203
29560699131,7cyclictest0-21swapper/312:39:183
29560699131,7cyclictest0-21swapper/310:39:203
29560699131,6cyclictest0-21swapper/310:29:103
29560699131,6cyclictest0-21swapper/309:44:183
29560699131,6cyclictest0-21swapper/309:44:183
29560699131,6cyclictest0-21swapper/309:39:233
29560699131,6cyclictest0-21swapper/309:09:183
29560699131,6cyclictest0-21swapper/308:29:173
29560699131,6cyclictest0-21swapper/308:29:173
29560699131,6cyclictest0-21swapper/307:49:203
29560699131,6cyclictest0-21swapper/307:14:183
29560699127,3cyclictest0-21swapper/308:45:023
29560699127,3cyclictest0-21swapper/308:14:243
29560699127,3cyclictest0-21swapper/308:09:233
29560699127,3cyclictest0-21swapper/307:44:313
29560699126,4cyclictest0-21swapper/309:14:233
29560699126,4cyclictest0-21swapper/307:34:193
29560699126,3cyclictest0-21swapper/307:29:423
29560699126,3cyclictest0-21swapper/307:29:423
29560699121,7cyclictest0-21swapper/312:19:113
29560699116,4cyclictest0-21swapper/312:10:113
29560699116,3cyclictest0-21swapper/309:50:023
29560699116,3cyclictest0-21swapper/309:50:023
29560699116,3cyclictest0-21swapper/308:05:013
29560699116,3cyclictest0-21swapper/307:54:063
29560699111,7cyclictest0-21swapper/309:05:023
29560699106,3cyclictest0-21swapper/307:39:103
29560699106,3cyclictest0-21swapper/307:20:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional