You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #2, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack2slot7s.osadl.org (updated Fri May 02, 2025 00:41:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
730476995951,5cyclictest788375-21munin-run21:40:011
730476995445,5cyclictest773838-21ls21:20:011
730475994837,6cyclictest151ktimers/021:07:250
730475994831,14cyclictest875427-21latency_hist23:40:020
730477994638,5cyclictest817384-21sendmail22:20:012
730476994534,7cyclictest422-21dbus-daemon22:40:011
730477994436,5cyclictest749912-21munin-run20:15:012
730477994436,5cyclictest749912-21munin-run20:15:012
730477994436,3cyclictest0-21swapper/219:40:012
730477994435,6cyclictest422-21dbus-daemon19:15:012
730477994421,14cyclictest911610-21sshd00:31:462
73047699446,29cyclictest909357-21ssh00:27:231
730476994436,5cyclictest833215-21aten_r2power_vo22:42:121
730476994334,6cyclictest453-21systemd-logind19:30:011
73047699431,31cyclictest905936-21ssh00:22:211
730475994331,9cyclictest859131-21modprobe23:17:290
730475994327,13cyclictest788375-21munin-run21:40:010
730476994234,5cyclictest914048-21idleruntime-cro00:35:011
730476994234,5cyclictest888718-21sed23:57:351
730476994234,5cyclictest790932-21unixbench_singl21:42:391
730476994234,5cyclictest790932-21unixbench_singl21:42:391
730476994233,6cyclictest770227-21/usr/sbin/munin21:15:021
730475994233,5cyclictest0-21swapper/022:37:260
730476994134,4cyclictest812586-21unixbench_singl22:12:311
730476994134,4cyclictest0-21swapper/121:25:011
730476994133,5cyclictest751463-21latency_hist20:20:021
730476994133,5cyclictest421-21avahi-daemon22:20:011
730476994132,5cyclictest863303-21sed23:22:351
730476994132,5cyclictest863303-21sed23:22:351
730477994034,3cyclictest0-21swapper/222:42:222
730476994033,4cyclictest0-21swapper/121:47:251
730476994032,5cyclictest1-21systemd23:00:011
730475994034,3cyclictest151ktimers/021:57:200
730475994033,4cyclictest151ktimers/020:42:270
730477993931,5cyclictest886252-21munin-run23:55:012
730477993927,7cyclictest0-21swapper/223:37:242
73047699396,6cyclictest171rcu_preempt23:02:261
73047699394,25cyclictest808444-21ssh22:07:281
730476993930,6cyclictest841307-21systemctl22:52:331
730476993930,6cyclictest422-21dbus-daemon22:04:361
730476993927,8cyclictest422-21dbus-daemon20:35:021
730476993924,9cyclictest899113-21sh00:12:271
730475993933,4cyclictest1-21systemd20:00:020
730477993830,5cyclictest868029-21cron23:30:002
730477993830,5cyclictest868029-21cron23:30:002
73047699385,28cyclictest866302-21ssh23:27:221
73047699385,28cyclictest866302-21ssh23:27:221
730476993833,2cyclictest0-21swapper/123:40:011
730476993830,5cyclictest856067-21head23:12:371
730476993829,6cyclictest421-21avahi-daemon00:15:241
730475993831,4cyclictest151ktimers/022:17:260
730475993830,5cyclictest869540-21cut23:32:160
730477993729,5cyclictest422-21dbus-daemon21:55:022
730476993729,5cyclictest757447-21systemctl20:40:011
730476993728,6cyclictest3149057-21cups-browsed22:27:261
730476993717,14cyclictest3149057-21cups-browsed22:22:251
730476993717,14cyclictest3149057-21cups-browsed22:22:251
730475993728,6cyclictest422-21dbus-daemon23:10:020
730475993727,7cyclictest915665-21missed_timers00:37:180
730477993627,6cyclictest773887-21systemd-run21:20:012
730477993626,7cyclictest3149057-21cups-browsed22:07:302
730476993629,3cyclictest0-21swapper/123:08:581
730475993629,5cyclictest421-21avahi-daemon22:27:150
730475993629,4cyclictest807941-21fschecks_time22:07:160
730475993627,7cyclictest742431-21/usr/sbin/munin19:50:010
730477993529,3cyclictest0-21swapper/221:27:242
730477993527,5cyclictest898759-21latency00:12:162
730477993526,6cyclictest737964-21systemd-run19:35:012
730476993529,3cyclictest0-21swapper/121:30:011
730476993528,4cyclictest860767-21cron23:20:011
730476993528,3cyclictest0-21swapper/100:05:011
730476993528,3cyclictest0-21swapper/100:05:011
730476993525,8cyclictest763236-21timerwakeupswit20:57:351
730475993528,4cyclictest828276-21idleruntime-cro22:35:010
730475993528,4cyclictest790791-21timerwakeupswit21:42:380
730475993528,4cyclictest790791-21timerwakeupswit21:42:380
730475993526,7cyclictest904207-21munin-plugin-st00:20:010
73047599351,27cyclictest906026-21ssh00:22:250
730477993428,3cyclictest0-21swapper/219:25:012
730477993425,6cyclictest859716-21unixbench_singl23:17:392
730477993422,9cyclictest750732-21needreboot20:17:212
730476993426,5cyclictest801646-21unixbench_multi21:57:331
730476993425,6cyclictest766432-21diskmemload00:37:291
730476993425,6cyclictest733490-21systemd-run19:20:011
730476993425,6cyclictest421-21avahi-daemon21:33:571
730476993425,5cyclictest746517-21mailstats20:02:301
730475993428,2cyclictest0-21swapper/000:00:010
730475993423,9cyclictest794544-21unixbench_singl21:47:360
730475993423,5cyclictest956-21mta-sts-daemon00:12:220
730475993422,7cyclictest151ktimers/021:27:290
730475993416,13cyclictest151ktimers/022:57:240
730475993415,12cyclictest12450irq/32-eth022:02:250
730477993325,5cyclictest857229-21latency_hist23:15:022
730477993325,5cyclictest839147-21munin-plugin-st22:50:012
730477993324,6cyclictest831854-21ls22:40:012
730477993324,6cyclictest766421-21cron21:10:012
730477993324,6cyclictest491-21gdbus21:47:122
73047699333,23cyclictest837398-21ssh22:47:221
730476993325,5cyclictest765463-21fschecks_count21:07:151
730476993324,6cyclictest894880-21sh00:07:121
730476993324,6cyclictest739468-21systemd-run19:40:011
730476993322,8cyclictest826363-21sh22:32:181
730475993326,4cyclictest791333-21kworker/u8:9-xprtiod21:55:010
730475993325,5cyclictest0-21swapper/021:22:160
730475993322,8cyclictest877789-21timerwakeupswit23:42:320
730475993320,8cyclictest12450irq/32-eth000:32:240
73047799323,25cyclictest790474-21grep21:42:332
73047799323,25cyclictest790474-21grep21:42:332
730477993227,2cyclictest0-21swapper/221:00:012
730477993226,3cyclictest0-21swapper/223:00:012
730477993226,3cyclictest0-21swapper/222:12:192
730477993225,4cyclictest760472-21latency_hist20:50:022
730477993225,4cyclictest760472-21latency_hist20:50:022
730477993225,4cyclictest731978-21kworker/u8:6+xprtiod20:05:022
730477993224,5cyclictest871712-21munin-plugin-st23:35:012
730477993224,5cyclictest756485-21fschecks_time20:37:132
730477993224,5cyclictest730614-21latency_hist19:10:032
730477993224,3cyclictest0-21swapper/200:17:222
730477993223,6cyclictest3149057-21cups-browsed22:29:482
730477993223,5cyclictest0-21swapper/223:57:282
73047699328,20cyclictest797635-21ssh21:52:311
73047699323,20cyclictest3149057-21cups-browsed20:27:261
730476993224,5cyclictest730481-21systemd-run19:10:011
730476993222,7cyclictest881354-21timerandwakeup23:47:381
730476993220,9cyclictest749267-21ntp_kernel_err20:12:211
730476993220,9cyclictest749267-21ntp_kernel_err20:12:211
730476993215,11cyclictest869509-21df_inode23:32:141
73047699321,23cyclictest3149052-21cupsd19:52:201
730475993225,4cyclictest910776-21sh00:30:010
730475993224,5cyclictest838002-21grep22:47:330
730475993219,11cyclictest886240-21cron23:55:010
730477993126,2cyclictest0-21swapper/200:25:012
730477993124,4cyclictest959-21mta-sts-daemon20:52:142
730477993124,4cyclictest957-21mta-sts-daemon00:38:072
730477993124,4cyclictest955-21mta-sts-daemon23:07:162
730477993124,4cyclictest770204-21munin-plugin-st21:15:012
730477993123,5cyclictest828265-21ssh22:35:012
730477993123,5cyclictest775179-21ssh21:22:162
730477993123,5cyclictest739970-21fschecks_count19:42:202
730477993123,5cyclictest739970-21fschecks_count19:42:202
730477993123,5cyclictest3149057-21cups-browsed21:37:332
730477993123,5cyclictest266-21systemd-journal23:45:012
730477993122,6cyclictest0-21swapper/200:27:272
730477993120,9cyclictest841538-21timerwakeupswit22:52:362
730476993125,3cyclictest0-21swapper/121:02:301
730476993123,5cyclictest731954-21idleruntime-cro19:15:011
730475993126,3cyclictest713164-21kworker/u8:12-nfsiod19:20:020
730475993125,3cyclictest0-21swapper/021:17:290
73047599312,20cyclictest891903-21ntpq00:02:250
73047599312,20cyclictest891903-21ntpq00:02:250
73047599311,25cyclictest855540-21ssh23:12:290
73047599311,23cyclictest833921-21ssh22:42:290
730477993025,2cyclictest0-21swapper/222:25:012
730477993025,2cyclictest0-21swapper/222:25:012
730477993024,3cyclictest0-21swapper/220:35:012
730477993023,4cyclictest956-21mta-sts-daemon21:31:312
730477993022,5cyclictest802831-21sendmail22:00:012
730477993022,5cyclictest753454-21fschecks_time20:27:172
73047699305,17cyclictest437-21mta-sts-daemon19:32:201
730476993021,6cyclictest886244-21sh23:55:011
73047599302,21cyclictest768467-21ssh21:12:240
73047599301,9cyclictest955-21mta-sts-daemon23:27:210
73047599301,9cyclictest955-21mta-sts-daemon23:27:210
73047599301,11cyclictest875399-21kworker/u8:2+xprtiod23:47:310
730477992922,4cyclictest957-21mta-sts-daemon23:47:112
730477992922,4cyclictest745667-21kworker/u8:12-xprtiod21:02:192
730477992921,5cyclictest957-21mta-sts-daemon19:47:342
730476992925,2cyclictest0-21swapper/120:55:011
730476992922,3cyclictest0-21swapper/123:42:201
730475992923,4cyclictest748958-21fschecks_time20:12:160
730475992923,4cyclictest748958-21fschecks_time20:12:160
730475992923,4cyclictest736524-21latency_hist19:30:020
730477992819,6cyclictest3149057-21cups-browsed22:00:232
730477992818,7cyclictest848225-21ntp_kernel_err23:02:202
730477992817,8cyclictest955-21mta-sts-daemon23:21:502
730477992817,8cyclictest955-21mta-sts-daemon23:21:502
730476992823,2cyclictest0-21swapper/119:25:011
730476992820,5cyclictest748418-21sh20:10:011
730476992818,7cyclictest740937-21idleruntime-cro19:45:011
730476992818,7cyclictest740937-21idleruntime-cro19:45:011
73047599284,15cyclictest12450irq/32-eth020:27:300
730475992823,2cyclictest0-21swapper/020:20:010
730475992822,4cyclictest841642-21unixbench_singl22:52:370
730475992819,7cyclictest821051-21ssh22:25:010
730475992819,7cyclictest821051-21ssh22:25:010
730475992813,11cyclictest12450irq/32-eth020:57:270
73047899272,12cyclictest0-21swapper/300:12:273
730477992722,2cyclictest0-21swapper/220:00:012
730477992721,3cyclictest957-21mta-sts-daemon00:08:592
73047699275,16cyclictest437-21mta-sts-daemon19:57:251
730476992718,6cyclictest758020-21idleruntime20:42:191
730475992721,4cyclictest752800-21tail20:22:340
730475992717,7cyclictest848038-21latency23:02:170
730475992714,9cyclictest12450irq/32-eth020:52:260
730477992620,3cyclictest0-21swapper/219:20:012
730477992619,4cyclictest955-21mta-sts-daemon20:22:232
73047799261,20cyclictest891875-21sh00:02:242
73047799261,20cyclictest891875-21sh00:02:242
730476992620,3cyclictest0-21swapper/120:22:291
73047599262,20cyclictest783052-21ssh21:32:280
73047599262,18cyclictest811895-21ntpq22:12:170
730475992620,3cyclictest956-21mta-sts-daemon19:52:200
730475992620,3cyclictest0-21swapper/020:40:010
730475992617,7cyclictest755939-21cron20:35:010
730475992617,5cyclictest862702-21ntpq23:22:250
730475992617,5cyclictest862702-21ntpq23:22:250
730477992517,5cyclictest758839-21unixbench_multi20:42:362
730476992519,4cyclictest720775-21kworker/u8:10-xprtiod20:50:011
730476992519,4cyclictest720775-21kworker/u8:10-xprtiod20:50:011
730475992515,8cyclictest0-21swapper/019:25:010
73047599251,20cyclictest895372-21ntpq00:07:220
730475992418,4cyclictest731526-21sendmail_mailqu19:12:280
730475992417,4cyclictest422-21dbus-daemon19:34:500
73047799234,13cyclictest437-21mta-sts-daemon19:27:192
730477992314,6cyclictest748413-21cron20:10:012
730475992315,5cyclictest748423-21munin-plugin-st20:10:010
73047899225,10cyclictest0-21swapper/322:22:253
73047899225,10cyclictest0-21swapper/322:22:253
730477992216,3cyclictest0-21swapper/219:52:162
730476992214,5cyclictest741640-21latency19:47:201
73047599223,12cyclictest437-21mta-sts-daemon20:47:250
73047599223,12cyclictest437-21mta-sts-daemon20:47:250
73047599222,15cyclictest764435-21ntpq21:02:300
730475992217,2cyclictest0-21swapper/019:10:020
730475992211,8cyclictest0-21swapper/019:37:300
730475992210,2cyclictest0-21swapper/019:42:290
730475992210,2cyclictest0-21swapper/019:42:290
73047899212,12cyclictest0-21swapper/322:07:303
730475992117,2cyclictest0-21swapper/020:02:340
73047899203,10cyclictest0-21swapper/321:27:293
73047899202,10cyclictest0-21swapper/322:47:273
73047899192,10cyclictest0-21swapper/323:22:233
73047899192,10cyclictest0-21swapper/323:22:233
73047899191,11cyclictest0-21swapper/321:17:223
73047899184,8cyclictest0-21swapper/323:12:263
73047899184,8cyclictest0-21swapper/322:12:233
73047899183,8cyclictest0-21swapper/321:42:333
73047899183,8cyclictest0-21swapper/321:42:333
73047899182,9cyclictest0-21swapper/319:17:233
73047899182,8cyclictest0-21swapper/323:17:253
73047899182,8cyclictest0-21swapper/321:32:283
73047899182,11cyclictest0-21swapper/300:02:223
73047899182,11cyclictest0-21swapper/300:02:223
73047899181,9cyclictest0-21swapper/320:27:273
73047899173,8cyclictest0-21swapper/321:37:313
73047899172,9cyclictest0-21swapper/322:52:263
73047899172,9cyclictest0-21swapper/322:17:243
73047899172,8cyclictest0-21swapper/322:37:273
73047899172,8cyclictest0-21swapper/300:17:233
73047899172,7cyclictest0-21swapper/322:57:213
73047899172,7cyclictest0-21swapper/300:07:203
73047899171,9cyclictest0-21swapper/322:32:163
73047899171,9cyclictest0-21swapper/300:37:303
73047899171,8cyclictest0-21swapper/323:47:243
73047899171,8cyclictest0-21swapper/321:47:203
73047899171,10cyclictest0-21swapper/323:52:323
73047899171,10cyclictest0-21swapper/321:22:253
73047899171,10cyclictest0-21swapper/320:07:233
73047899162,9cyclictest0-21swapper/322:41:293
73047899162,8cyclictest0-21swapper/323:27:193
73047899162,8cyclictest0-21swapper/323:27:193
73047899162,8cyclictest0-21swapper/323:07:243
73047899162,8cyclictest0-21swapper/321:52:323
73047899162,8cyclictest0-21swapper/300:00:013
73047899162,7cyclictest0-21swapper/321:57:153
73047899162,7cyclictest0-21swapper/320:17:263
73047899162,7cyclictest0-21swapper/300:27:193
73047899161,9cyclictest0-21swapper/323:37:243
73047899161,9cyclictest0-21swapper/300:32:213
73047899161,9cyclictest0-21swapper/300:22:223
73047899161,8cyclictest0-21swapper/322:27:243
73047899161,8cyclictest0-21swapper/320:42:293
73047899161,8cyclictest0-21swapper/320:32:253
73047899161,8cyclictest0-21swapper/319:47:283
73047899161,7cyclictest0-21swapper/323:42:213
73047899153,8cyclictest0-21swapper/323:32:233
73047899151,8cyclictest0-21swapper/323:02:253
73047899151,8cyclictest0-21swapper/320:02:293
73047899151,7cyclictest0-21swapper/319:32:203
73047899151,6cyclictest0-21swapper/319:27:193
73047899151,6cyclictest0-21swapper/319:22:263
73047899143,7cyclictest0-21swapper/321:12:233
73047899142,6cyclictest0-21swapper/322:02:113
73047899141,7cyclictest0-21swapper/319:12:233
73047899141,6cyclictest0-21swapper/320:12:223
73047899141,6cyclictest0-21swapper/320:12:223
73047899141,6cyclictest0-21swapper/319:57:213
73047899131,7cyclictest0-21swapper/321:07:213
73047899131,6cyclictest0-21swapper/320:57:233
73047899131,6cyclictest0-21swapper/320:47:253
73047899131,6cyclictest0-21swapper/320:47:253
73047899131,6cyclictest0-21swapper/319:42:253
73047899131,6cyclictest0-21swapper/319:42:253
73047899131,6cyclictest0-21swapper/319:37:193
73047899122,6cyclictest0-21swapper/320:37:203
73047899121,6cyclictest0-21swapper/321:02:243
73047899111,6cyclictest0-21swapper/319:52:223
73047899101,5cyclictest0-21swapper/320:22:213
73047899100,5cyclictest0-21swapper/320:52:223
7304789981,4cyclictest0-21swapper/319:10:033
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional