You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-04 - 05:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot5s.osadl.org (updated Fri Jul 04, 2025 00:44:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3686752995745,7cyclictest3786844-21sendmail00:00:003
3686748995347,4cyclictest3754157-21kworker/u8:2+events_unbound22:27:562
3686743994937,7cyclictest3711364-21munin-run20:20:011
3686748994842,4cyclictest3713023-21kworker/u8:0+events_unbound20:27:422
3686752994732,12cyclictest391ktimers/319:15:143
3686743994736,7cyclictest231ktimers/122:50:151
3686739994737,6cyclictest0-21swapper/019:15:020
3686748994436,5cyclictest3711364-21munin-run20:20:002
3686743994429,10cyclictest231ktimers/122:00:141
3686743994429,10cyclictest231ktimers/122:00:131
3686748994334,6cyclictest3700671-21grep19:45:282
3686748994332,7cyclictest3697733-21sendmail19:40:002
3686748994330,9cyclictest311ktimers/200:15:152
3686752994232,6cyclictest3704542-21cut20:00:003
3686748994232,7cyclictest3728419-21sh21:10:012
3686748994225,3cyclictest131rcu_preempt19:35:012
3686743994232,7cyclictest3697306-21unixbench_singl19:35:271
3686743994231,8cyclictest231ktimers/123:25:111
3686743994231,8cyclictest231ktimers/123:25:111
3686743994231,7cyclictest3706698-21fschecks_time20:05:181
3686743994231,7cyclictest3706698-21fschecks_time20:05:181
3686752994133,4cyclictest0-21swapper/323:20:003
3686752994117,6cyclictest131rcu_preempt21:10:143
3686752994034,4cyclictest3719864-21kworker/u8:2+events_unbound20:57:473
3686748994019,12cyclictest131rcu_preempt19:29:522
3686743994031,6cyclictest231ktimers/121:50:131
3686743994029,8cyclictest231ktimers/122:30:141
3686752993933,4cyclictest3761502-21kworker/u8:1+events_unbound23:13:023
3686752993932,4cyclictest3780014-21sh23:39:593
3686752993930,7cyclictest3749071-21kworker/u8:0+events_unbound22:12:563
3686752993930,7cyclictest3749071-21kworker/u8:0+events_unbound22:12:563
3686752993928,7cyclictest0-21swapper/319:30:273
3686752993924,11cyclictest391ktimers/321:20:163
3686748993922,10cyclictest3757633-21kworker/u8:0+events_unbound22:47:432
3686743993931,4cyclictest0-21swapper/122:42:561
3686743993927,9cyclictest231ktimers/100:15:161
3686739993933,4cyclictest3772276-21kworker/u8:3+events_unbound23:33:080
3686739993929,7cyclictest3694305-21latency_hist19:30:010
3686739993928,8cyclictest440-21cron21:30:000
3686752993832,3cyclictest3698196-21runrttasks19:40:193
3686748993832,4cyclictest3769637-21kworker/u8:2+events_unbound23:13:432
3686748993832,4cyclictest3721583-21kworker/u8:1+events_unbound20:52:462
3686743993827,8cyclictest231ktimers/122:10:141
3686743993827,8cyclictest231ktimers/122:10:141
3686739993831,4cyclictest3697718-21munin-run19:40:000
3686739993829,5cyclictest3749312-21cron22:10:010
3686739993829,5cyclictest3749312-21cron22:10:000
3686752993723,5cyclictest3761502-21kworker/u8:1+events_unbound22:57:063
3686752993719,11cyclictest131rcu_preempt22:45:183
3686752993712,13cyclictest131rcu_preempt21:30:123
368674899376,19cyclictest32-21ksoftirqd/221:55:012
3686748993731,4cyclictest3779756-21kworker/u8:0+events_unbound23:38:092
3686748993723,4cyclictest131rcu_preempt21:55:222
3686748993722,4cyclictest131rcu_preempt20:46:302
3686748993721,5cyclictest131rcu_preempt19:15:172
3686748993719,11cyclictest131rcu_preempt21:45:242
3686748993717,13cyclictest131rcu_preempt19:20:162
3686743993728,4cyclictest0-21swapper/121:25:161
3686752993628,6cyclictest3766064-21kworker/u8:0+events_unbound23:08:033
3686752993611,14cyclictest40-21ksoftirqd/321:40:013
3686748993628,5cyclictest3724533-21unixbench_multi20:55:272
3686748993628,4cyclictest0-21swapper/221:35:282
3686748993619,4cyclictest131rcu_preempt22:20:152
3686748993618,6cyclictest3717156-21kworker/u8:1+events_unbound20:35:532
3686748993617,5cyclictest131rcu_preempt23:05:012
3686748993616,5cyclictest131rcu_preempt22:05:242
3686748993616,5cyclictest131rcu_preempt22:05:242
3686743993630,4cyclictest3773198-21cron23:20:011
3686743993625,7cyclictest3766358-21cron23:00:001
368675299353,11cyclictest40-21ksoftirqd/321:25:123
3686752993528,5cyclictest3797520-21kworker/u8:0+events_unbound00:33:193
3686752993528,5cyclictest3797520-21kworker/u8:0+events_unbound00:33:193
3686752993527,4cyclictest3709665-21munin-plugin-st20:15:003
3686752993527,4cyclictest3709665-21munin-plugin-st20:15:003
3686752993518,5cyclictest131rcu_preempt00:05:003
3686748993528,5cyclictest3790103-21kworker/u8:0+events_unbound00:08:132
3686748993528,5cyclictest3742071-21kworker/u8:1+events_unbound22:04:442
3686748993528,5cyclictest3742071-21kworker/u8:1+events_unbound22:04:442
3686748993527,4cyclictest0-21swapper/221:00:322
3686748993522,5cyclictest3778017-21kworker/u8:1+events_unbound23:41:542
3686748993519,5cyclictest131rcu_preempt21:25:172
3686748993517,4cyclictest131rcu_preempt22:16:002
3686748993516,10cyclictest131rcu_preempt22:34:572
3686748993515,5cyclictest131rcu_preempt23:20:142
3686743993527,4cyclictest0-21swapper/100:39:591
3686743993525,7cyclictest3740154-21grep21:40:281
3686739993527,4cyclictest0-21swapper/000:00:270
368675299347,22cyclictest391ktimers/322:55:003
3686752993428,4cyclictest3740320-21kworker/u8:2+events_unbound21:52:513
3686752993427,5cyclictest3772276-21kworker/u8:3+events_unbound23:23:043
3686752993422,8cyclictest0-21swapper/323:05:013
3686752993416,4cyclictest131rcu_preempt20:30:133
3686752993414,3cyclictest131rcu_preempt22:25:003
368674899344,15cyclictest32-21ksoftirqd/220:40:222
3686748993425,4cyclictest0-21swapper/222:40:212
3686748993421,4cyclictest131rcu_preempt00:35:262
3686748993419,4cyclictest131rcu_preempt22:35:152
3686748993418,7cyclictest3766064-21kworker/u8:0+events_unbound22:56:422
3686748993417,5cyclictest131rcu_preempt00:30:222
3686748993417,5cyclictest131rcu_preempt00:30:222
3686748993416,4cyclictest131rcu_preempt00:26:452
3686748993416,4cyclictest131rcu_preempt00:26:452
3686748993414,10cyclictest131rcu_preempt20:05:172
3686748993414,10cyclictest131rcu_preempt20:05:162
3686743993423,7cyclictest3731415-21grep21:15:281
3686752993325,5cyclictest0-21swapper/322:35:013
3686752993325,5cyclictest0-21swapper/300:40:003
3686752993325,4cyclictest0-21swapper/300:15:293
3686752993323,6cyclictest3726309-21grep21:00:313
3686752993323,6cyclictest0-21swapper/321:10:003
3686752993321,7cyclictest0-21swapper/319:10:273
3686752993319,8cyclictest131rcu_preempt00:25:163
3686752993319,8cyclictest131rcu_preempt00:25:163
3686752993319,7cyclictest3711298-21kworker/u8:1+events_unbound20:25:213
3686748993326,4cyclictest3713085-21idleruntime-cro20:25:002
3686748993325,4cyclictest0-21swapper/221:35:012
3686748993319,4cyclictest131rcu_preempt20:10:222
3686748993319,4cyclictest131rcu_preempt20:10:212
3686748993318,5cyclictest131rcu_preempt23:45:232
3686748993318,5cyclictest131rcu_preempt00:00:172
3686748993318,4cyclictest131rcu_preempt21:20:222
3686748993317,7cyclictest3776251-21kworker/u8:0+events_unbound23:26:582
3686748993317,7cyclictest3776251-21kworker/u8:0+events_unbound23:26:582
3686748993317,6cyclictest3688149-21kworker/u8:2+events_unbound19:12:452
3686748993316,5cyclictest3771356-21kworker/u8:0+events_unbound23:18:302
3686748993314,4cyclictest131rcu_preempt23:30:162
3686748993313,5cyclictest131rcu_preempt00:00:012
3686748993313,10cyclictest131rcu_preempt21:15:282
3686748993310,13cyclictest32-21ksoftirqd/219:55:202
3686743993323,7cyclictest3732272-21grep21:20:191
3686743993323,5cyclictest0-21swapper/122:15:261
3686743993322,7cyclictest0-21swapper/100:00:001
3686739993325,6cyclictest3686730-21cyclictest21:35:270
3686739993323,7cyclictest3760847-21grep22:40:300
3686739993322,8cyclictest3785130-21sh23:55:010
3686739993322,7cyclictest3793411-21grep00:15:290
3686752993223,4cyclictest0-21swapper/320:20:183
3686752993222,6cyclictest0-21swapper/322:00:013
3686752993220,4cyclictest131rcu_preempt20:35:213
3686752993219,4cyclictest131rcu_preempt20:15:193
3686752993212,14cyclictest0-21swapper/323:45:173
368674899327,14cyclictest32-21ksoftirqd/222:50:182
3686748993217,4cyclictest131rcu_preempt22:13:172
3686748993217,4cyclictest131rcu_preempt22:13:172
3686748993216,4cyclictest131rcu_preempt20:30:202
3686748993211,11cyclictest131rcu_preempt21:40:182
3686743993224,4cyclictest0-21swapper/123:45:241
3686743993224,3cyclictest0-21swapper/123:40:281
3686739993224,4cyclictest0-21swapper/023:00:260
3686739993222,7cyclictest3744195-21apt-key21:55:020
368675299319,5cyclictest131rcu_preempt20:40:263
368675299318,14cyclictest40-21ksoftirqd/323:40:213
3686752993123,6cyclictest3695973-21kworker/u8:1+events_unbound19:36:443
3686752993122,5cyclictest0-21swapper/320:10:013
3686752993122,5cyclictest0-21swapper/320:10:013
3686752993118,7cyclictest3776251-21kworker/u8:0+events_unbound23:30:293
3686752993115,9cyclictest131rcu_preempt20:45:313
368674899319,13cyclictest3701473-21df_inode19:50:172
368674899319,13cyclictest3701473-21df_inode19:50:172
368674899319,13cyclictest32-21ksoftirqd/223:50:242
368674899314,5cyclictest32-21ksoftirqd/200:15:002
3686748993119,3cyclictest131rcu_preempt00:20:222
3686748993118,4cyclictest3769637-21kworker/u8:2+events_unbound23:08:322
3686748993116,5cyclictest3692245-21kworker/u8:2+events_unbound19:42:242
3686739993122,4cyclictest3732304-21grep21:20:200
3686752993018,5cyclictest131rcu_preempt19:52:383
3686752993018,5cyclictest131rcu_preempt19:52:373
3686752993018,4cyclictest131rcu_preempt00:20:203
3686752993016,5cyclictest131rcu_preempt22:35:263
3686752993015,9cyclictest131rcu_preempt22:08:093
3686752993015,9cyclictest131rcu_preempt22:08:093
3686752993015,9cyclictest131rcu_preempt22:05:003
3686752993015,9cyclictest131rcu_preempt22:05:003
3686752993012,9cyclictest131rcu_preempt22:45:013
3686752993011,12cyclictest131rcu_preempt20:50:303
3686752993010,5cyclictest131rcu_preempt00:10:143
3686748993023,3cyclictest0-21swapper/221:15:002
3686743993022,4cyclictest0-21swapper/120:25:261
3686743993022,4cyclictest0-21swapper/119:45:191
3686743993021,4cyclictest0-21swapper/122:30:011
3686739993023,4cyclictest3745498-21unixbench_singl21:55:260
3686739993022,4cyclictest0-21swapper/020:45:330
3686739993020,7cyclictest3766371-21sendmail23:00:010
368675299299,12cyclictest40-21ksoftirqd/319:45:203
368675299298,13cyclictest40-21ksoftirqd/322:25:243
3686752992923,4cyclictest3692870-21fschecks_time19:25:193
3686752992912,9cyclictest0-21swapper/321:40:263
3686743992921,4cyclictest0-21swapper/120:55:271
3686739992922,4cyclictest3757843-21sh22:35:010
3686739992921,5cyclictest0-21swapper/020:15:310
3686752992813,9cyclictest131rcu_preempt20:00:173
3686752992810,9cyclictest131rcu_preempt21:45:253
3686743992820,4cyclictest0-21swapper/119:40:271
3686743992820,4cyclictest0-21swapper/100:15:011
3686739992821,3cyclictest0-21swapper/023:25:150
3686739992821,3cyclictest0-21swapper/023:25:140
3686739992820,4cyclictest3750779-21kworker/u8:1+events_unbound22:17:550
3686739992819,6cyclictest0-21swapper/000:05:260
3686739992818,7cyclictest3800914-21munin-plugin-st00:40:000
3686739992816,7cyclictest3729667-21grep21:10:280
3686739992815,10cyclictest121ktimers/019:45:160
3686752992713,7cyclictest40-21ksoftirqd/323:30:013
3686752992713,7cyclictest40-21ksoftirqd/323:30:013
3686752992712,9cyclictest131rcu_preempt19:23:033
3686743992719,4cyclictest3737080-21grep21:35:001
3686743992718,5cyclictest0-21swapper/123:20:271
3686743992718,4cyclictest0-21swapper/121:10:291
3686743992717,6cyclictest3726254-21grep21:00:311
3686743992715,8cyclictest3779578-21grep23:35:251
3686739992723,2cyclictest0-21swapper/023:45:130
3686739992720,3cyclictest0-21swapper/023:39:590
3686739992720,3cyclictest0-21swapper/020:00:000
3686739992719,5cyclictest0-21swapper/020:50:310
3686739992716,9cyclictest3758073-21/usr/sbin/munin22:35:160
3686743992620,3cyclictest0-21swapper/119:15:171
3686743992619,3cyclictest0-21swapper/119:15:021
3686743992615,7cyclictest3723294-21sh20:55:001
3686739992619,4cyclictest3719898-21cron20:45:000
3686739992618,4cyclictest0-21swapper/020:00:300
368673999261,22cyclictest3774020-21ntp_states23:20:240
368675299251,7cyclictest0-21swapper/300:05:143
3686743992519,3cyclictest0-21swapper/100:30:211
3686743992519,3cyclictest0-21swapper/100:30:211
3686743992518,4cyclictest3784679-21grep23:50:241
3686743992518,4cyclictest0-21swapper/123:00:251
3686743992518,3cyclictest0-21swapper/119:25:281
3686743992517,4cyclictest0-21swapper/122:45:171
3686743992517,3cyclictest0-21swapper/100:05:181
368673999251,8cyclictest3726245-21grep21:00:310
3686739992518,4cyclictest0-21swapper/021:30:260
3686739992517,5cyclictest0-21swapper/022:25:240
368673999251,5cyclictest3716821-21df20:35:160
3686739992515,6cyclictest0-21swapper/000:00:000
3686739992514,8cyclictest3686730-21cyclictest20:55:140
368673999251,3cyclictest3795828-21apt-get00:25:130
368673999251,3cyclictest3795828-21apt-get00:25:130
368673999251,20cyclictest3690672-21cron19:20:010
368673999251,13cyclictest0-21swapper/021:05:260
368673999250,21cyclictest0-21swapper/020:20:200
3686743992416,6cyclictest231ktimers/122:00:001
3686743992416,4cyclictest0-21swapper/123:10:251
3686743992416,4cyclictest0-21swapper/119:20:281
3686739992418,4cyclictest3757633-21kworker/u8:0+events_unbound22:46:210
3686739992417,5cyclictest3779756-21kworker/u8:0+events_unbound23:43:090
3686739992417,5cyclictest3707730-21kworker/u8:0+events_unbound20:07:410
3686739992417,5cyclictest3707730-21kworker/u8:0+events_unbound20:07:400
3686739992416,4cyclictest3797742-21fschecks_time00:30:190
3686739992416,4cyclictest3797742-21fschecks_time00:30:180
3686739992415,4cyclictest0-21swapper/021:42:490
368673999241,19cyclictest0-21swapper/021:48:550
368673999241,19cyclictest0-21swapper/021:16:460
368673999240,20cyclictest0-21swapper/020:10:510
368673999240,20cyclictest0-21swapper/020:10:510
3686743992317,3cyclictest0-21swapper/120:40:231
3686743992316,3cyclictest0-21swapper/122:35:161
3686743992313,6cyclictest0-21swapper/121:40:011
3686739992317,2cyclictest0-21swapper/022:01:150
3686739992317,2cyclictest0-21swapper/022:01:150
3686739992316,4cyclictest0-21swapper/000:20:270
3686739992316,3cyclictest0-21swapper/020:34:020
3686739992315,4cyclictest3702396-21grep19:50:260
3686739992315,4cyclictest3702396-21grep19:50:260
3686739992315,4cyclictest3698100-21fschecks_count19:40:190
3686739992314,6cyclictest3769334-21grep23:05:260
368673999231,20cyclictest0-21swapper/019:30:240
368673999231,19cyclictest0-21swapper/023:19:040
368673999230,19cyclictest0-21swapper/020:25:030
368673999230,19cyclictest0-21swapper/000:13:170
3686743992215,5cyclictest231ktimers/120:10:211
3686743992215,5cyclictest231ktimers/120:10:201
3686743992215,4cyclictest3777573-21mailstats23:30:231
3686743992214,5cyclictest3716473-21cron20:35:011
3686743992213,5cyclictest0-21swapper/119:50:171
3686743992213,5cyclictest0-21swapper/119:50:171
3686743992210,8cyclictest1-21systemd20:37:431
3686739992217,3cyclictest13650irq/42-30be0000.ethernet23:10:220
3686739992215,4cyclictest0-21swapper/022:10:260
3686739992215,4cyclictest0-21swapper/022:10:250
3686739992213,4cyclictest3764805-21latency_hist22:55:010
368673999220,18cyclictest0-21swapper/022:21:130
3686743992112,6cyclictest3703449-21sort19:55:191
3686743992112,6cyclictest0-21swapper/121:05:171
3686743992112,5cyclictest231ktimers/123:05:261
3686743992111,5cyclictest3720354-21fschecks_time20:45:231
3686743992013,5cyclictest231ktimers/120:25:001
3686743992013,5cyclictest231ktimers/119:30:241
3686743992012,5cyclictest0-21swapper/100:25:171
3686743992012,5cyclictest0-21swapper/100:25:161
3686743992010,7cyclictest3748046-21cut22:05:171
3686743992010,7cyclictest3748046-21cut22:05:171
3686743991813,3cyclictest3753153-21fschecks_count22:20:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional