You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 13:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot5s.osadl.org (updated Tue Dec 02, 2025 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
669707996143,10cyclictest725374-21kworker/u8:1+events_unbound22:08:152
669702995641,11cyclictest231ktimers/121:45:151
669710995220,9cyclictest131rcu_preempt20:30:153
669707995042,5cyclictest722044-21cut21:40:012
669702995037,10cyclictest231ktimers/120:50:141
669707994936,9cyclictest311ktimers/220:50:122
669710994842,4cyclictest697436-21kworker/u8:0+events_unbound20:58:033
669710994838,6cyclictest728442-21grep21:55:283
669707994842,4cyclictest677111-21kworker/u8:1+events_unbound19:32:392
669707994740,5cyclictest671011-21kworker/u8:0+events_unbound19:27:392
669707994739,5cyclictest311ktimers/222:00:122
669707994634,8cyclictest728428-21grep21:55:282
669698994638,4cyclictest701151-21munin-plugin-st20:40:010
669707994536,5cyclictest687509-21idleruntime-cro20:00:012
669707994422,13cyclictest131rcu_preempt20:10:162
669702994434,5cyclictest299-21systemd-logind00:23:511
669710994322,12cyclictest756125-21kworker/u8:1+events_unbound23:40:143
669707994336,4cyclictest440-21cron21:50:002
669707994325,5cyclictest131rcu_preempt00:21:142
669707994236,4cyclictest771667-21kworker/u8:1+events_unbound00:28:532
669707994236,4cyclictest711458-21kworker/u8:2+events_unbound21:23:062
669707994135,4cyclictest756125-21kworker/u8:1+events_unbound23:33:372
669707994131,6cyclictest299-21systemd-logind22:28:212
669702994126,11cyclictest231ktimers/123:55:141
669702994122,6cyclictest131rcu_preempt21:40:151
669698994130,7cyclictest772959-21grep00:05:250
669707994034,4cyclictest759766-21kworker/u8:2+events_unbound23:58:472
669707994027,5cyclictest131rcu_preempt20:00:182
669707994026,4cyclictest733795-21kworker/u8:0+events_unbound22:18:202
669707994022,5cyclictest131rcu_preempt22:43:452
669707994018,4cyclictest0-21swapper/219:11:192
669710993927,8cyclictest391ktimers/323:35:153
669707993931,4cyclictest0-21swapper/221:10:012
669707993925,4cyclictest131rcu_preempt00:02:322
669698993932,4cyclictest0-21swapper/023:19:590
669710993831,4cyclictest680681-21sh19:40:013
669710993829,6cyclictest748085-21fschecks_time22:55:173
669710993823,9cyclictest678866-21kworker/u8:0+events_unbound19:45:023
669710993821,5cyclictest131rcu_preempt19:10:133
669707993829,7cyclictest704187-21kworker/u8:1+events_unbound20:47:592
669707993823,4cyclictest131rcu_preempt20:44:392
669707993819,10cyclictest131rcu_preempt00:19:512
669707993819,10cyclictest131rcu_preempt00:19:502
669707993818,4cyclictest0-21swapper/221:52:132
669707993817,10cyclictest131rcu_preempt00:39:582
669702993826,7cyclictest0-21swapper/121:28:071
669698993830,4cyclictest770012-21cron00:00:000
669710993729,5cyclictest687498-21cron20:00:003
669707993722,5cyclictest131rcu_preempt19:39:132
669707993720,5cyclictest0-21swapper/222:55:032
669707993717,9cyclictest131rcu_preempt00:09:542
669707993714,11cyclictest131rcu_preempt23:45:142
669707993712,5cyclictest131rcu_preempt23:50:162
669702993727,6cyclictest0-21swapper/119:55:011
669702993722,10cyclictest673674-21basename19:20:001
669707993630,4cyclictest733795-21kworker/u8:0+events_unbound22:13:432
669707993630,4cyclictest733795-21kworker/u8:0+events_unbound22:13:422
669702993628,6cyclictest781662-21kworker/u8:0+events_unbound00:33:531
669702993627,6cyclictest728853-21cron22:00:011
669702993626,6cyclictest0-21swapper/120:55:301
669698993626,7cyclictest678990-21basename19:35:010
669698993624,8cyclictest709887-21cron21:05:010
669710993527,6cyclictest673401-21kworker/u8:2+events_unbound19:22:373
669710993527,4cyclictest0-21swapper/300:15:173
669710993527,4cyclictest0-21swapper/300:15:163
669710993521,5cyclictest721825-21kworker/u8:1+events_unbound21:40:263
669710993519,5cyclictest653129-21kworker/u8:1+events_unbound19:15:093
669707993529,4cyclictest699146-21kworker/u8:1+events_unbound20:32:562
669707993526,4cyclictest0-21swapper/222:55:012
669707993522,4cyclictest131rcu_preempt21:41:352
669707993521,5cyclictest131rcu_preempt22:20:132
669707993517,10cyclictest131rcu_preempt23:35:242
669707993516,9cyclictest131rcu_preempt21:18:122
669707993515,4cyclictest131rcu_preempt22:31:392
669702993525,6cyclictest0-21swapper/123:40:261
669702993525,6cyclictest0-21swapper/121:30:261
669702993524,6cyclictest0-21swapper/122:45:261
669702993524,5cyclictest711458-21kworker/u8:2+events_unbound21:10:031
669702993523,5cyclictest721825-21kworker/u8:1+events_unbound21:38:111
66970299351,11cyclictest0-21swapper/119:35:131
669698993527,4cyclictest0-21swapper/022:15:260
669698993524,7cyclictest733552-21grep22:10:250
669698993524,7cyclictest733552-21grep22:10:250
669710993428,4cyclictest740593-21kworker/u8:1+events_unbound22:43:253
669710993425,6cyclictest690522-21unixbench_singl20:05:313
669710993424,6cyclictest0-21swapper/319:35:013
669710993418,4cyclictest131rcu_preempt20:49:123
66970799346,10cyclictest32-21ksoftirqd/222:50:012
669707993428,4cyclictest699146-21kworker/u8:1+events_unbound20:37:572
669707993426,4cyclictest0-21swapper/221:25:292
669707993425,4cyclictest0-21swapper/221:30:162
669707993421,5cyclictest131rcu_preempt19:50:252
669707993418,4cyclictest131rcu_preempt23:23:262
669707993414,12cyclictest131rcu_preempt23:00:252
669702993424,6cyclictest0-21swapper/119:40:261
669702993424,6cyclictest0-21swapper/119:20:261
669698993425,5cyclictest0-21swapper/019:40:010
669710993324,6cyclictest391ktimers/321:05:143
669710993323,6cyclictest0-21swapper/300:00:263
669710993322,6cyclictest0-21swapper/319:25:273
669707993325,4cyclictest0-21swapper/221:00:182
669707993320,4cyclictest131rcu_preempt19:45:172
669707993319,4cyclictest131rcu_preempt21:10:152
669707993318,4cyclictest131rcu_preempt23:40:222
669707993316,6cyclictest131rcu_preempt20:20:222
669707993316,4cyclictest131rcu_preempt23:09:122
669707993315,4cyclictest131rcu_preempt23:18:192
669707993312,4cyclictest131rcu_preempt00:12:492
669702993326,5cyclictest687189-21kworker/u8:1+events_unbound19:57:511
669702993324,4cyclictest0-21swapper/123:20:001
669698993325,4cyclictest0-21swapper/020:25:000
669698993322,7cyclictest0-21swapper/022:45:010
669710993225,5cyclictest694256-21kworker/u8:2+events_unbound20:25:213
669710993225,4cyclictest0-21swapper/322:50:253
669710993225,3cyclictest0-21swapper/319:40:273
669710993224,4cyclictest0-21swapper/320:20:183
669710993224,4cyclictest0-21swapper/300:10:003
669710993223,6cyclictest759792-21cron23:30:013
669710993222,6cyclictest0-21swapper/321:05:013
669710993218,5cyclictest714972-21kworker/u8:1+events_unbound21:20:043
669710993216,8cyclictest131rcu_preempt00:10:263
669707993215,3cyclictest0-21swapper/220:28:422
669707993212,9cyclictest131rcu_preempt20:56:442
669707993211,5cyclictest131rcu_preempt19:20:252
669702993226,4cyclictest771667-21kworker/u8:1+events_unbound00:08:491
669702993226,4cyclictest746160-21kworker/u8:2+events_unbound22:58:311
669702993222,6cyclictest0-21swapper/122:20:171
669702993221,6cyclictest0-21swapper/119:25:281
669698993221,7cyclictest0-21swapper/020:30:280
669698993221,5cyclictest757891-21munin-plugin-st23:25:000
669698993220,7cyclictest0-21swapper/021:50:260
669710993124,3cyclictest0-21swapper/300:25:163
669710993123,4cyclictest0-21swapper/322:10:253
669710993123,4cyclictest0-21swapper/322:10:253
669710993120,7cyclictest721600-21grep21:35:253
669710993120,7cyclictest0-21swapper/319:50:253
669710993120,6cyclictest0-21swapper/322:24:593
669710993117,6cyclictest700857-21kworker/u8:2+events_unbound20:43:353
66970799315,7cyclictest32-21ksoftirqd/223:25:142
669707993124,4cyclictest0-21swapper/222:35:282
669707993119,8cyclictest311ktimers/220:05:312
669707993114,9cyclictest131rcu_preempt20:15:252
669707993114,9cyclictest131rcu_preempt20:15:252
669707993114,9cyclictest131rcu_preempt00:31:072
669707993114,5cyclictest131rcu_preempt23:10:212
669698993112,4cyclictest748015-21fschecks_count22:55:170
669710993026,2cyclictest711458-21kworker/u8:2+events_unbound21:18:083
669710993022,4cyclictest0-21swapper/321:30:263
669710993021,5cyclictest0-21swapper/323:15:263
669710993011,6cyclictest40-21ksoftirqd/323:09:193
669710993010,3cyclictest781502-21unixbench_multi00:30:243
669702993023,3cyclictest0-21swapper/123:00:181
669702993022,5cyclictest732022-21kworker/u8:2+events_unbound22:25:151
669702993022,4cyclictest0-21swapper/123:50:251
669702993022,4cyclictest0-21swapper/121:05:261
669702993021,6cyclictest0-21swapper/119:15:011
669702993020,6cyclictest0-21swapper/100:00:251
66971099296,11cyclictest131rcu_preempt21:50:263
669710992923,4cyclictest692457-21kworker/u8:0+events_unbound20:17:423
669710992923,4cyclictest692457-21kworker/u8:0+events_unbound20:17:413
669710992921,4cyclictest0-21swapper/323:55:183
669710992920,6cyclictest725022-21grep21:45:273
669710992920,6cyclictest381rcuc/322:48:213
669710992919,6cyclictest0-21swapper/321:30:003
669707992917,4cyclictest131rcu_preempt19:15:192
669707992914,4cyclictest131rcu_preempt19:40:272
669702992920,6cyclictest711458-21kworker/u8:2+events_unbound21:18:061
669698992919,6cyclictest0-21swapper/023:05:280
669698992916,10cyclictest121ktimers/019:15:130
669698992914,11cyclictest669689-21cyclictest23:35:150
66969899290,27cyclictest0-21swapper/020:00:150
66971099285,6cyclictest131rcu_preempt20:50:143
669710992822,4cyclictest746160-21kworker/u8:2+events_unbound23:20:143
669710992821,5cyclictest776815-21kworker/u8:0+events_unbound00:23:433
669710992820,4cyclictest0-21swapper/320:00:193
66971099280,7cyclictest0-21swapper/322:00:113
66970299286,5cyclictest0-21swapper/122:30:141
669702992819,6cyclictest0-21swapper/120:15:271
669702992819,6cyclictest0-21swapper/120:15:271
669702992818,7cyclictest742529-21sendmail22:40:001
669702992818,6cyclictest0-21swapper/123:05:281
669698992820,4cyclictest0-21swapper/022:35:010
669710992721,4cyclictest750898-21kworker/u8:1+events_unbound23:04:193
669710992720,5cyclictest732022-21kworker/u8:2+events_unbound22:36:013
669710992718,7cyclictest766386-21kworker/u8:1+events_unbound23:49:513
669710992717,6cyclictest0-21swapper/320:12:533
669710992715,4cyclictest711458-21kworker/u8:2+events_unbound21:10:543
669702992721,4cyclictest697436-21kworker/u8:0+events_unbound20:29:431
669702992719,5cyclictest0-21swapper/122:15:011
669702992719,5cyclictest0-21swapper/122:15:001
669702992719,4cyclictest0-21swapper/122:00:261
669702992719,4cyclictest0-21swapper/120:35:171
669702992712,7cyclictest24-21ksoftirqd/121:20:061
669698992723,2cyclictest13650irq/42-30be0000.ethernet20:25:170
669698992719,5cyclictest0-21swapper/023:40:250
669698992719,4cyclictest13950irq/36-mmc122:40:010
669698992719,4cyclictest13650irq/42-30be0000.ethernet22:03:140
669698992719,3cyclictest0-21swapper/000:00:250
669698992718,6cyclictest713743-21fschecks_time21:15:160
669698992716,7cyclictest215-21dbus-daemon22:28:210
66969899270,10cyclictest775664-21awk00:15:180
66969899270,10cyclictest775664-21awk00:15:180
669710992616,6cyclictest0-21swapper/320:40:013
669710992615,4cyclictest766386-21kworker/u8:1+events_unbound23:50:213
66971099261,11cyclictest0-21swapper/322:17:473
66970299267,9cyclictest0-21swapper/122:41:081
66970299267,7cyclictest131rcu_preempt22:50:241
669702992618,5cyclictest0-21swapper/123:45:251
669702992616,6cyclictest0-21swapper/119:30:271
669702992615,7cyclictest1-21systemd19:47:471
66970299261,4cyclictest131rcu_preempt23:35:131
66969899263,18cyclictest754040-21grep23:10:240
669698992619,4cyclictest737001-21grep22:20:260
669698992611,12cyclictest669689-21cyclictest20:45:150
669710992516,6cyclictest732421-21latency_hist22:10:013
66971099251,11cyclictest0-21swapper/323:34:533
669702992517,6cyclictest758533-21cut23:25:181
669702992517,5cyclictest0-21swapper/123:10:251
669702992517,4cyclictest0-21swapper/123:20:251
669702992516,5cyclictest0-21swapper/123:30:271
669702992516,5cyclictest0-21swapper/120:00:281
669702992512,10cyclictest231ktimers/120:45:151
669698992519,4cyclictest687189-21kworker/u8:1+events_unbound20:07:530
669698992518,3cyclictest0-21swapper/022:45:200
66969899251,6cyclictest684898-21ntp_states19:50:220
669698992516,6cyclictest727552-21tune2fs21:55:170
669698992514,8cyclictest758678-21cut23:25:200
669698992514,7cyclictest215-21dbus-daemon19:42:450
66971099244,13cyclictest40-21ksoftirqd/300:35:133
669710992418,4cyclictest740593-21kworker/u8:1+events_unbound22:32:333
66971099241,3cyclictest0-21swapper/322:25:133
66970299248,3cyclictest131rcu_preempt21:05:001
669702992418,4cyclictest690636-21kworker/u8:2+events_unbound20:12:031
669698992419,3cyclictest0-21swapper/021:45:150
669698992417,4cyclictest0-21swapper/021:40:290
669698992417,4cyclictest0-21swapper/021:05:250
669698992417,4cyclictest0-21swapper/019:55:270
669698992417,3cyclictest0-21swapper/000:38:460
669698992417,3cyclictest0-21swapper/000:30:340
669698992416,6cyclictest121ktimers/020:55:160
669698992416,4cyclictest0-21swapper/020:15:260
669698992416,4cyclictest0-21swapper/020:15:260
669698992415,6cyclictest706283-21gpgv20:55:010
66969899241,20cyclictest215-21dbus-daemon20:12:530
66969899241,19cyclictest0-21swapper/023:34:310
66970299239,6cyclictest24-21ksoftirqd/120:20:211
669702992317,4cyclictest711458-21kworker/u8:2+events_unbound21:53:101
669702992315,5cyclictest0-21swapper/122:15:251
669702992315,5cyclictest0-21swapper/100:25:151
669702992314,5cyclictest0-21swapper/100:15:011
66970299231,13cyclictest783694-21awk00:40:011
669702992311,10cyclictest24-21ksoftirqd/100:15:141
669702992311,10cyclictest24-21ksoftirqd/100:15:141
669698992316,3cyclictest0-21swapper/022:08:080
669698992315,4cyclictest0-21swapper/019:45:200
669698992314,5cyclictest121ktimers/021:20:260
66969899231,18cyclictest0-21swapper/023:49:390
66969899231,18cyclictest0-21swapper/021:38:080
66969899231,18cyclictest0-21swapper/021:27:490
66969899230,19cyclictest0-21swapper/021:10:560
66969899230,19cyclictest0-21swapper/000:20:080
669710992212,3cyclictest131rcu_preempt23:10:223
66970299227,7cyclictest131rcu_preempt20:30:231
669702992215,4cyclictest690636-21kworker/u8:2+events_unbound20:07:221
669702992210,2cyclictest131rcu_preempt22:05:171
669698992217,3cyclictest13950irq/36-mmc119:30:010
669698992214,4cyclictest675568-21auditctl19:25:000
669698992212,7cyclictest13650irq/42-30be0000.ethernet19:10:130
66969899221,19cyclictest0-21swapper/022:51:320
66969899221,18cyclictest749710-21diskstats23:00:180
66969899221,18cyclictest701389-21munin-node20:40:110
66969899221,17cyclictest0-21swapper/000:12:490
66969899220,18cyclictest0-21swapper/021:30:090
66970299215,7cyclictest131rcu_preempt20:40:191
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional