You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 10:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 31, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18212236162,24sleep20-21swapper/219:09:192
15652235164,57sleep30-21swapper/319:06:063
15352230161,57sleep00-21swapper/019:05:550
17422202166,23sleep10-21swapper/119:08:181
118321340,5sleep2196799cyclictest22:45:182
118321340,5sleep2196799cyclictest22:45:182
1967998985,3cyclictest0-21swapper/220:29:542
1967998985,3cyclictest0-21swapper/220:29:542
44082600,1sleep30-21swapper/321:40:123
42642560,3sleep3197299cyclictest19:15:013
54022500,2sleep30-21swapper/322:55:183
291232450,3sleep028260-21/usr/sbin/munin22:35:200
195599341,25cyclictest0-21swapper/021:18:190
195599333,22cyclictest392-21kworker/u8:023:59:450
1963993115,2cyclictest0-21swapper/100:19:401
195599313,13cyclictest8887-21idleruntime-cro23:05:010
195599311,24cyclictest0-21swapper/021:51:250
196399296,10cyclictest9551-21cpuspeed_turbos19:25:121
195599291,13cyclictest22414-21perf19:55:020
196399273,13cyclictest87650irq/129-enp1s0-22:10:141
195599271,24cyclictest0-21swapper/019:45:160
196399264,6cyclictest28693-40dnf20:06:181
196399261,21cyclictest27420-21munin-run21:20:011
196399261,17cyclictest0-21swapper/122:25:191
195599267,6cyclictest1047-21cat22:45:160
195599267,6cyclictest1047-21cat22:45:160
195599267,13cyclictest87350irq/126-enp1s0-22:29:140
195599261,22cyclictest808-21snmpd19:55:260
160412260,3sleep20-21swapper/222:05:202
1963992510,12cyclictest21180-21awk19:50:161
195599251,16cyclictest0-21swapper/021:14:290
195599250,18cyclictest0-21swapper/023:05:430
196399242,12cyclictest4138-21munin-run19:15:011
196399241,21cyclictest0-21swapper/121:58:261
196399241,19cyclictest22742-21ntp_states22:20:221
195599248,3cyclictest13836-21/usr/sbin/munin19:35:230
195599247,10cyclictest87350irq/126-enp1s0-00:07:560
195599241,20cyclictest23022-21diskmemload00:25:180
195599241,20cyclictest0-21swapper/023:52:510
195599241,20cyclictest0-21swapper/023:14:570
195599241,20cyclictest0-21swapper/022:42:320
195599241,20cyclictest0-21swapper/022:14:420
195599241,20cyclictest0-21swapper/000:15:480
195599241,15cyclictest0-21swapper/021:44:430
195599241,15cyclictest0-21swapper/021:38:300
195599241,15cyclictest0-21swapper/021:38:300
195599241,12cyclictest12871-21/usr/sbin/munin22:00:160
195599240,22cyclictest0-21swapper/023:35:180
197299231,16cyclictest0-21swapper/322:05:253
197299231,11cyclictest0-21swapper/300:15:263
196399235,10cyclictest87550irq/128-enp1s0-20:18:391
196399234,7cyclictest10852-40dnf23:07:241
1963992316,4cyclictest579-21systemd-journal20:55:021
1963992312,3cyclictest1590-21df20:20:141
195599239,3cyclictest14449-21/usr/sbin/munin20:50:230
195599234,10cyclictest13209-21kworker/u8:223:45:310
195599232,18cyclictest26982-21memory22:30:220
195599232,10cyclictest24822-21/usr/sbin/munin20:00:140
195599231,20cyclictest0-21swapper/022:58:440
195599231,11cyclictest21050-21/usr/sbin/munin21:05:230
195599231,11cyclictest18235-21sshd00:39:150
195599230,6cyclictest0-21swapper/022:53:390
195599230,6cyclictest0-21swapper/022:07:010
195599230,10cyclictest6669-21kworker/0:019:20:030
1972992219,2cyclictest0-21swapper/323:41:023
197299221,11cyclictest21967-21memory21:05:203
196799222,7cyclictest0-21swapper/221:15:242
1967992210,3cyclictest15808-21awk19:40:012
196399221,8cyclictest0-21swapper/123:45:191
196399221,19cyclictest0-21swapper/121:12:101
196399221,19cyclictest0-21swapper/119:15:141
196399221,19cyclictest0-21swapper/119:15:131
196399221,18cyclictest4513-21timerandwakeup00:05:261
196399221,15cyclictest0-21swapper/122:38:581
1963992210,3cyclictest7604-21diskstats19:20:171
195599228,3cyclictest21668-21/usr/sbin/munin22:20:270
195599227,3cyclictest29939-21kworker/0:000:01:150
195599222,9cyclictest7005-21kworker/0:300:21:200
195599222,18cyclictest87350irq/126-enp1s0-21:20:170
195599222,11cyclictest12227-21kworker/0:021:59:020
195599221,9cyclictest14476-21df00:30:140
195599221,9cyclictest0-21swapper/023:40:280
195599221,6cyclictest0-21swapper/022:18:240
195599221,6cyclictest0-21swapper/021:28:510
195599221,10cyclictest30971-21sshd20:12:150
195599221,10cyclictest23022-21diskmemload23:32:070
195599221,10cyclictest0-21swapper/020:20:300
197299216,8cyclictest87650irq/129-enp1s0-21:50:203
197299216,3cyclictest0-21swapper/319:16:333
197299216,3cyclictest0-21swapper/319:16:323
197299214,9cyclictest29058-21cstates23:50:113
197299213,9cyclictest30293-21ntp_states20:10:213
197299212,11cyclictest19472-21/usr/sbin/munin22:15:133
197299212,10cyclictest4728-21users00:05:273
197299211,6cyclictest5430-21latency_hist20:30:023
197299211,6cyclictest5430-21latency_hist20:30:023
197299211,4cyclictest0-21swapper/300:27:133
197299211,11cyclictest7859-21/usr/sbin/munin20:35:253
197299211,11cyclictest30229-21munin-run22:40:013
197299211,11cyclictest19857-21latency_hist23:30:013
197299211,11cyclictest16414-21latency_hist20:55:023
197299211,11cyclictest15355-21processes00:30:253
197299211,10cyclictest7639-21munin-run20:35:013
197299210,14cyclictest0-21swapper/323:57:153
196799216,4cyclictest17359-21idleruntime20:55:162
196799215,4cyclictest20647-21ps22:15:232
196799214,15cyclictest0-21swapper/223:05:292
196799211,8cyclictest6496-21head00:10:232
1967992116,3cyclictest5190-21tr19:15:122
1967992116,3cyclictest5190-21tr19:15:112
196799211,11cyclictest0-21swapper/200:29:522
196399217,12cyclictest3739-21kworker/u8:022:42:261
196399216,13cyclictest23-21ksoftirqd/121:35:081
196399216,13cyclictest23-21ksoftirqd/121:35:081
196399213,9cyclictest7176-21unixbench_multi20:30:261
1963992114,4cyclictest30364-21tune2fs21:25:171
196399211,15cyclictest0-21swapper/100:29:061
196399211,11cyclictest810-21sshd21:01:341
196399211,11cyclictest476-21cstates22:45:111
196399211,11cyclictest476-21cstates22:45:111
196399211,11cyclictest13207-21missed_timers20:45:211
196399211,10cyclictest6668-21munin-run23:00:011
195599214,3cyclictest4986-21kworker/0:100:10:240
195599211,11cyclictest0-21swapper/023:26:360
195599211,10cyclictest0-21swapper/023:18:010
195599210,1cyclictest0-21swapper/020:45:150
172842210,3sleep121-21rcuc/119:40:241
160482211,3sleep20-21swapper/200:34:222
24382200,4sleep011-21rcuc/019:10:120
197299208,11cyclictest87350irq/126-enp1s0-20:55:223
197299204,8cyclictest87350irq/126-enp1s0-23:10:193
197299202,5cyclictest909-21sssd_nss23:07:033
197299202,10cyclictest21498-21ntp_states19:50:203
197299201,8cyclictest0-21swapper/322:40:103
197299201,7cyclictest17150-21kernelversion00:35:173
197299201,7cyclictest11271-21head21:55:163
1972992017,2cyclictest0-21swapper/323:15:213
197299201,6cyclictest27373-21diskstats20:05:143
197299201,6cyclictest12033-21perf20:45:013
197299201,6cyclictest0-21swapper/323:02:113
1972992015,3cyclictest0-21swapper/321:10:403
197299201,4cyclictest0-21swapper/322:25:023
1972992011,3cyclictest7784-21grep21:45:243
197299201,11cyclictest29841-21/usr/sbin/munin21:25:243
197299201,11cyclictest1239-21/usr/sbin/munin20:20:253
197299201,11cyclictest0-21swapper/323:32:203
197299201,10cyclictest26407-21/usr/sbin/munin22:30:163
197299201,10cyclictest0-21swapper/319:40:143
196799202,9cyclictest0-21swapper/222:40:162
196799201,7cyclictest32067-21strings23:55:212
1967992015,3cyclictest24280-21cat23:40:012
1967992010,7cyclictest22141-21kworker/u8:222:35:342
1967992010,4cyclictest0-21swapper/222:25:112
196399209,3cyclictest8040-21cstates20:35:121
196399209,3cyclictest32219-21cstates21:30:131
196399205,2cyclictest15472-21perf23:20:021
196399201,8cyclictest0-21swapper/123:38:281
1963992016,3cyclictest0-21swapper/120:12:221
1963992016,3cyclictest0-21swapper/100:31:591
196399201,18cyclictest0-21swapper/119:38:421
196399201,17cyclictest0-21swapper/120:00:211
196399201,14cyclictest87650irq/129-enp1s0-21:52:241
196399201,14cyclictest0-21swapper/123:00:481
196399201,13cyclictest0-21swapper/123:14:211
196399201,11cyclictest30214-21switchtime23:50:251
196399201,11cyclictest23022-21diskmemload00:10:131
196399201,11cyclictest19638-21grep19:45:251
196399201,11cyclictest19405-21grep23:25:261
196399201,11cyclictest13607-21latency_hist19:35:021
196399201,10cyclictest3048-21perl22:50:141
196399201,10cyclictest27827-21cstates21:20:151
195599203,15cyclictest0-21swapper/021:33:500
195599201,4cyclictest5977-21missed_timers19:15:190
195599201,4cyclictest5977-21missed_timers19:15:190
195599201,16cyclictest7648-21sed21:45:240
197299196,3cyclictest87350irq/126-enp1s0-20:15:123
197299193,8cyclictest12050irq/123-ahci[0019:35:013
197299193,4cyclictest27136-21fschecks_count23:45:153
197299193,3cyclictest0-21swapper/300:10:323
197299192,4cyclictest26520-21ntp_states21:15:213
197299191,7cyclictest0-21swapper/322:00:103
197299191,6cyclictest32039-21/usr/sbin/munin21:30:143
197299191,6cyclictest19654-21irqstats21:00:173
197299191,6cyclictest18229-21/usr/sbin/munin19:45:133
197299191,6cyclictest1698-21if_err_enp1s000:00:173
197299191,6cyclictest14751-21memory19:35:183
197299191,6cyclictest12253-21/usr/sbin/munin20:45:163
197299191,6cyclictest0-21swapper/322:10:143
197299191,6cyclictest0-21swapper/321:35:173
197299191,6cyclictest0-21swapper/321:35:173
197299191,6cyclictest0-21swapper/321:20:523
197299191,5cyclictest0-21swapper/322:53:323
197299191,4cyclictest808-21snmpd19:20:193
197299191,11cyclictest0-21swapper/322:23:093
196799198,3cyclictest26463-21awk23:45:012
196799193,6cyclictest32-21ksoftirqd/220:45:252
196799192,15cyclictest0-21swapper/219:27:392
196799191,5cyclictest0-21swapper/200:15:172
196799191,15cyclictest3489-21dnf00:05:152
196799191,12cyclictest0-21swapper/221:50:232
196799191,11cyclictest0-21swapper/222:30:362
196799191,10cyclictest0-21swapper/223:21:282
196399199,2cyclictest4572-21pmu-power20:25:211
196399199,2cyclictest4572-21pmu-power20:25:211
196399195,3cyclictest0-21swapper/123:30:211
196399191,8cyclictest0-21swapper/121:40:331
196399191,7cyclictest0-21swapper/122:15:061
1963991915,3cyclictest0-21swapper/123:24:451
1963991915,3cyclictest0-21swapper/121:47:561
196399191,16cyclictest0-21swapper/120:40:401
196399191,12cyclictest0-21swapper/122:08:001
196399191,10cyclictest31124-21cpu23:55:131
196399191,10cyclictest23022-21diskmemload00:20:161
196399191,10cyclictest22973-21diskstats19:55:151
1963991910,2cyclictest28055-21wc22:35:001
195599192,14cyclictest10053-21/usr/sbin/munin20:40:210
195599191,8cyclictest5364-21sshd20:29:160
195599191,8cyclictest5364-21sshd20:29:150
195599191,7cyclictest10518-21sh19:25:210
195599191,6cyclictest28615-21users20:05:250
195599191,6cyclictest0-21swapper/021:02:410
195599191,5cyclictest8775-21memory20:35:190
195599190,1cyclictest0-21swapper/019:30:170
283122181,2sleep20-21swapper/223:45:502
25662181,2sleep20-21swapper/200:00:552
197299181,5cyclictest22625-21/usr/sbin/munin19:55:233
197299181,5cyclictest0-21swapper/323:22:003
197299181,4cyclictest10854-21pmu-power00:20:233
197299181,4cyclictest0-21swapper/323:35:513
197299181,4cyclictest0-21swapper/322:47:483
197299181,4cyclictest0-21swapper/322:47:483
197299180,12cyclictest0-21swapper/320:03:503
196799189,2cyclictest87650irq/129-enp1s0-00:35:072
196799186,3cyclictest16419-21fschecks_count19:40:152
196799184,3cyclictest87350irq/126-enp1s0-23:30:222
196799183,6cyclictest9-21rcu_preempt22:55:262
196799182,6cyclictest32-21ksoftirqd/219:55:162
196799182,5cyclictest30853-21munin-run23:55:012
196799182,15cyclictest0-21swapper/223:19:142
196799181,7cyclictest0-21swapper/221:21:322
196799181,6cyclictest27155-21cpuspeed_turbos20:05:122
196799181,6cyclictest11710-21dump-pmu-power00:25:022
196799181,6cyclictest0-21swapper/221:43:012
196799181,5cyclictest579-21systemd-journal20:25:012
196799181,5cyclictest32685-21smartctl20:15:242
196799181,4cyclictest0-21swapper/222:23:252
196799181,4cyclictest0-21swapper/222:10:082
1967991812,4cyclictest6802-21tr20:30:242
196799181,15cyclictest0-21swapper/221:25:572
1967991811,3cyclictest25222-21perf21:15:002
196399187,6cyclictest0-21swapper/123:41:351
196399187,3cyclictest16994-21diskstats20:55:151
196399184,2cyclictest0-21swapper/121:07:431
196399181,9cyclictest13695-21kernelversion22:00:171
196399181,5cyclictest1918-21cut00:00:201
196399181,16cyclictest0-21swapper/100:35:051
195599181,5cyclictest18219-21unixbench_singl20:55:260
195599181,3cyclictest0-21swapper/023:20:140
1955991812,3cyclictest7205-21unixbench_singl20:30:270
41122171,2sleep20-21swapper/222:50:352
197299171,4cyclictest9444-21/usr/sbin/munin19:25:143
196799173,4cyclictest11407-21cpuspeed_turbos23:10:112
196799172,14cyclictest0-21swapper/220:41:182
196799171,7cyclictest910-21systemd-logind22:00:012
196799171,7cyclictest21971-21missed_timers21:05:192
196799171,7cyclictest13940-21ntp_states22:00:212
196799171,6cyclictest18992-21if_enp2s019:45:172
196799171,5cyclictest0-21swapper/223:00:592
196799171,5cyclictest0-21swapper/221:35:182
196799171,5cyclictest0-21swapper/221:35:182
196799171,3cyclictest0-21swapper/219:24:332
1967991712,3cyclictest2159-21/usr/sbin/munin19:10:122
196799171,14cyclictest0-21swapper/221:02:092
1967991710,4cyclictest25426-21grep20:00:172
1955991712,3cyclictest32485-21ntp_states20:15:200
1955991712,3cyclictest16100-21wc19:40:130
196799162,6cyclictest0-21swapper/220:51:032
196799161,9cyclictest0-21swapper/221:45:082
196799161,6cyclictest0-21swapper/219:30:132
196799161,5cyclictest0-21swapper/221:30:122
196799161,12cyclictest30025-21cat20:10:172
1967991610,3cyclictest827-21runrttasks19:50:222
1967991610,3cyclictest18983-21open_inodes23:25:222
1967991510,3cyclictest9249-21cat20:35:252
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional