You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 07:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 10, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24322240164,24sleep30-21swapper/319:06:213
24322240164,24sleep30-21swapper/319:06:213
27062238164,23sleep20-21swapper/219:09:552
27062238164,23sleep20-21swapper/219:09:552
25542237162,24sleep10-21swapper/119:07:571
25542237162,24sleep10-21swapper/119:07:561
24642235162,60sleep00-21swapper/019:06:480
24642235162,60sleep00-21swapper/019:06:480
1622321360,4sleep3281599cyclictest22:05:123
2815997774,2cyclictest0-21swapper/323:41:393
2815997774,2cyclictest0-21swapper/323:41:393
145692660,2sleep30-21swapper/323:15:113
16342550,1sleep20-21swapper/200:00:012
14542530,5sleep07-21ksoftirqd/023:59:190
222502490,4sleep123-21ksoftirqd/121:05:151
52832480,4sleep1221ktimersoftd/121:40:151
157472450,4sleep121-21rcuc/100:30:181
2798993910,13cyclictest19163-21munin-run00:40:010
279899389,13cyclictest16653-21munin-run19:40:020
206012370,4sleep0279899cyclictest21:00:160
279899368,13cyclictest11937-21awk23:10:010
18592340,3sleep1280499cyclictest21:30:231
279899313,13cyclictest16973-21awk00:35:010
279899291,13cyclictest32047-21munin-run20:15:010
280499281,19cyclictest0-21swapper/123:03:241
280499281,17cyclictest3991-21awk21:35:231
279899276,7cyclictest29123-21users23:45:260
279899274,10cyclictest30126-21fschecks_count23:50:140
264682270,2sleep10-21swapper/123:40:221
264682270,2sleep10-21swapper/123:40:221
280499266,11cyclictest0-21swapper/122:35:221
319732250,2sleep3401ktimersoftd/321:25:243
319732250,2sleep3401ktimersoftd/321:25:233
281599251,9cyclictest0-21swapper/323:02:193
2815992516,5cyclictest24454-21sort23:35:233
280499257,5cyclictest16326-21awk23:20:011
280499251,17cyclictest0-21swapper/121:15:231
280499251,16cyclictest0-21swapper/122:25:581
279899250,7cyclictest0-21swapper/023:23:130
47982240,2sleep09-21rcu_preempt21:40:020
281599241,21cyclictest0-21swapper/322:40:243
280499245,8cyclictest0-21swapper/122:21:181
280499241,4cyclictest4977-21perf19:15:001
2804992412,3cyclictest5960-21chrt20:26:031
280499241,17cyclictest0-21swapper/120:51:241
280499241,16cyclictest0-21swapper/122:05:071
280499241,16cyclictest0-21swapper/121:46:361
280499241,15cyclictest0-21swapper/122:15:141
280499241,14cyclictest0-21swapper/122:49:381
280499241,11cyclictest19167-21turbostat00:40:001
280499241,10cyclictest15071-21sh20:50:011
281599238,4cyclictest125712pluginstate00:25:023
281599231,16cyclictest0-21swapper/322:11:483
281599230,21cyclictest0-21swapper/323:55:213
280999236,5cyclictest1214-21cat22:45:152
280999233,7cyclictest24209-40dnf23:35:232
280499239,3cyclictest31710-21munin-run23:55:011
280499233,8cyclictest0-21swapper/123:29:211
280499232,11cyclictest29754-21smart_sda21:20:221
280499231,4cyclictest11750-21kworker/u8:123:10:561
280499231,3cyclictest0-21swapper/123:22:031
280499231,2cyclictest18873-21if_enp2s022:10:181
280499231,19cyclictest28018-21open_inodes22:30:221
280499231,19cyclictest28018-21open_inodes22:30:221
280499231,17cyclictest0-21swapper/100:29:481
280499231,16cyclictest0-21swapper/122:52:401
280499231,16cyclictest0-21swapper/119:46:021
280499231,16cyclictest0-21swapper/119:33:381
2804992311,3cyclictest24914-21irqstats21:10:181
280499231,10cyclictest32545-21sendmail_mailqu22:40:221
280499231,10cyclictest1877-21turbostat.cron20:20:011
280499231,10cyclictest1877-21turbostat.cron20:20:011
280499230,6cyclictest0-21swapper/100:22:161
280499230,17cyclictest0-21swapper/123:32:431
280499230,15cyclictest0-21swapper/120:00:161
280499230,10cyclictest0-21swapper/100:09:021
279899238,4cyclictest4978-21awk19:15:000
279899231,6cyclictest0-21swapper/021:28:190
279899231,6cyclictest0-21swapper/021:28:190
281599221,19cyclictest0-21swapper/322:30:253
281599221,19cyclictest0-21swapper/322:30:253
281599221,10cyclictest496-21jbd2/dm-0-822:39:233
281599221,10cyclictest2281-21sshd21:32:203
281599220,20cyclictest0-21swapper/323:30:243
280999224,3cyclictest0-21swapper/221:30:162
280999222,3cyclictest88150irq/126-enp1s0-21:41:192
280499226,8cyclictest0-21swapper/121:25:311
280499226,8cyclictest0-21swapper/121:25:311
280499222,12cyclictest88450irq/129-enp1s0-23:09:051
280499222,11cyclictest12199-21hddtemp_smartct21:55:161
280499221,4cyclictest0-21swapper/100:18:551
2804992214,2cyclictest0-21swapper/123:55:121
2804992213,3cyclictest0-21swapper/120:21:521
2804992213,3cyclictest0-21swapper/119:55:391
280499221,2cyclictest0-21swapper/122:04:251
280499221,15cyclictest0-21swapper/120:43:461
280499221,15cyclictest0-21swapper/120:35:441
280499221,15cyclictest0-21swapper/119:51:571
280499221,11cyclictest10466-21cstates19:25:121
2804992210,3cyclictest6547-21diskstats00:10:141
280499220,15cyclictest0-21swapper/100:00:101
279899224,12cyclictest17391-21unixbench-2d22:05:260
279899222,12cyclictest17778-21kworker/u8:122:30:240
279899222,12cyclictest17778-21kworker/u8:122:30:240
279899221,5cyclictest780-21in:imjournal21:50:020
279899221,5cyclictest0-21swapper/022:58:140
279899221,5cyclictest0-21swapper/022:49:480
279899221,4cyclictest6147-21kworker/u8:221:19:020
279899221,4cyclictest0-21swapper/023:00:510
9502211,16sleep2962-21ntp_states20:15:242
9502211,16sleep2962-21ntp_states20:15:242
281599217,8cyclictest0-21swapper/322:15:463
281599215,4cyclictest21450-21sshd21:02:193
281599212,7cyclictest1709-21fschecks_time22:45:173
281599211,9cyclictest31404-21sshd23:51:203
281599211,9cyclictest15944-21smart_sda19:35:223
281599211,8cyclictest13104-21/usr/sbin/munin20:45:263
281599211,6cyclictest0-21swapper/323:12:583
281599211,10cyclictest0-21swapper/300:17:493
2815992110,3cyclictest0-21swapper/300:28:573
281599210,20cyclictest0-21swapper/320:20:213
280999218,3cyclictest31644-21missed_timers21:25:182
280999218,3cyclictest31644-21missed_timers21:25:182
280999213,3cyclictest0-21swapper/221:35:122
280999211,5cyclictest4702-21idleruntime00:05:162
2809992114,4cyclictest5329-21taskset19:15:012
280999211,13cyclictest0-21swapper/220:00:112
280499216,3cyclictest0-21swapper/122:58:541
280499211,8cyclictest0-21swapper/121:50:101
280499211,17cyclictest29846-21munin-run20:10:011
280499211,15cyclictest0-21swapper/123:36:241
280499211,10cyclictest8438-21diskstats19:20:121
280499211,10cyclictest20822-21sed21:00:201
280499211,10cyclictest17228-21sa119:40:131
280499211,10cyclictest15019-21df19:35:121
279899213,14cyclictest88450irq/129-enp1s0-21:23:130
279899213,11cyclictest6767-21kworker/u8:222:40:060
279899212,11cyclictest27875-21/usr/sbin/munin20:05:200
279899211,6cyclictest0-21swapper/000:21:230
279899211,5cyclictest0-21swapper/000:16:450
279899211,4cyclictest0-21swapper/022:02:050
279899211,13cyclictest29115-21/usr/sbin/munin22:35:220
281599207,3cyclictest0-21swapper/321:35:343
281599206,3cyclictest12482-21/usr/sbin/munin19:30:113
281599204,10cyclictest8415-40dnf20:34:213
281599201,9cyclictest26910-21kworker/u8:019:29:183
281599201,8cyclictest0-21swapper/322:04:493
281599201,7cyclictest0-21swapper/323:25:523
281599201,7cyclictest0-21swapper/322:57:133
281599201,7cyclictest0-21swapper/322:27:533
281599201,7cyclictest0-21swapper/300:30:023
281599201,5cyclictest23869-21diskmemload21:45:183
281599201,10cyclictest0-21swapper/323:22:133
280999201,10cyclictest30990-21memory20:10:182
280999200,5cyclictest0-21swapper/222:30:232
280999200,5cyclictest0-21swapper/222:30:222
280499206,12cyclictest18942-21cat20:55:251
280499201,17cyclictest0-21swapper/120:12:071
280499201,15cyclictest0-21swapper/123:46:331
280499201,14cyclictest0-21swapper/120:30:311
2804992011,2cyclictest0-21swapper/119:19:541
279899206,6cyclictest6767-21kworker/u8:223:11:080
279899206,6cyclictest11-21rcuc/023:25:140
279899204,4cyclictest0-21swapper/000:10:130
279899203,13cyclictest88450irq/129-enp1s0-21:42:040
279899203,13cyclictest88450irq/129-enp1s0-00:28:300
279899202,4cyclictest18884-21ethtool22:10:180
279899202,10cyclictest10285-21/usr/sbin/munin19:25:190
279899201,6cyclictest0-21swapper/021:12:090
279899201,5cyclictest21384-21cat22:15:210
279899201,3cyclictest0-21swapper/022:52:060
279899201,2cyclictest0-21swapper/021:51:520
2798992011,8cyclictest0-21swapper/023:15:470
279899201,12cyclictest0-21swapper/023:30:260
279899201,11cyclictest25666-21/usr/sbin/munin20:00:220
281599195,3cyclictest0-21swapper/319:40:163
281599195,11cyclictest39-21rcuc/322:20:243
281599191,7cyclictest0-21swapper/321:23:443
281599191,6cyclictest0-21swapper/321:40:263
281599191,6cyclictest0-21swapper/321:13:493
281599191,6cyclictest0-21swapper/300:13:563
2815991916,2cyclictest0-21swapper/320:03:413
281599191,17cyclictest0-21swapper/320:53:093
281599191,16cyclictest0-21swapper/319:56:173
281599191,11cyclictest0-21swapper/323:47:043
281599191,10cyclictest0-21swapper/320:07:333
280999192,3cyclictest88150irq/126-enp1s0-22:00:132
280999192,14cyclictest22510-21find19:50:242
280999191,9cyclictest18143-21smart_sda19:40:222
280999191,9cyclictest0-21swapper/221:58:082
280999191,7cyclictest13104-21/usr/sbin/munin20:45:222
280999191,7cyclictest0-21swapper/219:13:052
280999191,6cyclictest8968-21ps23:00:252
280999191,6cyclictest0-21swapper/223:40:192
280999191,6cyclictest0-21swapper/223:40:192
280999191,6cyclictest0-21swapper/223:21:382
280999191,6cyclictest0-21swapper/223:15:052
280999191,6cyclictest0-21swapper/222:50:392
280999191,6cyclictest0-21swapper/222:40:092
280999191,6cyclictest0-21swapper/222:09:152
280999191,6cyclictest0-21swapper/221:50:512
280999191,6cyclictest0-21swapper/221:18:082
280999191,6cyclictest0-21swapper/221:10:202
280999191,6cyclictest0-21swapper/200:23:572
280999191,5cyclictest913-21sssd_nss19:29:122
280999191,5cyclictest0-21swapper/222:27:482
280999191,5cyclictest0-21swapper/222:14:222
280999191,4cyclictest0-21swapper/223:05:362
280999191,3cyclictest88150irq/126-enp1s0-21:22:492
279899192,9cyclictest3371-21users00:00:250
279899192,9cyclictest24317-21processes23:35:200
279899192,10cyclictest24717-21/usr/sbin/munin22:25:230
279899191,5cyclictest758-21dbus-daemon21:30:190
279899191,4cyclictest0-21swapper/020:40:220
279899191,3cyclictest0-21swapper/022:24:250
279899191,11cyclictest23869-21diskmemload21:55:160
279899191,10cyclictest23869-21diskmemload23:42:050
279899191,10cyclictest23869-21diskmemload23:42:050
279899191,10cyclictest2111-21/usr/sbin/munin20:20:240
77582180,3sleep30-21swapper/319:18:523
281599187,1cyclictest0-21swapper/321:50:153
281599184,3cyclictest11523-21/usr/sbin/munin21:55:123
281599182,3cyclictest4313-21/usr/sbin/munin20:25:203
281599181,6cyclictest17187-21/usr/sbin/munin00:35:133
281599181,5cyclictest3991-21/usr/sbin/munin00:05:123
2815991813,3cyclictest3373-21grep19:10:153
2815991813,3cyclictest3279-21grep00:00:253
281599181,15cyclictest0-21swapper/319:51:573
281599181,14cyclictest30071-21/usr/sbin/munin20:10:163
2815991810,3cyclictest9618-21cat20:35:193
281599180,9cyclictest0-21swapper/320:55:223
281599180,9cyclictest0-21swapper/320:17:173
281599180,9cyclictest0-21swapper/320:17:173
281599180,15cyclictest10905-21/usr/sbin/munin20:40:133
280999185,2cyclictest88450irq/129-enp1s0-00:10:142
280999185,2cyclictest88150irq/126-enp1s0-00:15:202
280999184,3cyclictest23869-21diskmemload22:35:152
280999184,2cyclictest88350irq/128-enp1s0-23:30:172
280999182,4cyclictest17272-21munin-run20:55:022
280999181,6cyclictest25437-21latency_hist20:00:002
280999181,6cyclictest0-21swapper/223:50:512
280999181,6cyclictest0-21swapper/223:27:492
280999181,6cyclictest0-21swapper/222:22:502
280999181,6cyclictest0-21swapper/200:31:462
280999181,6cyclictest0-21swapper/200:02:432
280999181,5cyclictest0-21swapper/223:46:072
280999181,5cyclictest0-21swapper/223:13:162
280999181,5cyclictest0-21swapper/221:48:002
280999181,5cyclictest0-21swapper/219:45:202
280999181,5cyclictest0-21swapper/200:26:242
280999181,4cyclictest5646-21grep22:55:122
280999181,4cyclictest0-21swapper/222:16:072
280999181,4cyclictest0-21swapper/220:31:292
280999181,3cyclictest29298-21timerwakeupswit20:05:242
2809991813,3cyclictest13163-21hddtemp_smartct19:30:152
280999181,11cyclictest0-21swapper/200:35:252
280999180,5cyclictest0-21swapper/219:35:122
279899188,8cyclictest88150irq/126-enp1s0-20:50:170
279899186,3cyclictest88350irq/128-enp1s0-20:35:190
279899185,3cyclictest18874-21sort20:55:240
279899183,8cyclictest4212-21expr00:05:130
279899182,3cyclictest17872-21strings19:40:190
279899181,4cyclictest0-21swapper/019:30:120
2798991811,4cyclictest21376-21tr19:50:140
82222171,3sleep00-21swapper/020:31:520
281599171,6cyclictest0-21swapper/323:07:113
281599171,5cyclictest9680-21who19:20:253
281599171,14cyclictest0-21swapper/322:50:163
2815991710,4cyclictest26584-21expr21:15:153
281599170,9cyclictest0-21swapper/319:45:153
280999172,5cyclictest0-21swapper/221:02:542
280999171,5cyclictest0-21swapper/220:22:362
280999171,4cyclictest21901-21/usr/sbin/munin21:05:132
280999171,4cyclictest0-21swapper/220:27:092
2809991712,3cyclictest8646-21perl19:20:142
279899178,2cyclictest88150irq/126-enp1s0-19:58:580
279899172,8cyclictest21029-21cat19:49:570
279899171,7cyclictest0-21swapper/021:05:170
279899171,3cyclictest0-21swapper/019:20:080
279899171,2cyclictest88150irq/126-enp1s0-20:25:120
279899171,2cyclictest0-21swapper/019:15:140
281599161,12cyclictest21901-21/usr/sbin/munin21:05:213
280999161,3cyclictest0-21swapper/220:35:172
280999161,14cyclictest0-21swapper/220:40:442
280999160,6cyclictest0-21swapper/220:57:512
279899161,14cyclictest0-21swapper/020:16:590
279899161,14cyclictest0-21swapper/020:16:590
2798991611,3cyclictest13597-21fschecks_time20:45:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional