You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-11-15 - 06:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Nov 15, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7622238162,25sleep00-21swapper/019:08:060
6652236163,24sleep20-21swapper/219:07:002
5782236163,59sleep30-21swapper/319:06:003
6152202165,24sleep10-21swapper/119:06:261
260421680,3sleep2106499cyclictest00:05:132
260421680,3sleep2106499cyclictest00:05:132
1373021600,4sleep3107199cyclictest00:30:153
2452621330,2sleep0105299cyclictest23:40:190
2631421280,2sleep20-21swapper/221:17:152
1064991181,115cyclictest0-21swapper/222:13:382
106499104101,2cyclictest0-21swapper/221:44:372
152652530,1sleep00-21swapper/022:05:200
125932530,1sleep10-21swapper/120:45:231
131942500,1sleep20-21swapper/223:15:132
223582450,4sleep011-21rcuc/021:10:020
199192350,5sleep1105999cyclictest22:15:241
107199331,24cyclictest0-21swapper/319:44:293
105299333,14cyclictest14925-21munin-run19:40:010
1052993321,6cyclictest8496-40dnf00:18:240
105299328,5cyclictest26766-21/usr/sbin/munin21:20:190
105299326,12cyclictest26557-21perf21:20:010
105299324,19cyclictest1603-21unixbench_singl00:00:270
105299323,13cyclictest13986-21awk22:05:010
105299319,11cyclictest3833-21chrt21:40:160
107199300,23cyclictest0-21swapper/321:53:453
105299305,13cyclictest22148-21diskmemload23:50:130
105299304,12cyclictest6764-21idleruntime-cro20:35:010
105999290,22cyclictest0-21swapper/122:35:311
105299297,13cyclictest983-21/usr/sbin/munin21:35:230
105299294,12cyclictest8336-21munin-run19:25:010
105299294,10cyclictest28526-21tr22:35:230
105299291,13cyclictest23726-21awk20:00:010
107199281,9cyclictest0-21swapper/323:35:173
107199281,10cyclictest0-21swapper/319:49:273
107199280,13cyclictest0-21swapper/300:21:433
107199280,12cyclictest0-21swapper/321:21:533
107199280,11cyclictest0-21swapper/322:08:373
105299289,7cyclictest6019-21/usr/sbin/munin23:00:220
105299287,9cyclictest20348-40dnf22:17:230
105299286,13cyclictest30197-21/usr/sbin/munin23:55:240
105299285,4cyclictest0-21swapper/023:19:290
1052992812,4cyclictest11237-21irqstats23:10:180
107199271,10cyclictest0-21swapper/321:45:033
107199270,13cyclictest0-21swapper/323:30:373
107199270,11cyclictest0-21swapper/319:35:033
105299275,4cyclictest11063-21/usr/sbin/munin00:25:240
105299275,18cyclictest10380-21unixbench_multi00:20:250
105299275,10cyclictest13257-21runrttasks00:30:100
107199261,12cyclictest0-21swapper/320:21:273
107199261,10cyclictest6294-21taskset00:13:153
107199260,13cyclictest1042-21cyclictest19:51:353
107199260,10cyclictest0-21swapper/321:55:173
1059992612,12cyclictest8281-21kworker/u8:219:16:431
105999261,16cyclictest7546-21awk23:00:261
105299265,6cyclictest30540-21/usr/sbin/munin20:15:240
1052992612,6cyclictest28343-21/usr/sbin/munin20:10:140
14712250,1sleep20-21swapper/219:10:132
107199251,12cyclictest0-21swapper/322:01:153
107199251,11cyclictest0-21swapper/322:11:143
107199251,11cyclictest0-21swapper/319:22:093
107199251,10cyclictest0-21swapper/323:42:253
107199250,9cyclictest0-21swapper/300:15:093
107199250,13cyclictest0-21swapper/321:34:373
107199250,12cyclictest0-21swapper/320:50:253
107199250,10cyclictest0-21swapper/322:51:293
107199250,10cyclictest0-21swapper/321:36:093
105999252,19cyclictest26310-40dnf21:17:141
105299257,6cyclictest26896-21/usr/sbin/munin23:45:220
105299256,10cyclictest0-21swapper/022:50:220
107199248,3cyclictest0-21swapper/321:15:393
107199241,9cyclictest0-21swapper/323:48:513
107199241,12cyclictest0-21swapper/320:33:473
107199241,10cyclictest0-21swapper/321:13:593
107199240,11cyclictest0-21swapper/323:05:293
107199240,11cyclictest0-21swapper/322:45:133
107199240,10cyclictest0-21swapper/323:14:113
107199240,10cyclictest0-21swapper/300:06:063
107199240,10cyclictest0-21swapper/300:06:053
106499244,11cyclictest0-21swapper/222:35:212
105999249,12cyclictest87650irq/129-enp1s0-23:40:141
105299246,4cyclictest4469-21/usr/sbin/munin00:10:240
105299245,6cyclictest22152-21munin-run21:10:010
105299244,8cyclictest32739-21/usr/sbin/munin20:20:240
105299244,5cyclictest87350irq/126-enp1s0-23:25:100
105299243,17cyclictest2262-21/usr/sbin/munin00:05:130
105299243,17cyclictest2262-21/usr/sbin/munin00:05:130
1052992414,6cyclictest15803-21diskstats00:35:140
1052992410,5cyclictest7616-21egrep21:50:120
103102240,4sleep09-21rcu_preempt23:10:010
107199238,3cyclictest0-21swapper/300:26:353
107199237,3cyclictest0-21swapper/323:15:473
107199237,3cyclictest0-21swapper/319:15:403
107199231,16cyclictest0-21swapper/320:41:213
107199231,10cyclictest0-21swapper/322:16:183
107199230,10cyclictest0-21swapper/323:55:553
105999231,4cyclictest6439-21turbostat00:15:011
1059992314,3cyclictest0-21swapper/120:01:431
1059992312,3cyclictest25423-21expr22:30:141
105999231,20cyclictest0-21swapper/123:15:021
105999231,10cyclictest14876-21hddtemp_smartct22:05:151
105299239,10cyclictest14923-21cpuspeed_turbos23:20:100
105299238,6cyclictest0-21swapper/022:10:450
105299234,6cyclictest10763-21/usr/sbin/munin19:30:280
105299233,18cyclictest0-21swapper/022:40:260
105299233,11cyclictest87350irq/126-enp1s0-19:40:110
105299232,9cyclictest5889-21sshd19:16:550
1052992315,5cyclictest30230-21smart_sda21:25:250
105299231,11cyclictest31773-21/usr/sbin/munin22:45:260
105299231,11cyclictest0-21swapper/019:25:250
105299230,14cyclictest5401-21/usr/sbin/munin21:45:190
105299230,12cyclictest4977-1kworker/0:2H20:55:010
107199228,3cyclictest0-21swapper/320:07:393
107199228,3cyclictest0-21swapper/300:38:073
107199227,3cyclictest0-21swapper/323:21:433
107199227,3cyclictest0-21swapper/321:27:433
107199226,3cyclictest0-21swapper/321:00:553
107199225,3cyclictest0-21swapper/319:58:233
107199220,9cyclictest0-21swapper/322:34:333
107199220,9cyclictest0-21swapper/320:58:113
107199220,11cyclictest0-21swapper/323:50:273
107199220,11cyclictest0-21swapper/322:29:553
107199220,10cyclictest0-21swapper/322:36:473
106499224,5cyclictest0-21swapper/222:20:232
105999228,4cyclictest22780-21sleep22:25:011
1059992216,3cyclictest31107-21tail22:40:281
1059992215,4cyclictest528-21perl00:00:171
1059992212,3cyclictest18614-21smart_sda19:45:241
105999221,18cyclictest7284-21memory19:20:181
105999221,15cyclictest0-21swapper/122:48:311
105999221,14cyclictest87650irq/129-enp1s0-21:39:241
105999220,20cyclictest0-21swapper/123:27:381
105299225,9cyclictest87550irq/128-enp1s0-21:00:250
105299223,8cyclictest0-21swapper/020:40:130
105299223,16cyclictest19704-21fschecks_time23:30:150
1052992215,4cyclictest31974-21irqstats21:30:170
1052992214,5cyclictest3848-21smart_sda20:25:240
1052992214,5cyclictest11723-21diskstats20:45:140
1052992213,6cyclictest24082-21/usr/sbin/munin22:25:210
1052992213,5cyclictest27410-21smart_sda20:05:240
1052992213,5cyclictest20813-21smart_sda19:50:250
1052992213,5cyclictest10148-21diskstats21:55:150
105299221,12cyclictest25192-21/usr/sbin/munin22:30:150
105299220,6cyclictest87350irq/126-enp1s0-22:20:160
107199217,3cyclictest0-21swapper/319:34:433
107199211,7cyclictest30378-21if_err_enp2s022:40:203
107199211,11cyclictest30540-21/usr/sbin/munin20:15:123
106499214,5cyclictest0-21swapper/200:32:592
106499213,4cyclictest5971-21grep23:00:012
106499211,9cyclictest0-21swapper/222:19:112
106499211,6cyclictest25363-21chrt23:42:232
106499211,3cyclictest9838-21missed_timers00:20:182
105999216,2cyclictest87550irq/128-enp1s0-23:21:121
1059992117,3cyclictest0-21swapper/120:58:351
1059992116,3cyclictest0-21swapper/119:34:551
105999211,3cyclictest15645-21latency_hist20:55:011
1059992113,3cyclictest0-21swapper/119:59:031
1059992113,3cyclictest0-21swapper/100:08:381
1059992113,3cyclictest0-21swapper/100:08:381
105999211,17cyclictest27805-21chrt20:06:111
105999211,14cyclictest0-21swapper/123:37:111
105999211,14cyclictest0-21swapper/123:09:311
105999211,11cyclictest7621-21egrep21:50:121
105999210,5cyclictest0-21swapper/121:22:271
105999210,16cyclictest0-21swapper/122:29:301
105999210,16cyclictest0-21swapper/119:10:181
105299216,3cyclictest808-21snmpd19:45:040
105299215,5cyclictest87650irq/129-enp1s0-19:10:240
1052992115,4cyclictest15829-21/usr/sbin/munin20:55:120
107199209,4cyclictest0-21swapper/322:20:233
107199206,3cyclictest0-21swapper/319:12:233
107199205,3cyclictest0-21swapper/321:09:353
107199204,3cyclictest0-21swapper/320:37:273
107199204,3cyclictest0-21swapper/320:37:273
107199203,3cyclictest0-21swapper/320:13:113
107199201,7cyclictest6358-21sh23:00:133
107199201,11cyclictest2585-21/usr/sbin/munin20:25:203
106499201,4cyclictest12997-21netstat22:00:202
105999209,3cyclictest26148-21df_inode23:45:131
105999206,3cyclictest0-21swapper/120:27:501
105999205,4cyclictest2951-21switchtime22:50:241
105999202,11cyclictest13498-21expr00:30:121
1059992016,3cyclictest0-21swapper/120:12:321
105999201,18cyclictest0-21swapper/121:25:011
105999201,17cyclictest0-21swapper/121:43:101
105999201,16cyclictest827-21runrttasks21:13:161
105999201,14cyclictest0-21swapper/122:55:261
105999201,14cyclictest0-21swapper/100:19:281
105999201,11cyclictest29073-21ntp_states23:50:211
105999201,10cyclictest11852-21cat22:00:011
105999200,2cyclictest87650irq/129-enp1s0-23:32:441
105999200,18cyclictest0-21swapper/121:00:171
105999200,14cyclictest0-21swapper/100:21:321
105999200,13cyclictest87650irq/129-enp1s0-23:15:271
1052992012,5cyclictest25528-21users20:00:260
1052992011,6cyclictest909-21sssd_nss20:35:220
1052992011,6cyclictest909-21sssd_nss20:35:220
1052992011,6cyclictest5081-21smart_sda22:55:220
1052992011,6cyclictest21412-21/usr/sbin/munin23:35:170
321002191,2sleep20-21swapper/223:59:212
107199194,3cyclictest9635-21ntp_states19:25:233
107199194,3cyclictest788-21interrupts00:00:193
107199191,11cyclictest3203-21/usr/sbin/munin21:40:253
106499198,3cyclictest2362-21awk20:25:012
106499196,4cyclictest22965-21gltestperf21:10:162
106499196,3cyclictest17751-21awk21:00:012
106499194,9cyclictest0-21swapper/221:21:012
106499194,9cyclictest0-21swapper/200:17:412
106499193,9cyclictest0-21swapper/221:59:152
106499192,7cyclictest1615-21/usr/sbin/munin22:50:252
106499191,7cyclictest30961-21fschecks_count20:15:142
106499191,7cyclictest152662uname22:05:202
106499191,7cyclictest0-21swapper/223:04:422
106499191,7cyclictest0-21swapper/219:40:172
106499191,6cyclictest0-21swapper/223:50:042
106499191,12cyclictest0-21swapper/200:11:172
106499191,10cyclictest0-21swapper/200:35:212
105999194,12cyclictest14320-21/usr/sbin/munin19:35:221
1059991915,3cyclictest0-21swapper/121:30:591
1059991915,3cyclictest0-21swapper/120:40:281
105999191,2cyclictest0-21swapper/100:25:521
105999191,17cyclictest0-21swapper/119:40:411
105999191,14cyclictest0-21swapper/100:35:351
105999191,12cyclictest0-21swapper/122:11:121
105999190,6cyclictest0-21swapper/121:47:581
105999190,14cyclictest0-21swapper/123:56:391
220932180,2sleep20-21swapper/221:09:332
136272181,3sleep10-21swapper/122:00:531
107199181,7cyclictest11440-21sshd20:45:123
107199181,3cyclictest0-21swapper/322:56:033
107199181,15cyclictest0-21swapper/320:00:543
107199180,10cyclictest0-21swapper/323:29:493
106499188,7cyclictest30240-21gltestperf22:40:182
106499187,3cyclictest12050irq/123-ahci[0022:34:562
106499182,10cyclictest0-21swapper/221:54:062
106499181,7cyclictest30931-21turbostat21:30:002
106499181,6cyclictest18991-21munin-run23:30:012
106499181,5cyclictest0-21swapper/222:28:112
106499181,3cyclictest18614-21smart_sda19:45:242
106499181,11cyclictest0-21swapper/223:06:352
106499181,11cyclictest0-21swapper/220:25:202
1059991814,3cyclictest0-21swapper/120:18:411
105999181,16cyclictest0-21swapper/120:35:111
105999181,16cyclictest0-21swapper/120:35:101
105999181,16cyclictest0-21swapper/120:21:571
105999181,16cyclictest0-21swapper/119:25:091
105999180,17cyclictest0-21swapper/121:08:001
210622170,2sleep20-21swapper/223:33:372
106499177,2cyclictest87650irq/129-enp1s0-19:55:262
106499174,4cyclictest32591-21timerwakeupswit21:30:252
106499174,4cyclictest32-21ksoftirqd/223:45:182
106499172,3cyclictest6033-21gltestperf21:45:152
106499171,8cyclictest936-21load00:00:202
106499171,7cyclictest22171-21awk23:35:182
106499171,6cyclictest0-21swapper/200:27:102
1064991713,3cyclictest0-21swapper/220:39:052
1064991713,3cyclictest0-21swapper/220:39:052
1064991712,3cyclictest7654-21sort19:20:232
1064991712,3cyclictest4167-21/usr/sbin/munin19:15:252
106499171,13cyclictest11397-21head19:30:162
1064991710,4cyclictest26621-21cut20:05:152
1064991710,4cyclictest11064-21cut23:10:162
1064991710,4cyclictest0-21swapper/223:20:122
1059991713,3cyclictest0-21swapper/119:51:481
1059991711,4cyclictest6754-21grep20:35:021
106499164,3cyclictest10392-21processes20:40:242
106499163,4cyclictest14850-21sshd19:39:122
106499161,7cyclictest32555-21cat22:45:162
106499161,6cyclictest20106-21cut19:50:152
106499161,6cyclictest13922-21diskstats20:50:142
1064991612,3cyclictest0-21swapper/219:26:282
106499161,14cyclictest0-21swapper/220:03:182
1064991611,3cyclictest4782-21/usr/sbin/munin20:30:112
1064991611,3cyclictest17966-21/usr/sbin/munin21:00:142
1064991611,3cyclictest11794-21fschecks_count20:45:152
106499152,3cyclictest87350irq/126-enp1s0-21:35:132
1064991510,3cyclictest28886-21cut20:10:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional