You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 13:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Jul 15, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113752238163,24sleep30-21swapper/319:08:253
113672238164,24sleep20-21swapper/219:08:202
111652233163,57sleep00-21swapper/019:05:410
113402232165,22sleep10-21swapper/119:08:001
3064321260,4sleep01157399cyclictest22:15:270
11587998783,3cyclictest0-21swapper/219:15:062
170902560,3sleep339-21rcuc/319:20:133
119512560,2sleep20-21swapper/221:35:162
67192460,2sleep30-21swapper/320:10:143
1157399431,26cyclictest30051-21/usr/sbin/munin19:50:210
1158199422,20cyclictest17263-21munin-run20:35:011
11587993936,2cyclictest0-21swapper/222:17:172
1157399391,18cyclictest0-21swapper/000:00:210
1157399390,31cyclictest13809-21awk00:05:220
1157399388,22cyclictest3337-21/usr/sbin/munin22:30:180
1157399381,24cyclictest32647-21crond21:10:020
11581993712,3cyclictest669-21expr21:10:121
1157399371,23cyclictest31296-21/usr/sbin/munin22:20:190
224412360,1sleep10-21swapper/120:45:141
1157399362,25cyclictest1464-21diskstats22:25:140
1157399361,26cyclictest9929-21/usr/sbin/munin22:45:170
1157399361,25cyclictest19265-21sed00:20:020
1159199351,25cyclictest0-21swapper/323:32:303
1157399352,18cyclictest0-21swapper/023:30:130
1157399351,26cyclictest12264-21if_enp2s021:35:180
1157399351,21cyclictest10677-21basename20:20:000
1157399351,20cyclictest23459-21/usr/sbin/munin19:35:170
1157399351,19cyclictest13046-21memory22:50:190
1159199340,27cyclictest0-21swapper/320:15:263
1157399345,20cyclictest26275-21/usr/sbin/munin20:55:290
1157399341,24cyclictest32650-21diskmemload21:29:240
1157399341,24cyclictest23766-21/usr/sbin/munin00:30:140
1157399341,23cyclictest29597-21chrt19:47:140
1157399341,23cyclictest0-21swapper/021:00:560
1159199331,12cyclictest0-21swapper/320:29:043
1158199331,15cyclictest0-21swapper/119:14:371
1157399336,19cyclictest87550irq/128-enp1s0-23:35:120
1157399334,19cyclictest15754-21cat22:55:260
1157399334,19cyclictest15754-21cat22:55:260
1157399332,22cyclictest87550irq/128-enp1s0-22:40:180
1157399332,16cyclictest6268-21munin-run20:10:010
1157399331,23cyclictest16870-21/usr/sbin/munin19:20:250
1157399330,22cyclictest0-21swapper/021:10:160
1159199320,25cyclictest0-21swapper/322:14:133
1158199321,14cyclictest0-21swapper/122:26:091
1157399324,20cyclictest87550irq/128-enp1s0-23:45:010
1157399322,26cyclictest1633-40dnf19:57:200
1157399322,22cyclictest18737-21/usr/sbin/munin23:05:230
1157399321,22cyclictest16530-21/usr/sbin/munin23:00:190
1157399321,21cyclictest27524-21/usr/sbin/munin23:25:230
1157399321,21cyclictest20114-21date21:55:010
1157399321,19cyclictest21263-21/usr/sbin/munin19:30:140
1157399321,19cyclictest10205-21runrttasks23:58:380
1157399321,18cyclictest15512-21expr20:30:120
1157399321,18cyclictest11783-21/usr/sbin/munin19:10:170
1157399321,16cyclictest0-21swapper/000:26:410
1157399320,24cyclictest20639-21swap00:20:250
1158199311,13cyclictest0-21swapper/120:40:451
1157399316,22cyclictest0-21swapper/020:10:260
1157399311,23cyclictest5991-21ntpq21:20:230
1157399311,22cyclictest23477-21diskstats23:15:140
1157399311,19cyclictest26989-21runrttasks00:35:210
1157399311,18cyclictest27361-21fschecks_count22:10:160
1159199302,12cyclictest20440-21ntp_states00:20:213
1159199301,21cyclictest0-21swapper/323:19:183
1159199301,13cyclictest0-21swapper/319:55:053
1159199300,23cyclictest0-21swapper/323:03:183
1159199300,23cyclictest0-21swapper/322:33:423
1158199301,26cyclictest0-21swapper/100:13:091
1158199301,15cyclictest0-21swapper/121:38:141
1158199301,14cyclictest0-21swapper/120:04:091
1158199301,12cyclictest0-21swapper/121:07:371
1157399301,26cyclictest6388-21sort22:35:170
1157399301,26cyclictest24708-21/usr/sbin/munin22:05:180
1157399301,26cyclictest24077-21/usr/sbin/munin20:50:200
1157399301,25cyclictest22129-40dnf21:58:220
1157399301,25cyclictest20132-21ntp_states19:25:220
1157399301,25cyclictest20132-21ntp_states19:25:220
1159199291,13cyclictest0-21swapper/321:12:493
1158199291,12cyclictest0-21swapper/123:08:091
1158199291,12cyclictest0-21swapper/121:03:211
1157399295,21cyclictest14376-21wc20:25:250
1157399291,25cyclictest6154-21/usr/sbin/munin23:50:140
1157399291,25cyclictest3953-21/usr/sbin/munin23:45:220
1157399291,25cyclictest26480-21/usr/sbin/munin23:20:230
1157399291,25cyclictest20933-21/usr/sbin/munin23:10:200
1157399291,25cyclictest19683-21/usr/sbin/munin20:40:150
1157399291,25cyclictest15905-21/usr/sbin/munin21:45:170
1157399291,25cyclictest15905-21/usr/sbin/munin21:45:170
1157399291,24cyclictest22843-21df_inode22:00:140
1157399291,24cyclictest14056-21/usr/sbin/munin21:40:150
1157399291,19cyclictest87550irq/128-enp1s0-00:14:410
1157399291,18cyclictest87350irq/126-enp1s0-21:30:160
1159199285,15cyclictest0-21swapper/321:45:153
1159199285,15cyclictest0-21swapper/321:45:153
11591992812,3cyclictest2814-21ntp_states23:40:203
1159199281,13cyclictest0-21swapper/300:16:013
1159199281,10cyclictest0-21swapper/321:08:253
1159199280,12cyclictest0-21swapper/323:58:213
1159199280,12cyclictest0-21swapper/322:39:023
1159199280,12cyclictest0-21swapper/300:01:013
1159199280,11cyclictest0-21swapper/320:42:423
1158199281,18cyclictest0-21swapper/123:00:131
1158199281,15cyclictest0-21swapper/120:56:571
1158199281,13cyclictest0-21swapper/122:35:571
1158199281,13cyclictest0-21swapper/119:15:491
1158199281,11cyclictest0-21swapper/122:44:171
1158199281,11cyclictest0-21swapper/122:02:571
1158199281,11cyclictest0-21swapper/100:33:341
1158199281,10cyclictest0-21swapper/121:45:421
1158199281,10cyclictest0-21swapper/121:45:421
1158199281,10cyclictest0-21swapper/120:16:491
1157399289,16cyclictest15014-21sh19:15:130
1157399281,24cyclictest2717-21/usr/sbin/munin21:15:180
1157399281,24cyclictest17487-21/usr/sbin/munin20:35:130
1157399281,24cyclictest17487-21/usr/sbin/munin20:35:130
1157399281,24cyclictest11578-21/usr/sbin/munin20:20:170
11573992812,13cyclictest22763-21sort20:45:170
11573992811,14cyclictest3011-21cat20:00:190
11573992811,14cyclictest25833-21cstates19:40:110
1159199271,9cyclictest0-21swapper/320:53:223
1159199271,18cyclictest0-21swapper/319:17:183
1159199271,17cyclictest0-21swapper/320:30:303
1159199271,11cyclictest0-21swapper/321:40:473
1159199271,11cyclictest0-21swapper/300:28:493
1159199270,3cyclictest0-21swapper/320:45:403
1159199270,12cyclictest0-21swapper/323:39:103
1159199270,12cyclictest0-21swapper/300:32:413
1159199270,11cyclictest0-21swapper/323:47:383
1159199270,11cyclictest0-21swapper/322:54:223
1158799277,11cyclictest0-21swapper/222:45:192
1158199271,18cyclictest0-21swapper/122:56:411
1158199271,18cyclictest0-21swapper/122:56:411
1158199271,18cyclictest0-21swapper/121:29:221
1158199271,13cyclictest0-21swapper/123:50:331
1158199271,13cyclictest0-21swapper/122:51:371
1158199271,13cyclictest0-21swapper/121:24:141
1158199271,10cyclictest0-21swapper/119:21:331
1158199271,10cyclictest0-21swapper/100:38:221
1159199261,9cyclictest0-21swapper/321:57:013
1159199261,16cyclictest0-21swapper/322:19:253
1159199261,16cyclictest0-21swapper/321:04:133
1159199261,11cyclictest0-21swapper/322:20:533
1159199261,11cyclictest0-21swapper/320:35:583
1159199261,11cyclictest0-21swapper/320:35:583
1159199261,10cyclictest32650-21diskmemload00:08:373
1159199260,17cyclictest0-21swapper/321:52:053
1159199260,12cyclictest0-21swapper/323:53:053
1158199262,11cyclictest26673-21grep23:20:261
1158199261,12cyclictest0-21swapper/121:53:021
1158199261,11cyclictest32650-21diskmemload23:46:011
1158199261,11cyclictest0-21swapper/122:49:571
1159199251,12cyclictest32650-21diskmemload22:01:333
1159199251,12cyclictest0-21swapper/321:37:413
1159199251,12cyclictest0-21swapper/320:03:253
1159199251,11cyclictest0-21swapper/321:15:093
1159199250,9cyclictest0-21swapper/320:56:093
1159199250,9cyclictest0-21swapper/319:53:253
1159199250,17cyclictest0-21swapper/322:57:343
1159199250,17cyclictest0-21swapper/322:57:343
1159199250,12cyclictest0-21swapper/321:34:333
1159199250,12cyclictest0-21swapper/321:28:413
1159199250,11cyclictest0-21swapper/320:07:393
1159199250,11cyclictest0-21swapper/319:38:053
1158799255,7cyclictest28026-40dnf20:57:242
1158199259,4cyclictest12157-21users00:00:291
1158199254,12cyclictest0-21swapper/122:10:191
1158199251,14cyclictest0-21swapper/121:58:341
1158199251,11cyclictest0-21swapper/123:10:251
1158199251,11cyclictest0-21swapper/122:31:331
1158199251,11cyclictest0-21swapper/100:09:051
1158199251,10cyclictest0-21swapper/123:39:331
1158199251,10cyclictest0-21swapper/119:56:381
11581992510,3cyclictest0-21swapper/120:20:291
1159199241,7cyclictest15897-21memory00:10:193
1159199241,11cyclictest827-21runrttasks19:14:403
1159199241,11cyclictest0-21swapper/322:06:253
1159199241,10cyclictest0-21swapper/319:25:513
1159199241,10cyclictest0-21swapper/319:25:513
1159199240,9cyclictest0-21swapper/323:29:023
1159199240,11cyclictest0-21swapper/322:42:143
1158799241,7cyclictest0-21swapper/223:06:332
1158199241,12cyclictest27246-21users19:40:261
1158199241,11cyclictest0-21swapper/119:34:381
1158199241,10cyclictest0-21swapper/123:41:011
11581992410,3cyclictest0-21swapper/120:10:211
1159199231,7cyclictest0-21swapper/300:38:253
1159199230,9cyclictest0-21swapper/323:07:143
11587992310,3cyclictest1523-21nfs4_client21:10:192
1158199238,3cyclictest24424-21diskstats20:50:151
1158199237,9cyclictest0-21swapper/100:20:081
1158199231,19cyclictest29444-21users19:45:281
1158199231,16cyclictest0-21swapper/100:26:131
1158199231,14cyclictest0-21swapper/119:29:541
1158199231,14cyclictest0-21swapper/119:29:531
1158199231,11cyclictest0-21swapper/121:16:291
1158199231,11cyclictest0-21swapper/119:53:101
1158199231,10cyclictest0-21swapper/122:06:571
1158199231,10cyclictest0-21swapper/120:36:131
1158199231,10cyclictest0-21swapper/120:36:131
1158199231,10cyclictest0-21swapper/120:08:571
1159199226,10cyclictest0-21swapper/320:20:543
1159199221,12cyclictest0-21swapper/321:22:453
1159199220,9cyclictest0-21swapper/322:47:023
1158199227,3cyclictest32209-21memory22:20:191
1158199227,3cyclictest0-21swapper/121:43:061
1158199227,3cyclictest0-21swapper/100:16:291
1158199226,3cyclictest0-21swapper/123:27:421
1158199221,9cyclictest32650-21diskmemload23:57:411
1158199221,10cyclictest0-21swapper/123:17:331
1159199216,3cyclictest0-21swapper/319:46:493
1159199215,8cyclictest0-21swapper/319:44:413
1159199214,9cyclictest0-21swapper/319:30:573
1159199211,6cyclictest0-21swapper/322:26:023
1159199210,9cyclictest0-21swapper/323:14:543
1158799212,16cyclictest30661-21tail19:50:172
1158799211,5cyclictest0-21swapper/222:41:442
1158799211,14cyclictest0-21swapper/221:55:552
1158199216,4cyclictest0-21swapper/123:32:091
1158199211,13cyclictest0-21swapper/119:39:021
1158199211,12cyclictest0-21swapper/120:25:051
1158199211,10cyclictest0-21swapper/122:16:291
1159199206,3cyclictest0-21swapper/323:20:123
1158799206,4cyclictest12050irq/123-ahci[0020:40:012
1158799206,4cyclictest12050irq/123-ahci[0020:40:012
1158799202,4cyclictest0-21swapper/221:32:572
1158799201,7cyclictest0-21swapper/200:15:482
1158799201,6cyclictest13763-21perf19:15:012
1158799201,6cyclictest0-21swapper/200:30:232
11587992014,3cyclictest5532-21find20:05:232
1158199207,3cyclictest0-21swapper/121:30:061
60832191,3sleep20-21swapper/221:20:242
1158799199,3cyclictest87550irq/128-enp1s0-23:29:252
1158799196,3cyclictest645-21grep19:55:202
1158799191,8cyclictest18097-21taskset21:50:102
1158799191,7cyclictest0-21swapper/220:15:182
1158799191,5cyclictest10198-40dnf23:58:352
1158799191,5cyclictest0-21swapper/223:50:162
1158799191,4cyclictest0-21swapper/223:36:542
1158799191,4cyclictest0-21swapper/200:05:062
1158799191,15cyclictest9091-21perf21:30:002
1158799191,12cyclictest0-21swapper/222:12:582
11587991910,2cyclictest21667-21awk20:45:012
1158799182,4cyclictest15460-21sshd21:42:202
1158799181,9cyclictest0-21swapper/223:02:202
1158799181,7cyclictest790-21in:imjournal20:20:212
1158799181,7cyclictest12547-21turbostat00:05:012
1158799181,7cyclictest0-21swapper/223:14:172
1158799181,7cyclictest0-21swapper/222:52:222
1158799181,6cyclictest0-21swapper/222:30:272
1158799181,6cyclictest0-21swapper/221:46:122
1158799181,6cyclictest0-21swapper/221:46:122
1158799181,5cyclictest0-21swapper/222:23:122
1158799181,5cyclictest0-21swapper/220:11:242
1158799181,5cyclictest0-21swapper/200:28:352
1158799181,4cyclictest5540-21users23:45:252
1158799181,4cyclictest0-21swapper/222:36:292
1158799181,4cyclictest0-21swapper/222:27:162
11587991813,3cyclictest30680-21/usr/sbin/munin21:05:202
11587991813,3cyclictest25519-21irqstats22:05:212
1158799181,15cyclictest0-21swapper/220:26:502
1158799172,6cyclictest2088-21/usr/sbin/munin20:00:122
1158799172,12cyclictest29822-21munin-plugin-st19:50:012
1158799171,9cyclictest0-21swapper/222:55:042
1158799171,9cyclictest0-21swapper/222:55:032
1158799171,6cyclictest0-21swapper/223:20:202
1158799171,6cyclictest0-21swapper/222:00:452
1158799171,5cyclictest21889-21gltestperf19:30:142
1158799171,4cyclictest0-21swapper/200:21:362
1158799171,15cyclictest0-21swapper/223:40:262
1158799171,15cyclictest0-21swapper/200:36:552
11587991711,3cyclictest16460-21grep00:10:262
1158799162,5cyclictest30-21rcuc/220:30:242
1158799162,5cyclictest22433-21perl20:45:142
1158799161,6cyclictest23921-21/usr/sbin/munin23:15:182
1158799161,6cyclictest0-21swapper/220:50:172
1158799161,6cyclictest0-21swapper/219:25:152
1158799161,6cyclictest0-21swapper/219:25:152
1158799161,5cyclictest17181-21expr19:20:132
1158799161,5cyclictest0-21swapper/223:32:152
1158799161,4cyclictest0-21swapper/221:16:162
1158799161,3cyclictest0-21swapper/219:39:402
11587991611,3cyclictest25754-21tr19:40:112
11587991610,4cyclictest802-21sssd_be21:01:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional