You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 09:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 30, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
307972244178,55sleep20-21swapper/219:08:232
306862202165,25sleep30-21swapper/319:06:553
306152201165,24sleep10-21swapper/119:06:021
306362198162,24sleep00-21swapper/019:06:170
31029991111,2cyclictest0-21swapper/322:14:453
3102999107104,2cyclictest0-21swapper/322:22:393
3102999105101,3cyclictest0-21swapper/322:17:593
310299910097,2cyclictest0-21swapper/323:14:343
310299910091,3cyclictest0-21swapper/323:26:443
31029999591,3cyclictest0-21swapper/300:34:083
31029996056,3cyclictest0-21swapper/321:35:403
31029995956,2cyclictest0-21swapper/323:18:383
31029995551,3cyclictest0-21swapper/320:32:593
117852550,3sleep111789-21gltestperf20:50:161
3101899401,18cyclictest6950-21irqstats19:25:181
31011994010,14cyclictest12512-21munin-run19:40:020
3101899392,3cyclictest8727-21munin-run20:45:021
3101899380,17cyclictest12047-21cat22:05:121
31011993810,13cyclictest1809-21munin-run00:10:010
3102399371,33cyclictest0-21swapper/223:29:062
3101899361,15cyclictest32572-21/usr/sbin/munin22:50:181
3102399331,23cyclictest0-21swapper/223:16:442
3101899331,18cyclictest32003-21ntp_states21:35:191
3102399321,22cyclictest0-21swapper/221:32:432
3102399321,22cyclictest0-21swapper/221:32:422
3102399321,20cyclictest0-21swapper/221:19:432
3102399321,19cyclictest0-21swapper/222:37:142
3102399321,19cyclictest0-21swapper/221:47:142
3102399321,19cyclictest0-21swapper/200:06:352
3102399321,12cyclictest0-21swapper/223:50:462
3101199329,11cyclictest7-21ksoftirqd/022:05:020
3102399311,13cyclictest0-21swapper/223:58:582
3102399311,12cyclictest0-21swapper/223:49:062
3102399311,12cyclictest0-21swapper/222:20:182
3101199319,10cyclictest26318-21idleruntime-cro21:25:000
31029993027,2cyclictest0-21swapper/323:39:443
3102399301,18cyclictest0-21swapper/220:39:392
3102399301,17cyclictest0-21swapper/220:46:132
3102399301,16cyclictest24669-21df21:20:132
3102399301,10cyclictest31968-21fschecks_count22:50:132
3102399300,12cyclictest0-21swapper/220:15:262
41862290,1sleep30-21swapper/319:20:133
3102399291,19cyclictest0-21swapper/219:31:542
3102399291,14cyclictest0-21swapper/221:14:512
3102399291,13cyclictest0-21swapper/222:59:112
3102399291,13cyclictest0-21swapper/200:10:432
3102399291,12cyclictest0-21swapper/223:40:382
3102399291,12cyclictest0-21swapper/222:11:302
3102399291,11cyclictest0-21swapper/220:00:402
3102399290,19cyclictest0-21swapper/221:05:152
3102399281,19cyclictest0-21swapper/222:19:582
3102399281,13cyclictest0-21swapper/221:28:112
3102399281,12cyclictest0-21swapper/222:43:232
3102399281,11cyclictest0-21swapper/223:01:392
3102399281,11cyclictest0-21swapper/219:54:522
3102399281,11cyclictest0-21swapper/219:42:322
3102399281,10cyclictest0-21swapper/223:31:222
3102399281,10cyclictest0-21swapper/221:51:382
3102399281,10cyclictest0-21swapper/220:13:242
3102399281,10cyclictest0-21swapper/219:39:182
3102399281,10cyclictest0-21swapper/200:02:422
31029992723,3cyclictest0-21swapper/321:33:463
31029992723,3cyclictest0-21swapper/321:33:463
3102399271,13cyclictest0-21swapper/200:16:392
3102399271,12cyclictest0-21swapper/221:40:342
3102399271,11cyclictest0-21swapper/223:10:152
3102399271,11cyclictest0-21swapper/220:26:162
3102399271,11cyclictest0-21swapper/219:18:212
3102399271,10cyclictest0-21swapper/223:06:292
3102399271,10cyclictest0-21swapper/222:03:502
3102399271,10cyclictest0-21swapper/221:01:492
3102399270,10cyclictest0-21swapper/220:05:182
31018992715,3cyclictest1172-21sort21:40:141
3101199278,16cyclictest3998-21munin-run00:15:010
3102399261,15cyclictest0-21swapper/223:20:022
3102399261,12cyclictest0-21swapper/220:55:392
3102399261,12cyclictest0-21swapper/220:31:382
3102399261,11cyclictest0-21swapper/219:23:002
3102399261,10cyclictest0-21swapper/219:45:362
3102399260,9cyclictest0-21swapper/222:49:252
3102399260,9cyclictest0-21swapper/222:49:252
3102399260,24cyclictest0-21swapper/221:35:162
3101899261,18cyclictest22515-21sadc22:30:011
3101899261,12cyclictest0-21swapper/100:00:131
3102399258,7cyclictest0-21swapper/200:40:022
3102399251,16cyclictest0-21swapper/219:57:562
3102399251,14cyclictest0-21swapper/221:58:542
3102399251,13cyclictest0-21swapper/200:22:152
3102399251,12cyclictest0-21swapper/223:36:182
3102399251,12cyclictest0-21swapper/222:31:022
3102399251,12cyclictest0-21swapper/219:14:392
3102399251,11cyclictest0-21swapper/222:28:582
3102399251,11cyclictest0-21swapper/220:22:482
3102399251,11cyclictest0-21swapper/220:22:482
3102399251,11cyclictest0-21swapper/200:32:232
3102399251,11cyclictest0-21swapper/200:25:352
3102399251,10cyclictest0-21swapper/220:43:492
3102399250,12cyclictest0-21swapper/222:07:082
3101899259,3cyclictest0-21swapper/121:10:001
3101899251,9cyclictest5011-21ntp_states19:20:201
3101899251,12cyclictest0-21swapper/119:56:361
3101899250,13cyclictest0-21swapper/121:04:131
3101199251,12cyclictest6541-21awk20:40:010
259002250,4sleep011-21rcuc/020:10:020
3101899241,3cyclictest10620-21ntp_states22:00:201
3101899241,15cyclictest29344-21head21:30:151
3101899241,15cyclictest29344-21head21:30:151
3101899241,14cyclictest0-21swapper/100:20:171
31018992410,3cyclictest28115-21/usr/sbin/munin20:15:211
3101899240,16cyclictest0-21swapper/119:18:561
3101899240,10cyclictest0-21swapper/122:10:411
3102999231,12cyclictest27760-21/usr/sbin/munin23:55:113
3102399231,14cyclictest0-21swapper/220:50:352
3102399231,10cyclictest0-21swapper/219:29:382
3101899238,3cyclictest0-21swapper/120:48:131
3101899238,3cyclictest0-21swapper/119:37:521
3101899232,18cyclictest0-21swapper/120:30:491
3101899231,9cyclictest0-21swapper/123:21:521
3101899231,3cyclictest17030-21latency22:15:201
3101899231,3cyclictest1174-21ls00:05:261
3101899231,16cyclictest0-21swapper/122:45:271
3101899231,16cyclictest0-21swapper/122:45:271
3101899231,10cyclictest0-21swapper/123:42:281
3101899230,18cyclictest0-21swapper/100:30:461
3101199235,9cyclictest0-21swapper/021:55:370
3101199232,12cyclictest0-21swapper/022:10:130
304722230,5sleep33102999cyclictest22:45:233
304722230,5sleep33102999cyclictest22:45:233
3102999228,4cyclictest86550irq/126-enp1s0-22:28:403
3101899226,3cyclictest0-21swapper/120:26:571
3101899222,8cyclictest0-21swapper/123:55:211
31018992214,3cyclictest0-21swapper/123:06:591
3101899221,15cyclictest5616-21ntp_states20:35:211
3101899221,14cyclictest0-21swapper/122:20:171
3101899221,14cyclictest0-21swapper/100:12:091
3101899221,13cyclictest0-21swapper/123:01:031
3101899221,11cyclictest0-21swapper/121:58:481
3101899221,11cyclictest0-21swapper/121:19:281
3101899221,10cyclictest0-21swapper/123:49:081
3101199221,9cyclictest15536-21/usr/sbin/munin21:00:160
3102999212,10cyclictest3108750kworker/u8:023:09:363
3102999211,11cyclictest24777-21ntp_states20:05:193
3102999211,11cyclictest21170-21/usr/sbin/munin23:40:133
3102999211,11cyclictest19941-21/usr/sbin/munin21:10:193
3101899218,3cyclictest0-21swapper/121:10:081
3101899215,3cyclictest0-21swapper/123:16:241
3101899214,8cyclictest86850irq/129-enp1s0-20:11:151
3101899214,4cyclictest0-21swapper/123:14:541
3101899212,8cyclictest0-21swapper/123:35:431
3101899212,3cyclictest0-21swapper/119:12:591
3101899211,3cyclictest3365-21perf23:00:011
3101899211,3cyclictest0-21swapper/100:18:061
31018992112,3cyclictest0-21swapper/119:45:241
3101899211,15cyclictest0-21swapper/122:44:461
3101899211,15cyclictest0-21swapper/122:37:461
3101899211,15cyclictest0-21swapper/120:07:061
3101899211,15cyclictest0-21swapper/119:40:441
3101899211,15cyclictest0-21swapper/100:37:281
3101899211,12cyclictest0-21swapper/119:31:241
3101899211,11cyclictest16543-21latency_hist23:30:021
3101899211,11cyclictest15506-21cat21:00:021
3101899211,11cyclictest0-21swapper/121:29:401
3101899211,10cyclictest0-21swapper/123:31:001
3101899210,14cyclictest0-21swapper/121:21:201
3101199213,9cyclictest2834850kworker/u8:121:31:170
3101199213,9cyclictest2834850kworker/u8:121:31:170
3101199211,9cyclictest28093-21df23:55:140
3101199211,11cyclictest5293-21ntp_states00:15:230
3101199211,11cyclictest16160-21users23:25:260
3101199211,11cyclictest0-21swapper/023:13:130
3101199211,11cyclictest0-21swapper/023:06:030
3101199210,14cyclictest0-21swapper/022:31:470
3102999209,3cyclictest30310-21/usr/sbin/munin20:20:193
3102999209,3cyclictest30310-21/usr/sbin/munin20:20:193
3102999201,8cyclictest0-21swapper/322:58:543
3102999201,6cyclictest31087-21kworker/u8:023:20:133
3102999201,11cyclictest6480-21sshd20:39:373
3102999201,11cyclictest27765-21sshd20:13:373
3102999201,11cyclictest11142-21/usr/sbin/munin20:50:243
3102999201,11cyclictest10714-21sshd20:47:373
3102999201,10cyclictest0-21swapper/300:20:113
3101899201,14cyclictest0-21swapper/120:02:051
3101899201,11cyclictest7163-21latency_hist21:55:011
3101899201,11cyclictest22818-21cpuspeed_turbos22:30:131
3101899200,3cyclictest27363-21systemd-journal23:53:001
3101899200,15cyclictest0-21swapper/100:25:111
3101899200,14cyclictest0-21swapper/119:50:251
3101899200,12cyclictest0-21swapper/121:47:481
3101199208,3cyclictest29963-21/usr/sbin/munin00:00:250
3101199208,3cyclictest2969-21/usr/sbin/munin21:45:210
31011992015,3cyclictest30808-21grep22:45:260
31011992015,3cyclictest30808-21grep22:45:260
3101199201,11cyclictest8339-21/usr/sbin/munin19:30:130
3101199201,11cyclictest0-21swapper/023:42:220
3101199201,11cyclictest0-21swapper/023:17:130
3101199201,10cyclictest5168-21/usr/sbin/munin21:50:130
3101199201,10cyclictest1363-21/usr/sbin/munin22:55:130
3101199201,10cyclictest0-21swapper/022:20:030
320472190,4sleep339-21rcuc/300:05:013
3102999198,3cyclictest1746-21/usr/sbin/munin19:15:153
3102999191,7cyclictest24278-21chrt23:45:183
3102999191,6cyclictest4964-21cat00:15:183
3102999191,5cyclictest0-21swapper/321:57:483
3102999191,5cyclictest0-21swapper/321:45:383
3102999191,10cyclictest28115-21/usr/sbin/munin20:15:163
3102999191,10cyclictest17837-21ntp_states23:30:213
31018991915,2cyclictest0-21swapper/120:21:071
31018991915,2cyclictest0-21swapper/120:21:071
3101199197,3cyclictest7990-21sshd19:28:370
3101199196,4cyclictest19588-21expr19:55:110
3101199195,5cyclictest0-21swapper/021:28:500
3101199194,4cyclictest0-21swapper/022:19:070
3101199194,2cyclictest86550irq/126-enp1s0-23:33:120
3101199191,11cyclictest0-21swapper/000:39:510
3101199191,10cyclictest19711-21diskmemload21:15:240
3101199190,9cyclictest32549-21kworker/u8:221:44:010
3101199190,6cyclictest0-21swapper/022:44:190
112292191,2sleep30-21swapper/322:01:143
3102999186,3cyclictest29134-21awk22:45:003
3102999186,3cyclictest26487-21kworker/3:220:00:143
3102999182,12cyclictest14926-21/usr/sbin/munin19:45:223
3102999182,12cyclictest0-21swapper/319:27:403
3102999181,8cyclictest26487-21kworker/3:221:19:433
3102999181,8cyclictest0-21swapper/322:05:133
3102999181,7cyclictest5597-21fschecks_count21:50:163
3102999181,7cyclictest0-21swapper/322:30:203
3102999181,5cyclictest8339-21/usr/sbin/munin19:30:213
3102999181,3cyclictest14006-21taskset19:40:263
31029991812,3cyclictest6750-21/usr/sbin/munin20:40:253
3102999181,10cyclictest786-21NetworkManager22:54:183
3101199187,3cyclictest17123-21/usr/sbin/munin19:50:200
3101199185,4cyclictest4569-21strings23:00:170
3101199181,9cyclictest0-21swapper/023:20:240
3101199181,7cyclictest24947-21/usr/sbin/munin22:35:170
3101199181,7cyclictest20592-21gltestperf21:10:150
3101199181,6cyclictest0-21swapper/000:32:390
3101199181,3cyclictest0-21swapper/023:51:120
31011991811,4cyclictest32192-21gltestperf22:50:130
31011991811,4cyclictest0-21swapper/022:06:430
263172181,2sleep30-21swapper/321:25:023
3102999175,4cyclictest19711-21diskmemload21:41:093
3102999171,9cyclictest0-21swapper/321:20:153
31029991712,3cyclictest511-21if_err_enp2s000:05:183
31029991712,3cyclictest15536-21/usr/sbin/munin21:00:263
3102999171,14cyclictest0-21swapper/319:10:063
3102999171,14cyclictest0-21swapper/300:10:143
3102999171,13cyclictest3580-21/usr/sbin/munin23:00:233
31029991711,3cyclictest25316-21fschecks_count22:35:143
31029991711,3cyclictest19315-21/usr/sbin/munin19:55:153
31029991710,4cyclictest776-21dbus-daemon21:00:003
3101199176,3cyclictest17742-21/usr/sbin/munin21:05:160
3101199173,3cyclictest0-21swapper/000:23:200
3101199172,3cyclictest24427-21ntp_states23:45:210
3101199171,9cyclictest0-21swapper/021:36:160
3101199171,9cyclictest0-21swapper/000:25:280
3101199171,7cyclictest2359-21/usr/sbin/munin20:30:230
3101199171,3cyclictest3941-21/usr/sbin/munin19:20:240
3101199171,3cyclictest0-21swapper/022:25:250
31011991711,3cyclictest15514-21date21:00:010
31011991711,3cyclictest13274-21cut20:55:010
31011991711,3cyclictest1166-21df19:15:020
3102999161,6cyclictest18649-21grep19:50:243
3102999161,6cyclictest0-21swapper/319:35:143
3102999161,5cyclictest0-21swapper/323:50:193
31029991611,3cyclictest300-21cat20:25:113
31029991611,3cyclictest18079-21df21:05:143
31029991611,3cyclictest13008-21/usr/sbin/munin00:35:163
3102999161,10cyclictest0-21swapper/300:25:143
3101199169,4cyclictest0-21swapper/023:35:040
3101199168,5cyclictest851-21sssd_nss19:40:250
3101199165,3cyclictest31212-21/usr/sbin/munin19:10:110
3101199162,11cyclictest26289-21sort20:10:160
31011991611,3cyclictest30444-21tr20:20:130
31011991611,3cyclictest30444-21tr20:20:120
31011991611,3cyclictest28115-21/usr/sbin/munin20:15:220
31011991611,2cyclictest9860-21missed_timers20:45:180
3101199161,12cyclictest32512-21/usr/sbin/munin20:25:130
31011991610,3cyclictest6750-21/usr/sbin/munin20:40:210
31011991510,3cyclictest15492-21cut19:45:160
3101199150,9cyclictest0-21swapper/020:00:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional