You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-04 - 12:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Jul 04, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
288272239164,24sleep20-21swapper/219:07:162
287032202166,24sleep30-21swapper/319:05:403
287212201163,26sleep00-21swapper/019:05:520
287322200164,24sleep10-21swapper/119:06:011
306621770,4sleep02912999cyclictest23:00:230
68322570,3sleep36837-21munin-plugin-st20:45:013
16032550,3sleep30-21swapper/323:00:013
31222500,1sleep30-21swapper/319:20:213
283932480,2sleep09-21rcu_preempt21:30:250
61582460,1sleep10-21swapper/119:29:221
2912999462,14cyclictest14669-21munin-run23:30:010
98032430,19sleep39804-21sed23:15:263
98032430,19sleep39804-21sed23:15:253
29148994029,3cyclictest0-21swapper/319:41:533
2912999402,13cyclictest3060-21munin-run21:50:000
2914899380,29cyclictest0-21swapper/322:03:223
29129993510,12cyclictest22845-21munin-run22:35:020
29148993424,3cyclictest0-21swapper/320:10:143
2914899341,25cyclictest0-21swapper/323:46:333
2914899341,21cyclictest0-21swapper/323:56:533
2914899331,25cyclictest0-21swapper/322:18:223
2914899331,25cyclictest0-21swapper/322:07:543
2914899331,25cyclictest0-21swapper/322:07:543
2914899331,20cyclictest0-21swapper/321:37:253
2914899331,20cyclictest0-21swapper/300:17:423
2914899321,20cyclictest0-21swapper/321:52:383
2914899321,15cyclictest0-21swapper/300:24:503
2913699321,23cyclictest0-21swapper/121:36:461
2912999322,11cyclictest17136-21kworker/0:019:55:150
29148993117,7cyclictest0-21swapper/323:14:183
2914899311,21cyclictest0-21swapper/323:26:413
2914899311,21cyclictest0-21swapper/320:27:343
2914899311,21cyclictest0-21swapper/320:18:543
2914899311,18cyclictest17842-21diskmemload22:41:343
2914899310,16cyclictest0-21swapper/322:11:223
2912999311,12cyclictest31979-21kworker/0:300:20:590
2914899301,19cyclictest0-21swapper/320:39:583
2914899301,19cyclictest0-21swapper/320:04:183
2914899301,13cyclictest0-21swapper/300:33:463
2914899300,20cyclictest0-21swapper/322:29:023
2912999305,12cyclictest26011-21munin-run20:15:020
29148992918,3cyclictest0-21swapper/322:36:023
29148992914,3cyclictest0-21swapper/320:32:063
2914899291,19cyclictest0-21swapper/321:40:133
2914899291,13cyclictest0-21swapper/323:39:093
29148992911,3cyclictest0-21swapper/320:21:103
2914899291,13cyclictest0-21swapper/300:10:263
2914899291,12cyclictest0-21swapper/321:13:133
2914899290,18cyclictest0-21swapper/300:07:303
2914899290,17cyclictest0-21swapper/300:04:263
2914899290,14cyclictest0-21swapper/323:42:493
2914899290,13cyclictest0-21swapper/322:24:383
2912999293,12cyclictest20667-21perf22:30:020
92392280,14sleep232-21ksoftirqd/220:50:012
2914899289,8cyclictest0-21swapper/322:53:503
2914899281,3cyclictest0-21swapper/323:04:263
2914899281,19cyclictest0-21swapper/322:46:303
2914899281,19cyclictest0-21swapper/322:30:303
2914899281,19cyclictest0-21swapper/319:53:133
2914899281,11cyclictest0-21swapper/323:06:583
2914899280,14cyclictest0-21swapper/321:56:263
2914899280,11cyclictest0-21swapper/320:52:063
29136992824,3cyclictest0-21swapper/121:04:591
2913699281,16cyclictest0-21swapper/122:59:501
46422270,4sleep123-21ksoftirqd/121:50:241
29148992716,3cyclictest0-21swapper/321:15:493
29148992713,7cyclictest0-21swapper/323:51:013
2914899271,13cyclictest0-21swapper/321:26:333
2914899271,11cyclictest0-21swapper/323:30:173
2914899271,10cyclictest0-21swapper/320:07:303
29136992723,3cyclictest0-21swapper/120:18:591
2913699271,24cyclictest28834-21awk21:35:021
2912999271,9cyclictest8848-21kworker/0:220:55:120
2912999271,9cyclictest8848-21kworker/0:220:55:120
2912999271,9cyclictest26342-21kworker/0:121:42:310
2912999271,10cyclictest30291-21kworker/0:220:24:000
2912999271,10cyclictest25811-21kworker/0:020:16:240
285952270,3sleep22914199cyclictest20:20:112
2914899261,9cyclictest0-21swapper/319:12:043
2914899261,17cyclictest0-21swapper/320:55:093
2914899261,17cyclictest0-21swapper/320:55:093
2914899261,12cyclictest0-21swapper/319:17:283
2914899261,11cyclictest7828-21sort19:30:243
2914899261,11cyclictest0-21swapper/321:47:423
2914899260,12cyclictest0-21swapper/323:20:093
2914899260,12cyclictest0-21swapper/321:00:053
2913699261,22cyclictest4645-21tr00:20:111
2912999262,13cyclictest31312-21perf19:15:020
2912999262,10cyclictest1873-21kworker/0:100:13:590
2912999261,17cyclictest28039-21kworker/0:120:26:440
2912999261,10cyclictest8848-21kworker/0:221:00:040
2912999261,10cyclictest8848-21kworker/0:220:49:520
2912999261,10cyclictest31979-21kworker/0:300:28:030
2912999261,10cyclictest31979-21kworker/0:300:28:030
2914899258,3cyclictest0-21swapper/319:47:213
2914899258,3cyclictest0-21swapper/319:47:213
29148992512,4cyclictest0-21swapper/300:25:043
29148992512,4cyclictest0-21swapper/300:25:043
2914899251,11cyclictest0-21swapper/321:21:173
2914899251,11cyclictest0-21swapper/300:35:463
2914899250,12cyclictest0-21swapper/320:47:583
2914899250,11cyclictest0-21swapper/319:57:533
29136992521,3cyclictest0-21swapper/122:03:091
29136992521,3cyclictest0-21swapper/100:27:571
29136992521,3cyclictest0-21swapper/100:27:571
2913699251,10cyclictest13418-21turbostat.cron21:00:011
2913699251,10cyclictest13418-21turbostat.cron21:00:011
2912999257,15cyclictest6848-21idleruntime-cro20:45:010
2912999252,9cyclictest24110-21kworker/0:221:27:000
2912999252,9cyclictest18979-21kworker/0:021:15:560
2912999252,10cyclictest29115-21kworker/0:122:59:080
2912999251,9cyclictest28039-21kworker/0:120:30:040
294172241,3sleep229662-21df19:10:132
2914899246,3cyclictest0-21swapper/319:35:453
29148992411,8cyclictest0-21swapper/319:27:533
2914899241,10cyclictest0-21swapper/321:33:573
2913699244,5cyclictest0-21swapper/123:45:151
29136992421,2cyclictest0-21swapper/120:36:471
29136992421,2cyclictest0-21swapper/119:56:441
29136992421,2cyclictest0-21swapper/119:36:371
2913699242,11cyclictest909-21sssd_nss22:20:011
2913699241,21cyclictest7447-21turbostat.cron22:00:011
2913699241,20cyclictest9604-21df_inode20:50:141
2913699241,20cyclictest579-21systemd-journal23:45:021
2913699241,20cyclictest0-21swapper/100:35:211
2913699241,11cyclictest8082-21awk23:15:011
2913699241,11cyclictest3607-21/usr/sbin/munin00:15:251
2913699241,11cyclictest29008-21perl20:20:141
2913699241,11cyclictest24372-21df20:10:121
2913699241,11cyclictest23740-21taskset23:50:121
2913699241,10cyclictest0-21swapper/119:10:231
2912999242,10cyclictest20911-21kworker/0:223:41:360
2912999242,10cyclictest1224-21kworker/0:023:07:080
29129992412,2cyclictest31323-21kworker/0:222:52:120
29129992412,2cyclictest21383-21kworker/0:220:06:480
2912999241,11cyclictest14628-21kworker/0:323:32:400
2912999241,10cyclictest29692-21kworker/0:100:03:190
29129992410,3cyclictest1224-21kworker/0:023:14:160
93502231,2sleep20-21swapper/223:15:212
93502231,2sleep20-21swapper/223:15:212
40122230,5sleep22914199cyclictest20:35:242
2914899239,3cyclictest0-21swapper/321:06:173
2914199231,4cyclictest0-21swapper/221:48:592
2913699235,2cyclictest14708-21cat23:30:021
2913699231,9cyclictest0-21swapper/122:42:351
29136992319,3cyclictest0-21swapper/120:26:521
2913699231,19cyclictest26132-21expr23:55:121
2913699231,11cyclictest8326-21smart_sda20:45:251
2913699231,11cyclictest18623-21cat22:25:011
2913699231,11cyclictest0-21swapper/121:17:551
2913699231,10cyclictest3676-21latency_hist23:05:011
2913699231,10cyclictest31595-21sed00:05:251
2913699231,10cyclictest11556-21ntpq23:20:201
2913699231,10cyclictest0-21swapper/122:35:161
2913699230,21cyclictest0-21swapper/123:05:201
2913699230,16cyclictest0-21swapper/123:38:181
2913699230,15cyclictest0-21swapper/121:23:321
2912999238,3cyclictest9351-21kworker/0:022:20:160
2912999237,4cyclictest10483-21kworker/0:019:46:030
2912999237,4cyclictest10483-21kworker/0:019:46:030
2912999234,4cyclictest9351-21kworker/0:022:04:110
2912999232,10cyclictest8848-21kworker/0:220:53:040
2912999231,11cyclictest12646-21kworker/0:319:44:430
29129992310,2cyclictest9351-21kworker/0:022:07:110
29129992310,2cyclictest9351-21kworker/0:022:07:110
2914199221,8cyclictest6636-21cat00:25:022
2914199221,7cyclictest0-21swapper/222:02:292
2913699228,2cyclictest0-21swapper/120:05:151
2913699222,11cyclictest0-21swapper/120:40:171
2913699221,9cyclictest0-21swapper/122:46:011
2913699221,9cyclictest0-21swapper/121:11:121
2913699221,19cyclictest0-21swapper/119:30:131
2913699221,18cyclictest2260-21/usr/sbin/munin21:45:221
2913699221,10cyclictest25555-21latency21:25:181
2913699221,10cyclictest16886-21wc23:35:021
2913699220,21cyclictest0-21swapper/120:00:181
2913699220,20cyclictest0-21swapper/122:30:221
2912999229,3cyclictest31979-21kworker/0:300:18:390
2912999228,3cyclictest6016-21kworker/0:019:31:150
2912999228,3cyclictest14354-21kworker/0:123:37:000
2912999228,2cyclictest11820-21kworker/0:222:10:160
2912999227,5cyclictest0-21swapper/021:24:470
2912999227,4cyclictest25630-21kworker/0:023:59:510
2912999227,2cyclictest28716-21kworker/0:219:24:560
2912999226,2cyclictest18352-21kworker/0:222:35:360
2912999222,5cyclictest13105-21munin-run00:40:000
2912999222,10cyclictest12117-21kworker/0:223:24:080
2912999221,11cyclictest18720-21expr21:10:180
107872220,3sleep19-21rcu_preempt22:05:161
107872220,3sleep19-21rcu_preempt22:05:161
2914199215,3cyclictest17435-21diskstats23:35:132
2914199213,10cyclictest16475-21/usr/sbin/munin22:20:162
2914199212,6cyclictest19632-21diskstats23:40:142
2914199211,8cyclictest0-21swapper/221:26:302
2914199211,7cyclictest24657-21users22:35:262
2914199211,7cyclictest13044-21users20:55:252
2914199211,7cyclictest13044-21users20:55:252
29141992117,3cyclictest0-21swapper/200:29:412
29141992117,3cyclictest0-21swapper/200:29:412
2913699218,3cyclictest8944-21gltestperf23:15:161
2913699218,3cyclictest8944-21gltestperf23:15:161
2913699217,3cyclictest0-21swapper/119:15:041
2913699216,4cyclictest0-21swapper/100:02:001
2913699212,17cyclictest0-21swapper/120:30:021
2913699211,8cyclictest0-21swapper/122:54:471
29136992117,3cyclictest0-21swapper/119:41:571
29136992117,3cyclictest0-21swapper/100:31:261
2913699211,6cyclictest0-21swapper/122:26:031
2913699211,5cyclictest2932-21kworker/u8:122:11:411
2913699211,3cyclictest16045-21irqstats19:50:161
2913699211,18cyclictest0-21swapper/119:20:451
2913699210,19cyclictest0-21swapper/100:10:161
2912999218,2cyclictest14923-21kworker/0:119:53:030
2912999216,4cyclictest28716-21kworker/0:219:26:430
2912999216,3cyclictest1564-21kworker/0:019:19:200
2912999212,2cyclictest26342-21kworker/0:121:38:110
2912999211,12cyclictest31360-21ntp_states00:05:200
2912999211,11cyclictest23416-21kworker/0:323:50:120
2914199207,3cyclictest87650irq/129-enp1s0-20:30:212
2914199206,3cyclictest87650irq/129-enp1s0-22:55:232
2914199206,3cyclictest23712-21runrttasks20:08:542
2914199204,4cyclictest22615-21sshd23:45:232
2914199203,6cyclictest0-21swapper/221:39:302
2914199202,3cyclictest87650irq/129-enp1s0-23:27:222
2914199201,8cyclictest0-21swapper/222:25:182
2914199201,7cyclictest8994-21diskstats19:35:142
2914199201,7cyclictest0-21swapper/222:09:262
2914199201,7cyclictest0-21swapper/222:09:262
2914199201,7cyclictest0-21swapper/220:00:242
2914199201,6cyclictest0-21swapper/200:32:162
2914199201,6cyclictest0-21swapper/200:10:312
2914199201,5cyclictest0-21swapper/200:05:152
2914199201,10cyclictest22455-21/usr/sbin/munin21:20:222
2914199201,10cyclictest13039-21/usr/sbin/munin19:45:132
2914199201,10cyclictest13039-21/usr/sbin/munin19:45:132
2913699202,17cyclictest0-21swapper/121:09:061
2913699201,10cyclictest0-21swapper/119:45:381
2913699201,10cyclictest0-21swapper/119:45:381
2912999208,4cyclictest3780-21kworker/0:119:35:190
2912999206,3cyclictest20607-21kworker/0:322:45:240
2912999206,2cyclictest12646-21kworker/0:320:00:040
2912999205,3cyclictest31979-21kworker/0:300:30:510
2912999205,2cyclictest9351-21kworker/0:022:16:080
2912999201,11cyclictest3285-21/usr/sbin/munin21:50:240
2912999201,10cyclictest2132-21kworker/0:320:38:520
127552201,2sleep20-21swapper/222:10:172
2914199195,6cyclictest87650irq/129-enp1s0-23:30:112
2914199195,3cyclictest87650irq/129-enp1s0-21:10:152
2914199195,3cyclictest16502-21sh19:50:232
2914199195,3cyclictest12050irq/123-ahci[0019:15:142
2914199194,8cyclictest0-21swapper/223:03:112
2914199193,4cyclictest7447-21grep23:10:242
2914199193,4cyclictest26691-21irqstats23:55:172
2914199191,7cyclictest0-21swapper/222:15:152
2914199191,6cyclictest5418-21ps19:25:222
2914199191,6cyclictest21937-21ntp_states22:30:202
2914199191,6cyclictest17842-21diskmemload21:50:172
2914199191,6cyclictest0-21swapper/220:30:002
2914199191,5cyclictest0-21swapper/221:44:242
2914199191,4cyclictest0-21swapper/200:35:082
2914199191,10cyclictest0-21swapper/220:53:052
2913699195,5cyclictest0-21swapper/121:40:231
2912999196,2cyclictest28755-21kworker/0:321:56:350
2912999195,7cyclictest87350irq/126-enp1s0-23:47:300
2912999194,3cyclictest1224-21kworker/0:023:18:040
2912999194,3cyclictest1224-21kworker/0:023:18:040
2912999192,11cyclictest8848-21kworker/0:221:06:000
2912999190,2cyclictest87650irq/129-enp1s0-22:40:430
2914199185,3cyclictest87650irq/129-enp1s0-22:40:222
2914199185,3cyclictest87650irq/129-enp1s0-19:30:172
2914199184,3cyclictest0-21swapper/221:55:162
2914199181,9cyclictest4862-21/usr/sbin/munin20:40:252
2914199181,7cyclictest0-21swapper/223:09:592
2914199181,6cyclictest16578-21basename21:05:162
2914199181,6cyclictest0-21swapper/223:20:172
2914199181,6cyclictest0-21swapper/222:45:092
2914199181,6cyclictest0-21swapper/219:59:052
2914199181,5cyclictest28063-21sshd20:18:212
2914199181,4cyclictest14880-21grep21:00:232
29141991813,4cyclictest0-21swapper/200:15:172
124352181,2sleep20-21swapper/219:40:282
2914199174,3cyclictest87350irq/126-enp1s0-00:00:172
2914199173,5cyclictest0-21swapper/221:15:332
2914199171,7cyclictest0-21swapper/222:50:472
2914199171,6cyclictest0-21swapper/223:52:102
2914199171,4cyclictest26850-21/usr/sbin/munin21:30:112
29141991710,4cyclictest3285-21sendmail_mailqu19:20:232
29141991610,3cyclictest24405-21fschecks_count20:10:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional