You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-16 - 05:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 16, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2415023070,5sleep339-21rcuc/319:05:143
256802226155,59sleep00-21swapper/019:07:440
258182203164,26sleep20-21swapper/219:09:322
257782202166,24sleep10-21swapper/119:09:021
617521000,2sleep30-21swapper/320:50:123
25961999693,2cyclictest0-21swapper/322:23:343
25961997268,3cyclictest0-21swapper/323:06:313
25961997168,2cyclictest0-21swapper/320:06:113
25961996866,1cyclictest0-21swapper/320:20:273
141792640,3sleep214183-21unixbench_multi21:05:242
25961995855,2cyclictest0-21swapper/321:57:313
97552570,1sleep00-21swapper/023:25:120
231622570,5sleep121-21rcuc/123:55:151
298272440,1sleep30-21swapper/300:10:153
25961994138,2cyclictest0-21swapper/320:44:543
25961993733,3cyclictest0-21swapper/323:10:563
2594499346,13cyclictest12455-21munin-run21:05:010
2594499336,12cyclictest5867-21awk20:50:010
2594499335,13cyclictest31007-21munin-run19:20:010
2594999312,19cyclictest9-21rcu_preempt22:31:201
2596199297,8cyclictest771-21rngd22:10:153
2594499295,11cyclictest29439-21idleruntime-cro20:30:000
2594499291,13cyclictest9934-21munin-run00:40:010
2594499279,15cyclictest28113-21munin-run19:15:010
2594999261,19cyclictest0-21swapper/122:44:001
2594999261,17cyclictest13489-21irqstats21:05:181
2594499269,14cyclictest3358-21perf00:25:010
2594499263,11cyclictest9312-21awk23:25:010
297822251,2sleep029908-21rt-features22:55:240
2596199259,3cyclictest28053-21sshd19:14:253
2596199251,10cyclictest24805-21sshd20:17:253
2594999257,10cyclictest23-21ksoftirqd/121:50:011
2594999251,16cyclictest19274-21/usr/sbin/munin21:20:171
2594499251,12cyclictest14668-21awk21:10:010
221512250,8sleep2311ktimersoftd/222:40:112
2596199249,3cyclictest27936-21irqstats00:05:173
2596199249,3cyclictest27451-21/usr/sbin/munin20:25:153
2596199249,3cyclictest1352-21/usr/sbin/munin00:20:173
25961992421,2cyclictest0-21swapper/323:29:113
2596199241,5cyclictest0-21swapper/321:44:223
2596199241,13cyclictest0-21swapper/323:30:243
2596199241,10cyclictest23023-21head21:25:253
2596199241,10cyclictest18039-21users20:00:283
2596199240,12cyclictest0-21swapper/321:15:163
2594999241,7cyclictest0-21swapper/123:08:271
2594999241,17cyclictest0-21swapper/123:20:111
2594999240,17cyclictest0-21swapper/100:01:071
2594499240,5cyclictest0-21swapper/022:46:360
2596199237,6cyclictest0-21swapper/322:00:503
2596199233,3cyclictest86850irq/129-enp1s0-21:53:193
2596199231,7cyclictest0-21swapper/300:39:443
2596199231,11cyclictest812-21snmpd20:30:523
2596199231,11cyclictest11319-21/usr/sbin/munin19:45:253
2595599237,5cyclictest407-1kworker/u9:021:43:512
2594999235,9cyclictest851-21sssd_nss00:20:001
2594999232,18cyclictest86550irq/126-enp1s0-23:18:311
2594999232,18cyclictest86550irq/126-enp1s0-23:18:311
2594999232,16cyclictest0-21swapper/100:22:411
2594999231,9cyclictest0-21swapper/122:52:171
2594999231,7cyclictest23-21ksoftirqd/121:50:121
2594999231,19cyclictest31309-21munin-run00:15:001
2594499239,8cyclictest0-21swapper/022:05:240
2594499239,8cyclictest0-21swapper/022:05:240
2594499233,5cyclictest0-21swapper/000:13:300
2594499230,5cyclictest0-21swapper/021:46:410
2596199226,5cyclictest26732-21cstates22:50:133
2596199222,2cyclictest86850irq/129-enp1s0-22:42:323
2596199220,10cyclictest0-21swapper/323:37:143
2595599221,10cyclictest30932expr23:10:122
2594999229,3cyclictest10878-21munin-run22:15:011
2594999221,8cyclictest0-21swapper/100:31:581
2594999221,8cyclictest0-21swapper/100:31:581
25949992214,3cyclictest0-21swapper/120:45:021
2594999221,18cyclictest4298-21/usr/sbin/munin19:30:201
2594999221,16cyclictest0-21swapper/122:21:141
2594999221,16cyclictest0-21swapper/120:11:261
2594999221,16cyclictest0-21swapper/119:26:031
2594999220,6cyclictest0-21swapper/100:36:261
2594499223,16cyclictest29112-21munin-run00:10:000
2594499222,12cyclictest0-21swapper/021:55:140
2594499221,6cyclictest0-21swapper/022:28:020
2594499221,5cyclictest0-21swapper/021:32:490
2594499221,12cyclictest9858-21/usr/sbin/munin19:45:110
2594499220,4cyclictest0-21swapper/021:41:250
82182210,5sleep22595599cyclictest00:35:132
2596199212,4cyclictest0-21swapper/319:35:303
2596199211,7cyclictest0-21swapper/323:19:293
2596199211,7cyclictest0-21swapper/323:19:293
2596199211,6cyclictest21578-21ntp_states23:50:203
2596199211,6cyclictest0-21swapper/322:45:133
2596199211,5cyclictest9288-21sshd19:40:433
2596199211,5cyclictest9288-21sshd19:40:423
2595599217,6cyclictest771-21rngd21:27:262
2595599211,9cyclictest11096-21taskset22:15:082
2595599211,7cyclictest0-21swapper/221:16:062
2595599211,12cyclictest0-21swapper/222:51:592
2594999212,17cyclictest0-21swapper/119:14:231
2594999211,6cyclictest0-21swapper/123:48:261
25949992114,4cyclictest31845-21/usr/sbin/munin20:35:191
25949992114,3cyclictest0-21swapper/121:00:091
2594999211,18cyclictest0-21swapper/123:25:061
2594999211,18cyclictest0-21swapper/121:37:201
2594999211,18cyclictest0-21swapper/121:15:131
2594999211,15cyclictest0-21swapper/122:48:091
2594999211,15cyclictest0-21swapper/122:35:381
2594999211,15cyclictest0-21swapper/122:05:081
2594999211,15cyclictest0-21swapper/122:05:081
2594999211,15cyclictest0-21swapper/120:44:081
2594999211,15cyclictest0-21swapper/120:26:451
2594999211,15cyclictest0-21swapper/120:20:361
2594999211,15cyclictest0-21swapper/119:57:431
2594999211,15cyclictest0-21swapper/119:23:431
2594999211,11cyclictest5888-21fschecks_count19:35:151
2594999210,16cyclictest0-21swapper/123:03:131
2594999210,15cyclictest0-21swapper/123:35:231
2594999210,15cyclictest0-21swapper/100:06:541
2594499216,6cyclictest0-21swapper/000:28:420
2594499214,2cyclictest0-21swapper/022:02:340
2594499212,4cyclictest0-21swapper/023:14:290
2594499211,8cyclictest61762cpuspeed_turbos20:50:130
2594499211,4cyclictest0-21swapper/022:19:530
2594499211,13cyclictest0-21swapper/022:43:560
2594499211,12cyclictest12055-21/usr/sbin/munin19:50:230
2594499211,10cyclictest1455-21perf20:40:010
2594499210,5cyclictest0-21swapper/023:17:530
2594499210,5cyclictest0-21swapper/023:17:530
2596199201,9cyclictest0-21swapper/322:56:243
2596199201,5cyclictest25257-21diskstats00:00:153
2596199200,6cyclictest0-21swapper/323:20:153
2595599207,5cyclictest407-1kworker/u9:021:53:112
2595599203,2cyclictest86850irq/129-enp1s0-23:35:012
2595599201,9cyclictest580-21systemd-journal20:05:012
2595599201,7cyclictest851-21sssd_nss00:30:182
2595599201,7cyclictest851-21sssd_nss00:30:182
2595599201,6cyclictest19161-21unixbench-2d22:30:262
2595599201,5cyclictest31830-21gltestperf19:20:142
2595599201,5cyclictest0-21swapper/200:18:282
25955992010,3cyclictest833-21runrttasks23:15:332
25955992010,3cyclictest833-21runrttasks23:15:332
2594999208,3cyclictest12169-21grep19:50:121
2594999204,4cyclictest0-21swapper/122:25:021
2594999201,6cyclictest0-21swapper/123:33:351
2594999201,5cyclictest21-21rcuc/123:50:231
25949992013,2cyclictest0-21swapper/122:00:201
2594999201,2cyclictest0-21swapper/120:15:561
2594999201,15cyclictest0-21swapper/123:13:571
2594999201,15cyclictest0-21swapper/122:58:341
2594999201,11cyclictest3992-21grep00:25:141
2594999201,11cyclictest22731-21smart_sda21:25:231
2594999201,11cyclictest17515-21ntp_states20:00:221
2594999201,11cyclictest17378-21smart_sda23:40:241
2594999201,10cyclictest6417-21df20:50:131
2594999201,10cyclictest3411-21pmu-power21:55:231
2594999200,15cyclictest0-21swapper/120:58:111
2594999200,15cyclictest0-21swapper/120:58:101
2594999200,15cyclictest0-21swapper/120:09:021
2594499209,3cyclictest13300-21/usr/sbin/munin22:20:160
2594499208,4cyclictest0-21swapper/000:02:590
2594499207,3cyclictest0-21swapper/023:37:370
2594499206,3cyclictest202632grep21:20:180
2594499205,4cyclictest0-21swapper/022:54:470
2594499202,5cyclictest407-1kworker/u9:023:05:550
2594499201,5cyclictest407-1kworker/u9:023:57:020
2594499201,11cyclictest0-21swapper/023:45:450
2594499201,11cyclictest0-21swapper/023:02:120
2594499201,11cyclictest0-21swapper/021:29:110
2594499201,11cyclictest0-21swapper/000:17:140
2594499201,10cyclictest0-21swapper/023:31:460
2596199191,6cyclictest0-21swapper/323:40:043
2596199191,6cyclictest0-21swapper/322:36:293
25961991916,2cyclictest0-21swapper/321:13:483
2596199191,5cyclictest0-21swapper/321:48:513
2596199191,4cyclictest0-21swapper/323:55:493
2596199191,3cyclictest8679-21turbostat22:10:023
2596199191,3cyclictest8679-21turbostat22:10:023
25961991913,3cyclictest4481-21memory00:25:193
25961991912,4cyclictest4524-21tail20:45:163
2596199191,11cyclictest0-21swapper/300:16:493
2596199191,10cyclictest836-21lldpd22:25:433
2595599198,3cyclictest833-21runrttasks23:43:512
2595599191,9cyclictest12929-21latency19:50:192
2595599191,9cyclictest0-21swapper/223:35:262
2595599191,8cyclictest0-21swapper/223:23:192
2595599191,8cyclictest0-21swapper/222:39:522
2595599191,6cyclictest8393-21kworker/u8:122:10:132
2595599191,6cyclictest31779-21grep21:45:282
2595599191,6cyclictest1549-21cstates00:20:132
2595599191,6cyclictest0-21swapper/223:00:132
2595599191,5cyclictest21493-21needreboot23:50:192
2595599191,5cyclictest15896-21fschecks_count22:25:142
2595599191,5cyclictest0-21swapper/222:58:132
25955991911,6cyclictest4168-21kworker/u8:222:07:552
25955991911,6cyclictest4168-21kworker/u8:222:07:552
2595599191,10cyclictest0-21swapper/222:46:422
2595599191,10cyclictest0-21swapper/221:23:042
2595599191,10cyclictest0-21swapper/200:02:272
2595599190,16cyclictest4279-21perf22:00:012
2594999199,2cyclictest7681-21cpu19:40:111
2594999199,2cyclictest7681-21cpu19:40:111
2594999191,8cyclictest0-21swapper/121:30:101
25949991914,1cyclictest0-21swapper/119:15:211
2594999191,13cyclictest8393-21kworker/u8:122:17:531
2594999191,10cyclictest30822-21ps20:30:231
2594499195,4cyclictest2063-21cut19:25:190
2594499191,9cyclictest0-21swapper/021:35:210
25944991912,4cyclictest25589-21df_inode20:20:140
2594499191,10cyclictest812-21snmpd21:15:570
2594499191,10cyclictest3272-21/usr/sbin/munin19:30:250
2594499191,10cyclictest16528-21sshd21:11:020
2594499191,10cyclictest0-21swapper/023:43:160
2594499191,10cyclictest0-21swapper/022:31:130
2594499191,10cyclictest0-21swapper/021:51:390
2594499190,5cyclictest407-1kworker/u9:023:52:220
156582190,2sleep10-21swapper/121:10:181
93542181,3sleep20-21swapper/219:41:352
93542181,3sleep20-21swapper/219:41:352
2596199181,9cyclictest31254-21cpu19:20:103
2596199181,7cyclictest0-21swapper/323:48:353
2596199181,7cyclictest0-21swapper/321:30:173
2596199181,6cyclictest17880-21cat22:30:133
2596199181,4cyclictest6642-21/usr/sbin/munin00:30:193
2596199181,4cyclictest6642-21/usr/sbin/munin00:30:193
2596199181,4cyclictest0-21swapper/321:35:243
2596199181,2cyclictest32449-21head20:35:153
25961991811,4cyclictest31537-21grep23:00:153
2596199180,10cyclictest0-21swapper/320:55:103
2596199180,10cyclictest0-21swapper/320:55:103
2595599185,5cyclictest407-1kworker/u9:023:28:562
2595599182,4cyclictest24220-21ps20:15:232
2595599181,9cyclictest22947-21ls23:55:132
2595599181,9cyclictest0-21swapper/222:20:132
2595599181,9cyclictest0-21swapper/221:11:412
2595599181,6cyclictest0-21swapper/223:45:252
2595599181,6cyclictest0-21swapper/221:38:402
2595599181,4cyclictest0-21swapper/222:00:172
25955991811,4cyclictest19727-21ntp_states20:05:202
2594999182,9cyclictest27807-21kworker/u8:021:40:261
2594999181,3cyclictest0-21swapper/119:46:561
2594499187,8cyclictest8194-21fschecks_time19:40:140
2594499187,8cyclictest8194-21fschecks_time19:40:140
2594499186,4cyclictest0-21swapper/022:10:180
2594499181,4cyclictest20237-21diskstats22:35:120
25944991812,3cyclictest20859-21/usr/sbin/munin20:10:190
2594499181,10cyclictest0-21swapper/000:30:150
2594499181,10cyclictest0-21swapper/000:30:150
2596199174,11cyclictest41-21ksoftirqd/319:15:153
25961991713,3cyclictest0-21swapper/321:00:213
2596199171,2cyclictest5827-21kworker/3:219:25:253
2596199171,2cyclictest0-21swapper/320:10:143
25961991712,3cyclictest14249-21/usr/sbin/munin19:55:173
2596199171,15cyclictest0-21swapper/321:22:183
2596199171,13cyclictest12678-21/usr/sbin/munin21:05:203
2596199171,13cyclictest12329-21cstates19:50:133
2595599177,3cyclictest86850irq/129-enp1s0-23:05:162
2595599172,3cyclictest27124-21/usr/sbin/munin00:05:122
2595599171,7cyclictest6995-21awk19:35:262
2595599171,5cyclictest0-21swapper/221:30:162
25955991712,3cyclictest30462-21irqstats20:30:182
25955991712,3cyclictest29350-21cpu00:10:122
25955991712,3cyclictest21041-21cstates20:10:132
25955991712,3cyclictest1863-21if_err_enp2s019:25:182
2595599171,14cyclictest0-21swapper/220:22:112
25955991711,3cyclictest3352-21cat19:30:132
2595599171,10cyclictest0-21swapper/219:17:222
2595599171,10cyclictest0-21swapper/200:25:072
2594499177,3cyclictest3239-21grep20:40:250
2594499176,3cyclictest18660-21/usr/sbin/munin20:05:150
2594499171,9cyclictest0-21swapper/020:59:460
2594499171,9cyclictest0-21swapper/020:59:460
2594499171,7cyclictest5652-21cstates19:35:120
2594499171,7cyclictest16452-21/usr/sbin/munin20:00:200
2596199161,6cyclictest0-21swapper/322:15:173
25961991611,3cyclictest4862-21vmstat19:30:273
2595599161,9cyclictest0-21swapper/220:39:332
2595599161,5cyclictest11604-21awk21:00:192
2595599161,5cyclictest0-21swapper/220:25:232
2595599161,14cyclictest0-21swapper/220:43:202
25955991611,3cyclictest8273-21/usr/sbin/munin20:55:212
25955991611,3cyclictest8273-21/usr/sbin/munin20:55:212
25955991611,3cyclictest6145-21cpuspeed_turbos20:50:122
2595599161,12cyclictest10040-21cstates19:45:132
2595599160,15cyclictest28162-21kworker/2:219:55:232
2594499166,3cyclictest812-21snmpd19:55:450
2594499165,3cyclictest86750irq/128-enp1s0-20:15:210
2594499161,9cyclictest0-21swapper/020:30:190
2594499161,7cyclictest31230-21/usr/sbin/munin19:20:140
2595599151,2cyclictest0-21swapper/220:45:142
25955991510,3cyclictest27214-21ntp_states19:10:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional