You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 10:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jun 25, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
186152240162,25sleep00-21swapper/019:06:520
187862216163,40sleep10-21swapper/119:09:071
185442202167,23sleep20-21swapper/219:05:582
168822202165,24sleep30-21swapper/319:05:023
37221660,5sleep31895699cyclictest20:50:253
2578521590,4sleep11894599cyclictest20:35:191
978621580,4sleep31895699cyclictest22:25:243
1895199111107,3cyclictest0-21swapper/219:14:292
1895199111107,3cyclictest0-21swapper/219:14:292
299172590,4sleep230-21rcuc/200:25:192
103692580,4sleep11894599cyclictest20:00:191
248032500,3sleep10-21swapper/119:20:141
24072480,3sleep011-21rcuc/020:55:200
1893999389,14cyclictest23029-21idleruntime-cro21:45:010
1893999389,14cyclictest23029-21idleruntime-cro21:45:010
18939993713,11cyclictest30893-21munin-run00:30:010
190832360,1sleep00-21swapper/021:35:130
1893999334,13cyclictest7039-21awk19:55:010
18956993217,8cyclictest0-21swapper/323:00:293
1893999323,14cyclictest30255-21munin-run23:15:010
262942310,1sleep30-21swapper/319:25:013
1895699311,16cyclictest0-21swapper/323:45:173
18945993122,4cyclictest21117-21munin-run19:15:011
18945993122,4cyclictest21117-21munin-run19:15:011
1895699301,12cyclictest0-21swapper/321:14:213
1893999306,11cyclictest1036-21idleruntime-cro20:55:010
1893999304,12cyclictest18030-21perf20:20:010
18956992914,10cyclictest0-21swapper/300:17:533
1893999296,11cyclictest11093-21perf23:45:010
185312290,4sleep11894599cyclictest20:20:131
1894599283,15cyclictest21-21rcuc/122:46:181
1895699279,3cyclictest0-21swapper/322:30:173
18956992713,7cyclictest0-21swapper/320:37:293
1895699271,14cyclictest0-21swapper/300:25:023
1895699271,10cyclictest0-21swapper/323:15:333
1895699271,10cyclictest0-21swapper/319:49:413
1895699270,17cyclictest0-21swapper/321:29:373
1894599271,16cyclictest29429-21interrupts19:30:181
18956992615,3cyclictest0-21swapper/323:27:333
18956992611,3cyclictest22931-21expr00:10:153
1895699261,10cyclictest0-21swapper/300:30:173
18956992610,8cyclictest0-21swapper/322:49:573
272612250,4sleep230-21rcuc/220:40:132
1895699257,4cyclictest0-21swapper/323:50:213
1895699251,12cyclictest0-21swapper/323:37:573
1895699251,12cyclictest0-21swapper/300:02:573
1895699251,11cyclictest0-21swapper/321:40:373
1895699251,11cyclictest0-21swapper/321:40:373
1895699251,11cyclictest0-21swapper/320:11:373
18956992510,3cyclictest0-21swapper/323:58:073
1894599253,8cyclictest13693-21runrttasks22:35:171
1894599253,15cyclictest2169-21kworker/u8:223:41:501
1893999252,10cyclictest2978-21kworker/0:121:05:310
1895699248,3cyclictest0-21swapper/322:50:573
1895699248,3cyclictest0-21swapper/322:50:573
1895699248,3cyclictest0-21swapper/319:40:213
1895699241,6cyclictest0-21swapper/322:35:113
18956992410,3cyclictest0-21swapper/323:12:193
18956992410,3cyclictest0-21swapper/323:07:113
1894599242,19cyclictest29512-21seq21:58:451
18945992415,3cyclictest0-21swapper/123:00:131
1894599241,18cyclictest0-21swapper/122:25:571
1893999242,8cyclictest23585-21kworker/0:023:01:350
1893999240,9cyclictest18374-21kworker/0:221:55:390
1893999240,12cyclictest2304-21awk23:25:010
52672231,2sleep25054-21/usr/sbin/munin19:50:132
311812230,10sleep00-21swapper/022:00:250
1895699239,4cyclictest9108-21/usr/sbin/munin23:40:173
1895699239,3cyclictest0-21swapper/319:50:113
1895699239,1cyclictest0-21swapper/321:30:113
1895699238,3cyclictest0-21swapper/322:23:183
1895699238,3cyclictest0-21swapper/320:29:513
1895699238,3cyclictest0-21swapper/320:29:513
1895699237,2cyclictest0-21swapper/320:01:133
1895699234,2cyclictest0-21swapper/321:38:403
1895699231,11cyclictest0-21swapper/322:09:453
1895699231,10cyclictest0-21swapper/320:41:573
1895699231,10cyclictest0-21swapper/319:34:373
1895199231,6cyclictest0-21swapper/200:15:102
1894599236,15cyclictest28659-21kworker/u8:100:35:071
18945992314,3cyclictest0-21swapper/123:11:061
18945992314,3cyclictest0-21swapper/122:20:171
1894599231,19cyclictest17118-21ntp_states20:15:221
1894599231,17cyclictest0-21swapper/122:10:091
1894599231,16cyclictest0-21swapper/121:38:031
1894599230,18cyclictest0-21swapper/121:29:241
1893999234,16cyclictest6695-21awk23:35:010
1893999231,7cyclictest18814-21kworker/0:219:24:390
1893999231,10cyclictest29020-21awk20:45:010
1893999230,8cyclictest2085-21kworker/0:323:29:210
1895699229,7cyclictest0-21swapper/323:31:453
1895699229,4cyclictest0-21swapper/300:28:053
1895699228,3cyclictest24678-21latency_hist20:35:013
1895699227,3cyclictest0-21swapper/322:57:593
1895699227,3cyclictest0-21swapper/321:50:543
1895699226,3cyclictest0-21swapper/322:00:203
1895699225,3cyclictest0-21swapper/319:16:383
1895699221,6cyclictest0-21swapper/321:57:193
18956992210,2cyclictest0-21swapper/300:08:033
18956992210,2cyclictest0-21swapper/300:08:033
1895199221,3cyclictest0-21swapper/200:08:392
1895199221,3cyclictest0-21swapper/200:08:392
1894599227,4cyclictest22492-21irqstats22:55:181
1894599224,17cyclictest0-21swapper/120:26:301
1894599224,17cyclictest0-21swapper/120:26:301
1894599223,2cyclictest87650irq/129-enp1s0-23:45:261
1894599222,15cyclictest0-21swapper/122:16:531
18945992214,3cyclictest0-21swapper/121:14:011
1894599221,19cyclictest0-21swapper/119:40:251
1894599221,15cyclictest87650irq/129-enp1s0-22:05:191
1894599221,15cyclictest0-21swapper/122:30:041
1894599221,15cyclictest0-21swapper/119:26:201
1894599221,14cyclictest0-21swapper/119:55:241
1894599221,12cyclictest17923-21smart_sda21:30:231
1894599220,16cyclictest0-21swapper/123:08:541
1893999222,8cyclictest12259-21kworker/0:021:25:210
1895699219,3cyclictest0-21swapper/300:35:333
1895699218,8cyclictest0-21swapper/321:07:053
1895699218,3cyclictest0-21swapper/319:36:233
1895699217,3cyclictest0-21swapper/319:11:173
1895699217,3cyclictest0-21swapper/319:11:173
1895699216,8cyclictest0-21swapper/320:47:573
1895699212,7cyclictest87550irq/128-enp1s0-20:59:133
1895699211,6cyclictest2252-40dnf23:24:443
1895699211,10cyclictest0-21swapper/321:45:213
1895199214,9cyclictest0-21swapper/221:44:592
1895199214,9cyclictest0-21swapper/221:44:592
1895199211,8cyclictest10096-21egrep21:15:132
1895199211,6cyclictest579-21systemd-journal21:25:012
1895199211,6cyclictest0-21swapper/223:09:232
1895199211,12cyclictest0-21swapper/223:15:092
1895199211,11cyclictest11172-21cat23:45:012
1895199211,11cyclictest0-21swapper/220:54:522
1894599217,12cyclictest32191-21kworker/u8:023:32:491
1894599211,9cyclictest0-21swapper/121:15:411
18945992113,3cyclictest0-21swapper/119:48:551
1894599211,2cyclictest0-21swapper/123:18:161
1894599211,17cyclictest24102-21taskset21:45:181
1894599211,15cyclictest0-21swapper/121:42:071
1894599211,15cyclictest0-21swapper/121:42:071
1894599211,14cyclictest0-21swapper/123:27:381
1894599211,14cyclictest0-21swapper/120:14:181
1894599211,14cyclictest0-21swapper/119:35:051
1894599211,13cyclictest0-21swapper/119:50:141
1894599211,11cyclictest645-21munin-run00:35:011
1894599211,11cyclictest19502-21egrep22:50:131
1894599211,11cyclictest19502-21egrep22:50:131
1894599211,11cyclictest1018-21latency_hist20:55:021
1893999216,6cyclictest0-21swapper/000:11:490
1893999216,5cyclictest0-21swapper/022:33:080
1893999213,3cyclictest0-21swapper/021:49:430
1893999211,8cyclictest8245-21kworker/0:222:29:210
1893999211,8cyclictest5260-21kworker/0:221:20:290
1893999211,8cyclictest21350-21kworker/0:222:59:040
155222211,2sleep215537-21sh21:25:212
89392201,2sleep00-21swapper/021:10:220
311592200,2sleep00-21swapper/019:35:150
1895699205,3cyclictest87350irq/126-enp1s0-22:40:113
18956992011,3cyclictest0-21swapper/319:29:343
1895699201,11cyclictest10075-21/usr/sbin/munin21:15:163
1895699200,7cyclictest0-21swapper/320:21:333
1895699200,10cyclictest0-21swapper/321:20:333
1895199205,3cyclictest12050irq/123-ahci[0000:25:012
1895199204,8cyclictest2640-21awk19:45:012
1895199203,5cyclictest22901-21cat20:30:132
1895199202,4cyclictest28402-21munin-run19:30:012
1895199202,10cyclictest22109-21perf00:14:562
1895199201,9cyclictest13269-21latency_hist23:50:012
1895199201,5cyclictest17047-21sh22:45:012
1895199201,5cyclictest0-21swapper/222:29:322
1895199201,10cyclictest2708-21irqstats22:10:182
1895199200,18cyclictest0-21swapper/222:20:132
1894599209,3cyclictest17673-21munin-run00:00:011
18945992017,2cyclictest0-21swapper/123:21:521
18945992013,3cyclictest0-21swapper/120:05:511
1894599201,14cyclictest0-21swapper/121:06:281
1894599201,14cyclictest0-21swapper/120:56:551
1894599201,13cyclictest0-21swapper/121:22:551
1894599201,11cyclictest6923-21cpu23:35:121
1894599201,11cyclictest29267-21diskstats00:25:131
1894599201,11cyclictest18273-21cut00:00:151
1894599201,11cyclictest0-21swapper/100:20:161
1894599201,10cyclictest27618-21perl20:40:161
1894599201,10cyclictest24033-21latency_hist19:20:011
1894599201,10cyclictest0-21swapper/100:15:181
1894599200,15cyclictest0-21swapper/121:03:181
1894599200,14cyclictest0-21swapper/100:10:191
1893999208,2cyclictest18374-21kworker/0:221:54:330
1893999207,2cyclictest5989-21kworker/0:022:19:330
1893999206,4cyclictest0-21swapper/023:55:060
1893999205,4cyclictest0-21swapper/000:02:100
1893999205,2cyclictest16834-21kworker/0:022:45:430
1893999202,10cyclictest24133-21kworker/0:000:17:270
1893999202,10cyclictest22645-21/usr/sbin/munin20:30:250
1893999201,9cyclictest17336-21kworker/0:100:05:410
1893999201,9cyclictest17336-21kworker/0:100:05:410
1893999201,7cyclictest14575-21chrt23:50:210
18939992015,3cyclictest10289-21kernelversion20:00:180
1893999201,11cyclictest22035-21/usr/sbin/munin19:15:130
1893999200,8cyclictest29842-21kworker/0:122:24:330
168942201,3sleep00-21swapper/022:43:080
1895699191,10cyclictest3465-21/usr/sbin/munin21:00:143
1895199199,3cyclictest11438-21awk20:05:002
1895199193,3cyclictest31199-21crond20:50:012
1895199191,7cyclictest0-21swapper/220:21:432
1895199191,7cyclictest0-21swapper/200:34:082
1895199191,6cyclictest17003-21ls20:15:202
1895199191,6cyclictest0-21swapper/222:54:262
1895199191,6cyclictest0-21swapper/222:54:262
1895199191,5cyclictest19357-21cut21:35:152
1895199191,5cyclictest18508-21sendmail_mailqu22:45:222
1894599191,9cyclictest30188-21df_inode22:00:131
1894599191,5cyclictest21-21rcuc/120:45:161
1894599191,16cyclictest0-21swapper/121:54:011
1894599191,16cyclictest0-21swapper/100:05:021
1894599191,16cyclictest0-21swapper/100:05:021
1893999199,3cyclictest2359-21sshd19:41:430
1893999198,4cyclictest0-21swapper/022:39:270
1893999198,3cyclictest16660-21/usr/sbin/munin21:30:170
1893999196,2cyclictest20616-21grep22:50:240
1893999196,2cyclictest20616-21grep22:50:230
1893999194,6cyclictest0-21swapper/022:06:510
1893999191,9cyclictest30737-21kworker/0:200:30:050
18939991912,4cyclictest12053-21idleruntime-cro21:20:010
1893999191,11cyclictest7646-21diskmemload23:05:230
1893999191,10cyclictest18241-21/usr/sbin/munin20:20:150
1893999191,10cyclictest15552-21sshd20:11:430
1895699187,6cyclictest0-21swapper/322:17:503
18956991813,3cyclictest4593-21kworker/u8:219:56:493
18956991812,3cyclictest13767-21latency_hist20:10:013
1895199184,8cyclictest0-21swapper/222:18:052
1895199184,5cyclictest755-21rngd00:35:072
1895199182,9cyclictest1495-21kworker/2:023:21:442
1895199181,7cyclictest7646-21diskmemload23:35:122
1895199181,7cyclictest0-21swapper/223:10:542
1895199181,6cyclictest0-21swapper/221:58:442
1895199181,4cyclictest579-21systemd-journal21:10:022
1895199181,4cyclictest12785-21pmu-power20:05:212
1895199181,15cyclictest26524-21ntp_states21:50:212
1895199181,11cyclictest0-21swapper/223:56:052
1895199181,11cyclictest0-21swapper/222:59:312
1894599186,5cyclictest23-21ksoftirqd/122:40:361
1894599182,3cyclictest14007-21grep23:50:151
1894599181,16cyclictest0-21swapper/120:31:521
1893999186,3cyclictest0-21swapper/000:36:120
1893999181,15cyclictest2531-21gltestperf22:10:160
1893999180,8cyclictest8724-21kworker/0:123:49:090
174842180,1sleep20-21swapper/221:30:182
221292171,2sleep00-21swapper/020:26:270
221292171,2sleep00-21swapper/020:26:270
1895699176,3cyclictest87350irq/126-enp1s0-22:10:193
1895699172,3cyclictest0-21swapper/320:15:163
1895199174,3cyclictest87350irq/126-enp1s0-23:00:232
1895199172,9cyclictest0-21swapper/223:30:082
1895199172,6cyclictest8794-21memory21:10:202
1895199171,9cyclictest0-21swapper/200:03:252
1895199171,7cyclictest14144-21smart_sda22:35:232
1895199171,5cyclictest7239-21sshd19:55:042
1895199171,4cyclictest25940-21open_inodes20:35:212
18951991713,2cyclictest22427-21awk20:30:012
18951991713,2cyclictest22427-21awk20:30:012
18951991712,3cyclictest4440-21uptime19:45:252
18951991711,3cyclictest14373-21latency23:50:182
1895199171,10cyclictest0-21swapper/220:10:182
1893999173,3cyclictest0-21swapper/023:16:050
1893999171,6cyclictest0-21swapper/000:21:250
18939991712,3cyclictest7252-21df_inode23:35:140
1893999171,14cyclictest0-21swapper/019:30:050
1893999171,10cyclictest6681-21kworker/0:119:55:120
125602170,2sleep20-21swapper/222:33:552
1895199169,4cyclictest2512-21/usr/sbin/munin20:55:212
1895199165,3cyclictest87650irq/129-enp1s0-19:15:232
1895199161,6cyclictest0-21swapper/223:26:342
1895199161,6cyclictest0-21swapper/221:45:202
1895199161,3cyclictest7646-21diskmemload22:00:122
18951991612,3cyclictest0-21swapper/219:32:302
18951991612,3cyclictest0-21swapper/219:21:342
1895199161,14cyclictest0-21swapper/219:36:392
18951991611,2cyclictest827-21runrttasks22:05:232
1895199161,12cyclictest3480-21bc21:00:122
1893999166,3cyclictest87550irq/128-enp1s0-20:05:070
1893999161,2cyclictest29965-21kworker/u8:019:45:140
18939991611,3cyclictest26675-21seq20:38:260
1893999160,10cyclictest0-21swapper/019:10:210
1893999160,10cyclictest0-21swapper/019:10:210
18939991510,3cyclictest3465-21/usr/sbin/munin21:00:140
18939991510,3cyclictest26798-21fschecks_count19:25:140
1893999149,3cyclictest29235-21seq20:45:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional