You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 07:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Mar 21, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
276822240164,24sleep20-21swapper/219:07:552
277162235164,58sleep00-21swapper/019:08:210
278122229163,53sleep30-21swapper/319:09:373
276242201166,23sleep10-21swapper/119:07:101
2863221670,4sleep12794399cyclictest20:20:251
274121620,5sleep02793899cyclictest00:15:260
1179421370,4sleep32795599cyclictest00:38:363
27955997672,3cyclictest0-21swapper/321:04:203
27955996662,3cyclictest0-21swapper/321:58:173
27955995854,3cyclictest0-21swapper/323:32:033
232202500,3sleep12794399cyclictest20:10:141
16072490,2sleep30-21swapper/319:20:163
249712480,2sleep3401ktimersoftd/321:25:243
27955994541,3cyclictest0-21swapper/323:19:313
304672440,2sleep20-21swapper/220:25:192
2793899367,13cyclictest32653-21munin-run23:00:010
287612320,2sleep02793899cyclictest22:50:140
2795599322,19cyclictest20497-21kworker/3:120:23:443
2795599312,19cyclictest7828-21kworker/3:100:12:403
27955993027,2cyclictest0-21swapper/321:33:333
2795599301,18cyclictest20497-21kworker/3:119:29:003
2793899303,12cyclictest7236-21idleruntime-cro19:35:000
27938993011,16cyclictest3449-21munin-run20:40:010
2795599281,18cyclictest7828-21kworker/3:123:56:403
2794899280,12cyclictest0-21swapper/219:51:362
2794399281,18cyclictest86750irq/128-enp1s0-22:24:341
2795599272,10cyclictest7828-21kworker/3:100:01:123
27955992711,3cyclictest20497-21kworker/3:119:51:083
2795599271,10cyclictest7828-21kworker/3:122:36:123
2794899270,9cyclictest0-21swapper/220:43:232
2794899270,10cyclictest0-21swapper/220:52:272
2794399279,3cyclictest8413-21diskstats20:50:131
2794399271,20cyclictest0-21swapper/122:00:491
2794399271,19cyclictest0-21swapper/121:23:451
2795599262,11cyclictest20497-21kworker/3:120:29:243
2795599261,9cyclictest7828-21kworker/3:100:17:123
2795599261,9cyclictest20497-21kworker/3:119:31:283
2795599261,11cyclictest7828-21kworker/3:122:58:083
2794899261,11cyclictest0-21swapper/221:40:522
2794899261,10cyclictest0-21swapper/222:36:482
2794899261,10cyclictest0-21swapper/221:07:392
2794899260,9cyclictest0-21swapper/220:14:082
2794899260,2cyclictest0-21swapper/223:44:402
2794899260,12cyclictest0-21swapper/222:13:282
2794899260,12cyclictest0-21swapper/200:08:242
2794399265,2cyclictest0-21swapper/121:35:181
2794399261,4cyclictest0-21swapper/123:02:481
2794399260,8cyclictest0-21swapper/100:05:211
174842260,2sleep20-21swapper/221:10:142
77522250,4sleep230-21rcuc/223:15:152
2795599252,10cyclictest7828-21kworker/3:121:47:483
27955992511,2cyclictest7828-21kworker/3:100:20:563
27955992511,2cyclictest7828-21kworker/3:100:20:563
27955992510,3cyclictest7828-21kworker/3:121:36:193
27955992510,3cyclictest20497-21kworker/3:119:13:323
2794899251,6cyclictest0-21swapper/222:20:412
27948992510,3cyclictest923-21awk00:15:012
2794899250,12cyclictest0-21swapper/222:46:402
2794899250,12cyclictest0-21swapper/221:54:282
2794399255,4cyclictest0-21swapper/121:41:331
2794399252,20cyclictest0-21swapper/123:50:061
2794399251,6cyclictest0-21swapper/123:15:011
2794399251,6cyclictest0-21swapper/123:15:011
2794399251,12cyclictest18405-21memory22:25:191
2794399251,11cyclictest0-21swapper/122:43:391
40822242,18sleep14116-21ntp_states19:25:201
2795599249,3cyclictest7828-21kworker/3:123:37:153
2795599242,7cyclictest7828-21kworker/3:122:20:263
2795599242,11cyclictest20497-21kworker/3:120:05:043
2795599242,10cyclictest7828-21kworker/3:123:06:203
2795599242,10cyclictest7828-21kworker/3:122:46:363
2795599241,5cyclictest0-21swapper/322:05:033
2795599241,10cyclictest20497-21kworker/3:119:37:323
2795599241,10cyclictest20497-21kworker/3:119:37:323
27955992410,3cyclictest7828-21kworker/3:123:40:193
27955992410,3cyclictest3079-21kworker/3:020:52:163
27955992410,3cyclictest20497-21kworker/3:120:32:323
27955992410,3cyclictest20497-21kworker/3:119:47:523
27955992410,3cyclictest20497-21kworker/3:119:18:433
2794899241,9cyclictest0-21swapper/223:10:042
2794899241,9cyclictest0-21swapper/223:10:032
2794899241,11cyclictest0-21swapper/221:03:032
2794899241,10cyclictest0-21swapper/221:35:462
2794899240,9cyclictest0-21swapper/222:17:482
2794899240,11cyclictest0-21swapper/200:28:282
2794899240,10cyclictest0-21swapper/219:41:002
2794399243,10cyclictest0-21swapper/122:07:481
2794399241,2cyclictest0-21swapper/123:38:531
2794399241,11cyclictest718-21memory21:45:171
2794399240,6cyclictest0-21swapper/100:30:311
2794399240,5cyclictest0-21swapper/121:10:291
2794399240,5cyclictest0-21swapper/100:17:491
2794399240,4cyclictest0-21swapper/121:51:081
2793899249,13cyclictest0-21swapper/021:20:010
2795599239,3cyclictest7828-21kworker/3:123:03:283
2795599238,3cyclictest20497-21kworker/3:120:16:203
2795599238,2cyclictest3079-21kworker/3:020:58:443
2795599236,3cyclictest7828-21kworker/3:123:10:403
2795599236,3cyclictest7828-21kworker/3:123:10:403
2795599232,9cyclictest7828-21kworker/3:123:23:043
2795599232,9cyclictest7828-21kworker/3:122:52:163
2795599232,8cyclictest7828-21kworker/3:123:52:443
2795599232,10cyclictest20497-21kworker/3:120:11:243
2795599231,9cyclictest7828-21kworker/3:100:05:123
2795599231,6cyclictest0-21swapper/322:31:433
2795599231,10cyclictest20497-21kworker/3:120:03:323
2795599231,10cyclictest0-21swapper/321:52:033
27955992310,3cyclictest7828-21kworker/3:122:03:443
27955992310,3cyclictest7828-21kworker/3:100:32:123
27955992310,2cyclictest20497-21kworker/3:119:59:003
2794899233,4cyclictest0-21swapper/221:15:482
2794899230,11cyclictest0-21swapper/221:28:202
2794899230,10cyclictest0-21swapper/223:58:162
2794399233,16cyclictest32368-40dnf22:56:411
2794399231,3cyclictest0-21swapper/122:45:481
2794399230,6cyclictest0-21swapper/100:25:521
2794399230,6cyclictest0-21swapper/100:22:201
2794399230,6cyclictest0-21swapper/100:22:201
2794399230,5cyclictest0-21swapper/123:05:141
2794399230,5cyclictest0-21swapper/122:30:491
2794399230,5cyclictest0-21swapper/122:18:591
2794399230,5cyclictest0-21swapper/121:15:551
2794399230,3cyclictest0-21swapper/123:28:331
2793899231,11cyclictest0-21swapper/021:43:470
163392230,2sleep00-21swapper/022:20:200
2795599229,3cyclictest20497-21kworker/3:119:42:363
2795599228,2cyclictest3079-21kworker/3:020:40:123
2795599222,4cyclictest0-21swapper/321:15:533
2795599222,10cyclictest7828-21kworker/3:122:27:163
2795599221,5cyclictest0-21swapper/323:29:363
2795599221,11cyclictest7828-21kworker/3:122:19:123
2795599221,10cyclictest7828-21kworker/3:122:11:483
2795599221,10cyclictest3079-21kworker/3:020:45:483
27955992210,2cyclictest7828-21kworker/3:123:48:083
2794899221,4cyclictest11272-21kworker/u8:100:00:272
2794899221,4cyclictest0-21swapper/222:06:392
27948992211,2cyclictest5642-21cat21:55:242
2794899221,12cyclictest0-21swapper/219:48:082
2794899221,12cyclictest0-21swapper/219:33:282
2794899220,9cyclictest0-21swapper/221:31:002
2794899220,8cyclictest0-21swapper/222:43:362
2794899220,10cyclictest0-21swapper/200:33:042
2794399225,14cyclictest25802-21pmu-power23:55:221
2794399221,4cyclictest10124-21irqstats23:20:181
2794399221,19cyclictest0-21swapper/120:58:401
2794399221,19cyclictest0-21swapper/119:20:211
2794399221,18cyclictest30157-21latency_hist19:15:011
27943992210,2cyclictest0-21swapper/122:10:501
2794399220,5cyclictest0-21swapper/121:28:041
2794399220,5cyclictest0-21swapper/100:12:031
2794399220,4cyclictest0-21swapper/121:31:401
2793899226,8cyclictest8667-21load00:30:200
2795599219,2cyclictest7828-21kworker/3:100:25:403
2795599216,2cyclictest3079-21kworker/3:020:37:323
2795599211,4cyclictest19240-21kworker/u8:122:42:383
2794899216,3cyclictest0-21swapper/222:55:202
2794899214,4cyclictest771-21rngd00:36:562
2794899211,9cyclictest0-21swapper/223:38:162
2794899211,10cyclictest0-21swapper/221:21:562
2794899210,9cyclictest0-21swapper/223:54:082
2794899210,11cyclictest0-21swapper/223:29:162
2794899210,10cyclictest0-21swapper/222:03:482
2794399212,17cyclictest0-21swapper/123:43:051
2794399212,17cyclictest0-21swapper/121:05:511
2794399212,17cyclictest0-21swapper/120:03:451
2794399212,17cyclictest0-21swapper/100:01:221
2794399211,7cyclictest0-21swapper/120:46:591
2794399211,18cyclictest0-21swapper/123:15:211
2794399210,3cyclictest0-21swapper/100:38:491
2794399210,19cyclictest0-21swapper/123:31:041
2794399210,13cyclictest86750irq/128-enp1s0-22:51:051
2793899216,6cyclictest8364-1kworker/u9:123:35:050
2793899211,11cyclictest8922-21unixbench-2d19:35:250
2793899211,11cyclictest8922-21unixbench-2d19:35:250
2795599207,7cyclictest86550irq/126-enp1s0-21:42:203
2795599201,11cyclictest16867-21/usr/sbin/munin21:10:193
2794899205,4cyclictest580-21systemd-journal23:10:012
2794899201,8cyclictest214992latency20:05:192
2794899201,5cyclictest0-21swapper/223:47:322
27948992013,4cyclictest17629-21tr22:25:132
2794899201,12cyclictest0-21swapper/220:03:162
2794899200,10cyclictest0-21swapper/200:23:482
2794899200,10cyclictest0-21swapper/200:23:482
2794399205,4cyclictest0-21swapper/121:56:561
2794399202,17cyclictest0-21swapper/120:18:401
2794399202,16cyclictest0-21swapper/121:00:321
2794399202,16cyclictest0-21swapper/120:32:201
2794399201,17cyclictest0-21swapper/120:28:331
2794399201,17cyclictest0-21swapper/119:50:141
2794399201,17cyclictest0-21swapper/119:49:341
2794399200,18cyclictest0-21swapper/119:35:211
2794399200,18cyclictest0-21swapper/119:35:211
2793899207,8cyclictest0-21swapper/020:13:020
2793899206,2cyclictest86750irq/128-enp1s0-00:14:080
2793899201,9cyclictest0-21swapper/022:38:380
2793899201,8cyclictest0-21swapper/000:09:190
2793899201,5cyclictest27968-21ntp_states00:00:200
2793899201,16cyclictest14080-21grep22:15:190
27938992011,4cyclictest0-21swapper/000:25:170
2793899201,11cyclictest4028-21sshd21:54:450
2793899201,11cyclictest17000-21ethtool19:55:170
2793899201,11cyclictest0-21swapper/021:32:000
2793899201,10cyclictest6744-21kworker/0:023:34:440
2793899201,10cyclictest0-21swapper/022:05:540
320162190,1sleep00-21swapper/019:15:200
2795599191,11cyclictest22801-21unixbench_singl21:20:253
2794899191,10cyclictest0-21swapper/221:46:162
2794399192,3cyclictest0-21swapper/120:41:521
2794399191,6cyclictest0-21swapper/122:35:081
2794399191,4cyclictest21145-21grep23:45:181
2794399191,16cyclictest0-21swapper/120:37:121
2794399191,16cyclictest0-21swapper/120:08:321
2794399191,16cyclictest0-21swapper/119:55:361
2794399191,16cyclictest0-21swapper/119:44:171
2794399191,16cyclictest0-21swapper/119:33:531
2793899195,6cyclictest86750irq/128-enp1s0-22:42:070
2793899193,2cyclictest7-21ksoftirqd/022:25:250
2793899191,9cyclictest12119-21sshd20:58:450
2793899191,9cyclictest0-21swapper/023:26:490
2793899191,9cyclictest0-21swapper/023:06:190
2793899191,6cyclictest5182-21unixbench_multi20:40:260
2793899191,4cyclictest0-21swapper/023:00:220
2793899191,3cyclictest25653-21kworker/0:221:37:290
2793899191,10cyclictest0-21swapper/023:21:470
27938991910,2cyclictest12038-21pmu-power22:10:220
15982191,2sleep00-21swapper/021:46:500
117292191,2sleep00-21swapper/000:37:460
2794899184,3cyclictest0-21swapper/220:46:292
2794899183,3cyclictest0-21swapper/223:32:462
2794899182,6cyclictest0-21swapper/219:22:452
2794899181,5cyclictest0-21swapper/223:20:592
2794899181,5cyclictest0-21swapper/200:15:022
2794899181,15cyclictest0-21swapper/219:14:092
2794899181,10cyclictest0-21swapper/219:25:182
2794899180,8cyclictest0-21swapper/220:37:272
2794399181,16cyclictest0-21swapper/119:16:331
2793899187,3cyclictest0-21swapper/000:22:070
2793899187,3cyclictest0-21swapper/000:22:070
2793899186,4cyclictest4222-21ps19:25:210
2793899186,11cyclictest0-21swapper/019:15:010
2793899184,6cyclictest11-21rcuc/020:04:050
2793899183,3cyclictest22476-21latency_hist23:50:020
2793899183,12cyclictest27237-21/usr/sbin/munin20:20:180
2793899181,9cyclictest0-21swapper/023:40:470
2793899181,9cyclictest0-21swapper/021:20:330
2793899181,8cyclictest0-21swapper/023:56:210
2793899181,5cyclictest20757-21ntp_states22:30:210
93432170,2sleep20-21swapper/219:39:142
93432170,2sleep20-21swapper/219:39:142
2795599175,2cyclictest3079-21kworker/3:021:07:323
2794899171,7cyclictest0-21swapper/223:01:282
2794899171,3cyclictest27237-21/usr/sbin/munin20:20:232
2794899170,9cyclictest0-21swapper/220:15:402
2794899170,10cyclictest0-21swapper/222:30:042
2793899175,3cyclictest0-21swapper/019:45:060
2793899171,9cyclictest7450-21df23:15:130
2793899171,8cyclictest0-21swapper/023:50:090
2793899171,8cyclictest0-21swapper/021:10:040
2793899171,6cyclictest0-21swapper/021:56:220
2793899171,5cyclictest7556-21sshd20:46:340
2793899171,5cyclictest16641-21diskmemload22:03:460
27938991714,2cyclictest0-21swapper/022:45:150
27938991713,3cyclictest0-21swapper/020:05:180
27938991712,3cyclictest15587-21awk19:50:250
27938991711,3cyclictest6359-21cat23:10:230
27938991711,3cyclictest6359-21cat23:10:230
27938991711,3cyclictest29747-21expr20:25:130
27938991711,3cyclictest1253-21fschecks_count19:20:140
2793899171,10cyclictest0-21swapper/021:29:380
2794899168,3cyclictest0-21swapper/219:15:532
2794899161,7cyclictest11400-21head20:55:212
2794899161,6cyclictest16241-21/usr/sbin/munin19:55:262
2794899161,5cyclictest8364-1kworker/u9:122:50:402
27948991612,3cyclictest0-21swapper/220:34:102
2793899161,4cyclictest26624-21users20:15:260
27938991612,3cyclictest0-21swapper/021:00:410
27938991611,3cyclictest10130-21cut19:40:150
2793899161,10cyclictest0-21swapper/020:50:220
27938991610,4cyclictest15770-21pluginstate21:05:220
27938991510,3cyclictest32545-21load20:30:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional