You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 10:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 30, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
245652234163,58sleep20-21swapper/219:09:182
244742203164,26sleep30-21swapper/319:08:103
243882201165,24sleep10-21swapper/119:07:041
243402200163,26sleep00-21swapper/019:06:250
24715991091,106cyclictest0-21swapper/222:50:352
24715991080,106cyclictest0-21swapper/222:05:422
24715999189,1cyclictest0-21swapper/222:03:442
24715998683,2cyclictest0-21swapper/219:49:222
24715997875,2cyclictest0-21swapper/221:46:172
53082460,2sleep20-21swapper/219:35:222
296322450,4sleep00-21swapper/020:30:220
24715993929,9cyclictest0-21swapper/223:25:462
152352380,5sleep013647-21/usr/sbin/munin21:10:270
2470599378,14cyclictest4614-21munin-run20:50:010
48352360,2sleep04839-21gltestperf19:35:150
2470599358,12cyclictest21166-21idleruntime-cro23:10:010
2470599338,12cyclictest9899-21munin-run22:30:020
2470599311,14cyclictest32582-21munin-run20:40:010
124512300,4sleep123-21ksoftirqd/100:30:161
2470599294,12cyclictest26889-21awk19:15:010
2472299281,20cyclictest0-21swapper/300:00:173
2471099281,5cyclictest0-21swapper/122:05:411
2470599283,12cyclictest14907-21munin-run22:45:010
2470599282,13cyclictest10597-21munin-run19:50:010
2470599280,17cyclictest7691-21kworker/0:100:14:370
2470599280,17cyclictest7691-21kworker/0:100:14:370
2470599280,13cyclictest23979-21awk23:20:010
2472299274,7cyclictest0-21swapper/323:28:453
2470599271,15cyclictest15827-21kworker/0:221:17:050
2472299251,10cyclictest28420-21users21:40:263
2471099252,3cyclictest27290-21ntp_states20:25:221
2471099251,3cyclictest579-21systemd-journal19:45:011
2471099251,2cyclictest0-21swapper/122:35:571
2471099251,16cyclictest0-21swapper/123:36:591
2471099250,2cyclictest0-21swapper/123:41:221
2471099250,2cyclictest0-21swapper/100:22:021
2470599259,14cyclictest0-21swapper/000:00:000
2470599252,8cyclictest17586-21kworker/0:023:03:490
2470599252,8cyclictest17586-21kworker/0:023:03:490
2470599252,8cyclictest17025-21kworker/0:220:04:250
2470599250,9cyclictest3547-21kworker/0:022:06:490
2472299242,14cyclictest18184-21kworker/u8:223:41:043
2471099245,10cyclictest0-21swapper/100:01:151
2471099241,6cyclictest18339-21perf23:00:011
2471099241,3cyclictest6190-21idleruntime-cro19:40:011
2471099241,17cyclictest0-21swapper/121:18:551
2471099241,11cyclictest5633-21ls22:10:211
2471099240,2cyclictest0-21swapper/100:17:311
2471099240,18cyclictest0-21swapper/119:22:061
2470599242,9cyclictest6461-21kworker/0:222:22:450
2470599242,9cyclictest32332-21kworker/0:320:51:410
2470599242,9cyclictest19642-21kworker/0:021:36:200
2470599242,9cyclictest17025-21kworker/0:220:23:250
2470599242,8cyclictest23094-21kworker/0:023:33:250
2470599242,8cyclictest11915-21kworker/0:322:49:450
2472299232,15cyclictest87350irq/126-enp1s0-23:00:013
2472299231,16cyclictest0-21swapper/321:10:143
2472299231,12cyclictest22612-40dnf23:10:223
2472299230,17cyclictest0-21swapper/322:35:493
2472299230,14cyclictest4163-21kworker/u8:122:17:323
2471599231,12cyclictest0-21swapper/200:10:152
2471599231,12cyclictest0-21swapper/200:10:152
2471099237,4cyclictest5704-21sort20:50:181
2471099231,6cyclictest0-21swapper/122:53:301
2471099231,3cyclictest23076-21chrt20:15:221
2471099231,16cyclictest0-21swapper/121:46:571
2471099231,16cyclictest0-21swapper/100:36:521
2471099231,13cyclictest13980-21df21:10:141
2471099231,11cyclictest0-21swapper/123:30:271
2471099230,6cyclictest0-21swapper/122:30:401
2470599239,2cyclictest17025-21kworker/0:220:07:590
2470599232,8cyclictest6461-21kworker/0:222:16:130
2470599232,8cyclictest23094-21kworker/0:023:28:330
2470599231,9cyclictest8832-21kworker/0:221:07:550
2470599231,9cyclictest5451-21kworker/0:000:15:130
2470599231,1cyclictest0-21swapper/020:15:070
2470599231,10cyclictest19642-21kworker/0:021:21:120
2470599230,9cyclictest509-21kworker/0:322:01:170
234692232,17sleep223498-21uname21:30:202
2472299229,4cyclictest839-21lldpd22:10:063
2472299229,4cyclictest24632-21/usr/sbin/munin21:35:123
2472299227,2cyclictest0-21swapper/300:39:153
2472299223,10cyclictest5642-21chrt00:09:293
2472299221,15cyclictest6413-21kworker/u8:222:24:053
2472299221,14cyclictest0-21swapper/300:20:403
2472299221,13cyclictest0-21swapper/300:17:313
2472299220,9cyclictest0-21swapper/322:07:543
2471599224,11cyclictest4500-21chrt22:10:082
2471599221,11cyclictest11378-21latency_hist21:05:022
2471099221,2cyclictest324752sleep120:38:501
2471099221,2cyclictest0-21swapper/120:45:211
2471099221,19cyclictest0-21swapper/119:10:281
2470599222,8cyclictest8832-21kworker/0:221:04:070
2470599222,8cyclictest29562-21kworker/0:019:21:070
2470599222,8cyclictest17025-21kworker/0:220:12:270
2470599222,11cyclictest0-21swapper/022:10:070
2470599222,10cyclictest8357-21kworker/0:300:21:030
2470599221,9cyclictest5451-21kworker/0:000:08:100
2470599221,9cyclictest1048-21kworker/0:219:19:510
2470599221,5cyclictest0-21swapper/021:48:500
24705992210,2cyclictest19909-21kworker/0:321:32:520
2470599220,8cyclictest17586-21kworker/0:022:59:290
2472299219,3cyclictest0-21swapper/321:50:103
2472299219,3cyclictest0-21swapper/321:50:103
2472299213,4cyclictest30267-21fgrep21:45:243
2472299211,8cyclictest7670-21sed20:55:153
2472299211,13cyclictest0-21swapper/321:28:593
2472299211,11cyclictest14983-21cat22:45:013
2472299210,3cyclictest24867-21gltestperf23:20:153
2472299210,14cyclictest0-21swapper/323:19:483
2471599212,8cyclictest579-21systemd-journal00:05:012
24715992110,3cyclictest1442-21awk22:00:012
24715992110,2cyclictest13418-21awk21:10:012
2471099219,3cyclictest24444-21cstates23:20:121
2471099216,9cyclictest0-21swapper/122:16:551
2471099216,8cyclictest0-21swapper/123:48:321
2471099215,8cyclictest0-21swapper/121:55:431
24710992117,3cyclictest0-21swapper/122:40:121
2471099211,18cyclictest0-21swapper/123:10:231
2471099211,18cyclictest0-21swapper/122:20:081
2471099211,18cyclictest0-21swapper/121:41:551
2471099211,18cyclictest0-21swapper/121:27:501
2471099211,18cyclictest0-21swapper/119:29:211
2471099211,11cyclictest6595-21hddtemp_smartct00:10:151
2471099211,11cyclictest6595-21hddtemp_smartct00:10:141
2471099211,11cyclictest0-21swapper/122:26:301
2471099211,10cyclictest13417-21diskmemload23:50:131
2471099211,10cyclictest10766-21cat19:50:011
2470599219,2cyclictest25888-21kworker/0:020:25:090
2470599216,2cyclictest29562-21kworker/0:019:25:240
2470599212,8cyclictest28580-21kworker/0:021:43:520
2470599211,9cyclictest8319-21kworker/0:319:51:470
2470599211,9cyclictest31881-21kworker/0:119:32:060
2470599211,9cyclictest31881-21kworker/0:119:32:060
2470599211,8cyclictest8964-21kworker/0:022:30:270
2470599211,8cyclictest26644-21kworker/0:223:35:210
2470599211,4cyclictest0-21swapper/023:44:310
24705992114,4cyclictest13650-21cut19:55:150
2470599211,12cyclictest0-21swapper/022:51:520
2470599210,9cyclictest28580-21kworker/0:021:55:170
97572200,4sleep39761-21pmu-power19:45:223
2472299209,3cyclictest19648-21cpu20:10:133
2472299209,2cyclictest25899-21grep19:10:193
2472299208,3cyclictest11323-21sa100:30:013
2472299202,9cyclictest41-21ksoftirqd/321:20:253
2472299202,13cyclictest87350irq/126-enp1s0-21:30:263
2472299201,6cyclictest26017-21sort20:25:013
2472299201,16cyclictest19186-21/usr/sbin/munin23:00:163
2472299201,16cyclictest19186-21/usr/sbin/munin23:00:163
2472299201,14cyclictest0-21swapper/322:45:163
2472299201,13cyclictest87350irq/126-enp1s0-22:31:143
2472299201,13cyclictest4163-21kworker/u8:122:50:273
24722992011,2cyclictest0-21swapper/321:55:533
2472299201,11cyclictest14745-21sshd19:57:043
2472299201,11cyclictest13417-21diskmemload00:30:253
2472299201,11cyclictest1323-21ntp_states23:50:213
2472299200,7cyclictest0-21swapper/323:31:403
2471599202,6cyclictest28804-21munin-run21:45:012
2471599202,16cyclictest0-21swapper/220:06:142
2471599201,8cyclictest0-21swapper/222:23:582
2471599201,5cyclictest0-21swapper/222:19:572
2471599201,12cyclictest0-21swapper/200:35:382
2471099208,3cyclictest20148-21users23:00:261
2471099208,3cyclictest20148-21users23:00:261
2471099207,4cyclictest0-21swapper/123:27:061
2471099203,10cyclictest0-21swapper/121:38:581
2471099201,5cyclictest0-21swapper/100:06:121
2471099201,18cyclictest0-21swapper/123:08:311
2471099201,17cyclictest0-21swapper/122:46:401
2471099201,17cyclictest0-21swapper/121:33:221
2471099201,17cyclictest0-21swapper/100:25:361
2471099201,13cyclictest0-21swapper/121:50:101
2471099201,13cyclictest0-21swapper/121:50:101
2471099201,11cyclictest0-21swapper/121:22:431
24710992010,3cyclictest19385-21latency_hist20:10:021
2470599208,3cyclictest19642-21kworker/0:021:26:320
2470599208,2cyclictest32019-21kworker/0:200:01:180
2470599205,2cyclictest4834-21kworker/0:020:56:450
2470599202,9cyclictest28164-21kworker/0:120:40:030
2470599202,9cyclictest28164-21kworker/0:120:40:030
2470599201,9cyclictest7691-21kworker/0:100:30:070
2470599201,8cyclictest28580-21kworker/0:021:53:430
2470599201,8cyclictest28580-21kworker/0:021:53:430
2472299198,3cyclictest4844-21/usr/sbin/munin20:50:263
2472299193,3cyclictest87350irq/126-enp1s0-19:30:013
2472299191,6cyclictest0-21swapper/320:05:443
2472299191,5cyclictest9378-21grep21:00:113
2472299191,5cyclictest10841-21egrep19:50:113
2472299191,15cyclictest17375-21unixbench_singl21:15:253
2472299191,10cyclictest808-21snmpd19:15:073
2472299191,10cyclictest11445-21/usr/sbin/munin21:05:123
2472299190,4cyclictest0-21swapper/319:35:013
2472299190,4cyclictest0-21swapper/319:35:013
2472299190,1cyclictest0-21swapper/323:36:343
2471599196,5cyclictest30-21rcuc/221:37:582
2471599195,4cyclictest23094-21switchtime20:15:222
2471599194,5cyclictest755-21rngd23:06:192
2471599192,6cyclictest21383-21/usr/sbin/munin23:10:232
2471599192,15cyclictest0-21swapper/219:15:482
24715991911,4cyclictest14797-21grep21:10:232
24715991910,3cyclictest87350irq/126-enp1s0-20:59:342
2471099199,8cyclictest0-21swapper/120:10:161
2471099198,3cyclictest2018-21df22:00:131
24710991915,3cyclictest0-21swapper/119:18:521
2471099191,17cyclictest0-21swapper/121:05:071
2471099191,17cyclictest0-21swapper/121:00:471
2471099191,17cyclictest0-21swapper/120:33:561
2471099191,17cyclictest0-21swapper/120:03:571
2471099191,17cyclictest0-21swapper/119:31:111
2471099191,17cyclictest0-21swapper/119:31:111
2471099191,16cyclictest0-21swapper/119:55:381
2471099191,10cyclictest1730-21/usr/sbin/munin20:40:251
2471099191,10cyclictest1730-21/usr/sbin/munin20:40:251
2471099191,10cyclictest0-21swapper/123:55:371
2471099190,18cyclictest0-21swapper/119:52:241
2470599198,2cyclictest23747-21kworker/0:223:21:210
2470599198,2cyclictest10568-21kworker/0:200:26:590
2470599190,9cyclictest31881-21kworker/0:119:40:270
2472299189,2cyclictest3033-21fschecks_count20:45:153
2472299185,4cyclictest0-21swapper/323:56:513
2472299181,9cyclictest808-21snmpd20:15:213
2472299181,9cyclictest1422-21strings20:40:203
2472299181,9cyclictest1422-21strings20:40:203
2472299181,7cyclictest2714-21grep22:00:213
2472299181,12cyclictest0-21swapper/300:10:063
2472299181,12cyclictest0-21swapper/300:10:063
2472299181,11cyclictest0-21swapper/323:06:443
2472299181,10cyclictest0-21swapper/323:47:433
2471599182,6cyclictest910-21systemd-logind21:30:012
2471599182,3cyclictest0-21swapper/223:36:232
2471599182,13cyclictest13526-21grep22:40:142
2471599182,10cyclictest0-21swapper/219:25:592
2471599181,8cyclictest5699-21munin-run00:10:012
2471599181,5cyclictest0-21swapper/221:19:352
24715991814,3cyclictest0-21swapper/223:04:082
24715991814,3cyclictest0-21swapper/223:04:082
2471599181,3cyclictest25552-21unixbench_multi20:20:252
2471599181,3cyclictest0-21swapper/200:17:442
24715991813,3cyclictest32-21ksoftirqd/220:30:002
24715991813,3cyclictest30191-21grep23:40:152
24715991812,3cyclictest24195-21/usr/sbin/munin23:20:162
2471599181,15cyclictest0-21swapper/223:50:532
2471599181,15cyclictest0-21swapper/222:55:422
2471599181,11cyclictest0-21swapper/219:30:552
2471599181,11cyclictest0-21swapper/219:30:552
2471599181,10cyclictest0-21swapper/221:23:022
2471099182,2cyclictest87550irq/128-enp1s0-23:17:271
24710991815,2cyclictest0-21swapper/120:57:281
2471099181,15cyclictest0-21swapper/120:20:081
2470599181,16cyclictest18279-21kworker/0:123:10:490
95372171,3sleep20-21swapper/222:25:352
316242171,3sleep30-21swapper/319:20:473
2472299177,2cyclictest39-21rcuc/320:25:233
2472299171,9cyclictest0-21swapper/322:28:233
2472299171,9cyclictest0-21swapper/320:35:133
2472299171,9cyclictest0-21swapper/319:40:123
2472299171,8cyclictest87350irq/126-enp1s0-20:30:133
2472299171,5cyclictest15947-21if_enp1s020:00:173
2471599179,4cyclictest0-21swapper/220:30:112
2471599176,8cyclictest30-21rcuc/200:30:142
2471599171,7cyclictest0-21swapper/223:59:342
2471599171,6cyclictest31529-21memory20:35:192
2471599171,6cyclictest0-21swapper/223:49:222
24715991713,3cyclictest0-21swapper/222:47:162
24715991712,3cyclictest3711-21ntp_states20:45:212
2471599171,15cyclictest0-21swapper/219:53:582
2471599171,14cyclictest0-21swapper/223:15:152
2471599171,14cyclictest0-21swapper/220:10:582
24715991711,3cyclictest13018-21/usr/sbin/munin19:55:242
24715991710,4cyclictest0-21swapper/223:30:212
2470599176,3cyclictest13472-21kworker/0:200:35:570
2470599171,6cyclictest0-21swapper/023:46:410
2472299161,5cyclictest0-21swapper/319:35:163
2471599168,2cyclictest0-21swapper/219:14:022
2471599161,7cyclictest0-21swapper/222:38:132
2471599161,6cyclictest0-21swapper/200:25:322
2471599161,5cyclictest1411-21needreboot20:40:192
2471599161,5cyclictest1411-21needreboot20:40:192
2471599161,5cyclictest0-21swapper/222:31:362
2471599161,5cyclictest0-21swapper/219:40:242
24715991612,3cyclictest0-21swapper/219:23:302
24715991612,3cyclictest0-21swapper/200:21:122
2471599161,10cyclictest0-21swapper/220:51:092
2470599163,2cyclictest29578-21kworker/0:023:50:240
2470599161,14cyclictest0-21swapper/022:36:270
2471599155,3cyclictest87350irq/126-enp1s0-20:00:172
2471599151,5cyclictest1-21systemd21:50:052
2471599151,5cyclictest1-21systemd21:50:052
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional