You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-20 - 05:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Jun 20, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
852210574,9sleep00-21swapper/019:06:300
86528366,13sleep10-21swapper/119:06:381
82428370,9sleep20-21swapper/219:06:082
78328369,9sleep30-21swapper/319:05:373
1228992928,1cyclictest27414-21kworker/3:100:04:433
122499280,24cyclictest0-21swapper/223:52:522
1228992725,1cyclictest27414-21kworker/3:123:40:433
122499270,2cyclictest0-21swapper/200:20:122
1228992625,1cyclictest0-21swapper/323:28:363
1228992625,1cyclictest0-21swapper/323:28:363
122499260,26cyclictest0-21swapper/223:04:472
1228992521,4cyclictest27414-21kworker/3:122:48:033
1224992524,1cyclictest0-21swapper/222:22:052
122499250,25cyclictest0-21swapper/223:46:302
1228992421,3cyclictest27414-21kworker/3:123:19:133
1228992421,2cyclictest27414-21kworker/3:123:20:433
1228992420,3cyclictest27414-21kworker/3:123:30:423
122499240,22cyclictest0-21swapper/221:28:022
1218992422,1cyclictest13-21ksoftirqd/000:25:560
122499230,22cyclictest0-21swapper/223:37:082
122299234,13cyclictest0-21swapper/119:55:131
122299230,23cyclictest0-21swapper/123:39:381
323592220,0sleep10-21swapper/123:28:051
323592220,0sleep10-21swapper/123:28:041
1228992219,2cyclictest27414-21kworker/3:100:06:423
1224992222,0cyclictest0-21swapper/222:11:402
122499220,21cyclictest0-21swapper/221:37:522
1222992219,1cyclictest0-21swapper/121:40:541
1222992219,1cyclictest0-21swapper/121:40:541
122899210,3cyclictest0-21swapper/321:25:113
122899210,20cyclictest0-21swapper/322:57:173
122499210,21cyclictest0-21swapper/200:01:042
122499210,20cyclictest0-21swapper/222:04:512
122299210,20cyclictest0-21swapper/122:20:141
1218992112,6cyclictest537-21snmpd21:31:160
1228992018,2cyclictest27414-21kworker/3:122:51:233
1228992016,3cyclictest27414-21kworker/3:123:08:233
122499200,2cyclictest0-21swapper/223:06:372
122499200,20cyclictest0-21swapper/222:44:172
122499200,20cyclictest0-21swapper/222:33:432
122499200,18cyclictest0-21swapper/223:26:552
122499200,18cyclictest0-21swapper/223:26:542
1218992018,1cyclictest13-21ksoftirqd/022:01:160
53732190,0chrt0-21swapper/323:11:353
1228991918,1cyclictest27414-21kworker/3:123:52:033
122499190,19cyclictest0-21swapper/223:55:322
122299195,12cyclictest0-21swapper/120:30:131
121899193,11cyclictest0-21swapper/000:05:180
1218991917,1cyclictest13-21ksoftirqd/023:55:330
1218991916,2cyclictest13-21ksoftirqd/019:15:000
121899190,3cyclictest0-21swapper/021:51:570
1228991816,1cyclictest27414-21kworker/3:123:38:333
122499180,18cyclictest0-21swapper/223:15:532
122299180,18cyclictest0-21swapper/121:21:421
1218991811,7cyclictest13-21ksoftirqd/022:47:090
121899180,18cyclictest0-21swapper/021:39:300
1228991716,1cyclictest4877-21kworker/3:200:25:533
1228991715,2cyclictest4877-21kworker/3:200:16:233
122499170,17cyclictest0-21swapper/221:47:332
122499170,16cyclictest0-21swapper/222:55:092
122299172,1cyclictest26640-21diskmemload00:17:291
122299170,17cyclictest0-21swapper/100:25:001
121899174,7cyclictest9652-21awk23:15:000
121899170,16cyclictest0-21swapper/021:29:480
122499162,13cyclictest0-21swapper/222:25:562
1224991615,1cyclictest0-21swapper/221:19:152
122499160,16cyclictest0-21swapper/222:07:412
122499160,16cyclictest0-21swapper/221:41:272
122499160,16cyclictest0-21swapper/221:41:272
122499160,15cyclictest0-21swapper/200:10:462
1222991615,1cyclictest0-21swapper/122:05:011
1222991612,3cyclictest7305-21sed00:10:271
122299160,16cyclictest0-21swapper/122:35:061
121899163,6cyclictest13289-21sh22:00:010
1218991614,1cyclictest13-21ksoftirqd/022:59:280
1218991613,1cyclictest13-21ksoftirqd/023:04:060
121899160,1cyclictest141rcu_preempt23:29:050
121899160,1cyclictest141rcu_preempt23:29:050
1228991514,1cyclictest27414-21kworker/3:123:56:533
1228991513,2cyclictest4877-21kworker/3:200:11:133
1228991513,1cyclictest4877-21kworker/3:200:30:333
1224991515,0cyclictest0-21swapper/221:20:352
1224991513,2cyclictest7352-21ssh00:30:152
1224991511,3cyclictest14354-21wc22:20:012
1224991510,3cyclictest3634-21cat19:15:002
122499150,14cyclictest0-21swapper/223:31:212
1222991512,2cyclictest0-21swapper/122:15:451
122299151,1cyclictest141rcu_preempt21:54:011
121899152,6cyclictest3658-21awk23:50:000
121899151,1cyclictest141rcu_preempt00:10:400
1218991511,1cyclictest141rcu_preempt00:36:120
121899150,12cyclictest0-21swapper/022:17:310
122899145,5cyclictest26640-21diskmemload21:12:163
1228991411,2cyclictest4748-21sh20:25:013
1228991411,2cyclictest20560-21ls21:45:003
1228991411,2cyclictest20560-21ls21:44:593
122899140,12cyclictest13567-21tr19:35:123
1224991410,2cyclictest1853-21grep20:15:262
122299142,4cyclictest0-21swapper/122:09:361
122299141,1cyclictest21658-21systemctl20:59:591
122299140,14cyclictest0-21swapper/121:30:421
122299140,12cyclictest28602-21head20:05:171
121899149,2cyclictest13-21ksoftirqd/000:15:070
1218991413,1cyclictest13-21ksoftirqd/022:44:000
1218991412,1cyclictest13-21ksoftirqd/022:20:140
1218991411,2cyclictest32371-21grep22:10:260
1218991410,3cyclictest15064-21sed19:35:250
121899140,14cyclictest0-21swapper/023:19:280
121899140,13cyclictest15800-21cron22:40:000
122899139,3cyclictest24324-21grep00:20:263
122899139,3cyclictest16446-21tr22:40:133
122899134,4cyclictest5102-21ssh21:15:263
1228991310,2cyclictest8325-21cron19:25:003
1228991310,2cyclictest8147-21grep20:30:163
1228991310,2cyclictest22770-21sort19:55:013
1228991310,2cyclictest22770-21sort19:55:013
1228991310,2cyclictest1939-21grep20:15:273
1228991310,2cyclictest17038-21sed00:35:263
1228991310,2cyclictest16667-21cat20:50:003
122499139,3cyclictest28128-21ssh22:46:332
122499139,3cyclictest11439-21grep20:35:252
122499132,4cyclictest225-21systemd-journal00:26:102
122499131,7cyclictest225-21systemd-journal21:10:002
122499131,12cyclictest0-21swapper/223:44:252
1224991310,2cyclictest9721-21tr19:25:202
1224991310,2cyclictest2872-21tr20:20:152
122499130,12cyclictest0-21swapper/223:24:432
122499130,12cyclictest0-21swapper/223:14:322
122499130,12cyclictest0-21swapper/200:05:032
122299139,3cyclictest4600-21grep19:15:171
122299138,4cyclictest26685-21ssh21:48:171
122299137,6cyclictest38150irq/131-snd_hda19:44:451
1222991312,1cyclictest0-21swapper/100:08:321
1222991310,2cyclictest26209-21grep20:00:161
1222991310,2cyclictest12838-21tr23:55:151
122299130,1cyclictest7345-21rm22:54:041
122299130,12cyclictest5426-21tr23:50:191
121899139,3cyclictest24027-21cat00:20:240
121899134,8cyclictest5976-21awk19:19:590
121899131,6cyclictest19235-21munin-run23:40:010
1218991310,2cyclictest5298-21tr20:25:140
1218991310,2cyclictest28130-21cut20:05:150
1218991310,2cyclictest19836-21cut23:40:140
1218991310,2cyclictest12268-21ssh00:33:100
121899130,4cyclictest11057-21grep21:20:000
121899130,2cyclictest0-21swapper/023:52:060
121899130,1cyclictest141rcu_preempt23:06:040
121899130,1cyclictest0-21swapper/021:44:240
121899130,1cyclictest0-21swapper/021:44:230
122899129,2cyclictest5930-21ssh21:55:153
122899129,2cyclictest25325-21head22:25:263
122899129,2cyclictest25246-21head21:05:183
122899129,2cyclictest12255-21cut20:40:143
122899129,2cyclictest12201-21sendmail-msp21:40:003
122899129,2cyclictest1039-21ssh21:52:003
122899128,3cyclictest29524-21head23:45:213
122499129,2cyclictest28363-21grep21:30:122
122499129,2cyclictest1408-21cut22:50:152
122499128,3cyclictest17407-21sed19:40:262
122499128,3cyclictest14604-21wc19:35:202
122499127,4cyclictest730-21apache220:00:152
122499120,3cyclictest0-21swapper/200:16:522
122299129,2cyclictest371-21tr20:15:151
122299129,2cyclictest30510-21tr20:10:141
122299129,2cyclictest24205-21apt-config21:05:001
122299129,2cyclictest21806-21grep21:25:241
122299129,2cyclictest19459-21cut23:20:171
122299129,2cyclictest18864-21cut19:45:161
122299129,2cyclictest17035-21cat23:00:001
122299129,2cyclictest16311-21ssh22:40:111
122299129,2cyclictest10226-21grep19:25:251
122299128,2cyclictest30541-21ssh22:10:141
122299123,5cyclictest225-21systemd-journal00:39:571
122299120,12cyclictest0-21swapper/100:25:261
122299120,11cyclictest0-21swapper/123:41:501
122299120,11cyclictest0-21swapper/123:00:121
121899129,2cyclictest3380-21head20:20:170
121899129,2cyclictest20254-21wc19:50:000
121899129,2cyclictest20026-21wc23:20:220
121899129,2cyclictest10950-21ls20:35:210
121899123,2cyclictest13-21ksoftirqd/022:25:290
122899119,1cyclictest4587-21sed19:15:173
122899118,2cyclictest31881-21wc22:30:123
122899118,2cyclictest30429-21cat20:10:143
122899118,2cyclictest28043-21ssh21:30:013
122899118,2cyclictest25722-21ssh22:06:413
122899118,2cyclictest2375-21ssh22:12:113
122899118,2cyclictest23624-21grep19:55:153
122899118,2cyclictest22025-21cat21:45:173
122899118,2cyclictest21844-21wc21:00:003
122899118,2cyclictest19219-21grep19:45:193
122899118,2cyclictest17735-21basename20:50:163
122899118,2cyclictest17728-21grep23:00:143
122899118,2cyclictest17106-21ssh22:20:413
122899118,2cyclictest16556-21ssh22:01:013
122899118,2cyclictest16402-21grep19:40:173
122899118,2cyclictest11522-21ssh21:20:113
122899118,2cyclictest11229-21ssh22:36:253
122899118,2cyclictest10664-21ssh22:17:123
122499118,2cyclictest5458-21cut21:55:142
122499118,2cyclictest30445-21tr20:10:142
122499118,2cyclictest29948-21cron20:09:592
122499118,2cyclictest29044-21tail21:10:252
122499118,2cyclictest24240-21cut21:05:002
122499118,2cyclictest19876-21chrt19:45:392
122499118,2cyclictest18030-21tr20:50:182
122499118,2cyclictest11710-21tail19:30:172
122499117,3cyclictest31197-21wc21:50:252
122499112,8cyclictest4624-21hddtemp_smartct19:15:172
122499110,10cyclictest16128-21cat00:35:182
122299118,2cyclictest9368-21ssh21:56:581
122299118,2cyclictest5524-21sed21:35:191
122299118,2cyclictest497-21ssh21:13:241
122299118,2cyclictest2916-21ssh23:49:261
122299118,2cyclictest27144-21ssh22:45:471
122299118,2cyclictest24024-21grep22:25:151
122299118,2cyclictest21307-21expr00:00:141
122299118,2cyclictest2093-21ssh22:30:591
122299118,2cyclictest2073-21cut19:10:151
122299118,2cyclictest17389-21df20:50:141
122299118,2cyclictest16012-21ssh23:18:251
122299118,2cyclictest1347-21ssh23:10:001
122299118,2cyclictest10408-21ssh21:19:311
122299117,3cyclictest6569-21ssh23:12:321
122299117,3cyclictest0-21swapper/120:20:151
122299116,4cyclictest7771-21ssh23:32:371
121899119,2cyclictest0-21swapper/022:33:390
121899118,2cyclictest388-21grep20:15:150
121899118,2cyclictest30513-21sh21:11:300
121899118,2cyclictest24625-21ssh22:05:490
121899118,2cyclictest22423-21ssh21:45:210
121899118,2cyclictest21880-21cat00:00:180
121899118,2cyclictest17049-21wc20:50:130
121899118,2cyclictest14969-21expr20:45:160
121899118,2cyclictest1481-21head22:50:140
121899118,2cyclictest11990-21load19:30:190
121899118,2cyclictest11839-21tr21:20:130
121899117,3cyclictest6487-21tr19:20:140
121899114,6cyclictest25223-21idleruntime-cro20:00:000
121899110,7cyclictest22592-21cron19:55:000
121899110,7cyclictest22592-21cron19:54:590
121899110,10cyclictest3089-21wget23:30:130
122899108,1cyclictest27865-21cut20:05:143
122899108,1cyclictest27349-21taskset20:02:083
122899108,1cyclictest22041-21cut21:00:153
122899107,2cyclictest8891-21cut19:25:153
122899107,2cyclictest1563-21basename19:10:123
122899107,2cyclictest11346-21expr19:30:153
122899107,2cyclictest11294-21wc20:35:253
122499108,1cyclictest8075-21chrt19:21:472
122499108,1cyclictest62962sleep220:25:222
122499108,1cyclictest22347-21chrt19:51:562
122499108,1cyclictest22347-21chrt19:51:562
122499108,1cyclictest214482sleep220:57:422
122499107,2cyclictest23833-21tail19:55:162
122499107,2cyclictest13095-21cut20:40:182
122499107,2cyclictest10419-21sh22:35:472
122499100,9cyclictest8383-21cat20:30:192
122299108,2cyclictest12833-21grep20:40:161
122299108,1cyclictest9184-21ssh00:30:451
122299108,1cyclictest6156-21ntp_offset20:25:201
122299108,1cyclictest14937-21expr20:45:141
122299108,1cyclictest128632sleep119:32:201
122299107,2cyclictest6476-21cut19:20:141
122299107,2cyclictest13359-21cut19:35:121
122299106,3cyclictest22599-21cron19:55:001
122299106,3cyclictest22599-21cron19:55:001
122299102,2cyclictest225-21systemd-journal21:10:001
122299100,9cyclictest10048-21tr20:35:141
121899108,2cyclictest7964-21diskstats20:30:160
121899108,1cyclictest8886-21expr19:25:150
121899108,1cyclictest31550-21sendmail_mailst20:10:220
121899108,1cyclictest14133-21chrt20:43:310
121899107,2cyclictest25897-21ls20:00:140
121899107,2cyclictest24657-21tr21:05:150
121899107,2cyclictest22628-21grep21:00:180
121899107,2cyclictest20576-21grep20:55:200
12289997,1cyclictest5125-21awk20:25:143
12249993,4cyclictest16662-21perf20:50:002
12189997,1cyclictest16643-21if_err_enp2s019:40:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional