You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-31 - 04:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jul 31, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1192721290,0sleep00-21swapper/019:19:560
3006321000,1sleep3225-21systemd-journal21:05:003
679229785,8sleep00-21swapper/019:05:330
697528976,9sleep20-21swapper/219:07:562
707928768,15sleep10-21swapper/119:09:151
697128370,9sleep30-21swapper/319:07:523
141292620,0sleep00-21swapper/021:37:030
279772440,0sleep20-21swapper/200:19:352
722999270,2cyclictest0-21swapper/300:26:473
722999240,23cyclictest0-21swapper/321:15:023
722499240,22cyclictest0-21swapper/123:29:171
722699230,0cyclictest0-21swapper/223:05:522
722099230,23cyclictest0-21swapper/000:08:320
7229992221,1cyclictest0-21swapper/323:22:043
722099227,1cyclictest32456-21diskmemload22:51:290
722999210,2cyclictest0-21swapper/300:33:543
722699210,21cyclictest0-21swapper/223:33:582
7229992020,0cyclictest0-21swapper/322:18:353
722699200,3cyclictest0-21swapper/200:03:362
722699200,2cyclictest0-21swapper/221:10:002
7224992019,1cyclictest0-21swapper/122:52:361
722499200,17cyclictest0-21swapper/120:25:261
722499200,15cyclictest0-21swapper/119:40:121
722999190,1cyclictest0-21swapper/323:33:533
722999190,13cyclictest0-21swapper/319:45:123
722699190,19cyclictest0-21swapper/222:43:192
7220991913,5cyclictest29406-21apache222:10:180
7226991817,1cyclictest0-21swapper/222:15:572
722699180,18cyclictest0-21swapper/223:20:212
722699180,17cyclictest0-21swapper/223:59:042
722699180,12cyclictest0-21swapper/223:28:542
722499180,17cyclictest0-21swapper/122:45:341
722099180,1cyclictest0-21swapper/023:18:110
147352180,1sleep10-21swapper/123:14:101
7226991717,0cyclictest0-21swapper/219:36:432
7226991714,1cyclictest0-21swapper/222:39:502
722499170,3cyclictest0-21swapper/121:12:361
722499170,1cyclictest0-21swapper/122:05:031
722499170,17cyclictest0-21swapper/123:32:191
722499170,17cyclictest0-21swapper/121:31:131
722499170,16cyclictest0-21swapper/122:21:101
722499170,0cyclictest0-21swapper/123:15:571
722099175,6cyclictest23877-21perf19:45:000
722099173,8cyclictest32681-21awk23:24:590
722099170,1cyclictest0-21swapper/021:22:030
7229991614,1cyclictest0-21swapper/322:30:333
722999161,14cyclictest0-21swapper/323:08:053
7226991615,1cyclictest0-21swapper/219:30:152
7226991614,2cyclictest4432-21kworker/u8:022:00:112
7226991613,2cyclictest32-21ksoftirqd/221:30:142
722699160,1cyclictest0-21swapper/200:23:342
722699160,1cyclictest0-21swapper/200:23:342
722699160,0cyclictest0-21swapper/223:49:382
722499160,1cyclictest0-21swapper/121:38:061
7220991615,0cyclictest32456-21diskmemload23:29:590
722099160,1cyclictest0-21swapper/022:23:470
722099160,13cyclictest0-21swapper/000:01:300
722999158,5cyclictest23674-21ssh23:00:183
722999156,5cyclictest225-21systemd-journal22:50:003
722999154,4cyclictest15825-21munin-run23:14:593
722999150,14cyclictest0-21swapper/323:38:593
722999150,11cyclictest0-21swapper/322:40:133
722999150,0cyclictest0-21swapper/320:40:123
722699150,15cyclictest0-21swapper/200:34:502
7224991515,0cyclictest0-21swapper/100:38:271
7224991514,1cyclictest0-21swapper/123:06:181
722499151,3cyclictest0-21swapper/122:15:121
7224991510,3cyclictest19874-21grep20:40:241
722499150,1cyclictest0-21swapper/100:22:281
722499150,1cyclictest0-21swapper/100:22:281
722499150,15cyclictest0-21swapper/123:48:541
722999149,3cyclictest13687-21grep20:30:143
7229991412,1cyclictest0-21swapper/321:40:143
7229991411,2cyclictest8226-21tail19:10:173
7229991411,2cyclictest1668-21tail23:25:193
7229991410,2cyclictest3190-21grep21:50:143
722999140,12cyclictest0-21swapper/323:45:513
722699145,3cyclictest0-21swapper/223:00:302
7226991410,3cyclictest15976-21grep22:55:182
722499142,12cyclictest8450-21kworker/1:021:56:141
722499141,3cyclictest141rcu_preempt23:23:291
722099143,7cyclictest31488-21perf23:04:590
722099143,1cyclictest141rcu_preempt21:33:530
7220991413,1cyclictest13-21ksoftirqd/022:37:180
7220991410,3cyclictest26542-21grep23:40:170
722099140,7cyclictest161rcuc/000:20:000
722099140,13cyclictest4472-21cron00:25:000
722099140,13cyclictest4472-21cron00:25:000
722999139,3cyclictest13322-21tail19:20:193
7229991310,2cyclictest9203-21grep20:20:153
7229991310,2cyclictest28926-21sed21:45:273
7229991310,2cyclictest20878-21ssh22:00:193
7229991310,2cyclictest13758-21ps00:10:273
7229991310,2cyclictest11703-21cut20:25:173
722999130,1cyclictest0-21swapper/322:56:273
722999130,1cyclictest0-21swapper/322:52:533
722699139,3cyclictest14332-21cron19:25:002
722699136,6cyclictest11448-21apt-get21:55:142
722699134,8cyclictest0-21swapper/221:16:362
722699132,5cyclictest29364-21apache222:30:122
7226991312,1cyclictest0-21swapper/223:50:252
7226991310,2cyclictest3360-21grep21:50:152
7226991310,2cyclictest26561-21chrt00:37:382
7226991310,2cyclictest12318-21grep00:10:182
722699130,1cyclictest0-21swapper/200:07:352
722699130,0cyclictest0-21swapper/222:51:212
722499139,4cyclictest29406-21apache223:52:251
722499139,3cyclictest25750-21cut21:25:141
722499137,2cyclictest225-21systemd-journal23:04:591
722499133,9cyclictest23024-21grep22:40:161
7224991312,1cyclictest0-21swapper/120:30:121
7224991310,2cyclictest38150irq/131-snd_hda00:15:171
7224991310,2cyclictest27311-21cut19:50:161
7224991310,2cyclictest20417-21unixbench_singl21:40:271
7224991310,2cyclictest18785-21tr23:35:201
7224991310,2cyclictest16051-21cat19:25:261
722499130,2cyclictest0-21swapper/100:26:011
722099139,2cyclictest3976-21grep20:10:130
722099138,3cyclictest17215-21tr19:30:160
722099136,3cyclictest225-21systemd-journal21:44:590
722099136,3cyclictest225-21systemd-journal20:50:000
722099135,7cyclictest19313-21sendmail-msp22:00:010
722099135,6cyclictest1089-21ssh21:10:170
722099132,8cyclictest11149-21df_inode23:50:150
7220991310,2cyclictest6317-21head22:30:190
7220991310,2cyclictest32140-21grep20:00:160
7220991310,2cyclictest26621-21sort21:25:180
7220991310,2cyclictest21452-21cron19:39:590
7220991310,2cyclictest18341-21grep23:35:160
7220991310,2cyclictest1534-21grep20:05:140
722099130,1cyclictest0-21swapper/023:56:390
722099130,12cyclictest3640-21fgrep23:45:210
722099130,12cyclictest23731-21grep00:35:300
722999129,2cyclictest3666-21sh22:10:013
722999129,2cyclictest27917-21grep23:40:273
722999129,2cyclictest26766-21cron00:00:003
722999129,2cyclictest18706-21head21:20:213
722999129,2cyclictest1222-21cron20:05:013
722999129,1cyclictest0-21swapper/300:35:293
722999128,3cyclictest16824-21wc19:30:013
722999128,2cyclictest5076-21cut20:10:213
722999125,6cyclictest29576-21apache_volume22:25:133
722999124,4cyclictest30399-21ntp_states00:20:223
722999124,4cyclictest30399-21ntp_states00:20:213
7229991211,1cyclictest0-21swapper/300:15:153
722999120,5cyclictest0-21swapper/321:55:583
722999120,1cyclictest0-21swapper/321:31:353
722999120,1cyclictest0-21swapper/319:15:143
722999120,11cyclictest10662-21basename23:50:143
722999120,10cyclictest0-21swapper/322:11:533
722699129,2cyclictest27680-21sed21:25:272
722699129,2cyclictest22000-21grep19:40:132
722699129,2cyclictest17207-21head23:15:192
722699129,2cyclictest128032sleep200:29:562
722699129,2cyclictest11748-21head20:25:172
722699123,6cyclictest27414-21munin-run21:00:012
7226991210,1cyclictest26490-21cp19:50:002
722699120,12cyclictest0-21swapper/222:49:152
722699120,11cyclictest30724-21chrt21:46:432
722699120,11cyclictest26103-21tr23:40:142
722499129,2cyclictest9207-21grep21:15:161
722499129,2cyclictest8194-21head19:10:161
722499129,2cyclictest7033-21head20:15:171
722499129,2cyclictest4994-21sed22:10:161
722499129,2cyclictest4263-21ssh22:29:271
722499129,2cyclictest31259-21latency_hist19:59:591
722499129,2cyclictest309582sleep100:01:301
722499129,2cyclictest20102-21sh00:15:001
722499129,2cyclictest11976-21basename19:20:011
722499128,3cyclictest326002sleep120:00:191
722499127,2cyclictest22071-21ssh22:59:161
722099129,2cyclictest9208-21tr21:15:160
722099129,2cyclictest8874-21tr20:20:140
722099129,2cyclictest8674-21grep19:10:200
722099129,2cyclictest6213-21ssh22:50:000
722099129,2cyclictest16509-21head20:35:160
722099128,2cyclictest9406-21tr23:30:150
7220991210,1cyclictest7509-21wc20:15:210
722099120,11cyclictest29842-21grep22:25:140
722999119,1cyclictest31068-21taskset19:57:463
722999118,2cyclictest7981-21awk20:15:253
722999118,2cyclictest4049-21ssh21:11:383
722999118,2cyclictest27635-21grep21:25:273
722999118,2cyclictest27460-21cut21:00:013
722999118,2cyclictest23860-21ls19:44:593
722999118,2cyclictest21840-21sort22:20:153
722999118,2cyclictest19418-21hostname19:35:143
722999118,2cyclictest16167-21ssh23:15:063
722999118,2cyclictest14248-21ls22:35:153
722999118,2cyclictest1043-21ssh00:03:293
722999118,1cyclictest15771-21ssh21:38:193
722999117,3cyclictest4672-21cat00:05:213
722999110,10cyclictest16870-21grep20:35:203
722699118,2cyclictest809-21tr21:10:162
722699118,2cyclictest7550-21ssh22:11:312
722699118,2cyclictest28414-21sed19:50:262
722699118,2cyclictest24657-21ssh22:21:372
722699118,2cyclictest2172-21rm22:08:522
722699118,2cyclictest19465-21sed23:35:262
722699118,2cyclictest17994-21sort21:20:172
722699118,2cyclictest14845-21ssh21:37:372
722699118,2cyclictest12019-21sh23:12:032
722699117,3cyclictest25017-21cron20:55:012
722699117,3cyclictest25017-21cron20:55:002
722699117,3cyclictest2287-21head20:05:172
722699117,2cyclictest29530-21acpi22:25:112
722699117,1cyclictest19615-21load21:40:202
722699113,4cyclictest22669-21idleruntime-cro20:50:002
722699112,4cyclictest0-21swapper/220:35:132
722699110,10cyclictest32526-21wc20:00:202
722499118,2cyclictest8928-21sh22:31:341
722499118,2cyclictest3218-21sed20:05:271
722499118,2cyclictest27005-21sh22:04:301
722499118,2cyclictest26253-21expr23:40:151
722499118,2cyclictest1973-21ssh21:49:331
722499118,2cyclictest17593-21sh00:32:101
722499118,2cyclictest13906-21grep22:35:131
722499118,2cyclictest12463-21tr19:20:141
722499118,2cyclictest10589-21ssh21:54:451
722499118,2cyclictest10238-21ssh00:08:541
7224991110,1cyclictest32456-21diskmemload23:58:031
722499110,10cyclictest19970-21ssh21:20:581
722099118,2cyclictest6566-21ssh21:51:370
722099118,2cyclictest5906-21cut00:25:200
722099118,2cyclictest32698-21ssh22:07:220
722099118,2cyclictest32050-21ssh23:05:150
722099118,2cyclictest31080-21grep21:05:150
722099118,2cyclictest27896-21grep21:45:180
722099118,2cyclictest22588-21chrt22:01:030
722099118,2cyclictest19790-21grep20:40:250
722099118,2cyclictest15495-21ssh00:11:290
722099118,2cyclictest15400-21which19:25:170
722099118,2cyclictest15111-21ssh22:55:110
722099118,2cyclictest13612-21rm22:15:160
722099118,2cyclictest13284-21ssh00:30:080
722099118,2cyclictest11291-21grep20:25:140
722099110,10cyclictest29186-21ssh22:44:180
722099110,10cyclictest11246-21ssh23:11:260
722999108,1cyclictest32176-21seq21:06:453
722999108,1cyclictest21681-21ntpq20:45:193
722999108,1cyclictest17581-21awk19:30:183
722999108,1cyclictest1693-21wc20:05:143
722999107,2cyclictest24003-21cut20:50:183
722999107,2cyclictest24003-21cut20:50:183
722699108,1cyclictest7076-21basename20:15:162
722699108,1cyclictest4076-21cat20:10:142
722699108,1cyclictest27956-21aten2_r7power_e21:00:142
722699108,1cyclictest19930-21chrt20:40:432
722699108,1cyclictest16465-21chrt19:26:552
722699108,1cyclictest154772sleep220:34:462
722699108,1cyclictest10547-21taskset20:22:222
722699107,2cyclictest29504-21cat19:55:152
722699102,4cyclictest225-21systemd-journal19:15:082
722699100,9cyclictest8852-21python19:10:242
722499108,1cyclictest26619-21taskset20:55:231
722499108,1cyclictest21192-21taskset19:36:381
722499107,2cyclictest9144-21df20:20:151
722499107,2cyclictest30440-21aten2_r7power_v21:05:121
722499107,2cyclictest29820-21seq21:02:011
722499107,2cyclictest25809-21sed19:45:251
722499107,2cyclictest23191-21tr20:50:141
722499107,2cyclictest23191-21tr20:50:131
722499107,2cyclictest18041-21cat19:30:211
722099108,1cyclictest14479-21sed19:25:000
722099108,1cyclictest13534-21cut20:30:130
722099107,2cyclictest29923-21taskset21:03:190
722099107,2cyclictest29147-21cut19:55:140
722099107,2cyclictest27935-21grep19:50:210
722099107,2cyclictest26421-21ntp_kernel_pll_20:55:210
722099107,2cyclictest25230-21head19:45:190
722099100,9cyclictest234902sleep020:50:160
722099100,9cyclictest234902sleep020:50:150
72299997,1cyclictest26718-21dpkg19:50:113
72249997,1cyclictest21427-21cat20:45:171
72249996,2cyclictest3831-21taskset20:10:041
72249996,2cyclictest17421-21sed20:35:251
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional