You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-18 - 08:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 18, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31957210090,6sleep00-21swapper/019:07:500
3203129165,8sleep20-21swapper/219:08:482
3189528564,6sleep10-21swapper/119:07:031
3197028468,12sleep30-21swapper/319:08:023
126542740,1sleep232-21ksoftirqd/222:37:532
32232992725,1cyclictest12819-21apache221:35:591
32232992413,11cyclictest25-21ksoftirqd/121:24:231
3223299230,1cyclictest0-21swapper/121:14:041
32237992222,0cyclictest0-21swapper/322:29:113
3223799222,15cyclictest31002-21apache_accesses00:25:103
3223799220,3cyclictest0-21swapper/322:59:073
3223799220,2cyclictest141rcu_preempt23:56:083
3223799220,21cyclictest0-21swapper/322:35:363
3223799211,3cyclictest0-21swapper/321:46:433
3223799210,2cyclictest0-21swapper/322:03:433
3223799210,2cyclictest0-21swapper/300:12:393
32232992118,2cyclictest12807-21apache221:16:341
3223299211,13cyclictest0-21swapper/122:20:121
3223299210,2cyclictest0-21swapper/123:20:201
32237992019,1cyclictest0-21swapper/323:19:373
32234992017,2cyclictest0-21swapper/223:05:192
3223499200,17cyclictest0-21swapper/222:10:452
3223299201,15cyclictest0-21swapper/121:55:141
3223799180,4cyclictest0-21swapper/321:23:183
3223799180,18cyclictest0-21swapper/323:28:483
3223499180,18cyclictest0-21swapper/223:21:262
32232991815,3cyclictest25-21ksoftirqd/123:50:141
3223299181,16cyclictest28967-21kworker/1:200:05:491
3223299180,17cyclictest0-21swapper/121:42:301
32227991815,2cyclictest13-21ksoftirqd/022:20:030
3222799180,1cyclictest0-21swapper/021:39:200
3222799180,0cyclictest0-21swapper/023:04:190
3223799179,8cyclictest0-21swapper/322:41:403
32234991714,1cyclictest141rcu_preempt23:01:072
3223499170,1cyclictest0-21swapper/200:38:002
32232991715,1cyclictest0-21swapper/100:30:171
3223299170,16cyclictest25447-21diskmemload23:05:191
3223799165,9cyclictest23813-21cron20:00:003
3223799160,1cyclictest0-21swapper/320:55:143
32234991615,1cyclictest0-21swapper/200:30:242
3223499160,16cyclictest0-21swapper/222:16:222
3223499160,14cyclictest0-21swapper/222:00:222
3223299166,5cyclictest225-21systemd-journal20:50:001
32237991512,2cyclictest19409-21kworker/u8:300:03:023
3223799150,1cyclictest0-21swapper/323:53:463
32234991515,0cyclictest0-21swapper/221:18:092
3223499151,12cyclictest24484-21ntp_states22:25:212
3222799158,7cyclictest16193-21munin-node23:34:540
32227991510,2cyclictest32222-21cyclictest21:24:230
32237991414,0cyclictest0-21swapper/300:38:373
32237991411,2cyclictest23050-21cron21:05:013
3223799140,12cyclictest0-21swapper/321:56:063
3223499140,1cyclictest0-21swapper/200:27:472
3223499140,14cyclictest0-21swapper/222:41:262
3223499140,14cyclictest0-21swapper/222:31:002
32232991414,0cyclictest0-21swapper/120:50:141
3223299141,2cyclictest225-21systemd-journal23:12:181
32232991410,2cyclictest24828-21rm22:45:141
3223799139,3cyclictest3211-21wc20:20:263
3223799139,3cyclictest11056-21cron23:35:003
3223799139,2cyclictest602-21grep22:50:143
32237991313,0cyclictest0-21swapper/300:34:493
32237991312,1cyclictest39-21ksoftirqd/323:21:473
32237991310,2cyclictest25063-21grep20:00:153
32237991310,2cyclictest2124-21cron19:15:003
32237991310,2cyclictest13544-21apt20:45:003
3223799130,13cyclictest0-21swapper/321:11:113
3223499139,2cyclictest4232-21sh21:55:002
3223499139,2cyclictest4229-21tr23:30:192
32234991312,1cyclictest0-21swapper/223:45:412
32234991310,2cyclictest22812-21sh00:20:142
32234991310,2cyclictest22156-21head00:00:202
32234991310,2cyclictest1638-21sed19:10:252
32234991310,2cyclictest1219-21grep22:50:162
3223499130,13cyclictest0-21swapper/223:26:382
3223499130,12cyclictest0-21swapper/223:35:132
3223299139,3cyclictest3976-21tail23:30:171
3223299139,3cyclictest3948-21sh23:50:001
3223299139,3cyclictest18564-21basename20:55:141
3223299139,3cyclictest13482-21df23:55:141
3223299134,7cyclictest0-21swapper/100:00:231
3223299132,9cyclictest25447-21diskmemload21:34:501
32232991311,2cyclictest0-21swapper/122:36:401
32232991310,2cyclictest5540-21grep19:20:171
32232991310,2cyclictest31758-21cat22:10:261
32232991310,2cyclictest22957-21tr00:20:141
32232991310,2cyclictest1781-21grep20:20:141
32232991310,2cyclictest11881-21turbostat19:35:001
3223299130,1cyclictest25447-21diskmemload22:41:041
3223299130,1cyclictest141rcu_preempt22:17:581
3222799139,3cyclictest7955-21unixbench_singl22:15:260
3222799132,7cyclictest0-21swapper/021:15:130
3222799131,3cyclictest0-21swapper/023:36:590
32227991310,2cyclictest787-21ssh22:11:210
32227991310,2cyclictest4372-21ssh00:28:200
32227991310,2cyclictest20518-21wc21:25:260
32227991310,2cyclictest17920-21ssh22:40:260
32227991310,2cyclictest13934-21basename20:45:140
3222799130,13cyclictest0-21swapper/021:32:090
3222799130,12cyclictest28202-21turbostat.cron21:50:000
3222799130,12cyclictest0-21swapper/022:58:230
3223799129,2cyclictest9267-21tr20:35:153
3223799129,2cyclictest6866-21tail22:15:163
3223799129,2cyclictest4718-21df20:25:163
3223799129,2cyclictest30488-21grep00:05:173
3223799129,2cyclictest29555-21df20:10:133
3223799129,2cyclictest13323-21tr19:35:193
3223799124,6cyclictest25447-21diskmemload23:00:143
3223799122,9cyclictest22997-21grep22:05:203
3223499129,2cyclictest5995-21cut19:20:202
3223499129,2cyclictest5128-21ssh23:50:172
3223499129,2cyclictest4470-21grep20:25:152
3223499129,2cyclictest4171-21grep21:35:162
3223499129,2cyclictest28280-21sh21:30:192
3223499129,2cyclictest26942-21tr20:05:142
3223499129,2cyclictest25942-21sed20:00:252
3223499129,2cyclictest22418-21grep19:55:182
3223499129,2cyclictest21668-21head21:00:172
3223499129,2cyclictest20041-21grep19:50:162
3223499129,2cyclictest19120-21grep21:25:152
3223499129,2cyclictest17711-21sed19:45:172
3223499129,2cyclictest14875-21tr19:40:132
3223499129,2cyclictest14117-21head23:55:192
3223499129,2cyclictest13893-21head19:35:252
3223499121,10cyclictest10350-21chrt21:58:132
32234991210,1cyclictest0-21swapper/222:07:572
3223299129,2cyclictest32474-21grep19:10:131
3223299129,2cyclictest32050-21ssh22:30:141
3223299129,2cyclictest30736-21grep20:10:241
3223299129,2cyclictest29138-21sed23:25:281
3223299129,2cyclictest26746-21cat21:30:001
3223299129,2cyclictest21935-21sed23:40:261
3223299128,1cyclictest776-21ssh00:25:321
3223299127,3cyclictest15457-21apt-get00:35:131
3223299127,2cyclictest2818-21ssh22:50:431
3223299121,5cyclictest23969-21date20:00:001
3223299121,2cyclictest225-21systemd-journal23:20:001
3223299121,1cyclictest17643-21cpuspeed_turbos23:00:141
3223299120,8cyclictest0-21swapper/119:25:121
3223299120,10cyclictest25447-21diskmemload21:52:181
3222799129,2cyclictest8622-21sendmail_mailtr00:30:220
3222799129,2cyclictest6672-21tr20:30:130
3222799129,2cyclictest5275-21turbostat.cron00:10:010
3222799129,2cyclictest27183-21df20:05:140
3222799129,2cyclictest236192sleep023:22:380
3222799129,2cyclictest18805-21grep20:55:150
3222799129,2cyclictest15778-21ls19:40:200
3222799129,2cyclictest13843-21sed19:35:250
3222799125,6cyclictest31057-21munin-run22:30:000
3222799124,3cyclictest225-21systemd-journal00:05:000
32227991210,1cyclictest5499-21grep19:20:160
3222799120,11cyclictest6432-21sed23:50:280
3223799119,1cyclictest28967-21sed23:45:143
3223799119,1cyclictest23460-21tr00:20:183
3223799118,2cyclictest9161-21seq19:28:373
3223799118,2cyclictest686-21awk22:30:223
3223799118,2cyclictest5547-21ssh21:16:403
3223799118,2cyclictest5201-21expr19:20:133
3223799118,2cyclictest4264-21hwlatdetect21:35:183
3223799118,2cyclictest4159-21seq19:15:393
3223799118,2cyclictest380-21rm22:11:033
3223799118,2cyclictest31103-21sh21:50:433
3223799118,2cyclictest30339-21ssh22:48:273
3223799118,2cyclictest29494-21sh23:06:483
3223799118,2cyclictest27254-21ssh21:30:133
3223799118,2cyclictest23949-21sort21:05:173
3223799118,2cyclictest23805-21ssh21:27:453
3223799118,2cyclictest2372-21sh23:10:163
3223799118,2cyclictest19582-21tr19:50:143
3223799118,2cyclictest17943-21ssh23:38:463
3223799118,2cyclictest16326-21sensors22:20:223
3223799118,2cyclictest15085-21ssh21:41:223
3223799118,2cyclictest14181-21expr20:45:153
3223799118,2cyclictest10035-21grep19:30:143
3223799117,3cyclictest20874-21head23:40:173
3223799117,3cyclictest16727-21cat19:45:013
3223799116,4cyclictest15135-21head00:15:183
3223499118,2cyclictest8294-21ntpq19:25:192
3223499118,2cyclictest6648-21grep00:10:212
3223499118,2cyclictest4634-21ssh23:11:082
3223499118,2cyclictest29746-21hostname00:05:142
3223499118,2cyclictest29297-21cut20:10:132
3223499118,2cyclictest24837-21expr22:45:142
3223499118,2cyclictest24053-21grep21:05:172
3223499118,2cyclictest21475-21grep21:45:192
3223499118,2cyclictest20156-21cut23:40:132
3223499118,2cyclictest1878-21tr20:20:142
3223499118,2cyclictest14530-21tail20:45:172
3223499118,2cyclictest12972-21sort21:40:182
3223499118,2cyclictest12314-21grep21:20:262
3223499118,2cyclictest10402-21cut19:30:172
3223499117,3cyclictest15893-21wc22:20:182
3223499115,4cyclictest10286-21apt-get23:15:122
3223499110,4cyclictest0-21swapper/221:14:352
3223299119,1cyclictest0-21swapper/119:40:121
3223299118,2cyclictest9666-21cut22:55:181
3223299118,2cyclictest32233-21sed20:15:171
3223299118,2cyclictest25043-21head20:00:151
3223299118,2cyclictest21144-21ls22:05:001
3223299118,2cyclictest21027-21cut21:00:141
3223299118,2cyclictest20331-21ssh21:45:131
3223299118,2cyclictest17718-21sed19:45:171
3223299118,2cyclictest14294-21cron19:40:001
3223299118,2cyclictest12729-21cat23:35:191
3223299117,3cyclictest21436-21ssh22:05:041
3223299113,4cyclictest225-21systemd-journal20:37:301
3223299112,4cyclictest0-21swapper/100:15:141
32232991110,1cyclictest0-21swapper/119:50:151
3222799119,1cyclictest28785-21grep23:25:240
3222799119,1cyclictest18262-21sleep23:20:000
3222799119,1cyclictest10406-21grep19:30:160
3222799118,2cyclictest7767-21grep22:35:130
3222799118,2cyclictest4368-21tr20:25:150
3222799118,2cyclictest27916-21grep21:10:260
3222799118,2cyclictest26169-21ls23:05:140
3222799118,2cyclictest26072-21cat22:45:250
3222799118,2cyclictest22809-21grep19:55:220
3222799118,2cyclictest18060-21grep19:45:200
3222799118,2cyclictest15999-21cat00:15:260
3222799118,2cyclictest14843-21ssh22:00:240
3222799118,2cyclictest1458-21ssh22:30:400
3222799118,2cyclictest13670-21grep23:55:160
3222799118,2cyclictest13133-21sed20:40:270
3222799118,2cyclictest10321-21ssh00:12:200
3222799118,1cyclictest0-21swapper/021:58:240
3222799110,10cyclictest280792sleep022:08:530
3223799108,1cyclictest6578-21basename20:30:103
3223799108,1cyclictest17644-21cat20:50:273
3223799108,1cyclictest17414-21expr19:45:153
3223799107,2cyclictest31713-21grep20:15:163
3223799105,2cyclictest225-21systemd-journal20:10:003
3223499109,1cyclictest0-21swapper/222:56:322
3223499108,1cyclictest6568-21taskset20:30:092
3223499108,1cyclictest32415-21sort20:15:202
3223499108,1cyclictest16892-21grep20:50:182
3223499108,1cyclictest13383-21taskset20:43:092
3223499107,2cyclictest14321-21chrt00:15:142
3223499101,2cyclictest225-21systemd-journal19:16:322
3223299108,1cyclictest6788-21grep20:30:131
3223299108,1cyclictest26840-21egrep20:05:151
3223299108,1cyclictest25245-21chrt21:07:431
3223299107,2cyclictest5961-21basename00:10:181
3223299107,2cyclictest23135-21ssh22:25:121
3222799108,2cyclictest22235-21sh23:40:340
3222799108,1cyclictest91352sleep019:28:170
3222799108,1cyclictest8072sleep020:16:010
3222799108,1cyclictest32700-21sh22:50:070
3222799108,1cyclictest3188-21wc23:10:220
3222799108,1cyclictest29848-21grep20:10:150
3222799108,1cyclictest285392sleep021:50:060
3222799108,1cyclictest22454-21ssh00:20:030
3222799108,1cyclictest2126-21df20:20:150
3222799108,1cyclictest19823-21df19:50:150
3222799108,1cyclictest17401-21grep20:50:230
3222799107,2cyclictest3664-21wc19:15:240
3222799107,2cyclictest28330-21rm23:45:080
3222799107,2cyclictest21002-21grep21:00:140
3222799107,2cyclictest13409-21ssh21:40:220
3222799102,4cyclictest225-21systemd-journal21:10:000
322349997,1cyclictest10687-21head20:35:262
322349994,2cyclictest452-21systemd-logind20:55:232
322329997,1cyclictest4164-21sed20:25:011
322329997,1cyclictest11672-21cat20:40:151
322329993,2cyclictest225-21systemd-journal19:16:211
322279997,1cyclictest24461-21grep20:00:130
322279997,1cyclictest15227-21ssh00:35:030
322279997,1cyclictest0-21swapper/020:35:140
322279987,1cyclictest393-21cat19:10:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional