You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-09 - 06:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Thu Jul 09, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19403210780,8sleep00-21swapper/019:07:260
1945129877,16sleep20-21swapper/219:08:042
1939629366,9sleep30-21swapper/319:07:213
1958428764,19sleep10-21swapper/119:09:481
1971599300,18cyclictest0-21swapper/221:40:082
1971399281,25cyclictest0-21swapper/100:30:201
1971399281,25cyclictest0-21swapper/100:30:201
1971599271,1cyclictest16282-21cron20:10:002
1971599270,27cyclictest0-21swapper/222:15:562
1971599250,25cyclictest0-21swapper/221:22:532
1971399250,22cyclictest0-21swapper/121:27:261
19715992419,4cyclictest15638-21kworker/u8:022:30:152
1971599231,5cyclictest0-21swapper/223:10:152
19715992312,10cyclictest0-21swapper/223:16:532
1971599230,23cyclictest0-21swapper/200:22:222
1971599230,20cyclictest0-21swapper/200:32:412
1971599230,20cyclictest0-21swapper/200:32:412
1971599230,1cyclictest0-21swapper/222:48:312
1971099230,1cyclictest0-21swapper/022:03:420
1971599228,7cyclictest31850-21apache200:02:092
1971599224,2cyclictest22493-21turbostat.cron21:35:002
1971599220,2cyclictest0-21swapper/200:06:452
1971599220,22cyclictest0-21swapper/223:04:152
1971599220,22cyclictest0-21swapper/200:19:512
1971599220,20cyclictest0-21swapper/221:28:232
1971599220,0cyclictest0-21swapper/222:42:352
1971599220,0cyclictest0-21swapper/222:42:352
19710992221,1cyclictest0-21swapper/022:30:380
1971599210,18cyclictest0-21swapper/221:58:242
1971599210,18cyclictest0-21swapper/221:47:482
1971399210,2cyclictest0-21swapper/122:11:221
1971399210,20cyclictest0-21swapper/123:48:341
1971599203,4cyclictest0-21swapper/200:25:272
1971599201,5cyclictest0-21swapper/222:38:152
1971599200,20cyclictest0-21swapper/223:55:342
19713992018,1cyclictest0-21swapper/121:47:321
1971399200,3cyclictest0-21swapper/123:05:441
1971599193,14cyclictest32-21ksoftirqd/220:35:132
1971599190,19cyclictest0-21swapper/222:13:292
19713991918,1cyclictest0-21swapper/122:33:181
1971399190,18cyclictest0-21swapper/123:31:351
1971599180,18cyclictest0-21swapper/200:11:492
1971599180,17cyclictest6682-21cat23:40:002
19710991818,0cyclictest0-21swapper/023:13:250
19710991817,1cyclictest0-21swapper/022:25:490
1971099180,16cyclictest0-21swapper/021:11:560
1971099180,0cyclictest0-21swapper/021:44:370
1971599178,2cyclictest16068-21chrt21:50:122
19715991710,6cyclictest22927-21wc21:15:192
1971599170,9cyclictest0-21swapper/223:25:212
1971599170,17cyclictest0-21swapper/223:40:212
1971399171,15cyclictest0-21swapper/120:20:131
1971399170,2cyclictest0-21swapper/123:28:031
1971399170,17cyclictest0-21swapper/123:57:451
1972199161,10cyclictest0-21swapper/323:20:133
1971599168,2cyclictest2598-21cat00:35:142
19715991614,2cyclictest32-21ksoftirqd/222:01:492
1971599160,1cyclictest0-21swapper/223:46:482
1971599160,16cyclictest0-21swapper/222:54:222
1971599160,16cyclictest0-21swapper/222:09:092
1971399167,6cyclictest38150irq/131-snd_hda00:28:311
19713991616,0cyclictest0-21swapper/123:01:331
19713991612,4cyclictest0-21swapper/122:25:421
1971399161,1cyclictest12924-21diskmemload00:11:521
1971099167,8cyclictest4066-21munin-run23:00:000
19710991613,2cyclictest13-21ksoftirqd/022:25:000
1971099160,16cyclictest0-21swapper/022:11:380
1972199159,5cyclictest6948-21cron21:45:003
1972199159,2cyclictest452-21systemd-logind23:17:043
19721991511,3cyclictest31600-21df20:40:133
1971599158,2cyclictest7484-21tail20:55:262
1971599158,2cyclictest7484-21tail20:55:262
1971599158,2cyclictest2769-21grep20:45:272
1971599158,2cyclictest23334-21sh21:35:132
1971599158,2cyclictest21042-21tr19:10:212
1971599157,2cyclictest21492-21grep21:15:012
1971599155,4cyclictest798-21ls20:45:002
19713991512,2cyclictest38150irq/131-snd_hda22:00:261
19713991511,4cyclictest38150irq/131-snd_hda23:10:341
1971399150,15cyclictest0-21swapper/123:15:381
1971399150,14cyclictest0-21swapper/121:30:371
1971399150,12cyclictest0-21swapper/122:35:121
1971099155,5cyclictest225-21systemd-journal20:17:160
1971099152,6cyclictest26740-21tr00:30:140
1971099152,6cyclictest26740-21tr00:30:140
1971099150,1cyclictest0-21swapper/019:10:130
1972199148,5cyclictest1213-21cron22:20:003
19721991414,0cyclictest0-21swapper/322:44:033
19721991414,0cyclictest0-21swapper/322:44:033
1972199141,2cyclictest0-21swapper/322:08:423
19721991411,2cyclictest25320-21head21:55:163
1971599147,2cyclictest23730-21wc23:30:182
1971599146,2cyclictest6755-21cron19:50:002
19715991410,3cyclictest5101-21tail20:50:262
1971599140,13cyclictest8998-21chrt21:00:152
19713991413,1cyclictest25-21ksoftirqd/122:56:541
19713991413,1cyclictest25-21ksoftirqd/100:01:301
19713991411,2cyclictest9324-21cat19:55:001
19713991410,2cyclictest31313-21tr20:40:131
1971399140,1cyclictest0-21swapper/100:20:011
1971099146,7cyclictest22504-21munin-run21:35:000
1971099145,2cyclictest15505-21ssh23:45:130
1971099140,1cyclictest0-21swapper/021:51:390
195272140,0sleep3381rcuc/321:51:333
1972199139,3cyclictest16070-21ssh21:10:563
19721991310,2cyclictest4168-21cron19:45:003
19721991310,2cyclictest32269-21head21:20:273
19721991310,2cyclictest32131-21cut19:35:123
19721991310,2cyclictest26627-21perf22:35:003
19721991310,2cyclictest24708-21ssh21:35:243
19721991310,2cyclictest24503-21wc19:20:013
19721991310,2cyclictest18204-21cron22:30:003
19721991310,2cyclictest12924-21diskmemload23:44:473
1971599139,2cyclictest3672-21grep19:40:262
19715991310,2cyclictest18157-21grep20:10:262
1971599130,13cyclictest0-21swapper/222:23:322
1971399139,3cyclictest5250-21ssh00:17:001
1971399139,3cyclictest483-21wc19:35:161
1971399139,3cyclictest21935-21tr21:15:141
19713991313,0cyclictest0-21swapper/123:24:001
19713991311,2cyclictest0-21swapper/122:54:501
19713991311,1cyclictest12924-21diskmemload22:16:161
19713991311,1cyclictest12924-21diskmemload21:36:461
19713991310,2cyclictest9667-21tr19:55:131
19713991310,2cyclictest4889-21tr19:45:121
19713991310,2cyclictest30991-21wc19:30:271
19713991310,2cyclictest29761-21grep20:35:191
19713991310,2cyclictest20028-21grep20:15:181
1971399130,12cyclictest0-21swapper/123:42:141
1971099139,3cyclictest22931-21sed20:20:260
19710991312,1cyclictest13-21ksoftirqd/023:32:000
19710991310,2cyclictest6046-21grep20:55:140
19710991310,2cyclictest6046-21grep20:55:140
19710991310,2cyclictest22348-21df21:15:150
19710991310,2cyclictest1297-21tr20:45:140
1971099130,0cyclictest0-21swapper/021:39:370
1972199129,2cyclictest7311-21tr19:50:143
1972199129,2cyclictest6753-21cron19:50:003
1972199129,2cyclictest25753-21cat19:20:183
1972199129,2cyclictest24631-21grep20:25:163
1972199129,2cyclictest23154-21grep23:30:153
1972199129,2cyclictest22804-21cat20:20:253
1972199129,2cyclictest20336-21rm23:47:393
1972199129,2cyclictest1806-21basename22:20:123
1972199129,2cyclictest17910-21strings00:05:193
1972199129,2cyclictest17778-21grep20:10:213
1972199129,2cyclictest15272-21grep20:05:203
1972199129,2cyclictest1201-21cut22:00:173
1972199128,3cyclictest7034-21grep21:25:163
1972199128,3cyclictest39-21ksoftirqd/323:59:493
1972199128,3cyclictest10540-21munin-run21:05:003
1972199125,3cyclictest225-21systemd-journal00:26:163
1972199124,6cyclictest0-21swapper/321:49:183
1971599129,2cyclictest24643-21grep20:25:162
1971599129,2cyclictest19410-21grep20:15:142
1971599128,3cyclictest9650-21tr19:55:132
1971599124,5cyclictest0-21swapper/223:20:262
1971599120,11cyclictest25238-21sed23:50:212
1971399129,2cyclictest4630-21grep22:40:201
1971399129,2cyclictest4630-21grep22:40:201
1971399129,2cyclictest4166-21head20:50:171
1971399129,2cyclictest32415-21cron22:00:001
1971399129,2cyclictest24661-21head23:50:161
1971399129,2cyclictest13978-21wc20:05:001
1971399129,2cyclictest12120-21wc21:05:211
1971399128,3cyclictest31850-21apache221:54:311
1971399128,3cyclictest27942-21grep20:30:271
1971399127,4cyclictest3097-21taskset20:49:071
1971399120,12cyclictest0-21swapper/120:55:231
1971399120,12cyclictest0-21swapper/120:55:221
1971399120,0cyclictest0-21swapper/121:13:141
1971099129,2cyclictest9692-21uname21:00:220
1971099129,2cyclictest31054-21sed21:20:170
1971099129,2cyclictest23001-21grep19:15:170
1971099129,2cyclictest10282-21ls22:05:210
1971099129,2cyclictest10248-21grep00:00:260
1971099127,3cyclictest18369-21sed00:05:250
1971099120,0cyclictest0-21swapper/019:20:120
191842120,1sleep20-21swapper/223:08:432
1972199119,1cyclictest24112-21cut23:50:143
1972199119,1cyclictest12470-21grep21:05:253
1972199118,2cyclictest9589-21ssh23:02:463
1972199118,2cyclictest7055-21grep20:55:213
1972199118,2cyclictest7055-21grep20:55:213
1972199118,2cyclictest5186-21rm00:16:583
1972199118,2cyclictest3683-21tr20:50:143
1972199118,2cyclictest32068-21ssh00:33:173
1972199118,2cyclictest32068-21ssh00:33:173
1972199118,2cyclictest30272-21grep20:35:253
1972199118,2cyclictest29074-21expr22:55:153
1972199118,2cyclictest21039-21head22:50:183
1972199118,2cyclictest20980-21sort19:10:203
1972199118,2cyclictest20114-21ssh22:11:053
1972199118,2cyclictest17529-21rm22:48:213
1972199118,2cyclictest14516-21rm21:30:093
1972199118,2cyclictest13899-21cron20:05:003
1972199118,2cyclictest13526-21ssh00:21:543
1972199118,2cyclictest13026-21ssh23:05:133
1972199118,2cyclictest11808-21ssh00:01:353
1972199118,2cyclictest10237-21latency_hist00:40:003
1972199115,4cyclictest12924-21diskmemload00:10:213
1972199113,2cyclictest22451-21diskstats21:15:163
1972199110,10cyclictest5076-21ssh23:38:513
1972199110,10cyclictest26367-21cut20:30:143
1972199110,10cyclictest22034-21fschecks_count23:10:153
1971599118,2cyclictest8815-21chrt19:50:592
1971599118,2cyclictest32510-21ssh22:56:552
1971599118,2cyclictest28056-21taskset20:30:482
1971599118,2cyclictest26892-21cat19:25:012
1971599118,2cyclictest22787-21df19:15:152
1971599118,2cyclictest14829-21ssh22:27:232
1971599117,2cyclictest13098-21cat21:10:012
1971399118,2cyclictest32514-21cut23:35:221
1971399118,2cyclictest30231-21ssh21:20:131
1971399118,2cyclictest2817-21sed19:40:171
1971399118,2cyclictest27783-21head19:25:161
1971399118,2cyclictest24669-21sed20:25:161
1971399118,2cyclictest21404-21sort19:10:261
1971399118,2cyclictest15852-21ssh22:47:031
1971399118,2cyclictest11856-21ssh22:06:091
1971399117,2cyclictest9131-21tr21:00:161
19713991110,1cyclictest0-21swapper/121:40:051
1971399110,10cyclictest17756-21cut20:10:211
1971099118,2cyclictest8360-21awk19:50:210
1971099118,2cyclictest6068-21grep23:20:140
1971099118,2cyclictest4303-21basename22:40:180
1971099118,2cyclictest4303-21basename22:40:180
1971099118,2cyclictest32499-21cut19:35:130
1971099118,2cyclictest30096-21sort19:30:170
1971099118,2cyclictest29374-21ssh22:16:440
1971099118,2cyclictest28666-21ssh23:52:380
1971099118,2cyclictest27157-21rm22:35:060
1971099118,2cyclictest26570-21ssh00:10:200
1971099118,2cyclictest26105-21ssh21:55:230
1971099118,2cyclictest20976-21grep22:50:170
1971099118,2cyclictest18884-21ssh23:27:300
1971099118,2cyclictest18301-21expr00:25:150
1971099118,2cyclictest1789-21grep23:55:250
1971099118,2cyclictest14969-21awk23:45:000
1971099118,2cyclictest14793-21ssh00:22:530
1971099118,2cyclictest13269-21ssh23:05:150
1971099118,2cyclictest12039-21grep22:45:120
1971099118,2cyclictest10457-21ssh21:46:110
1971099116,4cyclictest2072-21apt-get19:40:130
1971099112,2cyclictest0-21swapper/023:00:440
1971099111,1cyclictest29806-21apt-get23:15:140
1971099110,6cyclictest28415-21perf20:35:000
1971099110,10cyclictest6526-21tr21:25:130
1971099110,10cyclictest0-21swapper/023:35:130
1972199108,1cyclictest30577-21ntpq19:30:213
1972199108,1cyclictest27707-21expr22:35:143
1972199108,1cyclictest27314-21grep19:25:133
1972199108,1cyclictest1277-21grep20:45:143
1972199107,2cyclictest9799-21cat19:55:133
1972199100,9cyclictest14829-21rm23:25:143
1971599108,1cyclictest29400-21seq19:30:022
1971599108,1cyclictest21567-21expr20:20:142
1971599107,2cyclictest27172-21tr19:25:132
1971599107,2cyclictest1587-21perf19:40:002
1971599107,2cyclictest12506-21grep20:00:192
1971399108,1cyclictest1842-21cut22:20:131
1971399107,2cyclictest2459-21cut00:35:131
1971399107,2cyclictest24540-21cat19:20:011
1971399107,2cyclictest18868-21sh00:05:341
1971099108,1cyclictest9374-21basename19:55:100
1971099108,1cyclictest6451-21taskset19:46:070
1971099108,1cyclictest32270-21ntpq20:40:190
1971099108,1cyclictest14610-21expr20:05:150
1971099108,1cyclictest11798-21apache_processe20:00:140
1971099108,1cyclictest1027-21ssh00:15:040
1971099107,2cyclictest29037-21taskset19:28:190
1971099104,3cyclictest2896-21df00:35:150
1971099102,7cyclictest0-21swapper/020:35:130
197139990,4cyclictest14881-21grep20:05:171
197109998,1cyclictest3915-21expr20:50:150
197109997,1cyclictest238752sleep020:25:070
197109997,1cyclictest11803-21cat21:05:170
197109996,2cyclictest17707-21taskset20:10:210
197219985,2cyclictest19669-21grep20:15:163
197139981,5cyclictest25-21ksoftirqd/119:20:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional