You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-29 - 12:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Mar 29, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1374229481,9sleep20-21swapper/219:09:082
1358828976,9sleep00-21swapper/019:07:090
1366028370,9sleep30-21swapper/319:08:053
1364828164,13sleep10-21swapper/119:07:551
1390099280,28cyclictest0-21swapper/223:21:442
1389899280,28cyclictest0-21swapper/123:22:411
1390399265,10cyclictest225-21systemd-journal00:09:243
13900992625,1cyclictest32-21ksoftirqd/222:44:192
1390099250,23cyclictest0-21swapper/222:23:432
1389899243,9cyclictest0-21swapper/119:20:131
13900992321,2cyclictest32-21ksoftirqd/200:06:282
13900992218,3cyclictest4243-21cron21:05:002
13900992217,3cyclictest32-21ksoftirqd/221:43:562
1390099220,22cyclictest0-21swapper/221:25:202
1389899221,20cyclictest6624-21diskmemload00:30:041
13893992213,5cyclictest13-21ksoftirqd/022:15:210
1390399210,20cyclictest0-21swapper/322:00:133
1390099200,20cyclictest0-21swapper/223:53:232
1389899201,18cyclictest0-21swapper/123:56:421
1389399208,6cyclictest20928-21perf00:30:010
13893992020,0cyclictest0-21swapper/000:13:120
227882190,0sleep10-21swapper/123:14:251
1390099191,18cyclictest0-21swapper/222:54:342
1390099190,18cyclictest0-21swapper/222:27:372
1390099190,10cyclictest0-21swapper/223:32:182
1389899190,2cyclictest0-21swapper/123:26:181
1390099183,12cyclictest311rcuc/200:18:052
1390099181,3cyclictest0-21swapper/223:13:302
13900991811,7cyclictest32-21ksoftirqd/223:39:082
13898991817,1cyclictest0-21swapper/123:07:541
1389399184,7cyclictest21028-21munin-run19:25:000
76702170,2sleep30-21swapper/321:29:463
1390099172,12cyclictest0-21swapper/222:33:212
13900991717,0cyclictest0-21swapper/221:34:182
13900991715,1cyclictest13529-21cron22:50:002
13900991711,4cyclictest311rcuc/221:54:052
1389899170,17cyclictest0-21swapper/121:38:051
1390399163,13cyclictest15579-21kworker/u8:000:17:053
1390399160,0cyclictest0-21swapper/322:36:023
1390099169,1cyclictest32-21ksoftirqd/222:35:312
1390099165,3cyclictest32-21ksoftirqd/221:18:382
13900991614,1cyclictest0-21swapper/223:42:252
1390099161,14cyclictest30988-21apt-get23:00:132
1390099160,16cyclictest0-21swapper/222:10:092
1390099160,14cyclictest0-21swapper/200:25:332
1389899162,5cyclictest141rcu_preempt23:30:201
1389399168,5cyclictest21385-21ssh00:30:030
1389399165,6cyclictest6464-21munin-run23:05:000
13893991613,1cyclictest0-21swapper/022:50:130
1389399160,0cyclictest0-21swapper/022:20:040
184812150,0sleep00-21swapper/000:09:190
13903991511,2cyclictest6657-21grep21:10:003
1390399150,15cyclictest0-21swapper/323:05:393
1390099154,9cyclictest32183-21taskset00:35:332
1390099152,1cyclictest141rcu_preempt22:57:432
13900991513,1cyclictest6624-21diskmemload00:14:572
1390099150,1cyclictest0-21swapper/222:18:092
1390099150,13cyclictest0-21swapper/222:04:062
1389899152,1cyclictest6624-21diskmemload00:29:131
1389899151,11cyclictest0-21swapper/122:00:211
1389399151,1cyclictest6624-21diskmemload22:43:160
13903991413,1cyclictest19470-21kworker/3:022:51:033
13903991410,4cyclictest6624-21diskmemload23:32:183
1390399140,13cyclictest0-21swapper/323:11:433
1390099148,2cyclictest225-21systemd-journal19:35:002
1390099146,8cyclictest32-21ksoftirqd/200:32:042
13900991412,1cyclictest7195-21apt-get23:05:132
13900991410,3cyclictest16743-21grep20:20:272
13900991410,3cyclictest11034-21wc00:05:002
1389899140,14cyclictest0-21swapper/122:07:081
1389899140,13cyclictest6624-21diskmemload21:26:121
1389899140,13cyclictest0-21swapper/122:15:131
1389899140,11cyclictest38150irq/131-snd_hda19:55:151
1389399145,5cyclictest27016-21apache_processe23:55:140
1389399142,6cyclictest10985-21perf00:05:000
1390399139,3cyclictest26484-21sed20:40:273
1390399139,3cyclictest24536-21cron20:40:003
1390399139,2cyclictest25680-21grep23:35:143
13903991312,1cyclictest19470-21kworker/3:021:24:333
13903991310,2cyclictest28829-21grep20:45:263
13903991310,2cyclictest23415-21cat19:30:003
13903991310,2cyclictest16496-21cat21:35:003
1390399130,12cyclictest0-21swapper/320:50:133
1390099138,4cyclictest32-21ksoftirqd/221:45:192
1390099138,3cyclictest1830-21ls21:00:002
1390099136,1cyclictest141rcu_preempt21:59:372
1390099135,5cyclictest32-21ksoftirqd/220:00:192
1390099133,6cyclictest32-21ksoftirqd/221:24:282
1390099132,8cyclictest32-21ksoftirqd/223:47:032
13900991310,2cyclictest27086-21cat20:45:012
1390099130,12cyclictest25940-21grep23:15:262
1390099130,12cyclictest24892-21cat21:40:002
1389899139,3cyclictest11894-21grep21:50:261
1389899138,4cyclictest24318-21grep23:15:141
1389899131,11cyclictest12667-21/usr/sbin/munin20:15:171
13898991310,2cyclictest31791-21grep23:00:171
13898991310,2cyclictest27954-21grep20:45:171
13898991310,2cyclictest25394-21grep21:20:211
13898991310,2cyclictest21562-21tr19:25:151
1389399134,2cyclictest141rcu_preempt21:19:000
13893991310,2cyclictest8013-21grep21:10:170
13893991310,2cyclictest4748-21sed19:55:270
13893991310,2cyclictest27830-21grep20:45:170
13893991310,2cyclictest11182-21grep21:50:200
1389399130,1cyclictest0-21swapper/022:10:360
1389399130,12cyclictest23697-21wc20:35:210
1390399129,2cyclictest9567-21head23:25:163
1390399129,2cyclictest39-21ksoftirqd/320:20:113
1390399129,2cyclictest31460-21grep00:35:233
1390399129,2cyclictest31438-21grep22:40:263
1390399129,2cyclictest2867-21tr23:40:183
1390399129,2cyclictest26760-21ls21:40:213
1390399129,2cyclictest21675-21grep20:30:253
1390399129,2cyclictest19796-21ssh00:10:043
1390399129,2cyclictest19253-21ssh22:15:003
1390399129,2cyclictest184502sleep321:35:223
1390399129,2cyclictest1767-21wc23:20:233
1390399129,2cyclictest14572-21grep22:30:253
1390399129,2cyclictest12386-21chrt23:45:373
1390399129,2cyclictest12219-21ssh21:50:383
1390399128,3cyclictest32139-21cat23:20:003
1390399128,3cyclictest23151-21rm22:55:163
1390399128,3cyclictest22685-21sed00:30:193
1390399128,3cyclictest15076-21cron21:15:003
1390399128,3cyclictest14333-21sed20:15:273
1390399128,1cyclictest7827-21ssh00:21:263
1390399127,1cyclictest0-21swapper/322:17:003
13903991211,1cyclictest19470-21kworker/3:021:17:233
13903991210,2cyclictest39-21ksoftirqd/323:00:533
1390399120,11cyclictest2492-21run-parts00:00:003
1390099129,2cyclictest6053-21wc00:20:232
1390099129,2cyclictest28689-21grep19:40:142
1390099129,2cyclictest27618-21ssh23:55:172
1390099129,2cyclictest21290-21tr20:30:212
1390099129,2cyclictest21005-21cron19:25:012
1390099128,3cyclictest12921-21tr20:15:162
1390099121,8cyclictest0-21swapper/221:11:132
1390099120,11cyclictest5812-21chrt22:06:102
1389899129,2cyclictest7677-21cron20:05:001
1389899129,2cyclictest31471-21head00:35:231
1389899129,2cyclictest305112sleep122:20:401
1389899129,2cyclictest28665-21grep19:40:141
1389899129,2cyclictest26288-21dump-pmu-power22:00:001
1389899129,2cyclictest2491-21grep23:40:161
1389899129,2cyclictest23463-21sh21:20:001
1389899129,2cyclictest17047-21taskset19:15:151
1389899129,2cyclictest14037-21sed22:30:201
1389899129,2cyclictest12420-21ls00:25:001
1389899128,3cyclictest25715-21tr21:40:141
1389899122,1cyclictest26494-21ssh22:56:581
13898991210,1cyclictest591-21ssh22:41:331
13898991210,1cyclictest3309-21grep21:45:261
1389899120,11cyclictest156782sleep120:20:151
1389399129,2cyclictest9446-21grep23:25:160
1389399129,2cyclictest8007-21cron21:30:000
1389399129,2cyclictest5964-21grep00:20:220
1389399129,2cyclictest32660-21grep20:55:180
1389399129,2cyclictest28671-21grep19:40:140
1389399129,2cyclictest27591-21grep23:35:260
1389399129,2cyclictest23795-21tr22:55:210
1389399129,2cyclictest19118-21grep23:30:260
1389399129,2cyclictest17494-21cat23:10:250
1389399129,2cyclictest16520-21cat20:20:260
1389399129,2cyclictest14242-21grep22:30:210
1389399128,3cyclictest11029-21grep20:10:190
1389399128,3cyclictest11029-21grep20:10:180
1389399128,2cyclictest25315-21uname23:15:200
1389399122,1cyclictest7954-21apt-get20:05:130
1389399121,4cyclictest141rcu_preempt20:40:170
1389399120,5cyclictest0-21swapper/021:40:040
1389399120,11cyclictest14865-21cat20:20:000
1390399118,2cyclictest6779-21sh22:25:533
1390399118,2cyclictest4645-21cat19:55:273
1390399118,2cyclictest448-21basename19:50:003
1390399118,2cyclictest3330-21tr00:00:153
1390399118,2cyclictest3001-21grep21:45:223
1390399118,2cyclictest27495-21cat19:35:323
1390399118,2cyclictest25772-21ls19:35:003
1390399118,2cyclictest2387-21tr21:00:143
1390399118,2cyclictest227932sleep321:57:183
1390399118,2cyclictest2208-21ssh22:23:513
1390399118,2cyclictest19938-21grep23:50:203
1390399118,2cyclictest15481-21ssh00:25:513
1390399118,2cyclictest1233-21grep19:50:153
1390399118,2cyclictest1152-21sed20:55:273
1390399118,2cyclictest10873-21basename22:10:003
1390399118,2cyclictest10774-21ssh22:47:523
1390099118,2cyclictest9583-21grep23:25:162
1390099118,2cyclictest4793-21tr21:05:142
1390099118,2cyclictest3829-21grep19:55:192
1390099118,2cyclictest24361-21taskset20:37:462
1390099118,2cyclictest18429-21chrt20:25:162
1390099118,2cyclictest14395-21expr19:10:142
1390099115,6cyclictest32-21ksoftirqd/219:35:182
1390099115,5cyclictest0-21swapper/220:45:132
1390099114,5cyclictest32-21ksoftirqd/219:52:302
1390099114,5cyclictest32-21ksoftirqd/219:26:372
1390099114,5cyclictest32-21ksoftirqd/219:16:082
1389899118,2cyclictest7543-21tr21:10:151
1389899118,2cyclictest6208-21ssh00:01:221
1389899118,2cyclictest3574-21chrt00:19:411
1389899118,2cyclictest3500-21mailstats21:00:221
1389899118,2cyclictest28233-21cut19:40:001
1389899118,2cyclictest25759-21sh19:35:001
1389899118,2cyclictest25681-21cut23:35:141
1389899118,2cyclictest25165-21expr20:40:151
1389899118,2cyclictest22280-21cat20:35:011
1389899118,2cyclictest22027-21grep00:10:261
1389899118,2cyclictest20866-21taskset23:50:391
1389899118,2cyclictest16255-21awk19:15:001
1389899118,2cyclictest14763-21sort22:50:161
1389899118,2cyclictest14167-21ssh22:11:041
1389899118,2cyclictest12170-21taskset21:31:421
1389899118,2cyclictest10072-21ssh23:45:111
1389899113,4cyclictest225-21systemd-journal21:10:001
1389899113,4cyclictest225-21systemd-journal20:30:001
13898991110,1cyclictest0-21swapper/122:36:241
1389899110,2cyclictest0-21swapper/100:06:481
1389899110,10cyclictest5848-21tr22:45:141
1389399119,1cyclictest107932sleep022:28:560
1389399118,2cyclictest8312-21grep23:05:190
1389399118,2cyclictest6330-21chrt21:06:210
1389399118,2cyclictest5695-21cut22:45:140
1389399118,2cyclictest5584-21cut20:00:140
1389399118,2cyclictest4307-21ssh21:46:070
1389399118,2cyclictest29837-21tr20:50:140
1389399118,2cyclictest29361-21grep00:15:170
1389399118,2cyclictest26761-21ssh22:38:150
1389399118,2cyclictest26754-21ssh22:00:110
1389399118,2cyclictest24298-21rm21:58:280
1389399118,2cyclictest18783-21ls21:35:250
1389399118,2cyclictest1842-21cat19:50:200
1389399118,2cyclictest18173-21grep19:15:270
1389399118,2cyclictest1784-21ssh00:37:080
1389399118,2cyclictest14775-21cut19:10:170
1389399118,2cyclictest14401-21ssh21:33:250
1389399118,2cyclictest1256-21basename23:20:190
1389399116,2cyclictest225-21systemd-journal20:35:010
1389399114,3cyclictest17647-21apt-get20:25:030
1389399111,9cyclictest2813-21/usr/sbin/munin22:05:140
1389399110,11cyclictest6624-21diskmemload21:20:480
1390399108,1cyclictest6102-21cat20:00:173
1390399108,1cyclictest29486-21cat19:40:193
1390399108,1cyclictest19538-21taskset20:26:493
1390399108,1cyclictest16999-21sed19:15:153
1390399108,1cyclictest11848-21cat20:10:273
1390399108,1cyclictest11848-21cat20:10:273
1390399107,2cyclictest9645-21grep20:05:263
1390399107,2cyclictest14848-21expr19:10:163
1390099108,1cyclictest326722sleep219:47:142
1390099107,2cyclictest311rcuc/220:50:262
1390099107,2cyclictest10548-21expr20:10:162
1390099107,2cyclictest10548-21expr20:10:162
1390099105,3cyclictest32-21ksoftirqd/220:05:242
1389899108,1cyclictest4290-21tr22:25:141
1389899107,2cyclictest8039-21aten2_r7power_p20:05:141
1389899107,2cyclictest647-21tr20:55:211
1389899107,2cyclictest30806-21tr20:50:221
1389899106,3cyclictest23163-21grep20:35:161
1389899104,4cyclictest872-21cut19:50:141
1389399108,1cyclictest3123-21wc21:00:180
1389399108,1cyclictest31110-21expr19:45:140
1389399108,1cyclictest26492-21expr19:35:140
1389399108,1cyclictest18684-21wget23:50:130
1389399108,1cyclictest10560-21expr23:45:150
1389399107,2cyclictest25705-21chrt19:34:260
1389399107,2cyclictest21513-21grep19:25:150
1389399107,2cyclictest2067-21expr23:40:140
139039997,1cyclictest20302-21python19:20:243
138989997,1cyclictest30790-21taskset19:45:131
138989997,1cyclictest10406-21cut20:10:151
138989997,1cyclictest10406-21cut20:10:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional