You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 11:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Jun 30, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
333329885,9sleep20-21swapper/219:08:322
342429078,8sleep00-21swapper/019:09:420
322128864,13sleep10-21swapper/119:07:061
331728569,10sleep30-21swapper/319:08:213
150022690,0sleep30-21swapper/322:37:533
279612610,0sleep20-21swapper/200:21:342
205492600,0sleep00-21swapper/023:58:320
100762510,0sleep00-21swapper/022:54:410
353299280,27cyclictest0-21swapper/123:24:541
353299250,22cyclictest0-21swapper/121:52:351
353299230,1cyclictest0-21swapper/119:50:161
258732230,0chrt0-21swapper/322:05:273
353899222,11cyclictest1394-21wget22:30:143
353899222,11cyclictest1394-21wget22:30:143
353599220,0cyclictest0-21swapper/219:20:142
353899211,6cyclictest0-21swapper/322:56:403
353899211,19cyclictest20724-21ntpq23:00:203
353599210,21cyclictest0-21swapper/222:08:162
353299211,12cyclictest0-21swapper/120:25:141
353299210,19cyclictest0-21swapper/122:48:111
352899217,7cyclictest27381-21readlink20:00:000
353899200,19cyclictest0-21swapper/320:15:113
353599200,1cyclictest16907-21ntp_states22:00:212
3535991918,1cyclictest0-21swapper/221:27:082
353599190,16cyclictest0-21swapper/223:32:242
353299190,1cyclictest0-21swapper/122:05:171
353299190,19cyclictest0-21swapper/100:25:451
38682180,0sleep30-21swapper/323:48:373
3538991817,1cyclictest0-21swapper/300:14:283
353599180,1cyclictest0-21swapper/219:55:132
353599180,18cyclictest0-21swapper/221:44:352
353599180,16cyclictest0-21swapper/223:41:382
353899170,17cyclictest6866-21kworker/3:000:30:533
353899170,0cyclictest0-21swapper/323:18:123
3535991715,2cyclictest0-21swapper/223:53:492
353599170,16cyclictest0-21swapper/222:45:152
3532991716,1cyclictest17680-21python00:15:211
3532991716,1cyclictest0-21swapper/123:54:451
353299170,16cyclictest0-21swapper/123:41:221
352899173,8cyclictest29714-21awk23:45:000
353899161,15cyclictest22745-21kworker/3:123:08:193
353899160,1cyclictest0-21swapper/321:40:223
353899160,16cyclictest0-21swapper/300:25:033
3535991613,2cyclictest14213-21munin-run20:40:002
3532991616,0cyclictest0-21swapper/121:32:051
352899166,9cyclictest19641-21apache222:41:340
352899160,1cyclictest0-21swapper/023:12:570
353899155,7cyclictest28551-21diskmemload00:17:373
353899150,1cyclictest0-21swapper/323:34:333
353599152,3cyclictest141rcu_preempt23:25:422
353599150,15cyclictest0-21swapper/223:00:542
352899150,2cyclictest0-21swapper/020:25:140
352899150,15cyclictest0-21swapper/021:27:000
353899145,6cyclictest1235-21ntpd00:37:413
353899143,1cyclictest0-21swapper/322:49:463
3538991412,1cyclictest29579-21kworker/3:021:10:433
3538991410,3cyclictest8536-21sh21:35:443
3538991410,3cyclictest29682-21cron23:44:593
3538991410,3cyclictest11056-21cut19:25:153
3538991410,3cyclictest10652-21latency_hist19:25:003
353899140,4cyclictest0-21swapper/322:17:333
353599149,4cyclictest9587-21wc22:15:192
353599140,13cyclictest0-21swapper/222:26:182
353599140,13cyclictest0-21swapper/221:46:342
353299140,1cyclictest28551-21diskmemload21:45:541
352899142,5cyclictest28321-21kworker/u8:322:34:140
352899142,5cyclictest28321-21kworker/u8:322:34:140
3528991413,1cyclictest0-21swapper/023:08:210
352899140,0cyclictest0-21swapper/022:06:230
353899139,2cyclictest23357-21grep19:50:153
353899139,2cyclictest21990-21wc23:20:223
353899138,4cyclictest19080-21head22:40:163
353899134,6cyclictest0-21swapper/300:20:243
353899133,7cyclictest0-21swapper/322:20:193
3538991312,1cyclictest39-21ksoftirqd/323:27:273
3538991310,2cyclictest4593-21cut23:10:163
3538991310,2cyclictest32654-21wc00:05:253
3538991310,2cyclictest27978-21wc21:05:253
3538991310,2cyclictest14023-21cron23:55:003
3538991310,2cyclictest1088-21cat22:10:203
353599138,3cyclictest0-21swapper/223:47:552
353599137,4cyclictest32-21ksoftirqd/220:05:152
3535991312,1cyclictest0-21swapper/221:19:122
353599131,1cyclictest1904-21diskstats22:30:162
353599131,1cyclictest1904-21diskstats22:30:162
3535991310,2cyclictest9503-21cat00:10:292
3535991310,2cyclictest849-21cat00:05:322
3535991310,2cyclictest8343-21latency_hist00:30:002
3535991310,2cyclictest4502-21cat22:50:282
3535991310,2cyclictest20295-21grep22:40:272
3535991310,2cyclictest1599-21cat20:10:252
353599130,1cyclictest0-21swapper/222:12:232
353599130,13cyclictest0-21swapper/221:22:342
353599130,13cyclictest0-21swapper/200:37:122
3532991310,2cyclictest30710-21ssh21:10:261
3532991310,2cyclictest26720-21tr21:05:141
3532991310,2cyclictest25488-21tr19:55:141
3532991310,2cyclictest22311-21grep20:55:171
3532991310,2cyclictest12909-21grep23:15:181
3532991310,2cyclictest12111-21grep19:25:221
353299130,1cyclictest0-21swapper/122:25:201
352899139,3cyclictest29075-21ssh23:25:140
352899139,3cyclictest25258-21wc00:40:000
352899133,7cyclictest6946-21latency_hist00:10:010
352899131,7cyclictest30801-21idleruntime-cro21:50:000
352899131,6cyclictest28551-21diskmemload22:20:170
352899131,2cyclictest225-21systemd-journal23:35:000
3528991312,1cyclictest0-21swapper/021:32:150
3528991312,1cyclictest0-21swapper/000:15:230
3528991310,2cyclictest5034-21grep20:20:160
3528991310,2cyclictest21935-21tr20:55:150
3528991310,2cyclictest21910-21wc19:45:250
3528991310,2cyclictest17540-21chrt21:41:150
3528991310,2cyclictest12955-21grep23:15:180
3528991310,2cyclictest11566-21cat22:35:260
352899130,8cyclictest15073-21awk22:00:000
352899130,12cyclictest0-21swapper/019:10:140
353899129,2cyclictest7035-21ssh21:15:283
353899129,2cyclictest32078-21sort21:50:173
353899129,2cyclictest31236-21head20:05:223
353899129,2cyclictest30850-21cron00:05:003
353899129,2cyclictest30850-21cron00:05:003
353899129,2cyclictest26860-21sed19:55:263
353899129,2cyclictest22599-21sh19:50:003
353899129,2cyclictest17270-21grep19:35:263
353899129,2cyclictest11400-21grep20:30:263
353899128,3cyclictest7554-21tr21:55:153
353899128,3cyclictest18379-21sh22:01:033
353899128,3cyclictest15300-21tail23:55:153
353899128,3cyclictest13678-21tr21:20:143
353899126,2cyclictest225-21systemd-journal21:35:003
3538991210,2cyclictest23375-21kworker/3:222:51:033
353899120,11cyclictest27239-21ssh21:28:063
353599129,2cyclictest839-21sed21:50:272
353599129,2cyclictest24534-21sed19:50:252
353599129,2cyclictest15883-21ls23:55:192
353599128,3cyclictest4105-21wc20:15:242
353599121,7cyclictest0-21swapper/220:00:142
353299129,2cyclictest7481-21ssh00:10:071
353299129,2cyclictest6921-21grep19:15:181
353299129,2cyclictest4814-21ssh23:10:191
353299129,2cyclictest29187-21cut23:05:221
353299129,2cyclictest24908-21grep00:00:241
353299129,2cyclictest24908-21grep00:00:241
353299129,2cyclictest24614-21ssh22:25:001
353299129,2cyclictest21468-21wc20:55:001
353299129,2cyclictest20841-21grep23:00:211
353299129,2cyclictest18779-21grep19:40:171
353299129,2cyclictest17734-21sh22:40:001
353299129,2cyclictest14890-21grep21:20:221
353299129,2cyclictest10171-21cat22:15:251
353299128,4cyclictest3433-21sh00:07:231
353299128,3cyclictest23814-21sed21:25:281
353299127,4cyclictest22538-21taskset19:49:201
353299127,4cyclictest10141-21ssh23:32:501
353299125,6cyclictest38150irq/131-snd_hda21:41:341
353299123,4cyclictest225-21systemd-journal20:35:001
3532991210,1cyclictest0-21swapper/122:55:201
353299120,12cyclictest0-21swapper/121:19:121
353299120,11cyclictest15070-21systemctl22:00:001
352899129,2cyclictest8486-21grep00:10:200
352899129,2cyclictest629-21ssh00:25:140
352899129,2cyclictest379-21grep20:10:140
352899129,2cyclictest27069-21grep22:25:280
352899129,2cyclictest25942-21chrt00:20:240
352899129,2cyclictest24506-21grep19:50:250
352899129,2cyclictest23029-21tr00:00:120
352899129,2cyclictest23029-21tr00:00:120
352899129,2cyclictest22253-21wc23:20:260
352899129,2cyclictest21056-21ssh23:00:230
352899129,2cyclictest16418-21tail22:00:170
352899129,2cyclictest13572-21tr19:30:150
352899129,2cyclictest10229-21grep19:20:280
352899128,3cyclictest19641-21apache221:21:270
352899128,3cyclictest12580-21sed19:25:270
352899126,2cyclictest225-21systemd-journal19:40:010
3528991210,1cyclictest31732-21grep20:05:260
352899120,0cyclictest0-21swapper/022:56:300
353899118,2cyclictest897-21tail20:10:163
353899118,2cyclictest7269-21date20:25:003
353899118,2cyclictest5383-21sed19:10:273
353899118,2cyclictest27907-21tr20:00:163
353899118,2cyclictest26036-21ssh22:25:193
353899118,2cyclictest26002-21sh21:46:163
353899118,2cyclictest21763-21cut20:55:153
353899118,2cyclictest18206-21cut20:45:213
353899118,2cyclictest15386-21cron19:35:003
353899118,2cyclictest15249-21cut20:40:173
353899118,2cyclictest13709-21sed20:35:253
353899115,4cyclictest14141-21grep23:35:163
353899111,4cyclictest18630-21df_abs19:40:153
353899111,1cyclictest0-21swapper/320:50:153
353899110,7cyclictest0-21swapper/319:15:143
353599119,1cyclictest4530-21cat21:15:002
353599119,1cyclictest12511-21expr23:15:162
353599118,2cyclictest9117-21rm23:13:042
353599118,2cyclictest6637-21expr21:35:152
353599118,2cyclictest6475-21cat20:20:282
353599118,2cyclictest17810-21memory22:20:172
353599118,2cyclictest15515-21ssh22:38:172
353599118,2cyclictest15412-21perf19:35:002
353599118,2cyclictest14737-21grep20:40:152
353599118,2cyclictest10796-21sh22:55:042
353599118,2cyclictest0-21swapper/223:37:122
353599112,3cyclictest19641-21apache221:55:122
353599110,2cyclictest0-21swapper/200:33:412
353599110,10cyclictest17571-21grep20:45:162
353299118,2cyclictest32642-21expr22:10:141
353299118,2cyclictest29626-21grep23:25:171
353299118,2cyclictest2804-21tr20:15:121
353299118,2cyclictest24815-21grep21:00:171
353299118,2cyclictest24321-21ssh00:20:051
353299118,2cyclictest18135-21rm22:40:081
353299118,2cyclictest17259-21ssh00:35:121
353299118,2cyclictest16888-21grep19:35:221
353299118,2cyclictest16380-21fw_packets22:00:171
353299118,2cyclictest15255-21sed20:40:171
353299118,2cyclictest14518-21grep19:30:221
353299118,2cyclictest1395-21grep22:30:141
353299118,2cyclictest1395-21grep22:30:141
353299118,2cyclictest13950-21rm00:32:471
353299118,2cyclictest12759-21grep20:35:161
353299112,7cyclictest38150irq/131-snd_hda21:35:051
353299110,11cyclictest0-21swapper/123:55:391
353299110,10cyclictest9207-21ssh22:53:591
352899119,1cyclictest9243-21tail22:15:160
352899118,2cyclictest30261-21grep23:45:140
352899118,2cyclictest27222-21grep22:45:130
352899118,2cyclictest1845-21ssh21:12:560
352899117,3cyclictest7456-21ssh23:50:200
352899117,2cyclictest25777-21grep21:00:260
352899110,5cyclictest5316-21grep21:15:150
353899108,1cyclictest7397-21cut20:25:153
353599108,1cyclictest8481-21cat19:20:012
353599108,1cyclictest3720-21seq19:10:012
353599108,1cyclictest23942-21ls21:00:002
353599108,1cyclictest21344-21ssh23:20:172
353599108,1cyclictest19836-21df20:50:172
353599108,1cyclictest18055-21apache_accesses19:40:122
353599108,1cyclictest16425-21expr00:15:122
353599108,1cyclictest15616-21date19:35:002
353599107,2cyclictest11319-21sed20:30:262
353599105,4cyclictest0-21swapper/221:30:242
353599103,4cyclictest225-21systemd-journal20:26:412
353599100,9cyclictest22705-21taskset00:00:012
353599100,9cyclictest22705-21taskset00:00:012
353599100,9cyclictest0-21swapper/223:09:382
353299108,2cyclictest14401-21cat23:35:171
353299108,1cyclictest56712sleep119:12:311
353299108,1cyclictest29103-21sed20:00:261
353299108,1cyclictest2424-21cut20:15:001
353299107,2cyclictest8604-21basename19:20:151
353299107,2cyclictest5338-21expr20:20:171
353299107,2cyclictest32130-21grep23:45:271
353299107,2cyclictest17880-21cat20:45:181
353299102,4cyclictest31250-21grep20:05:221
352899108,1cyclictest868-21sh21:50:270
352899108,1cyclictest3657-21grep20:15:180
352899108,1cyclictest18430-21sed20:45:240
352899108,1cyclictest18117-21basename19:40:140
352899108,1cyclictest141142sleep020:39:080
352899108,1cyclictest11031-21fgrep20:30:220
352899107,2cyclictest9134-21grep00:30:150
352899107,2cyclictest8119-21head21:35:260
352899107,2cyclictest32056-21ssh22:10:020
352899107,2cyclictest27803-21tr21:05:220
352899107,2cyclictest20285-21sed20:50:200
352899107,2cyclictest15105-21sort20:40:170
352899107,2cyclictest14934-21wc23:35:210
352899106,3cyclictest28344-21grep20:00:180
35389996,2cyclictest25268-21cut21:00:223
35359998,1cyclictest24058-21apache_volume21:00:132
35359998,1cyclictest20977-21df_abs19:45:142
35359998,1cyclictest12128-21cut19:25:222
35359997,2cyclictest28219-21taskset21:05:582
35289997,1cyclictest7997-21chrt19:16:500
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional