You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 10:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Tue Aug 04, 2026 00:45:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
189182236213,16sleep20-21swapper/219:09:372
2683821690,1sleep30-21swapper/300:39:113
2275521690,1sleep30-21swapper/323:46:503
695721610,1sleep10-21swapper/121:58:281
186982149128,15sleep30-21swapper/319:06:453
188502143107,25sleep10-21swapper/119:08:431
186802136102,22sleep00-21swapper/019:06:300
8192840,2sleep30-21swapper/321:30:013
224682600,2sleep10-21swapper/123:46:321
167332590,3sleep334-21ksoftirqd/321:41:093
260972580,1sleep00-21swapper/019:25:150
172962560,1sleep30-21swapper/321:17:373
182822540,3sleep018278-21ssh22:30:260
236152520,1sleep20-21swapper/200:35:532
109432520,2sleep10-21swapper/123:14:331
49222510,1sleep24924-21gltestperf21:05:162
166482510,2sleep00-21swapper/022:54:300
146332500,3sleep10-21swapper/121:40:151
111802500,1sleep30-21swapper/322:50:153
187852450,2sleep10-21swapper/121:19:071
228562440,1sleep30-21swapper/321:21:273
258132390,1sleep30-21swapper/300:13:593
19223993711,6cyclictest1751-21cut19:45:001
19224993513,5cyclictest20911-21sh22:57:032
19224993411,5cyclictest8530-21seq23:35:552
19224993411,5cyclictest18182-21cat23:20:012
19223993410,6cyclictest24899-21seq21:47:271
19223993410,5cyclictest10738-21ls20:05:011
19224993312,3cyclictest20454-21sh22:32:302
19224993311,3cyclictest63232diskmemload00:16:442
19224993310,5cyclictest28200-21awk19:30:142
1922399339,5cyclictest1664-21grep00:20:141
19223993310,5cyclictest26434-21expr23:50:131
1922499329,5cyclictest11863-21cat22:50:212
1922499328,6cyclictest5260-21cat22:45:222
1922499327,9cyclictest0-21swapper/223:54:342
1922499327,9cyclictest0-21swapper/222:16:452
1922499326,9cyclictest2657-21forks21:55:162
1922499321,7cyclictest6744-21seq21:10:122
19224993212,3cyclictest12640-21sh22:26:202
19224993211,3cyclictest25495-21sh21:48:042
19224993210,5cyclictest800-21cut19:40:242
1922399329,5cyclictest5394-21seq23:34:311
1922399328,6cyclictest20094-21uname19:10:191
1922399327,6cyclictest2375-21tune2fs19:45:311
1922399321,6cyclictest0-21swapper/123:28:561
1922499319,7cyclictest19601-21diskstats19:10:152
1922499319,5cyclictest31690-21kernelversion23:05:182
1922499317,9cyclictest0-21swapper/200:21:192
19224993113,3cyclictest11974-21sh00:03:292
19224993111,5cyclictest28090-21cat20:45:002
19224993111,5cyclictest28090-21cat20:44:592
1922399312,8cyclictest75-21kswapd022:25:131
1922399312,10cyclictest1073-21tune2fs23:55:161
1922399311,7cyclictest22016-21cat00:10:231
1922399311,14cyclictest24813-21perl23:00:171
1922499307,7cyclictest30848-21seq21:51:332
1922499302,7cyclictest20577-21apt-get00:10:122
1922499301,5cyclictest0-21swapper/223:46:572
19224993012,3cyclictest18059-21sh00:07:522
19224993010,6cyclictest28-21ksoftirqd/223:20:122
1922399309,5cyclictest25320-21grep20:35:251
1922399308,5cyclictest25706-21awk19:25:101
1922399308,5cyclictest22-21ksoftirqd/122:10:121
1922399308,5cyclictest22-21ksoftirqd/122:10:121
1922399307,10cyclictest2352-21smartctl21:30:231
1922399301,7cyclictest13954-21ssh00:05:131
19223993013,3cyclictest8329-21sh00:00:211
19223993011,4cyclictest11400-21sh23:38:511
1922399301,13cyclictest7776-21cpuspeed_turbos21:35:121
1922399301,13cyclictest28507-21uname19:30:171
1922399300,5cyclictest0-21swapper/123:05:481
139782300,1sleep00-21swapper/021:15:180
1922499295,9cyclictest0-21swapper/222:11:182
1922499295,9cyclictest0-21swapper/222:11:182
19224992913,3cyclictest1878-21sendmail_mailtr23:55:242
19224992913,3cyclictest12155-21sh21:38:162
19224992911,4cyclictest6323-21diskmemload21:18:042
1922499291,13cyclictest31076-21sendmail_mailqu19:35:222
1922399299,6cyclictest22-21ksoftirqd/119:55:201
1922399299,5cyclictest30183-21expr22:40:131
1922399294,9cyclictest0-21swapper/121:20:181
1922399291,8cyclictest15526-21ssh00:30:181
19223992914,2cyclictest6323-21diskmemload00:15:211
19223992910,4cyclictest6323-21diskmemload23:19:211
19223992910,4cyclictest6323-21diskmemload22:59:431
1922499282,7cyclictest30411-21apt-get20:50:132
1922499281,11cyclictest30435-21diskstats22:40:152
1922399289,5cyclictest6323-21diskmemload22:07:361
1922399289,5cyclictest28776-21sh21:25:411
1922399288,5cyclictest22-21ksoftirqd/123:20:251
19223992811,3cyclictest26890-21sh22:37:151
19223992810,4cyclictest28394-21grep20:45:141
19222992819,7cyclictest20531-21ls23:45:200
280862270,3sleep334-21ksoftirqd/321:25:243
1922499279,4cyclictest6323-21diskmemload22:07:192
1922499274,2cyclictest141rcu_preempt22:35:562
19224992715,9cyclictest4553-21diskmemload23:33:402
19224992711,4cyclictest6323-21diskmemload23:00:422
1922399279,5cyclictest22-21ksoftirqd/121:00:111
1922399278,5cyclictest22-21ksoftirqd/100:35:251
19223992710,3cyclictest31003-21sh21:51:431
1922499261,11cyclictest25390-21apt-get23:25:132
1922399269,5cyclictest22-21ksoftirqd/120:05:251
1922399262,10cyclictest26135-21apt-get20:40:121
1922399262,10cyclictest26135-21apt-get20:40:121
1922399261,3cyclictest141rcu_preempt23:40:541
19225992523,1cyclictest0-21swapper/319:12:003
19224992516,6cyclictest28826-21hddtemp_smartct20:45:172
19224992515,8cyclictest25796-21ssh21:24:252
1922399251,6cyclictest12322-21ssh22:02:081
1922399251,10cyclictest17553-21apt-get20:20:141
19222992523,1cyclictest0-21swapper/019:11:580
134312250,2chrt0-21swapper/320:10:113
97532240,2sleep30-21swapper/320:00:243
217882240,3sleep321789-21grep21:45:183
1922599242,15cyclictest6323-21diskmemload22:55:113
19225992415,7cyclictest3964-21users21:55:263
1922299249,4cyclictest66050lldpd20:33:490
127872240,9sleep122-21ksoftirqd/122:50:341
284972230,2sleep00-21swapper/019:30:150
19225992312,9cyclictest8060-21sh21:35:163
1922499235,15cyclictest4386-21grep22:20:192
1922499232,4cyclictest1520150kworker/u9:121:25:432
19224992313,7cyclictest17881-21df_inode20:20:162
19222992312,10cyclictest12-21ksoftirqd/023:05:000
155312230,1sleep00-21swapper/022:05:100
55382220,1sleep10-21swapper/121:05:221
19224992214,6cyclictest15725-21diskstats20:15:162
1922299223,13cyclictest20125-21ssh21:20:120
19222992214,5cyclictest6851-40mandb00:00:000
19222992214,5cyclictest4935-21sed22:45:190
19222992213,7cyclictest25208-21seq19:21:580
19222992213,7cyclictest21410-21apt19:15:130
1922299221,17cyclictest10044-21apt-get22:25:120
315822211,1sleep30-21swapper/320:50:263
19225992113,6cyclictest2864-21seq23:08:063
19225992113,6cyclictest27253-21basename00:15:133
19225992113,6cyclictest15224-21uniq00:30:163
19225992112,6cyclictest1758-21ps23:55:233
1922499217,3cyclictest141rcu_preempt22:00:132
1922499214,3cyclictest6323-21diskmemload23:10:402
1922499211,5cyclictest0-21swapper/200:31:032
19224992112,6cyclictest24955-21grep19:20:262
1922499211,11cyclictest0-21swapper/221:40:282
19223992113,6cyclictest15951-21smartctl20:15:171
19223992112,6cyclictest22951-21seq19:15:431
19223992110,5cyclictest5387-21sed19:50:241
1922299217,5cyclictest28481-21tune2fs21:50:160
1922299212,13cyclictest22285-21ssh22:10:030
1922299212,13cyclictest22285-21ssh22:10:030
19222992114,5cyclictest3444-21sensors21:00:220
19222992114,4cyclictest4906-21sh21:56:220
19222992113,6cyclictest672-21ntp_states23:30:210
19222992113,6cyclictest560-21dbus-daemon00:30:000
19222992113,5cyclictest26492-21cpuspeed_turbos23:50:130
19222992112,7cyclictest8967-21seq21:10:560
19222992111,8cyclictest15120-21sed21:40:190
19222992110,6cyclictest6323-21diskmemload23:40:100
116712210,1sleep30-21swapper/323:39:073
1922599209,7cyclictest53050irq/142-enp3s0-22:10:243
1922599209,7cyclictest53050irq/142-enp3s0-22:10:243
19225992012,6cyclictest6860-21awk23:10:263
19225992012,6cyclictest29350-21seq22:39:443
19225992012,6cyclictest24289-21awk23:00:133
19224992013,5cyclictest9201-21cat20:00:182
19224992013,4cyclictest636-21runrttasks00:29:152
19224992010,8cyclictest636-21runrttasks23:43:502
1922399204,3cyclictest141rcu_preempt22:15:191
19223992012,6cyclictest8770-21smartctl00:25:161
19223992012,6cyclictest23879-21cat20:35:001
19223992011,6cyclictest30455-21fw_packets19:35:161
19223992011,6cyclictest18147-21sed22:30:251
1922299203,15cyclictest525-21avahi-daemon20:45:000
1922299203,15cyclictest525-21avahi-daemon20:45:000
1922299203,13cyclictest6195-21apt-get19:55:120
1922299201,8cyclictest26808-21ssh21:25:130
19222992013,4cyclictest18709-21pmu-power22:55:210
19222992012,6cyclictest4856-21cat19:50:190
19222992012,6cyclictest19509-21grep20:25:000
19222992012,6cyclictest10227-21seq20:02:180
19222992010,6cyclictest52750irq/139-enp3s0-21:45:150
154962200,4sleep3331rcuc/323:17:243
1922599198,9cyclictest0-21swapper/323:32:153
1922599192,12cyclictest25221-21ssh23:25:033
1922599191,4cyclictest141rcu_preempt21:10:123
19225991914,3cyclictest23629-21cpuspeed_turbos19:20:153
19225991913,4cyclictest29227-21sh23:51:373
19225991911,6cyclictest9136-21expr22:00:143
19225991911,6cyclictest7781-21cut00:00:163
19225991911,6cyclictest17708-21ntpq22:30:213
19225991911,5cyclictest29155-21ntp_kernel_err20:45:213
19225991911,5cyclictest16755-21cat20:15:263
19225991910,6cyclictest53050irq/142-enp3s0-22:45:163
1922499197,10cyclictest52850irq/140-enp3s0-20:30:252
1922499193,11cyclictest21412-21apt-get19:15:142
19224991912,5cyclictest25340-21cat20:35:252
19224991911,6cyclictest2875-21basename21:00:162
19224991910,6cyclictest25697-21cat19:25:112
1922399194,12cyclictest32681-21grep20:55:131
1922399193,3cyclictest6323-21diskmemload22:20:221
19223991912,4cyclictest8341-21sh22:47:471
19223991911,6cyclictest30680-21ls20:50:151
19223991910,7cyclictest23904-21df_inode19:20:161
19223991910,7cyclictest10738-21ssh21:12:441
19223991910,6cyclictest14354-21tr20:10:211
1922299194,13cyclictest3507-21ssh23:08:450
1922299192,14cyclictest1576-21taskset22:18:370
1922299192,12cyclictest15390-21apt-get20:15:130
19222991914,3cyclictest2308-21sendmail_mailtr21:30:230
19222991913,4cyclictest30934-21sh23:29:200
19222991913,4cyclictest12-21ksoftirqd/000:20:000
19222991912,4cyclictest24439-21sh00:36:440
19222991911,6cyclictest3502-21seq22:44:390
19222991911,6cyclictest32336-21date20:55:000
19222991911,6cyclictest3079-21tr19:45:380
19222991911,6cyclictest21834-21which00:10:220
19222991911,6cyclictest18131-21sed23:20:010
19222991911,5cyclictest8733-21sed00:00:250
19222991910,6cyclictest278-21systemd-journal20:39:550
1922599189,7cyclictest2545-21ntpq00:20:213
1922599189,6cyclictest4006-21cat19:50:003
1922599189,6cyclictest278-21systemd-journal20:29:553
1922599189,6cyclictest14323-21cat23:40:223
1922599188,8cyclictest28043-21ssh21:50:123
1922599182,11cyclictest4059-21apt-get19:50:113
1922599182,11cyclictest18534-21ssh23:20:133
19225991812,4cyclictest1908-21sh22:43:013
19225991811,5cyclictest6635-21grep19:55:163
19225991811,5cyclictest30167-21ntp_kernel_pll_22:15:203
19225991811,5cyclictest28064-21ls19:30:133
19225991811,5cyclictest16961-21swap22:05:263
19225991811,5cyclictest10767-21cat22:25:193
19225991811,4cyclictest6323-21diskmemload22:23:233
19225991811,4cyclictest2878-21idleruntime21:00:163
19225991810,6cyclictest25054-21gzip20:35:223
19225991810,6cyclictest1785-21cat19:44:593
1922499188,8cyclictest5707-21seq19:52:072
1922499184,2cyclictest141rcu_preempt21:30:252
19224991811,5cyclictest13540-21expr20:10:122
19224991810,6cyclictest7580-21grep19:55:262
19224991810,6cyclictest2404-21fw_packets19:45:312
19224991810,6cyclictest11703-21cat20:05:182
19223991811,5cyclictest21653-21cat20:30:001
1922299189,7cyclictest14166-21ntpq20:10:190
1922299182,11cyclictest8924-21apt-get22:00:130
1922299182,11cyclictest29433-21taskset20:45:240
1922299181,8cyclictest13888-21ssh00:05:110
19222991813,3cyclictest9325-21sh23:12:540
19222991811,5cyclictest4894-21sed22:20:230
19222991810,6cyclictest20280-21basename20:25:160
1922599178,6cyclictest23054-21cat20:30:223
1922599178,6cyclictest11017-21cut20:05:123
1922599175,9cyclictest6472-21sort21:10:013
1922599171,7cyclictest10668-21ssh00:26:153
19225991712,3cyclictest26780-21interrupts20:40:173
19225991712,3cyclictest26780-21interrupts20:40:163
19225991711,4cyclictest14666-21interrupts00:05:183
1922499179,6cyclictest1109-21awk20:55:222
1922299179,6cyclictest32362-21cat19:40:140
1922299179,6cyclictest30071-21wc19:35:130
1922299178,7cyclictest5931-21users21:05:260
1922299172,11cyclictest16136-21python300:30:240
1922299171,8cyclictest24673-21ssh23:24:560
19222991712,3cyclictest12022-21sh21:38:080
19222991711,4cyclictest6323-21diskmemload22:36:020
1922299171,12cyclictest1536-21ssh00:20:140
19222991710,5cyclictest757-21cat20:55:180
19222991710,5cyclictest11646-21sed20:05:170
19222991710,4cyclictest8539-21sh23:35:560
1922599162,9cyclictest29993-21apt-get19:35:133
1922599161,11cyclictest25710-21apt-get19:25:123
1922499161,11cyclictest19696-21apt-get20:25:122
1922599159,4cyclictest21665-21expr19:15:153
1922599157,6cyclictest17883-21df20:20:153
19225991510,3cyclictest32548-21apt20:55:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional