You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 07:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Jul 15, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24060216596,58sleep00-21swapper/019:05:480
240732148106,29sleep30-21swapper/319:05:593
243692141106,25sleep10-21swapper/119:09:491
241542139103,25sleep20-21swapper/219:07:012
2544721150,4sleep0560-21dbus-daemon21:45:010
249382920,2chrt0-21swapper/200:10:132
176212780,2sleep20-21swapper/200:29:082
304502650,1sleep10-21swapper/123:25:221
147382650,1sleep30-21swapper/321:11:453
277912600,2sleep10-21swapper/123:23:501
199982570,1sleep30-21swapper/322:53:293
317692530,1sleep331774-21cpu21:25:133
227322520,1sleep20-21swapper/200:32:332
176972520,1sleep30-21swapper/322:02:503
93972510,1sleep30-21swapper/321:05:113
301372510,2sleep00-21swapper/022:36:090
313572500,2sleep122-21ksoftirqd/119:25:131
70242480,1sleep00-21swapper/023:32:110
167682470,3chrt0-21swapper/020:05:170
290212440,2sleep122-21ksoftirqd/122:35:221
275872440,2sleep10-21swapper/121:45:261
154462440,2chrt0-21swapper/221:12:272
153422420,2sleep00-21swapper/022:50:130
299232410,1sleep20-21swapper/223:00:232
275022410,1sleep30-21swapper/323:23:313
191992390,1sleep20-21swapper/222:28:322
137812350,2chrt0-21swapper/022:24:500
132762350,3sleep00-21swapper/000:25:210
63222340,4chrt0-21swapper/321:54:513
310572330,2sleep028526-21kworker/u8:019:25:000
24659993319,8cyclictest11293-21diskmemload22:55:133
62182310,2chrt0-21swapper/221:30:132
223292300,2chrt0-21swapper/100:08:031
290062290,4sleep10-21swapper/100:13:061
24658992719,6cyclictest13930-21ssh22:24:592
2465699274,10cyclictest2405-21perf23:05:000
2465699271,12cyclictest141rcu_preempt20:30:120
157502270,3sleep315707-21ssh23:15:003
2465799261,24cyclictest0-21swapper/119:10:251
2465699261,24cyclictest0-21swapper/019:10:260
2465999251,18cyclictest0-21swapper/319:10:263
24658992516,5cyclictest53050irq/142-enp3s0-22:05:132
2465699250,14cyclictest52950irq/141-enp3s0-21:35:430
312372240,2sleep20-21swapper/222:13:082
24659992410,3cyclictest34-21ksoftirqd/300:00:143
24658992422,1cyclictest0-21swapper/219:10:292
24658992417,6cyclictest19980-21ssh22:53:282
39812230,1sleep20-21swapper/222:16:362
28232230,2sleep10-21swapper/123:05:121
2465999232,15cyclictest16850-21ssh23:40:113
24659992312,7cyclictest7483-21grep21:55:163
24659992311,8cyclictest30402-21apt-get23:50:133
24659992310,9cyclictest2412-21munin-run23:05:003
24658992314,7cyclictest14286-21diskstats20:00:132
24658992311,8cyclictest9405-21apt-get21:05:122
2465799238,12cyclictest278-21systemd-journal00:00:001
2465799231,20cyclictest2706-21cron20:50:001
2465799230,20cyclictest14650-21open_files22:00:211
2465699232,14cyclictest141rcu_preempt22:07:260
53712220,13sleep122-21ksoftirqd/121:29:521
2465999228,4cyclictest34-21ksoftirqd/320:50:193
2465999222,14cyclictest636-21runrttasks20:00:123
2465999222,14cyclictest11293-21diskmemload21:35:133
24659992214,5cyclictest657-21sshd23:56:323
24658992213,6cyclictest6172-21sed23:55:252
24658992212,7cyclictest30433-21sh23:25:222
2465799222,7cyclictest31112-21chrt20:40:131
24657992213,7cyclictest6439-21perl21:30:161
24657992213,7cyclictest4551-21seq20:51:161
24657992213,6cyclictest15792-21grep22:25:251
2465799221,15cyclictest9462-21ssh23:10:141
24656992215,4cyclictest11507-21mailstats23:35:240
96562210,2sleep10-21swapper/122:45:221
2465999211,6cyclictest1561-21ssh22:15:163
2465999211,5cyclictest141rcu_preempt22:20:123
2465999211,5cyclictest0-21swapper/323:06:013
24659992113,6cyclictest27727-21ntp_states20:30:203
24659992113,5cyclictest52850irq/140-enp3s0-22:05:143
24659992111,7cyclictest19990-21awk00:30:203
24659992110,7cyclictest18227-21expr00:05:133
24658992114,4cyclictest16423-21sh23:40:002
24658992113,6cyclictest5722-21forks19:40:132
24658992113,6cyclictest23613-21ls23:20:202
24658992113,6cyclictest16853-21seq21:38:032
24658992113,5cyclictest454-21sed00:15:282
24658992113,5cyclictest454-21sed00:15:272
24658992110,7cyclictest53050irq/142-enp3s0-21:40:132
2465799212,14cyclictest75-21kswapd022:40:111
2465799212,13cyclictest25599-21sh00:35:121
24657992113,6cyclictest23821-21seq22:07:201
24657992113,6cyclictest21563-21cut22:30:151
24657992112,7cyclictest9740-21diskstats21:05:141
24657992112,6cyclictest7730-21fw_packets21:00:181
24657992110,8cyclictest20363-21apt-get20:15:121
2465699218,3cyclictest12-21ksoftirqd/022:13:510
2465699214,4cyclictest12-21ksoftirqd/000:10:130
2465699212,16cyclictest12-21ksoftirqd/000:30:110
2465699212,13cyclictest31745-21ssh21:25:120
2465699212,10cyclictest12-21ksoftirqd/021:31:530
24656992116,4cyclictest12-21ksoftirqd/021:00:000
24656992112,7cyclictest25562-21ntp_kernel_pll_20:25:210
24656992112,7cyclictest25562-21ntp_kernel_pll_20:25:200
24656992112,5cyclictest52950irq/141-enp3s0-22:00:130
24656992112,5cyclictest52950irq/141-enp3s0-21:20:190
24656992111,8cyclictest15327-21ls20:00:250
2465699211,13cyclictest141rcu_preempt21:48:430
24656992110,6cyclictest52950irq/141-enp3s0-22:30:120
102212210,1sleep00-21swapper/023:59:210
37972200,2sleep00-21swapper/019:35:200
321052200,1sleep30-21swapper/322:14:013
2465999209,7cyclictest52850irq/140-enp3s0-23:25:143
2465999206,12cyclictest141rcu_preempt21:19:133
2465999203,3cyclictest11293-21diskmemload22:41:203
2465999203,11cyclictest29005-21apt-get19:20:133
2465999202,6cyclictest0-21swapper/323:19:383
2465999201,7cyclictest624-21gdbus19:20:003
2465999201,6cyclictest34-21ksoftirqd/322:37:493
24659992014,4cyclictest34-21ksoftirqd/322:30:003
24659992013,5cyclictest26713-21sh00:10:453
24659992013,5cyclictest13684-21pluginstate20:00:003
24659992011,7cyclictest23935-21cut23:45:143
24659992011,6cyclictest4338-21sed19:35:263
2465899202,13cyclictest29008-21chrt19:20:132
24658992013,5cyclictest24909-21sh22:56:432
24658992012,6cyclictest670-21grep19:30:002
24658992012,6cyclictest28133-21tune2fs22:35:132
24658992012,6cyclictest27858-21which19:15:212
24658992012,6cyclictest2217-21cat19:30:232
24658992012,6cyclictest1928-21find22:40:122
24658992011,7cyclictest7631-21grep19:45:152
24658992011,6cyclictest5687-21idleruntime20:55:192
2465799202,13cyclictest11737-21ssh21:10:131
2465799201,5cyclictest0-21swapper/122:56:361
24657992014,4cyclictest7294-21sendmail22:20:011
24657992012,6cyclictest25570-21ntp_kernel_pll_20:25:211
24657992012,6cyclictest25570-21ntp_kernel_pll_20:25:201
24657992012,6cyclictest15235-21seq00:02:381
24657992012,6cyclictest11632-21ls23:35:241
24657992011,6cyclictest8095-21cut21:55:231
24657992010,7cyclictest29615-21ntpq23:00:191
2465699209,9cyclictest6930-21rm22:19:380
2465699201,8cyclictest0-21swapper/023:46:050
24656992012,6cyclictest3904-21seq22:40:390
24656992012,6cyclictest18954-21cut21:15:160
2465699200,2cyclictest141rcu_preempt22:48:060
2465699200,12cyclictest141rcu_preempt00:01:290
203032200,1sleep10-21swapper/122:53:471
16402200,1sleep20-21swapper/221:50:242
2465999195,12cyclictest0-21swapper/321:30:173
2465999193,3cyclictest141rcu_preempt21:49:193
2465999191,5cyclictest0-21swapper/322:46:183
2465999191,4cyclictest0-21swapper/300:36:303
24659991914,3cyclictest20112-21dump-pmu-power20:15:003
24659991913,4cyclictest1149-21sh20:45:143
24659991913,4cyclictest11357-21grep23:35:233
24659991912,5cyclictest5807-21sed20:55:203
24659991912,4cyclictest20894-21sh22:30:023
24659991912,4cyclictest19372-21tr21:40:143
24659991911,6cyclictest31341-21cat20:40:143
24659991911,6cyclictest278-21systemd-journal20:40:013
24659991911,6cyclictest1205-21awk19:30:133
2465899191,6cyclictest0-21swapper/222:30:372
24658991913,4cyclictest8959-21sh00:22:052
24658991913,4cyclictest53050irq/142-enp3s0-00:07:312
24658991913,4cyclictest2705-21cron20:50:002
24658991912,5cyclictest12456-21seq23:11:452
24658991912,4cyclictest3408-21sh23:30:022
24658991912,4cyclictest13536-21/usr/sbin/munin22:00:152
24658991911,6cyclictest3183-21cat20:50:152
2465899191,16cyclictest0-21swapper/200:04:552
24658991910,7cyclictest29019-21cat20:35:142
2465799198,8cyclictest18356-21apt-get21:15:121
2465799197,9cyclictest0-21swapper/121:20:131
2465799197,10cyclictest18899-21taskset00:30:121
2465799192,12cyclictest18227-21apt-get20:10:121
2465799191,5cyclictest0-21swapper/100:15:141
2465799191,5cyclictest0-21swapper/100:15:141
24657991914,3cyclictest6600-21netstat00:20:201
24657991914,3cyclictest31071-21wc19:25:011
24657991914,3cyclictest13295-21ntp_offset00:25:201
24657991912,5cyclictest52750irq/139-enp3s0-20:55:121
24657991911,6cyclictest28688-21seq19:19:381
24657991911,6cyclictest28538-21seq20:33:091
24657991911,6cyclictest14764-21sed20:00:201
24657991911,6cyclictest1-21systemd22:10:241
24657991911,6cyclictest10535-21ntp_kernel_err19:50:211
24657991910,7cyclictest29048-21grep20:35:141
24657991910,7cyclictest13616-21seq19:59:221
2465699193,4cyclictest12-21ksoftirqd/023:06:150
2465699192,9cyclictest141rcu_preempt21:50:200
2465699191,9cyclictest141rcu_preempt20:40:190
2465699191,3cyclictest141rcu_preempt00:35:140
24656991913,4cyclictest20998-21sh23:42:530
24656991912,5cyclictest12475-21ssh23:11:450
24656991911,6cyclictest7590-21df21:00:150
24656991911,6cyclictest7336-21cut19:45:000
24656991911,6cyclictest28927-21cron23:24:590
24656991911,6cyclictest23345-21date20:20:200
24656991911,6cyclictest23193-21tr22:55:240
24656991911,6cyclictest14940-21expr22:25:160
24656991911,6cyclictest10937-21seq21:05:360
24656991911,5cyclictest31899-21memory19:25:190
24656991910,7cyclictest11705-21apt21:10:100
24656991910,5cyclictest52950irq/141-enp3s0-00:15:140
24656991910,5cyclictest52950irq/141-enp3s0-00:15:130
142072191,1sleep30-21swapper/300:25:413
2465999189,7cyclictest32107-21cut19:25:223
2465999183,10cyclictest7516-21apt-get19:45:143
24659991813,3cyclictest6939-21sh23:32:053
24659991811,5cyclictest6661-21ntpq00:20:213
24659991811,5cyclictest6155-21sed19:40:173
2465899189,6cyclictest21669-21cat20:15:252
2465899187,9cyclictest8991-21sh22:45:142
2465899187,7cyclictest53050irq/142-enp3s0-21:55:132
2465899181,8cyclictest18455-21/usr/sbin/munin23:40:232
24658991813,3cyclictest27770-21sh00:35:532
24658991813,3cyclictest17002-21missed_timers23:15:192
24658991812,4cyclictest3995-21sh21:28:272
24658991812,4cyclictest32613-21sh23:50:552
24658991812,4cyclictest27194-21sh21:20:502
24658991811,5cyclictest7434-21cat21:00:142
24658991811,5cyclictest3001-21ls23:05:132
2465899181,14cyclictest18988-21hddtemp_smartct21:15:172
24658991810,6cyclictest27689-21unixbench_multi21:45:272
24658991810,6cyclictest26238-21seq20:26:042
24658991810,6cyclictest26238-21seq20:26:032
24658991810,6cyclictest24360-21load23:45:182
24657991813,3cyclictest2222-21sh21:50:371
24657991812,4cyclictest24130-21sed23:45:151
24657991812,3cyclictest16482-21entropy23:15:151
24657991811,5cyclictest6302-21awk19:40:201
24657991811,5cyclictest30641-21expr23:50:141
24657991811,5cyclictest16072-21find20:05:091
24657991810,6cyclictest22923-21tune2fs20:20:161
24657991810,5cyclictest8105-21idleruntime19:45:181
2465699189,6cyclictest20351-21cat20:15:110
24656991813,3cyclictest22895-21sh00:08:380
24656991811,4cyclictest3682-21latency20:50:190
2465699181,14cyclictest17711-21unixbench_multi23:15:260
2465699181,10cyclictest31333-21ssh23:25:410
24656991810,6cyclictest560-21dbus-daemon20:40:010
24656991810,6cyclictest30127-21sort23:50:010
24656991810,6cyclictest28954-21sed19:20:010
24656991810,6cyclictest2381-21seq19:30:250
24656991810,6cyclictest1219-21tune2fs20:45:150
2465999177,7cyclictest8714-21unixbench_multi21:00:263
24659991711,4cyclictest26557-21latency_hist20:30:013
24659991711,4cyclictest26557-21latency_hist20:30:003
24659991710,5cyclictest24610-21cut20:25:013
24659991710,5cyclictest10257-21cat19:50:173
2465999170,4cyclictest0-21swapper/300:15:063
2465999170,4cyclictest0-21swapper/300:15:053
2465999170,15cyclictest25490-21grep21:20:163
2465899179,5cyclictest3004-21apt-get19:35:122
2465899178,7cyclictest0-21swapper/220:10:162
24658991712,3cyclictest17969-21sh20:10:002
24658991710,5cyclictest9848-21expr19:50:132
24658991710,5cyclictest31289-21expr20:40:142
24658991710,5cyclictest27833-21ps20:30:232
2465799179,6cyclictest20064-21cat21:40:181
2465799178,6cyclictest16865-21cpu23:40:111
2465799177,8cyclictest13098-21grep21:35:171
2465799172,11cyclictest3595-21apt-get23:30:141
24657991712,3cyclictest8825-21open_inodes22:20:211
24657991711,4cyclictest4942-21apt-key19:40:001
2465699179,6cyclictest12919-21which19:55:240
2465699175,3cyclictest12-21ksoftirqd/000:20:170
2465699174,11cyclictest12-21ksoftirqd/021:55:130
24656991712,3cyclictest19351-21sendmail_mailtr20:10:240
24656991711,4cyclictest12-21ksoftirqd/019:50:010
2465999169,5cyclictest34-21ksoftirqd/320:15:133
2465999162,8cyclictest16087-21apt-get20:05:113
2465899169,5cyclictest12168-21expr19:55:162
24658991610,4cyclictest23077-21expr20:20:162
2465699168,6cyclictest10838-21cat19:50:230
2465799159,4cyclictest2106-21grep19:30:241
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional