You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-05 - 09:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sun Jul 05, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32277996538,20cyclictest0-21swapper/200:35:132
3226899643,46cyclictest29116-21ssh22:45:131
3225799644,39cyclictest22409-21ssh21:30:120
3226899594,43cyclictest16059-21ssh22:20:121
3226899572,8cyclictest131rcu_preempt23:30:131
3226899564,40cyclictest687-21ssh23:05:141
3225799492,36cyclictest12144-21ssh00:05:150
3228599465,28cyclictest35966irq/58-eth000:10:123
32268994618,10cyclictest131rcu_preempt22:05:121
3225799452,35cyclictest3084-21ssh21:55:120
3226899432,37cyclictest21111-21ssh00:10:121
3225799431,36cyclictest4149-21ssh21:20:140
3227799421,22cyclictest10955-21ssh22:35:132
3226899424,30cyclictest9700-21rm23:10:091
3227799411,36cyclictest25121-21ssh22:25:122
3225799411,30cyclictest18413-21ssh21:10:140
3226899404,26cyclictest6980-21ssh22:15:121
3226899401,31cyclictest4130-21ssh21:20:131
32277993917,18cyclictest0-21swapper/220:05:122
3228599383,20cyclictest35966irq/58-eth021:10:143
3228599382,19cyclictest35966irq/58-eth000:25:113
3228599374,21cyclictest35966irq/58-eth023:25:123
3227799372,29cyclictest26606-21ssh23:55:152
3226899374,21cyclictest21-21ksoftirqd/123:45:131
3226899374,21cyclictest21-21ksoftirqd/123:45:131
3225799374,22cyclictest15734-21ssh00:25:120
3228599362,21cyclictest35966irq/58-eth023:05:153
3227799368,21cyclictest2656-1kworker/u9:100:00:152
3227799362,28cyclictest27985-21df_inode23:20:142
3227799362,18cyclictest6973-21sh22:30:492
3226899361,29cyclictest14849-21ssh22:55:131
3228599354,24cyclictest33-21ksoftirqd/300:35:133
3228599353,16cyclictest35966irq/58-eth019:30:143
3227799352,3cyclictest0-21swapper/222:55:132
3227799351,28cyclictest23945-21ssh23:00:132
3226899357,20cyclictest21-21ksoftirqd/123:40:121
3226899353,8cyclictest131rcu_preempt20:25:131
3226899351,24cyclictest30226-21ssh22:10:121
3227799342,16cyclictest13635-21ssh22:17:382
3227799342,16cyclictest10017-21ssh23:26:122
3227799342,11cyclictest17574-21df23:50:142
3226899347,16cyclictest27879-21ssh23:20:121
3226899342,24cyclictest18432-21ssh21:10:151
32268993411,15cyclictest630-21Async09:36:311
3228599331,17cyclictest35966irq/58-eth021:40:123
3228599331,17cyclictest35966irq/58-eth021:40:113
3227799339,18cyclictest0-21swapper/222:22:382
32257993310,17cyclictest26306-21diskmemload22:05:490
3228599326,23cyclictest35966irq/58-eth000:15:123
32277993214,13cyclictest536-21usb-storage21:25:122
3227799321,24cyclictest18012-21diskmemload22:45:132
32285993114,11cyclictest35966irq/58-eth022:52:463
3227799314,22cyclictest27-21ksoftirqd/200:30:152
32277993110,6cyclictest0-21swapper/223:15:122
3225799312,23cyclictest1797-21ssh23:22:130
3228599302,17cyclictest35966irq/58-eth000:20:123
3228599302,17cyclictest35966irq/58-eth000:20:123
32285993012,10cyclictest35966irq/58-eth022:35:133
3227799303,18cyclictest27346irq/40-dwc2_hso20:35:122
32268993014,10cyclictest0-21swapper/100:00:131
3225799302,22cyclictest29149-21ssh23:00:560
3228599293,17cyclictest0-21swapper/321:50:113
3228599293,15cyclictest35966irq/58-eth023:35:153
3228599292,18cyclictest35966irq/58-eth022:05:133
3227799292,20cyclictest0-21swapper/219:20:142
3226899292,24cyclictest5775-21ssh22:50:131
3225799292,21cyclictest22732-21sh22:22:390
3225799291,22cyclictest31727-21ssh23:40:140
3228599283,18cyclictest0-21swapper/323:23:123
3228599283,13cyclictest35966irq/58-eth023:10:103
3228599282,19cyclictest0-21swapper/323:18:143
3228599281,13cyclictest35966irq/58-eth000:05:143
3227799283,20cyclictest0-21swapper/223:11:142
3227799283,18cyclictest27346irq/40-dwc2_hso21:10:142
3227799283,17cyclictest27346irq/40-dwc2_hso20:55:142
3227799282,20cyclictest0-21swapper/219:45:122
3227799282,20cyclictest0-21swapper/219:30:152
3227799282,11cyclictest16545-21ssh21:44:462
3227799282,11cyclictest16545-21ssh21:44:462
32277992812,13cyclictest0-21swapper/222:05:122
3226899282,19cyclictest201rcuc/122:30:151
3226899281,24cyclictest10939-21ssh22:35:121
3225799282,20cyclictest31499-21ssh21:35:130
3227799273,17cyclictest27346irq/40-dwc2_hso21:00:122
3226899272,20cyclictest17578-21rm23:50:141
3226899272,19cyclictest23965-21ssh23:00:141
3225799272,20cyclictest19447-21ssh22:39:510
3225799272,20cyclictest10374-21ssh22:34:490
3228599269,9cyclictest35966irq/58-eth021:55:133
3228599263,16cyclictest0-21swapper/323:00:153
3228599262,4cyclictest0-21swapper/322:25:283
3228599261,8cyclictest0-21swapper/322:55:553
3227799263,10cyclictest27346irq/40-dwc2_hso20:15:112
3227799263,10cyclictest27346irq/40-dwc2_hso20:15:112
3227799262,16cyclictest11017-21grep19:55:172
32277992613,8cyclictest11979-1kworker/u9:222:52:462
3227799261,17cyclictest0-21swapper/221:35:122
3227799260,22cyclictest2492-21find20:20:112
3226899264,15cyclictest0-21swapper/100:25:131
3228599258,4cyclictest0-21swapper/322:20:413
3228599253,14cyclictest0-21swapper/322:10:133
3228599251,5cyclictest0-21swapper/320:15:213
3228599251,5cyclictest0-21swapper/320:15:213
3228599251,5cyclictest0-21swapper/319:40:143
3228599251,10cyclictest0-21swapper/321:15:213
3227799253,7cyclictest30056-21ssh00:15:122
3227799253,6cyclictest0-21swapper/223:05:132
3227799252,16cyclictest0-21swapper/221:30:052
3227799251,9cyclictest12346-21ssh00:21:272
3227799251,9cyclictest12346-21ssh00:21:272
3227799251,15cyclictest27346irq/40-dwc2_hso21:20:132
32277992511,12cyclictest0-21swapper/200:15:012
3225799252,15cyclictest1309-21ssh00:35:130
3228599241,7cyclictest0-21swapper/323:45:213
3228599241,7cyclictest0-21swapper/323:45:203
3228599241,7cyclictest0-21swapper/323:30:203
3228599241,7cyclictest0-21swapper/322:40:203
3228599241,6cyclictest0-21swapper/323:40:293
3228599241,6cyclictest0-21swapper/322:05:013
3228599241,5cyclictest0-21swapper/323:50:133
3228599241,4cyclictest0-21swapper/321:45:183
3227799242,8cyclictest18012-21diskmemload22:40:552
3227799242,17cyclictest12113-21apt-get00:05:132
3226899243,18cyclictest12142-21ssh22:00:151
3226899242,16cyclictest3072-21apt-get21:55:121
3226899242,15cyclictest22658-21ssh23:35:151
3226899241,20cyclictest17241-21ssh21:45:121
3225799243,16cyclictest20033-21ssh22:40:140
3225799242,17cyclictest17489-21ssh23:50:140
3228599231,8cyclictest0-21swapper/321:10:013
3228599231,7cyclictest0-21swapper/322:45:183
3228599231,7cyclictest0-21swapper/322:15:193
3228599231,6cyclictest0-21swapper/322:35:013
3228599231,6cyclictest0-21swapper/321:35:173
3228599231,6cyclictest0-21swapper/321:30:213
3228599231,6cyclictest0-21swapper/321:25:213
3228599231,6cyclictest0-21swapper/321:00:273
3228599231,6cyclictest0-21swapper/320:55:023
3228599231,6cyclictest0-21swapper/320:35:193
3228599231,6cyclictest0-21swapper/320:30:313
3228599231,6cyclictest0-21swapper/320:25:193
3228599231,6cyclictest0-21swapper/320:20:153
3228599231,6cyclictest0-21swapper/300:30:213
3228599231,6cyclictest0-21swapper/300:00:183
3228599231,5cyclictest0-21swapper/323:55:093
3227799239,11cyclictest0-21swapper/221:05:312
3227799232,16cyclictest0-21swapper/223:47:142
3227799232,16cyclictest0-21swapper/223:47:132
3226899235,13cyclictest22312-21kworker/u8:123:17:491
3226899232,19cyclictest14038-21ssh00:23:271
3226899232,19cyclictest14038-21ssh00:23:261
3226899232,16cyclictest12130-21ssh00:05:141
3226899231,19cyclictest1474-21ls19:45:171
3225799238,12cyclictest692-21ssh23:05:140
3225799236,13cyclictest6690-21sh00:20:140
3225799236,13cyclictest6690-21sh00:20:130
3225799233,15cyclictest26395-21ssh21:50:120
3225799233,15cyclictest17255-21diskmemload21:45:140
32257992315,5cyclictest18960-21apt-get19:30:050
3225799231,18cyclictest30084-21ssh00:15:140
3225799231,14cyclictest13581-21cut21:05:150
3228599222,2cyclictest0-21swapper/321:20:133
3228599221,6cyclictest0-21swapper/320:55:203
3228599221,6cyclictest0-21swapper/320:50:013
3228599221,6cyclictest0-21swapper/320:05:213
3228599221,6cyclictest0-21swapper/320:00:293
3228599221,6cyclictest0-21swapper/319:55:223
3228599221,6cyclictest0-21swapper/319:45:143
3228599221,6cyclictest0-21swapper/319:35:203
3228599221,6cyclictest0-21swapper/319:25:283
3228599221,6cyclictest0-21swapper/319:20:183
3228599221,5cyclictest0-21swapper/319:15:153
32285992215,2cyclictest0-21swapper/320:10:133
3227799223,14cyclictest27346irq/40-dwc2_hso23:35:142
3227799223,14cyclictest0-21swapper/221:50:122
3227799222,15cyclictest0-21swapper/222:14:492
3227799222,15cyclictest0-21swapper/222:03:462
32257992212,5cyclictest0-21swapper/020:30:150
3228599211,6cyclictest0-21swapper/320:40:153
3228599211,6cyclictest0-21swapper/320:40:153
3228599211,6cyclictest0-21swapper/319:50:183
3228599211,6cyclictest0-21swapper/319:10:163
3227799213,14cyclictest27346irq/40-dwc2_hso23:30:132
3227799211,13cyclictest12226-21ls20:30:162
3227799211,11cyclictest4512-21idleruntime-cro19:15:002
3226899213,8cyclictest24760-21ssh00:30:151
3226899213,4cyclictest131rcu_preempt23:25:131
32268992116,3cyclictest0-21swapper/120:05:281
32268992115,4cyclictest536-21usb-storage21:00:051
32268992112,5cyclictest0-21swapper/119:35:121
32268992111,6cyclictest26453-21/usr/sbin/munin23:55:291
3226899210,16cyclictest17231-21timerandwakeup21:25:291
3225799218,4cyclictest131rcu_preempt19:45:120
3225799215,11cyclictest12-21ksoftirqd/022:49:550
3225799211,7cyclictest23741-21cat23:00:010
32257992115,4cyclictest8095-21/usr/sbin/munin21:40:170
32257992115,4cyclictest8095-21/usr/sbin/munin21:40:160
32257992111,7cyclictest7098-21diskstats22:15:160
3225799211,17cyclictest20711-21cut20:05:150
3227799209,4cyclictest15699-21/usr/sbin/munin00:25:262
3227799207,9cyclictest0-21swapper/220:40:132
3227799207,9cyclictest0-21swapper/220:40:132
3227799205,5cyclictest2112-21kthreadcore23:40:222
3227799202,9cyclictest32217-21kthreadcore20:50:192
3227799201,9cyclictest930-21snmpd19:15:142
3227799201,9cyclictest6177-21ssh21:55:222
3227799201,16cyclictest20241-21sh21:45:232
3227799201,12cyclictest24237-21cut19:35:152
32277992010,8cyclictest20276-21sshd20:01:462
3227799200,11cyclictest10123-21cut20:25:212
3226899206,7cyclictest131rcu_preempt00:35:341
3226899202,9cyclictest13991-21irqrtprio21:05:171
32268992015,3cyclictest0-21swapper/122:40:151
3225799203,11cyclictest0-21swapper/019:20:130
3225799202,12cyclictest10990-21ssh22:50:550
3225799202,11cyclictest18803-21ssh23:15:140
3225799201,9cyclictest13760-21kthreadcore22:00:200
3225799201,3cyclictest0-21swapper/023:10:230
32257992010,7cyclictest7403-21cat00:00:320
3227799195,12cyclictest25626-21irqcore20:10:182
3227799192,8cyclictest14537-21sort19:25:162
3227799191,9cyclictest7473-21kthreadcore19:50:202
3227799191,9cyclictest26711-21fschecks_time20:45:152
3227799191,15cyclictest29057-21fschecks_time19:40:162
3227799191,10cyclictest27533-21ssh21:15:152
3226899195,8cyclictest32248-21cyclictest20:00:131
3226899192,14cyclictest9545-21apt-get19:20:121
3226899191,9cyclictest19923-21kernelversion19:30:191
32268991913,4cyclictest0-21swapper/100:15:121
3226899191,15cyclictest25094-21/usr/sbin/munin22:25:131
3226899191,14cyclictest21946-21apt-get21:30:031
3226899191,12cyclictest32636-21ls19:10:151
32268991910,6cyclictest31091-21cron21:35:021
3225799199,8cyclictest10950-21df_inode19:55:140
3225799199,7cyclictest10618-21wc20:25:220
3225799197,9cyclictest4686-21df23:25:150
3225799197,4cyclictest131rcu_preempt23:30:180
3225799195,12cyclictest5598-21sed20:55:210
3225799192,14cyclictest12-21ksoftirqd/000:30:140
3225799192,11cyclictest26475-21ssh23:55:110
3225799191,8cyclictest25219-21df22:25:140
3225799191,15cyclictest31485-21latency_hist23:40:020
32257991911,5cyclictest2247-21cron20:20:010
32257991911,5cyclictest2247-21cron20:20:000
3226899187,3cyclictest131rcu_preempt20:20:191
3226899182,3cyclictest131rcu_preempt19:55:141
3226899182,13cyclictest28810-21kthreadcore20:45:191
3226899181,9cyclictest25283-21/usr/sbin/munin20:10:281
3226899181,4cyclictest131rcu_preempt19:40:211
3226899181,15cyclictest14887-21kthreadcore20:30:231
3225799188,7cyclictest12-21ksoftirqd/019:30:010
3225799186,3cyclictest131rcu_preempt21:15:230
32257991811,5cyclictest28623-21cron19:40:010
3225799181,14cyclictest32545-21fschecks_count19:10:150
3227799171,8cyclictest32364-21latency_hist19:10:022
3226899179,3cyclictest21-21ksoftirqd/119:25:131
3226899174,10cyclictest27988-21kthreadcore21:50:201
3226899173,11cyclictest9375-21missed_timers19:50:221
3226899171,2cyclictest0-21swapper/120:15:171
3226899171,2cyclictest0-21swapper/120:15:161
32268991711,4cyclictest8121-21ssh21:40:121
32268991711,4cyclictest8121-21ssh21:40:121
3226899170,14cyclictest17182-21ls20:35:161
3225799176,8cyclictest21099-21apt-get00:10:110
3225799175,9cyclictest32457-21tr22:10:190
3225799174,10cyclictest3732-21kthreadcore20:20:190
3225799172,10cyclictest0-21swapper/020:50:140
3225799172,10cyclictest0-21swapper/020:45:130
3225799171,7cyclictest12776-21apt-get21:25:040
3225799171,13cyclictest22456-21sort20:40:180
3225799171,13cyclictest22456-21sort20:40:170
3225799171,13cyclictest18637-21kthreadcore20:35:180
3225799171,13cyclictest15693-21apt-get20:00:130
3225799171,12cyclictest12148-21ntp_states23:45:260
3225799171,12cyclictest12148-21ntp_states23:45:250
3227799161,8cyclictest32311-21latency_hist19:10:012
3226899166,6cyclictest0-21swapper/120:40:121
3226899166,6cyclictest0-21swapper/120:40:121
3226899163,9cyclictest0-21swapper/119:15:131
3226899162,12cyclictest31861-21ps20:50:191
3226899161,8cyclictest3707-21/usr/sbin/munin20:55:251
3225799169,5cyclictest7855-21cut19:50:210
3225799162,9cyclictest0-21swapper/019:40:140
32257991611,3cyclictest29113-21tail20:10:290
3225799161,12cyclictest8185-21ntp_states19:15:260
3225799161,10cyclictest32157-21latency_hist19:10:020
3225799161,10cyclictest32157-21latency_hist19:10:020
32257991610,4cyclictest9139-21irqrtprio21:00:180
3226899131,8cyclictest32351-21latency_hist19:10:021
3226899131,8cyclictest32351-21latency_hist19:10:021
3228599102,5cyclictest0-21swapper/319:10:013
3228599102,5cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional