You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-19 - 15:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Aug 19, 2026 12:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
130372141104,26sleep30-21swapper/307:07:353
113222141105,25sleep10-21swapper/107:05:101
131082138104,24sleep00-21swapper/007:08:300
129712138102,25sleep20-21swapper/207:06:432
281492720,2chrt0-21swapper/009:53:530
290432620,1sleep30-21swapper/309:00:153
217482610,2sleep30-21swapper/310:12:503
123632600,2sleep20-21swapper/209:18:322
182112580,2sleep20-21swapper/210:58:162
121512550,2sleep30-21swapper/312:05:183
286872460,1sleep10-21swapper/112:17:411
83992440,2chrt0-21swapper/010:03:000
103892440,1sleep00-21swapper/012:04:360
231112430,2chrt0-21swapper/010:37:470
1349899437,20cyclictest8714-21sh12:02:551
1349899431,29cyclictest32699-21diskmemload10:47:361
1349899431,24cyclictest9104-21ssh12:27:001
88802420,3sleep213495-21cyclictest12:03:052
279612420,2chrt0-21swapper/311:53:153
1349899421,28cyclictest32699-21diskmemload11:48:241
1349899421,28cyclictest32699-21diskmemload11:48:231
1349899421,28cyclictest32699-21diskmemload10:21:271
1349899421,27cyclictest32699-21diskmemload10:08:511
1349899421,26cyclictest0-21swapper/111:28:331
1349899420,31cyclictest29622-21sh09:55:071
199742410,1sleep30-21swapper/308:40:003
1349899411,27cyclictest32699-21diskmemload11:52:131
1349899411,27cyclictest32699-21diskmemload09:54:311
1349899411,26cyclictest32699-21diskmemload09:33:551
1349899411,26cyclictest32699-21diskmemload09:33:551
1349899411,26cyclictest21478-21ssh10:12:351
1349899411,23cyclictest32699-21diskmemload09:21:271
1349899410,21cyclictest32699-21diskmemload10:25:041
1349899401,27cyclictest32699-21diskmemload10:42:531
1349899401,27cyclictest30651-21sh11:07:111
1349899401,27cyclictest30651-21sh11:07:111
1349899401,27cyclictest20932-21sh12:11:391
1349899401,26cyclictest32699-21diskmemload12:30:561
1349899401,26cyclictest32699-21diskmemload12:30:551
1349899401,24cyclictest9805-21sh09:16:001
1349899401,22cyclictest26294-21sh09:28:361
1349899401,21cyclictest32699-21diskmemload11:39:591
1349899401,21cyclictest32699-21diskmemload11:14:431
1349899401,21cyclictest29129-21perl09:00:151
1349899400,29cyclictest13794-21sh11:42:351
1349899400,23cyclictest5868-21ssh09:13:511
1349899391,26cyclictest32699-21diskmemload11:02:561
1349899391,22cyclictest8208-21sh11:15:111
1349899391,21cyclictest20795-21ssh10:35:291
1349899391,21cyclictest16506-21sh10:56:341
1349899391,20cyclictest32699-21diskmemload12:05:421
1349899391,20cyclictest32699-21diskmemload10:52:301
1349899390,23cyclictest0-21swapper/110:15:141
1349899390,21cyclictest1815-21ssh12:21:261
1349899389,15cyclictest32699-21diskmemload09:45:301
1349899381,20cyclictest32699-21diskmemload12:38:261
1349899381,20cyclictest32699-21diskmemload11:59:151
1349899381,20cyclictest15055-21ssh09:44:201
1349899381,19cyclictest20169-21ssh11:23:471
1349899380,25cyclictest1513-21seq11:33:501
1349899380,22cyclictest8458-21ssh10:03:041
1349899380,21cyclictest32699-21diskmemload09:37:191
1349899371,19cyclictest15973-21ssh10:32:251
1349899361,21cyclictest30562-21seq07:49:451
1349899340,19cyclictest15289-21seq07:13:251
1349899331,19cyclictest636-21runrttasks08:17:061
13500993228,3cyclictest331rcuc/311:34:323
1349899321,16cyclictest0-21swapper/108:30:111
13500993112,11cyclictest34-21ksoftirqd/311:40:343
1349899311,27cyclictest23420-21cat07:30:241
1349899311,16cyclictest0-21swapper/108:25:261
13500993025,3cyclictest331rcuc/312:01:593
1349899302,25cyclictest32265-21grep07:50:251
1349899302,25cyclictest22723-21grep08:45:161
1349899302,25cyclictest22723-21grep08:45:151
1349899300,17cyclictest0-21swapper/107:40:251
309302290,1sleep20-21swapper/211:30:512
278742290,3sleep334-21ksoftirqd/310:17:113
1350099295,7cyclictest34-21ksoftirqd/310:39:583
1350099290,12cyclictest0-21swapper/311:06:073
1350099290,12cyclictest0-21swapper/311:06:063
1349899299,17cyclictest30927-21expr09:05:141
87232280,10sleep334-21ksoftirqd/312:26:383
50122280,3sleep334-21ksoftirqd/308:05:133
1350099289,10cyclictest34-21ksoftirqd/312:30:153
1350099289,10cyclictest34-21ksoftirqd/312:30:153
1350099283,7cyclictest34-21ksoftirqd/312:36:483
1349899289,16cyclictest8644-21seq08:12:041
1349899281,24cyclictest12943-21grep08:20:251
1349899281,13cyclictest0-21swapper/108:53:211
1349899280,15cyclictest0-21swapper/108:35:141
1350099274,7cyclictest34-21ksoftirqd/311:26:073
1350099273,7cyclictest34-21ksoftirqd/309:27:143
1350099273,4cyclictest34-21ksoftirqd/311:38:233
1350099272,6cyclictest34-21ksoftirqd/309:10:093
13500992723,3cyclictest331rcuc/310:00:593
13500992723,3cyclictest331rcuc/309:43:073
13500992723,3cyclictest331rcuc/309:31:423
13500992723,3cyclictest331rcuc/309:31:413
13500992722,3cyclictest331rcuc/312:18:093
13499992715,7cyclictest52750irq/139-enp3s0-09:45:152
1349899278,16cyclictest5020-21cstates08:05:141
1349899278,16cyclictest21013-21uname08:40:211
1349899278,16cyclictest1561-21pmu-power07:55:211
1349899272,22cyclictest24684-21grep07:35:161
1349899271,23cyclictest4624-21sleep08:04:591
1349899271,23cyclictest18241-21tune2fs07:20:151
1349899270,24cyclictest21064-21/usr/sbin/munin07:25:221
1350099266,7cyclictest34-21ksoftirqd/310:50:243
1350099266,5cyclictest331rcuc/310:49:033
1350099264,6cyclictest34-21ksoftirqd/309:16:163
1350099263,7cyclictest34-21ksoftirqd/312:12:483
1350099263,6cyclictest34-21ksoftirqd/309:22:293
13500992622,3cyclictest331rcuc/311:47:543
13500992622,3cyclictest331rcuc/311:47:543
13500992622,3cyclictest331rcuc/309:45:313
13500992621,3cyclictest331rcuc/312:20:453
1349999261,23cyclictest0-21swapper/207:14:142
1349899261,22cyclictest27511-21/usr/sbin/munin08:55:221
1349899261,21cyclictest16318-21cat07:15:171
1349799261,1cyclictest0-21swapper/007:14:110
318612250,2sleep20-21swapper/209:05:232
285342250,2sleep00-21swapper/007:44:590
1350099258,10cyclictest34-21ksoftirqd/310:40:363
1350099257,9cyclictest141rcu_preempt10:34:273
1350099254,7cyclictest34-21ksoftirqd/310:21:593
1350099253,6cyclictest424-21sh11:56:273
1350099252,6cyclictest15112-21timerandwakeup10:55:263
13500992521,3cyclictest331rcuc/311:01:433
1350099251,7cyclictest34-21ksoftirqd/311:12:453
1350099251,23cyclictest0-21swapper/307:15:513
1350099251,23cyclictest0-21swapper/307:13:473
13497992515,8cyclictest17183-21ssh09:21:330
28362240,3sleep30-21swapper/309:35:143
227282240,11sleep228-21ksoftirqd/211:01:012
1350099246,6cyclictest34-21ksoftirqd/310:25:213
1350099243,7cyclictest34-21ksoftirqd/310:06:423
1350099243,7cyclictest34-21ksoftirqd/307:39:003
1350099242,6cyclictest34-21ksoftirqd/311:15:553
13500992420,3cyclictest331rcuc/309:55:253
1349799240,1cyclictest0-21swapper/007:15:040
54162230,9sleep228-21ksoftirqd/210:25:092
157382230,3chrt28-21ksoftirqd/208:30:002
1350099234,6cyclictest34-21ksoftirqd/307:42:483
1350099232,7cyclictest34-21ksoftirqd/307:33:463
13500992319,3cyclictest331rcuc/311:22:163
13500992319,3cyclictest331rcuc/309:54:153
13500992318,4cyclictest331rcuc/308:45:513
13500992318,4cyclictest331rcuc/308:45:513
13499992316,4cyclictest25710-21grep12:15:212
1349999231,10cyclictest285362date07:44:592
1350099222,8cyclictest34-21ksoftirqd/307:26:233
1350099221,7cyclictest34-21ksoftirqd/307:47:303
13500992217,4cyclictest331rcuc/308:24:483
13499992211,7cyclictest52750irq/139-enp3s0-12:25:132
1349999221,10cyclictest66050lldpd11:24:032
1349799222,14cyclictest20444-21ssh11:00:130
13497992214,6cyclictest19040-21ntp_states12:10:220
13497992212,7cyclictest22449-21df_inode07:30:130
184952210,1sleep20-21swapper/207:20:182
13500992118,2cyclictest331rcuc/308:30:583
1350099211,7cyclictest34-21ksoftirqd/308:52:353
1350099211,7cyclictest34-21ksoftirqd/307:55:413
1350099211,7cyclictest34-21ksoftirqd/307:22:063
13500992116,4cyclictest34-21ksoftirqd/308:20:013
13500992116,3cyclictest331rcuc/307:50:163
13499992115,4cyclictest8915-21sendmail09:40:002
13499992113,6cyclictest9366-21awk11:15:232
13499992112,7cyclictest24013-21sed11:50:152
13499992112,6cyclictest24543-21grep09:50:232
13499992112,6cyclictest24463-21cut12:15:012
1349999211,17cyclictest21193-21ssh09:25:022
1349799212,10cyclictest4825-21unixbench_singl11:35:270
13497992114,5cyclictest11489-21ssh12:29:240
13497992113,6cyclictest3076-21seq10:23:030
13497992113,6cyclictest27202-21seq10:16:320
13497992113,6cyclictest23148-21ntp_states08:45:210
13497992113,6cyclictest23148-21ntp_states08:45:210
13497992113,6cyclictest1719-21sensors_temp07:55:220
13497992112,6cyclictest52950irq/141-enp3s0-11:32:410
13497992112,6cyclictest52950irq/141-enp3s0-09:10:240
13497992112,6cyclictest278-21systemd-journal09:05:010
13497992112,6cyclictest21044-21awk08:40:200
13497992111,8cyclictest18411-21sed07:20:170
315262200,8sleep012-21ksoftirqd/010:44:260
1350099203,14cyclictest30907-21tr09:05:133
1350099202,4cyclictest34-21ksoftirqd/308:25:153
13500992016,3cyclictest331rcuc/308:56:023
13500992011,6cyclictest3607-21ntp_kernel_err08:00:213
1349999201,9cyclictest6858-21ssh10:50:102
1349999201,7cyclictest17555-21ssh10:34:002
13499992012,5cyclictest26945-21grep10:40:222
13499992011,6cyclictest31219-21ntp_states11:55:222
13499992010,7cyclictest15843-21cut09:45:012
1349999200,10cyclictest1072-21ls10:45:212
1349799208,5cyclictest30480-21expr11:55:150
1349799202,8cyclictest3738-21ssh10:00:010
13497992013,4cyclictest74972sleep009:15:120
13497992013,4cyclictest636-21runrttasks11:15:080
13497992013,4cyclictest32699-21diskmemload10:10:240
13497992012,6cyclictest32699-21diskmemload11:30:010
13497992012,6cyclictest18133-21sed11:45:230
13497992012,6cyclictest18133-21sed11:45:230
13497992012,6cyclictest13630-21seq08:25:090
13497992012,6cyclictest13094-21seq08:21:150
13497992012,6cyclictest12900-21df12:30:140
13497992012,6cyclictest12900-21df12:30:130
13497992011,7cyclictest8899-21pluginstate08:15:010
1349799200,4cyclictest0-21swapper/010:57:510
196282191,16sleep20-21swapper/211:46:412
196282191,16sleep20-21swapper/211:46:412
13500991915,3cyclictest331rcuc/308:44:203
13500991914,4cyclictest331rcuc/308:12:013
1349999198,7cyclictest52750irq/139-enp3s0-10:05:122
1349999197,8cyclictest0-21swapper/207:30:132
1349999195,5cyclictest0-21swapper/212:05:212
1349999192,13cyclictest3126-21apt-get11:35:132
1349999192,12cyclictest22019-21apt-get11:25:132
13499991912,5cyclictest2607-21seq11:10:222
13499991912,5cyclictest17111-21grep07:15:242
13499991911,6cyclictest28430-21expr09:30:142
13499991911,6cyclictest28430-21expr09:30:142
13499991911,6cyclictest25581-21cat08:50:242
13499991911,4cyclictest0-21swapper/208:40:132
13499991910,7cyclictest24780-21diskstats10:15:172
13499991910,6cyclictest29209-21sed09:00:152
1349799199,6cyclictest1174-21ntp_offset10:45:220
1349799197,10cyclictest12681-21uniq10:30:150
1349799196,3cyclictest141rcu_preempt11:10:150
13497991914,3cyclictest30600-21sh07:50:000
13497991913,4cyclictest28096-21sh11:53:230
13497991912,5cyclictest12-21ksoftirqd/012:40:000
13497991911,6cyclictest5109-21seq09:36:140
13497991911,6cyclictest32183-21ls09:05:260
13497991911,6cyclictest26316-21cut08:55:000
13497991911,6cyclictest25686-21date12:15:200
13497991911,6cyclictest17349-21seq08:30:500
1349799190,9cyclictest16157-21/usr/sbin/munin09:45:250
1349999189,7cyclictest28001-21cat11:05:182
1349999189,7cyclictest28001-21cat11:05:182
1349999188,7cyclictest11369-21grep11:40:262
1349999182,14cyclictest640-21snmpd09:10:242
1349999182,11cyclictest640-21snmpd07:50:112
13499991813,3cyclictest7331-21sh10:01:562
13499991813,3cyclictest29298-21idleruntime07:45:172
13499991812,4cyclictest20046-21ntp_offset10:35:192
13499991811,5cyclictest636-21runrttasks09:55:052
13499991811,5cyclictest301-21mailstats12:20:232
13499991811,5cyclictest18098-21cut10:10:152
13499991811,5cyclictest14892-21grep09:20:172
13499991811,5cyclictest1369-21sed07:55:192
13499991810,6cyclictest8287-21cat08:10:242
13499991810,6cyclictest19413-21grep12:35:122
13499991810,6cyclictest12321-21sed08:20:192
1349999180,9cyclictest14206-21grep12:30:262
1349999180,9cyclictest14206-21grep12:30:262
1349799189,6cyclictest25151-21grep07:35:200
1349799188,8cyclictest9903-21gltestperf09:40:160
1349799182,13cyclictest15172-21tr11:20:120
13497991813,3cyclictest32695-21rm11:09:130
13497991813,3cyclictest32695-21rm11:09:130
13497991813,3cyclictest31248-21fschecks_time07:50:150
13497991813,3cyclictest24524-21sh09:26:510
13497991811,5cyclictest19479-21seq08:35:440
1349799181,12cyclictest6488-21ssh10:25:180
13497991810,6cyclictest7010-21cut10:50:120
13497991810,6cyclictest645-21grep12:20:260
13497991810,6cyclictest5228-21cut08:05:150
13497991810,6cyclictest28610-21cut09:30:160
13497991810,6cyclictest28610-21cut09:30:160
13497991810,6cyclictest26899-21cut08:55:160
13497991810,6cyclictest10818-21find10:05:120
1349999172,10cyclictest26511-21apt-get08:55:132
13499991712,3cyclictest31968-21latency10:20:192
13499991712,3cyclictest22209-21/usr/sbin/munin08:45:182
13499991712,3cyclictest22209-21/usr/sbin/munin08:45:172
1349999171,11cyclictest9140-21apt-get08:15:122
1349999171,11cyclictest4868-21taskset08:05:122
13499991710,5cyclictest560-21dbus-daemon08:40:002
13499991710,5cyclictest24292-21cat07:35:122
1349799179,6cyclictest20733-21load07:25:180
1349799171,5cyclictest10763-21ls11:40:200
13497991712,3cyclictest2701-21/usr/sbin/munin08:00:270
13497991712,3cyclictest11961-21if_err_enp3s012:05:170
1349999169,5cyclictest16233-21perl08:30:152
1349999163,4cyclictest3582-21netstat08:00:202
1349799168,6cyclictest9523-21fschecks_count08:15:160
13499991510,4cyclictest21900-21latency_hist07:30:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional