You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 04:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackaslot8s.osadl.org (updated Wed Mar 25, 2026 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305721630,2sleep20-21swapper/222:30:182
58042159122,26sleep00-21swapper/019:07:190
56712158105,28sleep10-21swapper/119:05:371
4111214698,23sleep30-21swapper/319:05:103
57962141105,25sleep20-21swapper/219:07:142
169162970,3sleep10-21swapper/123:54:001
262092640,2sleep10-21swapper/121:34:541
309412610,2sleep30-21swapper/322:02:223
109462570,2sleep20-21swapper/223:00:202
79902560,3sleep10-21swapper/100:11:021
229562550,2sleep30-21swapper/322:20:243
298912540,1sleep10-21swapper/123:15:041
266082540,1sleep20-21swapper/222:47:562
257492510,1sleep30-21swapper/323:11:043
181762510,1sleep10-21swapper/123:30:171
167932510,1sleep00-21swapper/021:27:080
189192500,1sleep30-21swapper/320:55:003
37452490,1sleep30-21swapper/300:32:473
184092490,2sleep30-21swapper/323:30:193
14722470,7sleep3331rcuc/323:41:463
157302460,1sleep20-21swapper/221:26:052
170802450,2chrt0-21swapper/221:51:382
83142440,2sleep00-21swapper/000:35:410
59232410,2chrt0-21swapper/022:32:210
4152410,1sleep10-21swapper/100:30:161
165582390,1sleep30-21swapper/323:29:263
6283993517,6cyclictest22-21ksoftirqd/100:00:001
628299354,6cyclictest560-21dbus-daemon19:30:010
6282993320,11cyclictest20906-21ssh00:20:420
88452310,1sleep10-21swapper/123:23:181
628399301,6cyclictest29021-21apt-get22:50:141
183432290,2sleep00-21swapper/023:05:270
6283992811,9cyclictest22-21ksoftirqd/122:11:171
628399277,8cyclictest22-21ksoftirqd/121:18:411
628399277,10cyclictest22-21ksoftirqd/100:00:151
628399276,10cyclictest22-21ksoftirqd/100:21:071
6283992720,5cyclictest22-21ksoftirqd/123:00:001
6283992714,4cyclictest22-21ksoftirqd/122:29:501
628399270,7cyclictest22-21ksoftirqd/123:13:581
6282992711,10cyclictest52750irq/139-enp3s0-00:15:120
628399266,9cyclictest22-21ksoftirqd/100:26:171
628399266,2cyclictest141rcu_preempt00:38:101
628399265,9cyclictest22-21ksoftirqd/123:07:081
628399263,9cyclictest22-21ksoftirqd/100:08:431
628399262,17cyclictest14812-21taskset22:15:111
628399261,12cyclictest22-21ksoftirqd/122:34:031
279732260,3sleep0131rcuc/021:35:220
6285992514,6cyclictest4269-21ssh23:20:113
6284992518,5cyclictest13763-21sh21:50:002
6284992515,7cyclictest1934-21df22:05:152
628399257,9cyclictest22-21ksoftirqd/123:03:041
628399256,9cyclictest22-21ksoftirqd/123:26:311
628399256,9cyclictest22-21ksoftirqd/122:05:231
628399256,9cyclictest22-21ksoftirqd/121:54:191
628399256,9cyclictest22-21ksoftirqd/121:47:581
628399256,11cyclictest22-21ksoftirqd/121:12:311
628399255,10cyclictest22-21ksoftirqd/123:42:131
628399255,10cyclictest22-21ksoftirqd/122:46:451
628399255,10cyclictest22-21ksoftirqd/100:16:461
628399254,11cyclictest22-21ksoftirqd/122:00:551
628399254,10cyclictest22-21ksoftirqd/121:37:351
628399253,10cyclictest22-21ksoftirqd/122:44:561
628399253,10cyclictest22-21ksoftirqd/121:22:261
6283992512,3cyclictest22-21ksoftirqd/123:38:351
6283992511,4cyclictest22-21ksoftirqd/121:43:361
628599248,3cyclictest34-21ksoftirqd/322:42:543
628599242,14cyclictest141rcu_preempt22:25:273
628599241,16cyclictest1593-21ssh22:05:133
628499242,16cyclictest20074-21ssh21:30:122
628399247,9cyclictest22-21ksoftirqd/120:52:241
628399245,9cyclictest22-21ksoftirqd/121:58:541
628399244,9cyclictest22-21ksoftirqd/121:26:131
6283992410,4cyclictest22-21ksoftirqd/122:35:381
274962240,2sleep30-21swapper/300:01:213
14742240,2sleep0131rcuc/023:17:350
628599239,2cyclictest141rcu_preempt21:35:113
628599237,3cyclictest141rcu_preempt22:57:163
628599235,5cyclictest34-21ksoftirqd/300:13:143
628599232,17cyclictest14311-21grep21:50:133
628499231,15cyclictest23232-21sh23:10:042
628399234,9cyclictest22-21ksoftirqd/122:20:471
6283992316,5cyclictest22-21ksoftirqd/119:35:011
6283992315,5cyclictest22-21ksoftirqd/120:20:011
6283992311,3cyclictest22-21ksoftirqd/123:48:341
6283992311,3cyclictest22-21ksoftirqd/123:48:331
6283992310,2cyclictest141rcu_preempt21:03:081
52422230,3chrt0-21swapper/300:09:593
214512230,1sleep20-21swapper/221:00:012
16922230,2sleep00-21swapper/000:30:390
153672230,2sleep20-21swapper/223:52:252
96402220,1sleep00-21swapper/021:21:340
94332220,2sleep00-21swapper/022:35:140
628599227,5cyclictest141rcu_preempt22:35:133
628599224,10cyclictest34-21ksoftirqd/321:58:363
628599221,11cyclictest8741-21rm00:36:083
6284992211,10cyclictest18636-21ssh22:41:332
628399229,4cyclictest22-21ksoftirqd/120:10:291
628399224,10cyclictest22-21ksoftirqd/120:41:251
628399222,9cyclictest22-21ksoftirqd/119:20:101
628399222,13cyclictest27996-21taskset20:00:131
6283992214,6cyclictest22-21ksoftirqd/120:09:591
6283992212,3cyclictest141rcu_preempt20:25:041
628399221,11cyclictest22-21ksoftirqd/119:45:181
6282992214,6cyclictest10893-21apt23:25:120
6282992213,7cyclictest22922-21fw_packets22:45:150
6282992212,7cyclictest31204-21df_inode23:40:140
6282992212,7cyclictest19883-21fw_packets19:40:150
52872220,7sleep0121ksoftirqd/000:10:000
49172220,2sleep00-21swapper/023:20:160
316342220,2sleep00-21swapper/020:05:260
69132210,7sleep228-21ksoftirqd/222:57:232
628599216,6cyclictest34-21ksoftirqd/300:17:343
628599215,8cyclictest141rcu_preempt23:50:523
628599214,4cyclictest6280-21cyclictest19:35:213
628599214,4cyclictest34-21ksoftirqd/322:10:173
628599212,13cyclictest32270-21apt-get20:10:143
6285992113,6cyclictest20256-21python300:20:243
6285992112,7cyclictest3140-21python320:15:193
6285992112,6cyclictest8051-21wc21:20:233
628499211,6cyclictest25559-21apt-get00:25:122
6284992113,6cyclictest22346-21ls21:00:202
6284992112,7cyclictest19024-21cat00:20:132
6284992112,7cyclictest10951-21df19:20:132
6284992112,6cyclictest24527-40find00:00:002
628399214,9cyclictest22-21ksoftirqd/119:59:261
628399214,9cyclictest22-21ksoftirqd/119:59:261
628399213,7cyclictest22-21ksoftirqd/120:55:161
628399212,6cyclictest22-21ksoftirqd/120:45:141
628399212,6cyclictest22-21ksoftirqd/120:45:141
628399212,13cyclictest17265-21taskset19:35:131
6283992113,5cyclictest22-21ksoftirqd/120:25:011
628399210,7cyclictest22-21ksoftirqd/119:10:151
628399210,11cyclictest22-21ksoftirqd/120:30:161
628299212,14cyclictest25552-21diskmemload22:55:120
628299211,6cyclictest0-21swapper/022:11:410
279872210,2sleep00-21swapper/000:01:500
628599208,6cyclictest7110-21sed19:10:193
628599208,6cyclictest141rcu_preempt22:18:333
628599207,10cyclictest0-21swapper/321:45:223
628599206,5cyclictest34-21ksoftirqd/322:50:203
628599202,5cyclictest34-21ksoftirqd/300:28:343
6285992015,3cyclictest25967-21apt21:10:123
6285992013,4cyclictest25552-21diskmemload21:40:203
6285992012,4cyclictest30158-21tr23:15:113
6285992011,7cyclictest6974-21ssh23:45:353
6285992011,7cyclictest6974-21ssh23:45:353
6285992011,7cyclictest12832-21grep20:40:123
6285992011,7cyclictest12067-21awk20:35:263
6285992011,6cyclictest23013-21cut23:10:013
628499207,5cyclictest32230-21cat23:40:242
628499202,9cyclictest141rcu_preempt00:35:002
6284992014,4cyclictest5767-21timerwakeupswit23:20:242
6284992013,5cyclictest21779-21sh23:08:572
6284992012,6cyclictest32248-21find20:10:132
6284992012,6cyclictest28485-21expr20:00:172
6284992012,6cyclictest27332-21users19:55:242
6284992012,6cyclictest27332-21users19:55:242
6284992012,6cyclictest25535-21cron21:10:012
6284992012,5cyclictest278-21systemd-journal23:30:012
6284992011,7cyclictest25275-21ntp_states23:35:232
6284992011,7cyclictest12683-21date00:40:012
6284992011,6cyclictest5101-21pluginstate20:20:222
6284992010,9cyclictest11478-21ssh21:23:312
628399206,3cyclictest22-21ksoftirqd/119:50:231
628399201,6cyclictest22-21ksoftirqd/119:40:161
6283992011,7cyclictest8905-21perl19:15:161
628399200,7cyclictest22-21ksoftirqd/121:05:191
628299208,9cyclictest12-21ksoftirqd/022:01:190
6282992014,4cyclictest12-21ksoftirqd/021:00:000
6282992013,4cyclictest4025-21sh22:06:120
6282992013,4cyclictest22435-21sh21:31:050
6282992013,4cyclictest21871-21ssh23:57:190
6282992013,4cyclictest12319-21rm23:50:160
6282992012,6cyclictest7944-21forks21:45:160
6282992012,6cyclictest20465-21cat21:55:000
6282992012,6cyclictest19135-21grep23:30:250
6282992012,5cyclictest25755-21cat21:10:010
6282992011,7cyclictest24472-21seq21:57:330
6282992011,6cyclictest15147-21cat22:15:120
6282992011,6cyclictest11355-21cat20:35:200
6282992010,7cyclictest52750irq/139-enp3s0-23:10:030
6282992010,7cyclictest1986-21apt-get20:15:090
287732200,1sleep00-21swapper/021:11:300
628599196,4cyclictest141rcu_preempt23:55:263
628599195,1cyclictest141rcu_preempt21:16:543
628599192,12cyclictest14908-21apt-get19:30:103
628599192,11cyclictest19358-21apt-get20:55:113
628599192,10cyclictest24228-21ssh23:35:133
628599192,10cyclictest21621-1kworker/u9:022:45:273
6285991912,4cyclictest25552-21diskmemload21:28:373
6285991911,6cyclictest4739-21ls20:20:173
6285991911,6cyclictest20749-21cat21:30:173
6285991911,5cyclictest19774-21grep19:40:153
6285991910,7cyclictest10444-21perl23:00:153
6285991910,6cyclictest13988-21sed19:25:243
628499199,7cyclictest30454-21df20:05:142
628499199,7cyclictest15824-21ntp_states19:30:202
628499198,9cyclictest28-21ksoftirqd/221:17:262
628499192,11cyclictest8558-21apt-get22:35:042
628499191,9cyclictest30092-21ssh22:50:222
6284991914,3cyclictest18325-21processes19:35:232
6284991913,4cyclictest7865-21sh00:10:542
6284991913,4cyclictest30514-21fschecks_time23:15:142
6284991913,4cyclictest28218-21sh00:02:052
6284991911,6cyclictest32585-21ntpq00:05:212
6284991911,6cyclictest27013-21expr21:35:132
6284991911,6cyclictest22256-21cat19:45:172
6284991911,6cyclictest16238-21cat20:45:242
6284991911,6cyclictest16238-21cat20:45:242
6284991911,6cyclictest1205-21ls21:40:142
6284991911,5cyclictest12464-21grep00:15:142
6284991910,7cyclictest7432-21tr20:25:232
6283991913,5cyclictest22-21ksoftirqd/119:30:011
6282991913,4cyclictest9774-21sh00:12:520
6282991913,4cyclictest23668-21sh22:20:540
6282991913,3cyclictest26013-21sh23:35:320
6282991913,3cyclictest25378-21sh00:25:030
6282991912,4cyclictest6269-21sh21:19:570
6282991911,6cyclictest7722-21grep19:10:250
6282991911,6cyclictest7477-21cat20:25:220
6282991911,6cyclictest5826-21seq20:22:270
6282991911,6cyclictest410-21basename20:10:190
6282991911,6cyclictest11065-21perl19:20:150
6282991910,7cyclictest16854-21ntpq22:40:200
6282991910,6cyclictest5432-21df23:45:140
6282991910,6cyclictest5432-21df23:45:140
6282991910,6cyclictest22876-21timerwakeupswit19:45:240
628599189,7cyclictest23400-21cron21:05:003
628599189,6cyclictest26266-21perl19:55:143
628599189,6cyclictest26266-21perl19:55:133
628599188,5cyclictest62922sleep320:25:123
628599181,6cyclictest0-21swapper/322:31:443
6285991811,5cyclictest9870-21sed20:30:273
6285991811,5cyclictest8755-21expr19:15:143
6285991811,5cyclictest25654-21sed19:55:003
6285991811,5cyclictest14939-21cat20:45:123
6285991811,5cyclictest14939-21cat20:45:123
6285991811,5cyclictest11262-21cat19:20:173
6285991810,6cyclictest30494-21cut20:05:153
628499182,11cyclictest25552-21diskmemload22:00:112
6284991813,3cyclictest25032-21timerwakeupswit19:50:232
6284991813,3cyclictest21669-21sh23:32:522
6284991813,3cyclictest21405-21sh22:20:092
6284991812,4cyclictest6893-21hwlatdetect19:10:172
6284991812,4cyclictest21645-21sh21:55:202
6284991812,4cyclictest13863-21sendmail_mailst19:25:222
6284991811,5cyclictest3490-21vmstat20:15:232
6284991811,5cyclictest28924-21sed22:25:182
6284991811,5cyclictest15685-21memory22:15:182
6284991810,6cyclictest9456-21readlink22:10:242
6284991810,6cyclictest9174-21cat20:30:202
6284991810,6cyclictest13558-21ls20:40:192
6283991811,5cyclictest10777-21expr20:35:141
628299186,3cyclictest101002grep23:00:130
628299182,11cyclictest880-21apt-get21:40:130
628299181,8cyclictest29247-21proc_pri22:25:230
628299181,8cyclictest28989-21/usr/sbin/munin22:50:250
6282991813,3cyclictest8472-21cpu19:15:120
6282991813,3cyclictest28430-21fschecks_time20:00:150
6282991811,5cyclictest25282-21seq19:51:210
6282991811,5cyclictest23096-21seq21:01:110
6282991810,6cyclictest9361-21ntpq20:30:230
6282991810,6cyclictest16727-21seq19:34:160
6282991810,6cyclictest14927-21ls20:45:120
6282991810,6cyclictest14927-21ls20:45:110
50352180,1sleep20-21swapper/223:45:102
50352180,1sleep20-21swapper/223:45:102
628499179,6cyclictest20885-21awk19:40:252
628499171,5cyclictest27698-21users21:10:262
628499170,5cyclictest18958-21dpkg20:55:002
628299179,6cyclictest26187-21uniq19:55:140
628299179,6cyclictest26187-21uniq19:55:140
628299172,10cyclictest12734-21apt-get20:40:120
6282991712,3cyclictest17768-21load20:50:180
628299171,11cyclictest17262-21apt-get19:35:130
628599167,6cyclictest24014-21fschecks_count21:05:173
6285991611,3cyclictest29153-21swap20:00:243
628499168,6cyclictest8598-21expr19:15:132
628599159,4cyclictest22125-21cat19:45:163
628499158,5cyclictest10790-21cat20:35:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional