You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-07 - 22:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Jun 07, 2026 12:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
230092185110,27sleep20-21swapper/207:07:152
230702162102,31sleep10-21swapper/107:08:031
230302162105,27sleep30-21swapper/307:07:323
2047521600,1sleep00-21swapper/011:16:520
2047521600,1sleep00-21swapper/011:16:520
806021590,2sleep10-21swapper/110:20:031
229142159105,27sleep00-21swapper/007:06:010
81002670,3sleep20-21swapper/211:07:552
228122660,1sleep30-21swapper/309:42:233
124892520,2sleep20-21swapper/210:46:442
276652510,2sleep00-21swapper/009:21:580
274242510,1sleep20-21swapper/209:21:452
255372490,2sleep025534-21ssh11:45:130
204192490,1sleep10-21swapper/111:16:491
204192490,1sleep10-21swapper/111:16:481
17622490,1sleep10-21swapper/110:15:131
107832490,3sleep10-21swapper/109:10:081
71982450,1sleep10-21swapper/111:55:181
175182450,2sleep20-21swapper/211:15:142
175182450,2sleep20-21swapper/211:15:132
169362390,1sleep10-21swapper/112:26:041
239082380,2chrt0-21swapper/109:43:311
197192380,2chrt0-21swapper/010:04:450
280992350,2sleep00-21swapper/008:35:120
25162330,3chrt0-21swapper/010:15:200
18322320,2sleep00-21swapper/010:39:320
2349899311,27cyclictest32168-21latency_hist08:45:002
52382300,3chrt0-21swapper/308:55:213
2349899301,5cyclictest20356-21timerandwakeup08:15:252
2349899301,4cyclictest6733-21df09:00:132
2349899301,4cyclictest6733-21df09:00:132
2349899291,25cyclictest28852-21memory08:35:172
23499992821,5cyclictest515-21irqbalance08:45:303
23498992818,8cyclictest32062-21sshd10:13:372
2349799281,25cyclictest0-21swapper/109:05:171
2349799281,24cyclictest0-21swapper/108:15:261
23496992823,3cyclictest19066-21/usr/sbin/munin08:15:220
2349699281,26cyclictest0-21swapper/008:41:110
309812270,4sleep2281ksoftirqd/212:12:522
2349999271,24cyclictest0-21swapper/308:50:263
23497992724,1cyclictest0-21swapper/108:35:181
2349799271,24cyclictest0-21swapper/108:55:091
2349799271,24cyclictest0-21swapper/108:31:261
2349799270,25cyclictest0-21swapper/108:50:231
2349699271,25cyclictest0-21swapper/008:55:410
2349699271,24cyclictest0-21swapper/009:05:290
2349699271,24cyclictest0-21swapper/009:01:520
2349699271,24cyclictest0-21swapper/009:01:520
2349699271,1cyclictest0-21swapper/008:45:260
23499992621,3cyclictest16314-21ssh09:14:083
2349999261,2cyclictest0-21swapper/308:37:283
2349999261,2cyclictest0-21swapper/308:15:193
2349999261,23cyclictest0-21swapper/308:40:523
23498992622,2cyclictest9389-21strings09:05:192
23498992622,2cyclictest0-21swapper/208:50:202
2349899261,3cyclictest0-21swapper/208:46:202
2349899261,23cyclictest0-21swapper/208:55:522
2349899261,23cyclictest0-21swapper/208:31:572
23497992620,3cyclictest10440-21diskmemload10:12:081
2349799261,24cyclictest0-21swapper/109:01:301
2349799261,24cyclictest0-21swapper/109:01:291
2349799261,24cyclictest0-21swapper/108:46:211
2349799261,24cyclictest0-21swapper/108:42:201
2349699261,24cyclictest0-21swapper/009:12:210
2349699260,24cyclictest0-21swapper/008:51:200
149872260,2sleep10-21swapper/110:25:121
2349999251,23cyclictest0-21swapper/309:01:023
2349999251,23cyclictest0-21swapper/309:01:023
2349999251,23cyclictest0-21swapper/308:33:403
2349999250,23cyclictest0-21swapper/309:05:213
23498992517,5cyclictest10263-21sh07:54:592
23498992515,7cyclictest12353-21df11:35:132
2349899250,23cyclictest25187-21ssh09:44:472
2349899250,23cyclictest0-21swapper/208:20:162
23496992523,1cyclictest0-21swapper/008:31:390
223452250,1sleep10-21swapper/108:20:201
236252240,2sleep00-21swapper/008:25:010
2349999244,13cyclictest640-21snmpd11:40:113
2349999242,15cyclictest30747-21ssh11:25:123
23499992416,6cyclictest8979-21irqstats10:20:183
23498992422,1cyclictest0-21swapper/209:11:502
23498992413,9cyclictest4561-21ssh12:17:092
2349799241,9cyclictest0-21swapper/111:04:271
23497992414,7cyclictest23921-21cron09:20:001
23497992414,6cyclictest52950irq/141-enp3s0-12:05:131
2349699241,10cyclictest180832ssh12:03:340
2349999239,11cyclictest11212-21cut11:10:173
2349999232,14cyclictest636-21runrttasks11:55:133
2349999232,14cyclictest27847-21chrt07:20:113
2349999232,14cyclictest27847-21chrt07:20:113
2349899231,9cyclictest192442unixbench_multi09:15:262
124792230,10sleep228-21ksoftirqd/209:59:172
123062230,1sleep30-21swapper/312:23:093
72842220,4sleep17286-21grep09:55:151
2349999222,14cyclictest75-21kswapd012:35:113
2349999222,13cyclictest5106-21sh11:30:043
23499992220,1cyclictest0-21swapper/308:25:043
23499992217,4cyclictest34-21ksoftirqd/307:20:003
23499992212,6cyclictest11216-21vmstat10:45:273
2349999220,10cyclictest306702rm12:12:343
2349899224,4cyclictest10440-21diskmemload11:34:162
2349899221,8cyclictest17082-21sed11:15:002
23498992214,6cyclictest7572-21pluginstate11:55:212
2349799223,5cyclictest0-21swapper/109:30:011
2349799222,10cyclictest10440-21diskmemload11:21:571
23497992213,7cyclictest18780-21seq08:14:301
23497992213,7cyclictest18780-21seq08:14:301
23497992212,7cyclictest21351-21sed10:05:191
2349699222,14cyclictest5109-21ssh11:30:040
2349699222,13cyclictest27149-21sh10:10:100
23496992220,1cyclictest0-21swapper/009:44:460
23496992213,7cyclictest14202-21seq11:12:190
23496992213,6cyclictest52850irq/140-enp3s0-11:35:130
23496992211,9cyclictest640-21snmpd09:53:390
210482220,2sleep30-21swapper/311:17:273
210482220,2sleep30-21swapper/311:17:273
138412220,4sleep2271rcuc/212:00:162
2349999213,11cyclictest17627-21diskmemload09:15:133
2349999211,6cyclictest0-21swapper/311:00:213
23499992112,7cyclictest24063-21awk10:55:223
23499992111,8cyclictest0-21swapper/309:30:103
2349999211,17cyclictest13492-21ssh11:35:243
23499992110,8cyclictest3101-21cat07:35:233
2349899219,8cyclictest53050irq/142-enp3s0-09:50:122
2349899212,4cyclictest141rcu_preempt11:40:112
23498992114,5cyclictest26495-21ssh11:45:192
23498992113,6cyclictest11503-21awk07:55:222
23498992113,6cyclictest1071-21python311:50:232
23498992113,5cyclictest24878-21sh12:32:162
23498992112,6cyclictest8905-21grep12:20:202
2349899210,4cyclictest0-21swapper/211:24:542
2349799219,9cyclictest21808-21grep10:30:111
2349799219,9cyclictest18310-21readlink11:40:011
2349799212,13cyclictest22932-21ssh10:55:131
23497992114,4cyclictest26787-21/usr/sbin/munin12:10:171
23497992114,4cyclictest12752-21sh09:35:151
23497992113,6cyclictest2265-21seq11:27:401
23497992112,7cyclictest5744-21seq07:41:591
23497992110,8cyclictest21934-21sh12:30:161
23496992113,6cyclictest26983-21sed07:15:250
23496992113,5cyclictest10440-21diskmemload11:40:100
23496992112,7cyclictest560-21dbus-daemon12:25:000
23496992111,7cyclictest13480-21pmu-power09:35:220
2349699210,7cyclictest8597-21ssh10:44:340
150702210,4chrt131rcuc/010:49:190
306692200,2sleep20-21swapper/211:25:092
254692200,3chrt0-21swapper/210:32:272
23499992015,3cyclictest8134-21dump-pmu-power07:50:003
23499992012,6cyclictest781-21ntpq11:50:213
23499992011,6cyclictest21577-21head10:05:213
2349899203,4cyclictest10440-21diskmemload11:04:042
2349899203,3cyclictest141rcu_preempt10:35:112
2349899203,3cyclictest10440-21diskmemload10:20:012
23498992014,4cyclictest17564-21sh10:02:352
23498992013,5cyclictest23594-21basename10:55:172
23498992012,6cyclictest9377-21seq09:32:262
23498992012,6cyclictest811-21seq09:25:362
23498992011,6cyclictest12835-21cut09:35:162
2349799203,4cyclictest22-21ksoftirqd/111:51:281
2349799202,12cyclictest14816-21apt-get08:05:131
2349799201,9cyclictest0-21swapper/110:47:511
23497992012,6cyclictest934-21ls07:30:231
23497992012,6cyclictest4081-21seq12:16:401
23497992012,6cyclictest14534-21ls10:00:181
23497992011,6cyclictest9075-21cat07:50:181
23497992010,8cyclictest2413-21cat07:35:171
2349699204,4cyclictest0-21swapper/009:31:500
23496992015,3cyclictest22128-21uptime10:05:260
23496992015,3cyclictest12397-21sh08:00:000
23496992015,3cyclictest12-21ksoftirqd/010:25:000
23496992013,5cyclictest31459-21sh12:13:220
23496992013,5cyclictest10155-21cron11:10:000
23496992013,5cyclictest10085-21seq07:52:510
23496992012,6cyclictest7616-21date07:45:250
23496992012,6cyclictest28099-21cut10:35:000
23496992012,6cyclictest25028-21awk07:10:240
23496992012,6cyclictest18191-21seq12:27:190
23496992011,7cyclictest18673-21seq08:13:050
23496992011,7cyclictest18673-21seq08:13:050
23496992011,6cyclictest1214-21timerandwakeup11:50:240
2349999199,6cyclictest52750irq/139-enp3s0-11:05:263
2349999193,14cyclictest2051-21grep10:15:153
2349999192,14cyclictest15344-21cut08:05:153
2349999191,9cyclictest15949-21ssh10:25:193
23499991914,3cyclictest15873-21sh12:01:213
23499991913,4cyclictest3878-21sh09:28:413
23499991913,4cyclictest28121-21sh09:45:593
23499991913,4cyclictest28121-21sh09:45:593
23499991913,4cyclictest24199-21sh09:20:023
23499991913,4cyclictest14188-21sh09:35:313
23499991913,3cyclictest14976-21python310:00:223
23499991912,5cyclictest21404-21which12:05:243
23499991912,5cyclictest12395-21cron07:59:593
23499991912,5cyclictest12371-21seq09:59:103
23499991911,6cyclictest21343-21sort08:20:013
23499991911,5cyclictest10440-21diskmemload11:20:543
23499991910,6cyclictest278-21systemd-journal07:29:593
23499991910,6cyclictest2540-21date12:15:253
23499991910,6cyclictest14026-21timerwakeupswit08:00:263
2349899198,2cyclictest28-21ksoftirqd/209:46:572
2349899198,2cyclictest28-21ksoftirqd/209:46:572
23498991914,3cyclictest4214-21timerandwakeup10:40:242
23498991914,3cyclictest16514-21sh12:25:382
23498991913,4cyclictest20975-21sh12:05:202
23498991913,4cyclictest17355-21proc_pri10:50:222
23498991912,5cyclictest6468-21cat07:45:142
23498991912,5cyclictest2538-21open_files10:15:212
23498991911,6cyclictest560-21dbus-daemon07:30:002
23498991911,6cyclictest16952-21apt-get08:10:112
23498991911,6cyclictest16952-21apt-get08:10:102
23498991911,5cyclictest24258-21latency07:10:182
23498991910,7cyclictest29542-21seq07:22:452
23498991910,7cyclictest29542-21seq07:22:452
2349899190,6cyclictest10440-21diskmemload10:08:562
2349899190,10cyclictest192882ssh10:28:032
2349799198,4cyclictest41242cat10:40:231
2349799194,3cyclictest141rcu_preempt11:30:021
23497991913,3cyclictest10440-21diskmemload11:06:491
23497991912,5cyclictest27332-21seq07:17:071
23497991911,6cyclictest957-21cat09:50:181
23497991911,6cyclictest6501-21wc09:30:181
23497991911,6cyclictest1886-21cron12:40:001
23497991911,6cyclictest14072-21grep08:00:261
23497991911,5cyclictest52950irq/141-enp3s0-12:00:121
23497991910,7cyclictest19905-21cut11:40:211
23497991910,7cyclictest10594-21cat11:10:121
23497991910,6cyclictest31877-21grep11:50:001
23497991910,5cyclictest52950irq/141-enp3s0-12:20:111
2349799190,4cyclictest0-21swapper/109:24:521
2349699192,5cyclictest0-21swapper/009:45:260
2349699192,5cyclictest0-21swapper/009:45:250
2349699191,4cyclictest0-21swapper/011:29:590
23496991914,3cyclictest17175-21netstat10:50:200
23496991913,3cyclictest13143-21gltestperf08:00:150
23496991912,5cyclictest2798-21pmu-power07:35:210
23496991912,5cyclictest18634-21ntp_kernel_pll_09:15:210
23496991911,6cyclictest607-21ntp_states07:30:200
23496991911,6cyclictest31360-21awk11:00:250
23496991911,6cyclictest28881-21seq11:23:340
23496991911,6cyclictest25736-21cat08:30:010
23496991911,6cyclictest11334-21seq11:58:340
23496991911,5cyclictest52850irq/140-enp3s0-12:05:130
2349999189,7cyclictest34-21ksoftirqd/310:35:123
2349999189,6cyclictest374-21latency07:30:173
2349999189,6cyclictest278-21systemd-journal11:45:213
2349999181,8cyclictest1003-21memory09:50:183
23499991813,3cyclictest23242-21sh12:30:393
23499991812,4cyclictest19881-21sh12:29:013
23499991812,4cyclictest10440-21diskmemload10:33:163
23499991811,5cyclictest8688-21expr07:50:143
23499991811,5cyclictest4113-21ls10:40:233
23498991816,1cyclictest0-21swapper/208:25:192
23498991811,4cyclictest10440-21diskmemload12:35:082
23498991810,6cyclictest26507-21ls07:15:202
2349799182,10cyclictest21414-21packagekitd08:25:041
2349799181,9cyclictest0-21swapper/109:45:151
2349799181,9cyclictest0-21swapper/109:45:151
2349799181,4cyclictest0-21swapper/110:35:261
23497991813,3cyclictest52950irq/141-enp3s0-10:52:531
23497991813,3cyclictest52750irq/139-enp3s0-07:25:161
23497991811,5cyclictest24761-21sed07:10:241
23497991810,6cyclictest11749-21cat07:55:241
23497991810,5cyclictest29158-21cat07:20:241
23497991810,5cyclictest29158-21cat07:20:241
2349699182,11cyclictest29982-21apt-get07:25:110
2349699182,10cyclictest30717-21apt-get09:25:040
2349699181,6cyclictest141rcu_preempt10:25:300
2349699181,5cyclictest0-21swapper/012:37:460
23496991812,4cyclictest4125-21sh12:16:420
23496991812,4cyclictest23598-21sh12:31:000
23496991811,5cyclictest14796-21cat08:05:110
23496991811,5cyclictest12-21ksoftirqd/010:59:270
23496991811,4cyclictest8701-21sh09:55:310
23496991810,6cyclictest5775-21seq07:42:230
23496991810,6cyclictest29715-21seq07:24:580
23496991810,6cyclictest29715-21seq07:24:570
2349999179,5cyclictest17418-21grep08:10:163
2349999179,5cyclictest17418-21grep08:10:153
23499991712,3cyclictest30552-21sh10:12:053
23499991711,4cyclictest5475-21sed07:40:263
2349899179,5cyclictest32097-21acpi07:30:102
23498991712,3cyclictest14791-21/usr/sbin/munin08:05:222
23498991710,5cyclictest640-21snmpd08:03:202
2349799178,7cyclictest6962-21grep07:45:181
2349999161,11cyclictest23539-21apt-get07:10:133
23499991610,4cyclictest17688-21expr10:50:253
2349899169,5cyclictest2165-21expr07:35:142
2349899161,11cyclictest4112-21taskset07:40:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional