You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-30 - 17:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat May 30, 2026 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
314182188107,74sleep20-21swapper/207:08:462
314212175101,67sleep00-21swapper/007:08:480
314422166110,28sleep10-21swapper/107:09:051
313472162106,27sleep30-21swapper/307:07:513
248252740,3chrt0-21swapper/011:15:140
265892660,11chrt12-21ksoftirqd/009:15:190
313202630,2sleep00-21swapper/012:32:270
29782580,2sleep10-21swapper/112:10:571
117762560,1sleep10-21swapper/112:18:071
232912550,1sleep30-21swapper/311:14:173
16602540,3sleep1211rcuc/109:45:141
302542530,2sleep30-21swapper/309:18:113
307122490,2sleep30-21swapper/310:30:223
239342490,2sleep10-21swapper/109:37:141
211742480,1sleep20-21swapper/210:48:072
89932470,2sleep30-21swapper/312:15:263
316042460,1sleep10-21swapper/110:07:051
84022450,3chrt0-21swapper/211:26:442
256922450,4chrt0-21swapper/310:02:503
285442440,2sleep10-21swapper/110:29:481
306182430,1sleep00-21swapper/008:20:220
306182430,1sleep00-21swapper/008:20:220
121942430,1sleep00-21swapper/011:05:230
323362420,1sleep10-21swapper/112:09:371
304002420,1sleep20-21swapper/211:43:452
38782410,2sleep20-21swapper/212:35:462
288622410,1sleep10-21swapper/111:42:121
154892410,2sleep30-21swapper/309:00:173
94862400,2sleep00-21swapper/010:15:150
282602400,1sleep30-21swapper/308:15:193
149842390,2sleep30-21swapper/311:31:383
31789993716,19cyclictest26027-21sh10:51:161
294122360,3chrt29414-21tr10:05:223
266212360,9sleep334-21ksoftirqd/311:15:533
31789993514,4cyclictest23656-21sh09:13:131
31789993514,4cyclictest23656-21sh09:13:131
31789993311,7cyclictest32408-21hwlatdetect07:10:181
31789993311,3cyclictest20024-21mailstats11:35:231
3178999330,1cyclictest0-21swapper/109:30:151
265042330,1sleep20-21swapper/210:27:452
214222330,2chrt0-21swapper/310:24:213
31789993214,3cyclictest14520-21sh10:19:061
3178999319,5cyclictest1925-21expr12:35:121
31789993113,3cyclictest29991-21sh10:55:031
31789993110,5cyclictest22583-21sed12:25:261
271272310,1sleep10-21swapper/109:15:241
3178999302,12cyclictest8073-21apt-get09:50:131
31789993013,3cyclictest10246-21sh09:26:581
31789993012,3cyclictest32159-21sh12:33:201
31789993011,5cyclictest22-21ksoftirqd/109:00:131
31789993011,4cyclictest17691-21sh11:34:231
223072300,6sleep30-21swapper/310:25:013
3178999299,6cyclictest307-21sed11:20:251
3178999299,6cyclictest307-21sed11:20:251
31789992911,3cyclictest17067-21sh11:57:391
3178999291,10cyclictest300-21apt-get09:20:151
3179199281,24cyclictest14086-21diskstats09:30:153
3178999289,6cyclictest11735-21seq07:36:321
3178999289,6cyclictest11735-21seq07:36:321
31788992818,7cyclictest28563-21ssh10:29:490
35842270,2sleep00-21swapper/012:35:280
31789992718,5cyclictest636-21runrttasks11:00:121
31789992713,2cyclictest18786-21/usr/sbin/munin07:55:131
3178999271,11cyclictest10102-21apt-get10:40:121
3178899271,24cyclictest0-21swapper/009:30:090
31790992622,2cyclictest7990-21strings09:25:202
3178999262,7cyclictest52950irq/141-enp3s0-09:55:121
3178999260,3cyclictest3362-21ls10:10:181
324052250,3sleep228-21ksoftirqd/207:10:182
31791992523,1cyclictest0-21swapper/309:23:493
31791992520,3cyclictest7107-21ssh09:25:133
31791992510,13cyclictest12032-21cron07:40:003
31791992510,13cyclictest12032-21cron07:39:593
31790992518,5cyclictest11391-21rm10:15:562
3179099251,12cyclictest32047-21ssh11:45:102
31788992523,1cyclictest0-21swapper/009:25:160
31788992520,3cyclictest9878-21sh07:35:000
31788992512,7cyclictest636-21runrttasks09:50:120
131782250,2sleep20-21swapper/212:19:332
31790992414,6cyclictest53050irq/142-enp3s0-10:35:132
3179099240,11cyclictest186922ssh10:21:342
3178999242,11cyclictest25244-21apt-get08:10:121
3178899247,9cyclictest321-21ssh09:20:140
298972240,2sleep20-21swapper/212:31:012
31791992314,6cyclictest19726-21ntp_states07:55:203
31791992313,7cyclictest278-21systemd-journal12:09:553
31791992313,6cyclictest28355-21cut12:30:173
31790992321,1cyclictest0-21swapper/209:30:172
31790992316,5cyclictest28-21ksoftirqd/210:55:002
31790992314,6cyclictest2470-21ntp_states07:15:212
3178899232,13cyclictest27079-21diskmemload09:40:110
31788992313,7cyclictest768-21users08:25:250
31788992312,8cyclictest14557-21cat11:55:220
243562230,10sleep334-21ksoftirqd/312:27:083
31790992211,9cyclictest22545-21unixbench-2d12:25:252
31789992214,6cyclictest15609-21python312:20:231
31789992213,7cyclictest9559-21users08:45:251
3178899226,4cyclictest66050lldpd11:34:030
124632220,2sleep00-21swapper/010:41:030
31791992116,3cyclictest24334-21rm12:03:153
31791992113,6cyclictest16671-21ssh10:45:053
31790992113,6cyclictest30271-21seq09:18:112
31790992113,6cyclictest17407-21smartctl09:05:172
31790992111,6cyclictest53050irq/142-enp3s0-12:20:132
3178899218,7cyclictest5365-21apt-get11:25:100
3178899212,14cyclictest0-21swapper/007:55:120
31788992112,7cyclictest7142-21seq09:49:300
287002210,4sleep30-21swapper/311:42:023
174882210,2sleep00-21swapper/012:21:560
47552200,2sleep10-21swapper/108:35:241
3179199204,13cyclictest15517-21cut09:55:163
31791992013,5cyclictest26863-21sed09:40:003
31791992012,6cyclictest2584-21seq11:22:363
31791992012,6cyclictest2584-21seq11:22:363
31791992012,6cyclictest10759-21ls10:15:263
31791992012,5cyclictest11295-21run-parts08:50:233
31791992011,7cyclictest1369-21cut07:15:003
3179199201,16cyclictest0-21swapper/311:55:003
31791992010,7cyclictest11642-21munin-run11:30:003
31791992010,6cyclictest52850irq/140-enp3s0-09:10:263
31791992010,6cyclictest52850irq/140-enp3s0-09:10:263
3179199200,5cyclictest0-21swapper/311:47:313
3179099202,13cyclictest2406-21kworker/u8:011:05:142
3179099202,12cyclictest5917-21apt-get08:40:122
31790992015,3cyclictest2102-21switchtime12:10:252
31790992014,4cyclictest323522sleep212:09:382
31790992014,4cyclictest31207-21sh10:55:212
31790992013,5cyclictest27836-21cron10:05:002
31790992012,6cyclictest7376-21ntp_states11:50:222
31790992012,6cyclictest18539-21cat11:35:012
31790992011,7cyclictest18947-21expr07:55:142
3178999202,12cyclictest20971-21apt-get08:00:121
31789992014,4cyclictest6229-21sh11:25:171
31789992014,4cyclictest27574-21sh11:16:511
31789992012,6cyclictest15234-21sed07:45:161
31789992012,5cyclictest10770-21cat08:50:181
31789992011,7cyclictest7068-21sensors_temp08:40:211
31789992011,6cyclictest6959-21seq11:50:171
31789992011,6cyclictest1365-21cut08:30:011
3178999200,18cyclictest25102-21ssh10:02:141
3178999200,18cyclictest18029-21gzip10:45:221
3178899202,12cyclictest19210-21ssh09:10:140
3178899202,12cyclictest19210-21ssh09:10:140
31788992014,4cyclictest14259-21tr09:00:000
31788992014,3cyclictest4569-21pmu-power10:35:230
31788992013,4cyclictest18738-21diskmemload10:55:190
31788992012,6cyclictest6587-21cut08:40:170
31788992012,6cyclictest24847-21cat10:50:250
31788992012,6cyclictest12016-21seq08:54:200
31788992010,6cyclictest7301-21grep07:25:250
3179199192,11cyclictest13145-21apt-get11:55:113
31791991914,3cyclictest24134-21hwlatdetect10:50:183
31791991913,4cyclictest12694-21sh09:53:243
31791991912,5cyclictest17411-21cut11:10:003
31791991912,4cyclictest0-21swapper/310:10:133
31791991911,6cyclictest9947-21seq10:40:033
31791991911,6cyclictest8478-21tune2fs08:45:153
31791991911,6cyclictest30616-21cut10:55:163
31791991911,6cyclictest16027-21awk07:45:253
31791991911,6cyclictest12823-21cut07:40:143
3179199191,15cyclictest31419-21cron08:25:003
3179199191,15cyclictest31419-21cron08:25:003
31791991910,6cyclictest32354-21sed08:25:173
31791991910,6cyclictest21701-21cat08:00:183
31790991914,3cyclictest6951-21mailstats07:25:232
31790991914,3cyclictest30537-21sh10:06:002
31790991914,3cyclictest27359-21sh11:16:382
31790991914,3cyclictest1918-21gltestperf09:45:162
31790991914,3cyclictest15570-21sh11:56:072
31790991914,3cyclictest1261-21sh11:21:162
31790991914,3cyclictest1261-21sh11:21:162
31790991914,3cyclictest12525-21sh09:53:132
31790991913,4cyclictest7558-21sh10:13:492
31790991913,4cyclictest32653-21sh09:20:082
31790991913,4cyclictest30489-21sh09:42:082
31790991912,5cyclictest560-21dbus-daemon07:40:002
31790991912,5cyclictest560-21dbus-daemon07:40:002
31790991912,5cyclictest5415-21ntp_kernel_err11:00:192
31790991912,4cyclictest636-21runrttasks11:38:502
31790991911,6cyclictest25191-21cat08:10:002
31790991911,6cyclictest20677-21ls12:00:172
31790991911,6cyclictest18183-21cat11:10:152
31790991911,6cyclictest13772-21ls07:40:242
31790991910,7cyclictest10072-21/usr/sbin/munin10:40:112
31790991910,6cyclictest27365-21acpi08:15:102
31790991910,6cyclictest15988-21unixbench_singl07:45:252
3179099190,17cyclictest18738-21diskmemload09:12:142
3179099190,17cyclictest18738-21diskmemload09:12:142
3178999199,6cyclictest52950irq/141-enp3s0-11:45:171
3178999198,8cyclictest20867-21sh12:00:201
3178999192,8cyclictest3822-21/usr/sbin/munin10:35:161
31789991913,4cyclictest31945-21sh09:43:341
31789991913,4cyclictest13402-21processes08:55:211
31789991911,6cyclictest2228-21sed08:30:181
31789991911,6cyclictest18530-21seq07:54:501
31789991911,5cyclictest12404-21grep07:40:121
3178899199,7cyclictest560-21dbus-daemon09:59:590
3178899197,3cyclictest141rcu_preempt12:30:010
3178899196,10cyclictest27873-21switchtime12:05:240
3178899195,5cyclictest2582-21switchtime08:30:230
3178899192,12cyclictest20407-21apt-get09:35:130
31788991912,5cyclictest18950irq/143-ahci[0012:20:010
31788991912,5cyclictest15491-21sed09:00:170
31788991911,6cyclictest560-21dbus-daemon12:15:000
31788991911,6cyclictest30795-21ssh10:06:150
31788991911,6cyclictest29259-21sh11:42:360
31788991911,6cyclictest28874-21grep08:15:250
31788991911,6cyclictest21956-21grep08:00:230
31788991911,6cyclictest1881-21df07:15:140
31788991911,6cyclictest18396-21seq07:53:050
31788991911,6cyclictest16823-21find09:05:110
31788991911,5cyclictest4664-21grep08:35:220
31788991911,4cyclictest4973-21grep11:00:150
312932190,3chrt0-21swapper/210:30:502
3179199189,6cyclictest23481-21tune2fs08:05:173
3179199183,12cyclictest4449-21wc11:00:123
31791991813,3cyclictest2411-21proc_pri08:30:213
31791991813,3cyclictest20245-21sh12:24:453
31791991812,4cyclictest22179-21sh11:37:093
31791991811,5cyclictest7031-21grep08:40:223
31791991811,5cyclictest2807-21ntp_kernel_err12:35:193
31791991810,5cyclictest692-21acpi12:10:113
31791991810,4cyclictest27230-21tr09:40:123
3179099182,11cyclictest3718-21apt-get07:20:132
3179099182,11cyclictest14704-21apt-get09:00:112
31790991813,3cyclictest16913-21sh09:55:432
31790991810,6cyclictest8272-21expr08:45:132
31790991810,6cyclictest3371-21cat08:35:012
31790991810,6cyclictest32279-21basename08:25:162
31790991810,6cyclictest21715-21expr08:00:182
31790991810,6cyclictest10047-21cat07:34:592
3178999183,6cyclictest0-21swapper/110:30:221
31789991813,3cyclictest24233-21sendmail_mailtr08:05:241
31789991811,5cyclictest29257-21cron08:20:001
31789991811,5cyclictest11612-21fschecks_time11:05:171
31789991810,6cyclictest9678-21seq07:32:351
31789991810,6cyclictest3109-21seq07:15:541
31789991810,6cyclictest30861-21expr08:20:241
31789991810,6cyclictest30861-21expr08:20:241
31789991810,6cyclictest18657-21memory11:10:181
31789991810,6cyclictest18054-21ls09:05:241
3178899189,6cyclictest7394-21open_files11:50:220
3178899181,5cyclictest31848-21/usr/sbin/munin11:20:170
3178899181,5cyclictest31848-21/usr/sbin/munin11:20:170
31788991813,3cyclictest5872-21latency_hist11:50:000
31788991812,4cyclictest12039-21latency_hist07:40:000
31788991812,4cyclictest12039-21latency_hist07:40:000
31788991811,4cyclictest5143-21sh10:11:220
31788991811,4cyclictest18738-21diskmemload12:01:380
31788991810,6cyclictest30608-21cat10:30:220
31788991810,6cyclictest22382-21cat10:00:170
31788991810,6cyclictest17356-21sed10:20:270
248582180,15sleep018738-21diskmemload11:39:550
3179199179,6cyclictest4221-21grep07:20:163
3179199179,5cyclictest10043-21sed07:35:003
31791991712,3cyclictest19234-21munin-node08:35:113
31791991712,3cyclictest1532-21acpi07:15:113
31791991711,4cyclictest7194-21timerwakeupswit07:25:253
31791991711,4cyclictest34-21ksoftirqd/307:55:003
3179199171,11cyclictest3441-21apt-get10:35:143
31791991710,5cyclictest2454-21ntpq09:45:213
31790991712,3cyclictest30095-21hddtemp_smartct08:20:162
31790991712,3cyclictest30095-21hddtemp_smartct08:20:162
31790991712,3cyclictest26370-21mailstats08:10:232
31790991712,3cyclictest10170-21/usr/sbin/munin08:50:122
31790991710,5cyclictest21361-21ls09:35:202
31790991710,5cyclictest17181-21expr07:50:192
3178999179,6cyclictest3943-21expr07:20:141
3178999171,7cyclictest18282-21ssh10:21:081
31789991710,5cyclictest6551-21cat07:25:171
3178899179,6cyclictest8055-21apt08:45:110
3178899179,6cyclictest4087-21cut07:20:150
3178899179,5cyclictest32758-21open_files07:10:220
3178899179,5cyclictest14735-21expr07:45:130
3178899178,6cyclictest25328-21grep08:10:130
3178899174,3cyclictest18738-21diskmemload11:10:100
31788991711,4cyclictest17282-21fschecks_time10:45:150
145132170,1sleep20-21swapper/209:00:002
3179199169,5cyclictest25524-21expr08:10:143
3179199169,5cyclictest12552-21expr08:55:123
3179199162,10cyclictest16845-21apt-get09:05:133
3178899160,6cyclictest23623-21sed08:05:170
3179099159,4cyclictest3719-21cat08:35:132
3178899151,13cyclictest0-21swapper/007:43:400
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional