You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-06 - 23:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jan 06, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
102702232162,23sleep00-21swapper/007:09:450
101402203165,25sleep10-21swapper/107:08:051
100432202166,25sleep20-21swapper/207:06:492
101622200163,24sleep30-21swapper/307:08:233
107482550,1sleep20-21swapper/208:20:212
81092540,6sleep1221ktimersoftd/109:30:111
120682540,3sleep07-21ksoftirqd/008:25:120
1037699508,13cyclictest22645-21munin-run08:50:000
1038199411,26cyclictest24848-21cat11:20:171
146922400,2sleep10-21swapper/107:15:231
1038199391,30cyclictest17869-21cat09:50:181
1038199391,29cyclictest21295-21/usr/sbin/munin10:00:121
1038199391,24cyclictest10312-21vmstat10:45:261
1038199391,21cyclictest4714-21sshd08:05:411
1037699386,15cyclictest19489-21munin-run11:10:010
134712370,1sleep10-21swapper/108:25:251
1038199371,33cyclictest21595-21/usr/sbin/munin07:30:231
1038199371,26cyclictest27262-21/usr/sbin/munin09:00:141
1038199371,26cyclictest27262-21/usr/sbin/munin09:00:141
1038199371,24cyclictest21074-21awk10:00:011
1038199371,22cyclictest1488-21/usr/sbin/munin09:15:241
10381993712,17cyclictest835-21runrttasks11:04:341
1037699377,14cyclictest21064-21munin-run10:00:010
1037699371,12cyclictest24229-21munin-run07:40:010
1037699368,12cyclictest15455-21idleruntime-cro07:20:020
1038199354,27cyclictest28007-40dnf11:26:421
1038199354,27cyclictest28007-40dnf11:26:421
1038199351,31cyclictest7887-21awk09:30:011
1038199351,31cyclictest577-21systemd-journal12:30:011
1038199351,31cyclictest20977-21smart_sda11:10:241
1038199351,31cyclictest1674-21irqstats08:00:171
1038199351,30cyclictest800-21sshd08:24:251
1038199350,28cyclictest32416-21kworker/u8:011:42:021
1038199350,24cyclictest0-21swapper/110:30:251
1037699358,12cyclictest21701-21basename11:15:010
1038199341,30cyclictest23898-21awk11:20:011
1038199341,30cyclictest0-21swapper/107:25:241
1038199341,25cyclictest0-21swapper/110:59:271
1038199341,24cyclictest28506-21/usr/sbin/munin11:30:221
1038199341,24cyclictest22869-21/usr/sbin/munin08:50:231
1038199341,21cyclictest13541-21ntp_kernel_pll_09:40:201
1038199341,20cyclictest17518-21/usr/sbin/munin11:05:151
1038199341,20cyclictest0-21swapper/110:40:021
1038199341,18cyclictest0-21swapper/110:15:251
1038199341,18cyclictest0-21swapper/110:15:251
1037699346,13cyclictest5065-21munin-run08:10:010
1038199331,29cyclictest11895-21diskstats12:05:141
1038199331,23cyclictest26823-21cat12:40:011
1038199331,23cyclictest26823-21cat12:40:011
1038199331,21cyclictest0-21swapper/112:15:161
10381993310,20cyclictest20101-21latency_hist12:25:011
1038199330,20cyclictest3106-21/usr/sbin/munin11:45:121
1038199326,23cyclictest5287-21uptime09:20:251
1038199321,24cyclictest26651-21/usr/sbin/munin07:45:211
1038199321,21cyclictest32350-21cpuspeed_turbos10:25:131
1038199321,20cyclictest31043-21/usr/sbin/munin07:55:161
1038199321,18cyclictest10304-21/usr/sbin/munin09:35:201
1038199320,23cyclictest0-21swapper/107:35:221
1038199319,19cyclictest6907-21dump-pmu-power11:55:011
1038199316,13cyclictest87150irq/128-enp1s0-10:20:111
1038199313,16cyclictest25065-21/usr/sbin/munin08:55:231
1038199313,13cyclictest27127-21timerwakeupswit10:10:251
1038199311,27cyclictest7256-21latency_hist08:15:021
1038199311,27cyclictest28924-21cat07:50:131
1038199311,26cyclictest19478-21netstat08:40:211
1038199311,26cyclictest17663-21timerandwakeup08:35:241
1038199311,20cyclictest0-21swapper/109:05:131
1038199311,18cyclictest32416-21kworker/u8:012:12:471
1038199302,16cyclictest31672-21/usr/sbin/munin09:10:161
1038199301,27cyclictest0-21swapper/112:00:151
1038199301,26cyclictest7486-21/usr/sbin/munin08:15:231
1038199301,17cyclictest31453-21diskmemload10:44:551
1038199301,16cyclictest11586-21hddtemp_smartct10:50:161
10381993010,17cyclictest23250-21if_enp1s012:30:171
1038199292,24cyclictest21585-21/usr/sbin/munin08:45:191
1038199291,25cyclictest31051-21diskstats11:35:131
1038199291,25cyclictest23837-21df_inode10:05:141
1038199291,25cyclictest14700-21/usr/sbin/munin09:45:221
1038199291,25cyclictest14083-21/usr/sbin/munin08:30:111
1038199291,16cyclictest0-21swapper/107:40:171
1037699291,15cyclictest4716-21munin-run11:50:010
1038199281,25cyclictest7298-21expr11:55:141
1038199281,24cyclictest15667-21/usr/sbin/munin07:20:181
1038199281,24cyclictest10576-21/usr/sbin/munin07:10:151
1037699281,13cyclictest27051-21awk09:00:000
1037699281,11cyclictest16303-21strings11:00:200
28722270,2sleep09-21rcu_preempt09:15:250
1039299271,17cyclictest0-21swapper/311:20:273
1039299271,10cyclictest0-21swapper/308:02:483
1037699274,2cyclictest87250irq/129-enp1s0-07:23:370
10376992713,2cyclictest0-21swapper/008:35:160
1037699271,13cyclictest8492-21munin-run10:45:000
1039299268,9cyclictest15268-21latency_hist11:00:023
1037699268,2cyclictest0-21swapper/011:53:160
1037699268,15cyclictest28293-21awk11:30:010
1037699268,15cyclictest28293-21awk11:30:010
10376992611,3cyclictest27888-21/usr/sbin/munin10:15:240
10376992611,3cyclictest27888-21/usr/sbin/munin10:15:240
267642250,4sleep31039299cyclictest10:10:223
1037699258,3cyclictest0-21swapper/007:59:400
1037699252,3cyclictest32025-21kworker/0:210:27:380
1037699251,10cyclictest0-21swapper/008:25:000
1039299245,3cyclictest0-21swapper/307:39:483
1039299241,6cyclictest19147-21kworker/u8:311:26:443
1039299241,6cyclictest19147-21kworker/u8:311:26:443
1039299241,10cyclictest127652/usr/sbin/munin12:05:233
1037699249,3cyclictest28992-21kworker/0:109:08:140
1037699245,3cyclictest0-21swapper/007:53:080
1037699245,16cyclictest15093-21munin-run11:00:020
1037699242,9cyclictest14396-21kworker/0:010:02:240
1037699242,2cyclictest87150irq/128-enp1s0-10:05:280
1037699241,3cyclictest28992-21kworker/0:109:20:540
1039299236,7cyclictest0-21swapper/311:34:343
1039299231,9cyclictest0-21swapper/310:04:403
10392992318,3cyclictest31742-21/usr/sbin/munin11:35:203
1039299231,15cyclictest0-21swapper/308:10:223
1039299231,10cyclictest0-21swapper/311:07:363
1037699237,4cyclictest20392-21df07:30:120
1037699236,3cyclictest7-21ksoftirqd/009:45:120
1037699235,8cyclictest0-21swapper/009:10:040
1037699233,3cyclictest0-21swapper/007:45:320
1037699232,3cyclictest0-21swapper/011:56:120
1037699231,11cyclictest32681-21munin-run11:40:010
185812222,2sleep20-21swapper/211:05:202
1039299221,7cyclictest0-21swapper/309:06:323
1039299221,5cyclictest0-21swapper/312:20:133
10392992210,3cyclictest0-21swapper/307:32:463
1039299220,7cyclictest0-21swapper/310:19:443
1039299220,7cyclictest0-21swapper/310:19:443
1038899222,9cyclictest28007-40dnf11:26:432
1038899222,9cyclictest28007-40dnf11:26:432
1037699228,3cyclictest0-21swapper/008:10:120
1037699227,3cyclictest0-21swapper/009:54:120
1037699226,3cyclictest0-21swapper/007:42:440
1037699225,3cyclictest0-21swapper/009:35:560
1037699223,3cyclictest17210-21swap12:15:240
1037699222,2cyclictest87250irq/129-enp1s0-08:40:560
1037699221,3cyclictest0-21swapper/008:00:520
156492210,4sleep339-21rcuc/307:20:023
134492210,2sleep20-21swapper/210:55:172
1039299211,15cyclictest0-21swapper/311:45:243
1039299211,14cyclictest0-21swapper/307:29:213
1039299211,10cyclictest20316-21gzip09:55:223
1038899211,7cyclictest0-21swapper/210:40:152
10388992116,3cyclictest13470-21/usr/sbin/munin08:25:252
1038899211,3cyclictest0-21swapper/209:38:492
1037699215,3cyclictest0-21swapper/012:27:400
1037699215,3cyclictest0-21swapper/011:31:560
1037699215,3cyclictest0-21swapper/008:18:120
1037699215,1cyclictest0-21swapper/012:11:320
1037699213,2cyclictest0-21swapper/008:30:220
1037699212,2cyclictest87150irq/128-enp1s0-11:40:060
1037699212,2cyclictest4901-21perl10:35:160
1037699211,11cyclictest27262-21/usr/sbin/munin09:00:110
1037699211,11cyclictest27262-21/usr/sbin/munin09:00:100
1037699210,12cyclictest0-21swapper/012:30:040
1039299205,5cyclictest0-21swapper/311:02:153
1039299204,4cyclictest0-21swapper/311:10:263
1039299201,9cyclictest6441-21grep11:50:253
1039299201,6cyclictest10664-1kworker/u9:111:41:553
1039299201,5cyclictest0-21swapper/310:09:033
10392992015,3cyclictest16086-21grep07:20:163
1039299201,11cyclictest6516-21/usr/sbin/munin10:40:113
1039299201,11cyclictest4622-21memory09:20:193
1039299201,11cyclictest24672-21NetworkManager08:53:033
1039299201,11cyclictest21913-21/usr/sbin/munin11:15:223
1039299201,10cyclictest0-21swapper/309:45:033
1039299201,10cyclictest0-21swapper/309:36:563
1039299200,10cyclictest2805-21switchtime09:15:243
1037699206,3cyclictest0-21swapper/007:29:000
1037699206,3cyclictest0-21swapper/007:14:120
1037699205,3cyclictest0-21swapper/009:40:440
1037699202,3cyclictest0-21swapper/012:08:120
1037699201,9cyclictest0-21swapper/012:37:500
1037699201,9cyclictest0-21swapper/012:37:500
1037699201,3cyclictest3773-21chrt10:31:020
1039299199,2cyclictest801-21snmpd08:42:253
1039299196,4cyclictest0-21swapper/312:30:273
1039299196,3cyclictest0-21swapper/312:00:023
1039299195,4cyclictest0-21swapper/312:37:363
1039299195,4cyclictest0-21swapper/312:37:353
1039299195,3cyclictest26133-21ntp_states08:55:223
1039299194,6cyclictest763-21avahi-daemon09:30:333
1039299194,4cyclictest0-21swapper/311:57:003
1039299193,3cyclictest0-21swapper/310:22:313
10392991912,4cyclictest16529-21grep12:15:153
1039299191,16cyclictest0-21swapper/309:45:023
1039299191,11cyclictest0-21swapper/310:50:223
1039299191,10cyclictest25365-21memory07:40:173
1039299191,10cyclictest21139-21irqstats12:25:183
1039299191,10cyclictest1166-21smart_sda10:25:243
1038899194,3cyclictest14717-21cpu09:45:102
1038899191,6cyclictest0-21swapper/212:26:372
1038899191,5cyclictest0-21swapper/210:13:232
1038899191,3cyclictest0-21swapper/209:31:592
10388991912,2cyclictest86950irq/126-enp1s0-11:57:582
1038899191,11cyclictest0-21swapper/209:58:522
1037699198,3cyclictest5907-21/usr/sbin/munin09:25:170
1037699196,3cyclictest0-21swapper/012:00:040
1037699192,2cyclictest87150irq/128-enp1s0-09:34:240
1037699191,7cyclictest0-21swapper/010:51:060
1037699191,4cyclictest0-21swapper/010:45:300
1037699191,2cyclictest87150irq/128-enp1s0-10:21:500
1037699190,3cyclictest21627-21kworker/0:211:19:060
241372180,2sleep20-21swapper/212:30:322
1039299186,3cyclictest27876-21gunzip07:45:223
1039299183,6cyclictest0-21swapper/309:27:513
1039299181,7cyclictest556-21sh08:00:013
1039299181,6cyclictest0-21swapper/310:31:243
1039299181,5cyclictest0-21swapper/310:35:063
10392991812,3cyclictest18652-21chrt09:52:283
10392991811,4cyclictest0-21swapper/312:10:173
1039299181,10cyclictest0-21swapper/309:10:273
1039299181,10cyclictest0-21swapper/308:24:553
1039299180,8cyclictest0-21swapper/310:47:263
1039299180,10cyclictest0-21swapper/308:37:553
1038899181,7cyclictest4212-21grep11:45:242
1038899181,5cyclictest0-21swapper/209:21:092
1038899181,4cyclictest0-21swapper/210:33:452
1038899181,11cyclictest0-21swapper/212:36:412
1038899181,11cyclictest0-21swapper/212:36:412
1038899181,11cyclictest0-21swapper/209:44:092
1038899181,10cyclictest0-21swapper/212:20:372
1037699185,3cyclictest87150irq/128-enp1s0-11:20:260
1037699183,3cyclictest0-21swapper/012:21:400
1037699181,3cyclictest0-21swapper/010:12:120
1037699181,1cyclictest0-21swapper/008:52:270
1039299175,9cyclictest0-21swapper/308:15:193
1039299173,3cyclictest16055-21munin-run08:35:023
1039299172,3cyclictest13473-21users08:25:263
1039299172,12cyclictest4557-21sed08:05:253
1039299171,6cyclictest11404-21grep07:10:173
10392991712,2cyclictest29005-21cpuspeed_turbos07:50:133
1039299171,15cyclictest0-21swapper/309:04:433
1039299171,15cyclictest0-21swapper/309:04:433
1038899179,5cyclictest14187-21cut08:30:122
1038899176,3cyclictest86950irq/126-enp1s0-10:45:212
1038899175,4cyclictest20066-21kworker/2:211:04:022
1038899174,4cyclictest782-21in:imjournal07:19:152
1038899173,4cyclictest17388-21sed08:35:222
1038899172,6cyclictest21020-21diskstats08:45:142
1038899171,9cyclictest0-21swapper/210:25:412
1038899171,9cyclictest0-21swapper/210:17:392
1038899171,9cyclictest0-21swapper/210:17:392
1038899171,7cyclictest32343-21hddtemp_smartct09:10:152
1038899171,6cyclictest0-21swapper/211:53:592
1038899171,6cyclictest0-21swapper/210:09:132
1038899171,5cyclictest1587-21ntp_states11:40:222
1038899171,5cyclictest0-21swapper/212:10:252
1038899171,4cyclictest0-21swapper/210:00:462
1038899171,3cyclictest0-21swapper/212:05:042
10388991713,3cyclictest0-21swapper/209:25:242
10388991712,3cyclictest29472-21/usr/sbin/munin09:05:132
10388991712,3cyclictest26128-21ntp_states08:55:212
10388991712,3cyclictest24111-21/usr/sbin/munin11:20:212
10388991712,3cyclictest22574-21gltestperf11:15:142
1038899171,14cyclictest19955-21grep08:40:272
10388991711,4cyclictest10833-21grep12:00:242
10392991612,3cyclictest0-21swapper/308:45:033
1038899166,3cyclictest87250irq/129-enp1s0-07:20:152
1038899166,3cyclictest86950irq/126-enp1s0-10:35:162
1038899165,3cyclictest30384-21taskset07:50:252
1038899165,3cyclictest12050irq/123-ahci[0008:00:232
1038899164,6cyclictest0-21swapper/209:19:292
1038899162,12cyclictest19345-21grep07:25:262
1038899161,6cyclictest0-21swapper/210:23:522
1038899161,5cyclictest23217-21diskstats08:50:142
1038899161,5cyclictest21120-21ntp_states07:30:192
1038899161,5cyclictest0-21swapper/211:37:232
1038899161,3cyclictest0-21swapper/212:15:492
1038899161,3cyclictest0-21swapper/211:14:262
1038899161,3cyclictest0-21swapper/208:05:082
1038899161,14cyclictest0-21swapper/208:13:312
10388991611,3cyclictest8619-21grep08:15:232
10388991611,3cyclictest31955-21/usr/sbin/munin07:55:182
10388991611,3cyclictest29137-21sed11:30:162
10388991611,3cyclictest28426-21sshd07:47:402
10388991611,3cyclictest27791-21grep09:00:142
10388991611,3cyclictest27791-21grep09:00:142
10388991611,3cyclictest25147-21idleruntime07:40:142
10388991611,3cyclictest22256-21/usr/sbin/munin07:35:152
10388991611,3cyclictest18070-21processes09:50:202
1038899151,5cyclictest10597-21egrep07:10:122
10388991510,3cyclictest11253-21df10:50:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional