You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-12 - 15:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Apr 12, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
307632236163,23sleep10-21swapper/107:08:561
306262230163,54sleep30-21swapper/307:07:103
308452223149,24sleep20-21swapper/207:09:572
307962202164,26sleep00-21swapper/007:09:190
615121670,4sleep03092599cyclictest07:25:110
1178821510,4sleep23094199cyclictest11:16:142
50102630,1sleep00-21swapper/007:20:220
325772510,4sleep23094199cyclictest07:10:252
3093599391,12cyclictest30981-21ntp_states12:00:251
3093599361,3cyclictest3304-21perf11:00:011
3092599369,12cyclictest26871-21perf10:40:010
3092599369,12cyclictest23096-21perf11:45:010
3092599369,12cyclictest23096-21perf11:45:010
3094699351,3cyclictest18189-21cat10:20:013
30935993512,3cyclictest13902-21/usr/sbin/munin10:10:181
3093599347,3cyclictest4338-21irqstats11:00:181
3093599347,3cyclictest0-21swapper/109:30:231
3093599347,3cyclictest0-21swapper/108:06:231
3093599341,3cyclictest926-21cat07:15:011
3093599341,3cyclictest10508-21latency_hist12:30:021
30935993412,3cyclictest30862-21/usr/sbin/munin09:35:151
3093599341,11cyclictest10229-21latency_hist07:35:011
30925993410,11cyclictest15243-21munin-run09:00:000
30935993314,3cyclictest18080-21awk10:20:011
30935993313,3cyclictest10871-21grep12:30:121
30925993310,11cyclictest672-21perf07:15:010
30935993215,2cyclictest0-21swapper/108:22:271
3093599321,4cyclictest26193-21diskstats08:10:141
30935993212,3cyclictest27262-21cat10:40:131
30935993111,4cyclictest23625-21head09:15:261
30935993111,4cyclictest12846-21grep10:05:221
3093599311,11cyclictest5692-21unixbench_singl12:15:251
3093599311,10cyclictest0-21swapper/110:45:161
3093599311,10cyclictest0-21swapper/110:25:131
3093599311,10cyclictest0-21swapper/110:25:121
3092599313,13cyclictest7710-21munin-run11:10:010
30935993016,2cyclictest0-21swapper/107:23:421
30935993012,3cyclictest9646-21seq11:11:541
30935993012,3cyclictest23372-21seq08:04:281
30935993012,3cyclictest23372-21seq08:04:281
3093599301,12cyclictest8874-21munin-node08:45:121
217962300,2sleep123-21ksoftirqd/111:40:161
217962300,2sleep123-21ksoftirqd/111:40:161
3093599291,5cyclictest6065-21/usr/sbin/munin07:25:231
30935992915,2cyclictest0-21swapper/107:41:081
3093599291,3cyclictest0-21swapper/111:23:471
3093599291,16cyclictest0-21swapper/111:50:031
3093599291,13cyclictest2866-21memory12:10:171
3093599290,3cyclictest0-21swapper/111:31:551
3093599290,17cyclictest0-21swapper/110:34:151
3093599287,3cyclictest12133-21ntpq08:50:211
3093599287,2cyclictest17112-21egrep07:50:121
3093599282,16cyclictest11300-21ps11:15:231
30935992815,2cyclictest0-21swapper/111:28:371
30935992815,2cyclictest0-21swapper/107:16:461
30935992812,2cyclictest0-21swapper/108:59:441
30935992812,2cyclictest0-21swapper/108:32:101
30935992811,2cyclictest0-21swapper/109:08:371
3093599281,11cyclictest8164-21grep08:40:261
3093599280,17cyclictest0-21swapper/108:19:571
3093599279,8cyclictest808-21snmpd09:28:391
3093599271,2cyclictest0-21swapper/110:51:541
3092599278,16cyclictest18068-21munin-run10:20:020
3093599261,4cyclictest25340-21ntp_states09:20:211
3092599261,12cyclictest27824-21munin-run08:15:020
3094199258,3cyclictest0-21swapper/209:59:392
3093599258,3cyclictest0-21swapper/107:47:391
3093599254,4cyclictest19642-21diskmemload12:38:431
3093599251,3cyclictest0-21swapper/109:55:181
3093599251,17cyclictest16540-21ntp_states09:00:221
3093599251,10cyclictest0-21swapper/109:10:391
3093599251,10cyclictest0-21swapper/109:10:391
3092599251,12cyclictest16827-21munin-run07:50:000
3092599250,12cyclictest3936-21awk12:15:020
3093599249,12cyclictest0-21swapper/112:20:221
3092599245,11cyclictest0-21swapper/012:30:220
3094699231,10cyclictest0-21swapper/307:30:133
3093599236,9cyclictest827-21runrttasks11:49:181
3093599231,19cyclictest12432-21perf07:40:001
3093599231,11cyclictest0-21swapper/109:54:351
3093599231,10cyclictest0-21swapper/109:43:071
3092599230,11cyclictest27490-21awk11:55:020
3094699224,4cyclictest0-21swapper/312:25:123
3094699221,9cyclictest0-21swapper/307:25:253
3094699221,8cyclictest0-21swapper/311:31:183
3094199223,11cyclictest4743-21taskset09:48:242
3094199222,6cyclictest29621-21df_inode10:45:132
30935992216,3cyclictest20828-21uptime07:55:241
3093599221,10cyclictest0-21swapper/109:45:311
3092599221,18cyclictest5513-21munin-run11:05:010
3092599221,11cyclictest0-21swapper/007:55:010
3094699219,4cyclictest7019-21hddtemp_smartct12:20:153
3094699219,3cyclictest0-21swapper/308:22:473
3094699217,3cyclictest257582sed10:35:193
3094699216,5cyclictest0-21swapper/311:25:003
3094699212,11cyclictest22691-21/usr/sbin/munin10:30:183
3094699212,11cyclictest20492-21/usr/sbin/munin10:25:213
3094699212,11cyclictest20492-21/usr/sbin/munin10:25:213
3094699211,2cyclictest0-21swapper/311:14:013
3094699211,13cyclictest16370-21kworker/u8:112:04:213
3094699211,11cyclictest606-21/usr/sbin/munin09:40:123
3094699210,9cyclictest0-21swapper/310:05:213
3094199211,8cyclictest0-21swapper/211:39:292
30941992115,4cyclictest23803-21tr08:05:122
3093599217,3cyclictest0-21swapper/112:06:031
3093599216,3cyclictest0-21swapper/111:36:351
3093599211,10cyclictest19360-21ntp_states10:20:211
3092599219,3cyclictest29902-21/usr/sbin/munin12:00:280
3092599214,15cyclictest19647-21munin-run09:10:010
3092599213,9cyclictest0-21swapper/009:30:200
3092599211,11cyclictest23652-21/usr/sbin/munin08:05:130
67122200,16sleep20-21swapper/212:20:152
42032201,2sleep20-21swapper/208:34:332
3094699207,3cyclictest14158-21cat10:10:123
3094699206,5cyclictest0-21swapper/309:21:133
3094699206,4cyclictest0-21swapper/310:57:353
3094699205,5cyclictest0-21swapper/310:20:043
3094699204,7cyclictest0-21swapper/311:56:013
3094699203,5cyclictest0-21swapper/310:45:193
3094699202,10cyclictest839-21lldpd08:59:113
3094699201,8cyclictest28552-21sed09:30:023
3094699201,11cyclictest19249-21/usr/sbin/munin07:55:123
3094699201,11cyclictest0-21swapper/311:49:443
3094699201,11cyclictest0-21swapper/311:44:293
3094699201,11cyclictest0-21swapper/311:44:293
3094199201,13cyclictest0-21swapper/212:26:502
3093599209,3cyclictest9843-21df_inode10:00:131
3093599201,14cyclictest0-21swapper/108:26:261
3092599201,4cyclictest3648-21perf07:20:020
259722200,4sleep232-21ksoftirqd/208:10:122
3094699197,5cyclictest0-21swapper/309:15:593
3094699197,3cyclictest28185-21open_inodes10:40:213
3094699196,6cyclictest0-21swapper/312:39:043
3094699195,6cyclictest0-21swapper/309:35:543
30946991914,3cyclictest22107-21cut08:00:163
30946991914,3cyclictest22107-21cut08:00:163
3094699191,11cyclictest0-21swapper/311:19:353
3094699191,11cyclictest0-21swapper/311:07:433
3094699191,11cyclictest0-21swapper/311:02:153
3094699191,11cyclictest0-21swapper/310:51:573
3094699191,10cyclictest3339-21sshd07:16:183
3094199199,3cyclictest87650irq/129-enp1s0-11:50:202
3094199192,5cyclictest405-1kworker/u9:011:05:112
3094199191,8cyclictest1-21systemd10:20:202
3094199191,6cyclictest0-21swapper/212:04:422
3094199191,11cyclictest0-21swapper/210:19:132
3094199191,11cyclictest0-21swapper/209:22:512
3093599196,4cyclictest0-21swapper/110:39:321
3093599191,11cyclictest0-21swapper/111:07:101
3093599191,10cyclictest29842-21cat12:00:011
3092599197,3cyclictest8543-21/usr/sbin/munin12:25:140
3092599196,4cyclictest87550irq/128-enp1s0-10:50:250
3092599195,5cyclictest0-21swapper/011:15:210
3092599191,9cyclictest0-21swapper/009:15:140
3092599191,8cyclictest2643-21seq09:44:450
30925991914,3cyclictest8317-21ls12:25:010
3092599191,10cyclictest8874-21/usr/sbin/munin08:45:170
3092599191,10cyclictest7305-21/usr/sbin/munin09:55:150
3092599191,10cyclictest4149-21/usr/sbin/munin12:15:120
3092599191,10cyclictest27700-21/usr/sbin/munin11:55:230
3092599191,10cyclictest24267-21/usr/sbin/munin09:20:240
3092599191,10cyclictest0-21swapper/010:59:210
30925991910,3cyclictest6419-21cat09:50:230
270672190,5sleep339-21rcuc/311:50:253
3094699183,3cyclictest87350irq/126-enp1s0-09:10:463
3094699183,3cyclictest87350irq/126-enp1s0-09:10:463
3094699182,13cyclictest19980-21ntp_states11:35:233
3094699181,9cyclictest808-21snmpd09:33:413
3094699181,6cyclictest0-21swapper/310:02:593
3094699181,5cyclictest4540-21kworker/u8:009:57:173
3094699181,3cyclictest2903-21/usr/sbin/munin09:45:233
3094699181,11cyclictest0-21swapper/312:07:083
3094699181,11cyclictest0-21swapper/311:26:533
3094699181,10cyclictest0-21swapper/309:50:543
3094699180,12cyclictest0-21swapper/307:20:163
3094699180,10cyclictest0-21swapper/308:50:113
3094199186,3cyclictest0-21swapper/207:57:312
3094199186,2cyclictest0-21swapper/207:43:592
3094199181,6cyclictest0-21swapper/212:15:502
3094199181,6cyclictest0-21swapper/211:45:132
3094199181,5cyclictest0-21swapper/210:02:472
30941991814,2cyclictest5024-21grep11:00:272
3094199181,3cyclictest0-21swapper/209:30:222
3094199181,2cyclictest0-21swapper/209:17:112
30941991812,3cyclictest3486-21unixbench_multi12:10:242
30941991812,3cyclictest21144-21sed10:25:142
30941991812,3cyclictest21144-21sed10:25:142
30941991812,3cyclictest16872-21tr11:30:102
30941991812,3cyclictest1273-21gltestperf09:40:152
3094199181,15cyclictest0-21swapper/210:05:262
3094199181,15cyclictest0-21swapper/207:22:112
30941991811,4cyclictest13052-21idleruntime-cro08:55:002
3094199181,12cyclictest0-21swapper/212:36:122
3094199181,11cyclictest0-21swapper/211:40:312
3094199181,11cyclictest0-21swapper/211:40:312
3094199181,10cyclictest0-21swapper/210:40:072
3093599188,2cyclictest5540-21/usr/sbin/munin08:35:201
3092599187,9cyclictest87350irq/126-enp1s0-11:35:130
3092599184,7cyclictest10034-21kworker/0:207:43:590
3092599182,3cyclictest87550irq/128-enp1s0-10:45:180
3092599181,5cyclictest0-21swapper/010:31:290
3092599181,4cyclictest9500-21/usr/sbin/munin10:00:220
30925991812,3cyclictest20750-21awk07:55:250
3092599181,12cyclictest0-21swapper/010:25:510
3092599181,12cyclictest0-21swapper/010:25:510
3092599181,10cyclictest0-21swapper/011:22:220
3092599180,10cyclictest0-21swapper/008:41:200
214632180,2sleep2311ktimersoftd/209:10:252
214632180,2sleep2311ktimersoftd/209:10:252
156582181,3sleep20-21swapper/210:12:252
95692171,2sleep00-21swapper/011:10:540
79942171,2sleep20-21swapper/207:29:412
45732171,2sleep00-21swapper/009:46:190
3094699173,8cyclictest0-21swapper/312:13:293
3094699172,5cyclictest10703-40dnf08:48:203
3094699171,4cyclictest808-21snmpd08:33:293
30946991712,3cyclictest4479-21/usr/sbin/munin08:35:143
3094699171,15cyclictest0-21swapper/307:41:583
3094699171,13cyclictest4813-21head12:15:153
30946991711,3cyclictest24253-21gltestperf08:05:153
3094699170,5cyclictest0-21swapper/312:34:153
3094199176,2cyclictest25939-21kworker/u8:008:59:252
3094199175,3cyclictest30-21rcuc/208:25:232
3094199171,7cyclictest31605-21timerandwakeup08:20:252
3094199171,6cyclictest28373-21hddtemp_smartct11:55:162
3094199171,6cyclictest16409-21grep07:45:262
3094199171,3cyclictest0-21swapper/208:49:522
3094199171,3cyclictest0-21swapper/208:17:282
30941991713,3cyclictest0-21swapper/212:34:132
30941991712,3cyclictest21564-21cut08:00:122
30941991712,3cyclictest21564-21cut08:00:122
3094199171,15cyclictest0-21swapper/207:50:242
3094199171,14cyclictest0-21swapper/207:33:152
3094199170,7cyclictest24657-21sh10:35:012
3092599173,3cyclictest10428-21kworker/0:112:35:130
3092599172,5cyclictest12428-21munin-run07:40:000
3092599171,7cyclictest32279-21sort09:35:250
3092599171,7cyclictest0-21swapper/010:40:160
3092599171,6cyclictest0-21swapper/011:32:220
3092599171,6cyclictest0-21swapper/009:26:070
3092599171,6cyclictest0-21swapper/008:25:160
3092599171,4cyclictest28042-21/usr/sbin/munin08:15:240
30925991713,2cyclictest19642-21diskmemload10:11:180
3092599171,14cyclictest0-21swapper/008:02:290
3092599171,14cyclictest0-21swapper/008:02:290
30925991711,3cyclictest20447-21grep10:25:020
30925991711,3cyclictest15650-21pmu-power11:25:200
30925991711,3cyclictest13098-21timerandwakeup10:05:250
296912171,2sleep30-21swapper/308:16:043
189012171,2sleep30-21swapper/307:53:373
3094699165,3cyclictest87350irq/126-enp1s0-08:40:213
3094699161,8cyclictest0-21swapper/309:05:133
3094699161,5cyclictest975-21memory08:25:183
3094699161,14cyclictest0-21swapper/307:10:453
30946991611,3cyclictest11861-21timerwakeupswit07:35:273
3094699160,9cyclictest0-21swapper/309:00:163
3094199165,5cyclictest0-21swapper/211:22:392
3094199162,7cyclictest0-21swapper/210:55:472
3094199161,6cyclictest31856-21needreboot09:35:202
3094199161,6cyclictest31487-21/usr/sbin/munin10:50:192
3094199161,6cyclictest26527-21grep09:25:122
3094199161,6cyclictest18282-21head09:05:182
3094199161,6cyclictest0-21swapper/211:25:132
3094199161,4cyclictest1258-21tail12:05:252
3094199161,4cyclictest0-21swapper/209:00:192
3094199161,4cyclictest0-21swapper/208:35:142
30941991612,3cyclictest0-21swapper/211:12:182
30941991612,3cyclictest0-21swapper/210:36:472
30941991612,3cyclictest0-21swapper/209:50:032
30941991611,3cyclictest1669-21/usr/sbin/munin07:15:172
3092599161,8cyclictest0-21swapper/011:47:020
3092599161,2cyclictest0-21swapper/008:35:090
3092599161,14cyclictest0-21swapper/009:04:350
30925991611,3cyclictest3535-21mailstats08:30:250
30925991611,3cyclictest32696-21grep12:05:160
30925991611,3cyclictest11074-21acpi08:50:110
30925991610,3cyclictest30337-21cpuspeed_turbos08:20:130
214842161,2sleep00-21swapper/009:10:310
214842161,2sleep00-21swapper/009:10:310
3094699154,4cyclictest808-21snmpd07:45:163
3094699153,11cyclictest401ktimersoftd/308:10:123
3094199152,10cyclictest10954-21fschecks_time07:35:162
3094199151,5cyclictest0-21swapper/208:40:142
30925991511,3cyclictest0-21swapper/007:30:400
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional