You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-15 - 01:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Jul 14, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10272239163,24sleep20-21swapper/207:06:222
9602233162,22sleep30-21swapper/307:05:313
11602206175,21sleep00-21swapper/007:08:050
317242203166,24sleep10-21swapper/107:05:041
1404998380,2cyclictest0-21swapper/208:52:572
298222620,4sleep011-21rcuc/009:25:170
1404995754,2cyclictest0-21swapper/207:14:412
161402560,1sleep10-21swapper/111:20:211
46062530,4sleep19-21rcu_preempt07:15:131
1404995047,2cyclictest0-21swapper/210:50:102
237102490,1sleep00-21swapper/010:25:140
131582460,2sleep10-21swapper/108:45:261
326842450,1sleep10-21swapper/112:00:011
1392993611,12cyclictest7763-21munin-run09:50:010
1392993610,13cyclictest7145-21munin-run08:35:020
139299349,12cyclictest15598-21awk12:35:010
139299335,13cyclictest24744-21perf09:15:000
139299335,13cyclictest24744-21perf09:15:000
51372320,1sleep20-21swapper/212:10:142
268592320,4sleep2140499cyclictest10:30:252
140499321,9cyclictest0-21swapper/207:28:162
139299321,9cyclictest8488-21kworker/0:107:25:530
139299314,12cyclictest6192-21perf11:00:020
139299313,13cyclictest8389-21munin-run11:05:020
140499300,10cyclictest0-21swapper/207:44:402
139299307,2cyclictest11284-21kworker/0:208:46:020
140499290,10cyclictest0-21swapper/212:37:072
139299291,21cyclictest7087-21kworker/0:108:40:560
1404992813,6cyclictest810-21sshd10:28:172
139299281,3cyclictest8212-21kworker/0:011:15:160
141099271,12cyclictest0-21swapper/310:51:073
140499271,9cyclictest0-21swapper/210:06:252
140499271,10cyclictest0-21swapper/211:26:412
140499271,10cyclictest0-21swapper/207:51:532
139299278,2cyclictest11284-21kworker/0:208:54:000
139299271,12cyclictest0-21swapper/007:25:010
140499261,9cyclictest0-21swapper/212:24:012
1404992612,3cyclictest87650irq/129-enp1s0-08:55:172
140499260,10cyclictest0-21swapper/209:16:382
140099260,19cyclictest0-21swapper/110:37:371
140099260,19cyclictest0-21swapper/109:34:161
139299268,15cyclictest6815-21idleruntime-cro12:15:010
139299265,3cyclictest12736-21kworker/0:007:48:130
139299262,9cyclictest26586-21kworker/0:309:25:010
139299262,7cyclictest21839-21kworker/0:011:50:430
139299262,7cyclictest13360-21kworker/0:212:37:320
139299261,9cyclictest13580-21kworker/0:008:57:530
139299261,10cyclictest32556-21kworker/0:108:22:580
213082250,2sleep081ktimersoftd/010:20:120
213082250,2sleep081ktimersoftd/010:20:120
178142250,4sleep3401ktimersoftd/310:10:203
141099257,11cyclictest0-21swapper/312:20:253
140499251,10cyclictest0-21swapper/211:43:082
140499250,11cyclictest0-21swapper/208:18:512
139299252,8cyclictest3244-21kworker/0:309:50:570
139299252,8cyclictest21839-21kworker/0:011:43:070
139299252,8cyclictest10231-21kworker/0:010:13:510
139299251,9cyclictest23062-21kworker/0:210:34:340
139299251,10cyclictest17484-21kworker/0:107:51:100
140499248,3cyclictest0-21swapper/209:22:382
140499247,3cyclictest0-21swapper/208:29:532
140499245,3cyclictest0-21swapper/211:16:572
140499241,11cyclictest0-21swapper/210:13:512
140499240,11cyclictest0-21swapper/209:54:282
140099242,4cyclictest14809-21processes12:30:231
140099241,17cyclictest0-21swapper/111:51:231
140099241,15cyclictest0-21swapper/111:13:211
140099240,6cyclictest0-21swapper/110:50:231
139299247,4cyclictest3244-21kworker/0:309:42:510
139299245,2cyclictest20023-21kworker/0:109:04:450
139299243,2cyclictest87450irq/127-enp1s0-08:36:360
139299242,8cyclictest3662-21kworker/0:010:52:280
139299241,8cyclictest12736-21kworker/0:007:37:250
141099233,5cyclictest39-21rcuc/310:05:013
141099231,16cyclictest0-21swapper/307:50:293
140499239,3cyclictest6116-21df09:45:122
140499238,5cyclictest22528-21diskmemload10:47:472
140499237,3cyclictest0-21swapper/208:49:132
140499237,3cyclictest0-21swapper/207:15:272
140499236,3cyclictest0-21swapper/210:02:052
140499235,5cyclictest484-21sed09:30:262
140499235,3cyclictest0-21swapper/208:14:172
140499231,5cyclictest774-21avahi-daemon09:35:382
140499231,11cyclictest5875-21sshd10:56:172
140099233,12cyclictest0-21swapper/110:00:141
140099232,11cyclictest17593-21irqstats10:10:171
140099231,9cyclictest0-21swapper/111:28:111
1400992319,3cyclictest0-21swapper/110:05:151
140099231,6cyclictest0-21swapper/112:19:091
140099231,5cyclictest7141-21if_enp1s011:00:181
140099231,16cyclictest0-21swapper/109:14:451
140099231,16cyclictest0-21swapper/109:14:451
140099231,16cyclictest0-21swapper/108:00:161
140099231,15cyclictest0-21swapper/111:18:451
140099231,12cyclictest0-21swapper/112:39:011
140099231,10cyclictest0-21swapper/110:15:131
139299237,7cyclictest25101-21kworker/0:310:35:400
139299232,2cyclictest12736-21kworker/0:008:04:360
139299232,2cyclictest0-21swapper/011:28:530
1392992317,3cyclictest2130-21sshd09:35:160
1392992317,3cyclictest18982-21/usr/sbin/munin10:15:220
211232220,3sleep339-21rcuc/311:30:253
211232220,3sleep339-21rcuc/311:30:253
141099229,4cyclictest0-21swapper/311:20:163
141099222,13cyclictest0-21swapper/309:49:043
141099221,10cyclictest0-21swapper/309:50:263
140499227,3cyclictest0-21swapper/208:44:132
140499227,3cyclictest0-21swapper/208:33:032
140499227,3cyclictest0-21swapper/207:55:572
140499227,3cyclictest0-21swapper/207:55:572
140499226,3cyclictest0-21swapper/210:21:332
140499226,3cyclictest0-21swapper/210:21:332
140499226,3cyclictest0-21swapper/209:05:082
140499226,2cyclictest0-21swapper/208:00:072
140499225,3cyclictest0-21swapper/211:14:172
140499225,3cyclictest0-21swapper/207:32:502
140499224,3cyclictest0-21swapper/209:13:422
140499224,3cyclictest0-21swapper/209:13:422
140499223,3cyclictest0-21swapper/208:37:512
1404992217,3cyclictest30329-21/usr/sbin/munin10:40:152
140499220,9cyclictest0-21swapper/211:59:442
140099228,4cyclictest0-21swapper/112:10:351
140099225,11cyclictest0-21swapper/110:30:011
1400992217,2cyclictest2643-21bc12:05:101
1400992215,2cyclictest0-21swapper/108:16:411
140099221,4cyclictest0-21swapper/110:58:141
140099221,2cyclictest5199-21cpu08:30:131
140099221,18cyclictest22247-21ntp_states10:20:211
140099221,18cyclictest22247-21ntp_states10:20:211
140099221,16cyclictest0-21swapper/109:39:181
140099221,12cyclictest0-21swapper/110:42:491
140099221,11cyclictest30710-21munin-run08:15:021
140099221,11cyclictest0-21swapper/112:01:101
140099221,11cyclictest0-21swapper/109:50:061
140099221,10cyclictest29412-21egrep09:25:151
139299226,7cyclictest8212-21kworker/0:011:06:240
139299222,9cyclictest7711-21kworker/0:109:57:250
139299222,4cyclictest31940-21munin-run10:45:020
139299222,11cyclictest1609-21/usr/sbin/munin07:10:180
139299221,8cyclictest26586-21kworker/0:309:19:370
139299221,15cyclictest87350irq/126-enp1s0-08:25:260
141099219,3cyclictest21725-21kworker/u8:107:25:523
141099216,5cyclictest0-21swapper/312:33:423
141099216,13cyclictest39-21rcuc/309:10:233
141099216,13cyclictest39-21rcuc/309:10:233
141099212,10cyclictest0-21swapper/312:15:093
141099211,6cyclictest0-21swapper/311:44:323
1410992116,3cyclictest5708-21grep10:55:283
141099211,12cyclictest87350irq/126-enp1s0-11:51:533
140499216,3cyclictest0-21swapper/211:36:182
140499215,3cyclictest0-21swapper/212:29:292
140499211,6cyclictest0-21swapper/212:18:382
140499211,10cyclictest0-21swapper/207:49:372
140499210,9cyclictest0-21swapper/211:04:312
140099215,9cyclictest23-21ksoftirqd/109:00:181
140099211,5cyclictest0-21swapper/110:45:201
1400992113,7cyclictest0-21swapper/109:24:061
140099211,18cyclictest0-21swapper/107:21:311
140099211,15cyclictest0-21swapper/110:31:591
140099211,14cyclictest0-21swapper/107:57:341
140099211,14cyclictest0-21swapper/107:57:341
140099211,14cyclictest0-21swapper/107:40:071
140099211,14cyclictest0-21swapper/107:37:311
1400992111,3cyclictest0-21swapper/108:20:231
139299217,2cyclictest12736-21kworker/0:008:06:280
139299215,2cyclictest19035-21kworker/0:111:33:530
139299215,2cyclictest19035-21kworker/0:111:33:530
139299213,7cyclictest30580-21/usr/sbin/munin11:55:170
139299212,11cyclictest21790-21/usr/sbin/munin11:35:240
139299211,9cyclictest8212-21kworker/0:011:24:420
139299211,9cyclictest21839-21kworker/0:012:23:380
139299211,9cyclictest12736-21kworker/0:007:42:330
1392992117,2cyclictest1687-21sshd10:49:010
161282200,3sleep3141099cyclictest10:05:263
141099209,3cyclictest24330-21/usr/sbin/munin08:00:233
141099208,4cyclictest0-21swapper/311:25:063
141099207,7cyclictest0-21swapper/311:47:063
141099207,2cyclictest39-21rcuc/308:30:253
141099206,3cyclictest0-21swapper/310:28:063
141099205,8cyclictest12761-21kworker/u8:112:36:093
141099205,7cyclictest0-21swapper/309:32:483
141099203,10cyclictest0-21swapper/309:35:163
141099201,11cyclictest0-21swapper/309:15:343
140499206,4cyclictest9217-21gltestperf11:05:152
140499205,3cyclictest0-21swapper/210:36:252
140499204,4cyclictest20943-21crond10:20:002
140499202,4cyclictest16267-21ntp_states11:20:222
140499201,6cyclictest4774-21cat12:10:022
140499201,5cyclictest0-21swapper/208:06:302
140499201,18cyclictest0-21swapper/209:41:492
140499201,10cyclictest0-21swapper/208:24:332
140499200,10cyclictest0-21swapper/209:55:162
1400992014,3cyclictest24907-21memory11:40:201
140099201,3cyclictest0-21swapper/108:57:221
1400992013,1cyclictest0-21swapper/109:40:121
1400992012,3cyclictest0-21swapper/107:48:371
140099201,18cyclictest0-21swapper/109:55:181
140099201,17cyclictest0-21swapper/111:05:111
140099201,14cyclictest0-21swapper/108:06:091
140099201,14cyclictest0-21swapper/107:25:041
140099201,10cyclictest1-21systemd08:30:021
140099201,10cyclictest0-21swapper/111:37:301
140099200,19cyclictest0-21swapper/108:39:511
139299209,3cyclictest0-21swapper/007:33:040
139299206,3cyclictest11420-21/usr/sbin/munin12:25:220
139299206,2cyclictest8212-21kworker/0:011:14:100
139299204,2cyclictest89050irq/133-enp2s0-08:16:500
139299203,4cyclictest9-21rcu_preempt11:45:220
1392992014,3cyclictest2628-21/usr/sbin/munin12:05:240
139299200,9cyclictest10231-21kworker/0:010:06:260
141099199,3cyclictest3691-21kworker/u8:011:10:483
141099192,3cyclictest19572-21sshd07:48:263
141099191,9cyclictest0-21swapper/312:00:233
141099191,8cyclictest808-21snmpd12:12:493
141099191,6cyclictest0-21swapper/310:15:583
1410991912,4cyclictest12127-21chrt09:59:533
141099191,12cyclictest0-21swapper/307:35:203
141099191,11cyclictest0-21swapper/311:39:373
141099191,10cyclictest30099-21expr10:40:133
141099190,9cyclictest0-21swapper/312:25:423
141099190,8cyclictest87350irq/126-enp1s0-10:20:223
141099190,8cyclictest87350irq/126-enp1s0-10:20:223
140499195,3cyclictest22528-21diskmemload11:33:002
140499195,3cyclictest22528-21diskmemload11:33:002
140499191,8cyclictest0-21swapper/207:35:222
140499191,7cyclictest0-21swapper/211:45:552
140499191,6cyclictest0-21swapper/212:02:002
140499191,6cyclictest0-21swapper/209:25:522
140499191,10cyclictest0-21swapper/209:03:502
140099198,3cyclictest21071-21grep11:30:261
140099198,3cyclictest21071-21grep11:30:261
1400991915,3cyclictest0-21swapper/109:48:281
140099191,17cyclictest0-21swapper/112:28:551
140099191,17cyclictest0-21swapper/111:47:561
140099191,17cyclictest0-21swapper/107:32:301
140099191,16cyclictest15020-21ntpq08:50:221
140099191,11cyclictest20040-21cpuspeed_turbos07:50:131
139299197,3cyclictest12736-21kworker/0:008:12:300
139299191,9cyclictest12736-21kworker/0:007:55:260
139299191,9cyclictest12736-21kworker/0:007:55:260
139299191,6cyclictest0-21swapper/009:33:580
139299191,5cyclictest5087-21tune2fs07:15:160
139299191,16cyclictest0-21swapper/010:00:120
141099189,6cyclictest87350irq/126-enp1s0-08:53:253
141099186,7cyclictest41-21ksoftirqd/309:41:183
141099182,3cyclictest9272-21sed11:05:153
141099181,9cyclictest0-21swapper/311:15:243
141099181,7cyclictest0-21swapper/311:55:233
141099181,11cyclictest0-21swapper/311:01:483
141099181,11cyclictest0-21swapper/310:31:233
141099181,11cyclictest0-21swapper/309:20:183
141099181,10cyclictest766-21ntpd10:45:203
141099181,10cyclictest0-21swapper/312:07:423
141099181,10cyclictest0-21swapper/309:26:113
140499181,5cyclictest6824-21sort07:20:132
140499181,5cyclictest28730-21diskstats11:50:162
140499181,5cyclictest0-21swapper/212:30:222
140099186,4cyclictest0-21swapper/112:20:171
140099182,13cyclictest21-21rcuc/107:10:161
1400991814,3cyclictest0-21swapper/108:40:071
140099181,16cyclictest0-21swapper/109:15:391
140099180,17cyclictest0-21swapper/109:06:461
139299181,7cyclictest0-21swapper/012:15:210
139299181,5cyclictest1067-21hddtemp_smartct12:00:160
139299181,4cyclictest21249-21sshd09:05:170
141099174,4cyclictest20108-21sshd09:02:173
141099173,6cyclictest41-21ksoftirqd/307:40:143
141099171,9cyclictest0-21swapper/308:40:153
141099171,6cyclictest17180-21netstat08:55:203
141099171,3cyclictest0-21swapper/307:13:383
141099171,14cyclictest0-21swapper/307:56:593
141099171,14cyclictest0-21swapper/307:56:593
141099170,10cyclictest0-21swapper/310:36:583
54142160,3sleep3141099cyclictest07:15:183
141099166,3cyclictest87350irq/126-enp1s0-08:25:143
141099166,3cyclictest20560-21/usr/sbin/munin09:05:263
141099165,3cyclictest87350irq/126-enp1s0-08:35:213
141099161,6cyclictest13186-21timerwakeupswit08:45:253
141099161,6cyclictest0-21swapper/308:15:143
141099161,3cyclictest674-21/usr/sbin/munin08:20:103
141099161,3cyclictest0-21swapper/308:05:263
1410991611,3cyclictest29990-21which08:10:233
1410991610,4cyclictest12422-21awk07:30:253
141099152,3cyclictest6851-21cpuspeed_turbos07:20:123
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional