You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-13 - 15:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 13, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1221323030,6sleep121-21rcuc/107:05:171
134562235163,59sleep00-21swapper/007:08:230
133092235168,55sleep30-21swapper/307:06:293
134202232163,57sleep20-21swapper/207:07:572
13684991171,114cyclictest0-21swapper/208:44:282
136849910092,7cyclictest0-21swapper/208:49:062
60422680,2sleep36039-21sed10:30:163
13684996657,3cyclictest0-21swapper/212:34:522
145662580,4sleep31369199cyclictest12:04:463
278402480,1sleep30-21swapper/311:20:163
13679994810,30cyclictest31983-21irqstats10:15:201
13679994717,19cyclictest14544-21diskstats10:50:141
228082430,2sleep30-21swapper/309:55:153
13679994215,16cyclictest6668-21tail11:45:181
13679994214,15cyclictest17019-21/usr/sbin/munin12:10:231
13679994214,15cyclictest17019-21/usr/sbin/munin12:10:231
1367999409,20cyclictest26978-21runrttasks11:17:481
1367999407,24cyclictest5726-40dnf11:44:251
13679994017,20cyclictest20862-21/usr/sbin/munin11:05:111
13679994014,15cyclictest12183-21taskset11:57:371
13679993918,18cyclictest2268-21smart_sda10:20:241
1367999388,19cyclictest25809-21/usr/sbin/munin12:30:201
13679993813,22cyclictest7260-21sshd10:33:441
13679993813,22cyclictest32116-21df11:30:141
13679993812,15cyclictest22719-21df_inode09:55:151
13679993811,14cyclictest17342-21unixbench_singl09:40:281
1367999377,17cyclictest0-21swapper/112:25:181
13679993717,17cyclictest15608-21/usr/sbin/munin12:05:221
13679993716,18cyclictest18957-21/usr/sbin/munin07:20:221
13679993716,18cyclictest18957-21/usr/sbin/munin07:20:221
13679993710,24cyclictest368-21idleruntime07:50:181
1367999366,14cyclictest18245-1kworker/u9:212:02:141
13679993614,18cyclictest909-21sssd_nss08:00:161
13679993613,12cyclictest27387-21/usr/sbin/munin11:20:201
13679993612,21cyclictest12964-21missed_timers10:45:221
13679993612,14cyclictest27128-21diskstats10:05:161
13679993611,12cyclictest0-21swapper/111:50:181
1367299368,13cyclictest17765-21awk09:45:010
13672993611,12cyclictest23732-21munin-run08:45:010
1369199350,24cyclictest0-21swapper/310:05:543
13679993515,17cyclictest10250-21ntp_states09:25:221
13679993514,11cyclictest18384-21chrt07:15:411
13679993513,19cyclictest30934-21crond10:15:011
13679993511,13cyclictest28565-21fschecks_time12:35:161
13679993511,13cyclictest0-21swapper/111:40:021
238992340,4sleep31369199cyclictest11:10:173
13679993414,17cyclictest10607-21irqstats10:40:171
13679993413,18cyclictest8174-21processes09:20:241
1367299349,11cyclictest0-21swapper/010:15:020
1367299335,13cyclictest16172-21munin-run10:55:000
1369199321,21cyclictest0-21swapper/311:05:153
1369199320,25cyclictest0-21swapper/307:17:033
1367999329,11cyclictest27506-21latency_hist07:40:021
13679993223,6cyclictest6054-21smart_sda09:15:261
13679993211,18cyclictest30322-21perf09:00:001
13679993210,18cyclictest18706-21awk08:30:251
13679993210,12cyclictest9550-21needreboot08:10:181
1369199310,24cyclictest0-21swapper/311:27:033
1369199310,24cyclictest0-21swapper/310:45:273
1369199310,24cyclictest0-21swapper/309:51:223
1369199310,12cyclictest0-21swapper/312:16:023
1367999318,9cyclictest22664-21/usr/sbin/munin12:20:241
1367999316,9cyclictest0-21swapper/109:30:231
1367999315,20cyclictest87650irq/129-enp1s0-08:38:041
13679993121,6cyclictest23530-40dnf08:42:421
13679993115,5cyclictest808-21snmpd11:13:011
13679993114,14cyclictest29685-21grep11:25:121
13679993111,12cyclictest9892-21kworker/u8:109:13:191
1369199301,22cyclictest0-21swapper/312:29:473
1367999306,13cyclictest19307-21cpuspeed_turbos12:15:141
1367999306,10cyclictest4095-21sed08:00:011
13679993021,5cyclictest29489-40dnf07:42:231
13679993017,10cyclictest26162-21users10:00:291
13679993010,17cyclictest23463-21grep07:30:131
1369199291,13cyclictest0-21swapper/309:08:373
1369199291,11cyclictest16595-21irqstats09:40:193
1369199290,22cyclictest0-21swapper/308:59:473
1369199290,17cyclictest0-21swapper/312:13:383
1369199290,17cyclictest0-21swapper/312:13:383
1369199290,13cyclictest0-21swapper/309:48:383
1369199290,13cyclictest0-21swapper/309:48:383
1369199290,11cyclictest0-21swapper/310:40:353
13679992919,7cyclictest16244-21/usr/sbin/munin08:25:221
13679992911,4cyclictest26173-21egrep08:50:121
13679992910,5cyclictest87650irq/129-enp1s0-09:05:131
13679992910,16cyclictest9042-21timerwakeupswit10:35:251
13679992910,15cyclictest30777-21cstates09:00:121
1367299291,13cyclictest18735-21munin-run07:20:010
1369199281,8cyclictest0-21swapper/307:51:073
1369199281,11cyclictest0-21swapper/310:52:593
1369199281,11cyclictest0-21swapper/308:51:473
1369199281,10cyclictest0-21swapper/309:04:073
1369199281,10cyclictest0-21swapper/308:49:473
1369199281,10cyclictest0-21swapper/308:07:193
1369199280,21cyclictest0-21swapper/308:12:393
1369199280,13cyclictest0-21swapper/311:41:113
1369199280,12cyclictest0-21swapper/311:00:113
1367999287,18cyclictest579-21systemd-journal11:00:011
1367999287,18cyclictest579-21systemd-journal11:00:011
13679992820,5cyclictest31195-21smart_sda07:45:251
13679992819,6cyclictest4020-21/usr/sbin/munin10:25:191
13679992819,6cyclictest14326-21/usr/sbin/munin09:35:161
13679992818,7cyclictest19505-21/usr/sbin/munin11:00:181
13679992814,11cyclictest15947-21cat07:15:001
13679992812,8cyclictest21147-21missed_timers09:50:191
13679992811,14cyclictest14923-21cat08:24:551
1367299282,14cyclictest20580-21munin-run11:05:010
1369199271,9cyclictest0-21swapper/311:30:173
1369199271,10cyclictest2363-21diskmemload09:34:433
1369199271,10cyclictest0-21swapper/307:21:223
1369199271,10cyclictest0-21swapper/307:21:223
1369199270,22cyclictest0-21swapper/310:12:503
1369199270,20cyclictest0-21swapper/310:24:183
1369199270,10cyclictest0-21swapper/311:48:553
13679992718,6cyclictest6375-21/usr/sbin/munin08:05:131
13679992718,6cyclictest17979-21/usr/sbin/munin09:45:161
13679992718,6cyclictest17979-21/usr/sbin/munin09:45:161
13679992718,6cyclictest10772-21/usr/sbin/munin08:15:131
13679992717,7cyclictest24299-21df_inode08:45:141
1367299279,15cyclictest11771-21munin-run10:45:000
1367299278,16cyclictest3945-21munin-run08:00:020
1367299272,13cyclictest27168-21munin-run11:20:010
1369199266,9cyclictest0-21swapper/307:26:233
1369199261,9cyclictest0-21swapper/309:27:313
1369199261,9cyclictest0-21swapper/308:32:433
1369199261,8cyclictest0-21swapper/311:18:513
1369199261,16cyclictest0-21swapper/310:28:153
1369199261,11cyclictest808-21snmpd12:30:593
1369199261,10cyclictest0-21swapper/311:39:393
1369199260,9cyclictest0-21swapper/309:19:193
1369199260,10cyclictest0-21swapper/307:30:273
1368499267,10cyclictest0-21swapper/211:35:252
13684992610,6cyclictest87350irq/126-enp1s0-12:00:142
13679992617,6cyclictest21154-21/usr/sbin/munin07:25:151
1369199251,9cyclictest26550-21netstat07:35:193
1369199251,9cyclictest0-21swapper/310:38:233
1369199251,8cyclictest0-21swapper/308:42:113
1369199251,11cyclictest0-21swapper/310:17:443
1369199251,11cyclictest0-21swapper/307:55:473
1369199251,10cyclictest28342-21grep07:40:153
1369199251,10cyclictest0-21swapper/309:22:513
1369199250,11cyclictest0-21swapper/310:01:583
1369199250,10cyclictest0-21swapper/312:08:343
1368499253,9cyclictest32-21ksoftirqd/210:50:212
1369199246,3cyclictest0-21swapper/311:50:113
1369199241,10cyclictest0-21swapper/312:20:503
1369199241,10cyclictest0-21swapper/311:58:593
1369199241,10cyclictest0-21swapper/308:28:113
1369199241,10cyclictest0-21swapper/307:49:113
1369199240,13cyclictest0-21swapper/309:37:033
1368499241,5cyclictest0-21swapper/210:46:382
1368499241,13cyclictest0-21swapper/210:40:222
1367299241,12cyclictest10191-21munin-run11:55:010
1369199238,3cyclictest0-21swapper/308:02:193
1369199231,10cyclictest0-21swapper/309:14:073
1369199230,9cyclictest0-21swapper/308:15:273
1369199230,19cyclictest0-21swapper/310:57:313
1369199230,19cyclictest0-21swapper/310:57:313
1369199230,10cyclictest0-21swapper/307:12:073
1368499234,8cyclictest311ktimersoftd/208:20:012
1368499233,3cyclictest27049-21sshd11:18:452
1368499230,8cyclictest0-21swapper/212:10:212
1368499230,8cyclictest0-21swapper/212:10:212
1367299230,11cyclictest21201-21idleruntime-cro12:20:010
1369199227,3cyclictest0-21swapper/308:37:553
13691992210,3cyclictest0-21swapper/312:35:243
1368499227,4cyclictest24970-21awk11:15:012
1368499225,7cyclictest311ktimersoftd/209:40:202
1368499223,7cyclictest311ktimersoftd/207:25:022
1368499223,7cyclictest311ktimersoftd/207:25:022
13684992211,3cyclictest87650irq/129-enp1s0-10:05:272
1368499221,12cyclictest0-21swapper/211:00:232
1367299224,10cyclictest827-21runrttasks09:55:230
130642220,2sleep339-21rcuc/308:20:133
1368499219,3cyclictest87350irq/126-enp1s0-10:55:172
1368499219,3cyclictest87350irq/126-enp1s0-10:55:172
1368499215,8cyclictest87350irq/126-enp1s0-10:10:192
1368499215,8cyclictest311ktimersoftd/209:30:202
1368499214,9cyclictest19690-21grep12:15:162
1368499214,9cyclictest0-21swapper/211:27:432
1368499214,8cyclictest20798-21/usr/sbin/munin11:05:212
1368499212,8cyclictest87550irq/128-enp1s0-10:25:272
1367299212,11cyclictest20798-21/usr/sbin/munin11:05:180
1367299211,6cyclictest0-21swapper/009:19:110
1367299211,3cyclictest0-21swapper/009:25:480
13672992112,2cyclictest0-21swapper/011:45:510
1367299211,11cyclictest13869-21/usr/sbin/munin07:10:150
1368499206,3cyclictest87350irq/126-enp1s0-10:30:122
1368499201,8cyclictest0-21swapper/209:15:492
1368499201,7cyclictest0-21swapper/210:02:152
1368499201,11cyclictest0-21swapper/209:10:212
1368499200,9cyclictest32-21ksoftirqd/211:59:582
1367299202,10cyclictest983-21/usr/sbin/munin10:20:150
1367299200,5cyclictest0-21swapper/010:18:000
37282191,3sleep20-21swapper/207:57:292
173282191,14sleep017366-21diskstats12:10:140
173282191,14sleep017366-21diskstats12:10:140
1368499199,2cyclictest87350irq/126-enp1s0-09:25:212
1368499198,4cyclictest87550irq/128-enp1s0-11:20:142
1368499197,2cyclictest87550irq/128-enp1s0-11:30:182
1368499194,5cyclictest0-21swapper/207:14:212
1368499192,9cyclictest87350irq/126-enp1s0-08:00:142
1368499192,8cyclictest311ktimersoftd/212:20:172
1368499192,6cyclictest30-21rcuc/211:40:112
1368499192,4cyclictest31910-21perf07:50:012
1368499192,14cyclictest13613-21gltestperf08:20:162
1368499191,5cyclictest6661-21tail11:45:172
1368499191,5cyclictest10539-21latency_hist08:15:012
1368499191,11cyclictest0-21swapper/212:05:172
1368499191,11cyclictest0-21swapper/211:50:282
1368499191,10cyclictest32-21ksoftirqd/210:20:232
1367299199,3cyclictest87550irq/128-enp1s0-12:38:280
1367299196,12cyclictest0-21swapper/007:35:000
1367299191,4cyclictest17952-21grep10:55:260
1367299191,4cyclictest17952-21grep10:55:260
109702190,3sleep00-21swapper/011:55:170
1368499184,6cyclictest0-21swapper/209:24:592
1368499183,8cyclictest87350irq/126-enp1s0-07:15:142
1368499183,7cyclictest87350irq/126-enp1s0-12:25:132
1368499182,7cyclictest87350irq/126-enp1s0-08:05:272
1368499181,8cyclictest18437-21ntp_states08:30:232
1368499181,5cyclictest28352-21diskstats12:35:152
13684991811,2cyclictest87550irq/128-enp1s0-10:39:332
1367299185,4cyclictest16340-21tail12:05:310
1367299181,9cyclictest14653-21ntp_states09:35:210
1367299181,2cyclictest0-21swapper/007:38:160
13672991812,3cyclictest5400-21/usr/sbin/munin10:30:130
1367299181,10cyclictest23121-21sshd12:21:480
1367299181,10cyclictest18245-1kworker/u9:209:20:140
1367299181,10cyclictest0-21swapper/012:00:140
44622171,2sleep00-21swapper/009:13:520
323152170,2sleep20-21swapper/209:02:312
258532171,2sleep00-21swapper/008:49:160
218592171,2sleep00-21swapper/009:51:290
1368499178,3cyclictest0-21swapper/207:40:212
1368499173,7cyclictest13577-21/usr/sbin/munin09:35:162
1368499173,2cyclictest87350irq/126-enp1s0-08:50:432
1368499171,9cyclictest18791-21irqstats09:45:182
1368499171,9cyclictest18791-21irqstats09:45:182
1368499171,8cyclictest28351-21/usr/sbin/munin08:55:192
1368499171,6cyclictest15391-21expr08:25:132
1368499171,6cyclictest0-21swapper/210:17:132
1368499171,6cyclictest0-21swapper/209:05:122
1368499171,15cyclictest0-21swapper/209:56:572
1368499171,14cyclictest0-21swapper/208:35:102
1368499171,10cyclictest0-21swapper/209:51:002
1367299176,2cyclictest8747-21awk10:35:220
1367299175,3cyclictest1948-21head09:05:260
1367299173,11cyclictest28351-21/usr/sbin/munin08:55:140
1367299172,6cyclictest12963-21/usr/sbin/munin08:20:170
1367299172,12cyclictest12032-21gltestperf09:30:160
1367299171,7cyclictest0-21swapper/011:41:290
1367299171,7cyclictest0-21swapper/009:47:310
1367299171,7cyclictest0-21swapper/009:47:310
1367299171,6cyclictest0-21swapper/012:32:130
1367299171,3cyclictest26780-21/usr/sbin/munin10:05:160
13672991713,3cyclictest0-21swapper/011:39:560
13672991713,3cyclictest0-21swapper/011:11:240
13672991712,3cyclictest32039-21unixbench_multi09:00:260
13672991712,3cyclictest29583-21/usr/sbin/munin11:25:150
13672991712,3cyclictest24855-21mailstats12:25:230
1367299171,15cyclictest0-21swapper/011:30:190
1367299171,15cyclictest0-21swapper/010:02:200
1367299171,14cyclictest0-21swapper/008:25:280
1367299171,14cyclictest0-21swapper/007:40:380
1367299171,10cyclictest0-21swapper/008:10:120
10162171,2sleep20-21swapper/207:50:272
1368499161,7cyclictest0-21swapper/207:30:262
13684991612,3cyclictest0-21swapper/207:25:392
13684991610,3cyclictest25667-21grep07:35:122
1367299164,3cyclictest0-21swapper/011:22:430
1367299161,9cyclictest0-21swapper/010:25:160
1367299161,6cyclictest0-21swapper/008:05:180
1367299161,6cyclictest0-21swapper/007:20:200
1367299161,6cyclictest0-21swapper/007:20:200
1367299161,2cyclictest0-21swapper/008:32:420
1367299161,2cyclictest0-21swapper/008:01:120
13672991611,3cyclictest12192-21expr10:45:140
13672991611,3cyclictest11418-21cut08:15:180
13672991610,3cyclictest29934-21/usr/sbin/munin07:45:150
13672991610,3cyclictest26738-21grep08:50:170
1367299159,3cyclictest20460-21cat08:35:180
1367299153,3cyclictest0-21swapper/007:28:050
1367299152,10cyclictest32248-21tr07:50:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional