You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-02 - 17:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Jul 02, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
256922240163,24sleep30-21swapper/307:09:583
254872239163,24sleep20-21swapper/207:07:202
255832203164,26sleep00-21swapper/007:08:350
254562203166,24sleep10-21swapper/107:06:541
351421780,3sleep12577999cyclictest09:55:261
691521660,5sleep02577199cyclictest11:20:010
3032021610,2sleep12577999cyclictest07:15:251
2578599103100,2cyclictest0-21swapper/210:00:222
263082520,3sleep230-21rcuc/209:35:162
2577199421,29cyclictest6304-21sh10:05:010
25785994037,2cyclictest0-21swapper/211:24:322
2577199391,26cyclictest0-21swapper/011:40:080
2577199381,24cyclictest0-21swapper/009:17:450
59552370,1sleep20-21swapper/212:30:142
2577199373,22cyclictest28871-21/usr/sbin/munin07:15:180
2577199372,26cyclictest7768-21/usr/sbin/munin12:35:140
2577199371,28cyclictest8108-21/usr/sbin/munin08:55:250
2577199371,27cyclictest26899-21perl10:50:160
2577199371,25cyclictest30096-21/usr/sbin/munin09:45:130
2577199361,22cyclictest31091-21fschecks_count11:00:130
25771993611,22cyclictest28339-21ntpq08:25:210
2577199352,30cyclictest20684-21/usr/sbin/munin08:10:120
2577199351,25cyclictest0-21swapper/011:25:230
2577199351,21cyclictest18985-21/usr/sbin/munin10:30:240
2578999340,21cyclictest0-21swapper/311:33:533
2577199346,13cyclictest16686-21munin-run09:15:010
2577199346,13cyclictest16686-21munin-run09:15:010
2577199344,20cyclictest0-21swapper/010:15:240
2577199343,16cyclictest87650irq/129-enp1s0-12:13:340
2577199342,26cyclictest87350irq/126-enp1s0-11:50:200
2577199342,24cyclictest24113-21/usr/sbin/munin10:45:150
2577199341,30cyclictest29478-21/usr/sbin/munin08:30:160
2577199341,29cyclictest24731-21/usr/sbin/munin12:00:170
2577199341,22cyclictest21305-21/usr/sbin/munin09:25:230
2577199341,21cyclictest827-21runrttasks09:57:410
2578599331,16cyclictest0-21swapper/210:17:592
2577199333,16cyclictest87350irq/126-enp1s0-10:55:230
2577199332,27cyclictest16586-21smart_sda10:25:240
25771993313,17cyclictest28201-21sort07:15:020
2577199331,29cyclictest25080-21/usr/sbin/munin08:20:200
2577199331,25cyclictest87350irq/126-enp1s0-12:18:390
2577199331,21cyclictest6643-21ntpq12:30:210
2577199331,20cyclictest7494-21/usr/sbin/munin07:40:200
2577199331,19cyclictest3374-21/usr/sbin/munin12:25:260
2577199331,18cyclictest14089-21/usr/sbin/munin07:55:250
2577199328,17cyclictest32294-21/usr/sbin/munin09:50:260
2577199326,12cyclictest29884-21perf09:45:010
2577199323,21cyclictest12511-21/usr/sbin/munin09:05:160
2577199322,19cyclictest87550irq/128-enp1s0-10:20:140
2577199321,28cyclictest7154-21/usr/sbin/munin11:20:230
2577199321,26cyclictest6983-21ntp_states08:50:220
2577199321,24cyclictest0-21swapper/009:20:130
2577199321,19cyclictest3558-21/usr/sbin/munin11:10:180
2578999319,11cyclictest0-21swapper/312:13:343
2578999319,10cyclictest0-21swapper/310:05:143
2578599311,14cyclictest0-21swapper/207:10:152
2577199318,20cyclictest17554-21swap08:00:250
2577199318,20cyclictest17554-21swap08:00:250
25771993113,15cyclictest19720-21/usr/sbin/munin10:35:260
2577199311,27cyclictest24096-21/usr/sbin/munin08:15:230
2577199311,27cyclictest23497-21/usr/sbin/munin09:30:140
2577199311,27cyclictest1492-21/usr/sbin/munin08:40:260
2577199311,27cyclictest10304-21/usr/sbin/munin09:00:160
2577199311,26cyclictest25697-21/usr/sbin/munin09:35:120
2577199311,23cyclictest8741-21/usr/sbin/munin10:10:120
2577199310,22cyclictest0-21swapper/007:30:230
2578999306,8cyclictest31850-40dnf09:47:263
2578599301,15cyclictest0-21swapper/207:47:312
2578599301,14cyclictest0-21swapper/208:54:032
2577999301,10cyclictest0-21swapper/112:31:231
2577199302,25cyclictest19401-21smart_sda11:45:240
2577199302,25cyclictest1824-21smart_sda11:05:230
2577199301,26cyclictest6536-21/usr/sbin/munin10:05:240
2577199301,26cyclictest5296-21/usr/sbin/munin07:35:250
2577199301,26cyclictest31677-21/usr/sbin/munin08:35:240
2577199301,26cyclictest28194-21awk12:05:230
2577199301,26cyclictest21915-21/usr/sbin/munin10:40:150
2577199301,26cyclictest19580-21/usr/sbin/munin08:05:200
2577199301,22cyclictest0-21swapper/007:45:140
2578599291,10cyclictest4732-21netstat08:45:192
2577199297,19cyclictest14484-21diskmemload11:55:180
2577199291,25cyclictest4056-21diskstats08:45:140
2577199291,25cyclictest31068-21/usr/sbin/munin07:20:210
2577199291,25cyclictest11894-21/usr/sbin/munin07:50:160
2577199291,25cyclictest1154-21/usr/sbin/munin12:20:130
2577199291,18cyclictest0-21swapper/011:30:210
2577199290,1cyclictest0-21swapper/011:35:150
2577199290,1cyclictest0-21swapper/011:35:150
2578599281,13cyclictest0-21swapper/208:41:592
2577999281,24cyclictest1366-21cut11:05:181
2577199281,24cyclictest864-21/usr/sbin/munin07:25:240
25789992713,2cyclictest2538-21awk11:10:013
2578599270,11cyclictest0-21swapper/209:51:152
25779992724,2cyclictest0-21swapper/111:19:471
25779992723,3cyclictest0-21swapper/109:04:501
2578599261,12cyclictest0-21swapper/210:46:312
2578599261,10cyclictest0-21swapper/210:07:472
2578599260,12cyclictest0-21swapper/211:30:312
25779992623,2cyclictest0-21swapper/110:44:501
25779992623,2cyclictest0-21swapper/108:59:131
2577999261,22cyclictest10699-21crond10:15:011
2577999261,22cyclictest0-21swapper/111:10:131
2578599251,9cyclictest0-21swapper/208:24:392
2578599251,9cyclictest0-21swapper/207:35:272
25779992522,2cyclictest0-21swapper/110:03:491
25779992522,2cyclictest0-21swapper/109:43:531
25779992522,2cyclictest0-21swapper/108:09:431
2577999251,13cyclictest15116-21awk10:25:011
2577999251,11cyclictest1579-21cpuspeed_turbos08:40:131
2578599241,14cyclictest0-21swapper/211:56:422
2578599241,13cyclictest0-21swapper/207:43:232
2578599241,11cyclictest0-21swapper/212:17:032
2578599240,11cyclictest0-21swapper/212:05:192
2578599240,11cyclictest0-21swapper/211:48:392
2578599240,10cyclictest0-21swapper/212:27:592
25779992412,9cyclictest31142-21grep07:20:121
25779992412,7cyclictest0-21swapper/112:13:201
2577999241,20cyclictest0-21swapper/107:45:011
2577999241,11cyclictest7830-21hddtemp_smartct11:20:161
2577999241,11cyclictest6360-21ntp_states07:35:201
2577999241,11cyclictest22308-21latency_hist11:55:021
2577999241,10cyclictest10692-21timerandwakeup11:25:231
2577999240,19cyclictest0-21swapper/110:33:121
1813422411,2sleep10-21swapper/108:03:301
1813422411,2sleep10-21swapper/108:03:301
2578999231,12cyclictest9694-21/usr/sbin/munin07:45:203
2578599236,9cyclictest12050irq/123-ahci[0008:20:012
2578599231,12cyclictest0-21swapper/207:53:432
2578599230,13cyclictest0-21swapper/211:04:072
2577999239,3cyclictest23055-21wc08:15:121
25779992320,2cyclictest0-21swapper/109:11:031
25779992320,2cyclictest0-21swapper/109:11:031
25779992312,2cyclictest0-21swapper/112:23:231
2577999231,20cyclictest0-21swapper/110:28:451
2577999231,19cyclictest579-21systemd-journal10:07:451
2577999231,19cyclictest28001-21ntp_offset12:05:211
2577999231,19cyclictest0-21swapper/107:30:141
2577999231,14cyclictest0-21swapper/112:38:131
2577999231,14cyclictest0-21swapper/112:25:311
2577999231,12cyclictest0-21swapper/109:24:391
2577999231,11cyclictest6118-21cstates08:50:131
25779992310,2cyclictest16565-21/usr/sbin/munin11:40:151
2578599229,2cyclictest0-21swapper/207:32:132
2578599228,3cyclictest0-21swapper/209:17:492
2578599227,6cyclictest0-21swapper/212:38:132
2578599223,2cyclictest87650irq/129-enp1s0-09:30:512
2578599221,8cyclictest0-21swapper/211:15:392
2578599221,12cyclictest0-21swapper/211:28:152
2578599220,11cyclictest0-21swapper/210:34:472
2577999221,4cyclictest24309-21irqstats09:30:171
25779992212,2cyclictest0-21swapper/111:59:331
2577999221,19cyclictest0-21swapper/110:54:581
2577999221,18cyclictest20215-21fschecks_time10:35:151
2577999221,13cyclictest0-21swapper/111:32:001
2577999221,10cyclictest28866-21diskstats10:55:151
2577999221,10cyclictest14373-21cstates07:55:131
2577999221,10cyclictest0-21swapper/112:15:321
2577999220,7cyclictest0-21swapper/109:53:061
2577999220,20cyclictest0-21swapper/109:45:451
2578999211,8cyclictest0-21swapper/310:24:463
2578999211,6cyclictest25664-21sed10:45:263
2578999211,6cyclictest0-21swapper/311:54:143
2578999211,5cyclictest0-21swapper/310:56:323
25789992110,2cyclictest32342-21sshd11:00:453
2578599217,7cyclictest29885-21awk09:45:012
2578599215,3cyclictest0-21swapper/212:13:512
2578599211,12cyclictest14484-21diskmemload09:20:122
2578599210,9cyclictest0-21swapper/210:52:072
2578599210,11cyclictest0-21swapper/210:42:552
2578599210,11cyclictest0-21swapper/208:59:512
2577999212,7cyclictest16908-21/usr/sbin/munin09:15:201
2577999211,3cyclictest26313-21df07:10:121
2577999211,3cyclictest25622-21sed12:00:181
2577999211,3cyclictest1126-21runrttasks08:38:221
2577999211,19cyclictest0-21swapper/107:45:111
2577999211,18cyclictest0-21swapper/109:06:401
2577999211,18cyclictest0-21swapper/108:25:021
2577999211,18cyclictest0-21swapper/108:13:391
2577999211,17cyclictest29827-21diskstats08:30:141
2577999211,13cyclictest0-21swapper/111:37:261
2577999211,13cyclictest0-21swapper/111:37:261
2577999211,13cyclictest0-21swapper/110:17:361
2577999211,10cyclictest0-21swapper/107:54:531
2577999210,19cyclictest0-21swapper/111:46:051
2577999210,19cyclictest0-21swapper/110:45:121
2577999210,19cyclictest0-21swapper/108:45:151
2578999201,8cyclictest16930-21cpu09:15:113
2578999201,7cyclictest0-21swapper/312:09:483
25789992014,3cyclictest1494-21gltestperf07:25:163
25789992013,4cyclictest4342-21gltestperf08:45:153
2578999201,17cyclictest0-21swapper/308:22:533
2578999201,10cyclictest18231-21sshd08:04:453
2578999201,10cyclictest18231-21sshd08:04:453
25789992010,8cyclictest30794-21sshd07:19:353
2578599207,3cyclictest0-21swapper/208:14:032
2578599206,3cyclictest87350irq/126-enp1s0-10:25:172
2578599205,8cyclictest87350irq/126-enp1s0-09:45:152
2578599204,3cyclictest0-21swapper/209:12:522
2578599204,3cyclictest0-21swapper/209:12:512
2578599203,9cyclictest87350irq/126-enp1s0-10:10:132
2578599202,7cyclictest1429-21latency11:05:172
2578599202,6cyclictest21309-21users10:35:272
2578599201,7cyclictest1233-21sshd08:39:452
2578599201,7cyclictest0-21swapper/209:56:322
2578599201,6cyclictest29781-21smart_sda10:55:252
2578599201,6cyclictest21252-21memory11:50:192
2578599201,12cyclictest87650irq/129-enp1s0-11:40:232
2578599201,11cyclictest0-21swapper/211:37:462
2578599201,11cyclictest0-21swapper/211:37:462
2578599201,11cyclictest0-21swapper/209:26:442
2577999201,18cyclictest0-21swapper/108:21:021
2577999201,10cyclictest0-21swapper/109:35:551
2578999198,3cyclictest11894-21/usr/sbin/munin07:50:253
2578999195,3cyclictest87650irq/129-enp1s0-07:59:453
2578999192,6cyclictest5345-21munin-run12:30:023
2578999192,14cyclictest19481-21strings08:05:183
2578999191,8cyclictest22298-21needreboot09:25:203
2578999191,7cyclictest18007-21kworker/u8:209:35:173
25789991913,3cyclictest10572-21gunzip11:25:213
25789991912,4cyclictest15048-21sh09:10:133
25789991912,4cyclictest15048-21sh09:10:133
2578999191,10cyclictest31338-21/usr/sbin/munin12:15:103
2578999191,10cyclictest30134-21kworker/3:212:30:243
2578999191,10cyclictest21424-21sshd10:36:443
2578999191,10cyclictest2134-21/usr/sbin/munin09:55:243
2578599195,3cyclictest87650irq/129-enp1s0-09:00:182
2578599194,8cyclictest87350irq/126-enp1s0-12:05:012
2578599194,4cyclictest29538-21cpu08:30:122
2578599192,9cyclictest2760-21/usr/sbin/munin11:10:212
2578599191,9cyclictest451-21sshd07:23:342
2578599191,6cyclictest30132-21smart_sda07:15:242
2578599191,6cyclictest17249-21sed08:00:202
2578599191,6cyclictest17249-21sed08:00:202
2578599191,10cyclictest14484-21diskmemload12:22:052
2577999196,2cyclictest31742-21ntp_kernel_err11:00:201
2577999191,9cyclictest1013-21kworker/u8:009:25:071
2577999190,18cyclictest0-21swapper/107:26:081
251962192,2sleep318007-21kworker/u8:209:31:423
2578999187,3cyclictest808-21snmpd10:18:103
2578999185,3cyclictest830-21grep08:35:273
2578999184,8cyclictest0-21swapper/307:35:193
2578999182,9cyclictest0-21swapper/310:10:393
2578999181,9cyclictest808-21snmpd10:43:143
2578999181,9cyclictest808-21snmpd08:17:483
2578999181,7cyclictest7060-21pmu-power08:50:233
2578999181,7cyclictest0-21swapper/312:23:033
2578999181,7cyclictest0-21swapper/311:17:323
2578999181,7cyclictest0-21swapper/310:02:333
2578999181,6cyclictest28051-21cat07:15:013
2578999181,6cyclictest0-21swapper/312:01:563
2578999181,6cyclictest0-21swapper/309:44:493
2578999181,5cyclictest9739-21munin-run12:40:023
2578999181,5cyclictest0-21swapper/311:23:043
2578999181,3cyclictest808-21snmpd10:28:113
2578999181,3cyclictest23638-21open_inodes11:55:233
25789991813,3cyclictest18139-21/usr/sbin/munin11:45:193
25789991811,4cyclictest3755-21tail07:30:153
2578999180,7cyclictest0-21swapper/311:35:153
2578999180,7cyclictest0-21swapper/311:35:153
2578599185,3cyclictest13502-21grep09:05:182
2578599184,7cyclictest87350irq/126-enp1s0-07:55:182
2578599181,5cyclictest13201-21cat10:20:122
2578599180,6cyclictest1907-21/usr/sbin/munin07:25:202
2578999177,2cyclictest19104-21/usr/sbin/munin09:20:143
2578999175,3cyclictest13649-21pmu-power09:05:213
2578999172,4cyclictest8170-21hddtemp_smartct07:40:163
2578999171,7cyclictest21721-21tr08:10:183
2578999171,6cyclictest19109-21users10:30:253
2578999171,6cyclictest0-21swapper/310:54:513
2578999171,5cyclictest0-21swapper/309:50:173
25789991712,3cyclictest29478-21/usr/sbin/munin08:30:143
25789991711,3cyclictest3380-21grep11:10:163
25789991711,3cyclictest31877-21irqstats07:20:173
25789991711,3cyclictest17266-21cat11:40:233
2578999171,10cyclictest0-21swapper/308:40:183
25785991714,2cyclictest0-21swapper/208:28:422
2578599171,2cyclictest0-21swapper/208:07:062
2578999169,4cyclictest10914-21head09:00:163
2578999168,5cyclictest10078-21dump-pmu-power09:00:013
2578999164,3cyclictest27528-21expr08:25:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional