You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-17 - 00:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Sep 16, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40822234163,23sleep10-21swapper/107:08:011
40582234163,57sleep20-21swapper/207:07:422
39502211178,22sleep00-21swapper/007:06:190
41872203166,24sleep30-21swapper/307:09:253
658821180,5sleep1432699cyclictest12:05:221
4331997068,1cyclictest0-21swapper/210:43:212
26452540,2sleep341-21ksoftirqd/310:45:023
432699466,26cyclictest22579-21awk07:50:011
109522450,1sleep00-21swapper/007:20:230
108822450,4sleep310884-21needreboot12:15:193
433199431,32cyclictest0-21swapper/210:08:352
4331994037,2cyclictest8564-1kworker/u9:111:28:252
4331994037,2cyclictest8564-1kworker/u9:111:28:252
431999388,14cyclictest29795-21munin-run09:20:010
4331993735,1cyclictest0-21swapper/212:09:462
114852360,3sleep2311ktimersoftd/212:15:262
433199341,25cyclictest0-21swapper/211:58:032
433199341,14cyclictest15701-21open_files08:45:222
431999348,12cyclictest17236-21awk10:05:010
433199331,22cyclictest0-21swapper/211:24:142
433199321,21cyclictest0-21swapper/209:05:022
433199321,20cyclictest0-21swapper/210:34:182
433199321,19cyclictest0-21swapper/209:53:552
433199320,13cyclictest0-21swapper/210:35:182
433899311,21cyclictest0-21swapper/309:04:223
4331993128,2cyclictest0-21swapper/207:35:122
4331993118,7cyclictest0-21swapper/208:53:502
433199311,21cyclictest0-21swapper/209:58:512
431999316,12cyclictest22575-21munin-run07:50:010
431999313,13cyclictest7817-21basename08:30:010
433199301,20cyclictest0-21swapper/209:14:302
433199301,13cyclictest0-21swapper/208:30:142
433199300,14cyclictest0-21swapper/210:14:072
433199300,13cyclictest0-21swapper/209:49:432
433199291,12cyclictest0-21swapper/210:17:062
433199291,12cyclictest0-21swapper/210:17:062
433199290,13cyclictest0-21swapper/211:19:302
433899280,21cyclictest0-21swapper/311:39:523
433199281,18cyclictest0-21swapper/209:43:352
433199281,13cyclictest0-21swapper/212:35:222
433199281,13cyclictest0-21swapper/212:35:222
433199281,13cyclictest0-21swapper/211:30:222
433199281,12cyclictest0-21swapper/211:01:062
433199280,13cyclictest0-21swapper/211:49:462
432699282,24cyclictest0-21swapper/109:53:391
432699281,25cyclictest0-21swapper/108:48:301
432699281,25cyclictest0-21swapper/108:25:451
432699281,25cyclictest0-21swapper/108:14:371
4326992811,4cyclictest2668-21/usr/sbin/munin10:45:201
431999282,14cyclictest6496-21munin-run07:15:010
4331992718,3cyclictest0-21swapper/209:04:062
433199271,2cyclictest0-21swapper/211:36:022
433199271,15cyclictest6131-21smart_sda10:50:222
433199271,11cyclictest0-21swapper/212:10:152
433199271,11cyclictest0-21swapper/209:25:262
432699274,20cyclictest730-21df10:40:151
432699271,24cyclictest0-21swapper/111:45:051
432699271,24cyclictest0-21swapper/111:04:511
432699271,24cyclictest0-21swapper/109:44:391
432699271,24cyclictest0-21swapper/109:35:141
432699271,24cyclictest0-21swapper/108:51:531
432699271,24cyclictest0-21swapper/107:54:451
432699271,24cyclictest0-21swapper/107:54:451
432699271,24cyclictest0-21swapper/107:37:131
432699270,25cyclictest0-21swapper/110:59:281
432699270,25cyclictest0-21swapper/108:20:511
433899262,13cyclictest15491-40dnf11:13:143
433199269,3cyclictest0-21swapper/207:46:392
433199261,16cyclictest0-21swapper/208:29:302
433199261,13cyclictest0-21swapper/208:21:102
433199261,13cyclictest0-21swapper/207:40:432
433199260,13cyclictest0-21swapper/209:23:502
432699261,3cyclictest9238-21grep09:45:161
432699261,23cyclictest0-21swapper/112:16:161
432699261,23cyclictest0-21swapper/110:11:031
432699261,23cyclictest0-21swapper/108:16:431
432699261,19cyclictest0-21swapper/111:33:511
432699261,16cyclictest0-21swapper/111:19:191
4326992611,13cyclictest17267-21kworker/u8:207:40:311
432699260,24cyclictest0-21swapper/110:06:391
432699260,24cyclictest0-21swapper/109:26:321
432699260,24cyclictest0-21swapper/108:58:001
432699260,24cyclictest0-21swapper/108:40:381
432699260,24cyclictest0-21swapper/108:37:181
432699260,24cyclictest0-21swapper/108:06:121
432699260,24cyclictest0-21swapper/107:22:231
432699260,24cyclictest0-21swapper/107:11:271
433899255,7cyclictest0-21swapper/312:10:183
433899254,12cyclictest0-21swapper/309:15:183
433899251,20cyclictest0-21swapper/312:26:323
433199251,14cyclictest0-21swapper/208:58:382
433199251,14cyclictest0-21swapper/207:28:472
433199251,12cyclictest0-21swapper/210:20:302
433199250,10cyclictest0-21swapper/211:13:342
433199250,10cyclictest0-21swapper/210:00:352
4326992514,3cyclictest0-21swapper/111:21:521
432699251,21cyclictest13777-21latency_hist07:30:011
432699251,20cyclictest26799-21processes09:10:231
432699251,12cyclictest22760-21memory10:15:191
432699251,12cyclictest22760-21memory10:15:191
432699250,23cyclictest0-21swapper/110:36:351
432699250,23cyclictest0-21swapper/108:00:111
433199241,9cyclictest0-21swapper/207:50:142
433199241,9cyclictest0-21swapper/207:50:142
433199241,13cyclictest0-21swapper/212:33:182
4331992411,3cyclictest0-21swapper/208:16:562
433199240,12cyclictest0-21swapper/209:31:192
433199240,10cyclictest0-21swapper/208:05:522
4326992414,2cyclictest0-21swapper/111:12:301
432699241,22cyclictest0-21swapper/111:05:031
432699241,21cyclictest0-21swapper/110:34:231
432699241,21cyclictest0-21swapper/107:33:411
432699241,20cyclictest19244-21fschecks_time12:35:161
432699241,20cyclictest19244-21fschecks_time12:35:161
432699241,20cyclictest17551-21ntp_states12:30:211
432699241,12cyclictest24684-21smart_sda09:05:241
432699241,11cyclictest8379-21diskstats08:30:111
432699241,11cyclictest5679-21irqstats10:50:161
432699241,11cyclictest15098-21irqstats12:25:171
432699240,22cyclictest0-21swapper/112:14:031
433899232,7cyclictest25400-21munin-run09:10:013
433899231,15cyclictest0-21swapper/310:45:313
4338992310,3cyclictest0-21swapper/310:08:423
433199238,3cyclictest0-21swapper/207:57:022
433199237,3cyclictest0-21swapper/212:21:582
433199237,3cyclictest0-21swapper/208:39:262
433199231,9cyclictest0-21swapper/208:43:542
433199231,8cyclictest0-21swapper/210:27:582
433199231,11cyclictest32506-21memory08:10:182
433199231,11cyclictest0-21swapper/211:44:222
433199231,11cyclictest0-21swapper/210:45:302
433199231,10cyclictest0-21swapper/209:16:182
432699231,2cyclictest0-21swapper/110:25:241
432699230,15cyclictest0-21swapper/111:36:381
220112231,3sleep00-21swapper/010:15:140
220112231,3sleep00-21swapper/010:15:140
433899225,5cyclictest20615-21cpu07:45:113
433899221,5cyclictest0-21swapper/311:30:253
433899221,3cyclictest0-21swapper/310:17:563
433899221,3cyclictest0-21swapper/310:17:553
433899221,15cyclictest0-21swapper/310:54:033
433899221,14cyclictest0-21swapper/308:40:153
433899221,13cyclictest13082-21needreboot12:20:203
433199227,3cyclictest0-21swapper/208:04:502
433199221,6cyclictest0-21swapper/212:03:402
433199221,12cyclictest0-21swapper/209:37:182
433199221,12cyclictest0-21swapper/207:21:302
433199221,12cyclictest0-21swapper/207:18:382
433199221,10cyclictest0-21swapper/207:14:262
4331992210,3cyclictest0-21swapper/210:56:222
432699221,8cyclictest0-21swapper/111:58:431
432699221,19cyclictest0-21swapper/109:56:461
432699220,8cyclictest0-21swapper/109:35:011
432699220,20cyclictest0-21swapper/109:00:331
432699220,15cyclictest0-21swapper/111:25:131
432699220,15cyclictest0-21swapper/111:25:131
431999226,6cyclictest0-21swapper/010:22:170
431999223,11cyclictest11476-21/usr/sbin/munin11:05:250
431999222,11cyclictest909-21sssd_nss08:23:390
156462220,2sleep015650-21latency_hist11:15:010
433899214,8cyclictest0-21swapper/308:27:223
433899211,7cyclictest579-21systemd-journal08:55:003
433899211,5cyclictest26854-21/usr/sbin/munin11:40:133
4338992115,3cyclictest9299-21gltestperf09:45:163
433899211,10cyclictest0-21swapper/309:50:223
433199211,10cyclictest0-21swapper/207:34:472
432699217,11cyclictest0-21swapper/111:40:171
432699216,3cyclictest4268-21cut12:00:181
432699216,14cyclictest0-21swapper/110:03:041
4326992117,3cyclictest0-21swapper/107:19:101
432699211,2cyclictest0-21swapper/109:15:131
432699211,18cyclictest0-21swapper/109:21:401
431999217,5cyclictest0-21swapper/010:41:450
431999213,10cyclictest32508-21smart_sda11:50:230
431999211,12cyclictest9470-21sshd12:12:400
433899204,4cyclictest11634-21cat07:25:013
433899202,10cyclictest6740-21sendmail_mailst12:05:243
433899201,9cyclictest0-21swapper/307:37:313
433899201,8cyclictest32713-21fschecks_time09:25:153
433899201,7cyclictest30594-21sed10:35:013
433899201,6cyclictest901-40dnf08:11:433
433899201,6cyclictest3587-21/usr/sbin/munin09:30:273
433899201,6cyclictest30534-21grep11:45:253
433899201,6cyclictest12290-21irqstats11:05:183
433899201,5cyclictest790-21in:imjournal07:15:013
433199201,12cyclictest0-21swapper/211:09:182
432699201,3cyclictest0-21swapper/112:23:281
432699201,18cyclictest0-21swapper/111:50:581
432699201,18cyclictest0-21swapper/110:20:581
432699201,18cyclictest0-21swapper/107:57:031
431999209,3cyclictest22458-21/usr/sbin/munin11:30:190
431999209,3cyclictest16715-21cat12:30:120
431999206,3cyclictest0-21swapper/012:22:370
431999201,7cyclictest27374-40dnf09:12:230
431999201,4cyclictest0-21swapper/012:09:300
431999201,4cyclictest0-21swapper/009:45:300
431999201,11cyclictest2860-21sshd11:57:400
431999201,10cyclictest4644-21cat12:00:230
431999201,10cyclictest25986-21needreboot07:55:190
431999201,10cyclictest16413-21sed12:30:010
431999200,12cyclictest2053-21/usr/sbin/munin09:30:210
433899196,3cyclictest87350irq/126-enp1s0-12:35:233
433899196,3cyclictest87350irq/126-enp1s0-12:35:233
433899194,8cyclictest0-21swapper/310:35:033
433899194,4cyclictest13513-21tune2fs09:55:153
433899194,3cyclictest15816-21fschecks_time10:00:143
433899193,9cyclictest0-21swapper/311:57:083
433899193,4cyclictest9618-21diskstats11:00:153
433899192,5cyclictest5779-21sed09:35:263
433899192,4cyclictest4193-21memory12:00:173
433899192,4cyclictest24383-21df_inode10:20:153
433899192,4cyclictest0-21swapper/311:20:153
433899191,9cyclictest0-21swapper/309:22:323
433899191,7cyclictest0-21swapper/311:17:273
433899191,7cyclictest0-21swapper/309:14:073
433899191,5cyclictest39-21rcuc/309:40:163
433899191,5cyclictest15887-21smart_sda08:45:243
4338991915,3cyclictest0-21swapper/310:25:233
4338991915,2cyclictest39-21rcuc/311:00:013
433899191,10cyclictest7416-21/usr/sbin/munin07:15:243
433199190,11cyclictest0-21swapper/211:50:182
431999198,3cyclictest14822-21irqstats07:30:180
431999197,3cyclictest5884-21kworker/0:309:55:160
431999195,7cyclictest0-21swapper/010:49:390
431999195,12cyclictest11-21rcuc/007:40:150
431999194,12cyclictest7-21ksoftirqd/008:35:200
431999194,12cyclictest11-21rcuc/012:15:160
431999193,3cyclictest0-21swapper/010:37:530
431999193,3cyclictest0-21swapper/009:36:540
431999193,13cyclictest17816-21needreboot08:50:210
431999191,9cyclictest17383-21sed11:15:250
431999191,6cyclictest21074-21irqstats11:25:170
431999191,6cyclictest21074-21irqstats11:25:170
4319991913,3cyclictest13110-21smartctl08:40:180
431999191,15cyclictest30971-21grep08:05:260
431999191,11cyclictest0-21swapper/011:35:230
431999191,11cyclictest0-21swapper/009:42:440
431999191,10cyclictest18828-21if_enp2s011:20:180
431999191,10cyclictest10079-21irqstats11:00:190
431999191,10cyclictest0-21swapper/010:05:590
431999190,13cyclictest0-21swapper/011:45:240
280882190,2sleep00-21swapper/010:28:380
97932180,3sleep00-21swapper/008:32:290
433899184,3cyclictest5750irq/9-acpi08:30:153
433899183,3cyclictest20707-21ntp_states10:10:203
433899181,9cyclictest18403-21/usr/sbin/munin07:40:243
433899181,6cyclictest41-21ksoftirqd/311:28:183
433899181,6cyclictest41-21ksoftirqd/311:28:183
433899181,6cyclictest0-21swapper/312:30:123
433899181,5cyclictest26958-21latency_hist08:00:023
433899181,5cyclictest12197-21latency_hist08:40:003
4338991812,3cyclictest31246-21/usr/sbin/munin11:50:123
433899181,11cyclictest87350irq/126-enp1s0-07:25:143
431999186,2cyclictest5884-21kworker/0:309:52:220
431999184,9cyclictest0-21swapper/008:45:170
431999181,7cyclictest0-21swapper/007:36:240
431999181,6cyclictest0-21swapper/010:12:550
431999181,15cyclictest0-21swapper/010:50:230
431999181,10cyclictest24273-21sed07:50:220
431999181,10cyclictest24273-21sed07:50:220
431999181,10cyclictest0-21swapper/012:36:360
431999181,10cyclictest0-21swapper/012:36:360
431999181,10cyclictest0-21swapper/010:56:230
431999180,11cyclictest0-21swapper/011:40:120
431999180,10cyclictest29493-21ntpq10:30:210
32302181,2sleep00-21swapper/008:17:550
293942181,2sleep30-21swapper/308:05:113
251232180,2sleep00-21swapper/009:06:430
208152181,2sleep00-21swapper/008:57:560
8952170,3sleep00-21swapper/008:11:370
433899176,3cyclictest27192-21/usr/sbin/munin08:00:193
433899171,9cyclictest0-21swapper/307:53:333
433899171,9cyclictest0-21swapper/307:53:333
433899171,5cyclictest21016-21sort09:00:013
433899171,4cyclictest579-21systemd-journal08:20:013
433899171,4cyclictest0-21swapper/307:30:133
4338991712,3cyclictest41-21ksoftirqd/308:23:393
433199173,7cyclictest0-21swapper/212:26:162
431999171,3cyclictest32278-21ls09:25:120
431999171,14cyclictest0-21swapper/009:00:050
316912171,2sleep00-21swapper/009:21:240
431999167,6cyclictest0-21swapper/007:25:160
4319991611,3cyclictest7416-21/usr/sbin/munin07:15:240
4319991611,3cyclictest27192-21/usr/sbin/munin08:00:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional