You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-27 - 18:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 27, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
197902240163,24sleep20-21swapper/207:06:042
200752234163,58sleep30-21swapper/307:09:433
199792224154,56sleep10-21swapper/107:08:331
200462220189,21sleep00-21swapper/007:09:230
1543821520,4sleep02018299cyclictest08:10:150
17162520,4sleep011-21rcuc/007:40:010
215052480,1sleep30-21swapper/310:50:193
103632450,4sleep121-21rcuc/111:40:131
2019999380,27cyclictest0-21swapper/312:10:243
2018899380,25cyclictest0-21swapper/109:11:541
20182993810,13cyclictest16094-21munin-run10:40:020
2018899371,12cyclictest0-21swapper/108:27:291
2018899370,10cyclictest0-21swapper/107:39:201
2019999360,13cyclictest0-21swapper/307:31:333
20188993615,7cyclictest5619-21systemd-journal09:00:211
2018899360,22cyclictest0-21swapper/110:07:331
217462350,1sleep30-21swapper/312:05:163
2019999351,25cyclictest0-21swapper/311:43:133
2019999350,25cyclictest0-21swapper/311:45:493
2018299359,13cyclictest8896-21idleruntime-cro09:10:020
226472340,4sleep22019399cyclictest09:40:152
2019999340,26cyclictest0-21swapper/308:19:383
2019999340,25cyclictest0-21swapper/310:09:463
2018899341,9cyclictest0-21swapper/108:50:311
2018899341,12cyclictest0-21swapper/109:40:221
2019999331,22cyclictest31322-21latency08:45:223
2019999331,22cyclictest31322-21latency08:45:223
2019999331,19cyclictest23951-21hddtemp_smartct07:15:173
2018899331,7cyclictest0-21swapper/111:06:301
2018899331,11cyclictest0-21swapper/110:44:251
2018299338,12cyclictest32442-21munin-run08:50:010
2018299338,12cyclictest32442-21munin-run08:50:010
2018299335,13cyclictest24891-21munin-run11:00:010
2018299334,14cyclictest4475-21munin-run09:00:020
2019999321,10cyclictest0-21swapper/308:10:463
2018899321,9cyclictest0-21swapper/109:47:541
2018899321,18cyclictest0-21swapper/112:24:371
2018899321,10cyclictest0-21swapper/112:06:451
2018899320,9cyclictest0-21swapper/111:03:021
2019999310,12cyclictest0-21swapper/307:20:453
2018899315,3cyclictest0-21swapper/108:02:171
2018899311,10cyclictest0-21swapper/110:16:331
2018899310,9cyclictest0-21swapper/112:02:331
2018899310,9cyclictest0-21swapper/111:45:291
2018899310,9cyclictest0-21swapper/111:15:381
2018899310,9cyclictest0-21swapper/111:15:381
2018899310,8cyclictest0-21swapper/111:27:541
2018899310,10cyclictest0-21swapper/111:30:221
2018899310,10cyclictest0-21swapper/110:58:451
2018899310,10cyclictest0-21swapper/109:39:021
2019999305,2cyclictest0-21swapper/308:22:303
2019999301,23cyclictest0-21swapper/311:52:043
2019999301,16cyclictest0-21swapper/307:39:053
2019999301,10cyclictest0-21swapper/310:24:103
2019999301,10cyclictest0-21swapper/310:00:013
2019999300,17cyclictest0-21swapper/308:32:583
2018899303,3cyclictest0-21swapper/107:17:171
2018899301,9cyclictest0-21swapper/110:38:491
2018899301,11cyclictest0-21swapper/111:35:211
2018899300,8cyclictest0-21swapper/110:49:211
2018899300,18cyclictest0-21swapper/110:21:491
2018299309,12cyclictest8146-21/usr/sbin/munin11:35:190
20182993011,7cyclictest4564-21/usr/sbin/munin11:25:190
20199992916,7cyclictest29518-21/usr/sbin/munin11:10:243
2019999291,14cyclictest0-21swapper/310:48:523
2019999291,12cyclictest0-21swapper/308:58:133
2019999291,11cyclictest0-21swapper/312:28:543
2019999291,11cyclictest0-21swapper/312:28:543
2019999290,9cyclictest0-21swapper/308:39:223
2019999290,12cyclictest0-21swapper/312:16:023
2019999290,12cyclictest0-21swapper/307:54:533
2019999290,11cyclictest0-21swapper/309:35:493
2018899296,3cyclictest0-21swapper/108:23:451
2018899291,9cyclictest0-21swapper/107:59:331
2018899291,8cyclictest0-21swapper/108:13:531
2018899291,25cyclictest0-21swapper/107:12:491
2018899291,11cyclictest0-21swapper/109:15:221
2018899291,10cyclictest0-21swapper/108:37:311
2018899291,10cyclictest0-21swapper/107:30:101
2018899290,9cyclictest0-21swapper/112:35:331
2018899290,9cyclictest0-21swapper/109:52:461
2018899290,9cyclictest0-21swapper/109:29:061
2018899290,10cyclictest0-21swapper/112:17:491
2018899290,10cyclictest0-21swapper/107:51:171
2018299299,8cyclictest5620-21/usr/sbin/munin09:00:220
20182992910,9cyclictest17683-21awk09:30:010
2019999281,22cyclictest0-21swapper/310:37:263
2019999281,17cyclictest0-21swapper/310:04:013
2019999281,17cyclictest0-21swapper/310:04:013
2019999281,12cyclictest0-21swapper/308:43:463
2019999280,9cyclictest0-21swapper/311:24:013
2019999280,14cyclictest0-21swapper/312:31:443
2018899285,7cyclictest620-21munin-run10:05:011
2018899285,7cyclictest620-21munin-run10:05:011
2018899285,3cyclictest0-21swapper/107:20:281
2018899281,9cyclictest0-21swapper/110:25:251
2018899281,9cyclictest0-21swapper/108:16:451
2018899281,7cyclictest0-21swapper/109:58:581
2018899281,21cyclictest0-21swapper/107:42:251
2018899281,20cyclictest7125-21kworker/u8:210:51:121
2018899281,10cyclictest0-21swapper/108:49:551
2018899281,10cyclictest0-21swapper/108:49:551
2018899280,9cyclictest0-21swapper/109:24:581
2018899280,21cyclictest0-21swapper/110:32:541
2018899280,14cyclictest0-21swapper/111:20:561
2018899280,10cyclictest0-21swapper/110:14:171
2018299288,9cyclictest11720-21kworker/u8:009:20:210
2018299287,13cyclictest19136-21/usr/sbin/munin12:00:210
2018299286,19cyclictest20726-21gltestperf09:35:150
2018299283,12cyclictest636-21basename10:05:010
2018299283,12cyclictest636-21basename10:05:010
2018299280,13cyclictest12329-21awk11:45:010
2019999271,12cyclictest0-21swapper/307:25:193
2019999271,10cyclictest0-21swapper/307:10:423
2019999270,19cyclictest0-21swapper/309:48:493
2019999270,15cyclictest0-21swapper/311:32:253
2019999270,15cyclictest0-21swapper/309:41:493
2019999270,13cyclictest0-21swapper/308:51:473
2019999270,11cyclictest0-21swapper/311:38:383
2019999270,10cyclictest0-21swapper/309:34:493
2018899276,2cyclictest0-21swapper/107:25:461
20188992712,9cyclictest0-21swapper/108:56:271
2018899271,22cyclictest6058-21latency_hist07:50:011
2018299276,8cyclictest6540-21head11:30:160
2018299276,12cyclictest21352-21/usr/sbin/munin12:05:240
2018299276,10cyclictest10380-21head10:25:180
20182992710,14cyclictest30122-21sshd08:43:370
2019999261,9cyclictest10102-21grep09:10:213
2019999261,8cyclictest0-21swapper/307:47:013
2019999261,11cyclictest0-21swapper/312:24:343
2019999260,13cyclictest0-21swapper/307:42:513
2019999260,12cyclictest0-21swapper/307:56:023
2019999260,11cyclictest0-21swapper/312:36:063
2019999260,11cyclictest0-21swapper/311:57:103
2019999260,11cyclictest0-21swapper/310:15:183
2019999260,10cyclictest0-21swapper/309:01:253
2018899261,8cyclictest0-21swapper/108:07:211
2018899261,11cyclictest0-21swapper/111:11:441
2018899261,11cyclictest0-21swapper/109:06:591
2018299269,5cyclictest5333-21/usr/sbin/munin10:15:260
2018299268,7cyclictest19031-21grep10:45:160
2018299268,15cyclictest12668-21perf08:05:010
20182992617,5cyclictest26098-21diskstats12:15:170
20182992611,8cyclictest14808-21cpuspeed_turbos11:50:140
2019999251,9cyclictest0-21swapper/309:09:133
2019999251,13cyclictest0-21swapper/311:28:073
2019999250,11cyclictest0-21swapper/311:06:253
2019999250,11cyclictest0-21swapper/309:53:213
2019999250,11cyclictest0-21swapper/308:27:223
2019999250,10cyclictest0-21swapper/310:29:463
2018899257,3cyclictest0-21swapper/112:13:511
20188992512,8cyclictest0-21swapper/108:30:261
2018899250,9cyclictest0-21swapper/112:25:331
2018899250,9cyclictest0-21swapper/112:25:331
2018899250,9cyclictest0-21swapper/108:41:571
2018299259,12cyclictest8107-21chrt10:20:130
2018299258,5cyclictest26690-21/usr/sbin/munin09:50:210
2018299256,16cyclictest31702-21sed11:15:010
2018299256,13cyclictest0-21swapper/010:05:240
2018299255,17cyclictest9109-21/usr/sbin/munin09:10:160
2018299253,19cyclictest25074-21sed12:10:260
20182992517,5cyclictest13188-21smart_sda10:30:250
20182992513,3cyclictest4645-21sed10:10:270
20182992511,11cyclictest17291-21diskstats11:55:170
20182992510,5cyclictest773-21NetworkManager08:33:090
311922240,5sleep22019399cyclictest12:25:212
311922240,5sleep22019399cyclictest12:25:212
2019999241,11cyclictest0-21swapper/308:03:543
2019999241,10cyclictest0-21swapper/309:23:153
2019999240,11cyclictest0-21swapper/312:02:583
2019999240,11cyclictest0-21swapper/310:40:103
2019999240,10cyclictest0-21swapper/311:18:253
2019999240,10cyclictest0-21swapper/311:18:253
2018899240,9cyclictest0-21swapper/112:31:171
2018299245,10cyclictest17898-21/usr/sbin/munin09:30:210
2018299242,13cyclictest17285-21/usr/sbin/munin08:15:190
2018299242,11cyclictest0-21swapper/011:45:170
20182992416,5cyclictest26427-21diskstats08:35:140
20182992415,6cyclictest21193-21/usr/sbin/munin10:50:170
20182992415,6cyclictest20754-21smart_sda08:20:230
20182992415,6cyclictest14815-40dnf08:09:370
2019999232,7cyclictest4049-40dnf10:10:223
2019999231,11cyclictest0-21swapper/308:07:283
2019999230,11cyclictest0-21swapper/310:31:183
20193992311,6cyclictest30956-21wc11:10:252
2019399231,10cyclictest0-21swapper/210:55:012
2018899231,10cyclictest0-21swapper/111:55:411
2018299234,3cyclictest87350irq/126-enp1s0-11:20:150
2018299234,11cyclictest0-21swapper/009:50:010
2018299233,15cyclictest87350irq/126-enp1s0-11:09:070
20182992316,4cyclictest29633-21/usr/sbin/munin09:55:160
20182992316,4cyclictest27667-21/usr/sbin/munin07:25:110
20182992315,5cyclictest12569-21smart_sda09:15:250
20182992314,6cyclictest313-21/usr/sbin/munin11:15:230
20182992314,6cyclictest313-21/usr/sbin/munin11:15:230
20182992314,6cyclictest30139-21/usr/sbin/munin12:25:230
20182992314,6cyclictest30139-21/usr/sbin/munin12:25:230
20182992314,6cyclictest25470-21/usr/sbin/munin07:20:130
20182992313,7cyclictest1356-21ntp_states08:50:220
2018299231,14cyclictest0-21swapper/011:01:250
2019999220,9cyclictest0-21swapper/311:04:053
2018899221,9cyclictest0-21swapper/109:33:321
2018899221,20cyclictest0-21swapper/111:51:051
2018299226,12cyclictest23579-21switchtime09:40:240
20182992215,4cyclictest30675-21irqstats07:30:180
20182992214,5cyclictest29013-21ntp_states12:20:210
20182992214,5cyclictest22755-21memory07:15:030
20182992213,6cyclictest8487-21/usr/sbin/munin07:55:210
20182992213,6cyclictest22766-21open_files08:25:220
20182992213,6cyclictest20567-21cstates07:10:120
20182992212,7cyclictest1041-21ntp_states12:30:220
2018299221,16cyclictest0-21swapper/012:38:240
95432211,16sleep29555-21ntp_states07:55:222
2019999211,11cyclictest0-21swapper/310:56:393
2019999211,10cyclictest0-21swapper/309:28:333
2019399214,12cyclictest808-21snmpd08:09:422
2019399211,6cyclictest0-21swapper/209:57:592
2019399211,6cyclictest0-21swapper/209:30:532
2019399211,5cyclictest25566-21ntp_states09:45:242
2019399211,4cyclictest0-21swapper/211:55:362
20182992114,4cyclictest4433-21/usr/sbin/munin07:45:150
20182992113,5cyclictest7551-21smart_sda07:50:220
20182992112,6cyclictest1896-21/usr/sbin/munin07:40:150
20182992112,6cyclictest16319-21/usr/sbin/munin10:40:190
2019999201,13cyclictest0-21swapper/309:19:373
2019399204,5cyclictest6760-21timerwakeupswit10:15:262
2019399202,15cyclictest11922-21grep11:40:272
2019399201,5cyclictest0-21swapper/212:38:182
2019399201,4cyclictest0-21swapper/211:53:432
2019399201,11cyclictest29288-21grep12:20:252
2019399195,6cyclictest0-21swapper/209:10:232
2019399193,3cyclictest87550irq/128-enp1s0-09:35:262
2019399192,7cyclictest22415-21ntp_states12:05:212
2019399192,14cyclictest27809-21grep07:25:132
2019399191,5cyclictest18245-1kworker/u9:210:34:112
2019399191,5cyclictest0-21swapper/211:06:002
2019399191,4cyclictest0-21swapper/210:38:042
20193991912,4cyclictest4642-21grep10:10:282
20193991912,4cyclictest31735-21sed10:00:172
20193991912,4cyclictest31735-21sed10:00:172
20193991912,4cyclictest18512-21grep08:15:222
2019399191,11cyclictest0-21swapper/212:15:302
2019399184,3cyclictest35862awk08:55:222
2019399181,7cyclictest0-21swapper/209:54:012
2019399181,6cyclictest31375-21grep07:30:272
2019399181,6cyclictest0-21swapper/209:29:292
2019399181,4cyclictest0-21swapper/212:14:232
20193991814,2cyclictest2000-21ntpq10:05:222
2019399181,3cyclictest11639-21df09:15:132
20193991812,3cyclictest26124-21tail07:20:152
20193991812,3cyclictest14313-21irqstats09:20:172
20193991811,4cyclictest2496-21sed11:20:212
20193991811,4cyclictest1827-21grep08:50:252
2019399180,10cyclictest0-21swapper/208:45:182
2019399180,10cyclictest0-21swapper/208:45:182
93722171,2sleep20-21swapper/210:23:292
86702171,2sleep20-21swapper/209:07:322
2019399176,3cyclictest0-21swapper/211:39:182
2019399175,3cyclictest24043-21tr08:30:122
2019399174,4cyclictest4096-21/usr/sbin/munin07:45:172
2019399172,6cyclictest0-21swapper/210:55:172
2019399171,7cyclictest18517-21/usr/sbin/munin10:45:142
2019399171,7cyclictest13391-21latency11:45:192
2019399171,7cyclictest0-21swapper/212:04:072
2019399171,5cyclictest0-21swapper/207:50:402
20193991712,3cyclictest3450-21head07:40:272
20193991712,3cyclictest32389-21sed11:15:182
20193991712,3cyclictest32389-21sed11:15:172
20193991712,3cyclictest11270-21unixbench_singl10:25:272
2019399171,15cyclictest0-21swapper/210:44:322
2019399171,15cyclictest0-21swapper/208:13:422
2019399171,15cyclictest0-21swapper/207:17:472
2019399171,14cyclictest0-21swapper/208:25:052
17172171,2sleep20-21swapper/212:31:562
2019399161,9cyclictest0-21swapper/211:25:412
2019399161,5cyclictest0-21swapper/207:35:182
2019399161,4cyclictest0-21swapper/207:10:192
2019399161,2cyclictest87650irq/129-enp1s0-09:00:292
20193991612,3cyclictest0-21swapper/211:00:042
20193991611,3cyclictest5948-21/usr/sbin/munin11:30:182
20193991611,3cyclictest29350-21ntpq08:40:222
20193991610,3cyclictest26080-21/usr/sbin/munin08:35:232
2019399160,15cyclictest0-21swapper/208:03:302
20193991511,2cyclictest19484-21/usr/sbin/munin08:20:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional