You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 21:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 20, 2025 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
283682251164,23sleep00-21swapper/007:09:170
280822237165,24sleep20-21swapper/207:05:322
284272230164,54sleep30-21swapper/307:09:593
282092213163,37sleep10-21swapper/107:07:131
2996321710,2sleep32852999cyclictest12:05:113
871321560,5sleep12851999cyclictest07:35:181
28524999693,2cyclictest0-21swapper/210:29:422
57792660,2sleep011-21rcuc/008:40:260
70802540,1sleep00-21swapper/011:10:270
28519994313,19cyclictest87150irq/128-enp1s0-09:35:021
2852499404,29cyclictest13797-40dnf11:26:572
2852499402,32cyclictest31621-40dnf08:25:312
2852499402,32cyclictest31621-40dnf08:25:302
2852499401,29cyclictest782-21in:imjournal08:00:292
2852499401,28cyclictest24034-21/usr/sbin/munin09:25:162
254892400,4sleep081ktimersoftd/011:55:130
2851299391,16cyclictest30846-21kworker/0:010:54:210
2852499381,26cyclictest24419-21sh10:40:012
2852499381,24cyclictest1465-21fschecks_count11:00:162
2852499371,22cyclictest10504-21find10:05:252
2852499371,21cyclictest30782-21cat07:15:022
2852499371,21cyclictest30782-21cat07:15:022
2852499361,24cyclictest359-21/usr/sbin/munin09:45:202
2852499361,22cyclictest17221-21diskmemload11:30:122
2852499361,21cyclictest18287-21cut09:10:192
2852499352,23cyclictest16828-21basename07:55:132
2852499351,25cyclictest13668-21/usr/sbin/munin10:15:182
2852499351,25cyclictest10773-21cat12:35:162
2852499351,21cyclictest29686-21mailstats09:35:222
2852499351,21cyclictest29686-21mailstats09:35:222
2852499350,24cyclictest0-21swapper/211:05:242
2851299358,13cyclictest10619-21munin-run08:55:020
2852499342,27cyclictest5178-21smart_sda12:20:292
2852499341,29cyclictest20113-21sort09:15:152
2852499341,25cyclictest9364-21runrttasks11:16:222
2852499341,22cyclictest1423-21/usr/sbin/munin07:20:182
2852499341,19cyclictest5448-21gltestperf09:55:162
2852499341,19cyclictest31267-21cpu10:55:142
2851299342,25cyclictest20927-21kworker/0:108:05:170
2852499332,27cyclictest25961-21diskstats08:15:152
2852499332,27cyclictest16211-21diskstats10:20:172
2852499332,26cyclictest10232-21/usr/sbin/munin07:40:282
2852499332,21cyclictest5840-21/usr/sbin/munin07:30:252
2852499331,29cyclictest32523-21gltestperf12:10:162
2852499331,28cyclictest4907-21smart_sda07:25:272
2852499331,25cyclictest8999-21chrt10:04:362
2852499331,24cyclictest28017-21perl12:00:162
2852499331,24cyclictest28017-21perl12:00:162
2852499331,22cyclictest26239-21sshd10:40:292
2852499331,22cyclictest10212-21cat11:20:132
2852499331,20cyclictest2021-21cstates12:15:152
2852499331,18cyclictest801-21snmpd08:58:002
2852499324,20cyclictest86950irq/126-enp1s0-09:00:222
2852499322,19cyclictest21054-21cstates11:45:132
2852499321,28cyclictest6450-21/usr/sbin/munin08:45:152
2852499321,27cyclictest7866-40dnf12:27:292
2852499321,21cyclictest26231-21/usr/sbin/munin09:30:152
2852499321,20cyclictest4489-21kworker/2:107:35:192
2852499321,19cyclictest21838-21/usr/sbin/munin09:20:182
2852499321,19cyclictest18233-21kworker/u8:011:55:192
2852499320,21cyclictest30634-21users10:50:282
2851299324,13cyclictest28839-21perf10:50:000
2851299324,13cyclictest28839-21perf10:50:000
2852499314,19cyclictest87150irq/128-enp1s0-08:40:252
2852499311,26cyclictest32563-21diskstats08:30:152
2852499311,26cyclictest30984-21/usr/sbin/munin09:40:172
2852499311,20cyclictest9219-21memory12:30:212
2852499311,19cyclictest30433-21awk12:05:162
2852499311,19cyclictest11814-21diskstats10:10:142
2852499311,18cyclictest6304-21kernelversion11:10:182
2852499311,18cyclictest23189-21latency_hist08:10:012
2852499310,16cyclictest0-21swapper/208:50:232
2851299311,24cyclictest17794-21kworker/0:110:26:330
2851299310,24cyclictest28066-21kworker/0:209:36:160
2851299310,24cyclictest28066-21kworker/0:209:36:160
2852499304,17cyclictest86950irq/126-enp1s0-10:45:262
2852499304,17cyclictest86950irq/126-enp1s0-10:45:262
2852499303,23cyclictest31540-21munin-node07:15:042
2852499303,23cyclictest31540-21munin-node07:15:042
2852499301,26cyclictest27810-21/usr/sbin/munin08:20:142
2852499301,26cyclictest23918-21/usr/sbin/munin11:50:212
2852499301,19cyclictest23420-21/usr/sbin/munin08:10:202
2852499301,18cyclictest19324-21head11:40:172
2852499301,17cyclictest2672-21/usr/sbin/munin09:50:182
2852499300,27cyclictest16476-21/usr/sbin/munin11:35:182
2852499300,26cyclictest12431-21/usr/sbin/munin07:45:182
2851999301,13cyclictest0-21swapper/112:31:431
2851999301,13cyclictest0-21swapper/110:32:011
2851999301,12cyclictest0-21swapper/109:26:021
2851299302,13cyclictest7737-21kworker/0:012:35:010
2851299301,9cyclictest4520-21kworker/0:009:55:130
2851299301,10cyclictest10003-21kworker/0:212:38:170
2852499292,24cyclictest20606-21diskstats10:30:162
2852499291,25cyclictest2058-21/usr/sbin/munin08:35:252
2852499291,25cyclictest15248-21/usr/sbin/munin09:05:152
2852499291,25cyclictest14626-21/usr/sbin/munin07:50:252
2851999291,9cyclictest801-21snmpd09:20:131
2851999291,21cyclictest0-21swapper/111:29:291
2851999291,21cyclictest0-21swapper/108:50:171
2851999291,13cyclictest0-21swapper/112:14:341
2851999291,13cyclictest0-21swapper/107:26:401
2851999291,12cyclictest0-21swapper/110:23:331
2851999291,11cyclictest0-21swapper/109:46:261
2851299292,10cyclictest11573-21kworker/0:011:24:560
2851299291,22cyclictest31502-21kworker/0:212:16:590
2851299291,20cyclictest8986-21kworker/0:010:08:150
2851299291,19cyclictest21862-21kworker/0:010:35:430
2851299291,10cyclictest11573-21kworker/0:011:33:340
86492280,5sleep339-21rcuc/312:30:163
2851999281,10cyclictest0-21swapper/111:47:301
2851299283,12cyclictest3074-21munin-run11:05:020
2851299282,25cyclictest4618-21kworker/0:112:23:430
2851299282,13cyclictest8080-21munin-run12:30:010
2851299282,12cyclictest14415-21perf07:50:010
2851299281,10cyclictest11573-21kworker/0:011:25:040
2851999271,8cyclictest0-21swapper/107:53:501
2851999271,11cyclictest0-21swapper/111:34:001
2851999271,11cyclictest0-21swapper/110:03:541
2851999270,9cyclictest0-21swapper/109:07:061
2851999270,10cyclictest0-21swapper/108:56:211
2851299272,9cyclictest8202-21kworker/0:108:49:520
2851299272,12cyclictest26634-21awk10:45:000
2851299271,9cyclictest11573-21kworker/0:011:36:300
2851299271,11cyclictest15554-21kworker/0:010:23:370
2851299271,11cyclictest13318-21kworker/0:110:14:170
2851299270,13cyclictest17794-21kworker/0:110:33:130
2851999261,21cyclictest6408-21grep09:55:271
2851999261,10cyclictest0-21swapper/110:16:311
2851999260,8cyclictest17444-21kworker/u8:009:11:121
2851999260,12cyclictest0-21swapper/108:02:151
2851299269,2cyclictest28066-21kworker/0:209:40:070
2851299267,4cyclictest18329-21kworker/0:111:48:580
2851299263,11cyclictest15647-21awk10:20:010
28512992612,2cyclictest10490-21kworker/0:009:01:300
2851299261,10cyclictest17192-21kworker/0:109:12:380
145482260,3sleep114550-21unixbench_multi09:00:251
2852999250,10cyclictest0-21swapper/311:25:253
2851999259,2cyclictest0-21swapper/112:05:181
2851999251,9cyclictest0-21swapper/109:36:321
2851999251,9cyclictest0-21swapper/109:36:321
2851999251,9cyclictest0-21swapper/108:24:231
28519992514,3cyclictest4262-21users09:50:311
2851999251,17cyclictest0-21swapper/110:56:261
2851999251,10cyclictest0-21swapper/108:30:311
2851999250,8cyclictest0-21swapper/111:24:391
2851299256,16cyclictest9049-21munin-run10:05:010
2851299252,12cyclictest9357-21kworker/0:111:17:360
2851299251,11cyclictest17192-21kworker/0:109:19:520
2851299251,10cyclictest30846-21kworker/0:011:07:460
2851299251,10cyclictest25835-21kworker/0:109:30:300
28529992418,3cyclictest18034-21grep11:35:273
2851999245,8cyclictest30883-21gltestperf07:15:021
2851999245,8cyclictest30883-21gltestperf07:15:011
2851999244,9cyclictest751-21rngd11:07:011
2851999241,9cyclictest0-21swapper/111:16:551
2851999241,9cyclictest0-21swapper/110:50:531
2851999241,14cyclictest0-21swapper/111:40:001
2851999241,11cyclictest0-21swapper/107:23:091
2851999241,10cyclictest0-21swapper/107:57:461
2851299247,3cyclictest14108-21kworker/0:207:55:250
2851299242,9cyclictest27022-21kworker/0:012:03:170
2851299242,9cyclictest27022-21kworker/0:012:03:170
2851299241,9cyclictest30846-21kworker/0:010:58:230
2851299241,9cyclictest23587-21kworker/0:009:22:560
2851299241,17cyclictest2296-21kworker/0:109:52:450
2851299241,10cyclictest29670-21kworker/0:308:24:310
2851299241,10cyclictest14108-21kworker/0:207:51:530
90022230,3sleep011-21rcuc/007:35:210
2851999237,3cyclictest0-21swapper/108:48:371
2851999237,2cyclictest12050irq/123-ahci[0007:50:021
2851999232,16cyclictest87150irq/128-enp1s0-11:14:371
2851999231,9cyclictest0-21swapper/107:44:261
2851999231,11cyclictest0-21swapper/110:12:561
2851299238,4cyclictest28193-21kworker/0:107:12:130
2851299238,4cyclictest28193-21kworker/0:107:12:130
2851299232,12cyclictest3258-21kworker/0:307:32:320
2851299231,16cyclictest31620-21kworker/0:312:11:330
2851299231,12cyclictest18329-21kworker/0:111:42:140
263982230,1sleep00-21swapper/008:15:170
2852999228,2cyclictest6246-21if_enp2s011:10:183
2852999224,9cyclictest19779-21pmu-power11:40:243
2852999221,9cyclictest28334-21unixbench_multi10:45:263
2852999221,9cyclictest28334-21unixbench_multi10:45:263
2852999221,18cyclictest24214-21pmu-power11:50:263
2852999221,15cyclictest0-21swapper/307:56:273
2851999229,11cyclictest421-21kworker/u8:111:00:351
2851999227,3cyclictest0-21swapper/108:15:051
2851999223,2cyclictest23-21ksoftirqd/107:15:251
2851999223,2cyclictest23-21ksoftirqd/107:15:251
2851999223,12cyclictest23-21ksoftirqd/108:10:151
2851999221,9cyclictest0-21swapper/111:51:281
2851999221,9cyclictest0-21swapper/108:08:171
2851999221,8cyclictest0-21swapper/112:28:011
2851999221,6cyclictest0-21swapper/110:05:121
28519992213,2cyclictest17221-21diskmemload11:58:561
2851999221,15cyclictest0-21swapper/110:25:161
2851999221,13cyclictest20342-21basename09:15:171
2851999221,12cyclictest25606-21ls10:40:211
2851999221,11cyclictest0-21swapper/112:20:481
2851999221,10cyclictest0-21swapper/108:25:031
2851999221,10cyclictest0-21swapper/108:25:031
2851999221,10cyclictest0-21swapper/107:32:221
28519992210,4cyclictest27808-21diskstats12:00:141
28519992210,4cyclictest27808-21diskstats12:00:141
2851299228,3cyclictest28193-21kworker/0:107:23:500
2851299228,2cyclictest22772-21kworker/0:211:50:140
2851299227,8cyclictest30488-21kworker/0:007:16:460
2851299227,8cyclictest30488-21kworker/0:007:16:460
2851299226,3cyclictest16342-21kworker/0:008:01:070
2851299225,2cyclictest3258-21kworker/0:307:26:400
2851299222,9cyclictest30188-21kworker/0:309:48:210
2851299221,9cyclictest14908-21kworker/0:209:05:060
28512992211,2cyclictest23587-21kworker/0:009:26:460
112302220,2sleep339-21rcuc/311:20:243
2852999218,3cyclictest13116-21sed07:45:173
2852999216,5cyclictest0-21swapper/310:38:073
2852999212,10cyclictest4249-21/usr/sbin/munin08:40:143
2852999211,11cyclictest634-21grep07:15:253
2852999211,11cyclictest634-21grep07:15:253
2852999211,11cyclictest25651-21fschecks_count11:55:153
2852999211,11cyclictest21519-21smart_sda10:30:253
2851999217,3cyclictest0-21swapper/110:37:451
28519992113,3cyclictest0-21swapper/108:36:001
2851999211,18cyclictest5303-21ntp_offset08:40:201
2851999211,15cyclictest0-21swapper/110:46:121
2851999211,15cyclictest0-21swapper/110:46:121
2851999211,11cyclictest3676-21sh12:20:021
2851999211,10cyclictest0-21swapper/111:41:221
2851299216,5cyclictest29277-21kworker/0:112:07:050
2851299212,11cyclictest16342-21kworker/0:008:10:230
285772200,4sleep339-21rcuc/308:20:183
2852999209,3cyclictest17453-21uptime10:20:283
2852999201,11cyclictest20603-21users08:00:283
2852999201,10cyclictest4865-21/usr/sbin/munin09:55:113
2852999201,10cyclictest28643-21processes12:00:223
2852999201,10cyclictest28643-21processes12:00:223
2852999201,10cyclictest25913-21smart_sda10:40:273
2852999201,10cyclictest0-21swapper/309:37:423
2852999201,10cyclictest0-21swapper/309:37:423
2852999200,7cyclictest0-21swapper/310:29:423
2851999201,14cyclictest0-21swapper/109:41:031
2851999201,11cyclictest12090-21users12:35:281
2851299206,2cyclictest87150irq/128-enp1s0-07:44:140
2851299201,5cyclictest31886-21kworker/0:008:33:490
220192201,15sleep322177-21df_inode09:20:143
36952191,3sleep3904-21sssd_nss09:50:243
2852999198,2cyclictest2205-21ethtool07:20:183
2852999196,4cyclictest0-21swapper/309:41:443
2852999195,4cyclictest0-21swapper/310:04:293
2852999195,3cyclictest0-21swapper/308:18:223
2852999194,8cyclictest9272-21/usr/sbin/munin10:05:193
2852999193,3cyclictest0-21swapper/312:39:413
2852999192,2cyclictest86950irq/126-enp1s0-10:55:213
2852999191,6cyclictest0-21swapper/312:29:263
2852999191,5cyclictest0-21swapper/312:10:073
2852999191,4cyclictest0-21swapper/309:25:283
2852999191,15cyclictest32328-21cpuspeed_turbos08:30:153
2852999191,15cyclictest30312-21vmstat07:10:273
2852999191,15cyclictest30312-21vmstat07:10:263
2852999191,10cyclictest5040-21pmu-power12:20:263
2852999191,10cyclictest3207-21unixbench_multi12:15:263
2852999191,10cyclictest29050-21/usr/sbin/munin10:50:273
2852999191,10cyclictest15267-21strings11:30:203
2852999191,10cyclictest0-21swapper/311:15:143
2852999191,10cyclictest0-21swapper/310:18:343
2852999191,10cyclictest0-21swapper/309:47:263
2852999191,10cyclictest0-21swapper/309:15:193
2852999191,10cyclictest0-21swapper/308:09:303
28512991912,4cyclictest30682-21sed08:25:170
28512991912,4cyclictest30682-21sed08:25:170
2852999184,12cyclictest39-21rcuc/308:45:163
2852999181,9cyclictest801-21snmpd09:33:073
2852999181,9cyclictest0-21swapper/311:45:233
2852999181,9cyclictest0-21swapper/311:06:393
2852999181,8cyclictest16383-21sshd07:52:273
2852999181,14cyclictest775-21dbus-daemon11:00:283
2852999181,10cyclictest22501-21kworker/3:110:10:243
2851299187,3cyclictest3998-21kworker/0:308:55:260
2851299182,11cyclictest31886-21kworker/0:008:36:320
2852999179,5cyclictest904-21sssd_nss07:30:163
2852999175,3cyclictest0-21swapper/308:51:003
2852999171,9cyclictest0-21swapper/308:10:193
28529991713,3cyclictest0-21swapper/309:09:343
28529991713,3cyclictest0-21swapper/308:37:483
2852999171,10cyclictest0-21swapper/309:10:023
28529991710,2cyclictest0-21swapper/308:26:033
28529991710,2cyclictest0-21swapper/308:26:033
2852999170,10cyclictest0-21swapper/308:58:173
2852999170,10cyclictest0-21swapper/307:40:193
2852999166,3cyclictest86950irq/126-enp1s0-07:35:253
2852999161,3cyclictest0-21swapper/307:25:193
28529991610,3cyclictest13139-21grep09:00:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional