You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 17:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 29, 2025 12:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
137602238162,24sleep20-21swapper/207:08:272
135842202165,25sleep10-21swapper/107:06:081
137382201164,25sleep00-21swapper/007:08:080
136262200164,24sleep30-21swapper/307:06:433
2744821020,4sleep21397299cyclictest08:50:192
54982880,1sleep30-21swapper/310:30:013
320272780,4sleep232-21ksoftirqd/209:00:232
156642530,2sleep00-21swapper/010:50:210
102832500,3sleep121-21rcuc/110:40:151
13956993510,12cyclictest19302-21munin-run12:15:020
1395699348,13cyclictest29045-21idleruntime-cro10:10:010
1395699348,13cyclictest24033-21munin-run08:45:010
1395699332,19cyclictest7036-21kworker/0:009:25:350
1395699331,18cyclictest24984-21kworker/0:211:22:380
1395699322,28cyclictest15147-21kworker/0:112:05:280
1395699311,28cyclictest10231-21kworker/0:312:02:060
307812300,5sleep230-21rcuc/210:10:262
1395699302,26cyclictest28969-21kworker/0:310:38:010
1395699302,12cyclictest24984-21kworker/0:211:15:180
1395699302,11cyclictest24984-21kworker/0:211:12:390
1395699301,27cyclictest10231-21kworker/0:311:53:180
1395699301,20cyclictest24984-21kworker/0:211:45:200
1395699301,19cyclictest26801-21kworker/0:209:14:310
1395699301,19cyclictest21696-21kworker/0:108:55:090
1395699298,3cyclictest24984-21kworker/0:211:35:180
1395699292,25cyclictest15147-21kworker/0:112:16:540
1395699291,26cyclictest2904-21kworker/0:110:24:050
1395699291,11cyclictest4837-21kworker/0:109:15:190
1395699281,8cyclictest24984-21kworker/0:211:29:290
1395699272,13cyclictest23084-21kworker/u8:010:02:570
1395699271,9cyclictest30006-21kworker/0:112:39:020
1395699271,8cyclictest5201-21kworker/0:011:43:290
1395699271,24cyclictest23568-21kworker/0:012:26:040
1395699271,24cyclictest20104-21kworker/0:009:57:250
1396699261,17cyclictest12357-21hddtemp_smartct09:30:181
1395699267,3cyclictest5261-21kworker/0:010:32:170
1395699267,2cyclictest9710-21kworker/0:010:42:170
1395699262,10cyclictest31185-21kworker/0:010:17:190
1395699261,23cyclictest24984-21kworker/0:211:30:320
1395699261,23cyclictest24984-21kworker/0:211:30:320
1395699261,10cyclictest14217-21kworker/0:211:02:410
13956992610,3cyclictest21098-21kworker/0:207:36:410
1397799258,9cyclictest3107-21processes11:35:233
1397799252,5cyclictest26674-40dnf10:02:593
1397799251,12cyclictest11681-21/usr/sbin/munin09:30:133
1397799251,11cyclictest23476-21sshd12:22:243
1397799251,10cyclictest11856-21sshd10:42:243
1396699251,16cyclictest835-21runrttasks08:22:371
1396699251,16cyclictest23582-21cat07:30:011
13966992510,3cyclictest288422expr08:55:141
1395699257,2cyclictest7036-21kworker/0:009:23:250
1395699252,11cyclictest14217-21kworker/0:210:57:260
1395699252,10cyclictest21098-21kworker/0:207:56:390
1395699252,10cyclictest21098-21kworker/0:207:56:390
1395699252,10cyclictest11140-21kworker/0:209:42:570
1395699251,9cyclictest21098-21kworker/0:208:06:170
1395699251,9cyclictest11140-21kworker/0:209:49:250
13956992511,2cyclictest23568-21kworker/0:012:23:560
1395699251,12cyclictest20415-21kworker/0:007:10:090
1395699251,12cyclictest11140-21kworker/0:209:35:590
1397799241,5cyclictest0-21swapper/309:39:513
1397799241,12cyclictest13265-21/usr/sbin/munin08:20:113
1396699242,3cyclictest9823-21kworker/u8:111:25:171
1396699240,19cyclictest0-21swapper/112:25:271
1395699243,2cyclictest5261-21kworker/0:010:28:010
1395699241,12cyclictest23370-21kworker/0:107:34:470
1395699241,10cyclictest21696-21kworker/0:108:53:110
13956992410,4cyclictest16096-21kworker/0:108:29:490
13956992410,3cyclictest20104-21kworker/0:009:50:170
194912231,2sleep319534-21smart_sda09:45:233
1397799238,3cyclictest5756-21sshd11:40:443
1397799237,3cyclictest0-21swapper/311:20:303
1397799233,10cyclictest0-21swapper/311:14:243
1397799231,11cyclictest10903-21cat09:25:243
1397299237,7cyclictest30421-1kworker/u9:209:14:032
1397299231,11cyclictest0-21swapper/211:11:362
1396699239,12cyclictest10369-21kworker/u8:112:04:371
1396699233,9cyclictest801-21snmpd10:08:101
1396699232,16cyclictest0-21swapper/109:28:001
1396699231,3cyclictest20653-21cstates09:50:131
1396699231,12cyclictest16134-21latency_hist07:15:021
1395699238,3cyclictest18859-21kworker/0:007:24:390
1395699237,2cyclictest26108-21kworker/0:212:33:380
1395699232,9cyclictest22762-21kworker/0:111:06:550
1395699232,9cyclictest12534-21kworker/0:011:58:330
1395699232,8cyclictest28969-21kworker/0:310:10:310
1395699232,11cyclictest11922-21kworker/0:107:18:110
1397799226,5cyclictest20423-21ps07:20:233
1397799226,3cyclictest5092-21/usr/sbin/munin09:15:143
1397799221,5cyclictest0-21swapper/311:25:573
13977992215,4cyclictest23271-21head09:55:153
1397799221,18cyclictest86950irq/126-enp1s0-10:10:283
1397299221,13cyclictest0-21swapper/211:37:102
1396699221,8cyclictest0-21swapper/110:10:091
13966992214,3cyclictest0-21swapper/111:46:101
13966992214,3cyclictest0-21swapper/110:17:291
1396699221,2cyclictest87250irq/129-enp1s0-08:15:211
13966992212,6cyclictest0-21swapper/111:58:311
1396699221,16cyclictest0-21swapper/110:35:491
1396699220,16cyclictest0-21swapper/109:00:161
1395699226,3cyclictest23925-21kworker/0:008:45:150
1395699226,2cyclictest21098-21kworker/0:207:26:330
1395699225,15cyclictest6455-21munin-run08:05:010
1395699222,11cyclictest11983-21kworker/0:110:48:170
1395699222,11cyclictest10706-21kworker/0:308:17:270
1395699221,10cyclictest21098-21kworker/0:207:52:030
59112210,4sleep31397799cyclictest08:00:243
314632211,2sleep30-21swapper/310:15:053
305672210,4sleep121-21rcuc/107:45:131
180152210,1sleep20-21swapper/212:10:172
179362210,1sleep30-21swapper/310:55:253
1397799216,3cyclictest87250irq/129-enp1s0-12:35:163
1397799214,5cyclictest1370-21idleruntime09:05:163
1397799214,3cyclictest2673-21diskmemload12:25:583
1397799212,7cyclictest28422-21munin-run08:55:013
1397799211,6cyclictest0-21swapper/309:50:033
1397299216,5cyclictest27464-21latency_hist11:20:012
1397299213,10cyclictest14436-21unixbench_multi12:00:282
1396699219,3cyclictest10852-21munin-run08:15:011
1396699214,7cyclictest21-21rcuc/109:45:221
1396699212,17cyclictest0-21swapper/109:08:381
1396699212,14cyclictest0-21swapper/109:14:511
1396699211,7cyclictest17687-21cat12:10:151
1396699211,3cyclictest23478-21irqstats09:55:181
13966992113,3cyclictest0-21swapper/111:16:101
13966992113,3cyclictest0-21swapper/110:30:191
13966992113,3cyclictest0-21swapper/107:38:161
1396699211,2cyclictest2673-21diskmemload11:51:241
1396699211,15cyclictest0-21swapper/111:01:371
1396699211,15cyclictest0-21swapper/110:59:431
1396699211,15cyclictest0-21swapper/109:43:431
1396699211,15cyclictest0-21swapper/109:17:551
1396699211,15cyclictest0-21swapper/107:23:461
1396699211,15cyclictest0-21swapper/107:18:301
1396699211,14cyclictest0-21swapper/110:54:511
1396699211,11cyclictest514-21grep11:30:191
1396699211,11cyclictest514-21grep11:30:191
1396699211,11cyclictest21695-21cat12:20:011
13966992110,3cyclictest22017-21memory11:05:181
1396699210,16cyclictest0-21swapper/108:34:361
1395699218,3cyclictest10706-21kworker/0:308:23:270
1395699212,8cyclictest26801-21kworker/0:209:03:030
1395699212,11cyclictest19477-21kworker/0:008:34:170
157652201,2sleep20-21swapper/209:39:042
1397799205,3cyclictest86950irq/126-enp1s0-08:05:193
1397799201,6cyclictest0-21swapper/308:40:273
1397799201,5cyclictest7903-21/usr/sbin/munin10:35:153
1397799201,5cyclictest0-21swapper/310:20:563
1397799201,12cyclictest0-21swapper/312:04:153
1397799200,10cyclictest0-21swapper/309:00:203
1397299201,7cyclictest0-21swapper/210:35:222
1397299201,6cyclictest0-21swapper/209:45:162
1397299201,12cyclictest0-21swapper/210:52:422
1397299201,12cyclictest0-21swapper/210:34:432
1397299201,11cyclictest0-21swapper/211:33:242
1397299201,11cyclictest0-21swapper/211:33:242
1397299201,11cyclictest0-21swapper/210:28:562
1397299201,11cyclictest0-21swapper/210:19:222
1397299201,11cyclictest0-21swapper/209:55:422
1396699202,2cyclictest23084-21kworker/u8:009:22:591
1396699202,16cyclictest0-21swapper/107:59:151
1396699202,16cyclictest0-21swapper/107:59:151
1396699202,10cyclictest4285-21expr11:40:151
13966992015,3cyclictest0-21swapper/108:05:261
1396699201,4cyclictest10369-21kworker/u8:112:21:531
1396699201,2cyclictest24425-21pmu-power11:10:221
13966992012,3cyclictest0-21swapper/112:07:381
13966992012,3cyclictest0-21swapper/109:38:481
1396699201,14cyclictest0-21swapper/112:33:561
1396699201,14cyclictest0-21swapper/111:21:371
1396699201,14cyclictest0-21swapper/110:25:521
1396699201,14cyclictest0-21swapper/110:22:121
1396699201,14cyclictest0-21swapper/110:03:041
1396699201,14cyclictest0-21swapper/108:51:401
1396699201,14cyclictest0-21swapper/108:48:171
1396699201,14cyclictest0-21swapper/108:37:371
1396699201,14cyclictest0-21swapper/108:28:401
1396699201,14cyclictest0-21swapper/108:01:131
1396699201,14cyclictest0-21swapper/107:52:271
1396699201,14cyclictest0-21swapper/107:43:441
1396699201,14cyclictest0-21swapper/107:30:371
1396699201,10cyclictest3175-21sendmail_mailst11:35:241
1395699202,9cyclictest8616-21kworker/0:008:11:330
1395699201,7cyclictest11140-21kworker/0:209:31:530
1395699201,11cyclictest19477-21kworker/0:008:35:290
1395699200,15cyclictest32041-21kworker/0:007:48:430
1395699200,11cyclictest32716-21kworker/0:109:05:170
288402190,3sleep31397799cyclictest08:55:143
248072191,3sleep324936-21smartctl08:45:163
17112191,3sleep20-21swapper/207:51:092
1397799192,8cyclictest0-21swapper/310:30:143
1397799191,6cyclictest19819-21cat12:15:153
1397799191,6cyclictest19488-21sshd08:33:143
1397799191,6cyclictest0-21swapper/311:48:223
1397799191,5cyclictest577-21systemd-journal08:25:223
1397799191,5cyclictest19085-21sshd12:12:243
1397799191,5cyclictest0-21swapper/311:08:253
1397799191,5cyclictest0-21swapper/309:10:113
1397799191,4cyclictest0-21swapper/312:33:453
13977991913,4cyclictest16981-21cut07:15:053
1397299195,7cyclictest751-21rngd12:35:322
1397299194,7cyclictest0-21swapper/212:24:182
1397299191,9cyclictest28989-21sshd10:09:242
1397299191,9cyclictest2615-21df_inode07:55:142
1397299191,9cyclictest2615-21df_inode07:55:142
1397299191,9cyclictest0-21swapper/209:30:512
1397299191,8cyclictest0-21swapper/212:16:472
1397299191,7cyclictest2673-21diskmemload10:42:392
1397299191,7cyclictest15481-21dnf12:05:142
1397299191,5cyclictest0-21swapper/211:43:132
1397299191,4cyclictest0-21swapper/209:15:182
1397299191,3cyclictest87250irq/129-enp1s0-10:21:312
1397299191,3cyclictest0-21swapper/211:08:142
13972991913,3cyclictest28207-21grep07:40:132
13972991912,2cyclictest30421-1kworker/u9:209:54:352
1397299191,15cyclictest28575-21wc11:20:182
1397299191,12cyclictest0-21swapper/209:43:092
1397299191,11cyclictest0-21swapper/212:34:192
1397299191,11cyclictest0-21swapper/210:45:112
1397299191,10cyclictest7121-21if_err_enp2s011:45:172
1396699191,6cyclictest0-21swapper/112:38:121
1396699191,14cyclictest0-21swapper/108:42:561
1396699191,10cyclictest12314-21cpu10:45:111
1395699192,16cyclictest21098-21kworker/0:207:42:250
1397799184,3cyclictest26048-21tune2fs11:15:143
1397799183,2cyclictest87250irq/129-enp1s0-10:07:283
1397799181,5cyclictest14498-21/usr/sbin/munin10:50:233
1397799181,5cyclictest0-21swapper/311:02:073
1397799181,15cyclictest0-21swapper/311:30:273
1397799181,15cyclictest0-21swapper/311:30:273
1397799181,10cyclictest0-21swapper/309:23:333
1397799180,9cyclictest0-21swapper/308:10:243
1397799180,5cyclictest0-21swapper/309:40:213
1397299182,3cyclictest0-21swapper/210:55:152
1397299181,9cyclictest801-21snmpd07:37:482
1397299181,8cyclictest0-21swapper/210:02:282
1397299181,6cyclictest0-21swapper/212:27:312
1397299181,5cyclictest22782-21basename08:40:192
1397299181,4cyclictest0-21swapper/211:00:042
1397299181,3cyclictest20515-21sensors_temp07:20:252
13972991813,3cyclictest2654-21crond09:10:012
1397299181,2cyclictest0-21swapper/211:55:022
13972991812,3cyclictest6679-21/usr/sbin/munin08:05:122
13972991812,3cyclictest19631-21idleruntime-cro08:35:012
13972991811,4cyclictest31199-21/usr/sbin/munin11:25:252
13972991811,4cyclictest15515-21cat07:10:252
1397299181,10cyclictest0-21swapper/211:56:562
1397299181,10cyclictest0-21swapper/209:24:462
1397799172,2cyclictest87250irq/129-enp1s0-11:55:423
1397799171,7cyclictest30871-21sed07:45:163
1397799171,6cyclictest0-21swapper/307:55:123
1397799171,6cyclictest0-21swapper/307:55:123
1397799171,15cyclictest0-21swapper/312:06:063
13977991711,3cyclictest26192-21diskstats07:35:133
1397799171,13cyclictest23651-21/usr/sbin/munin07:30:193
1397299173,2cyclictest0-21swapper/208:20:122
1397299172,3cyclictest87250irq/129-enp1s0-09:25:232
13972991712,3cyclictest21396-21grep08:35:262
13972991711,3cyclictest4468-21/usr/sbin/munin08:00:222
13972991711,3cyclictest17438-21dump-pmu-power08:30:012
1397799169,4cyclictest32595-21cut07:50:123
1397799161,4cyclictest0-21swapper/311:51:243
13977991611,3cyclictest14912-21if_enp1s007:10:173
13977991611,3cyclictest12305-21acpi10:45:113
13977991610,3cyclictest28044-21/usr/sbin/munin07:40:123
13977991610,3cyclictest22250-21interrupts07:25:183
13977991610,3cyclictest13034-21crond08:20:013
1397799160,14cyclictest0-21swapper/308:37:433
1397299162,2cyclictest87250irq/129-enp1s0-08:55:242
1397299161,5cyclictest22513-21ntp_states07:25:212
13972991612,3cyclictest0-21swapper/207:18:202
1397299161,14cyclictest0-21swapper/208:46:022
1397299161,14cyclictest0-21swapper/208:10:062
1397299161,14cyclictest0-21swapper/207:32:062
13972991611,3cyclictest31018-21if_err_enp1s007:45:172
13972991610,3cyclictest11406-21df08:15:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional