You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 04:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Tue Mar 24, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
191642237164,23sleep10-21swapper/119:08:311
189412202165,25sleep20-21swapper/219:05:392
173162202165,25sleep30-21swapper/319:05:083
189642200164,24sleep00-21swapper/019:05:570
495821570,4sleep01936199cyclictest21:00:220
1937899106102,3cyclictest0-21swapper/321:40:423
19378999693,2cyclictest0-21swapper/321:39:173
19378998581,3cyclictest0-21swapper/319:56:443
19378998581,3cyclictest0-21swapper/319:56:443
252582600,1sleep20-21swapper/223:00:202
19378995143,3cyclictest0-21swapper/320:09:373
83512470,2sleep00-21swapper/021:10:130
83512470,2sleep00-21swapper/021:10:130
1936999461,36cyclictest6032-21unixbench_multi22:15:271
1936999447,23cyclictest13693-21perf23:50:011
1936999421,29cyclictest12455-21latency_hist21:20:011
1936999401,35cyclictest5230-21tr23:30:121
1936999401,35cyclictest5230-21tr23:30:121
1937899392,23cyclictest6707-21/usr/sbin/munin22:20:253
1937899392,23cyclictest6707-21/usr/sbin/munin22:20:253
1937899391,9cyclictest833-21runrttasks22:58:533
1936999391,26cyclictest10106-21grep23:40:161
1937899388,3cyclictest8064-21diskmemload23:48:163
1937899381,8cyclictest8584-21smart_sda23:35:233
1937899381,24cyclictest10923-21wc23:40:253
1936999382,33cyclictest29102-21munin-run00:25:011
1936999381,27cyclictest26904-21latency_hist00:20:011
1936999380,29cyclictest0-21swapper/119:45:011
1936199380,26cyclictest0-21swapper/022:31:060
1937899371,10cyclictest0-21swapper/322:42:343
1936999371,33cyclictest21533-21crond19:15:011
1936999371,33cyclictest18437-21tr00:00:121
1936999371,32cyclictest336-21sshd23:19:261
1936999371,27cyclictest3272-21diskstats23:25:131
1936999371,27cyclictest10341-21cat21:15:011
1936999371,27cyclictest10341-21cat21:15:011
1936999371,23cyclictest0-21swapper/121:50:211
1937899361,9cyclictest11645-21proc_pri21:15:243
1937899361,5cyclictest0-21swapper/322:05:433
1937899361,10cyclictest0-21swapper/323:16:063
1936999366,19cyclictest0-21swapper/122:00:001
1936999363,21cyclictest32369-21latency_hist22:05:011
1936999361,29cyclictest9635-21latency_hist20:00:011
1936199360,26cyclictest0-21swapper/022:50:500
1937899358,3cyclictest812-21snmpd21:31:203
1937899355,3cyclictest12050irq/123-ahci[0019:40:023
1937899351,9cyclictest812-21snmpd23:01:353
1937899351,4cyclictest0-21swapper/321:46:013
1937899351,3cyclictest86550irq/126-enp1s0-21:55:243
1937899351,11cyclictest16027-21sshd20:12:253
1937899351,10cyclictest0-21swapper/323:13:403
1937899351,10cyclictest0-21swapper/322:50:003
1936999351,31cyclictest24424-21latency_hist19:20:021
1936999351,31cyclictest0-21swapper/123:10:131
1936199357,13cyclictest26288-21awk23:05:010
1937899344,4cyclictest8365-21wc21:10:143
1937899344,4cyclictest8365-21wc21:10:133
1937899342,20cyclictest14880-21kworker/3:000:26:463
1937899341,5cyclictest0-21swapper/322:02:273
1937899341,4cyclictest812-21snmpd20:41:093
1937899341,10cyclictest0-21swapper/321:28:303
1937899341,10cyclictest0-21swapper/321:22:513
1937399341,22cyclictest899-21irqstats22:05:162
1936999342,28cyclictest1158-21ntp_states22:05:211
1936999341,24cyclictest3889-21/usr/sbin/munin21:00:201
1936999341,23cyclictest786-21NetworkManager21:24:091
1936999341,23cyclictest22093-21/usr/sbin/munin22:55:091
1936999341,23cyclictest13915-21/usr/sbin/munin23:50:251
1936999341,21cyclictest0-21swapper/122:10:221
1936199341,30cyclictest0-21swapper/023:16:460
1937899333,5cyclictest23717-21smart_sda19:15:243
1937899331,9cyclictest86550irq/126-enp1s0-23:25:413
19378993312,18cyclictest21447-21memory00:05:203
19378993311,18cyclictest7160-21ntpq21:05:213
1937899331,10cyclictest26707-21kworker/u8:200:20:223
1937899331,10cyclictest0-21swapper/300:39:103
19378993310,20cyclictest14301-21cut22:35:193
1937899330,9cyclictest0-21swapper/322:10:203
1937399331,24cyclictest0-21swapper/220:31:042
1937399331,23cyclictest0-21swapper/220:53:272
1936999334,21cyclictest86750irq/128-enp1s0-20:11:041
1936999333,16cyclictest833-21runrttasks00:35:211
1936999332,19cyclictest1753-21runrttasks20:55:131
1936999331,5cyclictest19900-21/usr/sbin/munin22:50:251
1936999331,3cyclictest31309-21perf00:30:011
1936999331,3cyclictest19863-21tune2fs21:35:141
1936999330,4cyclictest0-21swapper/122:20:141
1936999330,4cyclictest0-21swapper/122:20:141
1936999330,2cyclictest0-21swapper/122:25:251
1937899325,4cyclictest0-21swapper/322:50:383
1937899321,8cyclictest0-21swapper/323:20:143
1937899321,5cyclictest7091-21latency_hist23:35:023
1937899321,5cyclictest7091-21latency_hist23:35:023
19378993213,16cyclictest24659-21users20:30:273
19378993211,18cyclictest17663-21grep23:55:263
1937899321,10cyclictest0-21swapper/322:32:023
1937899321,10cyclictest0-21swapper/300:30:073
1937899321,10cyclictest0-21swapper/300:30:073
1937399321,16cyclictest0-21swapper/200:29:452
1937399320,26cyclictest0-21swapper/200:24:012
1936999325,13cyclictest0-21swapper/100:10:291
1936999323,19cyclictest14408-21kworker/u8:221:46:531
1936999321,28cyclictest5242-21latency_hist19:50:011
1936999321,28cyclictest32460-21processes19:35:231
1936999321,28cyclictest11206-21grep22:30:111
1936999320,2cyclictest0-21swapper/119:31:531
1936999320,24cyclictest0-21swapper/100:30:261
1936999320,24cyclictest0-21swapper/100:30:251
1936199322,26cyclictest24046-21users19:15:270
1936199321,23cyclictest0-21swapper/023:25:280
1937899312,3cyclictest26574-21/usr/sbin/munin21:50:163
1937899311,9cyclictest812-21snmpd19:51:003
1937899311,6cyclictest0-21swapper/323:09:573
19378993111,17cyclictest5543-21basename21:01:013
1937899311,10cyclictest0-21swapper/300:01:383
1937399311,23cyclictest0-21swapper/223:26:442
1937399311,22cyclictest0-21swapper/220:41:202
1937399311,14cyclictest0-21swapper/222:00:402
1937399311,13cyclictest0-21swapper/200:34:532
1937399311,13cyclictest0-21swapper/200:34:532
1937399311,12cyclictest0-21swapper/221:52:202
1937399310,12cyclictest0-21swapper/221:25:252
1936999311,3cyclictest86750irq/128-enp1s0-23:05:171
1936999311,3cyclictest22839-21latency_hist20:30:021
1936999311,3cyclictest18088-21dump-pmu-power00:00:011
1936999311,3cyclictest0-21swapper/100:05:271
1936999311,27cyclictest2766-21cat23:25:011
1936999311,27cyclictest0-21swapper/121:10:011
1936999311,27cyclictest0-21swapper/120:45:191
19369993111,14cyclictest0-21swapper/122:40:131
1936199315,12cyclictest31317-21idleruntime-cro00:30:000
1936199311,13cyclictest0-21swapper/021:06:420
1937899309,3cyclictest19303-21gltestperf20:20:173
1937899307,3cyclictest31824-21cut20:50:013
1937899307,3cyclictest25045-21cpuspeed_turbos00:15:133
1937899302,6cyclictest24654-21/usr/sbin/munin19:20:273
1937899301,6cyclictest15506-21users23:50:263
1937899301,5cyclictest0-21swapper/322:15:213
19378993012,15cyclictest822-21sed20:50:273
19378993011,16cyclictest10935-21ntp_states20:00:233
19378993010,17cyclictest2242-21proc_pri19:40:223
1937399301,17cyclictest0-21swapper/222:48:412
1937399301,14cyclictest0-21swapper/222:35:412
1937399301,12cyclictest0-21swapper/222:17:162
1936999302,20cyclictest9345-21sshd19:56:261
1936999302,13cyclictest86850irq/129-enp1s0-20:19:101
1936999301,3cyclictest86750irq/128-enp1s0-21:30:151
1936999301,3cyclictest24848-21perl23:00:151
1936999301,2cyclictest0-21swapper/120:05:241
1936999301,26cyclictest7489-21cstates23:35:111
1936999301,25cyclictest328-21sed20:50:221
1936999301,21cyclictest23986-21memory20:30:211
1936999300,19cyclictest0-21swapper/122:36:311
1936199301,2cyclictest0-21swapper/023:42:270
1936199301,13cyclictest0-21swapper/021:44:260
1936199301,13cyclictest0-21swapper/020:43:440
1936199301,13cyclictest0-21swapper/020:27:580
1936199301,12cyclictest0-21swapper/020:55:580
1936199301,11cyclictest0-21swapper/019:29:360
1936199300,23cyclictest0-21swapper/023:23:380
1936199300,22cyclictest0-21swapper/020:02:530
1937899298,3cyclictest3793-21fschecks_time19:45:173
1937899296,3cyclictest1691-21/usr/sbin/munin20:55:263
1937899296,3cyclictest0-21swapper/300:12:213
1937399291,25cyclictest0-21swapper/221:08:352
1937399291,22cyclictest0-21swapper/221:32:412
1937399291,19cyclictest0-21swapper/223:39:362
19373992911,4cyclictest26181-21memory20:35:202
1937399291,14cyclictest0-21swapper/223:44:562
1937399291,14cyclictest0-21swapper/200:00:002
1937399291,13cyclictest0-21swapper/222:32:012
1937399291,11cyclictest24115-21timerandwakeup00:10:262
1937399291,11cyclictest0-21swapper/222:44:132
1936999297,19cyclictest5667-21expr19:50:111
1936999297,19cyclictest26218-21unixbench_singl19:20:261
1936999292,20cyclictest15003-21kworker/u8:022:46:081
1936999292,14cyclictest11842-21sh20:05:001
1936999291,3cyclictest28822-21sed20:40:231
1936999291,24cyclictest26367-21/usr/sbin/munin20:35:241
19369992911,15cyclictest14884-21/usr/sbin/munin21:25:241
1936199291,9cyclictest0-21swapper/020:38:380
1936199291,13cyclictest0-21swapper/023:34:270
1936199291,13cyclictest0-21swapper/023:34:260
1936199291,13cyclictest0-21swapper/020:20:310
1936199291,10cyclictest0-21swapper/020:09:230
1936199291,10cyclictest0-21swapper/019:51:070
1936199290,2cyclictest0-21swapper/022:42:220
1936199290,23cyclictest0-21swapper/000:17:180
1936199290,22cyclictest0-21swapper/023:47:510
1936199290,22cyclictest0-21swapper/022:35:520
1936199290,17cyclictest0-21swapper/000:07:470
1936199290,14cyclictest0-21swapper/000:34:280
1936199290,14cyclictest0-21swapper/000:34:280
1936199290,13cyclictest0-21swapper/021:49:050
1937899288,3cyclictest20868-21wc19:10:213
1937899288,3cyclictest10154-21sendmail_mailtr22:25:233
19373992815,3cyclictest0-21swapper/223:33:122
19373992815,3cyclictest0-21swapper/223:33:122
1937399281,17cyclictest0-21swapper/220:07:282
1937399281,15cyclictest0-21swapper/222:28:132
1937399281,15cyclictest0-21swapper/221:36:482
1937399281,12cyclictest0-21swapper/200:01:482
1937399281,11cyclictest7847-21cat19:55:122
1937399281,11cyclictest7847-21cat19:55:122
1937399281,11cyclictest0-21swapper/223:51:442
1937399281,10cyclictest0-21swapper/221:00:092
1937399281,10cyclictest0-21swapper/220:04:042
1937399280,21cyclictest0-21swapper/221:15:192
1937399280,11cyclictest0-21swapper/222:57:452
1936999281,2cyclictest0-21swapper/120:20:171
19369992812,13cyclictest21486-21/usr/sbin/munin21:40:211
1936199285,11cyclictest20308-21munin-run00:05:020
1936199281,13cyclictest0-21swapper/021:26:380
1936199281,11cyclictest0-21swapper/019:20:020
1936199280,2cyclictest0-21swapper/021:22:540
1936199280,13cyclictest0-21swapper/022:06:570
1937899278,3cyclictest16649-21cstates20:15:133
1937899271,8cyclictest25904-21gltestperf20:35:173
1937899271,11cyclictest29049-21/usr/sbin/munin19:30:233
1937399271,18cyclictest0-21swapper/220:22:042
1937399271,12cyclictest0-21swapper/200:39:012
1937399271,11cyclictest0-21swapper/223:48:242
1937399271,11cyclictest0-21swapper/222:22:362
1937399271,11cyclictest0-21swapper/222:22:362
1936999271,22cyclictest0-21swapper/119:25:251
1936199271,9cyclictest0-21swapper/022:59:460
1936199271,19cyclictest0-21swapper/023:09:100
1936199271,13cyclictest0-21swapper/021:39:140
1936199271,12cyclictest0-21swapper/023:35:390
1936199271,12cyclictest0-21swapper/020:45:120
1936199271,11cyclictest0-21swapper/019:48:490
1936199271,10cyclictest0-21swapper/022:23:300
1936199271,10cyclictest0-21swapper/022:23:300
1936199271,10cyclictest0-21swapper/019:30:180
1936199270,13cyclictest0-21swapper/023:58:270
1936199270,12cyclictest0-21swapper/022:45:020
1936199270,11cyclictest0-21swapper/021:53:430
1936199270,11cyclictest0-21swapper/000:14:230
1937899266,3cyclictest21233-21fschecks_count20:25:153
1937399266,11cyclictest0-21swapper/221:22:092
19373992615,3cyclictest0-21swapper/221:43:442
19373992613,8cyclictest0-21swapper/219:43:122
1937399261,12cyclictest0-21swapper/223:05:532
1937399261,11cyclictest0-21swapper/219:35:442
1936199261,11cyclictest0-21swapper/019:41:480
1936199261,10cyclictest0-21swapper/000:22:160
1936199260,13cyclictest0-21swapper/000:35:040
1936199260,12cyclictest0-21swapper/020:10:270
1936199260,11cyclictest0-21swapper/020:16:030
208362250,21sleep219562-21/usr/sbin/munin19:10:212
1937399259,3cyclictest0-21swapper/219:47:122
1937399258,10cyclictest0-21swapper/223:18:242
1937399251,9cyclictest9209-21memory21:10:212
1937399251,9cyclictest9209-21memory21:10:212
1937399251,8cyclictest0-21swapper/223:23:082
19373992514,3cyclictest0-21swapper/219:24:532
1937399251,11cyclictest0-21swapper/222:53:332
1937399251,11cyclictest0-21swapper/200:16:002
1937399251,10cyclictest0-21swapper/219:51:442
1937399251,10cyclictest0-21swapper/219:25:172
1936199251,9cyclictest0-21swapper/020:32:360
19361992511,7cyclictest0-21swapper/023:11:100
1936199251,14cyclictest0-21swapper/021:31:080
1936199251,11cyclictest0-21swapper/020:53:140
1936199251,10cyclictest0-21swapper/022:29:580
1936199251,10cyclictest0-21swapper/022:11:250
1936199250,9cyclictest0-21swapper/021:16:580
1936199250,16cyclictest0-21swapper/023:51:030
1936199250,10cyclictest0-21swapper/021:58:490
1937399249,3cyclictest0-21swapper/221:48:002
1937399241,9cyclictest0-21swapper/222:12:082
1937399241,8cyclictest0-21swapper/220:26:562
1937399241,11cyclictest0-21swapper/220:14:082
1937399241,11cyclictest0-21swapper/219:33:052
1937399241,10cyclictest0-21swapper/223:12:412
1936199248,7cyclictest0-21swapper/019:12:090
1936199241,9cyclictest0-21swapper/022:15:400
1936199241,11cyclictest0-21swapper/019:59:030
1936199241,11cyclictest0-21swapper/019:59:030
1936199241,10cyclictest0-21swapper/022:01:470
1937899239,3cyclictest14880-21kworker/3:019:26:353
1937399238,4cyclictest0-21swapper/219:19:042
1937399236,8cyclictest0-21swapper/220:46:072
1937399231,10cyclictest0-21swapper/200:05:442
1937399227,3cyclictest0-21swapper/220:16:402
1937399226,3cyclictest0-21swapper/221:58:322
1936199221,10cyclictest0-21swapper/019:39:480
1937399216,3cyclictest0-21swapper/220:55:292
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional