You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 00:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot5s.osadl.org (updated Wed Sep 17, 2025 12:44:24)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11032992619,6cyclictest8966-21latency_hist10:25:002
11030992522,2cyclictest15603-21turbostat08:25:000
1103399240,23cyclictest23392-21apt-get07:36:253
11033992321,1cyclictest6527-21sh10:20:013
11033992321,1cyclictest313-21systemd-journal08:40:003
1103299230,2cyclictest18584-21dump-pmu-power10:45:002
1103299230,0cyclictest0-21swapper/210:51:282
1103199230,2cyclictest28586-21idleruntime-cro11:05:001
1103199230,22cyclictest31286-21cron10:04:591
11030992322,1cyclictest3749-21apt-get10:15:020
1103099230,22cyclictest313-21systemd-journal11:49:590
1103099230,22cyclictest21702-21ls09:45:010
1103099230,22cyclictest18020-21awk08:30:000
1103099230,22cyclictest10352-21sh12:40:000
1103099230,0cyclictest337-21apt-get09:01:270
11033992220,1cyclictest32431-21idleruntime-cro09:00:003
11033992220,1cyclictest20424-21munin-run08:34:593
1103399220,21cyclictest313-21systemd-journal12:16:293
1103399220,21cyclictest0-21swapper/312:05:023
1103399220,21cyclictest0-21swapper/311:21:283
1103399220,21cyclictest0-21swapper/310:51:283
1103399220,21cyclictest0-21swapper/308:51:273
1103399220,1cyclictest5579-21idleruntime-cro08:05:013
11032992222,0cyclictest0-21swapper/210:56:292
11032992222,0cyclictest0-21swapper/207:31:272
11032992220,1cyclictest26871-21apt-get09:56:282
11032992220,1cyclictest0-21swapper/208:10:012
11032992220,1cyclictest0-21swapper/207:39:592
11032992219,2cyclictest15622-21perf08:24:592
1103299220,21cyclictest1089-21apt-get07:56:282
1103299220,21cyclictest0-21swapper/212:39:592
1103299220,21cyclictest0-21swapper/209:01:282
1103299220,0cyclictest0-21swapper/211:31:292
11031992221,1cyclictest25591-21apt-get08:46:281
1103199220,21cyclictest6212-21apt-get11:26:281
1103199220,21cyclictest583-21avahi-daemon10:35:011
1103199220,21cyclictest0-21swapper/109:01:271
1103199220,1cyclictest6549-21awk10:20:001
11030992221,1cyclictest10783-21wc08:14:590
11030992220,1cyclictest15840-21munin-run07:20:000
11030992220,1cyclictest0-21swapper/008:40:010
1103099220,21cyclictest0-21swapper/012:20:000
1103099220,21cyclictest0-21swapper/009:36:270
1103099220,21cyclictest0-21swapper/007:46:260
1103099220,1cyclictest25550-21sendmail-msp07:40:000
1103099220,1cyclictest20880-21apt-get11:55:010
11033992121,0cyclictest0-21swapper/310:01:253
11033992120,1cyclictest34-21ksoftirqd/311:15:343
11033992120,1cyclictest34-21ksoftirqd/308:16:213
11033992119,1cyclictest34-21ksoftirqd/311:44:593
11033992119,1cyclictest1293-21apt10:10:003
11033992119,1cyclictest0-21swapper/307:41:263
1103399210,20cyclictest0-21swapper/312:36:253
1103399210,20cyclictest0-21swapper/311:06:283
1103399210,20cyclictest0-21swapper/307:26:263
1103399210,0cyclictest0-21swapper/310:36:263
1103399210,0cyclictest0-21swapper/309:36:263
1103399210,0cyclictest0-21swapper/308:46:273
11032992121,0cyclictest0-21swapper/211:41:282
11032992121,0cyclictest0-21swapper/209:21:252
11032992120,1cyclictest1072-21apt-get12:21:262
11032992119,1cyclictest17261-21apt-get09:36:262
11032992119,1cyclictest13777-21turbostat10:35:012
1103299210,20cyclictest23447-21apt-get12:01:282
1103299210,20cyclictest0-21swapper/212:26:262
1103299210,0cyclictest27967-21munin-run07:44:592
1103299210,0cyclictest26231-21apt-get12:06:272
11031992121,0cyclictest0-21swapper/111:36:281
11031992121,0cyclictest0-21swapper/108:51:271
11031992120,1cyclictest13436-21apt-get11:41:281
11031992120,0cyclictest0-21swapper/112:01:281
11031992119,1cyclictest3201-21perf08:00:001
1103199210,20cyclictest24472-21apt-get09:51:271
1103199210,20cyclictest20987-21latency_hist10:50:001
1103199210,20cyclictest0-21swapper/112:06:271
1103199210,20cyclictest0-21swapper/111:46:271
1103199210,20cyclictest0-21swapper/110:26:281
1103199210,20cyclictest0-21swapper/107:11:271
1103199210,1cyclictest3818-21apt-get11:21:291
1103199210,0cyclictest0-21swapper/109:26:271
1103199210,0cyclictest0-21swapper/107:36:261
11030992121,0cyclictest16203-21apt-get07:21:290
11030992120,1cyclictest20990-21perf10:49:590
11030992120,0cyclictest3571-21apt-get08:01:250
11030992120,0cyclictest0-21swapper/009:26:280
11030992119,2cyclictest0-21swapper/008:56:240
11030992119,1cyclictest18236-21apt-get11:51:290
1103099210,21cyclictest0-21swapper/009:31:300
1103099210,21cyclictest0-21swapper/009:31:300
1103099210,20cyclictest0-21swapper/010:41:260
1103099210,20cyclictest0-21swapper/010:06:280
1103099210,0cyclictest0-21swapper/010:56:280
1103099210,0cyclictest0-21swapper/009:06:290
1103099210,0cyclictest0-21swapper/007:31:280
11033992020,0cyclictest0-21swapper/311:51:303
11033992020,0cyclictest0-21swapper/310:56:293
11033992020,0cyclictest0-21swapper/310:11:273
11033992020,0cyclictest0-21swapper/309:41:253
11033992019,1cyclictest31354-21apt-get11:11:263
11033992018,1cyclictest7283-21grep09:15:003
11033992017,2cyclictest18005-21perf08:29:593
1103399200,19cyclictest0-21swapper/312:11:273
1103399200,19cyclictest0-21swapper/310:21:263
1103399200,19cyclictest0-21swapper/309:31:293
1103399200,19cyclictest0-21swapper/309:31:293
1103399200,0cyclictest0-21swapper/312:21:263
1103399200,0cyclictest0-21swapper/311:26:263
1103399200,0cyclictest0-21swapper/308:11:273
1103399200,0cyclictest0-21swapper/307:56:283
11032992020,0cyclictest0-21swapper/208:36:272
11032992020,0cyclictest0-21swapper/207:21:282
11032992019,0cyclictest28959-21apt-get11:06:272
11032992019,0cyclictest0-21swapper/208:31:262
11032992018,1cyclictest29287-21apt-get10:01:252
11032992017,2cyclictest11880-21fschecks_time07:11:302
11032992016,4cyclictest0-21swapper/211:15:002
1103299200,20cyclictest0-21swapper/209:46:302
11031992020,0cyclictest5241-21apt-get09:11:251
11031992020,0cyclictest2792-21apt-get09:06:291
11031992020,0cyclictest25917-21apt-get07:41:271
11031992020,0cyclictest0-21swapper/112:31:291
11031992020,0cyclictest0-21swapper/111:51:291
11031992020,0cyclictest0-21swapper/108:41:281
1103199200,20cyclictest0-21swapper/107:51:281
1103199200,0cyclictest3534-21apt-get12:26:281
1103199200,0cyclictest0-21swapper/109:36:271
11030992020,0cyclictest32744-21apt-get07:55:010
11030992020,0cyclictest0-21swapper/012:26:270
11030992020,0cyclictest0-21swapper/011:26:280
11030992020,0cyclictest0-21swapper/008:41:280
11030992019,1cyclictest0-21swapper/012:01:290
1103099200,0cyclictest0-21swapper/012:06:280
1103099200,0cyclictest0-21swapper/009:21:270
1103099200,0cyclictest0-21swapper/007:11:270
11033991919,0cyclictest0-21swapper/312:01:293
11033991919,0cyclictest0-21swapper/310:31:273
11033991919,0cyclictest0-21swapper/309:46:293
11033991919,0cyclictest0-21swapper/309:16:273
11033991919,0cyclictest0-21swapper/307:51:273
11033991918,1cyclictest34-21ksoftirqd/311:50:013
11033991917,1cyclictest7957-21turbostat.cron12:34:593
11033991917,1cyclictest28605-21munin-run11:05:013
11033991917,1cyclictest20997-21munin-plugin-st10:50:003
11033991917,1cyclictest13412-21idleruntime-cro07:15:013
11032991919,0cyclictest12464-21apt-get09:26:272
11032991919,0cyclictest0-21swapper/211:26:262
11032991919,0cyclictest0-21swapper/210:16:262
11032991919,0cyclictest0-21swapper/209:16:272
11032991919,0cyclictest0-21swapper/209:06:302
11032991919,0cyclictest0-21swapper/208:56:242
11032991917,1cyclictest5834-21apt11:25:002
11032991917,1cyclictest3769-21munin-run10:15:002
11032991917,1cyclictest0-21swapper/212:11:262
1103299190,18cyclictest31688-21apt-get10:06:292
1103299190,11cyclictest0-21swapper/210:26:272
1103299190,0cyclictest0-21swapper/209:51:262
11031991919,0cyclictest8616-21apt-get11:31:291
11031991919,0cyclictest7990-21apt-get08:10:011
11031991919,0cyclictest5968-21apt-get08:06:251
11031991919,0cyclictest28320-21apt-get07:46:261
11031991919,0cyclictest18599-21apt-get07:26:251
11031991919,0cyclictest10058-21apt-get09:21:251
11031991919,0cyclictest0-21swapper/110:51:271
11031991919,0cyclictest0-21swapper/108:01:241
11031991919,0cyclictest0-21swapper/107:31:271
11031991918,1cyclictest31354-21apt-get11:11:261
11031991918,1cyclictest16900-21perf09:35:011
11031991918,1cyclictest16900-21perf09:35:001
1103199191,17cyclictest20265-21apt-get11:55:011
11030991919,0cyclictest0-21swapper/012:31:290
11030991919,0cyclictest0-21swapper/011:06:280
11030991919,0cyclictest0-21swapper/009:46:290
11033991818,0cyclictest0-21swapper/310:26:273
11033991818,0cyclictest0-21swapper/309:01:283
1103399180,17cyclictest0-21swapper/311:56:283
11032991818,0cyclictest26564-21apt-get11:01:282
11032991818,0cyclictest0-21swapper/212:16:292
11032991818,0cyclictest0-21swapper/211:56:282
11032991818,0cyclictest0-21swapper/209:31:292
11032991818,0cyclictest0-21swapper/209:31:292
11032991818,0cyclictest0-21swapper/208:46:272
11032991818,0cyclictest0-21swapper/208:01:252
11032991816,1cyclictest20959-21cron10:50:002
1103299180,17cyclictest0-21swapper/209:11:272
11031991818,0cyclictest31075-21apt-get12:16:301
11031991818,0cyclictest0-21swapper/112:11:261
11031991818,0cyclictest0-21swapper/110:36:271
11031991818,0cyclictest0-21swapper/109:46:301
1103199180,18cyclictest0-21swapper/108:36:281
11030991818,0cyclictest0-21swapper/008:16:280
11030991818,0cyclictest0-21swapper/008:06:260
11030991816,1cyclictest31303-21cron10:04:590
11030991815,2cyclictest20401-21turbostat08:35:000
1103099180,1cyclictest13758-21dump-pmu-power10:35:010
11033991717,0cyclictest20997-21apt-get07:31:273
11033991717,0cyclictest0-21swapper/310:41:273
11033991717,0cyclictest0-21swapper/309:56:283
11033991717,0cyclictest0-21swapper/309:51:263
11033991717,0cyclictest0-21swapper/308:41:273
11033991716,1cyclictest34-21ksoftirqd/309:27:473
11032991717,0cyclictest0-21swapper/211:16:262
11032991715,1cyclictest20270-21perf11:55:012
11032991715,1cyclictest13205-21ls08:19:592
11031991716,1cyclictest1308-21perf10:09:591
1103199170,0cyclictest0-21swapper/110:56:281
11030991717,0cyclictest11012-21apt-get11:36:270
11030991717,0cyclictest0-21swapper/009:51:260
11030991717,0cyclictest0-21swapper/009:11:250
11030991717,0cyclictest0-21swapper/008:46:270
11030991716,1cyclictest27962-21latency_hist07:45:000
11030991716,0cyclictest9334-21apt-get10:26:280
11030991716,0cyclictest0-21swapper/008:51:270
11030991714,2cyclictest28048-21awk09:56:340
11033991614,1cyclictest34-21ksoftirqd/308:21:283
11032991616,0cyclictest0-21swapper/208:26:262
11032991616,0cyclictest0-21swapper/207:46:262
11032991614,1cyclictest31348-21cut07:51:312
11032991614,1cyclictest0-21swapper/211:50:002
1103299160,6cyclictest0-21swapper/207:26:262
1103299160,14cyclictest0-21swapper/211:39:252
1103299160,14cyclictest0-21swapper/209:42:232
11031991616,0cyclictest1668-21apt-get10:11:261
1103199160,15cyclictest31181-21irqstats08:56:281
11030991616,0cyclictest0-21swapper/012:21:270
11030991616,0cyclictest0-21swapper/011:11:270
1103099160,15cyclictest0-21swapper/012:11:300
1103399153,12cyclictest0-21swapper/312:26:273
11033991515,0cyclictest0-21swapper/307:46:253
11033991512,2cyclictest16391-21tr07:21:293
11033991511,2cyclictest34-21ksoftirqd/307:16:123
1103399150,1cyclictest0-21swapper/309:21:293
1103399150,14cyclictest0-21swapper/308:06:323
11032991515,0cyclictest5936-21apt-get12:31:272
11032991513,1cyclictest8010-21munin-run08:10:012
11032991513,1cyclictest30031-21munin-run08:55:012
1103299150,13cyclictest0-21swapper/210:35:222
11031991515,0cyclictest19680-21apt-get09:41:261
11031991515,0cyclictest0-21swapper/109:16:261
11031991515,0cyclictest0-21swapper/108:21:271
11031991514,1cyclictest15981-21apt-get08:26:271
1103199150,1cyclictest18243-21chrt08:30:061
1103199150,1cyclictest1306-21acpi11:16:261
1103199150,1cyclictest0-21swapper/107:16:311
1103199150,0cyclictest0-21swapper/112:21:271
11030991515,0cyclictest0-21swapper/007:51:270
11030991514,1cyclictest0-21swapper/010:25:010
1103099150,1cyclictest2721-21ntp_states10:11:330
11033991414,0cyclictest0-21swapper/309:06:303
1103399140,14cyclictest0-21swapper/311:36:273
1103399140,14cyclictest0-21swapper/311:31:213
11032991412,1cyclictest0-21swapper/208:41:352
1103299140,14cyclictest0-21swapper/207:15:592
11031991414,0cyclictest0-21swapper/111:07:071
11031991414,0cyclictest0-21swapper/109:55:081
1103199140,14cyclictest0-21swapper/112:35:281
1103199140,14cyclictest0-21swapper/108:15:101
1103199140,0cyclictest0-21swapper/107:20:191
11030991414,0cyclictest0-21swapper/011:18:010
11030991414,0cyclictest0-21swapper/011:01:290
11030991414,0cyclictest0-21swapper/010:51:170
11030991414,0cyclictest0-21swapper/010:35:590
1103099140,14cyclictest0-21swapper/011:41:030
1103099140,0cyclictest0-21swapper/011:30:090
1103099140,0cyclictest0-21swapper/009:15:020
1103099140,0cyclictest0-21swapper/007:25:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional