You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 20:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Nov 25, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25402240167,24sleep30-21swapper/307:07:473
25382239163,24sleep10-21swapper/107:07:461
6652204167,24sleep00-21swapper/007:05:060
26382203165,25sleep20-21swapper/207:09:042
3210221620,4sleep2281899cyclictest08:15:012
850721040,4sleep0280599cyclictest12:15:140
258292790,4sleep121-21rcuc/108:00:151
192982640,3sleep319300-21pmu-power10:10:243
221882610,5sleep0280599cyclictest09:05:140
280202520,2sleep10-21swapper/108:05:131
280599511,16cyclictest1601-21munin-run12:00:020
2818995047,2cyclictest0-21swapper/207:34:302
2818995047,2cyclictest0-21swapper/207:34:302
157882470,2sleep215790-21timerandwakeup11:15:262
282399441,30cyclictest0-21swapper/310:32:233
2823994328,8cyclictest0-21swapper/311:38:353
2823994224,16cyclictest0-21swapper/307:49:193
282399421,31cyclictest0-21swapper/309:11:233
2823994126,8cyclictest0-21swapper/311:34:173
2823994023,3cyclictest0-21swapper/307:27:473
2823994017,11cyclictest0-21swapper/311:21:373
2823994016,11cyclictest0-21swapper/311:28:013
2823993827,3cyclictest0-21swapper/308:26:333
2823993813,8cyclictest0-21swapper/308:59:393
282399381,17cyclictest8218-21ntp_states09:45:223
2823993725,7cyclictest0-21swapper/308:54:033
2818993633,2cyclictest0-21swapper/210:22:092
2823993516,6cyclictest0-21swapper/311:01:153
2823993416,7cyclictest0-21swapper/308:11:273
282399331,30cyclictest0-21swapper/312:34:093
282399331,30cyclictest0-21swapper/311:58:273
282399331,30cyclictest0-21swapper/310:39:373
282399331,17cyclictest4338-21missed_timers10:50:193
280599337,13cyclictest12606-21munin-run12:25:020
280599335,13cyclictest5377-21idleruntime-cro10:55:020
142712330,1sleep20-21swapper/211:15:012
2823993217,7cyclictest0-21swapper/307:19:203
2823993216,3cyclictest0-21swapper/308:33:433
282399321,14cyclictest0-21swapper/309:32:233
282399321,12cyclictest0-21swapper/311:50:433
282399319,9cyclictest0-21swapper/307:38:493
2823993117,9cyclictest0-21swapper/307:32:253
2823993117,9cyclictest0-21swapper/307:32:253
282399311,23cyclictest0-21swapper/312:06:353
282399311,14cyclictest0-21swapper/309:40:103
282399311,13cyclictest0-21swapper/311:05:173
282399311,13cyclictest0-21swapper/310:58:273
282399311,13cyclictest0-21swapper/310:09:193
282399311,12cyclictest0-21swapper/309:38:443
282399311,12cyclictest0-21swapper/309:38:433
282399308,9cyclictest0-21swapper/307:53:353
2823993013,10cyclictest0-21swapper/310:25:233
282399301,27cyclictest0-21swapper/308:04:313
282399301,20cyclictest0-21swapper/310:41:513
282399301,20cyclictest0-21swapper/310:41:513
282399301,19cyclictest0-21swapper/309:51:463
282399301,12cyclictest0-21swapper/310:20:313
282399301,11cyclictest0-21swapper/308:05:313
282399300,20cyclictest0-21swapper/312:04:273
282399300,13cyclictest0-21swapper/309:29:373
282399291,12cyclictest0-21swapper/312:10:553
281399292,18cyclictest25672-21memory10:25:171
280599291,26cyclictest0-21swapper/008:12:340
2805992910,16cyclictest18567-21munin-run11:25:010
2823992822,5cyclictest0-21swapper/309:05:593
2823992817,3cyclictest0-21swapper/312:26:413
2823992814,9cyclictest0-21swapper/311:12:413
282399281,19cyclictest0-21swapper/310:45:533
282399281,18cyclictest0-21swapper/309:24:273
282399281,16cyclictest0-21swapper/312:20:113
282399281,12cyclictest0-21swapper/309:17:373
282399280,22cyclictest0-21swapper/311:41:593
282399280,19cyclictest0-21swapper/307:58:393
281399285,14cyclictest86750irq/128-enp1s0-08:49:551
280599281,13cyclictest24528-21munin-run10:25:010
282399279,7cyclictest0-21swapper/308:35:473
2823992716,3cyclictest0-21swapper/307:44:153
2823992712,3cyclictest0-21swapper/308:40:553
282399271,12cyclictest20055-21df09:00:153
282399271,12cyclictest0-21swapper/312:37:233
282399271,12cyclictest0-21swapper/311:47:233
282399271,12cyclictest0-21swapper/311:47:233
282399271,12cyclictest0-21swapper/310:17:293
282399271,11cyclictest0-21swapper/312:17:373
282399271,10cyclictest0-21swapper/309:59:363
280599278,16cyclictest25456-21munin-run08:00:020
280599271,13cyclictest20766-21munin-run11:30:010
282399268,8cyclictest0-21swapper/308:48:493
2823992614,3cyclictest0-21swapper/307:10:483
282399261,18cyclictest0-21swapper/308:18:473
282399261,13cyclictest0-21swapper/311:16:193
2823992611,3cyclictest0-21swapper/308:24:513
280599269,14cyclictest3176-21awk10:50:020
280599264,10cyclictest27676-21idleruntime-cro08:05:020
280599261,12cyclictest4747-21awk09:40:020
280599261,12cyclictest4747-21awk09:40:020
293962250,4sleep0280599cyclictest09:20:200
2823992511,4cyclictest0-21swapper/307:23:513
281399258,6cyclictest23119-40dnf07:53:331
281399251,3cyclictest20878-21tune2fs10:15:171
281399251,14cyclictest0-21swapper/110:55:051
281399251,13cyclictest10057-21latency_hist07:25:011
282399244,3cyclictest0-21swapper/310:01:163
281399242,12cyclictest939-21munin-run10:45:021
281399242,12cyclictest939-21munin-run10:45:011
281399241,6cyclictest0-21swapper/111:15:091
281399239,4cyclictest14422-21hddtemp_smartct10:00:161
281399232,2cyclictest28572-21kworker/u8:010:35:061
281399231,3cyclictest13224-21ntp_kernel_err11:10:201
281399231,3cyclictest10140-21tr11:05:141
281399231,21cyclictest0-21swapper/109:52:001
281399231,17cyclictest0-21swapper/110:24:231
281399231,14cyclictest0-21swapper/112:06:181
281399231,14cyclictest0-21swapper/111:38:361
281399230,6cyclictest0-21swapper/112:37:581
281399230,16cyclictest0-21swapper/109:40:141
2805992311,11cyclictest0-21swapper/009:30:010
194762231,18sleep119484-21latency_hist09:00:011
281899221,5cyclictest12259-21kworker/u8:212:28:292
281399222,15cyclictest0-21swapper/108:42:151
2813992218,3cyclictest0-21swapper/107:17:131
2813992216,3cyclictest3866-21uniq10:50:151
281399221,3cyclictest9501-21ntp_states12:15:221
281399221,18cyclictest6532-21expr12:10:141
281399221,16cyclictest0-21swapper/108:21:341
281399221,15cyclictest0-21swapper/110:13:141
281399221,12cyclictest0-21swapper/109:37:221
281399221,12cyclictest0-21swapper/109:37:221
281399221,10cyclictest0-21swapper/109:19:281
281399220,7cyclictest0-21swapper/109:31:161
281399220,6cyclictest0-21swapper/109:10:201
281399220,12cyclictest0-21swapper/112:02:091
280599222,17cyclictest11966-21munin-run11:10:020
2805992217,3cyclictest12415-21grep08:40:280
2805992216,3cyclictest5867-21grep08:25:270
280599221,5cyclictest0-21swapper/010:28:050
281899214,3cyclictest86550irq/126-enp1s0-09:20:192
281899211,6cyclictest0-21swapper/207:45:132
281399218,4cyclictest0-21swapper/112:24:381
281399212,3cyclictest0-21swapper/109:20:361
281399212,10cyclictest14791-21sh12:30:021
281399211,2cyclictest10700-21perf08:40:011
281399211,18cyclictest0-21swapper/109:28:221
281399211,15cyclictest0-21swapper/108:32:111
281399211,15cyclictest0-21swapper/108:17:581
281399211,13cyclictest0-21swapper/111:55:021
281399211,11cyclictest0-21swapper/110:07:341
281399211,11cyclictest0-21swapper/109:47:561
281399211,10cyclictest19142-21awk07:45:111
281399211,10cyclictest0-21swapper/111:54:421
281399210,19cyclictest0-21swapper/107:30:241
281399210,19cyclictest0-21swapper/107:30:241
280599212,16cyclictest6931-21munin-run09:45:020
280599212,11cyclictest29152-21/usr/sbin/munin10:35:180
280599211,11cyclictest15316-21/usr/sbin/munin08:50:120
280599211,11cyclictest12828-21/usr/sbin/munin12:25:180
281899206,3cyclictest15025-21runrttasks07:35:142
281899205,5cyclictest31629-21runrttasks11:53:292
281399206,7cyclictest28220-21kworker/u8:007:43:581
281399205,12cyclictest21-21rcuc/111:25:131
281399202,3cyclictest0-21swapper/112:34:181
281399202,3cyclictest0-21swapper/111:00:061
2813992016,3cyclictest0-21swapper/111:21:151
2813992013,2cyclictest0-21swapper/108:54:051
281399201,17cyclictest0-21swapper/108:10:151
281399201,15cyclictest0-21swapper/109:55:371
281399201,15cyclictest0-21swapper/108:28:391
281399201,14cyclictest0-21swapper/109:06:511
281399201,14cyclictest0-21swapper/109:00:421
281399201,12cyclictest0-21swapper/111:43:221
281399201,11cyclictest0-21swapper/110:49:431
281399201,10cyclictest5099-21cat07:15:011
281399200,6cyclictest0-21swapper/110:31:401
281399200,18cyclictest0-21swapper/107:25:401
280599209,3cyclictest27890-21/usr/sbin/munin08:05:250
280599209,3cyclictest17224-21/usr/sbin/munin12:35:270
280599204,6cyclictest22441-21timerwakeupswit11:30:260
280599201,6cyclictest23897-21diskmemload12:00:190
280599201,11cyclictest15960-21/usr/sbin/munin10:05:200
281899196,5cyclictest26112-21awk09:15:012
281899192,7cyclictest12047-21sshd07:27:282
281899191,8cyclictest15735-21munin-run10:05:012
281899191,8cyclictest0-21swapper/211:30:442
281899191,7cyclictest25651-21latency10:25:172
2818991914,3cyclictest29431-21grep08:05:262
2818991913,3cyclictest2124-21/usr/sbin/munin08:20:172
2818991912,4cyclictest8625-21uniq07:20:142
2818991912,4cyclictest29190-21/usr/sbin/munin11:45:402
2818991912,4cyclictest29190-21/usr/sbin/munin11:45:402
281899191,15cyclictest9267-21grep11:00:272
281399197,6cyclictest0-21swapper/111:34:381
281399192,8cyclictest792-21avahi-daemon11:45:151
281399192,8cyclictest792-21avahi-daemon11:45:141
281399191,3cyclictest0-21swapper/107:39:591
280599199,3cyclictest86750irq/128-enp1s0-11:42:450
280599197,3cyclictest22552-1kworker/u9:109:45:230
280599195,5cyclictest779-21ntpd12:07:160
2805991914,3cyclictest8188-21cpuspeed_turbos07:20:120
2805991914,3cyclictest2774-21gltestperf08:20:180
2805991913,3cyclictest836-21runrttasks07:10:130
2805991912,4cyclictest24241-21grep09:10:130
280599191,10cyclictest14634-21snmpd09:54:090
52482181,2sleep20-21swapper/210:53:432
281899183,9cyclictest792-21avahi-daemon10:55:032
281899183,3cyclictest86550irq/126-enp1s0-12:00:012
281899181,7cyclictest19492-21munin-run09:00:012
281899181,7cyclictest12185-21grep12:20:282
2818991813,3cyclictest932-21sh10:45:012
2818991813,3cyclictest932-21sh10:45:012
2818991813,3cyclictest8034-21grep08:30:252
2818991813,3cyclictest15111-21cpuspeed_turbos12:30:122
2818991812,3cyclictest18331-21cat07:40:262
2818991811,4cyclictest23962-21cat09:10:012
281399181,16cyclictest0-21swapper/107:56:201
280599188,6cyclictest0-21swapper/008:39:560
280599188,3cyclictest86750irq/128-enp1s0-10:57:220
280599183,12cyclictest16886-21/usr/sbin/munin07:40:170
280599182,13cyclictest28177-21grep11:45:170
280599182,13cyclictest28177-21grep11:45:170
280599181,7cyclictest0-21swapper/011:11:350
280599181,7cyclictest0-21swapper/011:00:580
280599181,3cyclictest0-21swapper/009:31:170
2805991812,3cyclictest30036-21expr11:50:130
2805991812,3cyclictest24255-21ntpq11:35:220
2805991812,3cyclictest15914-21awk11:15:270
2805991812,3cyclictest15325-21grep07:35:180
280599181,15cyclictest8488-21idleruntime-cro08:35:010
280599181,14cyclictest14634-21snmpd10:30:250
280599181,12cyclictest0-21swapper/008:45:230
248082181,2sleep20-21swapper/211:35:382
215122181,2sleep00-21swapper/009:02:500
281899175,4cyclictest23897-21diskmemload11:07:112
281899174,5cyclictest0-21swapper/210:15:202
281899174,4cyclictest696-21expr09:30:122
281899173,4cyclictest3110-21chrt09:35:142
281899173,4cyclictest3110-21chrt09:35:142
281899173,11cyclictest21669-21grep07:50:132
281899172,6cyclictest0-21swapper/211:20:172
281899172,4cyclictest25741-21diskstats11:40:142
281899171,9cyclictest0-21swapper/210:45:032
281899171,7cyclictest18488-21smartctl12:35:242
281899171,7cyclictest0-21swapper/212:00:152
281899171,6cyclictest8521-21ls12:15:132
281899171,6cyclictest0-21swapper/212:13:552
281899171,4cyclictest0-21swapper/208:00:272
281899171,3cyclictest30878-21grep09:25:132
2818991712,3cyclictest9141-21grep08:35:162
2818991712,3cyclictest4033-21ntp_kernel_err07:10:212
2818991712,3cyclictest32621-21sh08:15:132
2818991712,3cyclictest15316-21/usr/sbin/munin08:50:322
2818991712,3cyclictest11549-21/usr/sbin/munin09:55:222
2818991711,4cyclictest8022-21latency09:45:182
281899171,13cyclictest26856-21cut09:15:152
2818991710,5cyclictest8768-21kworker/u8:109:50:102
280599171,8cyclictest6888-21strings07:15:220
280599171,7cyclictest17518-21/usr/sbin/munin08:55:150
280599171,4cyclictest0-21swapper/010:14:120
2805991714,2cyclictest15036-21kworker/0:007:45:210
2805991711,4cyclictest575-21sshd10:41:280
2805991711,4cyclictest575-21sshd10:41:280
280599171,13cyclictest11549-21/usr/sbin/munin09:55:140
280599170,7cyclictest0-21swapper/010:15:160
280599170,7cyclictest0-21swapper/010:02:100
214892171,2sleep20-21swapper/209:02:312
198352171,2sleep20-21swapper/210:11:242
177382171,2sleep20-21swapper/210:07:422
281899165,3cyclictest86550irq/126-enp1s0-09:40:222
281899161,7cyclictest18795-21/usr/sbin/munin11:25:152
281899161,5cyclictest30710-21unixbench_singl10:35:262
281899161,3cyclictest0-21swapper/207:16:402
2818991612,3cyclictest0-21swapper/208:45:332
2818991612,3cyclictest0-21swapper/207:56:092
2818991611,3cyclictest5120-21open_files12:05:212
2818991611,3cyclictest4678-21entropy08:25:152
2818991610,4cyclictest11767-21/usr/sbin/munin08:40:202
2818991610,4cyclictest0-21swapper/210:30:242
280599165,3cyclictest86750irq/128-enp1s0-08:18:520
280599165,3cyclictest23897-21diskmemload12:10:170
280599161,6cyclictest10290-21/usr/sbin/munin07:25:130
280599161,6cyclictest0-21swapper/012:30:220
280599161,3cyclictest0-21swapper/009:15:260
2805991611,3cyclictest21642-21awk07:50:140
2805991511,3cyclictest0-21swapper/007:30:160
2805991511,3cyclictest0-21swapper/007:30:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional