You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-24 - 11:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 24, 2025 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
272422237163,24sleep00-21swapper/019:08:350
253922203168,24sleep30-21swapper/319:05:073
270452202165,24sleep20-21swapper/219:06:022
273132200165,23sleep10-21swapper/119:09:321
27445999087,2cyclictest0-21swapper/222:37:392
27445999087,2cyclictest0-21swapper/222:37:382
27445997773,3cyclictest0-21swapper/222:25:022
285552450,9sleep121-21rcuc/119:10:201
2744599424,31cyclictest30308-40dnf00:06:342
2744599404,25cyclictest18330-21basename23:40:182
2744599382,5cyclictest801-21snmpd22:05:022
2744599381,5cyclictest28065-21/usr/sbin/munin22:50:112
293932360,6sleep028624-21/usr/sbin/munin00:05:180
2744599362,6cyclictest13924-40dnf21:04:592
2744599361,22cyclictest22350-21/usr/sbin/munin20:10:272
2743599368,13cyclictest11126-21munin-run19:45:010
27435993510,12cyclictest8930-21munin-run19:40:020
211482350,4sleep22744599cyclictest23:45:272
211482350,4sleep22744599cyclictest23:45:272
2744599344,3cyclictest86950irq/126-enp1s0-21:40:152
2744599341,5cyclictest801-21snmpd00:20:282
2744599341,4cyclictest0-21swapper/221:45:182
2744599331,6cyclictest11017-21/usr/sbin/munin23:25:212
2744599331,19cyclictest0-21swapper/223:50:162
27445993311,19cyclictest2589-21needreboot21:50:202
2743599331,14cyclictest0-21swapper/023:21:560
264342330,1sleep30-21swapper/322:45:163
2744599326,14cyclictest86950irq/126-enp1s0-19:25:142
2744599325,3cyclictest86950irq/126-enp1s0-21:40:022
27445993213,16cyclictest4640-21grep20:40:282
2744599321,28cyclictest2361-21cat19:25:022
2744599321,13cyclictest15138-21kworker/2:220:24:422
27435993210,11cyclictest29622-21awk19:15:010
2744599315,14cyclictest16144-21diskmemload00:00:052
2744599313,3cyclictest87250irq/129-enp1s0-00:00:022
2744599312,5cyclictest9812-40dnf22:05:332
2744599312,4cyclictest23311-21diskstats21:25:142
2744599312,26cyclictest12509-21cat22:15:012
2744599312,25cyclictest32091-21smart_sda00:10:242
2744599311,7cyclictest2230-21/usr/sbin/munin23:05:172
2744599311,4cyclictest7302-21diskstats19:35:132
2744599311,4cyclictest1071-21ntp_states23:00:252
2744599311,4cyclictest10109-21/usr/sbin/munin20:55:122
2744599311,27cyclictest6614-21grep00:25:272
2744599310,3cyclictest12050irq/123-ahci[0023:40:012
2743599313,13cyclictest13333-21idleruntime-cro19:50:010
2743599310,9cyclictest16417-21kworker/0:222:22:100
2745299301,11cyclictest0-21swapper/323:09:053
2744599309,18cyclictest14337-21if_err_enp2s019:50:182
2744599309,18cyclictest14231-21cpu21:05:122
2744599306,21cyclictest18904-21df21:15:132
2744599305,3cyclictest87250irq/129-enp1s0-19:55:122
2744599305,13cyclictest30900-21idleruntime22:55:162
2744599302,4cyclictest15139-21diskstats22:20:142
2744599302,4cyclictest13859-21smart_sda22:15:252
2744599302,3cyclictest32402-21smart_sda20:30:252
2744599302,3cyclictest12446-21/usr/sbin/munin19:45:262
2744599302,3cyclictest12050irq/123-ahci[0022:50:012
2744599302,3cyclictest1085-21fschecks_count00:15:152
2744599301,3cyclictest6285-21memory20:45:182
2744599301,3cyclictest4208-21perf23:14:572
2744599301,3cyclictest25312-21/usr/sbin/munin20:15:202
27445993012,15cyclictest8300-21ntpq00:30:202
2743599308,7cyclictest13290-21kworker/0:019:54:060
2743599302,19cyclictest18173-21kworker/0:021:32:460
2743599301,15cyclictest0-21swapper/023:54:430
2743599301,14cyclictest7-21ksoftirqd/022:02:000
2743599301,14cyclictest28400-21munin-run00:05:000
2743599300,13cyclictest0-21swapper/023:49:480
2743599300,13cyclictest0-21swapper/023:49:480
2744599298,18cyclictest6624-21/usr/sbin/munin23:15:132
2744599298,18cyclictest1863-21grep20:35:202
2744599297,18cyclictest27758-21grep19:10:132
2744599292,5cyclictest30007-21users19:15:032
2744599291,4cyclictest21114-21diskstats21:20:142
2744599291,4cyclictest21114-21diskstats21:20:142
2744599291,3cyclictest9156-21df23:20:132
2744599291,3cyclictest5053-21/usr/sbin/munin21:55:252
2744599291,3cyclictest4957-21cat19:30:132
2744599291,3cyclictest20916-21/usr/sbin/munin20:05:162
2744599291,3cyclictest20916-21/usr/sbin/munin20:05:162
2744599291,2cyclictest0-21swapper/222:30:162
2744599291,15cyclictest11065-21sshd00:35:312
27445992910,16cyclictest8639-21sh20:50:232
2743599296,11cyclictest31548-21perf21:45:000
2743599292,13cyclictest6769-21kworker/0:100:35:140
2743599291,14cyclictest0-21swapper/023:03:080
2743599291,12cyclictest0-21swapper/020:46:200
2743599291,11cyclictest0-21swapper/020:33:560
2743599290,11cyclictest0-21swapper/019:34:120
10702290,4sleep32745299cyclictest23:00:253
2744599282,3cyclictest87150irq/128-enp1s0-20:00:222
2744599281,5cyclictest25520-21forks21:30:162
2744599281,4cyclictest29292-21diskstats20:25:152
2744599281,4cyclictest16718-21/usr/sbin/munin21:10:152
2744599281,3cyclictest9976-21grep19:40:162
2744599281,3cyclictest14088-21latency23:30:192
2743599282,12cyclictest13001-21idleruntime-cro23:30:010
2743599281,12cyclictest0-21swapper/021:57:070
2743599281,12cyclictest0-21swapper/021:23:080
2743599281,12cyclictest0-21swapper/021:23:080
2743599281,11cyclictest0-21swapper/022:37:080
2743599281,11cyclictest0-21swapper/022:37:080
27435992810,16cyclictest24938-21munin-run21:30:010
2743599280,14cyclictest32435-21kworker/0:200:12:250
2744599279,15cyclictest24811-21sendmail_mailqu22:40:222
2743599271,12cyclictest0-21swapper/019:58:440
27435992711,14cyclictest22134-21perf20:10:010
27435992711,14cyclictest22134-21perf20:10:010
2745299261,9cyclictest0-21swapper/319:45:283
2744199264,19cyclictest0-21swapper/122:05:251
2744199261,17cyclictest958-21/usr/sbin/munin20:35:201
2744199261,17cyclictest9152-21users20:50:271
27435992610,14cyclictest9243-21perf00:35:010
2743599260,13cyclictest0-21swapper/022:10:280
185812260,1sleep10-21swapper/123:40:201
2744199251,5cyclictest12605-21/usr/sbin/munin23:25:251
2743599259,2cyclictest13688-21kworker/0:221:12:160
2743599255,3cyclictest12595-21kworker/0:022:17:540
2743599253,3cyclictest18173-21kworker/0:021:37:060
2743599252,8cyclictest344-21kworker/0:120:43:200
2743599251,5cyclictest29773-21sshd22:52:510
27435992510,3cyclictest344-21kworker/0:120:35:340
2744199249,7cyclictest17520-21systemd-journal19:57:411
27441992415,2cyclictest3289-21/usr/sbin/munin19:25:181
27441992414,2cyclictest5992-21wc19:30:231
2744199241,18cyclictest0-21swapper/100:03:291
2743599247,15cyclictest4214-21awk23:10:020
2743599246,4cyclictest24222-21kworker/0:220:25:160
2743599244,2cyclictest28611-21kworker/0:000:17:410
2743599240,17cyclictest7892-21kworker/0:022:07:020
2744199231,16cyclictest0-21swapper/120:10:251
2744199231,14cyclictest87150irq/128-enp1s0-19:47:411
2743599238,3cyclictest5114-21kworker/0:220:50:140
2743599237,14cyclictest13963-21perf21:05:010
2743599236,3cyclictest31822-21kworker/0:222:59:440
27435992310,2cyclictest17029-21kworker/0:223:39:040
2745299221,6cyclictest0-21swapper/321:27:073
2745299221,4cyclictest0-21swapper/323:42:303
2745299221,15cyclictest0-21swapper/323:37:243
2745299221,15cyclictest0-21swapper/320:49:283
2744199221,18cyclictest29884-21df19:15:021
2744199221,18cyclictest29617-21hddtemp_smartct20:25:171
2744199221,16cyclictest0-21swapper/123:10:021
2744199221,16cyclictest0-21swapper/120:43:591
2744199221,15cyclictest0-21swapper/120:30:211
2744199220,16cyclictest0-21swapper/121:10:191
2743599227,3cyclictest31822-21kworker/0:223:16:260
2743599225,4cyclictest17472-21kworker/0:120:01:360
2743599222,11cyclictest19234-21kworker/0:323:57:070
2743599221,16cyclictest0-21swapper/022:29:340
2743599220,13cyclictest0-21swapper/000:22:350
2745299211,9cyclictest0-21swapper/321:02:253
2745299211,8cyclictest0-21swapper/322:18:343
27452992116,3cyclictest32604-21grep20:30:263
2745299210,2cyclictest12050irq/123-ahci[0019:15:023
2744199212,11cyclictest11472-21ntp_states22:10:231
2744199212,10cyclictest6630-21smart_sda20:45:241
27441992114,2cyclictest0-21swapper/123:38:271
2744199211,18cyclictest0-21swapper/100:10:161
2744199211,17cyclictest7631-21hddtemp_smartct19:35:161
2744199211,15cyclictest0-21swapper/121:09:451
2744199211,15cyclictest0-21swapper/120:00:211
2744199211,14cyclictest0-21swapper/119:50:081
2744199211,14cyclictest0-21swapper/119:21:001
2744199211,11cyclictest20958-21interrupts20:05:181
2744199211,11cyclictest20958-21interrupts20:05:181
2744199211,10cyclictest27896-21latency_hist22:50:021
2744199210,15cyclictest0-21swapper/121:00:301
2743599217,5cyclictest0-21swapper/022:30:110
2743599217,3cyclictest22801-21kworker/0:119:18:510
2743599213,2cyclictest3421-21kworker/0:021:54:490
2743599213,10cyclictest13215-21/usr/sbin/munin23:30:230
2743599211,13cyclictest0-21swapper/019:21:240
2743599211,12cyclictest0-21swapper/020:23:080
2743599211,11cyclictest23223-21kworker/0:222:47:480
2743599211,10cyclictest0-21swapper/021:09:400
27435992110,2cyclictest6175-21kworker/0:123:42:080
2743599210,13cyclictest0-21swapper/020:14:040
2745299204,3cyclictest0-21swapper/319:25:213
2745299201,9cyclictest21721-21kworker/u8:119:59:373
2745299201,6cyclictest0-21swapper/300:33:303
2745299201,4cyclictest0-21swapper/321:22:003
2745299201,4cyclictest0-21swapper/321:22:003
2745299201,14cyclictest0-21swapper/321:30:013
2745299201,11cyclictest23072-21sshd22:36:323
2745299201,11cyclictest23072-21sshd22:36:323
2745299201,11cyclictest13549-21/usr/sbin/munin19:50:233
2745299201,10cyclictest0-21swapper/320:50:223
2744199209,6cyclictest0-21swapper/121:25:021
2744199209,2cyclictest19843-21switchtime21:15:241
27441992016,3cyclictest0-21swapper/122:37:481
27441992016,3cyclictest0-21swapper/122:37:471
27441992013,4cyclictest25759-21tail21:30:171
2744199201,18cyclictest0-21swapper/123:47:181
2744199201,18cyclictest0-21swapper/123:47:181
2744199201,17cyclictest9167-21runrttasks00:34:221
2744199201,17cyclictest6378-21chrt23:14:561
2744199201,17cyclictest0-21swapper/122:41:041
2744199201,14cyclictest0-21swapper/120:55:381
2744199201,11cyclictest14085-21latency23:30:191
2744199201,10cyclictest13513-21/usr/sbin/munin22:15:191
2744199200,18cyclictest0-21swapper/122:20:011
2743599204,3cyclictest9196-21kworker/0:020:56:340
2743599203,2cyclictest7-21ksoftirqd/000:25:430
2743599201,9cyclictest0-21swapper/021:15:160
2743599201,11cyclictest5729-21sed23:10:260
2743599201,11cyclictest0-21swapper/021:47:390
2745299198,1cyclictest0-21swapper/322:27:523
2745299195,3cyclictest0-21swapper/323:18:413
2745299191,9cyclictest0-21swapper/319:19:103
2745299191,5cyclictest24571-21sort23:55:173
2745299191,15cyclictest27661-21memory20:20:203
2745299191,12cyclictest0-21swapper/300:20:233
2745299191,10cyclictest801-21snmpd20:04:393
2745299191,10cyclictest17964-21users21:10:273
2745299191,10cyclictest0-21swapper/321:16:243
2744199199,4cyclictest0-21swapper/121:40:441
2744199198,3cyclictest73172switchtime22:00:231
2744199192,3cyclictest10523-21ntp_states00:35:191
27441991916,2cyclictest0-21swapper/121:59:481
27441991915,3cyclictest0-21swapper/122:29:081
27441991915,3cyclictest0-21swapper/121:46:191
27441991913,3cyclictest3337-21grep00:20:161
2744199191,2cyclictest0-21swapper/121:27:531
27441991912,3cyclictest0-21swapper/120:16:341
2744199191,17cyclictest0-21swapper/123:55:511
2744199191,17cyclictest0-21swapper/122:56:541
2744199191,16cyclictest0-21swapper/123:00:581
27441991911,3cyclictest0-21swapper/119:40:351
2744199191,11cyclictest28942-21expr00:05:151
2744199191,10cyclictest2439-21kernelversion21:50:191
2744199191,10cyclictest20088-21sed22:30:181
2744199190,14cyclictest0-21swapper/120:20:301
2743599191,11cyclictest0-21swapper/020:15:250
2743599191,10cyclictest32251-21kworker/0:019:25:080
2743599190,12cyclictest0-21swapper/022:41:320
2745299187,3cyclictest22124-21awk20:10:013
2745299187,3cyclictest22124-21awk20:10:013
2745299187,2cyclictest32188-21timerandwakeup00:10:263
2745299182,13cyclictest10991-21find20:55:203
2745299181,9cyclictest0-21swapper/319:38:133
2745299181,8cyclictest12984-21latency_hist23:30:003
27452991814,3cyclictest0-21swapper/321:07:513
2745299181,16cyclictest0-21swapper/321:47:513
2745299181,14cyclictest5685-21sensors23:10:253
2745299181,10cyclictest23582-21/usr/sbin/munin22:40:163
2745299180,6cyclictest0-21swapper/322:00:213
2745299180,10cyclictest0-21swapper/322:24:253
2744199181,9cyclictest22989-21strings23:50:191
2744199181,16cyclictest0-21swapper/123:21:411
2744199181,16cyclictest0-21swapper/122:50:191
2744199181,16cyclictest0-21swapper/121:35:081
2744199181,16cyclictest0-21swapper/121:21:261
2744199181,16cyclictest0-21swapper/100:18:271
2744199181,12cyclictest0-21swapper/123:18:101
2744199180,17cyclictest0-21swapper/100:29:301
2745299174,3cyclictest87250irq/129-enp1s0-00:06:343
2745299172,6cyclictest9492-21seq19:40:133
2745299171,8cyclictest0-21swapper/323:51:133
2745299171,7cyclictest19683-21tune2fs22:30:163
2745299171,7cyclictest0-21swapper/323:34:353
2745299171,6cyclictest577-21systemd-journal00:20:023
2745299171,6cyclictest0-21swapper/322:50:163
2745299171,6cyclictest0-21swapper/322:05:293
2745299171,6cyclictest0-21swapper/319:20:163
2745299171,5cyclictest0-21swapper/300:27:543
2745299171,4cyclictest0-21swapper/321:51:483
27452991713,3cyclictest41-21ksoftirqd/300:40:013
27452991712,3cyclictest2133-21awk20:35:233
27452991712,3cyclictest10882-21sed23:25:003
2745299171,15cyclictest0-21swapper/323:46:133
2745299171,15cyclictest0-21swapper/323:46:123
2745299171,15cyclictest0-21swapper/300:04:163
27452991711,4cyclictest0-21swapper/322:10:253
2745299171,13cyclictest292852sleep320:25:133
2745299171,10cyclictest0-21swapper/320:15:163
27452991710,4cyclictest30208-21tail21:40:163
2745299161,6cyclictest0-21swapper/322:56:363
2745299161,5cyclictest0-21swapper/320:40:143
2745299161,2cyclictest0-21swapper/321:55:093
27452991611,3cyclictest5569-21irqstats19:30:183
27452991611,3cyclictest27695-21expr21:35:123
240002160,2sleep30-21swapper/320:11:063
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional