You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-08 - 08:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 08, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
279692238161,24sleep00-21swapper/019:08:280
279812236161,23sleep20-21swapper/219:08:382
278952236166,56sleep30-21swapper/319:07:313
261312232163,55sleep10-21swapper/119:05:111
1754921430,3sleep32819199cyclictest22:25:013
2818599102100,1cyclictest0-21swapper/221:30:392
283922710,4sleep10-21swapper/121:35:131
28185997067,2cyclictest0-21swapper/222:00:312
28185995754,2cyclictest0-21swapper/220:45:102
322982560,4sleep011-21rcuc/000:10:150
228622540,3sleep3581-21systemd-journal20:10:013
210272500,1sleep20-21swapper/220:05:142
28173994111,14cyclictest20376-21munin-run23:45:010
2819199400,29cyclictest0-21swapper/300:17:463
2817399359,12cyclictest30744-21munin-run22:55:010
2819199331,15cyclictest0-21swapper/322:13:163
2819199331,15cyclictest0-21swapper/319:10:353
2819199331,15cyclictest0-21swapper/319:10:353
2819199321,17cyclictest0-21swapper/300:29:543
2819199321,15cyclictest0-21swapper/322:49:263
2819199321,13cyclictest0-21swapper/300:04:003
2819199320,15cyclictest0-21swapper/300:22:203
2819199320,13cyclictest0-21swapper/319:25:133
2817399324,13cyclictest4990-21munin-run23:10:010
2817399324,13cyclictest22867-21sadc20:10:010
65912310,7sleep339-21rcuc/323:10:253
2819199311,20cyclictest0-21swapper/320:31:013
2819199311,15cyclictest0-21swapper/323:00:463
2819199311,15cyclictest0-21swapper/321:15:543
2819199311,13cyclictest0-21swapper/321:59:333
2819199311,13cyclictest0-21swapper/321:42:193
2819199311,13cyclictest0-21swapper/300:13:083
2819199310,13cyclictest0-21swapper/321:03:543
2819199301,15cyclictest0-21swapper/323:58:563
2819199301,14cyclictest16907-21diskmemload00:07:383
2819199301,14cyclictest0-21swapper/322:16:523
2819199301,13cyclictest0-21swapper/321:25:223
2819199301,12cyclictest0-21swapper/323:48:423
2819199301,12cyclictest0-21swapper/322:55:183
2819199301,12cyclictest0-21swapper/321:51:013
2817399307,11cyclictest29487-21awk20:25:010
28173993015,8cyclictest29985-21vmstat19:10:330
28173993015,8cyclictest29985-21vmstat19:10:330
2819199291,13cyclictest0-21swapper/300:37:023
2819199291,12cyclictest0-21swapper/323:07:503
2819199290,15cyclictest0-21swapper/322:40:583
2819199281,14cyclictest0-21swapper/323:15:343
2819199281,12cyclictest0-21swapper/323:33:023
2819199281,12cyclictest0-21swapper/322:50:183
2819199281,12cyclictest0-21swapper/322:35:573
2819199281,11cyclictest0-21swapper/323:20:303
2819199281,11cyclictest0-21swapper/320:16:323
2818199281,18cyclictest25364-21/usr/sbin/munin23:55:161
2819199271,9cyclictest0-21swapper/323:50:063
2819199271,9cyclictest0-21swapper/319:20:093
2819199271,13cyclictest0-21swapper/319:41:493
2819199271,12cyclictest0-21swapper/321:30:533
2819199271,11cyclictest16882-21hddtemp_smartct23:35:183
2819199271,11cyclictest16882-21hddtemp_smartct23:35:183
2819199271,11cyclictest0-21swapper/322:00:133
2819199271,10cyclictest0-21swapper/321:37:543
2819199270,12cyclictest0-21swapper/320:10:273
2818199272,22cyclictest27270-21munin-run20:20:011
28181992713,9cyclictest0-21swapper/122:29:211
28181992711,5cyclictest29718-40dnf19:10:331
28181992711,5cyclictest29718-40dnf19:10:331
2817399270,12cyclictest7486-21awk19:35:010
2819199261,9cyclictest0-21swapper/322:31:013
2819199261,9cyclictest0-21swapper/322:31:013
2819199261,6cyclictest0-21swapper/323:43:103
2819199261,19cyclictest0-21swapper/320:58:323
2819199261,15cyclictest0-21swapper/320:01:113
2819199261,11cyclictest0-21swapper/322:06:533
2819199261,10cyclictest0-21swapper/321:45:493
2819199261,10cyclictest0-21swapper/321:11:013
2819199261,10cyclictest0-21swapper/320:45:233
2819199261,10cyclictest0-21swapper/319:17:093
2819199261,10cyclictest0-21swapper/319:17:093
2818199261,17cyclictest29456-21sed22:50:171
2818199261,17cyclictest13251-21latency_hist22:15:011
2819199251,9cyclictest0-21swapper/322:26:423
2819199251,12cyclictest0-21swapper/323:29:583
2819199251,11cyclictest0-21swapper/319:54:213
2818199251,17cyclictest86750irq/128-enp1s0-21:45:221
2817399258,14cyclictest12501-21munin-run21:00:010
2819199249,3cyclictest0-21swapper/320:44:533
2819199241,10cyclictest0-21swapper/300:32:063
2819199240,13cyclictest0-21swapper/321:07:383
2819199240,12cyclictest0-21swapper/319:59:133
2818199241,3cyclictest10601-21df_inode00:35:141
2818199241,16cyclictest0-21swapper/122:45:141
218382240,7sleep121-21rcuc/123:45:231
2819199231,16cyclictest0-21swapper/319:48:413
2819199230,10cyclictest0-21swapper/320:22:003
2818599233,10cyclictest0-21swapper/222:00:012
2818199236,8cyclictest0-21swapper/100:15:161
2818199232,18cyclictest14634-21snmpd21:26:161
2818199232,12cyclictest0-21swapper/123:10:211
28181992315,3cyclictest0-21swapper/123:30:281
2818199231,3cyclictest21158-21ls20:05:141
28181992312,3cyclictest12942-21ls23:25:241
2818199231,17cyclictest0-21swapper/100:20:491
2818199231,16cyclictest0-21swapper/123:41:341
2819199221,9cyclictest0-21swapper/321:23:013
2819199221,13cyclictest0-21swapper/320:50:093
2819199221,13cyclictest0-21swapper/320:38:433
2819199221,13cyclictest0-21swapper/319:30:393
2819199221,12cyclictest0-21swapper/319:36:153
2818599221,4cyclictest0-21swapper/222:29:212
2818199229,11cyclictest2108-21kworker/u8:121:12:021
2818199222,2cyclictest86750irq/128-enp1s0-21:53:121
2818199222,11cyclictest31649-21hddtemp_smartct22:55:181
2818199221,7cyclictest21305-21latency_hist21:20:011
28181992215,2cyclictest0-21swapper/122:03:551
28181992214,3cyclictest0-21swapper/100:31:301
2818199221,3cyclictest13292-21processes19:45:231
2818199221,19cyclictest0-21swapper/100:14:371
2818199221,16cyclictest0-21swapper/122:17:511
2818199221,15cyclictest0-21swapper/123:15:191
2818199221,15cyclictest0-21swapper/123:01:121
2818199221,11cyclictest0-21swapper/123:23:021
2818199220,5cyclictest0-21swapper/122:30:241
2818199220,5cyclictest0-21swapper/122:30:241
2817399228,4cyclictest0-21swapper/000:00:190
2817399227,13cyclictest0-21swapper/023:55:010
2817399221,10cyclictest0-21swapper/022:05:010
128662220,4sleep011-21rcuc/023:25:230
2819199210,13cyclictest0-21swapper/320:29:283
2819199210,13cyclictest0-21swapper/320:29:283
2818599217,5cyclictest9048-21needreboot00:30:192
2818599215,6cyclictest0-21swapper/223:35:252
2818599215,6cyclictest0-21swapper/223:35:252
2818599213,5cyclictest32295-1kworker/u9:000:11:302
2818599211,3cyclictest0-21swapper/223:04:142
2818199211,6cyclictest0-21swapper/123:35:481
2818199211,6cyclictest0-21swapper/123:35:481
2818199211,6cyclictest0-21swapper/121:20:021
28181992113,3cyclictest0-21swapper/100:08:121
2818199211,2cyclictest4617-21hddtemp_smartct20:40:201
2818199211,2cyclictest0-21swapper/119:20:071
2818199211,18cyclictest0-21swapper/121:55:131
2818199211,18cyclictest0-21swapper/120:03:281
2818199211,18cyclictest0-21swapper/119:26:271
2818199211,17cyclictest10950-21munin-run22:10:011
2818199211,15cyclictest0-21swapper/123:07:011
2818199211,15cyclictest0-21swapper/122:24:511
2818199211,15cyclictest0-21swapper/100:00:271
28181992111,3cyclictest0-21swapper/123:50:331
2817399211,14cyclictest0-21swapper/023:04:400
2817399211,11cyclictest21540-21/usr/sbin/munin21:20:180
2817399211,11cyclictest1453-21/usr/sbin/munin00:15:220
2817399211,10cyclictest25978-21hddtemp_smartct20:15:190
2818599202,13cyclictest10958-21perf22:10:022
28185992014,3cyclictest18717-21grep21:10:282
2818599201,17cyclictest18465-21perf20:00:012
28181992016,3cyclictest0-21swapper/120:24:091
28181992016,3cyclictest0-21swapper/119:58:481
28181992013,2cyclictest0-21swapper/100:29:091
2818199201,18cyclictest0-21swapper/120:38:001
2818199201,17cyclictest0-21swapper/119:50:551
2818199201,11cyclictest31396-21ntpq21:40:221
2818199201,10cyclictest8290-21grep19:35:171
2818199200,6cyclictest0-21swapper/122:40:291
2817399201,6cyclictest0-21swapper/000:33:360
28173992014,3cyclictest15980-21grep23:35:010
28173992012,6cyclictest19204-21kworker/u8:221:15:060
2817399201,11cyclictest5214-21/usr/sbin/munin23:10:140
71912190,2sleep00-21swapper/020:45:230
2818599195,8cyclictest12865-40dnf22:11:382
2818599195,5cyclictest24571-21cstates22:40:132
2818599191,8cyclictest0-21swapper/221:43:062
2818599191,7cyclictest0-21swapper/221:21:432
2818599191,6cyclictest30273-21cat21:40:022
28185991913,3cyclictest29450-21ntp_states19:10:272
28185991913,3cyclictest29450-21ntp_states19:10:272
28185991912,4cyclictest31607-21df19:15:152
28185991912,4cyclictest31607-21df19:15:152
2818199198,3cyclictest86750irq/128-enp1s0-22:40:011
2818199192,8cyclictest14941-21/usr/sbin/munin21:05:301
2818199191,9cyclictest1292-21seq20:32:341
2818199191,7cyclictest0-21swapper/120:47:231
28181991915,3cyclictest0-21swapper/120:52:481
28181991915,2cyclictest830-21crond19:20:011
28181991915,2cyclictest830-21crond19:20:001
2818199191,17cyclictest0-21swapper/121:02:281
2818199191,17cyclictest0-21swapper/119:43:471
2818199191,16cyclictest0-21swapper/119:31:101
2817399198,8cyclictest31893-21gltestperf19:15:170
2817399198,8cyclictest31893-21gltestperf19:15:170
2817399193,13cyclictest18462-21munin-run20:00:010
2817399192,14cyclictest20200-21tail20:00:260
2817399191,9cyclictest14634-21snmpd22:36:300
2817399191,8cyclictest18171-21sh23:40:010
2817399191,8cyclictest18171-21sh23:40:010
2817399191,7cyclictest1266-21sendmail22:45:180
2817399191,6cyclictest0-21swapper/023:15:150
2817399191,3cyclictest0-21swapper/023:20:120
28173991912,4cyclictest6504-21sed00:25:190
2817399191,10cyclictest24371-21/usr/sbin/munin22:40:210
127962191,2sleep014634-21snmpd22:10:420
94192180,2sleep00-21swapper/019:36:590
42052180,2sleep00-21swapper/021:53:230
293962181,13sleep029490-21timerandwakeup21:35:250
2818599183,3cyclictest86550irq/126-enp1s0-22:45:152
2818599182,5cyclictest23719-21memory23:50:212
2818599181,7cyclictest581-21systemd-journal00:30:012
2818599181,7cyclictest30938-21fgrep20:25:252
2818599181,7cyclictest30938-21fgrep20:25:252
2818599181,7cyclictest16195-21sshd19:54:162
2818599181,7cyclictest12730-21/usr/sbin/munin21:00:242
2818599181,7cyclictest0-21swapper/221:45:402
2818599181,6cyclictest0-21swapper/222:52:302
2818599181,3cyclictest352-21memory20:30:192
2818599181,3cyclictest23120-21cpu20:10:152
28185991812,4cyclictest22364-21cat22:35:132
28185991812,3cyclictest1226-21grep19:20:132
2818599181,15cyclictest0-21swapper/222:20:422
2818599181,15cyclictest0-21swapper/220:56:332
2818599181,14cyclictest11514-21sensors00:35:252
2818599181,12cyclictest0-21swapper/221:09:522
2818599181,11cyclictest0-21swapper/200:03:272
2818599181,10cyclictest0-21swapper/223:46:342
2818599180,15cyclictest27176-21taskset20:18:592
2818199185,5cyclictest0-21swapper/121:34:131
28181991814,3cyclictest0-21swapper/120:27:271
28181991814,3cyclictest0-21swapper/120:27:271
2818199181,2cyclictest0-21swapper/120:10:341
28181991812,3cyclictest10767-21expr20:55:141
2817399189,6cyclictest1079-21head20:30:260
2817399182,3cyclictest0-21swapper/021:04:220
2817399182,15cyclictest0-21swapper/023:47:020
2817399181,9cyclictest0-21swapper/022:56:360
2817399181,7cyclictest0-21swapper/021:40:270
2817399181,5cyclictest0-21swapper/021:57:480
2817399181,4cyclictest19975-21/usr/sbin/munin22:30:240
2817399181,4cyclictest19975-21/usr/sbin/munin22:30:240
28173991812,3cyclictest3443-21grep19:25:130
28173991812,3cyclictest31692-21basename20:30:010
28173991812,3cyclictest31692-21basename20:30:010
28173991812,3cyclictest11749-21unixbench_multi00:35:270
28173991811,5cyclictest25433-21kworker/u8:021:29:050
2817399181,14cyclictest25134-21tr23:55:140
2817399180,16cyclictest0-21swapper/019:50:160
156582181,2sleep20-21swapper/223:31:042
152872181,2sleep20-21swapper/222:19:272
2818599179,5cyclictest0-21swapper/221:15:162
2818599176,3cyclictest86850irq/129-enp1s0-23:05:252
2818599175,3cyclictest6829-21sshd23:10:362
2818599173,5cyclictest10748-21grep23:20:232
2818599171,9cyclictest32295-1kworker/u9:022:55:192
2818599171,9cyclictest0-21swapper/221:25:202
2818599171,7cyclictest0-21swapper/223:29:052
2818599171,6cyclictest3327-21grep19:25:132
2818599171,5cyclictest29111-21vmstat20:20:302
2818599171,5cyclictest0-21swapper/200:05:192
28185991712,3cyclictest4479-21irqstats00:20:182
28185991712,3cyclictest21194-21rt-features22:30:252
28185991712,3cyclictest21194-21rt-features22:30:252
28185991712,3cyclictest18575-21cstates23:40:142
28185991712,3cyclictest1-21systemd19:30:092
28185991711,3cyclictest8996-21sed20:50:182
28185991711,3cyclictest4236-21expr20:40:172
28185991711,3cyclictest2996-21sendmail_mailtr20:35:262
28185991711,3cyclictest2376-21/usr/sbin/munin21:50:172
28185991711,3cyclictest18689-21/usr/sbin/munin20:00:212
2818599171,10cyclictest0-21swapper/223:55:142
28185991710,4cyclictest8778-21open_files19:35:232
28185991710,4cyclictest0-21swapper/219:40:172
2817399172,14cyclictest0-21swapper/022:24:220
2817399171,7cyclictest0-21swapper/022:07:110
2817399171,6cyclictest0-21swapper/021:10:430
2817399171,6cyclictest0-21swapper/000:05:260
2817399171,3cyclictest16907-21diskmemload21:31:430
28173991713,3cyclictest0-21swapper/019:40:040
28173991712,3cyclictest5722-21seq20:42:450
2817399171,14cyclictest0-21swapper/020:13:020
2817399171,14cyclictest0-21swapper/019:46:520
28173991711,3cyclictest1248-21expr19:20:130
2817399171,10cyclictest0-21swapper/021:45:250
28173991710,4cyclictest19282-21unixbench_multi22:25:270
28185991612,3cyclictest0-21swapper/219:47:292
28185991611,3cyclictest8291-21latency23:15:192
28185991610,3cyclictest1872-21cut00:15:172
2817399161,6cyclictest14289-21memory22:15:200
2817399161,5cyclictest0-21swapper/000:23:080
28173991612,3cyclictest0-21swapper/021:05:150
28173991611,3cyclictest3221-21grep20:35:280
28173991610,3cyclictest8364-21cpu20:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional