You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-04 - 17:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Jun 04, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
60462239162,25sleep00-21swapper/007:09:160
59932232163,56sleep20-21swapper/207:08:342
58212203165,25sleep30-21swapper/307:06:223
58462201164,25sleep10-21swapper/107:06:401
180592580,1sleep30-21swapper/310:00:193
277712540,4sleep339-21rcuc/311:35:213
99962480,1sleep10-21swapper/108:30:101
238052440,1sleep10-21swapper/107:45:251
234182430,1sleep00-21swapper/009:00:120
6210993812,16cyclictest31982-21ntpq11:45:203
6210993811,18cyclictest19419-21grep10:05:123
6210993720,11cyclictest25922-21sensors_temp07:50:223
6190993711,13cyclictest3074-21munin-run08:15:010
621099369,14cyclictest27484-21/usr/sbin/munin09:10:233
6210993618,4cyclictest9665-21/usr/sbin/munin10:55:163
6210993612,8cyclictest13650-21runrttasks12:19:173
621099358,20cyclictest31960-21cpuspeed_turbos09:20:113
6210993524,6cyclictest8415-40dnf10:51:333
6210993513,5cyclictest808-21snmpd07:57:113
6210993512,14cyclictest5578-21sshd08:20:133
621099347,14cyclictest0-21swapper/311:40:463
6210993414,11cyclictest14997-21idleruntime08:40:153
6210993414,11cyclictest14997-21idleruntime08:40:153
6210993412,13cyclictest7356-21/usr/sbin/munin12:05:233
6210993412,13cyclictest7356-21/usr/sbin/munin12:05:233
619099349,12cyclictest7483-21idleruntime-cro08:25:010
621099339,13cyclictest0-21swapper/309:30:133
6210993315,9cyclictest15238-21df09:55:143
6210993315,15cyclictest27868-21latency_hist10:25:023
6210993314,10cyclictest22232-21latency_hist07:45:023
6210993211,13cyclictest16820-21smartctl12:25:153
6210993211,12cyclictest11327-21cstates11:00:113
6210993211,12cyclictest11327-21cstates11:00:113
6210993210,9cyclictest13724-21netstat09:50:193
6210993210,9cyclictest13724-21netstat09:50:193
6210993210,8cyclictest20043-21perf07:40:013
6210993210,10cyclictest27255-21diskmemload09:15:133
621099318,6cyclictest3384-21ntp_offset10:40:213
621099316,19cyclictest16323-21cat11:10:163
621099316,15cyclictest0-21swapper/311:20:113
6210993113,6cyclictest2242-21/usr/sbin/munin11:50:243
6210993112,6cyclictest7414-21/usr/sbin/munin09:35:243
621099309,9cyclictest14288-21df_inode12:20:143
621099309,14cyclictest0-21swapper/311:25:163
621099308,10cyclictest0-21swapper/311:30:233
621099307,8cyclictest0-21swapper/309:30:003
6210993021,6cyclictest20271-21users08:50:263
6210993012,6cyclictest19614-21/usr/sbin/munin12:30:223
149102300,3sleep29-21rcu_preempt12:20:202
621099297,11cyclictest0-21swapper/308:00:243
621099295,13cyclictest0-21swapper/311:55:153
621099292,10cyclictest0-21swapper/310:27:153
6210992911,15cyclictest14362-21ntp_kernel_err11:05:203
619099291,13cyclictest2725-21munin-run11:55:010
621099287,9cyclictest0-21swapper/308:35:203
621099286,16cyclictest0-21swapper/309:41:523
6210992820,5cyclictest14935-21smart_sda07:25:233
6210992819,5cyclictest5145-21/usr/sbin/munin12:00:273
6210992812,5cyclictest3297-21/usr/sbin/munin08:15:113
6210992810,15cyclictest4523-21/usr/sbin/munin10:45:253
6210992810,10cyclictest23697-21/usr/sbin/munin10:15:173
621099279,15cyclictest17777-21cpu11:15:113
621099278,6cyclictest822-21latency_hist08:10:023
621099278,11cyclictest87350irq/126-enp1s0-09:47:583
621099277,17cyclictest21112-21fschecks_time12:35:173
6210992719,5cyclictest12537-21ntp_states07:20:193
6210992719,5cyclictest10816-21smart_sda12:10:263
6210992718,6cyclictest21949-21ntp_states08:55:203
6210992718,6cyclictest1368-21smart_sda10:35:233
6210992717,7cyclictest16943-21chrt07:30:223
6210992717,7cyclictest16943-21chrt07:30:223
6210992714,9cyclictest32409-21sort10:35:023
619799277,12cyclictest0-21swapper/110:18:381
619799276,12cyclictest0-21swapper/109:38:051
619099271,11cyclictest21289-21awk10:10:000
6210992619,4cyclictest23552-21/usr/sbin/munin07:45:223
6210992619,4cyclictest2014-21memory08:10:193
6210992619,4cyclictest17291-21irqstats08:45:173
6210992618,5cyclictest9103-21irqstats07:15:033
6210992618,5cyclictest22314-21irqstats10:10:173
6210992618,5cyclictest10239-21diskstats08:30:133
6210992617,6cyclictest6391-21/usr/sbin/munin07:10:193
6210992617,6cyclictest23815-21/usr/sbin/munin09:00:153
6197992612,12cyclictest18224-21kworker/u8:207:40:211
619099261,12cyclictest19696-21munin-run11:20:010
107382260,3sleep20-21swapper/209:45:132
6210992518,4cyclictest8038-21/usr/sbin/munin08:25:143
6210992518,4cyclictest25291-21/usr/sbin/munin09:05:113
619099257,16cyclictest13113-21munin-run11:05:010
619099257,16cyclictest13113-21munin-run11:05:010
620399246,7cyclictest0-21swapper/209:40:292
6203992421,2cyclictest0-21swapper/211:54:352
620399240,5cyclictest32-21ksoftirqd/212:15:142
619799241,20cyclictest0-21swapper/111:48:391
619799241,14cyclictest0-21swapper/110:10:181
323062241,3sleep232330-21ntp_states08:05:212
10002240,5sleep2999-21unixbench_multi09:20:252
619799236,3cyclictest0-21swapper/111:10:321
619799233,3cyclictest87550irq/128-enp1s0-09:55:151
619799231,16cyclictest0-21swapper/109:28:471
619099236,4cyclictest23770-21kworker/0:011:34:520
619099231,11cyclictest10912-21idleruntime-cro11:00:010
620399221,6cyclictest0-21swapper/212:01:332
620399221,10cyclictest0-21swapper/211:13:022
619799221,3cyclictest6740-21vmstat12:00:271
619799221,14cyclictest0-21swapper/111:38:101
619799221,11cyclictest0-21swapper/112:16:401
86152211,16sleep20-21swapper/208:25:202
6203992116,3cyclictest18066-21/usr/sbin/munin07:35:132
6203992115,3cyclictest26627-21turbostat.cron07:55:012
619799216,4cyclictest0-21swapper/109:48:551
6197992115,3cyclictest28420-21grep07:55:261
619799211,3cyclictest6501-21munin-run10:50:001
619799211,2cyclictest10143-21seq09:43:261
619799211,18cyclictest0-21swapper/110:42:511
619799211,18cyclictest0-21swapper/110:05:171
619799211,11cyclictest22130-21users12:35:281
619799211,11cyclictest14025-21dnf07:25:151
619799211,11cyclictest0-21swapper/111:25:531
619799211,10cyclictest0-21swapper/112:11:491
619799211,10cyclictest0-21swapper/110:01:511
619099216,4cyclictest159372ls07:30:120
619099216,4cyclictest159372ls07:30:120
619099212,8cyclictest22934-21fschecks_time07:45:140
619099212,15cyclictest30682-21munin-plugin-st11:45:010
85112201,2sleep18613-21missed_timers08:25:181
620399202,9cyclictest0-21swapper/211:08:342
619799204,3cyclictest30688-21awk11:45:011
619799202,15cyclictest0-21swapper/110:22:281
619799202,10cyclictest13548-21timerwakeupswit08:35:251
619799201,7cyclictest0-21swapper/110:26:511
6197992016,3cyclictest0-21swapper/112:25:071
6197992016,3cyclictest0-21swapper/112:09:121
6197992016,3cyclictest0-21swapper/112:09:121
6197992016,3cyclictest0-21swapper/109:01:411
619799201,17cyclictest0-21swapper/109:50:081
619799201,17cyclictest0-21swapper/109:50:081
619799201,17cyclictest0-21swapper/108:40:381
619799201,17cyclictest0-21swapper/108:40:381
619799201,17cyclictest0-21swapper/108:21:281
619799201,17cyclictest0-21swapper/107:54:421
619799201,15cyclictest0-21swapper/108:12:031
6197992011,2cyclictest12738-21smart_sda07:20:231
619799201,11cyclictest9602-21hddtemp_smartct10:55:141
619799201,11cyclictest19815-21pmu-power08:50:211
619799201,11cyclictest16829-21diskstats08:45:121
619799201,11cyclictest13349-21egrep11:05:111
619799201,11cyclictest0-21swapper/112:30:271
619799201,11cyclictest0-21swapper/110:54:041
619799201,10cyclictest0-21swapper/111:15:221
619799200,6cyclictest0-21swapper/109:10:471
619099201,7cyclictest0-21swapper/010:50:320
619099201,16cyclictest20317-21munin-run12:35:010
297962200,1sleep20-21swapper/208:00:202
258522200,1sleep10-21swapper/109:05:151
67392190,2sleep20-21swapper/207:10:152
620399199,3cyclictest12160-21sed07:20:152
620399191,7cyclictest26962-21ntp_states10:20:202
620399191,6cyclictest0-21swapper/211:41:282
620399191,4cyclictest0-21swapper/210:47:222
620399190,6cyclictest0-21swapper/210:50:262
619799197,3cyclictest0-21swapper/110:37:221
6197991915,3cyclictest0-21swapper/107:34:391
6197991915,3cyclictest0-21swapper/107:34:391
619799191,17cyclictest0-21swapper/109:33:361
619799191,16cyclictest0-21swapper/108:00:261
619799191,11cyclictest12052-21missed_timers11:00:171
619799191,11cyclictest12052-21missed_timers11:00:171
619799191,10cyclictest30641-21diskstats10:30:121
619799191,10cyclictest21906-21/usr/sbin/munin08:55:191
619799191,10cyclictest0-21swapper/111:50:281
619799191,10cyclictest0-21swapper/111:30:231
619799191,10cyclictest0-21swapper/109:15:101
619799190,9cyclictest2507-21kworker/u8:011:55:421
619799190,18cyclictest0-21swapper/108:17:091
619799190,17cyclictest0-21swapper/109:20:211
619099199,6cyclictest0-21swapper/008:49:560
619099198,2cyclictest3316-21grep10:40:190
619099192,14cyclictest4550-21cut09:30:160
619099191,4cyclictest0-21swapper/009:43:070
6190991914,3cyclictest31655-21awk09:20:020
6190991912,4cyclictest30709-210anacron08:01:000
619099191,15cyclictest14440-21sed11:05:220
619099191,10cyclictest30910-21/usr/sbin/munin11:45:120
619099191,10cyclictest12090-21/usr/sbin/munin08:35:270
5102191,2sleep00-21swapper/008:06:470
278202190,4sleep011-21rcuc/011:35:230
620399189,3cyclictest20052-21timerandwakeup08:50:242
620399181,9cyclictest909-21sssd_nss11:22:282
620399181,7cyclictest0-21swapper/209:54:152
620399181,7cyclictest0-21swapper/209:54:152
620399181,6cyclictest0-21swapper/211:17:362
620399181,6cyclictest0-21swapper/210:05:552
620399181,5cyclictest0-21swapper/211:57:342
620399181,4cyclictest0-21swapper/210:25:442
620399181,3cyclictest0-21swapper/210:30:062
620399181,3cyclictest0-21swapper/210:11:422
620399181,14cyclictest9930-21cut07:15:152
620399181,11cyclictest0-21swapper/210:39:342
619799181,9cyclictest14627-21smartctl12:20:171
6197991814,3cyclictest0-21swapper/107:39:191
6197991814,3cyclictest0-21swapper/107:17:091
619799181,2cyclictest0-21swapper/108:05:261
619799181,16cyclictest0-21swapper/111:21:181
619799181,16cyclictest0-21swapper/107:12:361
619099185,3cyclictest0-21swapper/011:25:200
619099182,7cyclictest28159-21hddtemp_smartct09:10:160
619099182,13cyclictest19052-21uniq08:50:140
619099181,7cyclictest26882-21grep10:20:180
619099181,6cyclictest29160-21ntp_states10:25:210
619099181,6cyclictest0-21swapper/010:03:470
619099181,15cyclictest0-21swapper/009:50:420
619099181,15cyclictest0-21swapper/009:50:420
619099181,15cyclictest0-21swapper/008:58:130
324792181,2sleep00-21swapper/010:35:040
190802181,3sleep219153-21irqstats12:30:162
620399178,3cyclictest0-21swapper/209:02:222
620399176,3cyclictest87350irq/126-enp1s0-07:30:222
620399176,3cyclictest87350irq/126-enp1s0-07:30:222
620399173,4cyclictest21949-21ntp_states08:55:202
620399172,6cyclictest25284-21/usr/sbin/munin09:05:222
620399171,5cyclictest0-21swapper/211:46:482
620399171,3cyclictest0-21swapper/212:28:532
620399171,15cyclictest0-21swapper/209:14:182
620399171,15cyclictest0-21swapper/208:38:502
620399171,15cyclictest0-21swapper/207:47:222
6203991711,3cyclictest10936-21ntpq08:30:192
6203991711,3cyclictest0-21swapper/209:30:152
620399171,10cyclictest0-21swapper/212:14:512
619099176,3cyclictest5145-21/usr/sbin/munin12:00:260
619099175,4cyclictest27255-21diskmemload09:48:390
619099173,5cyclictest0-21swapper/012:12:000
619099171,9cyclictest808-21snmpd09:07:290
619099171,9cyclictest0-21swapper/011:10:470
619099171,7cyclictest0-21swapper/009:59:050
619099171,6cyclictest18404-21df07:35:100
619099171,6cyclictest0-21swapper/007:25:110
619099171,5cyclictest7704-21diskstats12:05:150
619099171,5cyclictest7704-21diskstats12:05:150
619099171,2cyclictest27255-21diskmemload12:16:140
619099171,2cyclictest0-21swapper/010:14:270
6190991712,3cyclictest3642-21sh08:15:120
6190991712,3cyclictest26159-21unixbench_multi07:50:230
619099171,14cyclictest0-21swapper/009:21:530
619099171,13cyclictest9893-21/usr/sbin/munin08:30:210
619099171,13cyclictest14411-21tune2fs12:20:150
619099171,10cyclictest0-21swapper/010:15:110
619099170,7cyclictest0-21swapper/007:10:120
255442171,3sleep225579-21swap11:30:232
620399165,3cyclictest87350irq/126-enp1s0-11:25:122
620399165,3cyclictest12050irq/123-ahci[0008:45:002
620399165,3cyclictest12050irq/123-ahci[0008:45:002
620399164,4cyclictest3644-21sed10:40:232
620399162,6cyclictest29679-21/usr/sbin/munin09:15:202
620399161,8cyclictest0-21swapper/211:35:462
620399161,6cyclictest5275-21awk08:20:012
620399161,6cyclictest17098-21/usr/sbin/munin10:00:202
620399161,5cyclictest0-21swapper/210:56:332
620399161,3cyclictest0-21swapper/212:06:532
620399161,3cyclictest0-21swapper/212:06:532
620399161,3cyclictest0-21swapper/211:01:472
620399161,3cyclictest0-21swapper/211:01:472
6203991612,3cyclictest0-21swapper/209:35:022
620399161,14cyclictest0-21swapper/208:47:152
620399161,14cyclictest0-21swapper/207:40:422
620399161,14cyclictest0-21swapper/207:26:482
6203991611,3cyclictest2295-21gltestperf09:25:152
6203991611,3cyclictest14902-21/usr/sbin/munin09:55:142
6203991611,2cyclictest21578-21ntpq12:35:232
6203991610,3cyclictest27817-21cat07:55:192
6203991610,3cyclictest24041-21df_inode10:15:132
619099163,2cyclictest87350irq/126-enp1s0-12:26:030
619099161,8cyclictest0-21swapper/010:30:070
619099161,6cyclictest0-21swapper/009:35:320
619099161,4cyclictest0-21swapper/011:55:110
619099161,3cyclictest19923-21/usr/sbin/munin11:20:220
6190991612,3cyclictest0-21swapper/012:36:560
6190991611,3cyclictest20847-21fw_packets07:40:160
619099161,12cyclictest9284-21/usr/sbin/munin07:15:130
619099161,12cyclictest27471-21gltestperf07:55:140
6190991610,3cyclictest5767-21mailstats10:45:230
6190991610,3cyclictest14472-21ls08:40:110
6190991610,3cyclictest14472-21ls08:40:110
619099160,7cyclictest0-21swapper/008:29:080
619099160,7cyclictest0-21swapper/007:20:210
24112161,2sleep00-21swapper/009:25:170
620399151,3cyclictest6561-21ntp_offset08:20:212
6203991510,3cyclictest2005-21cat08:10:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional