You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-01 - 20:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 01, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
218312238163,24sleep20-21swapper/207:06:172
218532202166,23sleep30-21swapper/307:06:343
218292201164,25sleep00-21swapper/007:06:150
218082201165,24sleep10-21swapper/107:06:001
222129910198,2cyclictest0-21swapper/209:55:372
22212996461,2cyclictest0-21swapper/210:00:492
22212996461,2cyclictest0-21swapper/210:00:492
21242600,4sleep339-21rcuc/311:15:153
44222560,3sleep12220799cyclictest12:35:121
279042540,2sleep00-21swapper/008:35:000
267032540,1sleep30-21swapper/308:30:183
22212995147,3cyclictest0-21swapper/210:38:192
6102470,1sleep10-21swapper/111:10:241
2220099451,25cyclictest13311-21unixbench_singl10:25:270
2220099421,32cyclictest26365-21ntp_states07:15:210
2220099401,28cyclictest10517-21/usr/sbin/munin07:55:140
22200993910,13cyclictest24388-21basename07:15:000
2220099382,27cyclictest9562-21/usr/sbin/munin10:20:250
2220099381,27cyclictest8430-21ntp_states10:15:200
2221799372,12cyclictest0-21swapper/309:17:563
2220099372,21cyclictest1665-21cstates11:15:120
2220099368,19cyclictest26526-21/usr/sbin/munin09:45:160
2220099362,29cyclictest2561-40dnf10:02:500
2220099362,29cyclictest2561-40dnf10:02:500
2220099361,26cyclictest16153-21/usr/sbin/munin10:35:160
2220099361,24cyclictest21607-21ntp_states10:45:220
224292350,2sleep30-21swapper/312:00:223
2221799351,2cyclictest0-21swapper/309:10:123
2220099354,20cyclictest87550irq/128-enp1s0-12:05:200
2220099353,23cyclictest27087-21latency_hist11:00:010
2220099346,13cyclictest3973-21munin-run12:35:010
2220099343,22cyclictest0-21swapper/009:15:110
2220099341,22cyclictest0-21swapper/008:10:110
2220099341,21cyclictest31930-21latency_hist12:25:010
2220099341,21cyclictest31930-21latency_hist12:25:010
2220099341,20cyclictest22978-21expr10:50:120
2220099341,20cyclictest12199-21grep09:10:200
2220099334,26cyclictest0-21swapper/011:22:110
2220099332,19cyclictest87550irq/128-enp1s0-09:20:210
2220099331,29cyclictest18966-21/usr/sbin/munin11:55:220
2220099331,20cyclictest14297-21df_inode10:30:120
2220099322,26cyclictest6130-21diskstats11:25:140
2220099321,28cyclictest21157-21/usr/sbin/munin12:00:160
2220099321,22cyclictest4537-21/usr/sbin/munin08:55:160
2220099321,19cyclictest9212-21grep11:30:240
2220099321,17cyclictest87550irq/128-enp1s0-09:05:140
2220099321,17cyclictest6237-21ntp_states10:10:210
2220099311,3cyclictest31546-21/usr/sbin/munin11:10:220
2220099311,27cyclictest6734-21/usr/sbin/munin09:00:170
2220099311,27cyclictest18349-21/usr/sbin/munin10:40:200
2220099311,22cyclictest12954-21perl11:40:140
2220099311,20cyclictest22124-21/usr/sbin/munin09:35:130
2220099311,18cyclictest25574-21/usr/sbin/munin12:10:140
2220099309,18cyclictest8085-21latency_hist07:50:010
2220099302,25cyclictest13629-21memory08:00:180
2220099301,26cyclictest11451-21/usr/sbin/munin11:35:240
2220099300,1cyclictest0-21swapper/007:40:150
2220099299,17cyclictest2286-21fschecks_time07:35:160
2220099295,11cyclictest29739-21munin-run12:20:000
2220099293,23cyclictest1913-21wc12:30:010
2220099293,19cyclictest87550irq/128-enp1s0-08:15:190
2220099292,24cyclictest15567-21uname11:45:200
2220099291,2cyclictest27145-21/usr/sbin/munin11:00:150
2220099291,25cyclictest30921-21/usr/sbin/munin09:55:210
2220099291,25cyclictest25860-21tail09:40:260
2220099291,25cyclictest21273-21timerandwakeup09:30:240
2220099291,25cyclictest0-21swapper/009:50:230
2220099291,16cyclictest10911-21diskmemload11:53:270
22200992911,15cyclictest17731-21/usr/sbin/munin09:25:210
2220099290,20cyclictest0-21swapper/008:50:120
2220099290,20cyclictest0-21swapper/008:50:120
2220099287,18cyclictest30408-21cut08:40:100
2220099287,18cyclictest15535-21grep08:05:160
2220099282,13cyclictest87350irq/126-enp1s0-08:20:130
2220099281,24cyclictest4973-21/usr/sbin/munin12:35:160
2220099281,24cyclictest31322-21munin-run11:10:000
2220099281,24cyclictest0-21swapper/007:20:230
2220099281,23cyclictest30035-21chrt07:25:140
2221799271,15cyclictest0-21swapper/311:03:133
22207992710,7cyclictest23-21ksoftirqd/107:40:111
2220099278,16cyclictest24960-21mailstats08:25:240
2220099277,17cyclictest8387-21cpuspeed_turbos07:50:110
2220099271,23cyclictest2979-21/usr/sbin/munin10:05:120
2220099271,23cyclictest28271-21/usr/sbin/munin08:35:120
2220099271,23cyclictest1194-21/usr/sbin/munin08:45:210
2220099270,24cyclictest31890-21/usr/sbin/munin07:30:170
285762260,2sleep20-21swapper/211:00:262
2221799261,11cyclictest0-21swapper/310:47:273
2221799258,3cyclictest26636-21ntp_states12:10:213
22217992510,3cyclictest10911-21diskmemload09:43:073
22212992518,4cyclictest23049-21tail07:10:162
2220799251,11cyclictest0-21swapper/109:16:271
169652250,4sleep121-21rcuc/111:50:131
2221799249,3cyclictest0-21swapper/311:05:273
2221799246,4cyclictest25302-21users08:25:273
2221799241,11cyclictest0-21swapper/310:21:313
2221799241,10cyclictest10911-21diskmemload09:33:313
2220799241,15cyclictest0-21swapper/112:05:211
2221799239,3cyclictest0-21swapper/310:25:233
2221799238,3cyclictest26547-21sendmail_mailtr07:15:233
2221799237,4cyclictest23709-21df12:05:133
2221799237,4cyclictest0-21swapper/309:02:233
2221799237,3cyclictest4102-21cat07:40:113
2221799236,3cyclictest0-21swapper/307:10:273
2221799231,11cyclictest0-21swapper/310:42:513
22217992310,2cyclictest10911-21diskmemload10:34:073
2220799238,3cyclictest22166-21netstat12:00:181
22207992317,3cyclictest18682-21df10:40:121
2220799231,6cyclictest0-21swapper/111:28:231
22207992316,4cyclictest3531-21/usr/sbin/munin12:30:251
2220799231,2cyclictest87550irq/128-enp1s0-09:42:181
2220799231,19cyclictest22477-21diskstats09:35:131
2220799231,16cyclictest0-21swapper/112:10:061
2220799231,13cyclictest30921-21/usr/sbin/munin09:55:171
2220799231,11cyclictest0-21swapper/111:56:491
2221799228,3cyclictest0-21swapper/310:12:073
2221799227,3cyclictest0-21swapper/310:53:233
2221799226,3cyclictest0-21swapper/308:11:273
2221799222,8cyclictest3698-21munin-run07:40:003
2221799220,15cyclictest0-21swapper/311:51:493
2220799228,3cyclictest29059-21df09:50:131
2220799227,3cyclictest9738-21cat07:50:241
2220799223,10cyclictest0-21swapper/111:42:131
2220799221,5cyclictest1277-21fschecks_time10:00:151
2220799221,5cyclictest1277-21fschecks_time10:00:151
2220799221,4cyclictest16762-21tail10:35:161
2220799221,18cyclictest9868-21sshd11:34:111
2220799221,13cyclictest12098-21df10:25:141
2220799221,13cyclictest0-21swapper/110:49:091
2220799220,6cyclictest0-21swapper/110:13:171
2221799219,3cyclictest0-21swapper/310:00:193
2221799219,3cyclictest0-21swapper/310:00:193
2221799217,3cyclictest32076-21proc_pri09:55:233
2221799215,3cyclictest3884-21memory10:05:193
2221799215,3cyclictest29018-21memory08:35:193
2221799215,2cyclictest0-21swapper/311:20:133
2221799214,3cyclictest1985-21taskset08:48:313
2221799213,4cyclictest0-21swapper/312:35:133
22217992115,3cyclictest3968-21munin-plugin-st12:35:003
2221799211,11cyclictest0-21swapper/311:41:093
2221799210,10cyclictest0-21swapper/310:55:313
2221299216,5cyclictest23685-21grep09:35:252
22207992117,2cyclictest23384-21needreboot07:10:191
2220799211,4cyclictest0-21swapper/111:05:241
2220799211,14cyclictest0-21swapper/112:18:411
22207992110,3cyclictest26908-21latency_hist11:00:011
2220799210,15cyclictest0-21swapper/111:04:071
2221799209,3cyclictest4537-21/usr/sbin/munin08:55:263
2221799205,8cyclictest0-21swapper/311:37:573
2221799205,2cyclictest31026-21ntpq12:20:213
2221799205,2cyclictest31026-21ntpq12:20:213
2221799204,3cyclictest333-21cut07:30:193
2221799203,3cyclictest0-21swapper/311:26:453
2221799201,13cyclictest7462-21expr10:15:113
2221799201,12cyclictest0-21swapper/309:45:163
2221799201,11cyclictest16153-21/usr/sbin/munin10:35:223
2221799201,11cyclictest0-21swapper/311:49:063
2221799201,10cyclictest28680-21load12:15:183
2221299201,10cyclictest8496-40dnf09:02:232
2221299201,10cyclictest25962-21grep07:15:152
2221299201,10cyclictest0-21swapper/209:55:002
2220799209,8cyclictest24257-21unixbench_multi10:50:261
2220799201,7cyclictest11687-21sed11:35:251
2220799201,17cyclictest0-21swapper/108:39:511
2220799201,17cyclictest0-21swapper/107:26:111
2220799201,16cyclictest827-21runrttasks08:03:071
2220799201,16cyclictest5431-21systemd-journal11:23:351
2220799201,14cyclictest0-21swapper/112:25:101
2220799201,10cyclictest0-21swapper/110:20:091
22207992010,2cyclictest22781-21smartctl08:20:241
2220799200,14cyclictest0-21swapper/112:22:501
2220799200,14cyclictest0-21swapper/112:22:501
201412201,2sleep20-21swapper/210:42:572
2221799198,3cyclictest31545-21sshd11:10:103
2221799198,3cyclictest24057-21sshd09:39:363
2221799197,3cyclictest21508-21/usr/sbin/munin08:20:243
2221799197,3cyclictest0-21swapper/308:50:313
2221799197,3cyclictest0-21swapper/308:50:313
2221799195,3cyclictest0-21swapper/308:05:073
2221799193,2cyclictest89250irq/135-enp2s0-11:32:373
2221799192,3cyclictest0-21swapper/307:22:073
2221799191,14cyclictest0-21swapper/308:02:153
2221799191,10cyclictest0-21swapper/309:23:593
2221299194,7cyclictest30-21rcuc/210:30:182
2221299192,14cyclictest8970-21gltestperf07:50:152
2221299191,5cyclictest0-21swapper/212:17:392
2221299191,5cyclictest0-21swapper/209:21:562
22212991913,3cyclictest4934-21idleruntime-cro10:10:002
2221299191,11cyclictest0-21swapper/212:26:382
2220799199,2cyclictest28755-21smart_sda07:20:241
2220799194,2cyclictest0-21swapper/109:01:391
2220799192,8cyclictest14922-21/usr/sbin/munin08:05:241
2220799192,14cyclictest711-21sed08:45:151
22207991915,2cyclictest7889-21fschecks_time10:15:141
2220799191,3cyclictest30296-21/usr/sbin/munin08:40:211
2220799191,17cyclictest0-21swapper/108:59:291
2220799191,16cyclictest0-21swapper/111:45:441
2220799191,16cyclictest0-21swapper/109:34:111
2220799191,16cyclictest0-21swapper/109:07:081
2220799191,16cyclictest0-21swapper/108:30:401
2220799191,16cyclictest0-21swapper/107:38:361
2220799191,10cyclictest17463-21diskstats08:10:121
2220799191,10cyclictest0-21swapper/111:15:071
2220799190,6cyclictest0-21swapper/110:31:421
2220799190,2cyclictest0-21swapper/109:45:141
20772191,14sleep22181-21grep07:35:162
171532191,2sleep10-21swapper/109:20:371
2221799186,3cyclictest0-21swapper/307:29:033
2221799184,4cyclictest0-21swapper/309:26:143
2221799183,3cyclictest0-21swapper/312:25:053
2221799183,2cyclictest89250irq/135-enp2s0-07:54:393
2221799181,9cyclictest808-21snmpd09:07:433
2221799181,3cyclictest0-21swapper/308:42:393
22217991813,3cyclictest11764-21sendmail_mailtr07:55:233
2221799181,10cyclictest0-21swapper/309:54:433
2221299189,6cyclictest0-21swapper/207:45:142
2221299186,3cyclictest87350irq/126-enp1s0-12:05:202
2221299184,5cyclictest30-21rcuc/210:10:242
2221299184,4cyclictest3361-21munin-run11:20:002
2221299183,4cyclictest19845-21wc09:30:012
2221299181,9cyclictest808-21snmpd11:48:282
2221299181,7cyclictest20072-21cpuspeed_turbos09:30:122
2221299181,7cyclictest0-21swapper/210:46:262
2221299181,6cyclictest5892-21cat08:55:252
2221299181,6cyclictest25043-21tr10:55:122
22212991813,3cyclictest23480-21latency_hist08:25:012
22212991813,3cyclictest17738-21/usr/sbin/munin08:10:142
2221299181,15cyclictest0-21swapper/212:30:022
2221299181,15cyclictest0-21swapper/207:31:442
2221299181,10cyclictest0-21swapper/210:28:102
22207991814,3cyclictest0-21swapper/107:15:281
22207991812,3cyclictest2677-21df_inode08:50:121
22207991812,3cyclictest2677-21df_inode08:50:121
22207991812,3cyclictest12103-21users07:55:261
2220799181,16cyclictest0-21swapper/109:10:441
2220799181,16cyclictest0-21swapper/107:48:391
2220799181,10cyclictest25301-21users08:25:261
22207991810,2cyclictest0-21swapper/109:25:231
2220799180,5cyclictest0-21swapper/110:05:211
2221799171,13cyclictest19313-21/usr/sbin/munin08:15:173
2221799170,15cyclictest0-21swapper/311:55:103
2221299178,3cyclictest0-21swapper/208:50:132
2221299178,3cyclictest0-21swapper/208:50:132
2221299178,3cyclictest0-21swapper/208:26:502
2221299178,3cyclictest0-21swapper/207:21:482
2221299176,3cyclictest87350irq/126-enp1s0-08:50:012
2221299175,3cyclictest11366-21cstates09:10:132
2221299172,6cyclictest0-21swapper/212:20:172
2221299172,6cyclictest0-21swapper/212:20:172
2221299172,3cyclictest87650irq/129-enp1s0-07:25:182
2221299171,7cyclictest0-21swapper/212:02:482
2221299171,6cyclictest31559-21smart_sda08:40:222
2221299171,6cyclictest0-21swapper/209:40:132
2221299171,5cyclictest0-21swapper/208:31:082
2221299171,4cyclictest0-21swapper/210:17:372
2221299171,3cyclictest0-21swapper/210:20:072
22212991713,3cyclictest0-21swapper/211:25:162
22212991712,3cyclictest13023-21expr08:00:122
2221299171,15cyclictest0-21swapper/208:05:542
22212991711,3cyclictest4508-21gltestperf07:40:142
22212991711,3cyclictest27685-21latency_hist12:15:012
22212991711,3cyclictest11560-21ntpq07:55:202
2221299171,10cyclictest0-21swapper/211:08:212
22212991710,4cyclictest0-21swapper/211:30:222
2221299170,2cyclictest87550irq/128-enp1s0-11:13:092
2220799171,2cyclictest0-21swapper/107:30:011
2220799171,13cyclictest20901-21users08:15:261
2221299169,4cyclictest26994-21/usr/sbin/munin09:45:162
2221299169,4cyclictest0-21swapper/211:35:132
2221299165,3cyclictest87350irq/126-enp1s0-09:05:242
2221299161,5cyclictest0-21swapper/211:21:342
2221299161,2cyclictest0-21swapper/212:36:542
22212991612,3cyclictest0-21swapper/208:17:082
2221299161,14cyclictest0-21swapper/208:37:062
22212991611,3cyclictest24608-21munin-node11:40:102
22212991611,3cyclictest23560-21irqstats10:50:182
22212991611,3cyclictest20267-21switchtime11:55:252
22212991610,3cyclictest17569-21interrupts11:50:182
22212991610,3cyclictest14315-21needreboot09:15:182
2221799149,3cyclictest6920-21cat07:45:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional