You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-10 - 11:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun May 10, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
65132214163,38sleep30-21swapper/319:07:013
65912202166,24sleep20-21swapper/219:08:012
65162202165,24sleep10-21swapper/119:07:031
64462201163,25sleep00-21swapper/019:06:100
2142321800,4sleep3685899cyclictest00:35:133
2142321800,4sleep3685899cyclictest00:35:133
6852997571,3cyclictest0-21swapper/223:39:552
153942600,2sleep10-21swapper/123:05:251
129692520,1sleep30-21swapper/319:20:173
6852995147,3cyclictest0-21swapper/221:06:262
175362480,4sleep230-21rcuc/223:10:242
685299424,6cyclictest25478-40dnf21:02:252
6852993716,18cyclictest10645-21cut20:30:122
6852993611,3cyclictest27496-21/usr/sbin/munin19:55:252
685299351,8cyclictest27286-21awk19:55:012
6852993512,3cyclictest28736-21/usr/sbin/munin22:25:192
6852993510,4cyclictest15229-21grep00:20:162
684799351,12cyclictest0-21swapper/122:55:311
684199358,13cyclictest27284-21perf19:55:010
685299349,4cyclictest0-21swapper/223:20:262
685299342,5cyclictest9071-21ntp_states00:05:212
685299341,24cyclictest0-21swapper/222:43:232
6852993411,3cyclictest30324-21/usr/sbin/munin21:15:192
685299335,3cyclictest87650irq/129-enp1s0-21:10:262
685299333,3cyclictest2179-21fschecks_count20:10:142
6852993316,15cyclictest16286-21awk19:30:002
685299331,5cyclictest15935-21fschecks_count21:55:142
6852993310,3cyclictest827-21runrttasks19:35:112
685299322,12cyclictest2873-21grep23:50:242
6852993211,9cyclictest0-21swapper/219:40:242
6852993210,3cyclictest6515-21latency_hist21:35:022
685299316,6cyclictest0-21swapper/220:55:012
685299312,7cyclictest827-21runrttasks21:22:302
685299311,6cyclictest0-21swapper/222:45:322
685299311,6cyclictest0-21swapper/221:49:432
685299311,14cyclictest0-21swapper/219:14:272
684199313,13cyclictest12199-21perf00:15:010
685299309,4cyclictest11392-21proc_pri00:10:222
685299309,3cyclictest0-21swapper/200:35:012
685299302,6cyclictest3584-21/usr/sbin/munin23:55:222
685299301,6cyclictest22761-21/usr/sbin/munin23:25:252
685299301,6cyclictest22761-21/usr/sbin/munin23:25:252
6852993011,3cyclictest17049-21sed20:45:012
6852993010,3cyclictest10948-21seq19:15:212
685299300,13cyclictest22662-21tr20:55:232
685299293,4cyclictest23787-21hwlatdetect19:45:162
685299292,12cyclictest23736-21users22:10:272
685299291,7cyclictest0-21swapper/223:41:032
685299291,6cyclictest827-21runrttasks23:30:242
684199294,12cyclictest12516-21idleruntime-cro20:35:010
685299289,3cyclictest0-21swapper/220:29:192
685299281,6cyclictest28839-21kworker/u8:023:45:142
685299281,6cyclictest19948-21/usr/sbin/munin22:05:122
6852992811,3cyclictest12413-21/usr/sbin/munin00:15:172
6852992810,3cyclictest3312-21cat21:25:172
684799281,18cyclictest26988-21ntp_states21:05:221
91732270,3sleep230-21rcuc/221:40:142
685299274,3cyclictest0-21swapper/223:00:152
685299272,7cyclictest0-21swapper/200:03:152
685299271,6cyclictest30237-1kworker/u9:022:39:582
685299271,13cyclictest0-21swapper/222:53:122
685299271,13cyclictest0-21swapper/222:15:272
685299271,13cyclictest0-21swapper/222:15:272
685299271,12cyclictest0-21swapper/220:22:592
685299271,11cyclictest0-21swapper/219:31:082
684199273,12cyclictest5918-21awk20:20:010
685299261,2cyclictest0-21swapper/223:08:212
685299261,12cyclictest0-21swapper/220:19:312
685299261,11cyclictest0-21swapper/200:35:112
685299261,11cyclictest0-21swapper/200:35:112
6852992610,3cyclictest31822-21sed22:30:182
6852992610,3cyclictest18657-21memory22:00:192
6852992610,3cyclictest0-21swapper/220:03:272
684799261,17cyclictest22953-21/usr/sbin/munin22:10:191
684199268,15cyclictest10006-21awk00:10:010
684199261,13cyclictest9008-21munin-run19:15:010
684199261,13cyclictest30722-21basename22:30:010
684199260,7cyclictest27278-21if_enp2s022:20:170
685299259,3cyclictest0-21swapper/219:22:272
6852992512,2cyclictest0-21swapper/220:48:132
6852992511,3cyclictest0-21swapper/223:16:552
685299251,12cyclictest0-21swapper/222:23:432
685299251,10cyclictest0-21swapper/222:56:232
685299251,10cyclictest0-21swapper/200:26:472
6852992510,3cyclictest0-21swapper/221:50:072
684799250,18cyclictest87550irq/128-enp1s0-21:45:151
684199252,11cyclictest20687-21awk19:40:010
215682250,4sleep121-21rcuc/100:35:151
215682250,4sleep121-21rcuc/100:35:151
685899244,3cyclictest0-21swapper/322:10:143
685299249,3cyclictest0-21swapper/221:36:192
685299241,12cyclictest0-21swapper/220:36:112
6847992418,3cyclictest8683-21hddtemp_smartct00:05:161
684799241,15cyclictest0-21swapper/121:52:071
684199247,7cyclictest87550irq/128-enp1s0-23:05:150
684199244,18cyclictest3363-21munin-run23:55:010
299612240,2sleep081ktimersoftd/020:00:110
685899236,10cyclictest0-21swapper/322:40:013
685899231,16cyclictest0-21swapper/323:58:193
685899231,15cyclictest0-21swapper/323:14:233
685899231,15cyclictest0-21swapper/320:14:313
685899231,10cyclictest8625-21users19:10:283
684799231,6cyclictest0-21swapper/123:45:141
684799231,13cyclictest28410-21cstates21:10:131
684799231,10cyclictest15419-21irqstats00:20:181
684799230,17cyclictest0-21swapper/100:14:121
684199230,11cyclictest457-21perf22:35:010
280882230,1sleep30-21swapper/321:10:013
685899224,9cyclictest30417-21ntp_states23:40:193
685899221,6cyclictest0-21swapper/322:52:583
685899221,4cyclictest5790-21/usr/sbin/munin00:00:223
685899221,18cyclictest39-21rcuc/322:25:113
685899221,10cyclictest0-21swapper/323:26:103
685899221,10cyclictest0-21swapper/323:26:093
685899221,10cyclictest0-21swapper/320:42:003
684799228,3cyclictest0-21swapper/120:13:251
684799227,3cyclictest25870-21memory23:30:191
684799224,11cyclictest0-21swapper/119:34:011
684799222,3cyclictest773-21NetworkManager21:36:561
684799221,3cyclictest9022-21cpuspeed_turbos21:40:151
684799221,18cyclictest3937-21/usr/sbin/munin20:15:171
684799221,18cyclictest27499-21/usr/sbin/munin23:35:141
684799221,18cyclictest21674-21if_err_enp1s019:40:171
684799221,11cyclictest0-21swapper/122:01:231
684199223,7cyclictest81ktimersoftd/020:50:170
684199223,3cyclictest22680-21sshd19:42:400
6841992211,2cyclictest87350irq/126-enp1s0-00:35:140
6841992211,2cyclictest87350irq/126-enp1s0-00:35:140
684199221,11cyclictest0-21swapper/023:16:580
143932220,5sleep011-21rcuc/019:25:120
685899211,9cyclictest24448-21grep22:15:133
685899211,9cyclictest24448-21grep22:15:133
685899211,8cyclictest0-21swapper/319:16:593
685899211,12cyclictest30934-21/usr/sbin/munin22:30:163
685899211,11cyclictest2560-21cstates21:25:103
685899211,10cyclictest12318-21proc_pri21:45:243
685899211,10cyclictest10934-21basename00:10:153
684799214,3cyclictest24436-21basename21:00:171
684799212,11cyclictest20296-21diskstats22:05:141
684799211,9cyclictest0-21swapper/100:17:231
684799211,8cyclictest0-21swapper/123:25:151
684799211,8cyclictest0-21swapper/123:25:151
6847992114,2cyclictest0-21swapper/121:29:591
684799211,2cyclictest87350irq/126-enp1s0-00:27:021
684799211,18cyclictest0-21swapper/122:41:261
6847992111,8cyclictest0-21swapper/120:35:111
684799211,15cyclictest87550irq/128-enp1s0-21:20:251
684799211,14cyclictest0-21swapper/123:56:591
684799211,14cyclictest0-21swapper/119:54:391
684799211,13cyclictest0-21swapper/123:50:321
684799211,12cyclictest31912-21users21:15:271
684799211,12cyclictest30934-21/usr/sbin/munin22:30:181
684799211,10cyclictest31681-21munin-run20:05:011
6847992110,3cyclictest0-21swapper/122:17:081
6847992110,3cyclictest0-21swapper/122:17:071
684199213,10cyclictest15541-21/usr/sbin/munin21:55:230
684199213,10cyclictest12727-21/usr/sbin/munin20:35:230
684199212,5cyclictest18323-21/usr/sbin/munin22:00:160
684199211,11cyclictest0-21swapper/021:40:210
684199210,13cyclictest17718-21head23:10:250
685899207,7cyclictest87350irq/126-enp1s0-00:05:463
685899207,4cyclictest0-21swapper/300:19:163
685899206,6cyclictest0-21swapper/321:10:163
685899206,3cyclictest0-21swapper/322:03:553
685899202,10cyclictest0-21swapper/323:00:173
685899201,8cyclictest0-21swapper/320:54:323
685899201,5cyclictest0-21swapper/322:07:023
685899201,13cyclictest0-21swapper/322:20:363
685899201,11cyclictest21517-21/usr/sbin/munin20:55:233
685899201,10cyclictest2964-21/usr/sbin/munin22:40:173
685899201,10cyclictest1725-21df23:50:123
685899200,7cyclictest0-21swapper/323:31:593
685899200,14cyclictest0-21swapper/321:57:373
685299206,3cyclictest0-21swapper/220:07:312
684799208,3cyclictest16290-21perf19:30:001
6847992015,3cyclictest8019-21cut22:50:131
684799201,4cyclictest0-21swapper/123:15:171
6847992014,3cyclictest19935-21sendmail_mailqu19:35:231
684799201,2cyclictest8205-21date20:25:011
684799201,14cyclictest0-21swapper/121:56:151
684799201,14cyclictest0-21swapper/121:32:041
684799201,14cyclictest0-21swapper/120:53:471
684799201,14cyclictest0-21swapper/119:57:491
684799201,12cyclictest0-21swapper/122:27:051
684799201,11cyclictest0-21swapper/123:12:161
684799201,11cyclictest0-21swapper/122:47:361
684799201,10cyclictest790-21in:imjournal00:03:161
684799200,15cyclictest0-21swapper/120:59:001
684799200,15cyclictest0-21swapper/120:46:251
684199202,10cyclictest24959-21/usr/sbin/munin23:30:260
684199201,3cyclictest0-21swapper/020:55:240
684199201,11cyclictest5164-21/usr/sbin/munin22:45:210
684199201,11cyclictest22143-21/usr/sbin/munin22:10:170
684199201,11cyclictest13342-21/usr/sbin/munin21:50:140
6841992010,3cyclictest87550irq/128-enp1s0-00:00:220
685899198,3cyclictest10535-21/usr/sbin/munin20:30:223
685899194,2cyclictest0-21swapper/321:43:153
685899191,8cyclictest6517-21sh21:35:013
685899191,7cyclictest0-21swapper/300:21:053
685899191,11cyclictest0-21swapper/322:55:093
685899191,10cyclictest0-21swapper/321:35:243
685899190,6cyclictest0-21swapper/322:47:463
684799198,3cyclictest8915-21perl20:25:151
684799195,3cyclictest87550irq/128-enp1s0-20:05:231
6847991916,2cyclictest0-21swapper/123:21:191
684799191,5cyclictest1355-21head22:35:141
684799191,10cyclictest15924-21/usr/sbin/munin20:40:201
684799191,10cyclictest0-21swapper/100:30:341
684799190,17cyclictest0-21swapper/123:40:191
684199197,3cyclictest87350irq/126-enp1s0-23:37:300
684199197,3cyclictest24780-21ntp_states21:00:220
684199196,6cyclictest87550irq/128-enp1s0-23:40:190
684199193,7cyclictest87550irq/128-enp1s0-21:10:160
684199191,7cyclictest87550irq/128-enp1s0-19:55:230
6841991912,5cyclictest0-21swapper/020:10:260
684199191,12cyclictest0-21swapper/023:01:180
684199191,12cyclictest0-21swapper/020:25:200
684199191,10cyclictest30324-21/usr/sbin/munin21:15:180
684199191,10cyclictest19948-21/usr/sbin/munin22:05:180
684199191,10cyclictest0-21swapper/022:51:110
684199191,10cyclictest0-21swapper/021:36:230
684199190,8cyclictest20669-21cpuspeed_turbos23:20:140
685899186,3cyclictest17150-21df_inode00:25:153
685899186,3cyclictest0-21swapper/321:20:503
685899185,4cyclictest0-21swapper/323:45:113
685899184,4cyclictest0-21swapper/323:05:083
685899184,2cyclictest87350irq/126-enp1s0-21:51:043
685899181,9cyclictest808-21snmpd20:35:193
685899181,9cyclictest808-21snmpd19:50:103
685899181,3cyclictest19515-21irqstats19:35:173
6858991812,4cyclictest0-21swapper/300:30:203
6858991811,4cyclictest0-21swapper/321:16:033
685899181,10cyclictest0-21swapper/323:38:173
685899180,10cyclictest0-21swapper/323:20:263
684799183,3cyclictest12148-21fschecks_count23:00:141
684799181,5cyclictest8286-21sshd19:10:251
684799181,5cyclictest28130-21users22:20:271
6847991812,3cyclictest23435-21df19:45:131
6847991812,3cyclictest10734-21irqstats19:15:181
684799181,16cyclictest0-21swapper/120:30:031
684799181,16cyclictest0-21swapper/119:24:151
684199188,3cyclictest87550irq/128-enp1s0-21:30:160
684199188,2cyclictest6860-21if_enp1s020:20:160
684199186,4cyclictest87350irq/126-enp1s0-00:34:250
684199182,14cyclictest0-21swapper/000:21:550
6841991811,6cyclictest19561-21kworker/u8:120:05:370
684199181,13cyclictest808-21snmpd19:15:030
684199181,11cyclictest0-21swapper/023:56:500
684199181,10cyclictest0-21swapper/022:57:250
685899174,7cyclictest6876-21kworker/u8:223:15:063
685899172,9cyclictest41-21ksoftirqd/319:34:443
685899171,9cyclictest0-21swapper/320:45:183
685899171,9cyclictest0-21swapper/320:20:133
685899171,7cyclictest27496-21/usr/sbin/munin19:55:163
685899171,5cyclictest24977-21smart_sda21:00:243
6858991711,3cyclictest20902-21/usr/sbin/munin19:40:183
6858991711,3cyclictest15051-21cat19:25:173
685899171,10cyclictest0-21swapper/320:15:133
684199177,4cyclictest87350irq/126-enp1s0-21:20:250
684199176,3cyclictest2353-21/usr/sbin/munin21:25:220
684199172,12cyclictest32171-21tail23:45:150
684199171,7cyclictest808-21snmpd20:45:220
684199171,6cyclictest0-21swapper/020:40:160
684199171,4cyclictest23100-21/usr/sbin/munin19:45:120
6841991711,3cyclictest4428-21grep22:40:240
684199171,13cyclictest24341-21/usr/sbin/munin22:15:170
684199171,13cyclictest24341-21/usr/sbin/munin22:15:170
684199171,13cyclictest23292-21fschecks_time23:25:160
684199171,13cyclictest23292-21fschecks_time23:25:160
6841991711,3cyclictest1839-21ntp_states22:35:200
685899161,5cyclictest808-21snmpd20:00:113
6858991611,3cyclictest827-21runrttasks20:29:003
6858991611,3cyclictest32366-21fschecks_count20:05:143
685899161,12cyclictest808-21snmpd19:45:093
684199161,2cyclictest16809-21/usr/sbin/munin00:25:160
6841991611,3cyclictest27950-21cut21:10:010
6841991611,3cyclictest16512-21/usr/sbin/munin19:30:160
684199161,12cyclictest11150-21/usr/sbin/munin21:45:130
684199160,9cyclictest0-21swapper/000:15:260
684199158,1cyclictest0-21swapper/019:20:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional