You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-14 - 12:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Mar 14, 2026 00:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
312962232162,21sleep00-21swapper/019:06:240
312962232162,21sleep00-21swapper/019:06:240
314662227156,58sleep20-21swapper/219:08:352
314662227156,58sleep20-21swapper/219:08:352
313592202165,24sleep10-21swapper/119:07:131
313592202165,24sleep10-21swapper/119:07:131
312372202164,25sleep30-21swapper/319:05:383
312372202164,25sleep30-21swapper/319:05:383
1882121780,3sleep23167399cyclictest19:50:202
31677991101,107cyclictest0-21swapper/323:51:293
3167799104100,3cyclictest0-21swapper/323:22:393
316779910299,2cyclictest0-21swapper/321:29:173
316779910197,3cyclictest0-21swapper/322:19:373
31677999591,3cyclictest0-21swapper/323:06:073
31677998480,3cyclictest0-21swapper/322:24:183
31677996461,2cyclictest0-21swapper/300:33:463
31677996158,2cyclictest0-21swapper/321:16:013
31677995855,2cyclictest0-21swapper/323:31:463
98792480,4sleep23167399cyclictest23:10:212
31677994845,2cyclictest0-21swapper/300:13:053
320752430,1sleep10-21swapper/100:00:251
31677993713,14cyclictest11169-21cat22:00:183
3167799357,16cyclictest4905-21/usr/sbin/munin00:15:173
31677993514,8cyclictest1452-21latency_hist19:15:013
31677993514,11cyclictest24187-21latency_hist20:05:013
3166199349,12cyclictest9408-21idleruntime-cro20:45:010
3166199347,12cyclictest22615-21awk21:15:000
3167799339,7cyclictest8875-21kworker/3:123:45:203
31677993313,9cyclictest24415-21/usr/sbin/munin20:05:183
31677993310,7cyclictest31003-21/usr/sbin/munin20:20:233
31677993310,7cyclictest31003-21/usr/sbin/munin20:20:233
3167799327,16cyclictest18248-21cat21:05:013
31677993216,6cyclictest8860-21ls23:10:113
31677993215,14cyclictest8285-21sed19:25:253
31677993212,13cyclictest13126-21hddtemp_smartct22:05:143
31677993211,9cyclictest4271-21/usr/sbin/munin23:00:213
31677993211,14cyclictest22553-21switchtime22:25:243
31677993211,12cyclictest9374-21df_inode19:30:123
3167399321,11cyclictest0-21swapper/222:34:362
3167799319,10cyclictest4590-21/usr/sbin/munin19:20:213
31677993113,7cyclictest27437-21/usr/sbin/munin20:10:183
31677993112,6cyclictest4371-21munin-run19:20:013
31677993112,6cyclictest1626-21/usr/sbin/munin20:25:163
31677993110,8cyclictest9201-21proc_pri21:55:213
3167399311,11cyclictest0-21swapper/221:06:502
3167399310,18cyclictest0-21swapper/221:25:362
3167799309,9cyclictest0-21swapper/300:35:143
3167799309,7cyclictest5856-21/usr/sbin/munin21:50:193
3167799308,9cyclictest12186-21cat23:15:223
3167799308,7cyclictest20633-21/usr/sbin/munin21:10:153
31677993013,14cyclictest10627-21needreboot20:45:183
3167399301,9cyclictest0-21swapper/220:35:562
3167399301,10cyclictest0-21swapper/221:54:022
3167399300,17cyclictest0-21swapper/222:16:322
3167799299,12cyclictest28507-21hddtemp_smartct22:40:153
3167799299,11cyclictest0-21swapper/300:00:143
3167799297,10cyclictest30617-21gltestperf22:45:173
3167799297,10cyclictest30617-21gltestperf22:45:173
3167799296,10cyclictest10239-21cat00:25:183
31677992922,4cyclictest15291-21smart_sda20:55:233
31677992914,6cyclictest25372-21diskstats21:20:133
31677992911,6cyclictest30414-21latency_hist00:00:013
3167399291,13cyclictest0-21swapper/222:29:522
3167399291,13cyclictest0-21swapper/221:23:122
3167399290,11cyclictest0-21swapper/222:36:242
3167799289,10cyclictest29419-21/usr/sbin/munin21:30:253
3167799288,11cyclictest86850irq/129-enp1s0-23:25:253
31677992821,4cyclictest921-21ntp_states22:50:213
31677992820,5cyclictest2420-21diskstats22:55:133
31677992818,7cyclictest7458-21/usr/sbin/munin00:20:163
31677992812,3cyclictest8875-21kworker/3:120:30:123
31677992811,6cyclictest18431-21/usr/sbin/munin21:05:233
31677992811,4cyclictest0-21swapper/320:40:193
31677992810,6cyclictest23117-21/usr/sbin/munin23:40:233
3167399281,18cyclictest0-21swapper/221:32:402
3167399281,13cyclictest0-21swapper/220:27:282
3167399281,13cyclictest0-21swapper/200:09:562
3167399280,22cyclictest0-21swapper/222:00:002
3166199285,11cyclictest4384-21idleruntime-cro19:20:010
3166199283,12cyclictest6879-21munin-run00:20:010
31677992720,4cyclictest21068-21/usr/sbin/munin19:55:223
31677992719,5cyclictest5200-21/usr/sbin/munin21:45:263
31677992719,5cyclictest29505-21ntp_states23:55:203
31677992719,5cyclictest12140-21memory19:35:183
31677992718,6cyclictest6290-21/usr/sbin/munin20:35:193
31677992718,6cyclictest393-21/usr/sbin/munin00:05:233
31677992718,6cyclictest28807-21/usr/sbin/munin20:15:163
31677992718,6cyclictest24711-21/usr/sbin/munin22:30:223
31677992718,6cyclictest19653-21/usr/sbin/munin23:35:183
31677992718,6cyclictest14235-21/usr/sbin/munin19:40:163
31677992715,8cyclictest12621-21if_err_enp2s020:50:173
31677992712,7cyclictest25633-21/usr/sbin/munin22:35:173
31677992712,12cyclictest31617-21/usr/sbin/munin21:35:193
31677992712,12cyclictest31617-21/usr/sbin/munin21:35:193
3167399276,2cyclictest0-21swapper/222:45:202
3167399276,2cyclictest0-21swapper/222:45:202
3167399271,12cyclictest0-21swapper/221:12:322
3167399271,12cyclictest0-21swapper/220:55:342
3167399271,12cyclictest0-21swapper/220:11:122
3167399271,10cyclictest0-21swapper/223:41:142
3167399271,10cyclictest0-21swapper/221:03:002
3167399271,10cyclictest0-21swapper/200:27:322
3167399270,9cyclictest0-21swapper/219:41:482
3167399270,2cyclictest0-21swapper/222:24:322
3167399270,11cyclictest0-21swapper/222:56:302
3167399270,11cyclictest0-21swapper/221:41:202
3166199272,12cyclictest1855-21munin-run22:55:010
31677992619,4cyclictest15958-21df19:45:123
31677992618,5cyclictest18171-21/usr/sbin/munin19:50:153
31677992617,6cyclictest2564-21/usr/sbin/munin21:40:213
31677992613,10cyclictest14992-21/usr/sbin/munin22:10:123
3167399261,12cyclictest0-21swapper/220:18:182
3167399261,12cyclictest0-21swapper/219:48:562
3167399261,11cyclictest15322-21hddtemp_smartct22:10:142
3167399261,10cyclictest0-21swapper/221:45:142
3167399261,10cyclictest0-21swapper/220:50:582
3167399260,17cyclictest0-21swapper/222:08:362
3167399260,12cyclictest0-21swapper/219:26:342
3167399260,11cyclictest0-21swapper/220:24:562
3167399260,11cyclictest0-21swapper/220:24:562
3167399260,10cyclictest0-21swapper/219:19:122
3166199260,13cyclictest0-21swapper/000:10:220
3166199260,12cyclictest0-21swapper/021:20:220
61552251,20sleep0812-21snmpd20:35:180
3167399251,9cyclictest0-21swapper/200:21:482
3167399251,12cyclictest0-21swapper/223:27:322
3167399251,11cyclictest0-21swapper/223:48:202
3167399251,11cyclictest0-21swapper/222:01:082
3167399251,10cyclictest0-21swapper/221:37:262
3167399251,10cyclictest0-21swapper/221:37:262
3167399251,10cyclictest0-21swapper/219:59:042
3167399250,12cyclictest0-21swapper/220:48:162
3167399250,11cyclictest0-21swapper/223:37:082
31666992519,3cyclictest7219-21munin-run20:40:011
3166699250,18cyclictest0-21swapper/119:45:181
3166199257,16cyclictest24811-21awk21:20:000
3166199250,2cyclictest0-21swapper/023:53:500
48722240,9sleep121-21rcuc/100:15:011
3167399248,3cyclictest0-21swapper/220:43:562
3167399247,3cyclictest0-21swapper/219:23:442
3167399241,11cyclictest0-21swapper/219:38:542
3167399240,9cyclictest0-21swapper/223:56:182
3167399240,9cyclictest0-21swapper/223:56:182
3167399240,9cyclictest0-21swapper/219:31:482
3167399240,10cyclictest0-21swapper/222:52:562
3167399240,10cyclictest0-21swapper/200:38:322
3166699242,11cyclictest11005-21munin-run19:35:011
3166699241,16cyclictest0-21swapper/122:05:341
3166699240,18cyclictest0-21swapper/100:19:001
3166199248,3cyclictest0-21swapper/019:20:420
3166199241,10cyclictest0-21swapper/023:44:060
3166199241,10cyclictest0-21swapper/019:31:340
3166199240,12cyclictest0-21swapper/000:35:040
3167399238,3cyclictest24194-21munin-run20:05:002
3167399231,7cyclictest0-21swapper/223:05:382
3167399231,3cyclictest0-21swapper/223:23:382
3167399231,10cyclictest0-21swapper/222:40:082
3167399230,10cyclictest0-21swapper/223:30:442
3167399230,10cyclictest0-21swapper/223:16:442
3167399230,10cyclictest0-21swapper/223:00:562
3167399230,10cyclictest0-21swapper/200:34:242
3167399230,10cyclictest0-21swapper/200:01:002
3166699234,18cyclictest0-21swapper/100:36:011
3166699234,17cyclictest0-21swapper/123:11:451
3166699232,17cyclictest16008-21munin-run21:00:011
31666992317,3cyclictest23524-21idleruntime21:15:151
3166699231,5cyclictest0-21swapper/123:44:021
31666992314,3cyclictest0-21swapper/120:16:121
31666992314,2cyclictest18880-21ntpq19:50:211
3166699231,19cyclictest21-21rcuc/100:00:011
3166699231,19cyclictest21-21rcuc/100:00:011
3166199237,3cyclictest0-21swapper/022:29:260
3166199231,10cyclictest0-21swapper/019:48:380
3167399228,3cyclictest0-21swapper/220:06:362
3167399227,10cyclictest0-21swapper/200:16:202
3167399225,3cyclictest0-21swapper/200:12:022
3167399223,3cyclictest0-21swapper/219:13:222
3167399221,9cyclictest0-21swapper/220:30:542
3166699228,12cyclictest10612-21kworker/u8:220:27:051
3166699222,17cyclictest217742sleep119:57:291
3166699221,15cyclictest0-21swapper/121:55:071
3166699221,15cyclictest0-21swapper/119:27:041
3166699221,15cyclictest0-21swapper/119:24:441
3166699221,15cyclictest0-21swapper/119:14:421
3166199228,3cyclictest0-21swapper/023:20:260
3166199227,3cyclictest0-21swapper/019:42:300
31661992215,4cyclictest1642-21if_err_enp2s020:25:160
3167399210,11cyclictest0-21swapper/223:52:162
3166699218,3cyclictest10746-21pmu-power20:45:201
3166699217,12cyclictest26955-21kworker/u8:222:13:051
3166699212,16cyclictest29197-21munin-run21:30:001
3166699212,10cyclictest2722-21smart_sda21:40:231
31666992116,3cyclictest21039-21latency_hist22:25:001
3166699211,5cyclictest0-21swapper/121:49:531
3166699211,2cyclictest7546-21ntp_states23:05:211
3166699211,18cyclictest0-21swapper/122:45:201
3166699211,18cyclictest0-21swapper/122:45:201
31666992110,3cyclictest14493-21ntp_states19:40:201
3166699210,6cyclictest0-21swapper/122:38:161
3166199219,3cyclictest0-21swapper/022:24:340
3166199216,4cyclictest0-21swapper/021:55:430
3166199211,9cyclictest0-21swapper/023:11:070
3166199211,4cyclictest0-21swapper/022:48:350
3166199211,4cyclictest0-21swapper/022:48:350
3166199211,10cyclictest32171-21grep21:35:150
3166199211,10cyclictest32171-21grep21:35:150
3166199210,9cyclictest0-21swapper/000:00:120
24062211,16sleep02491-21ntp_kernel_err21:40:190
3167399201,17cyclictest0-21swapper/221:15:082
3166699209,3cyclictest4271-21/usr/sbin/munin23:00:151
3166699206,2cyclictest0-21swapper/122:00:361
3166699205,4cyclictest10009-21kworker/1:123:25:011
3166699201,3cyclictest0-21swapper/100:05:261
31666992012,5cyclictest3311-21fgrep22:55:231
3166699201,18cyclictest0-21swapper/123:28:301
3166699201,18cyclictest0-21swapper/122:29:581
3166699201,17cyclictest0-21swapper/122:30:421
3166699201,14cyclictest0-21swapper/120:13:151
3166699201,14cyclictest0-21swapper/100:34:571
3166699201,11cyclictest6189-21df21:50:121
3166699201,11cyclictest11116-21cat23:15:101
3166699201,11cyclictest0-21swapper/122:50:251
3166699201,10cyclictest9231-21cat00:25:011
31666992010,3cyclictest18814-21latency_hist22:20:011
3166199202,10cyclictest10249-21/usr/sbin/munin22:00:140
3166199201,14cyclictest0-21swapper/000:30:300
3166199200,8cyclictest0-21swapper/023:58:140
3166199200,8cyclictest0-21swapper/023:58:140
3166699194,4cyclictest0-21swapper/121:23:521
3166699191,9cyclictest11757-21fschecks_time19:35:141
31666991916,2cyclictest0-21swapper/121:10:131
31666991915,3cyclictest0-21swapper/121:35:171
31666991915,3cyclictest0-21swapper/121:35:171
3166699191,2cyclictest0-21swapper/119:16:381
31666991912,2cyclictest0-21swapper/120:05:591
3166699191,17cyclictest0-21swapper/122:43:441
3166699191,17cyclictest0-21swapper/120:20:141
3166699191,17cyclictest0-21swapper/120:20:141
3166699191,16cyclictest0-21swapper/123:45:291
3166699191,16cyclictest0-21swapper/123:30:301
3166699191,16cyclictest0-21swapper/120:32:551
3166699191,10cyclictest19684-21awk21:05:201
3166699190,18cyclictest0-21swapper/123:50:501
3166699190,18cyclictest0-21swapper/120:54:271
3166699190,14cyclictest0-21swapper/120:02:491
3166199199,3cyclictest29419-21/usr/sbin/munin21:30:150
3166199198,3cyclictest1292-21irqstats00:05:160
3166199196,4cyclictest0-21swapper/022:08:410
3166199196,3cyclictest0-21swapper/020:54:100
3166199195,5cyclictest0-21swapper/022:19:310
3166199192,2cyclictest0-21swapper/023:01:120
3166199191,3cyclictest4919-21kworker/u8:121:46:010
3166199191,3cyclictest0-21swapper/021:51:430
3166199191,10cyclictest4044-21latency_hist23:00:010
3166199191,10cyclictest0-21swapper/023:28:400
3166199191,10cyclictest0-21swapper/023:16:570
3166199191,10cyclictest0-21swapper/022:40:200
3166199190,10cyclictest0-21swapper/023:45:140
3166699182,12cyclictest9395-21cpuspeed_turbos00:25:111
31666991815,2cyclictest0-21swapper/121:01:481
3166699181,16cyclictest0-21swapper/123:36:091
3166699181,16cyclictest0-21swapper/121:30:021
3166699181,16cyclictest0-21swapper/120:43:141
3166199186,3cyclictest0-21swapper/000:29:540
3166199185,4cyclictest0-21swapper/023:05:550
3166199185,3cyclictest0-21swapper/019:10:260
3166199184,3cyclictest13208-21perf19:40:010
3166199183,6cyclictest0-21swapper/021:29:240
3166199181,9cyclictest10707-21ntp_states20:45:200
3166199181,9cyclictest0-21swapper/023:31:280
3166199181,8cyclictest0-21swapper/022:37:190
3166199181,7cyclictest7343-21expr00:20:140
3166199181,6cyclictest0-21swapper/023:35:450
31661991813,2cyclictest7956-21grep19:25:220
3166199181,10cyclictest0-21swapper/022:13:400
247012181,2sleep00-21swapper/022:30:230
3166199177,8cyclictest86550irq/126-enp1s0-20:30:120
3166199171,14cyclictest0-21swapper/020:05:010
31661991711,3cyclictest1608-21munin-node20:15:110
3166199171,11cyclictest0-21swapper/020:10:300
3166199161,8cyclictest0-21swapper/020:00:100
3166199161,6cyclictest0-21swapper/020:55:210
31661991612,3cyclictest0-21swapper/021:06:150
3166199161,12cyclictest16228-21/usr/sbin/munin21:00:140
31661991610,3cyclictest21520-21sed19:55:270
3166199151,5cyclictest19080-21smart_sda19:50:240
3166199150,14cyclictest0-21swapper/020:23:130
3166199150,14cyclictest0-21swapper/020:23:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional