You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-10 - 06:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot3s.osadl.org (updated Tue Mar 10, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
414143621410,1sleep290-21kswapd022:55:192
3820240211859,54sleep30-21swapper/319:07:553
382009729477,10sleep10-21swapper/119:06:081
382034328360,12sleep20-21swapper/219:09:142
382022727654,6sleep00-21swapper/019:07:430
3820642992919,8cyclictest3882996-21turbostat20:09:553
40840022260,2sleep00-21swapper/022:24:550
3820642992414,7cyclictest41550162sleep323:05:123
382064299222,16cyclictest0-21swapper/323:13:363
382064299222,16cyclictest0-21swapper/322:07:423
382064299221,17cyclictest4098977-21taskset22:31:113
3820642992118,1cyclictest0-21swapper/300:25:233
382064299211,16cyclictest4083277-21taskset22:23:593
382064299211,16cyclictest20948-21taskset23:38:513
382064299203,13cyclictest0-21swapper/323:57:443
3820642992017,1cyclictest0-21swapper/300:20:123
3820642992015,2cyclictest0-21swapper/323:55:003
382064299199,3cyclictest0-21swapper/322:27:503
382064299191,6cyclictest83203-21taskset00:14:023
3820642991911,5cyclictest0-21swapper/322:15:553
382064299191,15cyclictest0-21swapper/300:36:463
382064299191,15cyclictest0-21swapper/300:17:563
382064299191,14cyclictest0-21swapper/323:40:113
382064299191,14cyclictest0-21swapper/321:30:113
382064299191,14cyclictest0-21swapper/321:21:153
382064299191,14cyclictest0-21swapper/319:42:113
3820642991815,2cyclictest0-21swapper/322:48:373
382064299181,14cyclictest0-21swapper/323:46:363
382064299181,14cyclictest0-21swapper/323:30:573
382064299181,14cyclictest0-21swapper/323:27:443
382064299181,14cyclictest0-21swapper/323:21:153
382064299181,14cyclictest0-21swapper/323:15:043
382064299181,14cyclictest0-21swapper/323:02:413
382064299181,14cyclictest0-21swapper/322:50:193
382064299181,14cyclictest0-21swapper/322:40:503
382064299181,14cyclictest0-21swapper/322:35:133
382064299181,14cyclictest0-21swapper/322:12:073
382064299181,14cyclictest0-21swapper/321:55:193
382064299181,14cyclictest0-21swapper/321:40:043
382064299181,14cyclictest0-21swapper/321:27:363
382064299181,14cyclictest0-21swapper/321:10:493
382064299181,14cyclictest0-21swapper/320:48:183
382064299181,14cyclictest0-21swapper/320:35:133
382064299181,14cyclictest0-21swapper/300:30:313
382064299181,13cyclictest0-21swapper/322:55:203
382064299181,12cyclictest0-21swapper/320:10:113
382064299180,15cyclictest0-21swapper/300:09:493
937282171,3sleep20-21swapper/200:20:152
843262171,2sleep00-21swapper/000:15:130
575422171,8sleep157815-21dpkg00:00:021
547072171,3sleep10-21swapper/123:56:331
41284402171,4sleep14128451-21ssh22:50:061
41028512171,11sleep24104081-21grep22:35:162
40471692171,2sleep00-21swapper/022:02:550
39833702171,11sleep23983428-21ntpq21:25:212
382064299173,12cyclictest0-21swapper/321:15:173
3820642991712,4cyclictest0-21swapper/321:35:373
382064299171,13cyclictest0-21swapper/321:50:053
382064299171,13cyclictest0-21swapper/321:50:053
382064299171,13cyclictest0-21swapper/300:00:293
382064299171,12cyclictest0-21swapper/320:20:113
3820642991710,4cyclictest3909639-21taskset20:26:173
382064299170,14cyclictest0-21swapper/321:47:113
1067722171,2sleep00-21swapper/000:25:300
40184592161,11sleep04018525-21ssh21:45:230
38324022161,9sleep23832491-21ntpq19:15:192
382064299161,3cyclictest0-21swapper/320:55:183
382064299161,13cyclictest0-21swapper/320:30:253
382064299160,14cyclictest0-21swapper/321:01:123
382064299160,13cyclictest0-21swapper/322:00:063
382064299160,13cyclictest0-21swapper/319:55:263
382064299160,13cyclictest0-21swapper/319:45:153
1246332161,2sleep20-21swapper/200:35:582
1072822161,4sleep13952542-21diskmemload00:26:121
41882282150,2sleep00-21swapper/023:23:120
40911642151,3sleep00-21swapper/022:27:240
40725442151,9sleep13952542-21diskmemload22:16:511
40271372151,4sleep14027195-21ntpq21:50:221
40271372151,4sleep14027195-21ntpq21:50:211
39785302151,9sleep03978544-21ssh21:24:430
39048842151,2sleep20-21swapper/220:25:132
382064299151,3cyclictest0-21swapper/321:05:123
3820642991512,2cyclictest0-21swapper/319:35:113
382064299151,12cyclictest0-21swapper/320:52:393
382064299150,1cyclictest0-21swapper/319:50:273
382064299150,13cyclictest0-21swapper/319:30:213
382064299150,13cyclictest0-21swapper/319:20:133
1162392154,2sleep13952542-21diskmemload00:31:241
912592141,4sleep291274-21ssh00:18:062
820472140,9sleep23952542-21diskmemload00:12:292
59812141,8sleep27632-21cut23:30:162
41184232146,2sleep20-21swapper/222:43:462
40977572141,7sleep04097821-21ntpq22:30:210
40906242141,3sleep23952542-21diskmemload22:26:422
40229322141,8sleep04022950-21grep21:50:130
40229322141,8sleep04022950-21grep21:50:120
39665502140,4sleep10-21swapper/121:15:301
39195962145,5sleep13919646-21ntpq20:35:221
382064299148,3cyclictest38440232sleep319:27:373
382064299148,3cyclictest0-21swapper/319:15:113
382064299140,2cyclictest0-21swapper/320:00:123
382064299140,12cyclictest0-21swapper/320:40:173
382064299140,12cyclictest0-21swapper/320:15:173
834942131,7sleep13952542-21diskmemload00:14:251
625052131,7sleep23952542-21diskmemload00:00:242
470602131,3sleep13952542-21diskmemload23:52:561
41725312131,7sleep14172799-21df_inode23:15:121
41697792130,1sleep00-21swapper/023:11:540
41604062130,2sleep13952542-21diskmemload23:06:021
41592792131,6sleep04159335-21ntpq23:05:210
41543562131,9sleep04154534-21apt-config23:05:010
41025592131,7sleep04102771-21hddtemp_smartct22:35:150
40837652131,3sleep14083768-21ssh22:24:371
40134552131,3sleep04013466-21ssh21:44:350
40103942131,7sleep13952542-21diskmemload21:40:311
39745452131,6sleep13974598-21ntpq21:20:191
38764202131,8sleep03876467-21ntpq19:55:210
38432422131,6sleep13843296-21ntpq19:25:201
382064299131,8cyclictest3820963-21taskset19:10:023
370492131,7sleep13952542-21diskmemload23:46:151
282112131,8sleep028224-21rm23:41:280
836272121,5sleep00-21swapper/000:14:360
719022121,5sleep20-21swapper/200:05:402
600512121,7sleep061779-21kthreadcore00:00:170
489062121,7sleep048917-21ssh23:55:030
41701602121,3sleep24170171-21ssh23:12:252
41460202122,5sleep14146042-21apt-get23:00:131
41170932121,3sleep14117103-21ssh22:42:001
40938432121,6sleep14094065-21hddtemp_smartct22:30:141
40654502121,6sleep13952542-21diskmemload22:14:031
40637452121,6sleep23952542-21diskmemload22:11:472
40392332121,4sleep20-21swapper/221:59:002
39966192121,6sleep13952542-21diskmemload21:35:061
39863422121,7sleep13986353-21ssh21:28:281
39354772121,5sleep03935565-21ntpq20:50:200
38474282121,7sleep13848654-21cat19:30:171
32572121,7sleep23270-21ssh23:28:322
189512121,6sleep13952542-21diskmemload23:36:121
133712121,7sleep013382-21ssh23:35:030
1166342120,2sleep0116648-21ssh00:31:560
1148312121,6sleep2114890-21ntpq00:30:212
982072110,4sleep10-21swapper/100:20:461
668122111,6sleep13952542-21diskmemload00:05:111
493172111,4sleep23952542-21diskmemload23:55:132
461412111,5sleep03952542-21diskmemload23:51:440
41858052111,6sleep24185861-21cat23:20:262
41720772111,5sleep13952542-21diskmemload23:14:561
41615352111,5sleep23952542-21diskmemload23:07:312
41433522111,6sleep03952542-21diskmemload22:57:030
41422982111,5sleep13952542-21diskmemload22:55:391
41286832111,5sleep04128820-21diskstats22:50:130
41268892111,4sleep23952542-21diskmemload22:48:242
41063122111,5sleep14106411-21ntpq22:35:201
40988602111,3sleep20-21swapper/222:31:032
40799442111,5sleep24079996-21ntpq22:20:222
40665692111,4sleep23952542-21diskmemload22:15:122
40571932111,5sleep13952542-21diskmemload22:09:411
40377982111,3sleep10-21swapper/121:57:071
40195432111,4sleep13952542-21diskmemload21:46:041
40147572111,6sleep24015105-21kthreadcore21:45:172
40117792111,5sleep23952542-21diskmemload21:42:202
40017892111,6sleep24001803-21ssh21:35:442
39919402111,5sleep23992098-21ntpq21:30:182
39780482110,5sleep23952542-21diskmemload21:24:042
39673392111,7sleep03967354-21ssh21:16:330
39581782111,4sleep13952542-21diskmemload21:10:571
39146462111,5sleep23914890-21users20:30:262
38654922111,5sleep23865549-21ntp_states19:45:212
36542111,6sleep13952542-21diskmemload23:29:041
27812110,1sleep00-21swapper/023:27:550
276602111,4sleep13952542-21diskmemload23:40:451
129722111,2sleep00-21swapper/023:34:510
1257662111,5sleep03952542-21diskmemload00:37:290
1018902111,5sleep2102104-21diskstats00:25:132
986022101,3sleep00-21swapper/000:21:170
916012101,5sleep191613-21sh00:18:351
742632101,3sleep00-21swapper/000:08:460
41784582101,2sleep00-21swapper/023:16:510
41514272101,3sleep23952542-21diskmemload23:01:102
41284522101,2sleep20-21swapper/222:50:072
41259072101,3sleep10-21swapper/122:47:061
41193542101,2sleep0171rcu_preempt22:45:000
40864732101,5sleep14088161-21tr22:25:171
40642262100,2sleep00-21swapper/022:12:250
40571542101,2sleep20-21swapper/222:09:382
40543172101,3sleep00-21swapper/022:05:480
40470862101,4sleep23952542-21diskmemload22:02:492
40452942101,2sleep10-21swapper/122:00:271
40372832101,3sleep00-21swapper/021:56:260
40305052101,4sleep23952542-21diskmemload21:53:592
40305052101,4sleep23952542-21diskmemload21:53:592
40029152101,3sleep00-21swapper/021:37:130
39910152101,2sleep10-21swapper/121:30:171
39847442101,3sleep00-21swapper/021:26:240
39616002100,6sleep20-21swapper/221:15:102
39526052101,2sleep20-21swapper/221:10:022
39267352101,5sleep23927491-21cut20:45:182
39195492101,5sleep03919589-21ntpq20:35:220
39088392100,2sleep10-21swapper/120:25:201
38784982101,5sleep13878982-21grep20:00:151
38618512101,4sleep03861922-21irqstats19:45:160
38503692101,4sleep03850578-21hddtemp_smartct19:35:130
38263802101,4sleep13826792-21df19:15:021
368052101,2sleep20-21swapper/223:45:552
317902101,3sleep00-21swapper/023:45:130
276202101,2sleep20-21swapper/223:40:422
200732101,3sleep20-21swapper/223:37:432
1264812101,5sleep13952542-21diskmemload00:38:261
121252101,5sleep112128-21ssh23:33:441
47678291,2sleep20-21swapper/223:53:452
4182931291,4sleep14184619-21tr23:20:171
4177420291,2sleep20-21swapper/223:15:292
4119751291,4sleep04119765-21ssh22:45:120
4071145291,3sleep03952542-21diskmemload22:15:260
3987841291,2sleep00-21swapper/021:30:080
3958510291,2sleep00-21swapper/021:11:240
3947462290,4sleep23947481-21apt-get21:05:122
3942154291,4sleep13942412-21df_inode21:00:101
3903485291,4sleep13903602-21ntpq20:20:191
3883819291,4sleep13885198-21cat20:05:161
3883244291,3sleep03883263-21apt-get20:05:110
3882012291,3sleep23882062-21ntpq20:00:202
3855730291,4sleep13855745-21/usr/sbin/munin19:40:111
3839125291,4sleep03839144-21apt-get19:25:110
3834290291,4sleep03834359-21/usr/sbin/munin19:20:150
3833806291,4sleep13834004-21ls19:20:111
3826104291,4sleep03826379-21hddtemp_smartct19:15:010
3952529281,3sleep10-21swapper/121:10:001
3952171280,2sleep00-21swapper/021:05:300
3942133281,2sleep00-21swapper/021:00:100
3923867281,4sleep03924771-21sort20:40:180
3909844280,1sleep00-21swapper/020:28:510
3900103280,4sleep23901000-21kthreadcore20:20:172
3899122280,1sleep00-21swapper/020:18:170
3889144281,3sleep03890196-21kthreadcore20:10:170
3868052281,3sleep23869801-21kthreadcore19:50:182
3867261281,3sleep13868051-21grep19:50:171
3845723281,3sleep03847427-21grep19:30:160
3838867281,3sleep23839107-21latency_hist19:25:012
3825832280,1sleep00-21swapper/019:11:320
3946292271,3sleep23946371-21pmu-power21:00:192
3941568270,1sleep20-21swapper/220:56:272
3941553271,3sleep00-21swapper/020:56:170
3936550270,1sleep10-21swapper/120:54:451
3936409270,1sleep20-21swapper/220:52:582
3925615270,1sleep10-21swapper/120:41:151
3920850270,2sleep20-21swapper/220:40:122
3910188270,1sleep10-21swapper/120:30:051
3898899270,1sleep20-21swapper/220:15:312
3893927270,3sleep13893648-21apt-get20:15:021
3893477270,1sleep20-21swapper/220:12:582
3872302270,1sleep20-21swapper/219:55:042
3860480270,1sleep20-21swapper/219:40:442
3855163270,3sleep23855265-21touch19:40:002
3850079270,2sleep10-21swapper/119:35:101
3849625270,1sleep20-21swapper/219:32:312
3825984270,1sleep10-21swapper/119:13:301
3824047270,3sleep20-21swapper/219:10:182
3941843260,3sleep10-21swapper/120:59:541
3920418260,4sleep20-21swapper/220:38:072
3910767261,2sleep00-21swapper/020:30:160
3902772261,2sleep00-21swapper/020:20:180
3890199260,3sleep13891994-21cut20:10:171
3885199261,2sleep20-21swapper/220:05:172
3877924260,2sleep00-21swapper/020:00:020
3872283260,1sleep10-21swapper/119:55:021
3871438260,4sleep00-21swapper/019:50:440
3866231260,4sleep10-21swapper/119:47:241
3860508260,4sleep00-21swapper/019:41:060
3844042260,1sleep20-21swapper/219:27:532
3931241250,3sleep10-21swapper/120:49:561
3930560250,2sleep03930781-21/usr/sbin/munin20:45:270
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional