You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 07:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Thu Dec 04, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1449070212998,10sleep10-21swapper/119:06:091
1449189211760,53sleep30-21swapper/319:07:393
144915828659,19sleep00-21swapper/019:07:160
144928627661,9sleep20-21swapper/219:08:542
15787712291,18sleep01578801-21perf21:10:000
1449502992824,2cyclictest0-21swapper/321:12:393
1449502992622,2cyclictest0-21swapper/323:00:263
1449502992522,1cyclictest0-21swapper/321:40:003
1449502992520,3cyclictest0-21swapper/322:20:153
1449502992519,3cyclictest0-21swapper/323:35:203
1449502992421,1cyclictest0-21swapper/322:55:143
1449502992420,2cyclictest0-21swapper/322:25:193
1449502992320,1cyclictest0-21swapper/320:55:133
144950299231,19cyclictest1724634-21taskset22:37:543
144950299231,19cyclictest1724634-21taskset22:37:543
16748942221,11sleep21674900-21ssh22:07:112
144950299222,14cyclictest1594157-21taskset21:18:443
144950299222,13cyclictest1806217-21taskset23:26:493
1449502992219,1cyclictest0-21swapper/323:49:083
1449502992218,2cyclictest0-21swapper/321:50:003
1449502992218,2cyclictest0-21swapper/300:35:263
1449502992218,2cyclictest0-21swapper/300:00:173
144950299221,17cyclictest1644163-21taskset21:50:033
144950299212,13cyclictest1790359-21taskset23:17:473
144950299212,13cyclictest1774425-21taskset23:08:253
144950299212,13cyclictest1674286-21taskset22:06:153
1449502992118,1cyclictest0-21swapper/321:20:193
144950299211,14cyclictest1709507-21sleep22:34:553
144950299201,7cyclictest0-21swapper/323:12:103
1449502992016,2cyclictest0-21swapper/319:22:133
144950299201,14cyclictest1792729-21chrt23:20:153
144950299201,14cyclictest1578791-21turbostat21:10:003
18887932190,10sleep01578785-21diskmemload00:19:140
17895622191,2sleep00-21swapper/023:16:360
144950299192,13cyclictest1830451-21taskset23:41:083
144950299192,13cyclictest1618717-21taskset21:34:003
144950299192,12cyclictest1880298-21taskset00:12:473
144950299192,12cyclictest1667409-21taskset22:03:183
144950299192,11cyclictest1866163-21taskset00:05:093
1449502991916,1cyclictest0-21swapper/300:25:193
1449502991916,1cyclictest0-21swapper/300:25:183
1449502991915,2cyclictest0-21swapper/322:15:253
1449502991914,3cyclictest0-21swapper/300:15:203
1449502991912,4cyclictest0-21swapper/323:30:103
1449502991911,4cyclictest16352082sleep321:44:113
144950299191,14cyclictest0-21swapper/300:30:033
144950299191,10cyclictest1855327-21taskset23:56:233
1449502991910,5cyclictest16573882sleep321:55:333
144950299190,14cyclictest0-21swapper/321:25:093
18728732181,11sleep11578785-21diskmemload00:08:291
18388512181,3sleep10-21swapper/123:46:321
18354252181,12sleep21837166-21tr23:45:182
18224792181,8sleep11822490-21sh23:36:301
18097352181,11sleep01811349-21cut23:30:150
17986382181,11sleep11578785-21diskmemload23:22:471
15851092181,3sleep10-21swapper/121:12:141
144950299182,12cyclictest1734624-21taskset22:45:033
144950299182,12cyclictest1733050-21taskset22:43:033
144950299182,12cyclictest1733050-21taskset22:43:033
144950299182,12cyclictest1683864-21taskset22:13:283
144950299181,14cyclictest1750398-21chrt22:54:013
144950299181,12cyclictest1504792-21chrt20:00:133
144950299181,11cyclictest1557340-21taskset20:45:533
144950299181,10cyclictest1841688-21taskset23:50:063
144950299181,10cyclictest1457535-21taskset19:15:153
144950299180,14cyclictest0-21swapper/300:21:123
19124102171,4sleep21912436-21dump-pmu-power00:35:002
19042922171,4sleep01578785-21diskmemload00:29:170
19042922171,4sleep01578785-21diskmemload00:29:170
18651012171,4sleep11578785-21diskmemload00:03:521
18637762171,3sleep20-21swapper/200:01:492
18556052171,4sleep11578785-21diskmemload23:56:501
18487372171,1sleep10-21swapper/123:53:501
17914272171,4sleep11578785-21diskmemload23:19:261
17236732171,10sleep10-21swapper/122:36:281
17236732171,10sleep10-21swapper/122:36:281
17087632171,3sleep01578785-21diskmemload22:28:590
16661412171,11sleep11666148-21ssh22:01:251
16519282171,3sleep11578785-21diskmemload21:54:471
16188572171,4sleep01578785-21diskmemload21:34:140
16178312171,4sleep11578785-21diskmemload21:32:381
15457792171,11sleep01545825-21ntpq20:35:220
144950299172,12cyclictest14884742sleep319:43:213
1449502991714,1cyclictest0-21swapper/319:27:403
144950299171,10cyclictest1473406-21taskset19:30:143
18143722161,3sleep10-21swapper/123:31:411
17336852161,5sleep1171rcu_preempt22:44:021
17336852161,5sleep1171rcu_preempt22:44:021
17175422160,10sleep01578785-21diskmemload22:34:430
16997382161,9sleep01578785-21diskmemload22:22:440
16987782161,2sleep20-21swapper/222:21:182
16851332161,2sleep10-21swapper/122:15:041
16838332161,4sleep01578785-21diskmemload22:13:260
16685292167,2sleep0171rcu_preempt22:05:000
16579172161,2sleep00-21swapper/021:56:230
144950299167,6cyclictest1514901-21chrt20:08:063
1449502991614,1cyclictest0-21swapper/321:00:173
1449502991612,2cyclictest0-21swapper/320:40:133
1449502991612,2cyclictest0-21swapper/320:10:173
144950299161,10cyclictest1530934-21taskset20:24:493
144950299160,13cyclictest0-21swapper/319:35:203
144950299160,12cyclictest0-21swapper/319:50:113
144950299160,12cyclictest0-21swapper/319:50:113
144950299160,11cyclictest1454654-21taskset19:13:233
144950299160,10cyclictest1525449-21taskset20:17:243
19118942150,2sleep10-21swapper/100:33:591
18632522151,9sleep01578785-21diskmemload00:00:580
18473932151,10sleep21847407-21ssh23:51:422
18257062151,9sleep01825978-21grep23:40:180
18211052151,9sleep21821162-21ntpq23:35:242
17921732151,9sleep21792185-21rm23:20:112
16427982151,2sleep20-21swapper/221:48:192
16270312157,3sleep01578785-21diskmemload21:39:150
15247582151,6sleep01524841-21/usr/sbin/munin20:15:210
14886052151,9sleep11488816-21wc19:45:001
144950299151,9cyclictest1541163-21taskset20:31:103
144950299151,9cyclictest1531196-21taskset20:25:013
1449502991512,1cyclictest0-21swapper/319:55:153
144950299151,10cyclictest228-1kworker/3:1H+kblockd20:53:483
144950299151,10cyclictest228-1kworker/3:1H+kblockd20:53:483
144950299150,12cyclictest0-21swapper/320:35:223
144950299150,12cyclictest0-21swapper/319:45:193
18671492141,8sleep01868748-21tr00:05:190
18559322140,2sleep00-21swapper/023:57:220
18383592141,5sleep00-21swapper/023:45:470
18237822141,8sleep01578785-21diskmemload23:38:290
17892722140,1sleep20-21swapper/223:16:112
17656002142,8sleep01578785-21diskmemload23:02:170
17485572141,7sleep21578785-21diskmemload22:51:092
17425732141,8sleep21742930-21apt-config22:50:012
17263962141,3sleep21726422-21apt-get22:40:102
17263962141,3sleep21726422-21apt-get22:40:102
17161512141,7sleep11578785-21diskmemload22:32:361
16836182141,3sleep11683628-21ssh22:13:071
16025902141,9sleep11602604-21ssh21:24:121
15968702141,9sleep01598491-21kthreadcore21:20:180
15858082140,7sleep014-21ksoftirqd/021:13:210
15675042141,10sleep11567607-21/usr/sbin/munin20:55:261
15520812141,8sleep11552404-21dpkg20:45:001
15458272141,8sleep11545880-21ntp_states20:35:221
15355612141,9sleep11535820-21head20:25:241
14666882141,9sleep01466914-21sed19:20:260
19035352130,7sleep11578785-21diskmemload00:27:571
19035352130,7sleep11578785-21diskmemload00:27:571
19024632131,5sleep20-21swapper/200:26:062
19024632131,5sleep20-21swapper/200:26:062
18811462131,6sleep11578785-21diskmemload00:14:141
18081662131,6sleep21578785-21diskmemload23:29:462
18080062131,6sleep01578785-21diskmemload23:29:310
17832102131,3sleep11783219-21sh23:14:211
17749652131,6sleep11578785-21diskmemload23:09:161
17640552131,7sleep21764068-21sensors_temp23:00:272
17584622131,8sleep21758472-21ssh22:58:592
16934252134,3sleep1331ktimers/122:20:131
16770192131,7sleep21677021-21ssh22:10:082
16436912131,6sleep11578785-21diskmemload21:49:431
16256362131,3sleep11625645-21ssh21:37:041
16196892131,3sleep21619699-21ssh21:35:082
16169192131,3sleep20-21swapper/221:31:152
15949772132,6sleep21595045-21/usr/sbin/munin21:20:002
19192062121,5sleep21578785-21diskmemload00:39:292
18701872120,2sleep21870307-21strings00:05:222
18304382121,5sleep20-21swapper/223:41:082
18132712125,3sleep20-21swapper/223:30:262
18066882121,7sleep11578785-21diskmemload23:27:321
17815282121,7sleep01578785-21diskmemload23:11:470
17586222126,2sleep11578785-21diskmemload22:59:151
17392632121,7sleep01739527-21unixbench_multi22:45:260
17055702121,4sleep21705722-21strings22:25:192
16739622121,6sleep11578785-21diskmemload22:05:471
16341242121,4sleep20-21swapper/221:42:292
16285102121,6sleep11629147-21kthreadcore21:40:161
16083872121,6sleep01578785-21diskmemload21:25:370
15632422121,6sleep01563412-21cstates20:55:130
14743712121,6sleep11476105-21kthreadcore19:30:171
19191002111,5sleep11578785-21diskmemload00:39:161
19187712111,4sleep01578785-21diskmemload00:38:320
18953572111,3sleep20-21swapper/200:22:132
18950032111,5sleep11578785-21diskmemload00:21:361
18948112111,4sleep01578785-21diskmemload00:21:150
18871702111,4sleep11578785-21diskmemload00:16:231
18795462111,4sleep01578785-21diskmemload00:11:290
18794072111,4sleep21578785-21diskmemload00:11:152
18565832111,4sleep21578785-21diskmemload23:58:252
18423242111,6sleep01842897-21kthreadcore23:50:160
18314832111,6sleep11831492-21ssh23:42:431
17817642111,2sleep20-21swapper/223:12:092
17650262111,3sleep10-21swapper/123:01:261
17352262111,3sleep10-21swapper/122:45:141
17327082111,4sleep01578785-21diskmemload22:42:340
17327082111,4sleep01578785-21diskmemload22:42:330
17232842111,4sleep21578785-21diskmemload22:35:532
17232842111,4sleep21578785-21diskmemload22:35:532
17076802111,4sleep11578785-21diskmemload22:27:181
16927372111,4sleep21578785-21diskmemload22:19:382
16692262111,6sleep01669304-21fschecks_time22:05:170
16638522111,4sleep20-21swapper/222:00:172
16585542110,1sleep20-21swapper/221:57:212
16508532111,5sleep01578785-21diskmemload21:53:100
16507842111,3sleep20-21swapper/221:53:042
16413572111,4sleep01578785-21diskmemload21:46:040
16092722111,3sleep20-21swapper/221:26:582
15948222112,3sleep00-21swapper/021:19:460
15938302110,5sleep10-21swapper/121:18:151
15844382111,4sleep21578785-21diskmemload21:11:102
15619202111,5sleep01561971-21ntpq20:50:200
15619202111,5sleep01561971-21ntpq20:50:200
15372182111,6sleep01538859-21kthreadcore20:30:160
15197572111,4sleep21519984-21users20:10:262
15035112112,4sleep01503557-21ntpq19:55:210
14732912111,6sleep21473404-21fschecks_time19:30:142
14523832111,4sleep11453571-21memory19:10:171
19101912101,3sleep01578785-21diskmemload00:30:570
18879162101,2sleep20-21swapper/200:17:442
17974122101,3sleep00-21swapper/023:20:570
17744162101,3sleep20-21swapper/223:08:262
17676672101,3sleep00-21swapper/023:05:100
17582652101,3sleep01578785-21diskmemload22:58:400
17491112101,2sleep10-21swapper/122:52:001
17485782101,3sleep00-21swapper/022:51:110
17184712101,3sleep01578785-21diskmemload22:35:160
17184712101,3sleep01578785-21diskmemload22:35:150
17174762100,4sleep21578785-21diskmemload22:34:372
16913812101,3sleep01578785-21diskmemload22:17:320
16579282101,3sleep11578785-21diskmemload21:56:251
16322302101,2sleep00-21swapper/021:40:230
16095132101,3sleep11578785-21diskmemload21:27:211
16017322101,3sleep21578785-21diskmemload21:22:532
15724862101,4sleep21572532-21ntpq21:00:202
15619732101,4sleep11562019-21ntpq20:50:211
15619732101,4sleep11562019-21ntpq20:50:211
15471572101,4sleep01547336-21diskstats20:40:140
15408502101,4sleep21541094-21users20:30:262
15299732101,4sleep11530020-21ntpq20:20:231
15259412101,4sleep01525960-21apt-get20:20:140
14942032101,4sleep01494455-21df19:50:120
14942032101,4sleep01494455-21df19:50:120
14916972101,3sleep21492763-21memory19:45:182
14888662101,3sleep00-21swapper/019:45:110
1573762291,2sleep10-21swapper/121:05:121
1568760291,4sleep01568848-21tune2fs21:00:140
1519371291,4sleep01519417-21ntpq20:10:200
1515051291,4sleep01-21systemd20:10:000
1494456291,4sleep11494489-21fschecks_count19:50:131
1494456291,4sleep11494489-21fschecks_count19:50:131
1472528290,1sleep10-21swapper/119:27:421
1469016291,4sleep01470737-21cut19:25:150
1557632280,1sleep20-21swapper/220:49:382
1532774281,3sleep01533267-21kthreadcore20:25:160
1530022281,3sleep21525942-21/usr/sbin/munin20:20:242
1524645280,2sleep20-21swapper/220:15:192
1505278280,1sleep20-21swapper/220:00:162
1466244281,2sleep11466357-21ntpq19:20:191
1462940281,3sleep21463625-21kthreadcore19:20:162
1457614281,3sleep01458740-21kthreadcore19:15:170
1573547270,1sleep20-21swapper/221:05:012
1572344270,4sleep10-21swapper/121:00:191
1562771270,4sleep20-21swapper/220:52:402
1562771270,4sleep20-21swapper/220:52:392
1546605270,1sleep20-21swapper/220:38:002
1535119270,3sleep21535212-21ld20:25:172
1510332271,2sleep10-21swapper/120:05:161
1509781271,2sleep1567-21dbus-daemon20:05:001
1503618271,3sleep21503739-21sendmail_mailst19:55:232
1498894270,5sleep20-21swapper/219:50:592
1498894270,5sleep20-21swapper/219:50:582
1483265270,1sleep20-21swapper/219:39:282
1473011271,2sleep01473029-21apt-get19:30:120
1458744270,3sleep11460521-21cut19:15:181
1454585271,2sleep01454590-21systemd19:13:090
1553194261,2sleep00-21swapper/020:45:150
1524554261,2sleep10-21swapper/120:15:181
1519665261,2sleep10-21swapper/120:10:251
1509427260,4sleep00-21swapper/020:00:340
1493229261,2sleep11493233-21mailstats19:45:241
1487446260,2sleep00-21swapper/019:40:180
1483093260,4sleep10-21swapper/119:37:161
1478328261,2sleep00-21swapper/019:35:130
1457333261,2sleep20-21swapper/219:15:152
1454731260,1sleep20-21swapper/219:14:232
1567031250,2sleep20-21swapper/220:55:192
1557358250,3sleep10-21swapper/120:46:091
1547047250,1sleep20-21swapper/220:40:122
1510524250,1sleep20-21swapper/220:05:172
1484863250,2sleep21486589-21cut19:40:162
1471576250,1sleep20-21swapper/219:25:172
1536219240,1sleep10-21swapper/120:30:011
1499739240,1sleep10-21swapper/119:55:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional