You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-30 - 19:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sat Aug 30, 2025 12:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
114620021980,2sleep1331ktimers/110:10:161
97589421750,1sleep2411rcuc/207:45:162
9327212148103,16sleep30-21swapper/307:05:063
9371212146102,35sleep20-21swapper/207:06:462
9372762144100,35sleep00-21swapper/007:08:440
9327242140102,12sleep10-21swapper/107:05:081
11656942420,2sleep20-21swapper/210:22:082
937504993632,2cyclictest1319137-21gltestperf12:10:183
937504992820,5cyclictest1256455-21find11:25:243
937504992723,3cyclictest733-21gdbus11:25:013
937504992722,3cyclictest1237504-21sh11:15:003
937504992721,3cyclictest736-21gdbus12:35:073
937504992621,3cyclictest1202574-21idleruntime-cro10:50:003
937504992621,3cyclictest1105156-21ps09:40:173
937504992617,8cyclictest1351178-21gpgv12:30:003
937504992617,6cyclictest1081540-21ls09:25:013
93750499261,20cyclictest1123626-21runrttasks09:54:543
937504992520,3cyclictest970040-40dconf12:04:323
937504992513,8cyclictest1290323-21tr11:50:173
937504992419,3cyclictest1286320-21unixbench-2d11:45:263
937504992418,3cyclictest0-21swapper/309:35:013
937504992413,7cyclictest1060516-21cat09:10:013
937504992319,2cyclictest1045914-21gltestperf08:55:163
93750499231,8cyclictest1178769-21ssh10:30:203
937504992318,3cyclictest0-21swapper/311:20:003
93750499231,19cyclictest1054351-21ntpq09:00:203
93750499225,5cyclictest1301065-21sh11:56:133
937504992218,3cyclictest1266513-21idleruntime-cro11:35:003
937504992218,2cyclictest0-21swapper/308:40:003
93750499221,18cyclictest1169512-21kthreadcore10:25:183
93750499221,18cyclictest1135508-21ntp_states10:00:223
93750499221,18cyclictest1135508-21ntp_states10:00:223
937504992211,7cyclictest1125123-21tr09:55:203
937504992211,7cyclictest1066848-21sh09:14:043
93750499214,15cyclictest1308836-21diskmemload12:02:163
93750499212,15cyclictest1281625-21awk11:45:003
937504992117,2cyclictest1074261-21sendmail09:20:003
937504992116,3cyclictest709-21polkitd08:25:013
937504992114,5cyclictest1116644-21cron09:50:003
937504992112,8cyclictest0-21swapper/312:30:243
93750499211,18cyclictest984239-21runrttasks07:50:213
937504992111,8cyclictest1190329-21cat10:40:173
11743402217,9sleep01174416-21systemd-run10:30:010
93750499201,7cyclictest1317690-21ssh12:08:433
93750499201,7cyclictest1035839-21tune2fs08:45:163
937504992017,2cyclictest1101867-21ssh09:38:073
937504992015,3cyclictest0-21swapper/307:20:153
937504992014,3cyclictest709-21polkitd07:40:063
937504992014,3cyclictest1216763-21cpuspeed_turbos11:00:163
93750499201,17cyclictest1158305-21ssh10:16:373
93750499201,16cyclictest1228403-21taskset11:05:583
93750499201,14cyclictest1145486-21munin-run10:10:003
93750499199,6cyclictest1329313-21tr12:15:193
93750499194,12cyclictest1186386-21sh10:35:383
937504991915,2cyclictest1149621-21strings10:10:203
93750499191,16cyclictest942538-21munin-setwarn07:15:013
937504991911,6cyclictest1208600-21taskset10:52:413
93750499191,16cyclictest1010209-21runrttasks08:19:303
937504991911,6cyclictest0-21swapper/310:55:203
93750499191,15cyclictest1020302-21awk08:30:013
13325312190,2sleep10-21swapper/112:17:361
93750499189,6cyclictest1161014-21cut10:20:173
93750499188,7cyclictest995301-21basename08:05:003
937504991814,2cyclictest0-21swapper/311:35:143
937504991814,2cyclictest0-21swapper/308:55:003
937504991810,6cyclictest986526-21cut07:55:173
93750499180,5cyclictest954726-21/usr/sbin/munin07:25:253
13560982181,4sleep01356196-21ls12:30:310
937504991713,2cyclictest0-21swapper/308:15:003
937504991712,3cyclictest1087293-21sh09:27:413
937504991712,2cyclictest0-21swapper/308:05:163
93750499171,12cyclictest1334968-21grep12:20:163
12785652170,2sleep10-21swapper/111:40:251
12430032171,3sleep00-21swapper/011:16:540
10735552171,4sleep21060350-21diskmemload09:18:392
10159312171,4sleep21016282-21cut08:25:172
93750499167,6cyclictest944274-21grep07:15:023
937504991612,2cyclictest1024415-21ntpq08:30:223
93750499161,11cyclictest943-21runrttasks07:30:123
11500722160,2sleep00-21swapper/010:10:280
93750499159,4cyclictest262-21systemd-journal08:45:003
93750499151,12cyclictest978984-21cut07:45:193
13656362151,2sleep20-21swapper/212:36:382
12652792151,10sleep21265293-21ssh11:32:482
11882352151,9sleep00-21swapper/010:39:060
11156412151,9sleep00-21swapper/009:48:000
13159112141,7sleep01060350-21diskmemload12:05:570
12216312141,9sleep01221643-21rm11:01:390
10942442141,9sleep11094252-21ssh09:32:251
9992832130,8sleep10-21swapper/108:05:241
13176472131,8sleep21317654-21ssh12:08:402
12869372131,8sleep11286950-21sh11:46:221
12144742131,7sleep21060350-21diskmemload10:56:242
12082402131,8sleep10-21swapper/110:52:021
11937362130,8sleep20-21swapper/210:40:412
11884912131,6sleep11060350-21diskmemload10:39:361
11869732131,7sleep21060350-21diskmemload10:36:442
9804832122,5sleep2980502-21apt-get07:50:142
13653312120,8sleep01365343-21ssh12:36:120
13230062121,7sleep01323225-21unixbench-2d12:10:300
12793222120,8sleep21279332-21ssh11:41:022
12019952120,8sleep11202001-21ssh10:48:541
11230702120,8sleep11123079-21ssh09:53:461
11031962121,6sleep21060350-21diskmemload09:40:122
10555772122,5sleep21055596-21apt-get09:05:132
9493672110,7sleep0949368-21sshd07:19:540
13580672111,7sleep21358082-21ssh12:32:412
13500102111,3sleep00-21swapper/012:28:340
13497372111,3sleep20-21swapper/212:28:122
13366232111,6sleep11338353-21kthreadcore12:20:211
13238252111,3sleep10-21swapper/112:11:191
12799612110,1sleep00-21swapper/011:42:080
12722682111,3sleep10-21swapper/111:36:401
12511392111,5sleep21060350-21diskmemload11:23:302
12452342110,7sleep11245462-21basename11:20:161
12437122111,3sleep20-21swapper/211:18:132
12303312111,3sleep00-21swapper/011:09:410
12294312111,7sleep11229442-21sh11:08:061
12205182110,7sleep21220565-21ntpq11:00:252
12067592111,5sleep01206812-21ntpq10:50:250
11587042111,6sleep01060350-21diskmemload10:17:220
11449562111,3sleep10-21swapper/110:09:011
11293942111,3sleep10-21swapper/109:57:061
11158032110,7sleep21115812-21ssh09:48:182
10960342112,4sleep11096053-21apt-get09:35:141
10883032111,7sleep11088315-21sh09:29:451
10878892110,7sleep01087893-21sh09:28:560
10793112110,1sleep10-21swapper/109:20:551
10459822111,6sleep11046492-21cut08:55:181
9792802101,4sleep0979336-21ntp_states07:45:230
13648322101,3sleep10-21swapper/112:35:321
13425442101,4sleep21060350-21diskmemload12:24:432
13401642101,4sleep01060350-21diskmemload12:21:320
13324472101,2sleep20-21swapper/212:17:292
13242172101,4sleep21060350-21diskmemload12:11:562
13174692101,3sleep10-21swapper/112:08:221
13101152101,4sleep11060350-21diskmemload12:04:101
13081702101,4sleep01060350-21diskmemload12:01:120
13014642101,2sleep10-21swapper/111:56:571
13007152100,3sleep20-21swapper/211:55:292
12957552101,5sleep01060350-21diskmemload11:54:550
12938792101,3sleep20-21swapper/211:50:482
12866922101,4sleep21060350-21diskmemload11:45:552
12739852100,5sleep01266520-21perf11:39:550
12736162101,3sleep20-21swapper/211:39:052
12580382105,2sleep10-21swapper/111:27:381
12522042101,4sleep01060350-21diskmemload11:25:040
12426412101,4sleep11060350-21diskmemload11:15:591
12365452101,2sleep00-21swapper/011:13:140
12308742101,3sleep20-21swapper/211:10:112
12204742101,5sleep11220516-21ntpq11:00:241
11739222101,2sleep10-21swapper/110:29:171
11722472101,4sleep21060350-21diskmemload10:25:502
11653172100,3sleep10-21swapper/110:21:281
11651832101,3sleep00-21swapper/010:21:110
11590062101,4sleep11060350-21diskmemload10:17:551
11458432101,4sleep21145867-21apt-get10:10:142
11368192100,7sleep01136832-21sh10:01:540
11368192100,7sleep01136832-21sh10:01:540
11364662101,3sleep10-21swapper/110:01:111
11364662101,3sleep10-21swapper/110:01:111
11287822101,4sleep01060350-21diskmemload09:55:530
11216052100,3sleep00-21swapper/009:50:360
11170032101,4sleep21117025-21apt-get09:50:142
11165182101,3sleep10-21swapper/109:49:451
11026542100,3sleep20-21swapper/209:39:442
10936432101,4sleep01060350-21diskmemload09:31:160
10807612101,3sleep20-21swapper/209:23:522
10731652101,3sleep10-21swapper/109:17:341
10662652101,2sleep20-21swapper/209:12:302
10657932101,2sleep00-21swapper/009:11:360
10544352100,6sleep11054550-21/usr/sbin/munin09:00:231
959459291,4sleep0959959-21gpgv07:30:000
959412291,3sleep165-21kcompactd007:29:291
948378291,4sleep1948428-21ntpq07:15:221
1356891290,3sleep10-21swapper/112:31:071
1348104291,2sleep10-21swapper/112:25:581
1333338291,2sleep00-21swapper/012:18:400
1308682291,4sleep21308694-21ssh12:02:022
1301012291,3sleep01060350-21diskmemload11:56:070
1286758291,2sleep00-21swapper/011:46:020
1265153291,2sleep00-21swapper/011:32:340
1250186291,2sleep00-21swapper/011:21:400
1235558291,2sleep10-21swapper/111:10:381
1229443290,3sleep20-21swapper/211:08:072
1214378290,2sleep10-21swapper/110:56:111
1214244290,1sleep00-21swapper/010:55:340
1209557291,4sleep22021027-21cups-browsed10:55:002
1202321291,3sleep20-21swapper/210:49:352
1201241290,6sleep01201248-21ssh10:47:200
1180325291,3sleep11060350-21diskmemload10:32:341
1179789291,3sleep01060350-21diskmemload10:31:250
1145206291,3sleep21060350-21diskmemload10:09:312
1141243291,4sleep01142525-21memory10:05:190
1135347291,4sleep21135384-21ntpq10:00:202
1135347291,4sleep21135384-21ntpq10:00:202
1109067290,2sleep10-21swapper/109:43:401
1107120291,4sleep01107175-21ntpq09:40:220
1088684290,3sleep20-21swapper/209:30:102
1087444291,2sleep20-21swapper/209:27:572
1073580291,3sleep01060350-21diskmemload09:18:410
1065258291,4sleep11060350-21diskmemload09:10:331
1035168290,2sleep20-21swapper/208:43:082
1020548291,3sleep21020691-21df08:30:142
1004283291,3sleep11004333-21ntpq08:10:211
999337281,4sleep2999385-21open_files08:05:252
979605281,3sleep1979849-21users07:45:281
964523283,3sleep1331ktimers/107:32:141
948430280,5sleep20-21swapper/207:15:232
1290991281,3sleep11292728-21kthreadcore11:50:191
1259714281,2sleep11060350-21diskmemload11:30:141
1258239281,3sleep21060350-21diskmemload11:28:032
1193807281,3sleep01060350-21diskmemload10:40:470
1178892281,4sleep21178947-21sed10:30:232
1158329280,3sleep21060350-21diskmemload10:16:402
1129661280,2sleep20-21swapper/209:57:402
1079450281,2sleep00-21swapper/009:21:100
1055756281,4sleep01055966-21hddtemp_smartct09:05:150
1054995280,1sleep00-21swapper/009:00:540
1054987280,1sleep20-21swapper/209:00:482
1026043280,5sleep11027488-21kthreadcore08:35:181
1005684281,3sleep21005764-21fschecks_time08:15:152
989301271,3sleep10-21swapper/107:55:221
985498271,2sleep00-21swapper/007:55:130
970094270,2sleep10-21swapper/107:39:391
964393270,1sleep20-21swapper/207:30:342
949663270,2sleep20-21swapper/207:20:122
938083271,3sleep10-21swapper/107:10:171
1194151271,2sleep11194158-21ssh10:41:321
1100744270,2sleep00-21swapper/009:35:490
1045150270,1sleep10-21swapper/108:53:211
1040544270,1sleep00-21swapper/008:50:080
1040267270,1sleep20-21swapper/208:49:462
1030322271,2sleep21030354-21perf08:40:002
1015675270,4sleep01015698-21fschecks_count08:25:150
1010537271,2sleep00-21swapper/008:20:140
1009237270,1sleep00-21swapper/008:15:210
990999261,2sleep20-21swapper/208:00:162
988376261,2sleep20-21swapper/207:55:192
975245261,1sleep10-21swapper/107:45:001
974280260,2sleep00-21swapper/007:40:220
970425260,2sleep20-21swapper/207:40:122
970093260,2sleep00-21swapper/007:39:380
965277260,1sleep20-21swapper/207:35:112
954725260,2sleep00-21swapper/007:25:110
942112260,1sleep00-21swapper/007:11:130
938229260,3sleep2939547-21kthreadcore07:10:182
1059288260,1sleep10-21swapper/109:05:201
1046494261,2sleep20-21swapper/208:55:192
1029976260,1sleep00-21swapper/008:35:370
1020526260,2sleep10-21swapper/108:30:131
1002670261,2sleep21004109-21load08:10:182
1000524260,1sleep00-21swapper/008:10:120
1000089260,1sleep00-21swapper/008:07:480
990717250,1sleep00-21swapper/008:00:140
957129250,1sleep20-21swapper/207:25:172
949854250,1sleep10-21swapper/107:20:151
1049139250,1sleep00-21swapper/008:55:210
1044356250,1sleep20-21swapper/208:50:222
1038910250,2sleep11039240-21strings08:45:201
1019943251,2sleep10-21swapper/108:25:561
1010559250,1sleep10-21swapper/108:20:151
1005537250,1sleep10-21swapper/108:15:141
990528240,1sleep10-21swapper/108:00:071
980825240,1sleep10-21swapper/107:50:171
980661240,1sleep00-21swapper/007:50:160
949830240,1sleep00-21swapper/007:20:140
1035601240,1sleep00-21swapper/008:45:150
1030779240,1sleep10-21swapper/108:40:151
1030630240,1sleep00-21swapper/008:40:140
1020731240,1sleep00-21swapper/008:30:160
1010706240,1sleep20-21swapper/208:20:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional