You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 12:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat Jul 12, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1268429782,10sleep10-21swapper/119:08:561
1246829582,8sleep30-21swapper/319:06:093
1261129265,8sleep00-21swapper/019:07:590
1252428070,6sleep20-21swapper/219:06:522
275232700,0sleep00-21swapper/019:40:000
154662540,0sleep10-21swapper/121:39:171
12845993332,1cyclictest0-21swapper/123:42:491
12845992927,1cyclictest0-21swapper/121:48:291
12854992726,1cyclictest0-21swapper/323:04:093
12845992726,1cyclictest0-21swapper/123:14:091
12845992523,1cyclictest0-21swapper/123:27:491
1285499242,21cyclictest25056-21latency_hist19:35:003
12845992322,1cyclictest0-21swapper/123:38:291
1285499224,16cyclictest19862-21grep22:50:173
12845992120,1cyclictest0-21swapper/123:30:491
1284099210,1cyclictest0-21swapper/022:37:510
1284099210,1cyclictest0-21swapper/022:37:510
1284099202,14cyclictest20347-21/usr/sbin/munin21:20:250
1284099200,19cyclictest0-21swapper/000:36:580
1284099200,0cyclictest0-21swapper/021:42:110
12854991917,1cyclictest17603-21ssh23:10:443
1285099191,11cyclictest0-21swapper/222:25:122
1284599196,3cyclictest224-21systemd-journal21:20:011
1284599190,2cyclictest0-21swapper/100:27:561
12850991818,0cyclictest0-21swapper/221:48:152
1285099180,18cyclictest0-21swapper/223:12:532
12840991814,3cyclictest13-21ksoftirqd/023:06:470
1285499170,13cyclictest32387-21apache_accesses00:35:133
1285099172,6cyclictest0-21swapper/223:30:002
1285099170,1cyclictest0-21swapper/221:42:562
12840991713,3cyclictest26727-21ssh22:09:540
77192160,0sleep10-21swapper/121:55:321
1285499169,6cyclictest9162-21cron20:10:003
1285499160,15cyclictest0-21swapper/322:55:143
12850991613,3cyclictest0-21swapper/223:30:382
1285099160,16cyclictest0-21swapper/221:13:142
1285099160,12cyclictest0-21swapper/222:30:082
12840991612,3cyclictest25312sleep023:23:400
1284099160,15cyclictest0-21swapper/022:29:320
1285099151,1cyclictest0-21swapper/222:07:412
1284599150,4cyclictest0-21swapper/121:20:211
1284099153,6cyclictest636-21grep21:00:000
12840991515,0cyclictest0-21swapper/021:13:380
112162150,1sleep00-21swapper/021:35:290
1285499141,2cyclictest0-21swapper/323:44:123
1285499140,13cyclictest0-21swapper/321:38:293
12850991414,0cyclictest0-21swapper/220:35:142
12850991412,1cyclictest4548-21ssh00:10:292
12850991412,1cyclictest32-21ksoftirqd/221:37:022
12850991411,2cyclictest37850irq/131-snd_hda22:21:552
12850991410,3cyclictest32047-21sort21:50:222
12850991410,2cyclictest15894-21sh21:16:232
1285099140,14cyclictest0-21swapper/200:21:062
1285099140,14cyclictest0-21swapper/200:21:062
1285099140,0cyclictest0-21swapper/222:02:492
1284599149,3cyclictest20951-21tail19:25:181
12845991410,3cyclictest581-21cron21:00:001
12845991410,3cyclictest20690-21ssh22:50:221
12845991410,2cyclictest8978-21cut23:05:161
12845991410,2cyclictest26663-21grep22:55:151
1284599140,13cyclictest1789-21grep00:35:281
1284599140,13cyclictest0-21swapper/119:45:171
1284099140,1cyclictest0-21swapper/019:40:170
1284099140,13cyclictest27604-21wc23:40:250
1284099140,12cyclictest28449-21tr00:05:140
1285499139,3cyclictest691-21grep19:50:183
12854991310,2cyclictest7374-21grep22:40:303
12854991310,2cyclictest5868-21grep21:10:143
12854991310,2cyclictest27878-21munin-plugin-st00:05:003
12854991310,2cyclictest23024-21sed20:35:273
12854991310,2cyclictest19722-21wc22:05:003
1285499130,11cyclictest23029-21sed00:25:273
1285099138,4cyclictest19768-21head23:35:222
1285099137,2cyclictest17763-21uname20:25:202
1285099133,6cyclictest1549-21idleruntime-cro22:15:002
12850991311,2cyclictest32-21ksoftirqd/200:15:152
12850991310,2cyclictest32-21ksoftirqd/221:25:252
12850991310,2cyclictest23900-21grep20:40:162
12850991310,2cyclictest21129-21sh20:35:002
12850991310,1cyclictest32-21ksoftirqd/223:09:162
1284599139,3cyclictest2552-21sed21:00:281
1284599138,3cyclictest16722-21grep20:25:141
1284599138,3cyclictest12971-21tail20:15:191
1284599134,3cyclictest29159-21turbostat23:20:001
1284599133,4cyclictest1552-21dump-pmu-power22:15:001
1284599132,2cyclictest224-21systemd-journal22:01:121
12845991313,0cyclictest0-21swapper/122:36:571
12845991313,0cyclictest0-21swapper/122:36:571
12845991311,2cyclictest0-21swapper/119:50:151
12845991310,2cyclictest6612-21tail21:10:181
12845991310,2cyclictest4124-21wc23:25:001
12845991310,2cyclictest24433-21sed19:30:261
12845991310,2cyclictest18998-21tr19:20:191
1284099139,3cyclictest7131-21rm22:40:290
1284099139,3cyclictest642-21grep23:45:140
1284099139,2cyclictest6449-21sh21:55:210
1284099139,2cyclictest17999-21cut19:20:130
1284099134,8cyclictest11726-21packagekitd22:47:030
1284099134,1cyclictest13005-21apt-get22:00:150
1284099132,8cyclictest1419-21apache222:34:330
12840991310,2cyclictest7730-21tune2fs20:05:190
12840991310,2cyclictest3859-21cat22:15:300
12840991310,2cyclictest29993-21wc20:50:260
12840991310,2cyclictest2785-21df19:55:190
12840991310,2cyclictest20402-21sed22:50:200
12840991310,2cyclictest15796-21grep00:20:170
12840991310,2cyclictest15796-21grep00:20:170
12840991310,2cyclictest15244-21wc20:20:220
1284099130,3cyclictest0-21swapper/023:50:220
1284099130,1cyclictest0-21swapper/000:13:150
83072120,5sleep30-21swapper/300:15:003
1285499129,2cyclictest8447-21ssh23:05:133
1285499129,2cyclictest31993-21systemd-cgroups00:07:203
1285499129,2cyclictest25451-21sendmail-msp23:40:003
1285499129,2cyclictest22814-21tr23:15:163
1285499129,2cyclictest2077-21wc23:45:253
1285499129,2cyclictest19499-21seq21:19:533
1285499129,2cyclictest18161-21wc21:40:243
1285499129,2cyclictest15904-21grep20:20:293
1285499129,2cyclictest15063-21cut00:20:133
1285499129,2cyclictest15063-21cut00:20:133
1285499128,3cyclictest32033-21head22:35:243
1285499128,3cyclictest32033-21head22:35:233
1285499128,2cyclictest21498-21head22:05:233
1285499127,3cyclictest0-21swapper/322:25:193
1285499122,6cyclictest18338-21unixbench_singl20:25:263
12854991210,1cyclictest21243-21head21:20:213
1285099129,3cyclictest0-21swapper/222:59:512
1285099129,2cyclictest6789-21cron20:05:002
1285099129,2cyclictest23267-21grep23:15:182
1285099129,2cyclictest2045-21basename19:55:002
1285099129,2cyclictest20339-21grep22:50:202
1285099129,2cyclictest14705-21tr23:55:162
1285099129,2cyclictest12103-21grep20:15:142
1285099129,1cyclictest32-21ksoftirqd/200:26:492
1285099128,3cyclictest646-21grep21:00:002
1285099128,3cyclictest3906-21mailstats21:30:232
1285099126,5cyclictest922-21tr21:00:162
12850991211,1cyclictest0-21swapper/222:44:582
1285099120,1cyclictest0-21swapper/200:08:402
1285099120,12cyclictest0-21swapper/223:52:592
1285099120,11cyclictest0-21swapper/200:30:292
1284599129,2cyclictest9047-21tr00:15:171
1284599129,2cyclictest6248-21tail22:40:201
1284599129,2cyclictest4889-21sed21:05:261
1284599129,2cyclictest37412sleep121:53:351
1284599129,2cyclictest29505-21wc20:50:201
1284599129,2cyclictest27790-21grep00:30:191
1284599129,2cyclictest22621-21tr00:00:211
1284599129,2cyclictest1795-21cut23:00:161
1284599129,2cyclictest11144-21grep20:10:271
1284599128,2cyclictest6941-21cat20:05:001
1284599123,6cyclictest9690-21apache223:52:591
1284599121,9cyclictest5393-21diskmemload22:20:211
1284599121,6cyclictest0-21swapper/122:45:141
1284599120,10cyclictest0-21swapper/121:25:211
1284099129,2cyclictest5395-21tail20:00:180
1284099129,2cyclictest4695-21tr23:25:140
1284099129,2cyclictest4507-21tr21:05:210
1284099129,2cyclictest2985-21sed23:00:260
1284099129,2cyclictest29264-21grep22:10:280
1284099129,2cyclictest19830-21apt-config21:20:010
1284099129,2cyclictest1634-21head21:00:190
1284099129,2cyclictest13733-21cut19:10:160
1284099128,2cyclictest22107-21grep20:35:180
1284099122,8cyclictest7744-21snmpd00:25:360
1285499119,1cyclictest2850-21head22:15:213
1285499119,1cyclictest2603-21grep21:00:283
1285499119,1cyclictest13306-21grep19:10:143
1285499118,2cyclictest8296-21ssh23:50:203
1285499118,2cyclictest687-21ssh23:21:553
1285499118,2cyclictest6527-21grep21:55:213
1285499118,2cyclictest6362-21sed20:00:283
1285499118,2cyclictest5098-21grep23:25:163
1285499118,2cyclictest3532-21grep19:55:253
1285499118,2cyclictest32302-21ssh21:50:253
1285499118,2cyclictest30768-21ssh21:26:303
1285499118,2cyclictest27820-21grep20:45:273
1285499118,2cyclictest27820-21grep20:45:273
1285499118,2cyclictest27709-21expr22:10:163
1285499118,2cyclictest26916-21grep19:35:273
1285499118,2cyclictest23670-21ssh21:45:053
1285499118,2cyclictest23628-21ssh22:30:133
1285499118,2cyclictest21002-21head19:25:183
1285499118,2cyclictest19751-21grep20:30:173
1285499118,2cyclictest19626-21seq19:20:453
1285499118,2cyclictest15651-21ntpq23:55:213
1285499118,2cyclictest14225-21cat20:20:003
1285499118,2cyclictest13846-21ssh00:19:153
1285499118,2cyclictest12753-21ssh22:21:573
1285499118,2cyclictest12621-21grep23:30:173
1285499118,2cyclictest10586-21grep20:10:203
1285099118,2cyclictest7449-21ssh21:55:312
1285099118,2cyclictest5043-21cat22:40:002
1285099118,2cyclictest5043-21cat22:40:002
1285099118,2cyclictest4532-21wc21:05:222
1285099118,2cyclictest30344-21ssh23:42:592
1285099118,2cyclictest2859-21grep22:15:212
1285099118,2cyclictest26421-21tr20:45:162
1285099118,2cyclictest26421-21tr20:45:162
1285099118,2cyclictest2394-21ssh23:00:202
1285099118,2cyclictest20892-21expr21:20:192
1285099118,2cyclictest20690-21expr19:25:162
1285099118,2cyclictest1904-21ssh00:35:322
1285099118,2cyclictest17256-21chrt19:16:062
1285099118,2cyclictest12992-21expr22:45:162
1285099117,3cyclictest31679-21taskset23:20:322
1285099117,3cyclictest20125-21sed19:25:012
1285099110,10cyclictest32620-21sh23:45:042
1284599118,2cyclictest9690-21apache222:08:411
1284599118,2cyclictest8687-21sed20:05:281
1284599118,2cyclictest4429-21tail00:10:271
1284599118,2cyclictest4215-21sh22:15:411
1284599118,2cyclictest3820-21ssh23:46:561
1284599118,2cyclictest2399-21cut21:30:141
1284599118,2cyclictest23477-21sh22:30:071
1284599118,2cyclictest23027-21grep20:35:271
1284599118,2cyclictest19810-21sed20:30:171
1284599118,2cyclictest16600-21ssh21:40:091
1284599118,2cyclictest15306-21cat19:15:011
1284599118,2cyclictest15268-21grep00:20:151
1284599118,2cyclictest15268-21grep00:20:151
1284599118,2cyclictest14700-21tr23:55:161
1284599118,2cyclictest14573-21tr20:20:171
1284599116,4cyclictest11728-21gdbus00:06:341
1284599113,4cyclictest224-21systemd-journal19:45:001
1284599111,3cyclictest224-21systemd-journal20:49:251
1284599111,3cyclictest224-21systemd-journal20:49:241
1284099118,2cyclictest8560-21ssh00:15:010
1284099118,2cyclictest5396-21ssh21:31:300
1284099118,2cyclictest32306-21chrt19:50:100
1284099118,2cyclictest30242-21grep19:45:180
1284099118,2cyclictest28955-21ssh22:55:550
1284099118,2cyclictest28625-21ssh23:19:290
1284099118,2cyclictest28357-21ssh21:47:560
1284099118,2cyclictest27908-21ssh21:25:160
1284099118,2cyclictest25406-21ssh00:02:280
1284099118,2cyclictest20492-21ssh23:35:290
1284099118,2cyclictest20078-21ssh23:13:060
1284099118,2cyclictest18003-21ssh23:57:110
1284099118,2cyclictest15586-21ssh22:24:400
1284099118,2cyclictest1364-21ssh21:51:200
1284099118,2cyclictest12671-21ssh23:30:170
1284099118,2cyclictest12592-21grep20:15:170
1284099117,3cyclictest19674-21cut20:30:160
1284099113,4cyclictest224-21systemd-journal20:48:210
1284099113,4cyclictest224-21systemd-journal20:48:210
1285499108,1cyclictest3912-21tune2fs21:05:163
1285499108,1cyclictest31255-21expr20:55:153
1285499108,1cyclictest30108-21wget19:45:173
1285499108,1cyclictest2369-21apt21:30:133
1285499108,1cyclictest12388-21rm22:45:043
1285499107,2cyclictest29244-21ssh00:31:003
1285499107,2cyclictest24274-21expr20:40:173
1285499107,2cyclictest16165-21chrt19:15:193
1285499101,8cyclictest0-21swapper/320:50:143
1285099108,1cyclictest8259-21pmu-power20:05:232
1285099108,1cyclictest30201-21tr19:45:182
1285099108,1cyclictest2894-21grep19:55:202
1285099108,1cyclictest285542sleep220:50:142
1285099108,1cyclictest25403-21cat19:35:152
1285099108,1cyclictest155882sleep220:20:252
1285099107,2cyclictest28707-21cut19:40:242
1285099107,2cyclictest23888-21sh00:01:022
1285099107,2cyclictest23145-21expr19:30:152
1285099107,2cyclictest11538-21seq20:14:182
1284599108,1cyclictest25889-21cut19:35:171
1284599108,1cyclictest16651-21basename22:25:131
1284599107,2cyclictest17587-21munin-plugin-st19:20:001
1284599100,9cyclictest3655-21which19:55:261
1284599100,9cyclictest25703-21taskset20:42:331
1284099108,1cyclictest27175-21grep00:30:160
1284099108,1cyclictest23011-21egrep19:30:140
1284099108,1cyclictest18528-21chrt20:27:080
1284099108,1cyclictest10894-21taskset20:10:250
1284099107,2cyclictest20660-21expr19:25:160
1284099107,2cyclictest16211-21grep19:15:190
128549997,1cyclictest27667-21grep19:40:183
128509998,1cyclictest13134-21hostname19:10:132
128409997,1cyclictest25786-21chrt20:43:350
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional