You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 05:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 01, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26532210465,34sleep30-21swapper/319:06:373
26781210275,9sleep10-21swapper/119:09:491
2655628067,9sleep00-21swapper/019:06:530
2678227969,6sleep20-21swapper/219:09:502
307962650,2sleep220234-21kworker/2:221:17:232
2688799280,28cyclictest0-21swapper/221:37:212
2688599280,0cyclictest0-21swapper/100:25:461
2688599260,3cyclictest0-21swapper/122:57:021
2688599250,2cyclictest0-21swapper/121:30:471
2688599240,1cyclictest0-21swapper/123:07:321
2689099220,1cyclictest0-21swapper/322:58:143
2688799220,1cyclictest0-21swapper/221:49:402
2688599221,17cyclictest0-21swapper/121:40:151
136422220,0sleep30-21swapper/323:30:033
2689099216,7cyclictest224-21systemd-journal23:17:573
2688599213,3cyclictest0-21swapper/122:10:191
2688599211,3cyclictest0-21swapper/122:31:391
26885992020,0cyclictest0-21swapper/121:49:511
2688199208,6cyclictest24399-21dpkg22:49:590
2689099198,6cyclictest224-21systemd-journal22:46:403
2688799190,19cyclictest0-21swapper/222:54:572
2688599190,1cyclictest0-21swapper/100:20:291
2688599190,1cyclictest0-21swapper/100:20:281
26887991816,1cyclictest32-21ksoftirqd/223:10:002
2688799180,17cyclictest0-21swapper/223:23:272
2688799180,17cyclictest0-21swapper/222:10:192
26885991813,2cyclictest1201-21kworker/u8:022:35:071
2688599180,1cyclictest0-21swapper/122:19:511
2688599180,10cyclictest0-21swapper/120:15:181
26890991716,1cyclictest0-21swapper/300:10:103
2688799170,16cyclictest0-21swapper/223:13:522
26885991714,1cyclictest25-21ksoftirqd/121:50:311
2688599170,1cyclictest0-21swapper/121:21:451
2688599170,1cyclictest0-21swapper/121:17:481
26881991716,1cyclictest0-21swapper/023:07:320
12392170,2sleep11242-21latency_hist00:35:001
26890991615,1cyclictest0-21swapper/300:17:593
26890991611,4cyclictest16028-21pmu-power21:30:273
2689099160,1cyclictest0-21swapper/323:00:283
2689099160,1cyclictest0-21swapper/323:00:273
2688799161,1cyclictest141rcu_preempt23:17:442
2688799160,15cyclictest0-21swapper/223:40:162
2688599160,2cyclictest0-21swapper/123:17:371
2688599160,2cyclictest0-21swapper/100:36:091
2688199165,6cyclictest25588-21munin-run23:15:010
2688199160,1cyclictest0-21swapper/021:53:440
26887991513,1cyclictest32-21ksoftirqd/222:33:222
2688799151,1cyclictest10956-21ssh21:26:322
26885991510,3cyclictest32464-21ls20:25:281
26885991510,3cyclictest25842-21cron20:15:001
2688599150,1cyclictest0-21swapper/123:46:241
2688199159,2cyclictest28891-21/usr/sbin/munin22:05:230
26890991410,3cyclictest21242-21grep23:10:283
26887991413,1cyclictest32-21ksoftirqd/222:44:132
2688799140,14cyclictest0-21swapper/200:07:312
2688599149,3cyclictest8228-21tr21:25:161
26885991411,2cyclictest18720-21grep20:00:001
26885991410,2cyclictest25767-21grep22:50:191
2688599140,1cyclictest0-21swapper/123:54:471
2688599140,1cyclictest0-21swapper/123:34:151
2688599140,1cyclictest0-21swapper/100:07:491
2688599140,12cyclictest0-21swapper/122:26:261
2688199145,8cyclictest2972-21perf20:35:000
26881991410,3cyclictest32528-21grep20:25:280
2688199140,1cyclictest0-21swapper/022:30:160
2689099139,3cyclictest11100-21cut22:40:163
2689099134,8cyclictest7744-21snmpd23:24:553
26890991310,2cyclictest29221-21sh19:14:593
26890991310,2cyclictest2546-21tr21:45:203
26890991310,2cyclictest23774-21cut20:10:163
26890991310,2cyclictest10414-21cut19:40:273
2688799139,4cyclictest26875-21cyclictest00:16:242
2688799139,3cyclictest9013-21tr21:50:172
2688799139,3cyclictest29733-21tr19:15:132
2688799139,3cyclictest29733-21tr19:15:132
2688799139,3cyclictest28080-21cut21:40:152
2688799139,3cyclictest17062-21grep21:55:292
2688799138,4cyclictest2584-21basename22:09:592
2688799135,4cyclictest32-21ksoftirqd/219:55:002
26887991311,1cyclictest0-21swapper/222:25:152
26887991310,2cyclictest7382-21grep23:25:162
26887991310,2cyclictest32173-21grep22:55:182
26887991310,2cyclictest28080-21grep00:30:172
26887991310,2cyclictest24530-21date22:50:012
26887991310,2cyclictest20887-21tr23:35:152
26887991310,2cyclictest1562-21grep20:30:202
2688799130,12cyclictest0-21swapper/200:37:122
26885991311,2cyclictest25-21ksoftirqd/123:00:141
26885991311,2cyclictest25-21ksoftirqd/123:00:141
26885991310,2cyclictest26426-21sh21:15:011
26885991310,2cyclictest26426-21sh21:15:001
26885991310,2cyclictest22933-21ssh22:25:001
26885991310,2cyclictest1073-21cat23:45:001
2688599130,1cyclictest0-21swapper/100:01:311
2688199139,3cyclictest4675-21tr19:30:160
2688199134,2cyclictest0-21swapper/020:20:000
2688199131,6cyclictest16125-21munin-run19:55:000
2688199131,11cyclictest30192-21grep19:15:160
2688199131,11cyclictest30192-21grep19:15:150
26881991310,2cyclictest23740-21grep22:25:190
26881991310,2cyclictest23052-21ls22:25:000
26881991310,2cyclictest15699-21grep23:30:280
179922130,0sleep00-21swapper/000:21:460
179922130,0sleep00-21swapper/000:21:450
2689099129,2cyclictest8526-21tr23:50:173
2689099129,2cyclictest39-21ksoftirqd/322:15:013
2689099129,2cyclictest32729-21grep19:20:233
2689099129,2cyclictest3203-21cut22:10:203
2689099129,2cyclictest28599-21cut00:30:193
2689099129,2cyclictest27730-21wc21:40:003
2689099129,2cyclictest20219-21chrt22:22:033
2689099129,2cyclictest19319-21cat21:05:293
2689099129,2cyclictest16092-21tail21:55:203
2689099129,2cyclictest13258-21sed19:45:253
2689099128,3cyclictest5359-21tail22:35:213
2689099128,3cyclictest2708-21chrt21:20:243
2689099128,3cyclictest26877-21sed20:15:213
2689099128,2cyclictest4198-21cat22:35:003
2689099125,3cyclictest224-21systemd-journal00:06:183
2689099124,7cyclictest13420-21fschecks_count23:05:193
2689099122,1cyclictest0-21swapper/323:49:263
2689099122,1cyclictest0-21swapper/319:50:143
2689099121,3cyclictest224-21systemd-journal20:57:483
2689099120,1cyclictest0-21swapper/321:17:013
2689099120,11cyclictest1415-21chrt21:44:403
2688799129,2cyclictest9202-21grep20:45:222
2688799129,2cyclictest4868-21grep20:35:262
2688799129,2cyclictest18078-21grep22:20:252
2688799129,2cyclictest15225-21tr23:55:152
2688799129,2cyclictest14623-21grep23:30:192
2688799129,2cyclictest13046-21tr20:55:142
2688799129,2cyclictest11855-21tr19:45:142
2688799129,2cyclictest0-21swapper/222:35:162
2688799128,3cyclictest22978-21sed20:05:252
2688799121,3cyclictest224-21systemd-journal19:14:552
2688799121,1cyclictest0-21swapper/219:30:162
2688799121,10cyclictest21141-21kworker/u8:022:02:302
26887991210,1cyclictest66942sleep220:40:202
26887991210,1cyclictest2309-21tr19:25:162
2688799120,12cyclictest0-21swapper/223:45:302
2688599129,2cyclictest7920-21head23:25:191
2688599129,2cyclictest7070-21cut19:35:211
2688599129,2cyclictest4179-21awk19:30:011
2688599129,2cyclictest31114-21grep19:15:251
2688599129,2cyclictest31114-21grep19:15:251
2688599129,2cyclictest29234-21grep20:20:221
2688599129,2cyclictest22394-21wc20:05:191
2688599129,2cyclictest22211-21sh22:47:471
2688599129,2cyclictest20874-21grep23:35:151
2688599129,2cyclictest15840-21df21:55:191
2688599129,2cyclictest13417-21cut20:55:161
2688599128,3cyclictest2551-21grep20:30:291
2688599128,2cyclictest5191-21sed19:30:201
2688599128,2cyclictest20245-21wc20:00:221
2688599128,2cyclictest1859-21wc19:25:011
2688599120,10cyclictest0-21swapper/100:16:411
2688599120,0cyclictest0-21swapper/122:00:091
2688199129,2cyclictest7069-21tr19:35:210
2688199129,2cyclictest27931-21tr00:30:150
2688199129,2cyclictest25300-21cut22:50:170
2688199129,2cyclictest22843-21tr00:25:220
2688199129,2cyclictest2152-21cut21:20:200
2688199129,2cyclictest15534-21tail21:30:180
2688199129,2cyclictest10160-21cat20:50:000
2688199128,2cyclictest1155-21basename00:10:100
2688199124,6cyclictest27557-21sensors_temp23:15:250
2688199122,9cyclictest3042-21apt-get22:10:190
26881991211,1cyclictest0-21swapper/021:41:390
2688199120,1cyclictest0-21swapper/000:16:080
2689099118,2cyclictest8227-21cut20:45:163
2689099118,2cyclictest6218-21ssh00:38:343
2689099118,2cyclictest31798-21ssh23:43:233
2689099118,2cyclictest30995-21wget20:25:153
2689099118,2cyclictest29595-21grep19:15:133
2689099118,2cyclictest29595-21grep19:15:123
2689099118,2cyclictest28959-21taskset22:05:153
2689099118,2cyclictest27568-21ssh22:50:593
2689099118,2cyclictest2604-21grep19:25:163
2689099118,2cyclictest26010-21ssh22:02:253
2689099118,2cyclictest2554-21sed20:30:293
2689099118,2cyclictest23692-21taskset00:25:383
2689099118,2cyclictest22130-21head20:05:163
2689099118,2cyclictest21667-21ssh00:00:123
2689099118,2cyclictest21412-21head23:35:173
2689099118,2cyclictest16647-21cut19:55:143
2689099118,2cyclictest16105-21ssh23:55:213
2689099118,2cyclictest15625-21grep00:20:203
2689099118,2cyclictest15625-21grep00:20:203
2689099118,2cyclictest12528-21ls20:55:003
2689099118,2cyclictest11744-21ssh21:51:373
2689099118,2cyclictest10429-21ssh21:25:573
2689099114,6cyclictest21211-21gltestperf21:10:203
2689099114,6cyclictest21211-21gltestperf21:10:203
2689099110,10cyclictest7844-21fw_packets23:25:193
2689099110,10cyclictest29276-21grep20:20:223
2688799118,2cyclictest29231-21head20:20:222
2688799118,2cyclictest24196-21ssh00:01:022
2688799118,2cyclictest20668-21grep20:00:262
2688799118,2cyclictest15396-21fschecks_count21:30:192
2688799118,2cyclictest15139-21tr00:20:182
2688799118,2cyclictest15139-21tr00:20:172
2688799118,2cyclictest10828-21sed19:40:322
2688799116,4cyclictest2157-21fschecks_time21:20:192
2688799110,10cyclictest15724-21sed21:00:182
2688599118,2cyclictest8225-21tr20:45:151
2688599118,2cyclictest5130-21taskset20:37:411
2688599118,2cyclictest31446-21ssh22:06:071
2688599118,2cyclictest2590-21tr00:10:191
2688599118,2cyclictest22411-21uname21:35:231
2688599118,2cyclictest20193-21ssh23:10:191
2688599118,2cyclictest14294-21tr19:50:151
2688599118,2cyclictest10626-21grep20:50:151
2688599112,1cyclictest0-21swapper/119:45:131
2688199118,2cyclictest9093-21cut23:50:200
2688199118,2cyclictest6778-21ssh23:48:570
2688199118,2cyclictest3980-21head20:35:180
2688199118,2cyclictest32608-21ssh00:09:260
2688199118,2cyclictest3103-21ssh23:21:220
2688199118,2cyclictest30624-21sh20:25:000
2688199118,2cyclictest3046-21grep21:45:230
2688199118,2cyclictest27851-21grep19:10:190
2688199118,2cyclictest26894-21basename21:15:130
2688199118,2cyclictest26465-21ssh23:39:390
2688199118,2cyclictest25132-21ssh00:02:000
2688199118,2cyclictest25076-21rm22:01:300
2688199118,2cyclictest23714-21basename20:10:140
2688199118,2cyclictest23204-21ssh21:11:290
2688199118,2cyclictest23204-21ssh21:11:290
2688199118,2cyclictest2184-21grep00:35:160
2688199118,2cyclictest21397-21ssh21:35:170
2688199118,2cyclictest19754-21grep20:00:170
2688199118,2cyclictest19520-21ssh21:57:470
2688199118,2cyclictest18217-21ssh23:56:520
2688199118,2cyclictest16883-21tr19:55:160
2688199118,2cyclictest14369-21sh22:41:560
2688199118,2cyclictest13178-21ssh21:28:510
2688199118,2cyclictest13083-21cut20:55:150
2688199118,2cyclictest12504-21seq20:54:530
2688199118,2cyclictest11898-21sed22:15:310
2688199118,2cyclictest10432-21grep19:40:270
2688199112,8cyclictest1291-21runrttasks22:36:030
2688199110,9cyclictest0-21swapper/021:00:160
2688199110,10cyclictest11423-21ssh23:28:040
2689099108,1cyclictest52002sleep320:38:373
2689099108,1cyclictest23189-21ssh22:25:023
2689099107,2cyclictest19636-21sort20:00:173
2689099107,2cyclictest17387-21apt-config21:05:013
2689099103,1cyclictest6814-21apt-get19:35:193
2688799108,1cyclictest7334-21sed23:00:222
2688799108,1cyclictest7334-21sed23:00:222
2688799108,1cyclictest25745-21chrt20:13:522
2688799108,1cyclictest23370-21ssh21:11:392
2688799108,1cyclictest23370-21ssh21:11:382
2688799108,1cyclictest23332-21cat00:25:272
2688799108,1cyclictest19899-21sh21:09:592
2688799108,1cyclictest1493-21expr00:10:122
2688799108,1cyclictest11353-21ssh23:51:422
2688799107,2cyclictest17700-21cut19:55:202
2688799107,2cyclictest11593-21open_inodes20:50:212
2688799101,7cyclictest566-21munin-plugin-st20:29:592
2688799100,9cyclictest12724-21ssh22:16:192
2688599108,1cyclictest19737-21taskset21:08:161
2688599108,1cyclictest16114-21cut21:00:211
2688599108,1cyclictest14822-21ssh23:55:051
2688599108,1cyclictest12580-21sensors_fan22:40:251
2688599108,1cyclictest11486-21cat19:45:011
2688599107,2cyclictest6377-21grep20:40:181
2688599107,2cyclictest32671-21ssh23:20:111
2688599107,2cyclictest27153-21tr19:10:151
2688199108,2cyclictest11930-21expr19:45:150
2688199108,1cyclictest40022sleep019:28:060
2688199108,1cyclictest27084-21ssh23:40:020
2688199108,1cyclictest22188-21ls20:05:170
2688199107,2cyclictest6586-21cat23:00:170
2688199107,2cyclictest6586-21cat23:00:160
2688199107,2cyclictest5614-21dpkg20:40:140
2688199107,2cyclictest32302-21cat22:55:170
2688199107,2cyclictest32019-21cat19:20:000
2688199100,9cyclictest177322sleep021:05:010
268909997,1cyclictest7210-21unixbench-2d20:40:263
268909996,2cyclictest4742-21expr19:30:173
268879997,1cyclictest281472sleep220:19:042
268879996,2cyclictest7367-21sed19:35:222
268879996,2cyclictest1560-21taskset19:22:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional