You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-01-17 - 05:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Jan 17, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14199210187,9sleep30-21swapper/319:05:543
1429029885,9sleep10-21swapper/119:07:041
1431629469,9sleep00-21swapper/019:07:230
1450428565,16sleep20-21swapper/219:09:512
1461599270,27cyclictest0-21swapper/221:56:552
14611992318,4cyclictest27392-21latency_hist23:55:011
14611992316,6cyclictest5820-21diskmemload00:12:171
1461199231,11cyclictest0-21swapper/120:55:131
14615992221,1cyclictest0-21swapper/200:36:342
1460799220,3cyclictest0-21swapper/021:21:500
14618992120,1cyclictest0-21swapper/322:04:183
1461599210,20cyclictest22379-21kworker/2:221:48:492
14611992119,1cyclictest5820-21diskmemload00:22:421
1461599200,20cyclictest0-21swapper/222:10:422
14611992019,1cyclictest0-21swapper/123:38:491
14611992017,2cyclictest0-21swapper/119:20:111
1461199200,1cyclictest0-21swapper/123:23:141
1461199200,1cyclictest0-21swapper/100:34:221
14607992018,2cyclictest0-21swapper/023:10:020
14607992018,2cyclictest0-21swapper/023:05:020
14618991918,1cyclictest0-21swapper/300:14:383
1461199199,6cyclictest0-21swapper/123:56:431
14611991917,1cyclictest5820-21diskmemload22:46:191
1460799190,1cyclictest0-21swapper/021:40:140
14618991817,1cyclictest25976-21ssh21:50:183
1461599183,9cyclictest0-21swapper/200:30:092
1461199189,6cyclictest0-21swapper/100:37:171
1461899170,15cyclictest0-21swapper/322:30:263
14615991716,1cyclictest0-21swapper/222:49:222
1461599170,17cyclictest0-21swapper/223:18:512
1461199177,9cyclictest0-21swapper/123:15:121
1461199176,8cyclictest0-21swapper/121:28:591
1461199170,3cyclictest0-21swapper/121:15:111
1461599169,2cyclictest19640-21wc21:20:012
14615991616,0cyclictest0-21swapper/200:01:342
1461599160,16cyclictest0-21swapper/222:20:592
1461199169,6cyclictest0-21swapper/122:50:431
1461199168,5cyclictest0-21swapper/100:15:161
1461199167,5cyclictest31124-21perf22:20:001
1461199165,7cyclictest0-21swapper/122:26:391
14611991611,1cyclictest7743-21grep21:35:241
14611991610,6cyclictest5820-21diskmemload21:49:321
1460799163,6cyclictest224-21systemd-journal23:35:000
14618991514,1cyclictest0-21swapper/322:17:393
14618991511,2cyclictest21968-21wc20:30:233
1461199159,5cyclictest5820-21diskmemload21:21:391
1461199159,5cyclictest0-21swapper/122:55:231
1461199157,8cyclictest0-21swapper/122:03:501
1461199152,12cyclictest20303-21processes22:35:191
14611991513,2cyclictest25-21ksoftirqd/100:28:101
14611991513,1cyclictest5820-21diskmemload23:29:381
14611991511,3cyclictest15752-21chrt20:20:001
1461899145,8cyclictest0-21swapper/320:00:123
1461899145,8cyclictest0-21swapper/320:00:113
1461899145,7cyclictest815-21idleruntime-cro23:35:003
1461899142,6cyclictest28893-21hddtemp_smartct23:55:163
14618991412,2cyclictest39-21ksoftirqd/323:53:573
14618991410,2cyclictest614-21wc21:55:273
1461899140,14cyclictest0-21swapper/321:40:493
1461899140,12cyclictest15787-21awk20:20:003
1461599149,4cyclictest6191-21tr21:35:142
1461599148,5cyclictest12300-21grep22:30:132
14615991412,2cyclictest37850irq/131-snd_hda21:20:222
14615991412,1cyclictest0-21swapper/222:55:182
14615991410,3cyclictest12824-21systemd00:08:572
1461599140,7cyclictest0-21swapper/222:50:082
1461599140,6cyclictest0-21swapper/219:32:532
1461199148,6cyclictest0-21swapper/122:20:131
1461199148,5cyclictest0-21swapper/122:14:011
1461199148,4cyclictest4374-21sh23:13:311
1461199146,8cyclictest0-21swapper/121:12:281
1461199145,7cyclictest5820-21diskmemload23:41:441
14611991413,1cyclictest0-21swapper/100:08:391
14611991412,2cyclictest5820-21diskmemload22:31:441
14611991412,2cyclictest0-21swapper/121:34:281
1461199140,14cyclictest0-21swapper/123:48:131
1461199140,14cyclictest0-21swapper/123:48:121
1461199140,13cyclictest32547-21idleruntime19:45:181
14607991410,3cyclictest5823-21grep21:09:590
14607991410,3cyclictest1515-21cat19:50:000
1460799140,1cyclictest0-21swapper/019:15:130
23782130,0chrt0-21swapper/223:11:312
1461899139,3cyclictest24168-21munin-plugin-st21:50:013
1461899135,5cyclictest13954-21apache223:28:403
14618991312,1cyclictest0-21swapper/322:11:383
14618991312,1cyclictest0-21swapper/321:13:583
14618991312,1cyclictest0-21swapper/300:25:183
14618991310,2cyclictest4298-21tr19:55:163
14618991310,2cyclictest31626-21basename22:20:123
14618991310,2cyclictest31199-21dump-pmu-power21:30:013
14618991310,2cyclictest31050-21grep19:40:253
14618991310,2cyclictest27673-21grep19:35:183
14618991310,2cyclictest25299-21head19:30:203
14618991310,2cyclictest1385-21tr23:35:133
1461599138,3cyclictest15505-21grep00:10:212
14615991310,2cyclictest9543-21head20:05:152
14615991310,2cyclictest7163-21head21:10:202
14615991310,2cyclictest31033-21head19:40:252
14615991310,2cyclictest21625-21head23:00:272
14615991310,2cyclictest15352-21grep20:15:242
1461599130,12cyclictest0-21swapper/223:40:562
1461599130,12cyclictest0-21swapper/223:27:452
1461199139,3cyclictest19534-21cron19:19:591
1461199139,1cyclictest0-21swapper/123:31:201
1461199138,5cyclictest0-21swapper/121:56:201
1461199137,6cyclictest0-21swapper/120:50:241
1461199137,5cyclictest2506-21idleruntime19:50:241
1461199134,7cyclictest0-21swapper/120:30:031
14611991310,2cyclictest7567-21awk20:00:181
14611991310,2cyclictest7567-21awk20:00:181
1461199130,11cyclictest0-21swapper/121:53:171
1460799139,3cyclictest2321-21wc21:00:190
1460799136,3cyclictest0-21swapper/000:08:580
14607991310,2cyclictest24033-21latency_hist22:15:010
14607991310,2cyclictest21714-21grep20:30:200
14607991310,2cyclictest18613-21wc22:35:000
14607991310,2cyclictest17697-21sed20:20:260
14607991310,2cyclictest13471-21grep22:55:140
1461899129,2cyclictest9540-21chrt22:52:053
1461899129,2cyclictest32128-21sed20:55:183
1461899129,2cyclictest2994-21grep21:00:253
1461899129,2cyclictest15151-21wc23:45:003
1461899128,2cyclictest24319-21awk19:30:013
1461899127,3cyclictest20703-21cut00:15:163
1461899127,3cyclictest13832-21ssh23:20:123
1461899122,9cyclictest14068-21munin-node22:42:443
14618991211,1cyclictest39-21ksoftirqd/321:37:493
1461599129,2cyclictest6696-21tr20:00:142
1461599129,2cyclictest6696-21tr20:00:132
1461599129,2cyclictest28367-21cat23:55:132
1461599129,2cyclictest28130-21wc23:30:212
1461599129,2cyclictest27180-21cut19:35:152
1461599129,2cyclictest26262-21tr20:40:152
1461599129,2cyclictest25270-21ssh22:15:152
1461599129,2cyclictest23599-21chrt20:35:122
1461599129,2cyclictest21710-21grep20:30:212
1461599129,2cyclictest21641-21grep23:50:132
1461599129,2cyclictest21465-21cut00:15:192
1461599128,3cyclictest32182-21sh22:44:272
1461599128,3cyclictest17214-21cron22:10:012
1461599128,3cyclictest17181-21ssh23:46:592
1461599128,3cyclictest17181-21ssh23:46:582
1461599126,2cyclictest224-21systemd-journal21:00:002
1461599120,12cyclictest0-21swapper/221:52:302
1461199129,2cyclictest21571-21tail23:00:271
1461199124,6cyclictest0-21swapper/122:09:401
1461199120,11cyclictest10927-21sh20:10:001
1460799129,2cyclictest7090-21grep20:00:150
1460799129,2cyclictest7090-21grep20:00:150
1460799129,2cyclictest31191-21cron21:30:000
1460799129,2cyclictest30079-21grep19:40:160
1460799129,2cyclictest27190-21tr00:20:200
1460799129,2cyclictest25343-21sort22:15:150
1460799129,2cyclictest22213-21rm00:15:270
1460799129,2cyclictest21306-21cat19:20:240
1460799129,2cyclictest21090-21ssh23:50:120
1460799128,3cyclictest29157-21sh21:53:140
1460799124,4cyclictest224-21systemd-journal00:39:590
1460799121,6cyclictest26189-21munin-run23:05:000
1460799121,4cyclictest0-21swapper/023:15:130
14607991212,0cyclictest0-21swapper/021:30:100
14607991210,1cyclictest0-21swapper/021:36:480
1460799120,7cyclictest3804-21perf19:55:010
1460799120,2cyclictest0-21swapper/000:00:490
1461899119,1cyclictest4803-21grep21:05:193
1461899118,2cyclictest8350-21wc23:15:263
1461899118,2cyclictest6225-21ssh00:31:473
1461899118,2cyclictest32341-21df23:10:123
1461899118,2cyclictest3218-21cut00:00:183
1461899118,2cyclictest31898-21tr21:30:103
1461899118,2cyclictest27835-21ssh23:06:293
1461899118,2cyclictest25962-21basename00:20:103
1461899118,2cyclictest21458-21ssh22:36:013
1461899118,2cyclictest20816-21cat21:20:173
1461899118,2cyclictest19871-21aten2_r7power_e19:20:113
1461899118,2cyclictest19188-21taskset19:15:373
1461899118,2cyclictest19156-21tail20:25:173
1461899118,2cyclictest16958-21sleep19:14:593
1461899118,2cyclictest16411-21ssh22:09:113
1461899118,2cyclictest15064-21ssh21:15:283
1461899118,2cyclictest11986-21grep20:10:203
1461899118,2cyclictest10521-21rm00:35:233
1461899117,2cyclictest13054-21grep22:55:133
1461899116,2cyclictest224-21systemd-journal20:42:263
1461899112,8cyclictest16050-21apt-get20:20:153
14618991110,1cyclictest0-21swapper/322:25:393
1461899110,10cyclictest1519-21ssh22:45:593
1461599119,1cyclictest11479-21tr20:10:182
1461599118,2cyclictest6205-21ssh22:25:082
1461599118,2cyclictest31995-21grep19:45:162
1461599118,2cyclictest28632-21taskset20:49:042
1461599118,2cyclictest27229-21chrt21:25:532
1461599118,2cyclictest26586-21grep00:20:152
1461599118,2cyclictest2076-21sed23:35:172
1461599118,2cyclictest191622sleep219:15:272
1461599118,2cyclictest16326-21grep20:20:162
1461599118,2cyclictest13623-21ssh23:20:052
1461599118,2cyclictest12703-21apache_processe21:40:122
1461599118,2cyclictest10075-21ssh22:02:542
1461199118,2cyclictest17090-21sed20:20:211
1461199118,2cyclictest11447-21grep20:10:171
1461199118,2cyclictest0-21swapper/119:55:131
1461199117,3cyclictest19799-21cut20:25:221
1461199117,3cyclictest0-21swapper/119:25:141
1461199115,3cyclictest5820-21diskmemload21:40:131
14611991110,1cyclictest25-21ksoftirqd/120:36:511
1461199110,3cyclictest2441-21diskmemload00:00:121
1461199110,10cyclictest30234-21ssh23:08:521
1461199110,10cyclictest24586-21apt19:30:161
1460799118,2cyclictest8851-21ssh22:51:240
1460799118,2cyclictest8817-21apache_accesses20:05:110
1460799118,2cyclictest7524-21ssh22:26:280
1460799118,2cyclictest7007-21cut21:10:210
1460799118,2cyclictest5012-21sed00:30:230
1460799118,2cyclictest3426-21ssh22:22:400
1460799118,2cyclictest32116-21tail20:55:170
1460799118,2cyclictest3042-21ls23:35:250
1460799118,2cyclictest29018-21ssh23:55:160
1460799118,2cyclictest27723-21sensors_temp22:40:270
1460799118,2cyclictest27502-21grep19:35:150
1460799118,2cyclictest23626-21grep20:35:120
1460799118,2cyclictest19593-21cut22:35:140
1460799118,2cyclictest18964-21df20:25:150
1460799118,2cyclictest16494-21chrt23:46:150
1460799118,2cyclictest16494-21chrt23:46:150
1460799118,2cyclictest15651-21ssh21:16:030
1460799118,2cyclictest15375-21ssh00:10:200
1460799118,2cyclictest1407-21rm21:56:010
1460799118,2cyclictest13755-21ssh22:06:300
1460799110,10cyclictest784-21ssh22:45:140
1460799110,10cyclictest6042-21cut22:00:140
1461899108,2cyclictest10033-21ssh00:05:503
1461899108,1cyclictest31873-21grep19:45:163
1461899108,1cyclictest28955-21cat20:50:093
1461899108,1cyclictest23560-21taskset20:35:103
1461899108,1cyclictest1829-21egrep19:50:203
1461899107,2cyclictest9094-21cut20:05:123
1461899107,2cyclictest22882-21ssh23:01:433
1461899107,2cyclictest16019-21ssh23:45:473
1461899107,2cyclictest16019-21ssh23:45:473
1461899105,3cyclictest28719-21turbostat.cron20:49:593
1461599108,1cyclictest4083-21wget19:55:152
1461599108,1cyclictest29252-21grep20:50:122
1461599108,1cyclictest22644-21expr19:25:162
1461599107,2cyclictest4503-21sh21:05:172
1461599107,2cyclictest3328-21grep19:50:322
1461599107,2cyclictest32406-21grep21:30:132
1461599107,2cyclictest26680-21ssh23:05:192
1461599107,2cyclictest21317-21ssh22:35:522
1461599107,2cyclictest19863-21grep19:20:112
1461599107,2cyclictest19148-21grep20:25:172
1461599107,2cyclictest14905-21wget19:10:122
1461599103,5cyclictest502-21rs:main2
1461599100,9cyclictest621-21sh00:27:122
1461199108,1cyclictest3205-21chrt21:02:071
1461199108,1cyclictest30220-21cat19:40:171
1461199108,1cyclictest147912sleep119:10:011
1461199106,3cyclictest26279-21grep20:40:151
1461199105,4cyclictest0-21swapper/119:35:251
1461199101,2cyclictest5607-21taskset21:07:281
1461199100,9cyclictest26084-21ssh22:40:121
1460799108,1cyclictest4035-21cat19:55:120
1460799108,1cyclictest31381-21sh00:25:110
1460799107,2cyclictest8756-21expr23:40:140
1460799107,2cyclictest29529-21df20:50:140
1460799107,2cyclictest28660-21seq20:49:240
1460799107,2cyclictest26072-21cat19:30:300
1460799107,2cyclictest22434-21tr19:25:150
1460799107,2cyclictest21311-21ssh23:25:270
1460799107,2cyclictest20232-21ssh21:45:560
1460799107,2cyclictest15300-21expr19:10:140
1460799107,2cyclictest13720-21ssh23:20:110
146119997,1cyclictest28329-21taskset20:45:071
146079997,1cyclictest277392sleep020:40:300
146079997,1cyclictest135372sleep020:15:070
146079996,2cyclictest11453-21grep20:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional