You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-18 - 05:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Tue Mar 18, 2025 00:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11884210490,9sleep10-21swapper/119:05:311
12042210176,9sleep30-21swapper/319:06:413
1146628874,9sleep00-21swapper/019:05:260
1226428464,16sleep20-21swapper/219:09:332
1239299380,19cyclictest0-21swapper/322:40:103
263952280,1chrt0-21swapper/023:30:000
1238999270,27cyclictest0-21swapper/222:28:132
1238999260,25cyclictest0-21swapper/200:30:172
1238999240,24cyclictest0-21swapper/200:24:332
1238999240,21cyclictest0-21swapper/200:27:312
12383992423,1cyclictest0-21swapper/021:33:150
12383992422,1cyclictest0-21swapper/000:07:060
228922210,0sleep10-21swapper/123:01:521
1238999211,19cyclictest0-21swapper/222:52:302
1238999210,1cyclictest0-21swapper/221:28:392
12383992121,0cyclictest0-21swapper/023:19:340
1238999202,4cyclictest141rcu_preempt22:15:412
12389992018,1cyclictest32-21ksoftirqd/223:55:312
12389992018,1cyclictest32-21ksoftirqd/222:44:122
1238999200,18cyclictest0-21swapper/222:20:062
1238799202,1cyclictest0-21swapper/122:59:201
12387992019,1cyclictest0-21swapper/122:24:101
1238799200,19cyclictest0-21swapper/122:15:001
1238399200,1cyclictest0-21swapper/021:25:120
1238999195,2cyclictest0-21swapper/223:42:202
1238999195,2cyclictest0-21swapper/223:42:202
1238999193,3cyclictest141rcu_preempt19:10:212
1238999190,19cyclictest0-21swapper/223:50:132
1238999190,19cyclictest0-21swapper/223:19:182
1238799195,8cyclictest224-21systemd-journal21:20:001
1238399198,2cyclictest1388-21diskmemload22:50:000
1238399190,5cyclictest0-21swapper/021:23:530
12389991813,1cyclictest32-21ksoftirqd/221:47:352
1238399180,14cyclictest0-21swapper/023:49:000
1238399180,0cyclictest0-21swapper/023:10:330
1238999179,5cyclictest1388-21diskmemload22:07:272
1238999177,2cyclictest4725-21idleruntime-cro22:50:002
12389991714,3cyclictest32-21ksoftirqd/221:20:012
12389991714,1cyclictest141rcu_preempt23:35:262
1238999170,17cyclictest0-21swapper/223:13:482
1238999170,16cyclictest0-21swapper/221:57:332
1238999170,16cyclictest0-21swapper/221:42:392
1238999170,16cyclictest0-21swapper/221:34:532
1238799173,2cyclictest141rcu_preempt21:42:541
1238399170,1cyclictest0-21swapper/021:53:140
1239299160,1cyclictest0-21swapper/321:38:053
1238999166,6cyclictest32-21ksoftirqd/223:22:312
12389991616,0cyclictest0-21swapper/200:00:272
12389991615,1cyclictest32-21ksoftirqd/200:12:552
1238799164,2cyclictest141rcu_preempt00:19:441
1238799160,2cyclictest0-21swapper/122:37:181
12383991616,0cyclictest0-21swapper/022:20:150
12383991614,2cyclictest26562-21ssh00:16:530
12383991614,1cyclictest0-21swapper/023:41:420
12383991614,1cyclictest0-21swapper/023:41:420
12392991511,2cyclictest14685-21tr20:25:163
1239299151,10cyclictest224-21systemd-journal22:50:003
1239299150,2cyclictest141rcu_preempt22:52:473
1239299150,1cyclictest0-21swapper/322:58:003
1239299150,14cyclictest0-21swapper/321:15:143
1238999154,11cyclictest32-21ksoftirqd/221:10:502
12389991513,1cyclictest32-21ksoftirqd/223:47:142
1238999151,1cyclictest0-21swapper/221:35:122
1238999150,14cyclictest0-21swapper/223:25:152
12387991511,3cyclictest221irq_work/123:50:421
1238799150,8cyclictest28322-21sensors23:30:201
1238399154,6cyclictest24262-21munin-run19:35:000
1238399150,1cyclictest0-21swapper/022:41:350
12392991411,2cyclictest31953-21sh22:20:003
12392991411,2cyclictest29720-21head21:30:243
12392991410,3cyclictest19608-21cat19:25:013
1238999143,1cyclictest141rcu_preempt22:01:402
1238999141,7cyclictest1388-21diskmemload22:56:472
12389991413,1cyclictest37850irq/131-snd_hda21:15:202
1238999141,1cyclictest141rcu_preempt23:09:102
12389991411,2cyclictest32-21ksoftirqd/220:10:002
12389991411,2cyclictest32-21ksoftirqd/219:40:002
1238999140,1cyclictest141rcu_preempt20:35:282
1238999140,14cyclictest0-21swapper/200:36:142
1238999140,13cyclictest28333-21wc19:40:292
1238999140,0cyclictest0-21swapper/222:30:482
12387991413,1cyclictest1388-21diskmemload00:28:381
12387991411,2cyclictest1964-21wc21:10:111
1238799140,2cyclictest0-21swapper/121:55:151
1238399143,9cyclictest25358-21aten2_r7power_v22:15:150
1238399141,7cyclictest8308-21munin-run00:30:000
12383991410,3cyclictest16610-21grep19:15:290
12383991410,3cyclictest12442-21tail21:40:250
1238399140,13cyclictest7728-21ssh00:02:560
1239299139,3cyclictest9397-21head00:30:143
1239299139,3cyclictest20494-21grep19:25:193
1239299139,3cyclictest20405-21grep20:35:233
1239299139,2cyclictest30987-21cut19:50:163
1239299138,3cyclictest933-21cron23:35:003
1239299133,7cyclictest8209-21/usr/sbin/munin23:15:223
12392991310,2cyclictest6854-21turbostat.cron22:25:003
12392991310,2cyclictest32409-21ssh23:57:323
1239299130,7cyclictest0-21swapper/319:10:143
1239299130,12cyclictest8963-21grep20:10:313
1238999139,3cyclictest4002-21wc20:00:262
1238999139,3cyclictest20414-21cut19:25:182
1238999139,2cyclictest32-21ksoftirqd/221:52:392
12389991310,2cyclictest22193-21cut20:40:182
12389991310,2cyclictest15565-21ls00:10:002
1238799133,8cyclictest28933-21ssh23:55:171
1238799131,4cyclictest0-21swapper/119:20:141
12387991311,1cyclictest241rcuc/119:40:191
12387991310,2cyclictest18023-21cat21:20:261
1238399138,4cyclictest500-21in:imuxsock23:06:470
12383991310,2cyclictest26961-21cut20:50:180
12383991310,2cyclictest16070-21grep23:20:270
12383991310,2cyclictest0-21swapper/021:35:470
12383991310,1cyclictest27685-21sed19:40:210
1238399130,13cyclictest0-21swapper/022:01:180
1238399130,12cyclictest20395-21cut19:25:180
1239299129,2cyclictest6543-21tail20:05:273
1239299129,2cyclictest5315-21cut22:00:163
1239299129,2cyclictest31244-21uname00:20:203
1239299129,2cyclictest28980-21grep20:55:173
1239299129,2cyclictest25033-21cat22:40:013
1239299129,2cyclictest21809-21grep20:40:163
1239299129,2cyclictest20471-21sort23:50:003
1239299129,2cyclictest15752-21cut19:15:203
1239299129,2cyclictest12298-21tr20:20:133
1239299128,3cyclictest26454-21grep20:50:153
1239299128,3cyclictest19532-21basename23:00:143
1239299127,2cyclictest0-21swapper/323:50:013
12392991210,1cyclictest218482sleep322:12:293
1239299120,4cyclictest0-21swapper/300:28:103
1239299120,3cyclictest1388-21diskmemload00:12:083
1239299120,3cyclictest0-21swapper/323:21:283
1238999129,2cyclictest2196-21munin-run20:00:002
1238999129,2cyclictest19569-21cat19:25:012
1238999129,2cyclictest15132-21basename19:15:172
1238999128,3cyclictest24038-21if_enp2s000:15:192
1238999122,9cyclictest37850irq/131-snd_hda23:30:092
1238999122,8cyclictest0-21swapper/222:11:012
1238999120,1cyclictest0-21swapper/223:00:262
1238799129,2cyclictest5981-21sed22:00:191
1238799129,2cyclictest32734-21tr19:55:141
1238799129,2cyclictest23377-21grep19:30:251
1238799129,2cyclictest22725-21wc00:15:011
1238799129,2cyclictest1706-21grep23:35:161
1238799129,2cyclictest16526-21cron20:30:001
1238799129,2cyclictest14747-21basename19:15:001
1238799129,2cyclictest12292-21tr20:20:131
1238799129,2cyclictest10743-21cat20:15:211
1238799127,4cyclictest6929-21wc22:50:211
1238799123,6cyclictest0-21swapper/122:41:481
1238799123,1cyclictest141rcu_preempt21:50:511
1238799121,9cyclictest0-21swapper/122:30:091
1238799121,10cyclictest0-21swapper/123:20:141
1238799120,11cyclictest13323-21systemd00:07:571
1238399129,2cyclictest8485-21cut20:10:270
1238399129,2cyclictest32004-21sh19:55:000
1238399129,2cyclictest21782-21tail23:50:170
1238399129,2cyclictest18337-21tune2fs22:10:120
1238399129,2cyclictest17083-21wc00:10:200
1238399129,2cyclictest1674-21grep19:55:260
1238399129,2cyclictest12871-21grep19:10:150
1239299119,1cyclictest276132sleep321:29:183
1239299118,2cyclictest8337-21ssh22:26:123
1239299118,2cyclictest7050-21ssh21:13:303
1239299118,2cyclictest6060-21ssh00:01:203
1239299118,2cyclictest5715-21ssh23:37:363
1239299118,2cyclictest3233-21head20:00:173
1239299118,2cyclictest29289-21taskset19:45:413
1239299118,2cyclictest26956-21ssh23:05:323
1239299118,2cyclictest26221-21ssh21:51:323
1239299118,2cyclictest2614-21ssh23:11:263
1239299118,2cyclictest25131-21cut19:35:173
1239299118,2cyclictest23946-21cut00:15:183
1239299118,2cyclictest23268-21ssh23:27:003
1239299118,2cyclictest18386-21ssh21:45:513
1239299118,2cyclictest16745-21ssh00:35:343
1239299118,2cyclictest16622-21ssh22:32:143
1239299118,2cyclictest16484-21ssh22:09:153
1239299118,2cyclictest16227-21ssh21:20:093
1239299118,2cyclictest12948-21ssh23:42:423
1239299118,2cyclictest12948-21ssh23:42:423
1239299118,2cyclictest11259-21cut21:40:163
1239299118,2cyclictest10063-21expr20:15:163
1238999118,2cyclictest29701-21wc19:50:002
1238999118,2cyclictest21847-21ssh22:37:042
1238999118,2cyclictest15777-21grep20:25:242
1238999117,3cyclictest12081-21cut20:20:132
1238999115,3cyclictest1387-21cron21:10:002
1238999114,3cyclictest224-21systemd-journal21:00:002
1238799118,2cyclictest3796-21cut21:35:151
1238799118,2cyclictest374-21ssh22:45:471
1238799118,2cyclictest30747-21df00:20:161
1238799118,2cyclictest28718-21sh22:16:551
1238799118,2cyclictest28612-21ssh21:30:031
1238799118,2cyclictest26252-21ssh23:29:541
1238799118,2cyclictest24058-21rm21:25:521
1238799118,2cyclictest21296-21ls20:40:001
1238799118,2cyclictest20355-21ssh21:47:441
1238799118,2cyclictest13919-21ssh00:34:421
1238799118,2cyclictest12069-21ssh22:05:011
1238799118,2cyclictest10174-21seq23:16:421
1238799116,4cyclictest30876-21date21:00:001
1238799110,10cyclictest9520-21ssh22:27:201
1238799110,10cyclictest31908-21ssh23:10:041
1238399119,1cyclictest14189-21ssh22:30:180
1238399118,2cyclictest9851-21ssh22:52:570
1238399118,2cyclictest9292-21grep21:15:150
1238399118,2cyclictest9064-21expr00:30:120
1238399118,2cyclictest8027-21ssh21:14:280
1238399118,2cyclictest4628-21apt20:05:000
1238399118,2cyclictest30886-21sh21:55:170
1238399118,2cyclictest30487-21taskset20:56:120
1238399118,2cyclictest30464-21cat00:20:150
1238399118,2cyclictest27228-21expr23:30:110
1238399118,2cyclictest26058-21sleep20:50:000
1238399118,2cyclictest26058-21sleep20:50:000
1238399118,2cyclictest24774-21tr19:35:150
1238399118,2cyclictest2137-21ssh23:59:330
1238399118,2cyclictest20844-21grep23:00:210
1238399118,2cyclictest19709-21ssh21:47:070
1238399118,2cyclictest16149-21head20:25:280
1238399118,2cyclictest15737-21chrt22:56:500
1238399118,2cyclictest15091-21cat00:35:160
1238399118,2cyclictest14173-21ssh22:07:010
1239299108,1cyclictest32449-21chrt19:55:053
1239299108,1cyclictest31829-21gltestperf21:00:223
1239299108,1cyclictest23946-21apt-get20:45:143
1239299108,1cyclictest23946-21apt-get20:45:133
1239299108,1cyclictest18600-21taskset20:31:113
1239299108,1cyclictest10703-21sh00:05:353
1239299107,2cyclictest31916-21wc21:55:263
1239299107,2cyclictest23520-21cut19:30:263
1239299107,2cyclictest1033-21taskset21:05:423
1239299101,5cyclictest0-21swapper/319:40:253
1238999108,1cyclictest548-21taskset21:02:352
1238999107,2cyclictest8760-21wc20:10:312
1238999107,2cyclictest26236-21wc20:50:002
1238999107,2cyclictest26236-21wc20:50:002
1238999107,2cyclictest23461-21grep19:30:262
1238999107,2cyclictest18810-21taskset20:33:532
1238999107,2cyclictest10855-21cut20:15:222
1238799108,1cyclictest93312sleep120:14:311
1238799108,1cyclictest4866-21acpi20:05:121
1238799108,1cyclictest3515-21wc20:00:191
1238799108,1cyclictest27037-21ssh23:05:361
1238799108,1cyclictest24377-21expr20:45:151
1238799108,1cyclictest24377-21expr20:45:151
1238799108,1cyclictest14551-21ssh00:35:081
1238799107,2cyclictest9441-21head23:40:181
1238799107,2cyclictest9441-21head23:40:181
1238799107,2cyclictest30221-21expr19:50:111
1238799107,2cyclictest2921-21ssh00:00:021
1238799107,2cyclictest27911-21cat20:50:271
1238799107,2cyclictest16907-21cut20:30:181
1238799107,2cyclictest15500-21ssh23:45:061
1238799107,2cyclictest15083-21cut19:15:171
1238799107,2cyclictest1048-21taskset21:05:541
1238799105,2cyclictest224-21systemd-journal21:00:051
1238399108,2cyclictest17642-21basename20:30:220
1238399108,1cyclictest7632-21ssh22:25:320
1238399108,1cyclictest294082sleep019:47:130
1238399108,1cyclictest19675-21sh22:35:020
1238399107,2cyclictest31874-21head21:00:220
1238399107,2cyclictest18267-21if_enp2s019:20:180
1238399107,2cyclictest1794-21ssh23:35:160
1238399107,2cyclictest12742-21cut20:20:160
1238399105,4cyclictest11752-21cron20:20:010
123899997,1cyclictest26879-21sed20:50:172
123899995,3cyclictest31884-21chrt19:53:322
123879997,1cyclictest25272-21expr19:35:181
123879997,1cyclictest216252sleep120:40:151
123879992,4cyclictest224-21systemd-journal19:46:081
123879990,2cyclictest0-21swapper/119:25:241
123839997,1cyclictest5347-21expr20:05:150
123839997,1cyclictest21886-21cat20:40:160
123839997,1cyclictest19138-21taskset20:35:070
123839986,1cyclictest1430-21ssh21:10:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional