You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-19 - 02:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Sep 18, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11365210982,9sleep30-21swapper/307:05:523
1156829784,9sleep10-21swapper/107:08:321
1161628464,15sleep20-21swapper/207:09:082
1159428169,8sleep00-21swapper/007:08:510
302872640,0sleep30-21swapper/310:16:063
1178099301,28cyclictest0-21swapper/209:44:342
11775992927,2cyclictest13-21ksoftirqd/010:45:000
11780992624,1cyclictest4829-21diskmemload12:12:322
11784992524,1cyclictest0-21swapper/312:21:583
11780992518,6cyclictest0-21swapper/212:15:122
11780992514,8cyclictest4829-21diskmemload10:15:332
11780992514,10cyclictest4829-21diskmemload10:12:182
1178099247,16cyclictest0-21swapper/208:25:182
1178099240,24cyclictest0-21swapper/210:45:522
247112230,0sleep10-21swapper/108:40:271
1178099239,8cyclictest4829-21diskmemload11:30:272
1178099239,8cyclictest4829-21diskmemload11:30:272
1178099234,16cyclictest0-21swapper/209:56:332
1178099234,16cyclictest0-21swapper/209:56:322
11780992321,1cyclictest0-21swapper/211:35:132
11780992320,2cyclictest6425-21hddtemp_smartct09:10:172
1178099231,21cyclictest0-21swapper/210:01:452
11778992321,2cyclictest25-21ksoftirqd/111:11:361
1177899230,2cyclictest0-21swapper/112:05:171
1177899230,23cyclictest0-21swapper/111:43:301
1177599234,1cyclictest24442-21ssh10:55:190
11780992220,2cyclictest4829-21diskmemload09:34:532
11780992220,1cyclictest0-21swapper/211:45:592
11780992219,3cyclictest4829-21diskmemload09:36:352
11780992217,4cyclictest30132-21sh11:20:092
11780992216,5cyclictest0-21swapper/208:30:162
11780992215,6cyclictest0-21swapper/211:57:032
11780992213,8cyclictest4829-21diskmemload09:16:052
11780992213,7cyclictest0-21swapper/212:30:142
11780992211,8cyclictest4829-21diskmemload11:00:182
324102210,0sleep20-21swapper/211:41:562
11780992114,6cyclictest4829-21diskmemload10:05:232
11780992114,6cyclictest4600-21open_inodes08:00:212
11780992111,6cyclictest0-21swapper/210:35:002
11778992120,1cyclictest0-21swapper/110:07:161
92492200,0sleep30-21swapper/309:12:383
168002200,0sleep00-21swapper/011:31:440
168002200,0sleep00-21swapper/011:31:440
1178099207,10cyclictest0-21swapper/210:29:502
1178099206,13cyclictest0-21swapper/209:21:482
11780992013,6cyclictest3736-21ssh10:20:282
11780992012,7cyclictest5052-21ssh12:05:502
11780992012,1cyclictest4829-21diskmemload11:53:532
11780992010,8cyclictest0-21swapper/211:25:362
1177599203,2cyclictest18956-21ssh10:08:390
1177599200,2cyclictest0-21swapper/010:24:080
1178099199,9cyclictest0-21swapper/212:35:592
11780991915,3cyclictest19862-21ssh10:51:522
11780991914,4cyclictest28875-21nfs4_client08:50:212
11778991918,1cyclictest0-21swapper/110:58:311
1178499185,11cyclictest0-21swapper/309:23:463
11784991816,1cyclictest0-21swapper/310:08:183
1178099188,7cyclictest4829-21diskmemload12:25:262
1178099188,10cyclictest4829-21diskmemload09:51:392
1178099188,10cyclictest4829-21diskmemload09:51:392
11780991816,2cyclictest0-21swapper/209:27:032
11780991816,2cyclictest0-21swapper/209:05:192
11780991814,3cyclictest0-21swapper/208:35:152
11780991814,3cyclictest0-21swapper/207:30:202
11780991813,4cyclictest0-21swapper/208:05:162
11780991813,4cyclictest0-21swapper/208:05:152
11780991813,2cyclictest20329-21ntpq09:45:182
11780991811,4cyclictest0-21swapper/207:20:222
11780991810,8cyclictest0-21swapper/211:19:092
11780991810,7cyclictest4829-21diskmemload10:36:252
1177899180,14cyclictest0-21swapper/109:50:531
1177899180,14cyclictest0-21swapper/109:50:531
11775991810,7cyclictest9675-21expr12:30:140
1177599180,18cyclictest0-21swapper/012:02:570
1177599180,0cyclictest0-21swapper/011:21:260
11784991716,1cyclictest0-21swapper/312:02:183
1178099177,9cyclictest0-21swapper/210:43:342
11780991716,1cyclictest0-21swapper/207:15:122
11780991714,3cyclictest0-21swapper/211:12:432
11780991714,2cyclictest29484-21ssh10:58:372
11780991711,1cyclictest4829-21diskmemload12:23:252
1177599171,6cyclictest30783-21kworker/u8:009:28:160
11775991716,1cyclictest161rcuc/011:40:180
11775991716,1cyclictest13-21ksoftirqd/010:25:120
11775991713,4cyclictest161rcuc/009:37:380
1177599170,1cyclictest0-21swapper/012:18:530
1177599170,1cyclictest0-21swapper/010:49:380
1177599170,17cyclictest0-21swapper/011:39:590
11784991615,1cyclictest0-21swapper/311:51:183
11784991611,4cyclictest14537-21sh10:48:573
1178099163,7cyclictest0-21swapper/211:09:442
1178099160,15cyclictest0-21swapper/212:02:582
11778991612,1cyclictest14387-21apache_processe07:15:111
1177599161,4cyclictest0-21swapper/010:14:220
1177599161,13cyclictest0-21swapper/009:21:190
81582150,0chrt0-21swapper/312:08:303
1178499151,8cyclictest0-21swapper/311:32:223
1178499151,8cyclictest0-21swapper/311:32:223
11784991514,1cyclictest0-21swapper/311:45:583
11784991514,1cyclictest0-21swapper/309:48:583
11784991514,1cyclictest0-21swapper/309:32:383
1178099154,5cyclictest2427-21cron09:05:002
1177899153,8cyclictest18253-21/usr/sbin/munin09:20:241
11778991514,1cyclictest0-21swapper/109:45:291
1177899151,1cyclictest0-21swapper/111:00:561
1177899150,14cyclictest10588-21cron08:15:001
11775991514,1cyclictest13-21ksoftirqd/007:21:440
1178499147,6cyclictest61850irq/126-enp2s0-12:32:523
11784991413,1cyclictest0-21swapper/309:58:583
11784991413,1cyclictest0-21swapper/309:58:573
11784991411,2cyclictest1805-21sed07:55:213
11784991410,3cyclictest27118-21head09:50:213
11784991410,3cyclictest27118-21head09:50:203
1178499140,13cyclictest32096-21grep11:20:273
11780991411,3cyclictest0-21swapper/208:20:272
11780991411,2cyclictest20591-21wc07:25:312
11778991411,3cyclictest5399-21apache210:21:321
1177899140,1cyclictest0-21swapper/110:03:271
1177599146,6cyclictest4829-21diskmemload10:35:240
11775991413,1cyclictest4829-21diskmemload12:08:000
11775991413,1cyclictest161rcuc/009:40:430
11775991413,1cyclictest13-21ksoftirqd/012:38:500
11775991413,1cyclictest13-21ksoftirqd/011:13:300
11775991412,1cyclictest13-21ksoftirqd/012:25:450
1177599141,10cyclictest13-21ksoftirqd/011:59:200
1177599140,1cyclictest0-21swapper/011:09:330
1177599140,12cyclictest0-21swapper/011:54:020
1178499139,3cyclictest28500-21head08:50:173
1178499136,5cyclictest21505-21/usr/sbin/munin07:30:203
1178499133,5cyclictest224-21systemd-journal09:04:593
11784991311,2cyclictest0-21swapper/311:13:383
11784991310,2cyclictest21592-21sed11:35:153
11784991310,2cyclictest18956-21sed12:35:323
11784991310,2cyclictest10463-21cut12:10:133
1178099133,8cyclictest25597-21runrttasks08:45:162
11780991310,2cyclictest9471-21wc08:10:202
11780991310,2cyclictest29719-21wc07:45:202
11780991310,2cyclictest12714-21grep07:10:172
1177899139,3cyclictest28225-21sh11:40:001
1177899139,3cyclictest2296-21cat12:05:001
1177899139,3cyclictest20951-21grep10:10:131
1177899138,3cyclictest233052sleep107:30:431
1177899137,5cyclictest20606-21ssh11:35:011
1177899137,5cyclictest20606-21ssh11:35:001
1177899137,4cyclictest4829-21diskmemload09:26:411
11778991310,2cyclictest909-21grep10:40:151
11778991310,2cyclictest8752-21sleep12:29:591
11778991310,2cyclictest18668-21grep08:30:161
11778991310,2cyclictest14124-21wc11:10:001
11778991310,2cyclictest13644-21grep09:15:341
11778991310,2cyclictest1079-21grep09:00:171
1177899130,1cyclictest0-21swapper/112:18:031
1177599138,1cyclictest2425-21cron09:05:000
11775991310,2cyclictest4805-21sh09:09:590
11775991310,2cyclictest28400-21tail10:15:170
11775991310,2cyclictest22467-21cut07:30:170
1178499129,3cyclictest0-21swapper/310:31:163
1178499129,2cyclictest9208-21grep08:10:183
1178499129,2cyclictest8187-21grep10:00:183
1178499129,2cyclictest6824-21ssh11:05:123
1178499129,2cyclictest24269-21grep11:15:253
1178499129,2cyclictest23773-21head08:40:183
1178499129,2cyclictest21328-21head08:35:183
1178499129,2cyclictest18049-21wc07:20:213
1178499129,2cyclictest16706-21expr10:50:143
1178499129,2cyclictest15885-21tr08:25:123
1178499129,2cyclictest13550-21tr08:20:163
1178499129,2cyclictest12659-21cut09:15:253
1178499129,2cyclictest1047-21sed11:00:253
1178499128,3cyclictest7220-21cut08:05:223
1178499128,3cyclictest7220-21cut08:05:213
1178499128,3cyclictest28602-21sed07:45:003
1178499128,3cyclictest27090-21ls10:15:013
1178499120,12cyclictest0-21swapper/312:26:013
1178099128,2cyclictest0-21swapper/207:35:132
1178099124,4cyclictest224-21systemd-journal07:52:542
1178099124,2cyclictest0-21swapper/207:42:532
1178099123,6cyclictest31474-21awk08:55:212
1177899129,2cyclictest6760-21grep09:35:241
1177899129,2cyclictest4882-21gpgv09:10:001
1177899129,2cyclictest4356-21cut11:45:161
1177899129,2cyclictest28410-21sort08:50:171
1177899129,2cyclictest20798-21grep07:25:311
1177899129,2cyclictest19944-21ssh11:55:161
1177899129,2cyclictest12977-21cron08:20:001
1177899129,2cyclictest10435-21ssh12:30:191
1177899127,4cyclictest25-21ksoftirqd/110:35:181
1177899124,4cyclictest224-21systemd-journal12:24:551
1177599129,2cyclictest29290-21uniq07:45:150
1177599129,2cyclictest17839-21ls09:20:000
1177599128,3cyclictest13-21ksoftirqd/011:45:230
1177599128,2cyclictest4829-21diskmemload10:01:220
1177599128,1cyclictest32747-21rm09:30:270
1177599127,3cyclictest161rcuc/008:05:190
1177599127,3cyclictest161rcuc/008:05:190
1177599126,5cyclictest4414-21kworker/0:111:25:180
1177599123,8cyclictest161rcuc/012:23:080
1177599121,5cyclictest224-21systemd-journal08:40:000
1177599121,1cyclictest141rcu_preempt09:14:130
11775991211,1cyclictest13-21ksoftirqd/009:50:280
11775991211,1cyclictest13-21ksoftirqd/009:50:270
11775991210,2cyclictest13-21ksoftirqd/012:12:460
11775991210,1cyclictest14392-21apt-get07:15:130
1178499118,2cyclictest9778-21ssh10:25:063
1178499118,2cyclictest680-21rm11:42:433
1178499118,2cyclictest5846-21sh08:05:003
1178499118,2cyclictest5723-21cut09:35:173
1178499118,2cyclictest32053-21wc07:50:193
1178499118,2cyclictest3198-21sort10:20:243
1178499118,2cyclictest3014-21ssh10:40:493
1178499118,2cyclictest29448-21head07:45:173
1178499118,2cyclictest28544-21ssh10:57:483
1178499118,2cyclictest25466-21cut08:45:153
1178499118,2cyclictest25209-21ssh10:35:103
1178499118,2cyclictest24575-21ssh09:25:073
1178499118,2cyclictest21010-21tr11:55:233
1178499118,2cyclictest14294-21grep09:40:293
1178499118,2cyclictest10991-21ssh11:28:233
1178499113,4cyclictest18489-21cpuspeed_turbos12:15:153
1178099118,2cyclictest2892-21taskset07:57:072
1178099116,4cyclictest23658-21fschecks_time08:40:172
1178099113,7cyclictest11568-21tail08:15:162
1177899118,2cyclictest9393-21expr10:45:181
1177899118,2cyclictest8621-21ssh09:11:431
1177899118,2cyclictest5458-21ssh09:58:351
1177899118,2cyclictest5458-21ssh09:58:351
1177899118,2cyclictest32002-21grep08:55:271
1177899118,2cyclictest30811-21perf07:50:001
1177899118,2cyclictest30036-21ssh11:20:041
1177899118,2cyclictest2514-21cat07:55:291
1177899118,2cyclictest22809-21cat11:15:151
1177899118,2cyclictest19575-21grep10:30:271
1177899118,2cyclictest18495-21grep12:35:281
1177899118,2cyclictest17343-21cat10:30:001
1177899118,2cyclictest14893-21ssh12:12:371
1177899118,2cyclictest12971-21expr11:50:201
1177899118,2cyclictest12964-21expr09:40:161
1177899118,2cyclictest12452-21expr07:10:151
1177899115,4cyclictest472-21avahi-daemon10:51:111
1177899112,6cyclictest26026-21cron07:40:001
1177899111,2cyclictest141rcu_preempt11:28:121
1177899110,10cyclictest0-21swapper/110:16:401
1177599118,2cyclictest31689-21aten2_r7power_v11:00:130
1177599118,2cyclictest31306-21grep07:50:150
1177599118,2cyclictest25825-21ssh11:16:400
1177599118,2cyclictest18309-21grep08:30:140
1177599118,2cyclictest17733-21ssh10:50:220
1177599118,2cyclictest15649-21dpkg08:25:100
1177599118,2cyclictest13818-21df08:20:170
1177599118,1cyclictest2742-21ssh09:55:550
1177599118,1cyclictest2742-21ssh09:55:550
1177599118,1cyclictest21206-21ssh09:45:390
1177599117,4cyclictest161rcuc/010:30:130
1177599117,3cyclictest27080-21sed08:45:260
1177599113,4cyclictest224-21systemd-journal07:30:000
1177599110,1cyclictest141rcu_preempt08:43:010
1178499108,1cyclictest4693-21taskset09:08:403
1178499108,1cyclictest23959-21basename07:35:143
1178499108,1cyclictest20064-21cat07:25:243
1178499108,1cyclictest198982sleep308:32:073
1178499108,1cyclictest12007-21apache_volume07:10:123
1178499107,2cyclictest10896-21grep08:15:143
1178499106,3cyclictest30582-21grep08:55:153
1177899108,1cyclictest7316-21cut08:05:231
1177899108,1cyclictest7316-21cut08:05:231
1177899108,1cyclictest4063-21sort08:00:171
1177899108,1cyclictest17839-21cat08:30:001
1177899108,1cyclictest14408-21ntpq08:20:221
1177899107,2cyclictest32749-21grep07:50:271
1177899107,2cyclictest32431-21ntpq09:30:231
1177899107,2cyclictest17708-21uname07:20:181
1177899100,9cyclictest26500-21egrep07:40:131
1177899100,9cyclictest26437-21sed08:45:201
1177599108,2cyclictest12033-21open_files08:15:210
1177599108,1cyclictest9614-21grep08:10:210
1177599108,1cyclictest30417-21wget08:55:130
1177599108,1cyclictest2595-21ls07:55:290
1177599107,2cyclictest13-21ksoftirqd/007:39:590
1177599101,1cyclictest141rcu_preempt07:44:550
117789997,1cyclictest20504-21dpkg08:35:011
117759998,1cyclictest12196-21cpuspeed_turbos07:10:140
117759993,5cyclictest4621-21grep08:00:220
117759991,3cyclictest0-21swapper/008:50:140
117849981,5cyclictest15079-21fschecks_time07:15:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional