You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 21:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Nov 27, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
10260210489,10sleep30-21swapper/307:06:573
10196210087,9sleep10-21swapper/107:06:071
1018728464,15sleep20-21swapper/207:06:002
1048428167,9sleep00-21swapper/007:09:500
61792550,0sleep30-21swapper/311:51:513
159132520,0sleep10-21swapper/110:13:391
1059399310,28cyclictest0-21swapper/308:30:003
1059399280,2cyclictest0-21swapper/312:30:143
1059399272,22cyclictest0-21swapper/309:35:123
1059399270,26cyclictest0-21swapper/311:56:173
1059399251,1cyclictest3880-21apt-get09:10:143
1059399250,24cyclictest3426-21diskmemload10:17:033
1059399250,1cyclictest0-21swapper/310:00:373
1059399242,17cyclictest27358-21fschecks_count10:50:173
1059399241,2cyclictest0-21swapper/309:25:473
1059399241,1cyclictest3426-21diskmemload09:54:403
1059399241,1cyclictest0-21swapper/311:41:203
1059399240,23cyclictest0-21swapper/312:29:163
1059399236,16cyclictest3426-21diskmemload11:05:143
1059399233,19cyclictest3426-21diskmemload12:17:043
1059399230,2cyclictest0-21swapper/312:13:283
1059399230,1cyclictest0-21swapper/311:10:013
1059399230,1cyclictest0-21swapper/308:15:143
1059399222,19cyclictest21349-21awk08:40:003
1059399220,1cyclictest0-21swapper/312:24:573
1059399211,1cyclictest3426-21diskmemload12:04:073
1059399211,19cyclictest3426-21diskmemload09:46:343
1059399210,1cyclictest0-21swapper/311:23:063
1059399210,17cyclictest0-21swapper/310:23:433
1058899210,2cyclictest37850irq/131-snd_hda11:15:582
1058599210,4cyclictest0-21swapper/111:18:421
1059399202,17cyclictest12924-21idleruntime-cro07:15:003
1059399201,2cyclictest3426-21diskmemload09:57:173
1059399201,1cyclictest29711-21sh10:25:143
1059399200,1cyclictest0-21swapper/311:45:373
1059399193,15cyclictest30273-21ssh09:31:243
1059399191,2cyclictest23959-21/usr/sbin/munin12:05:213
1059399190,3cyclictest20673-21tr07:30:143
10588991911,6cyclictest17380-21apache209:34:232
1058899190,19cyclictest0-21swapper/212:13:532
1058899190,14cyclictest0-21swapper/210:07:332
1059399188,7cyclictest13667-21ssh09:17:563
1059399181,14cyclictest2915-21aten2_r7power_p10:30:133
1059399180,2cyclictest0-21swapper/308:30:223
1059399180,1cyclictest0-21swapper/310:13:213
1059399180,17cyclictest0-21swapper/312:37:243
1059399180,17cyclictest0-21swapper/311:33:363
1059399180,15cyclictest0-21swapper/308:00:013
10588991815,3cyclictest12484-21kworker/2:209:20:042
10588991814,4cyclictest0-21swapper/211:28:172
1058899180,18cyclictest0-21swapper/212:02:042
1059399172,14cyclictest3426-21diskmemload10:41:043
1059399172,10cyclictest3426-21diskmemload11:35:013
1059399171,1cyclictest1252-21latency_hist09:04:593
1059399170,2cyclictest2621-21sh10:56:003
1059399170,17cyclictest0-21swapper/310:09:153
1059399170,16cyclictest0-21swapper/308:40:133
1059399170,16cyclictest0-21swapper/308:00:123
1058899170,2cyclictest0-21swapper/209:39:252
1058899170,17cyclictest0-21swapper/211:59:472
1059399162,13cyclictest16133-21cut09:20:163
1059399161,14cyclictest0-21swapper/307:20:143
10593991610,5cyclictest62050irq/128-enp2s0-10:37:183
10593991610,4cyclictest13545-21ls09:45:013
1059399160,3cyclictest224-21systemd-journal08:07:123
1059399160,1cyclictest3426-21diskmemload11:17:563
1059399160,1cyclictest0-21swapper/308:10:253
1059399160,1cyclictest0-21swapper/307:45:133
1059399160,16cyclictest0-21swapper/311:25:153
10588991614,1cyclictest32-21ksoftirqd/210:53:242
10588991611,2cyclictest311rcuc/207:55:282
1058899160,0cyclictest0-21swapper/210:40:102
10585991613,2cyclictest25-21ksoftirqd/109:18:381
10583991614,1cyclictest25369-21ntp_states11:15:220
1059399151,1cyclictest0-21swapper/309:05:183
1059399151,12cyclictest0-21swapper/310:49:183
1058899153,8cyclictest0-21swapper/211:05:132
1058899151,8cyclictest224-21systemd-journal09:00:002
10588991514,1cyclictest4317-21kworker/u8:208:25:432
10588991512,2cyclictest27717-21grep09:55:272
1058899150,1cyclictest0-21swapper/212:19:152
10585991512,3cyclictest10550irq/122-ahci[0008:14:591
1058399157,7cyclictest13556-21perf09:45:010
10583991511,3cyclictest11784-21tr08:20:000
1058399150,14cyclictest0-21swapper/011:55:440
1059399148,5cyclictest30775-21taskset08:56:203
1059399148,5cyclictest30345-21ls07:50:183
1059399148,5cyclictest28913-21cat08:55:003
1059399148,5cyclictest24171-21chrt08:45:103
1059399147,6cyclictest7519-21sensors_fan11:00:243
1059399142,11cyclictest14556-21which07:15:233
1059399142,10cyclictest17984-21taskset07:25:013
1059399141,4cyclictest224-21systemd-journal07:40:003
1059399140,0cyclictest0-21swapper/307:40:243
1058899149,2cyclictest2405-21sh10:30:002
1058899141,6cyclictest224-21systemd-journal12:40:002
10588991411,3cyclictest37850irq/131-snd_hda11:11:152
1058899140,1cyclictest0-21swapper/209:11:222
1058899140,14cyclictest0-21swapper/212:33:302
1058899140,14cyclictest0-21swapper/212:05:352
1058599149,3cyclictest30450-21ssh09:58:351
10585991410,3cyclictest19298-21basename08:35:121
1058399143,7cyclictest21831-21apache_volume08:40:120
1058399141,6cyclictest7006-21idleruntime-cro08:10:000
1058399140,1cyclictest0-21swapper/012:20:130
1058399140,13cyclictest0-21swapper/008:30:130
1058899135,6cyclictest0-21swapper/210:55:132
10588991313,0cyclictest0-21swapper/212:28:322
10588991311,1cyclictest32-21ksoftirqd/209:53:532
10588991311,1cyclictest311rcuc/210:45:332
10588991311,1cyclictest0-21swapper/210:15:152
10588991310,2cyclictest9792-21tail10:35:182
10588991310,2cyclictest8301-21sort09:40:182
10588991310,2cyclictest17297-21grep07:20:262
1058899130,1cyclictest0-21swapper/210:11:312
1058899130,13cyclictest3426-21diskmemload09:49:352
1058599139,3cyclictest7862-21tr09:40:161
1058599139,3cyclictest6102-21head08:05:211
1058599139,3cyclictest24701-21df08:45:191
1058599139,3cyclictest1743-21grep07:55:281
1058599138,5cyclictest17380-21apache210:27:261
10585991310,2cyclictest29467-21grep09:30:291
10585991310,2cyclictest23948-21turbostat08:45:001
10585991310,2cyclictest21247-21head09:50:191
10585991310,2cyclictest1480-21grep10:00:271
10585991310,2cyclictest13478-21tr07:15:151
10585991310,2cyclictest11663-21cut11:05:131
1058599130,1cyclictest0-21swapper/112:35:391
1058599130,12cyclictest6050-21ssh11:51:411
1058399139,3cyclictest6348-21grep12:15:270
1058399139,2cyclictest239982sleep010:20:190
1058399139,2cyclictest11875-21grep11:05:140
10583991311,2cyclictest13-21ksoftirqd/010:06:410
10583991310,2cyclictest30770-21tune2fs07:50:200
10583991310,2cyclictest25662-21grep07:40:170
10583991310,2cyclictest24931-21grep11:40:220
1058399130,7cyclictest9382-21idleruntime-cro08:15:000
1058399130,13cyclictest0-21swapper/011:54:500
1059399126,5cyclictest12152-21cut08:20:133
1058899129,2cyclictest8870-21grep08:10:252
1058899129,2cyclictest5590-21grep08:05:162
1058899129,2cyclictest3488-21grep11:50:142
1058899129,2cyclictest28853-21sleep10:25:002
1058899129,2cyclictest25818-21grep08:45:292
1058899129,2cyclictest24697-21grep07:35:252
1058899129,2cyclictest2417-21grep09:05:222
1058899129,2cyclictest23459-21grep09:25:272
1058899129,2cyclictest19965-21grep08:35:142
1058899129,2cyclictest18966-21grep11:35:272
1058899129,2cyclictest18707-21sort07:25:162
1058899129,2cyclictest13916-21tail07:15:172
1058899129,2cyclictest11337-21grep08:15:272
1058899129,2cyclictest10093-21grep09:15:142
1058899128,3cyclictest21193-21head07:30:172
1058899127,5cyclictest2940-21diskmemload10:01:532
10588991210,2cyclictest32-21ksoftirqd/211:47:282
10588991210,1cyclictest6796-21grep11:00:182
10588991210,1cyclictest25400-21ssh11:40:272
10588991210,1cyclictest12743-21tail08:20:162
1058899120,12cyclictest0-21swapper/211:20:162
1058599129,2cyclictest9911-21tr08:15:151
1058599129,2cyclictest6797-21sed11:00:181
1058599129,2cyclictest5222-21head11:25:201
1058599129,2cyclictest29240-21perf11:45:011
1058599129,2cyclictest26797-21cut10:50:141
1058599129,2cyclictest23304-21cut10:20:151
1058599129,2cyclictest23125-21cron11:40:011
1058599129,2cyclictest1699-21tr10:55:191
1058599129,2cyclictest11188-21sed11:55:301
1058599128,3cyclictest19086-21cat07:25:191
1058599128,2cyclictest26815-21tr08:50:161
1058599126,6cyclictest17380-21apache209:38:551
1058599126,6cyclictest17379-21apache210:39:301
1058599126,2cyclictest224-21systemd-journal10:20:001
1058599123,2cyclictest0-21swapper/107:45:121
1058399129,2cyclictest5668-21sed09:10:270
1058399129,2cyclictest4714-21grep11:25:180
1058399129,2cyclictest29723-21grep08:55:170
1058399129,2cyclictest29147-21cat12:10:000
1058399129,2cyclictest28730-21ssh12:30:130
1058399129,2cyclictest2259-21sort10:55:360
1058399129,2cyclictest10652-21ssh11:30:030
1058399128,3cyclictest32280-21sed11:20:270
1058399128,3cyclictest2856-21perf11:49:590
1058399128,3cyclictest27361-21grep08:50:190
1058399128,3cyclictest2420-21perf10:30:000
1058399128,3cyclictest23545-21grep07:35:140
1058399128,2cyclictest11548-21grep07:10:170
1058399124,4cyclictest224-21systemd-journal08:00:010
1058399121,10cyclictest0-21swapper/011:13:580
1058399120,1cyclictest0-21swapper/009:35:230
1058399120,11cyclictest16749-21rm09:46:210
1058399120,11cyclictest15700-21grep10:40:170
1058899118,2cyclictest3609-21head10:30:162
1058899118,2cyclictest3580-21needreboot08:00:172
1058899118,2cyclictest32210-21chrt07:54:322
1058899118,2cyclictest28653-21chrt08:54:372
1058899118,2cyclictest25806-21expr07:40:172
1058899118,2cyclictest15904-21rm12:21:362
1058899118,2cyclictest11087-21cat11:30:152
1058899110,2cyclictest224-21systemd-journal08:35:002
1058899110,10cyclictest22194-21expr08:40:142
1058599118,2cyclictest5645-21ssh10:05:141
1058599118,2cyclictest4691-21grep12:15:161
1058599118,2cyclictest31282-21ntpq07:50:231
1058599118,2cyclictest30659-21sed12:30:271
1058599118,2cyclictest29576-21ssh11:45:091
1058599118,2cyclictest2836-21ssh11:24:001
1058599118,2cyclictest2813-21ssh12:14:041
1058599118,2cyclictest26495-21cat07:40:221
1058599118,2cyclictest25142-21tr07:40:001
1058599118,2cyclictest24683-21ssh09:26:561
1058599118,2cyclictest23227-21ssh12:25:561
1058599118,2cyclictest22444-21ssh12:05:051
1058599118,2cyclictest21113-21grep07:30:151
1058599118,2cyclictest20387-21ssh12:03:101
1058599118,2cyclictest20127-21ssh11:10:481
1058599118,2cyclictest19491-21ssh09:49:281
1058599118,2cyclictest1918-21sort09:05:191
1058599118,2cyclictest18103-21wc08:30:211
1058599118,2cyclictest16753-21sed10:40:281
1058599118,2cyclictest16217-21cut07:20:171
1058599118,2cyclictest12485-21sed07:10:261
1058599118,2cyclictest12269-21ssh12:20:031
1058599118,2cyclictest12191-21head11:30:221
1058599110,10cyclictest4841-21ssh10:30:361
1058599110,10cyclictest23066-21ssh10:46:141
1058599110,10cyclictest16167-21rm09:20:151
1058399118,2cyclictest7184-21grep11:00:200
1058399118,2cyclictest7054-21taskset10:33:140
1058399118,2cyclictest4761-21ls08:05:000
1058399118,2cyclictest4075-21ssh12:35:020
1058399118,2cyclictest32585-21ssh12:11:460
1058399118,2cyclictest31765-21taskset09:00:130
1058399118,2cyclictest30831-21ssh09:32:080
1058399118,2cyclictest28547-21grep07:45:160
1058399118,2cyclictest2275-21ssh10:01:110
1058399118,2cyclictest21874-21tr10:45:240
1058399118,2cyclictest21648-21ssh12:25:170
1058399118,2cyclictest19975-21tail08:35:150
1058399118,2cyclictest19217-21wc10:15:290
1058399118,2cyclictest18190-21cut11:35:200
1058399118,2cyclictest17903-21ssh09:20:390
1058399118,2cyclictest17225-21unixbench_multi07:20:260
1058399118,2cyclictest15371-21ssh12:00:050
1058399118,2cyclictest12174-21tr08:20:130
1058399118,2cyclictest11711-21ssh09:15:290
1058399117,3cyclictest14462-21cron10:40:000
1058399115,3cyclictest22444-21ssh09:50:430
1058399111,5cyclictest0-21swapper/009:55:160
1058399110,1cyclictest0-21swapper/009:25:380
1058399110,10cyclictest27556-21idleruntime10:50:180
1058399110,10cyclictest0-21swapper/010:11:030
1058899108,1cyclictest31560-21cat09:00:132
1058899108,1cyclictest298612sleep207:49:552
1058899108,1cyclictest12857-21taskset07:14:152
1058599108,1cyclictest31358-21apt09:00:111
1058599107,2cyclictest4120-21ls09:10:161
1058599107,2cyclictest29083-21cut08:55:141
1058599107,2cyclictest2642-21egrep08:00:121
1058599107,2cyclictest16432-21chrt08:29:011
1058599107,2cyclictest12240-21cat08:20:131
1058399107,2cyclictest3225-21taskset09:07:350
1058399107,2cyclictest26197-21chrt08:48:350
1058399107,2cyclictest18698-21cut07:25:160
105839997,1cyclictest14488-21aten2_r7power_p08:25:130
105839997,1cyclictest14127-21cut07:15:190
105839996,2cyclictest21330-21uname07:30:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional