You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-19 - 04:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Jun 19, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3253729861,33sleep00-21swapper/019:06:090
3251528760,9sleep30-21swapper/319:05:523
33828270,8sleep20-21swapper/219:09:362
3267528063,13sleep10-21swapper/119:07:541
226752470,0sleep30-21swapper/323:03:013
47099290,28cyclictest0-21swapper/223:01:452
474992827,1cyclictest0-21swapper/320:15:123
47099280,2cyclictest0-21swapper/221:30:032
47099280,28cyclictest0-21swapper/223:41:052
47099280,1cyclictest0-21swapper/200:25:362
47099270,26cyclictest0-21swapper/221:58:042
47099237,1cyclictest7489-21python00:10:242
47099232,4cyclictest6559-21/usr/sbin/munin22:15:142
47099232,3cyclictest26160-21diskmemload22:03:302
47099231,1cyclictest0-21swapper/223:32:282
47099230,4cyclictest0-21swapper/223:22:592
47099230,18cyclictest0-21swapper/222:51:002
46799230,2cyclictest0-21swapper/122:23:131
46799230,1cyclictest0-21swapper/122:03:331
47099223,2cyclictest225-21systemd-journal20:40:152
470992222,0cyclictest0-21swapper/222:10:362
47099220,20cyclictest0-21swapper/200:21:122
46799220,2cyclictest0-21swapper/123:00:581
47099213,2cyclictest0-21swapper/200:35:142
47099210,5cyclictest0-21swapper/222:45:042
47099210,21cyclictest0-21swapper/221:19:482
467992119,1cyclictest0-21swapper/119:55:121
46499210,0cyclictest0-21swapper/021:29:120
47499202,17cyclictest25529-21sh22:25:303
47099200,4cyclictest0-21swapper/223:10:252
47099200,1cyclictest0-21swapper/223:50:142
47099200,1cyclictest0-21swapper/222:08:002
46499200,1cyclictest141rcu_preempt23:14:290
6712190,0sleep20-21swapper/221:13:532
47499197,4cyclictest26160-21diskmemload23:10:003
464991919,0cyclictest0-21swapper/022:06:200
47499183,14cyclictest5875-21ssh00:10:133
47099182,2cyclictest26160-21diskmemload22:27:562
470991816,2cyclictest0-21swapper/223:25:302
470991815,3cyclictest0-21swapper/221:35:202
47099180,18cyclictest0-21swapper/222:32:422
467991816,1cyclictest26160-21diskmemload00:29:321
467991814,3cyclictest38150irq/131-snd_hda22:28:381
46799180,18cyclictest0-21swapper/121:15:321
46499180,16cyclictest0-21swapper/021:56:380
289962180,0sleep30-21swapper/321:50:043
47499173,7cyclictest0-21swapper/323:25:123
474991712,2cyclictest22763-21df_abs22:05:153
47499170,0cyclictest0-21swapper/321:49:013
467991717,0cyclictest0-21swapper/100:22:171
47499160,4cyclictest23249-21kworker/u8:221:25:293
47499160,3cyclictest0-21swapper/322:57:113
47499160,3cyclictest0-21swapper/322:22:543
47499160,16cyclictest0-21swapper/321:20:253
47099168,7cyclictest18041-21grep22:40:252
47099163,5cyclictest4784-21latency_hist21:55:002
47099162,4cyclictest27637-21tr20:05:142
47099160,6cyclictest0-21swapper/223:58:522
47099160,2cyclictest0-21swapper/223:45:252
47099160,2cyclictest0-21swapper/200:17:112
47099160,1cyclictest0-21swapper/200:04:242
47099160,16cyclictest0-21swapper/222:55:412
47099160,0cyclictest0-21swapper/221:45:302
46799169,6cyclictest28022-21grep23:25:171
467991613,2cyclictest8714-21df_abs19:25:151
46499164,7cyclictest23785-21perf21:05:010
46499164,6cyclictest7315-21idleruntime-cro22:35:000
85722150,0sleep10-21swapper/100:11:001
474991512,1cyclictest0-21swapper/322:10:133
47499150,13cyclictest5434-21cron19:20:003
47099158,6cyclictest31452-21tail20:10:252
47099158,6cyclictest29480-21sh23:06:462
47099158,6cyclictest17879-21wc20:50:202
47099158,2cyclictest21833-21ssh22:24:102
47099158,2cyclictest15027-21sendmail19:40:002
47099158,2cyclictest13537-21ssh23:35:272
47099158,2cyclictest10269-21ssh23:15:082
47099152,3cyclictest225-21systemd-journal21:40:282
470991512,2cyclictest22443-21hddtemp_smartct21:00:172
470991511,3cyclictest6118-21grep20:25:252
470991511,3cyclictest6118-21grep20:25:252
47099150,4cyclictest225-21systemd-journal20:00:002
47099150,0cyclictest0-21swapper/200:34:032
46799153,8cyclictest0-21swapper/122:45:121
467991513,2cyclictest25-21ksoftirqd/122:14:151
46799150,2cyclictest0-21swapper/122:56:481
46499154,0cyclictest141rcu_preempt21:11:120
46499152,6cyclictest20789-21munin-run00:00:000
464991512,2cyclictest0-21swapper/023:35:130
47499141,2cyclictest0-21swapper/323:35:383
474991410,3cyclictest14239-21cron20:45:003
47099148,2cyclictest22545-21ssh21:26:122
47099147,2cyclictest15815-21sed19:40:152
47099141,9cyclictest0-21swapper/221:21:342
47099141,2cyclictest225-21systemd-journal19:55:002
470991411,2cyclictest0-21swapper/222:35:182
467991410,3cyclictest29409-21head21:30:221
46799140,1cyclictest0-21swapper/119:15:141
46799140,14cyclictest0-21swapper/121:43:191
46499145,5cyclictest7752-21/usr/sbin/munin22:35:230
464991413,1cyclictest14872-21apt-get00:35:000
464991411,2cyclictest20696-21ssh23:40:170
46499140,14cyclictest0-21swapper/023:19:190
47499139,4cyclictest39-21ksoftirqd/300:15:103
47499136,7cyclictest6420-21apache219:55:253
474991310,2cyclictest5011-21grep23:30:253
474991310,2cyclictest2968-21grep23:10:203
474991310,2cyclictest25647-21grep21:05:263
474991310,2cyclictest21020-21sed19:50:163
474991310,2cyclictest18063-21grep22:40:263
47099139,3cyclictest24306-21grep19:55:252
47099139,3cyclictest15657-21tr20:45:232
47099138,1cyclictest75882sleep219:21:132
47099137,1cyclictest54152sleep219:19:492
47099137,1cyclictest3371-21awk20:20:202
47099137,1cyclictest1678-21cut19:10:182
47099136,2cyclictest11288-21cut19:30:162
470991310,2cyclictest20499-21tr20:55:222
47099130,4cyclictest452-21systemd-logind00:05:192
47099130,4cyclictest225-21systemd-journal21:10:002
47099130,4cyclictest225-21systemd-journal20:15:162
46799139,3cyclictest12939-21grep19:35:131
46799138,4cyclictest2204-21ssh22:31:051
46799138,3cyclictest10477-21basename23:15:131
467991310,2cyclictest27469-21ls21:30:001
467991310,2cyclictest20522-21grep19:50:131
467991310,2cyclictest19986-21cron19:50:001
467991310,2cyclictest15301-21sed20:45:181
467991310,2cyclictest1278-21df19:10:141
467991310,2cyclictest12274-21cut20:40:121
467991310,2cyclictest10794-21wc20:35:221
46499138,4cyclictest0-21swapper/023:50:030
46499135,4cyclictest225-21systemd-journal21:00:000
46499133,7cyclictest26160-21diskmemload21:18:130
464991310,2cyclictest30047-21grep20:10:130
464991310,2cyclictest21631-21grep00:00:150
464991310,2cyclictest16419-21grep00:35:170
464991310,2cyclictest13-21ksoftirqd/021:49:090
46499130,13cyclictest0-21swapper/023:00:580
46499130,12cyclictest1217-21cat20:15:240
47499129,2cyclictest9441-21ssh22:16:043
47499129,2cyclictest4050-21head21:15:223
47499129,2cyclictest32235-21grep00:25:223
47499129,2cyclictest29508-21cron20:10:003
47499129,2cyclictest20011-21sed20:55:173
47499129,2cyclictest17119-21tr20:50:153
47499129,2cyclictest15762-21ssh22:40:013
47499129,2cyclictest12308-21grep23:15:263
47499122,8cyclictest20385-21sh22:03:583
474991212,0cyclictest0-21swapper/322:53:523
474991210,1cyclictest28788-21cut23:45:153
47499120,11cyclictest24801-21tr22:45:133
47499120,11cyclictest0-21swapper/323:54:563
47099126,1cyclictest18831-21cut19:45:172
47099125,2cyclictest11788-21sort20:40:002
47099120,5cyclictest8766-21unixbench-2d20:30:252
46799129,2cyclictest9286-21basename20:35:001
46799129,2cyclictest6545-21sed19:20:151
46799129,2cyclictest29254-21ssh22:09:201
46799129,2cyclictest26105-21cstates23:05:161
46799129,2cyclictest21596-21tr00:00:151
46799129,2cyclictest1994-21grep23:10:141
46799129,2cyclictest1869-21perf20:20:001
46799129,2cyclictest13196-21tr23:55:141
46799129,2cyclictest11834-21grep21:20:161
46799129,2cyclictest1103-21sed22:50:151
46799126,2cyclictest225-21systemd-journal19:45:001
467991210,2cyclictest25-21ksoftirqd/100:18:271
46799120,12cyclictest10195-21ssh23:34:201
46799120,11cyclictest9752-21sed22:35:281
46499129,2cyclictest729-21apache200:16:260
46499129,2cyclictest3801-21sort23:30:160
46499129,2cyclictest29669-21sort23:45:200
46499129,2cyclictest26409-21grep22:45:260
46499129,2cyclictest24549-21cat22:25:190
46499129,2cyclictest20766-21head23:20:270
46499129,2cyclictest15520-21grep19:40:140
46499128,2cyclictest5374-21turbostat00:10:000
46499120,5cyclictest0-21swapper/022:57:180
47499118,2cyclictest8252-21ssh00:30:193
47499118,2cyclictest7563-21ssh21:55:413
47499118,2cyclictest7179-21python19:20:233
47499118,2cyclictest5058-21df20:25:153
47499118,2cyclictest5058-21df20:25:153
47499118,2cyclictest4245-21sh20:25:003
47499118,2cyclictest32700-21sh22:30:183
47499118,2cyclictest30701-21uname20:10:173
47499118,2cyclictest29867-21grep21:30:273
47499118,2cyclictest28186-21which21:10:223
47499118,2cyclictest24964-21seq00:20:443
47499118,2cyclictest24915-21sh00:01:423
47499118,2cyclictest2451-21head19:10:253
47499118,2cyclictest21060-21sh23:20:393
47499118,2cyclictest18878-21ls19:45:183
47499118,2cyclictest17603-21ssh23:57:323
47499118,2cyclictest16909-21cat00:35:213
47499118,2cyclictest16389-21grep19:40:193
47499118,2cyclictest15486-21rm21:41:173
47499114,4cyclictest21908-21/usr/sbin/munin23:40:273
47499111,9cyclictest30454-21fschecks_time00:05:163
47499110,10cyclictest4910-21ssh21:35:183
46799119,1cyclictest99192sleep121:57:311
46799118,2cyclictest957-21ssh23:47:431
46799118,2cyclictest4250-21grep21:35:141
46799118,2cyclictest3537-21ssh21:54:081
46799118,2cyclictest28517-21cut20:05:201
46799118,2cyclictest27327-21cat20:05:001
46799118,2cyclictest24038-21ssh23:42:001
46799118,2cyclictest2044-21rm21:14:571
46799118,2cyclictest19820-21ssh23:20:191
46799118,2cyclictest18575-21sed20:50:271
46799118,2cyclictest13254-21ssh23:35:261
46799118,2cyclictest11501-21ssh22:17:401
46799117,3cyclictest11643-21ssh23:54:271
46799115,5cyclictest16251-21chrt00:35:161
46799111,6cyclictest455-21dbus-daemon00:05:191
46799110,7cyclictest225-21systemd-journal20:20:111
46799110,1cyclictest0-21swapper/119:30:131
46499118,2cyclictest904-21cat22:50:150
46499118,2cyclictest8085-21cut20:30:180
46499118,2cyclictest6308-21ssh22:15:020
46499118,2cyclictest6097-21ssh00:29:430
46499118,2cyclictest5862-21cut20:25:220
46499118,2cyclictest5862-21cut20:25:220
46499118,2cyclictest5452-21dump-pmu-power19:20:000
46499118,2cyclictest30016-21grep21:50:150
46499118,2cyclictest29685-21sh23:25:490
46499118,2cyclictest27859-21sh21:30:080
46499118,2cyclictest2548-21expr20:20:130
46499118,2cyclictest24321-21grep21:05:150
46499118,2cyclictest22497-21ssh00:20:110
46499118,2cyclictest21707-21turbostat22:05:000
46499118,2cyclictest20950-21tune2fs19:50:150
46499118,2cyclictest20103-21wc19:50:000
46499118,2cyclictest16442-21ls22:20:230
46499118,2cyclictest13709-21ntpq21:40:200
46499118,2cyclictest12252-21basename20:40:120
46499118,2cyclictest11269-21cut21:20:140
46499118,2cyclictest10621-21grep20:35:200
46499117,3cyclictest32066-21wc22:10:220
46499115,4cyclictest29521-21munin-run20:10:000
46499115,1cyclictest5866-21apt-get00:10:130
46499112,7cyclictest16169-21apache_volume22:40:120
46499110,2cyclictest0-21swapper/023:05:170
148882110,3sleep114880-21munin-run00:35:001
47499108,1cyclictest165082sleep320:48:413
47499108,1cyclictest11351-21basename19:30:173
47499107,2cyclictest8770-21cut19:25:163
47499107,2cyclictest13365-21expr19:35:143
47499107,2cyclictest11891-21cat20:40:013
47499100,9cyclictest7584-21expr20:30:143
47499100,9cyclictest26528-21sed20:00:243
47099107,2cyclictest8435-21tr19:25:142
46799108,1cyclictest6254-21awk20:25:261
46799108,1cyclictest6254-21awk20:25:261
46799108,1cyclictest24300-21cut21:05:151
46799108,1cyclictest22119-21ssh21:45:211
46799107,2cyclictest22813-21ntpq21:00:211
46799107,1cyclictest30935-21sh20:10:201
46799103,6cyclictest6432-21apache222:40:341
46499108,1cyclictest3713-21ssh21:35:030
46499108,1cyclictest17532-21tune2fs20:50:170
46499108,1cyclictest15012-21sed20:45:160
46499107,2cyclictest9911-21taskset19:25:330
46499107,2cyclictest26127-21sed20:00:180
46499107,2cyclictest22780-21wc19:55:130
46499107,2cyclictest1497-21sed19:10:160
46499105,2cyclictest225-21systemd-journal19:39:550
4749990,8cyclictest455-21dbus-daemon21:00:233
4679992,1cyclictest19266-21apt-get20:55:141
4649997,1cyclictest6213-21sed19:20:130
4649997,1cyclictest10492-21chrt19:30:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional