You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 12:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Tue Jul 07, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1471829483,6sleep00-21swapper/019:06:050
1490829278,10sleep30-21swapper/319:08:343
1481929274,13sleep10-21swapper/119:07:231
1477128977,8sleep20-21swapper/219:06:482
1510899300,15cyclictest0-21swapper/223:15:442
1510499300,1cyclictest0-21swapper/122:25:271
1510499280,3cyclictest29596-21ssh22:40:211
1510499271,25cyclictest5940-21diskmemload23:52:541
1510499270,3cyclictest18086-21cron22:15:001
1510499261,4cyclictest0-21swapper/123:45:151
15108992524,1cyclictest0-21swapper/222:47:172
1510499259,3cyclictest38150irq/131-snd_hda00:05:201
1510499252,2cyclictest17405-21apache_processe21:55:131
1510499252,2cyclictest17405-21apache_processe21:55:131
1510499251,21cyclictest29195-21grep19:40:191
1510499241,4cyclictest0-21swapper/100:15:201
1509899245,3cyclictest225-21systemd-journal22:00:420
1510499233,19cyclictest1678-21proc_pri19:50:211
1510499232,2cyclictest5940-21diskmemload22:21:381
1510499231,3cyclictest7900-21ssh23:25:201
1510499230,18cyclictest0-21swapper/121:45:131
1510499223,15cyclictest38150irq/131-snd_hda19:45:111
15104992222,0cyclictest0-21swapper/123:04:361
1510499221,20cyclictest0-21swapper/121:15:151
1510499220,2cyclictest5940-21diskmemload22:35:251
1510499220,1cyclictest0-21swapper/119:35:131
1510899210,21cyclictest0-21swapper/200:39:102
1510899210,13cyclictest0-21swapper/200:20:482
1510499212,1cyclictest9204-21ntp_states21:30:231
1510499212,1cyclictest5940-21diskmemload00:37:091
1510499211,5cyclictest5940-21diskmemload23:10:201
1510499211,4cyclictest5940-21diskmemload23:40:051
1510499210,4cyclictest29625-21sh22:01:511
1510499210,20cyclictest0-21swapper/123:30:151
1510499210,20cyclictest0-21swapper/123:30:151
1510499210,1cyclictest0-21swapper/100:27:061
1509899217,8cyclictest13954-21-browsed00:07:070
15098992120,1cyclictest0-21swapper/023:51:290
1510899201,18cyclictest21349-21ps00:30:242
1510499207,5cyclictest11062-21sh23:08:061
1510499203,16cyclictest0-21swapper/123:55:231
1510499202,4cyclictest24146-21/usr/sbin/munin21:20:171
1510499201,5cyclictest20704-21runrttasks00:30:191
1510499201,4cyclictest0-21swapper/119:20:281
1510499200,20cyclictest0-21swapper/122:56:561
1511299190,19cyclictest0-21swapper/323:28:453
1511299190,17cyclictest0-21swapper/300:29:033
1510899190,12cyclictest0-21swapper/221:37:262
15104991919,0cyclictest0-21swapper/100:13:281
15104991915,2cyclictest0-21swapper/121:15:001
1510499190,2cyclictest24993-21unixbench_multi19:30:261
1510499190,16cyclictest0-21swapper/100:24:151
1510499190,14cyclictest24837-21ssh22:18:451
1509899190,4cyclictest0-21swapper/023:33:580
1509899190,4cyclictest0-21swapper/023:33:580
1510499186,6cyclictest19195-21ssh21:36:131
1510499181,5cyclictest14475-21cron20:20:011
1510499180,2cyclictest5940-21diskmemload23:35:151
1510499180,1cyclictest0-21swapper/123:17:071
1510499180,1cyclictest0-21swapper/120:25:191
303032170,0sleep30-21swapper/322:02:233
15112991713,3cyclictest11061-21ls22:10:153
1511299171,12cyclictest0-21swapper/323:05:123
1511299170,17cyclictest12482-21kworker/3:119:23:353
1511299170,15cyclictest0-21swapper/321:54:043
1510899179,2cyclictest16067-21grep19:15:122
15108991717,0cyclictest0-21swapper/200:05:012
15108991716,1cyclictest0-21swapper/223:51:572
15108991710,2cyclictest30629-21grep23:00:182
1510899170,13cyclictest0-21swapper/223:45:572
1510499177,5cyclictest29334-21ssh21:42:331
1510499173,1cyclictest5940-21diskmemload22:53:001
1510499170,9cyclictest0-21swapper/100:01:211
1510499170,2cyclictest19355-21latency_hist22:35:001
1509899175,6cyclictest14495-21munin-run20:20:010
1509899170,1cyclictest0-21swapper/000:38:050
1511299160,3cyclictest225-21systemd-journal00:33:413
1510499169,6cyclictest25965-21sed20:40:261
1510499161,1cyclictest5940-21diskmemload23:23:291
1509899160,2cyclictest0-21swapper/023:40:530
1509899160,1cyclictest0-21swapper/023:49:510
1511299156,7cyclictest0-21swapper/321:15:123
15112991514,1cyclictest39-21ksoftirqd/322:54:163
1510899159,5cyclictest4079-21wc22:25:212
1510899156,8cyclictest716-21munin-node21:50:512
1510899150,4cyclictest0-21swapper/221:20:162
1510899150,2cyclictest0-21swapper/222:59:382
1510499159,5cyclictest5391-21wc21:05:261
1510499158,6cyclictest9866-21cut21:50:161
1510499158,6cyclictest5302-21apache222:06:571
1510499155,6cyclictest225-21systemd-journal20:24:551
1510499150,3cyclictest1276-21unixbench_multi21:25:251
1510499150,2cyclictest0-21swapper/119:15:151
1510499150,14cyclictest20916-21chrt19:25:101
1509899155,5cyclictest14994-21ssh23:30:000
15098991512,2cyclictest28995-21cut22:40:160
1509899150,9cyclictest30386-21grep23:20:010
1509899150,3cyclictest0-21swapper/021:13:150
15112991410,3cyclictest26400-21turbostat.cron20:44:593
15112991410,3cyclictest21390-21ssh22:16:033
1511299140,2cyclictest0-21swapper/321:34:103
1511299140,1cyclictest0-21swapper/323:45:033
1510899149,3cyclictest31139-21cron20:55:002
1510899146,4cyclictest225-21systemd-journal20:25:002
1510899143,8cyclictest31792-21apt-get21:25:122
15108991411,2cyclictest8045-21cut21:30:162
1510899140,4cyclictest0-21swapper/223:25:132
1510899140,13cyclictest0-21swapper/222:07:382
1510499146,7cyclictest4909-21run-parts20:00:011
1510499144,2cyclictest225-21systemd-journal22:45:001
1510499141,3cyclictest225-21systemd-journal20:00:031
15098991413,1cyclictest0-21swapper/022:35:400
15098991413,1cyclictest0-21swapper/022:15:570
15098991411,2cyclictest27815-21ssh00:35:000
15098991410,2cyclictest7889-21grep20:05:140
15098991410,2cyclictest7889-21grep20:05:140
1509899140,14cyclictest0-21swapper/022:55:310
1509899140,14cyclictest0-21swapper/022:14:240
1509899140,11cyclictest0-21swapper/000:10:300
1511299139,3cyclictest10294-21sort00:05:183
1511299132,8cyclictest1942-21gltestperf00:00:153
1511299131,11cyclictest0-21swapper/323:39:073
15112991310,2cyclictest23318-21sort20:35:263
15112991310,2cyclictest16034-21cut19:15:123
15112991310,2cyclictest15558-21wget23:30:153
15112991310,2cyclictest15558-21wget23:30:143
1510899139,3cyclictest12972-21wc00:25:272
15108991310,2cyclictest26669-21cat22:20:012
15108991310,2cyclictest22913-21cat21:20:002
15108991310,2cyclictest20232-21tail19:20:282
1510899130,13cyclictest0-21swapper/200:15:302
1510499135,7cyclictest703-21sed20:55:281
1510499135,7cyclictest23152-21wc20:35:221
1510499131,4cyclictest12111-21dump-pmu-power20:15:001
1510499131,2cyclictest225-21systemd-journal21:04:591
1510499130,3cyclictest27706-21sed20:45:201
1509899135,5cyclictest4808-21cat22:25:270
1509899131,2cyclictest26960-21cstates00:15:150
15098991310,2cyclictest2494-21cat22:25:000
15098991310,2cyclictest2380-21cat00:20:010
15098991310,2cyclictest15441-21grep23:10:220
1509899130,13cyclictest0-21swapper/000:01:060
1509899130,12cyclictest26939-21grep20:45:150
1511299129,2cyclictest8849-21chrt22:28:243
1511299129,2cyclictest8103-21df20:05:153
1511299129,2cyclictest8103-21df20:05:143
1511299129,2cyclictest6211-21wc20:00:193
1511299129,2cyclictest4975-21chrt21:28:133
1511299129,2cyclictest3622-21grep22:05:253
1511299129,2cyclictest32621-21cat21:45:003
1511299129,2cyclictest3096-21grep00:20:133
1511299129,2cyclictest30337-21cron23:20:003
1511299129,2cyclictest30190-21cron19:45:013
1511299129,2cyclictest27584-21chrt22:58:533
1511299129,2cyclictest24735-21wc23:55:003
1511299129,2cyclictest21955-21basename23:15:013
1511299129,2cyclictest2022-21ssh00:40:003
1511299129,2cyclictest14497-21wc21:15:013
1511299129,2cyclictest10270-21seq22:48:293
1511299128,3cyclictest24011-21ssh21:20:173
1511299128,2cyclictest15057-21tr20:20:153
1511299123,8cyclictest18087-21entropy21:55:173
1511299123,8cyclictest18087-21entropy21:55:173
1511299123,7cyclictest28783-21apt20:49:593
1511299120,3cyclictest0-21swapper/322:42:003
1511299120,11cyclictest1130-21tr20:59:593
1510899129,2cyclictest5900-21sort20:00:162
1510899129,2cyclictest3892-21grep19:55:222
1510899129,2cyclictest3764-21munin-run22:44:592
1510899129,2cyclictest29788-21head19:40:252
1510899129,2cyclictest26513-21grep21:40:262
1510899129,2cyclictest24729-21sed23:35:172
1510899129,2cyclictest23045-21cron19:30:002
1510899129,2cyclictest21958-21sleep23:15:012
1510899129,2cyclictest2044-21grep19:50:252
1510899129,2cyclictest14263-21wc22:50:212
1510899129,2cyclictest12693-21cat22:30:212
1510899128,3cyclictest25955-21grep20:40:262
1510899128,2cyclictest31483-21grep20:55:152
1510899124,4cyclictest225-21systemd-journal00:14:552
1510899121,10cyclictest0-21swapper/222:37:312
1510899120,6cyclictest452-21systemd-logind23:59:042
1510899120,2cyclictest0-21swapper/223:40:022
1510899120,11cyclictest0-21swapper/223:07:472
1510499123,2cyclictest29083-21awk20:50:151
1510499121,2cyclictest225-21systemd-journal20:06:111
1510499121,2cyclictest225-21systemd-journal20:06:111
1510499120,5cyclictest15693-21cron19:15:001
1509899129,2cyclictest6154-21cat22:45:260
1509899129,2cyclictest31984-21wc23:40:000
1509899129,2cyclictest31952-21wget21:25:140
1509899129,2cyclictest28836-21grep19:40:160
1509899129,2cyclictest26412-21head19:35:170
1509899129,2cyclictest25484-21sed21:40:170
1509899129,2cyclictest2532-21turbostat19:55:000
1509899129,2cyclictest21170-21grep20:30:250
1509899129,2cyclictest15994-21ls20:20:230
1509899129,2cyclictest13644-21cat23:10:000
1509899129,2cyclictest12100-21sed00:25:180
1509899127,4cyclictest392-21sed23:20:270
1509899124,7cyclictest12127-21idleruntime-cro20:15:000
1509899124,6cyclictest9208-21ntpq21:30:230
15098991210,2cyclictest0-21swapper/021:59:480
15098991210,2cyclictest0-21swapper/021:59:470
1509899120,3cyclictest0-21swapper/023:00:520
1509899120,2cyclictest0-21swapper/021:22:130
1511299118,2cyclictest4055-21wc19:55:253
1511299118,2cyclictest3869-21grep21:05:143
1511299118,2cyclictest3656-21ssh21:46:173
1511299118,2cyclictest31196-21head19:45:143
1511299118,2cyclictest30966-21aten2_r7power_e23:20:143
1511299118,2cyclictest29857-21grep23:00:143
1511299118,2cyclictest29667-21sort20:50:173
1511299118,2cyclictest26532-21ssh00:15:093
1511299118,2cyclictest26396-21fw_packets19:35:173
1511299118,2cyclictest25914-21fw_packets23:55:173
1511299118,2cyclictest25576-21cat19:35:003
1511299118,2cyclictest24111-21awk21:40:003
1511299118,2cyclictest22517-21ssh22:35:583
1511299118,2cyclictest2028-21ssh23:40:333
1511299118,2cyclictest1648-21tr21:00:153
1511299118,2cyclictest1130-21grep19:50:163
1511299118,2cyclictest10232-21tr20:10:133
1511299117,3cyclictest27095-21grep22:20:133
1511299117,3cyclictest26129-21wc00:15:003
1511299117,2cyclictest11778-21grep22:30:153
1510899118,2cyclictest8370-21grep20:05:172
1510899118,2cyclictest8370-21grep20:05:162
1510899118,2cyclictest625-21sh23:20:342
1510899118,2cyclictest4501-21grep21:05:172
1510899118,2cyclictest31612-21grep19:45:182
1510899118,2cyclictest27093-21chrt19:35:242
1510899118,2cyclictest26712-21cat22:00:182
1510899118,2cyclictest2477-21head00:00:202
1510899118,2cyclictest2110-21awk21:45:232
1510899118,2cyclictest18790-21ssh23:31:272
1510899118,2cyclictest18790-21ssh23:31:262
1510899118,2cyclictest10172-21basename22:10:112
1510899117,3cyclictest8105-21python21:10:272
1510899113,4cyclictest225-21systemd-journal21:01:232
1510499110,5cyclictest20281-21head20:30:161
1509899119,1cyclictest4365-21cat19:55:290
1509899118,2cyclictest9131-21tr21:50:130
1509899118,2cyclictest8395-21ssh22:09:010
1509899118,2cyclictest5992-21gltestperf20:00:170
1509899118,2cyclictest29004-21chrt20:50:080
1509899118,2cyclictest25732-21df23:55:160
1509899118,2cyclictest21243-21cstates19:25:150
1509899118,2cyclictest19265-21grep20:30:000
1509899118,2cyclictest18117-21sendmail-msp19:20:000
1509899118,2cyclictest17120-21sed21:35:200
1509899118,2cyclictest16451-21taskset22:51:460
1509899117,3cyclictest4469-21grep21:05:170
1509899115,3cyclictest14861-21aten2_r7power_c21:15:130
1509899110,10cyclictest19407-21basename19:20:200
1509899110,10cyclictest0-21swapper/021:46:150
1511299108,1cyclictest20917-21taskset19:25:113
1511299108,1cyclictest153832sleep319:11:093
1511299108,1cyclictest13305-21uname20:15:203
1511299107,2cyclictest17808-21grep20:25:183
1510899108,1cyclictest26759-21ssh22:20:012
1510899108,1cyclictest22997-21grep20:35:212
1510899108,1cyclictest19768-21grep20:30:142
1510899108,1cyclictest156452sleep219:14:332
1510899108,1cyclictest143572sleep220:18:322
1510899108,1cyclictest11185-21pluginstate20:10:202
1510899107,2cyclictest27216-21grep20:45:162
1510899107,2cyclictest18865-21sed20:25:282
1510899105,3cyclictest25189-21sshd19:32:032
1510899104,2cyclictest225-21systemd-journal21:56:342
1510899104,2cyclictest225-21systemd-journal21:56:342
1509899109,1cyclictest131ksoftirqd/020:40:000
1509899108,2cyclictest24273-21grep19:30:180
1509899108,1cyclictest31882-21ls19:45:240
1509899108,1cyclictest31373-21chrt20:55:040
1509899107,2cyclictest2602-21grep21:00:220
1509899107,2cyclictest11396-21sh22:30:120
1509899100,6cyclictest15446-21sshd19:11:580
151129996,2cyclictest20250-21head20:30:163
150989990,5cyclictest0-21swapper/020:40:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional