You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-18 - 06:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri Sep 18, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
305888621170,4sleep30-21swapper/321:05:143
304810121130,1sleep00-21swapper/020:35:110
301725429042,13sleep30-21swapper/319:06:453
30165662810,6sleep20-21swapper/219:05:182
301642727447,16sleep10-21swapper/119:05:141
301602826743,15sleep00-21swapper/019:05:060
30491252430,4sleep33056-21PK-Backend20:35:223
31292932420,3sleep33056-21PK-Backend00:25:153
31170382390,3sleep30-21swapper/323:50:153
31136202390,7sleep126-21ksoftirqd/123:40:181
30756072310,7sleep1251rcuc/121:55:001
3017665992717,6cyclictest3028742-21apt19:40:123
301766599257,16cyclictest3132111-21/usr/sbin/munin00:35:013
301765899259,9cyclictest13-21ksoftirqd/021:40:000
139099250,4rtkit-daemon1389-21rtkit-daemon20:02:041
3017658992415,7cyclictest3034851-21python319:55:200
139099240,3rtkit-daemon1389-21rtkit-daemon21:55:191
3017660992320,2cyclictest3056-21PK-Backend20:23:361
3017660992318,3cyclictest3025750-21irqstats19:30:161
301765899239,8cyclictest13-21ksoftirqd/019:36:210
3017658992314,7cyclictest436-20systemd-journal22:55:000
31039692220,8chrt13-21ksoftirqd/023:13:390
3017665992220,1cyclictest3056-21PK-Backend21:30:573
301766599220,20cyclictest3056-21PK-Backend21:50:043
301766599220,20cyclictest3056-21PK-Backend19:31:133
3017664992220,1cyclictest0-21swapper/219:20:532
301766499220,21cyclictest0-21swapper/221:50:242
3017660992220,1cyclictest0-21swapper/120:05:101
3017660992220,1cyclictest0-21swapper/119:11:161
301765899226,6cyclictest13-21ksoftirqd/019:30:150
3017658992220,1cyclictest0-21swapper/023:25:200
3017658992220,1cyclictest0-21swapper/023:18:100
3017658992220,1cyclictest0-21swapper/019:16:450
3017658992220,1cyclictest0-21swapper/000:32:200
3017658992214,6cyclictest3089231-21ntp_kernel_pll_22:30:220
301765899220,21cyclictest3056-21PK-Backend20:49:260
139099220,4rtkit-daemon1389-21rtkit-daemon23:02:271
301766599216,12cyclictest3023300-21unin-run19:24:593
3017665992119,1cyclictest3056-21PK-Backend23:27:583
3017665992119,1cyclictest3056-21PK-Backend23:02:093
3017665992119,1cyclictest3056-21PK-Backend20:47:443
3017665992119,1cyclictest3056-21PK-Backend19:36:003
3017665992119,1cyclictest0-21swapper/323:11:303
3017665992119,1cyclictest0-21swapper/322:55:133
3017665992119,1cyclictest0-21swapper/320:50:083
3017665992119,1cyclictest0-21swapper/320:23:113
3017665992118,2cyclictest3056-21PK-Backend22:49:053
3017665992115,4cyclictest886-21NetworkManager23:45:003
3017665992114,5cyclictest3021526-21cron19:20:003
301766599210,20cyclictest3056-21PK-Backend22:53:373
301766599210,20cyclictest3056-21PK-Backend19:45:463
301766599210,20cyclictest0-21swapper/322:42:093
301766599210,20cyclictest0-21swapper/319:25:043
301766599210,19cyclictest3056-21PK-Backend22:30:393
301766599210,19cyclictest3056-21PK-Backend21:56:463
3017664992119,1cyclictest0-21swapper/223:12:582
3017664992119,1cyclictest0-21swapper/222:25:052
3017664992119,1cyclictest0-21swapper/222:10:112
3017664992119,1cyclictest0-21swapper/220:58:332
3017664992119,1cyclictest0-21swapper/220:44:482
3017664992113,6cyclictest3115758-21ntp_states23:45:192
3017664992110,10cyclictest3119988-40ogrotate00:00:012
3017660992119,1cyclictest3056-21PK-Backend23:53:251
3017660992119,1cyclictest3056-21PK-Backend23:12:271
3017660992119,1cyclictest3056-21PK-Backend21:10:061
3017660992119,1cyclictest3056-21PK-Backend00:34:131
3017660992119,1cyclictest0-21swapper/123:56:351
3017660992119,1cyclictest0-21swapper/123:22:351
3017660992119,1cyclictest0-21swapper/122:48:281
3017660992119,1cyclictest0-21swapper/121:31:081
3017660992117,2cyclictest3083279-21/usr/sbin/munin22:15:161
301766099211,19cyclictest0-21swapper/120:42:011
301766099210,19cyclictest3056-21PK-Backend22:43:071
301766099210,19cyclictest3056-21PK-Backend22:05:301
301766099210,19cyclictest3056-21PK-Backend19:42:121
301766099210,19cyclictest0-21swapper/123:19:241
3017658992119,1cyclictest3056-21PK-Backend23:44:470
3017658992119,1cyclictest0-21swapper/022:22:540
3017658992119,1cyclictest0-21swapper/022:22:540
3017658992119,1cyclictest0-21swapper/021:40:360
3017658992119,1cyclictest0-21swapper/021:23:540
3017658992119,1cyclictest0-21swapper/021:23:530
3017658992119,1cyclictest0-21swapper/021:19:340
3017658992119,1cyclictest0-21swapper/020:31:490
3017658992119,1cyclictest0-21swapper/020:21:410
3017658992119,1cyclictest0-21swapper/020:08:300
3017658992119,1cyclictest0-21swapper/019:41:330
3017658992119,1cyclictest0-21swapper/019:10:510
3017658992119,1cyclictest0-21swapper/000:29:090
301765899211,8cyclictest13-21ksoftirqd/019:22:570
3017658992118,2cyclictest161rcuc/021:32:030
3017658992117,3cyclictest161rcuc/021:45:160
301765899210,5cyclictest3031208-21ntp_states19:45:190
301765899210,20cyclictest0-21swapper/023:58:390
301765899210,20cyclictest0-21swapper/022:15:100
301765899210,20cyclictest0-21swapper/022:06:040
301765899210,20cyclictest0-21swapper/020:55:350
301765899210,20cyclictest0-21swapper/020:27:160
301765899210,20cyclictest0-21swapper/020:27:150
301765899210,20cyclictest0-21swapper/019:28:000
301765899210,20cyclictest0-21swapper/000:35:300
301765899210,20cyclictest0-21swapper/000:01:510
139099210,4rtkit-daemon1389-21rtkit-daemon19:24:191
30276802200,2sleep10-21swapper/119:35:171
3017665992018,1cyclictest3056-21PK-Backend23:31:093
3017665992018,1cyclictest3056-21PK-Backend22:29:513
3017665992018,1cyclictest3056-21PK-Backend19:57:563
3017665992011,7cyclictest3079051-21apt-get22:05:003
301766599200,19cyclictest3056-21PK-Backend22:22:553
301766599200,19cyclictest3056-21PK-Backend22:22:553
301766599200,19cyclictest3056-21PK-Backend22:13:363
301766599200,19cyclictest3056-21PK-Backend21:49:163
301766599200,19cyclictest0-21swapper/322:36:003
301766599200,19cyclictest0-21swapper/322:07:273
301766599200,19cyclictest0-21swapper/321:01:513
301766599200,19cyclictest0-21swapper/320:16:213
301766599200,18cyclictest3056-21PK-Backend23:09:063
301766599200,18cyclictest3056-21PK-Backend22:15:123
3017664992018,1cyclictest0-21swapper/222:15:522
3017664992018,1cyclictest0-21swapper/221:33:152
3017664992018,1cyclictest0-21swapper/220:27:572
3017664992018,1cyclictest0-21swapper/220:27:572
3017664992018,1cyclictest0-21swapper/200:17:132
3017664992018,1cyclictest0-21swapper/200:11:192
3017664992017,2cyclictest0-21swapper/200:05:232
3017664992016,3cyclictest3128638-21ls00:25:002
3017664992013,5cyclictest3081189-21cat22:10:012
3017664992012,6cyclictest3018208-21df19:10:122
3017664992011,7cyclictest3096303-21python322:50:232
301766499200,19cyclictest0-21swapper/222:39:112
301766499200,19cyclictest0-21swapper/221:59:242
301766499200,19cyclictest0-21swapper/220:53:262
301766499200,19cyclictest0-21swapper/220:06:212
301766499200,19cyclictest0-21swapper/219:18:432
301766499200,19cyclictest0-21swapper/200:38:562
3017660992018,1cyclictest0-21swapper/122:50:031
3017660992018,1cyclictest0-21swapper/122:20:091
3017660992018,1cyclictest0-21swapper/122:20:091
3017660992018,1cyclictest0-21swapper/122:11:371
3017660992018,1cyclictest0-21swapper/100:05:201
3017660992018,1cyclictest0-21swapper/100:00:331
3017660992013,5cyclictest3133876-21pluginstate00:40:001
301766099200,19cyclictest0-21swapper/123:09:161
301766099200,19cyclictest0-21swapper/122:55:251
301766099200,19cyclictest0-21swapper/122:00:561
301766099200,19cyclictest0-21swapper/121:38:251
301766099200,19cyclictest0-21swapper/120:31:191
301766099200,19cyclictest0-21swapper/119:27:531
301766099200,19cyclictest0-21swapper/119:15:491
3017658992018,1cyclictest0-21swapper/021:12:170
3017658992018,1cyclictest0-21swapper/021:07:580
3017658992018,1cyclictest0-21swapper/000:15:220
3017658992017,2cyclictest161rcuc/023:33:290
3017658992013,5cyclictest3050563-21ntp_kernel_pll_20:40:190
3017658992012,6cyclictest436-20systemd-journal23:25:000
301765899201,17cyclictest3118234-21idleruntime-cro23:55:000
301765899200,1cyclictest3056-21PK-Backend20:50:140
301765899200,19cyclictest0-21swapper/023:01:070
139099200,16rtkit-daemon26-21ksoftirqd/100:12:151
3017665991917,1cyclictest0-21swapper/321:25:513
3017665991917,1cyclictest0-21swapper/320:09:333
3017665991914,3cyclictest978-21snmpd20:10:023
3017665991914,3cyclictest3019306-21cron19:15:013
3017665991913,4cyclictest978-21snmpd23:38:273
3017665991911,6cyclictest3106833-21ntp_states23:20:203
301766599190,1cyclictest3056-21PK-Backend21:43:213
301766499192,15cyclictest3107552-21dump-pmu-power23:25:012
3017664991917,1cyclictest0-21swapper/223:40:082
3017664991917,1cyclictest0-21swapper/220:15:422
3017664991917,1cyclictest0-21swapper/219:32:482
3017664991916,2cyclictest3056-21PK-Backend21:20:232
3017664991916,2cyclictest3056-21PK-Backend21:20:232
3017664991913,4cyclictest3072109-21cron21:45:002
3017664991912,5cyclictest3100629-21perf23:05:002
3017664991912,5cyclictest3079214-21date22:05:022
3017664991912,5cyclictest3053242-21wc20:50:002
3017664991912,5cyclictest3029793-21sed19:40:222
3017664991911,6cyclictest3040903-21cron20:15:012
301766499190,1cyclictest0-21swapper/221:14:562
301766499190,1cyclictest0-21swapper/221:04:482
3017660991917,1cyclictest0-21swapper/123:31:201
3017660991917,1cyclictest0-21swapper/123:27:221
3017660991917,1cyclictest0-21swapper/122:27:541
3017660991917,1cyclictest0-21swapper/121:48:051
3017660991917,1cyclictest0-21swapper/121:17:231
3017660991917,1cyclictest0-21swapper/120:27:351
3017660991917,1cyclictest0-21swapper/120:27:341
3017660991917,1cyclictest0-21swapper/100:18:531
3017660991916,2cyclictest3056-21PK-Backend20:10:101
3017660991912,5cyclictest3032461-21cat19:50:131
3017660991911,6cyclictest3053684-21df_inode20:50:141
301766099190,1cyclictest3056-21PK-Backend23:49:251
301766099190,1cyclictest3056-21PK-Backend22:37:461
301766099190,1cyclictest3056-21PK-Backend21:28:111
301766099190,1cyclictest3056-21PK-Backend21:04:591
301766099190,18cyclictest3056-21PK-Backend20:59:521
301766099190,18cyclictest0-21swapper/122:32:271
301766099190,18cyclictest0-21swapper/121:44:211
301766099190,18cyclictest0-21swapper/120:48:581
301765899192,5cyclictest13-21ksoftirqd/020:19:590
3017658991917,1cyclictest0-21swapper/023:46:220
3017658991917,1cyclictest0-21swapper/022:42:420
3017658991917,1cyclictest0-21swapper/022:29:500
301765899191,6cyclictest13-21ksoftirqd/021:00:200
3017658991916,2cyclictest161rcuc/023:09:160
3017658991916,2cyclictest161rcuc/022:56:130
3017658991916,2cyclictest161rcuc/022:48:260
3017658991916,2cyclictest161rcuc/000:22:030
3017658991915,3cyclictest161rcuc/000:11:080
3017658991911,6cyclictest436-20systemd-journal22:15:000
3017658991911,6cyclictest3075846-21awk21:55:110
3017658991911,6cyclictest3033072-21swap19:50:200
301766599186,4cyclictest42-21ksoftirqd/323:45:113
301766599181,3cyclictest3119352-21python323:55:213
3017665991813,3cyclictest3051346-21munin-run20:45:003
3017665991812,5cyclictest3122941-21sed00:05:243
3017665991811,5cyclictest436-20systemd-journal21:25:003
3017665991811,5cyclictest436-20systemd-journal21:24:593
301766599180,16cyclictest3124569-21python300:10:243
3017664991816,1cyclictest0-21swapper/222:46:282
3017664991816,1cyclictest0-21swapper/221:16:172
3017664991816,1cyclictest0-21swapper/221:05:372
3017664991816,1cyclictest0-21swapper/219:37:552
3017664991816,1cyclictest0-21swapper/200:33:012
3017664991814,3cyclictest3035451-21sh20:00:002
3017664991812,4cyclictest978-21snmpd23:38:382
3017664991812,4cyclictest3117250-21uname23:50:182
3017664991810,6cyclictest3085401-21df22:20:142
3017664991810,6cyclictest3085401-21df22:20:142
3017664991810,6cyclictest3065649-21wc21:25:112
301766499180,16cyclictest3047856-21perf20:35:002
3017660991816,1cyclictest3056-21PK-Backend21:23:521
3017660991816,1cyclictest3056-21PK-Backend21:23:511
3017660991816,1cyclictest3056-21PK-Backend21:07:561
3017660991813,3cyclictest3035459-21latency_hist20:00:011
3017658991814,3cyclictest161rcuc/000:07:430
3017658991812,4cyclictest3074353-21ls21:50:110
301765899180,6cyclictest3056-21PK-Backend21:25:150
301765899180,1cyclictest0-21swapper/022:38:090
301765899180,18cyclictest0-21swapper/022:02:190
301766599179,6cyclictest3105812-21cron23:19:593
3017665991715,1cyclictest3056-21PK-Backend21:36:043
3017665991715,1cyclictest3056-21PK-Backend20:32:313
3017665991715,1cyclictest0-21swapper/320:25:493
3017665991715,1cyclictest0-21swapper/320:25:483
3017665991715,1cyclictest0-21swapper/320:04:323
3017665991711,5cyclictest42-21ksoftirqd/300:35:183
3017665991711,3cyclictest3063464-21turbostat21:19:593
3017664991715,1cyclictest0-21swapper/223:26:572
3017664991715,1cyclictest0-21swapper/220:37:452
3017664991715,1cyclictest0-21swapper/220:20:272
3017664991715,1cyclictest0-21swapper/220:01:482
3017664991713,3cyclictest3068896-21/usr/sbin/munin21:35:162
3017664991712,3cyclictest3101022-21tr23:05:112
301766499170,15cyclictest3023900-21df19:25:132
3017660991712,3cyclictest1095-21runrttasks20:35:201
3017658991714,2cyclictest161rcuc/020:01:490
3017665991614,1cyclictest3056-21PK-Backend21:12:053
3017665991611,4cyclictest0-21swapper/320:55:113
301766599161,13cyclictest3120214-21/usr/sbin/munin00:00:243
301766499169,5cyclictest3121032-21proc_pri00:00:242
301766499169,5cyclictest3105898-21date23:20:012
301766499169,5cyclictest3030589-21wc19:45:132
301766499168,6cyclictest3089092-21memory22:30:212
3017664991614,1cyclictest0-21swapper/223:31:162
3017664991610,5cyclictest3098882-21cron23:00:002
3017664991610,4cyclictest3093039-21uptime22:40:232
3017664991610,4cyclictest3032353-21tr19:50:122
301766099169,5cyclictest3031040-21sed19:45:161
3017658991610,4cyclictest3039605-21tr20:10:120
301765899160,5cyclictest13-21ksoftirqd/023:35:050
3017665991512,2cyclictest42-21ksoftirqd/300:15:483
3017665991510,4cyclictest42-21ksoftirqd/319:50:123
301766099158,5cyclictest3127164-21awk00:20:131
3017660991513,1cyclictest3056-21PK-Backend20:19:231
3017660991513,1cyclictest0-21swapper/123:35:191
3017660991510,3cyclictest3129675-21open_inodes00:25:201
30723822140,3sleep23072371-21acpi21:45:112
301766599141,7cyclictest978-21snmpd00:20:513
3017664991410,3cyclictest3129915-21grep00:25:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional