You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-08 - 07:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Aug 08, 2026 00:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
92141021310,4sleep3921412-21cpuspeed_turbos22:40:113
87780221290,3sleep30-21swapper/320:35:173
90092821140,4sleep13056-21PK-Backend21:40:181
90888621090,4sleep13056-21PK-Backend22:05:011
139099900,72rtkit-daemon0-21swapper/119:05:201
84508127847,21sleep30-21swapper/319:05:113
84631527455,14sleep00-21swapper/019:06:570
84653026341,14sleep20-21swapper/219:09:432
9287742520,3sleep342-21ksoftirqd/323:00:173
9129162360,1sleep30-21swapper/322:15:133
846686992814,12cyclictest919237-21idleruntime-cro22:35:010
9606142260,3chrt0-21swapper/200:30:152
846696992611,13cyclictest889251-21perf21:10:013
84669699260,8cyclictest0-21swapper/321:35:013
84669699260,16cyclictest0-21swapper/300:00:233
84669699260,14cyclictest0-21swapper/322:20:203
84668699262,22cyclictest943303-21ntpq23:40:180
846696992514,8cyclictest436-20systemd-journal20:50:013
84668699258,16cyclictest875384-21perf20:29:590
84668699251,20cyclictest877385-21apt20:35:120
84668699250,24cyclictest0-21swapper/022:23:500
84668699250,19cyclictest0-21swapper/023:00:280
9172682240,5sleep20-21swapper/222:27:112
8503332240,2chrt0-21swapper/219:17:012
84669699240,8cyclictest0-21swapper/321:00:163
84668699248,14cyclictest3056-21PK-Backend00:01:370
84668699247,16cyclictest0-21swapper/000:24:500
84668699247,15cyclictest941803-21sort23:35:210
84668699247,15cyclictest8802662chrt20:41:120
84668699247,15cyclictest0-21swapper/019:56:000
84668699246,16cyclictest954113-21cut00:10:190
84668699246,16cyclictest893278-21ls21:20:120
84668699246,16cyclictest893278-21ls21:20:120
84668699246,16cyclictest0-21swapper/023:09:370
84668699246,16cyclictest0-21swapper/023:09:360
84668699245,17cyclictest945807-21systemd-run23:50:000
84668699245,17cyclictest891768-21wc21:15:160
846686992411,11cyclictest858227-21cpuspeed_turbos19:40:100
846686992411,11cyclictest847382-21kernelversion19:10:160
139099240,16rtkit-daemon3056-21PK-Backend22:43:101
846696992318,3cyclictest849594-21irqstats19:15:183
84669699230,9cyclictest871632-21taskset20:16:403
84669299230,21cyclictest909467-21df22:05:122
846689992320,2cyclictest3056-21PK-Backend21:03:521
84668699236,15cyclictest0-21swapper/020:22:020
84668699235,16cyclictest961841-21grep00:35:010
84668699233,18cyclictest905371-21turbostat.cron21:55:000
84668699232,18cyclictest923836-21python322:45:200
84668699231,4cyclictest949336-40mandb00:00:010
846686992312,9cyclictest906645-21timerwakeupswit21:55:210
846686992312,10cyclictest916011-21/usr/sbin/munin22:25:160
84668699231,20cyclictest868458-21munin-run20:10:010
84668699230,10cyclictest947512-21cron23:54:590
84669699220,9cyclictest866666-21runrttasks20:04:363
846692992220,1cyclictest0-21swapper/222:45:152
846692992213,7cyclictest859105-21sed19:40:192
846689992220,1cyclictest3056-21PK-Backend23:21:281
846689992220,1cyclictest3056-21PK-Backend22:57:011
846689992220,1cyclictest3056-21PK-Backend20:27:091
84668699228,13cyclictest8609202chrt19:46:060
84668699228,12cyclictest913190-21missed_timers22:15:180
84668699228,12cyclictest13-21ksoftirqd/020:45:150
84668699226,14cyclictest889385-21sed21:10:020
84668699226,14cyclictest853427-21missed_timers19:25:160
84668699223,17cyclictest0-21swapper/021:00:370
84668699222,17cyclictest955417-21diskstats00:15:140
846686992220,1cyclictest3056-21PK-Backend22:00:260
846686992220,1cyclictest0-21swapper/023:32:320
846686992220,1cyclictest0-21swapper/023:17:440
846686992220,1cyclictest0-21swapper/022:40:470
846686992217,4cyclictest0-21swapper/022:13:160
846686992217,4cyclictest0-21swapper/022:13:160
846686992216,4cyclictest871920-21munin-run20:19:590
84668699221,4cyclictest919881-21diskstats22:35:130
84668699220,21cyclictest0-21swapper/020:12:030
84668699220,20cyclictest963465-21turbostat00:40:000
139099220,16rtkit-daemon3056-21PK-Backend22:17:151
139099220,14rtkit-daemon0-21swapper/019:39:320
9311702210,3sleep20-21swapper/223:05:222
9311702210,3sleep20-21swapper/223:05:212
84669699211,8cyclictest920250-21ntp_states22:35:193
84669699211,7cyclictest901045-21python321:40:213
846696992115,5cyclictest938616-21systemctl23:30:003
846696992114,5cyclictest903392-21/usr/sbin/munin21:50:003
846696992113,6cyclictest952451-21awk00:05:193
846696992113,6cyclictest862282-21awk19:50:193
84669699210,8cyclictest874469-21cat20:25:173
84669699210,7cyclictest854386-21latency_hist19:30:003
84669699210,19cyclictest0-21swapper/322:25:113
846692992119,1cyclictest0-21swapper/223:45:372
846692992119,1cyclictest0-21swapper/223:44:502
846692992119,1cyclictest0-21swapper/222:32:092
846692992119,1cyclictest0-21swapper/222:02:082
846692992119,1cyclictest0-21swapper/221:53:502
846692992119,1cyclictest0-21swapper/221:36:062
846692992119,1cyclictest0-21swapper/221:31:062
846692992119,1cyclictest0-21swapper/221:18:292
846692992119,1cyclictest0-21swapper/220:56:042
846692992119,1cyclictest0-21swapper/220:52:062
846692992119,1cyclictest0-21swapper/220:27:052
846692992119,1cyclictest0-21swapper/220:11:502
846692992119,1cyclictest0-21swapper/219:23:582
846692992116,3cyclictest963460-21turbostat.cron00:40:002
846692992114,6cyclictest940348-21sshd23:32:152
846692992113,6cyclictest910817-21cat22:10:112
846692992113,6cyclictest910817-21cat22:10:102
84669299210,20cyclictest0-21swapper/221:25:312
84669299210,20cyclictest0-21swapper/200:22:142
846689992119,1cyclictest3056-21PK-Backend23:49:401
846689992119,1cyclictest3056-21PK-Backend22:53:431
846689992119,1cyclictest3056-21PK-Backend20:39:321
846689992119,1cyclictest0-21swapper/100:31:101
846689992118,2cyclictest3056-21PK-Backend20:53:031
846689992117,2cyclictest26-21ksoftirqd/100:25:001
84668999210,20cyclictest3056-21PK-Backend21:26:161
84668999210,20cyclictest3056-21PK-Backend21:13:311
84668999210,20cyclictest3056-21PK-Backend00:15:021
84668999210,19cyclictest3056-21PK-Backend22:21:391
84668999210,19cyclictest3056-21PK-Backend21:45:291
84668999210,19cyclictest3056-21PK-Backend21:19:471
84668999210,19cyclictest3056-21PK-Backend19:23:341
84668699216,13cyclictest1095-21runrttasks00:05:200
84668699215,14cyclictest890065-21latency21:10:170
84668699214,16cyclictest0-21swapper/000:28:520
84668699213,3cyclictest0-21swapper/020:57:430
84668699213,3cyclictest0-21swapper/019:50:340
84668699213,16cyclictest0-21swapper/021:49:400
84668699212,17cyclictest0-21swapper/021:31:560
846686992119,1cyclictest0-21swapper/023:20:280
846686992119,1cyclictest0-21swapper/022:51:560
846686992119,1cyclictest0-21swapper/022:07:080
846686992119,1cyclictest0-21swapper/020:33:050
84668699211,18cyclictest865963-21grep20:00:160
84668699211,16cyclictest894721-21apt21:25:110
84668699210,5cyclictest854681-21grep19:30:110
84668699210,20cyclictest0-21swapper/021:44:570
84668699210,19cyclictest848599-21ntp_states19:15:020
139099210,4rtkit-daemon3056-21PK-Backend20:06:031
846696992014,5cyclictest856055-21/usr/sbin/munin19:35:003
846696992013,6cyclictest933362-21cron23:14:593
846696992013,5cyclictest883564-21cat20:50:213
846696992012,6cyclictest961710-21cron00:35:003
846696992012,6cyclictest42-21ksoftirqd/321:10:113
846696992011,7cyclictest436-20systemd-journal19:50:013
84669699200,8cyclictest0-21swapper/320:40:143
84669699200,8cyclictest0-21swapper/319:10:133
84669699200,19cyclictest0-21swapper/323:31:443
84669699200,19cyclictest0-21swapper/323:16:563
84669699200,19cyclictest0-21swapper/322:06:333
84669699200,19cyclictest0-21swapper/322:04:033
84669699200,19cyclictest0-21swapper/320:06:423
846692992018,1cyclictest0-21swapper/223:54:162
846692992018,1cyclictest0-21swapper/223:35:302
846692992018,1cyclictest0-21swapper/223:12:332
846692992018,1cyclictest0-21swapper/222:56:362
846692992018,1cyclictest0-21swapper/220:38:542
846692992018,1cyclictest0-21swapper/220:34:142
846692992018,1cyclictest0-21swapper/219:55:152
846692992018,1cyclictest0-21swapper/219:46:292
846692992018,1cyclictest0-21swapper/200:26:472
846692992018,1cyclictest0-21swapper/200:13:232
846692992017,2cyclictest0-21swapper/223:15:152
846692992014,5cyclictest28650irq/123-ahci[0023:55:192
846692992013,5cyclictest866090-21sensors_temp20:00:182
846692992012,6cyclictest436-20systemd-journal00:04:592
846692992012,6cyclictest1-21systemd19:55:012
84669299200,1cyclictest0-21swapper/222:43:192
84669299200,19cyclictest0-21swapper/223:25:232
84669299200,19cyclictest0-21swapper/221:49:592
84669299200,19cyclictest0-21swapper/220:16:172
84669299200,19cyclictest0-21swapper/219:28:182
846689992018,1cyclictest3056-21PK-Backend23:38:441
846689992018,1cyclictest3056-21PK-Backend23:29:251
846689992018,1cyclictest3056-21PK-Backend23:04:101
846689992018,1cyclictest3056-21PK-Backend22:47:341
846689992018,1cyclictest3056-21PK-Backend20:22:411
846689992018,1cyclictest3056-21PK-Backend19:37:201
846689992018,1cyclictest3056-21PK-Backend19:14:481
846689992018,1cyclictest3056-21PK-Backend00:01:491
846689992017,2cyclictest3056-21PK-Backend19:51:461
84668999200,1cyclictest3056-21PK-Backend20:02:421
84668999200,19cyclictest3056-21PK-Backend23:42:021
84668999200,19cyclictest3056-21PK-Backend22:13:001
84668999200,19cyclictest3056-21PK-Backend22:13:001
84668999200,19cyclictest3056-21PK-Backend22:06:511
84668999200,19cyclictest3056-21PK-Backend21:52:261
84668999200,19cyclictest3056-21PK-Backend21:32:591
84668999200,19cyclictest3056-21PK-Backend00:28:121
84668999200,19cyclictest3056-21PK-Backend00:12:031
84668999200,18cyclictest3056-21PK-Backend19:27:061
84668699201,3cyclictest937977-21python323:25:190
84668699201,3cyclictest899196-21ntp_states21:35:200
84668699201,3cyclictest851486-21diskstats19:20:120
84668699201,3cyclictest0-21swapper/022:56:130
84668699200,7cyclictest882587-21grep20:50:110
84668699200,5cyclictest931986-21sshd23:10:110
139099200,14rtkit-daemon3056-21PK-Backend20:41:431
139099200,14rtkit-daemon3056-21PK-Backend19:55:321
139099200,13rtkit-daemon3056-21PK-Backend23:57:211
8814282190,3sleep13056-21PK-Backend20:45:171
846696991917,1cyclictest0-21swapper/323:36:233
846696991917,1cyclictest0-21swapper/323:24:193
846696991917,1cyclictest0-21swapper/322:55:473
846696991917,1cyclictest0-21swapper/322:12:083
846696991917,1cyclictest0-21swapper/322:12:083
846696991917,1cyclictest0-21swapper/321:25:433
846696991917,1cyclictest0-21swapper/319:35:053
846696991917,1cyclictest0-21swapper/300:35:303
846696991914,3cyclictest864107-21python319:55:193
846696991912,5cyclictest930727-21cat23:05:173
846696991912,5cyclictest930727-21cat23:05:163
846696991912,5cyclictest918772-21cat22:30:223
846696991910,7cyclictest893391-21which21:20:153
846696991910,7cyclictest893391-21which21:20:153
84669699190,7cyclictest0-21swapper/300:20:153
84669699190,6cyclictest899293-21processes21:35:213
84669699190,1cyclictest0-21swapper/322:52:293
84669699190,1cyclictest0-21swapper/320:59:403
84669699190,1cyclictest0-21swapper/320:13:383
84669699190,18cyclictest0-21swapper/323:41:023
84669699190,18cyclictest0-21swapper/320:23:383
84669699190,18cyclictest0-21swapper/300:17:463
84669699190,18cyclictest0-21swapper/300:11:513
846692991917,1cyclictest3056-21PK-Backend21:57:422
846692991917,1cyclictest0-21swapper/222:50:482
846692991917,1cyclictest0-21swapper/220:23:132
846692991913,4cyclictest856885-21load19:35:162
846692991912,5cyclictest34-21ksoftirqd/222:35:102
84669299190,1cyclictest0-21swapper/220:40:162
84669299190,1cyclictest0-21swapper/220:07:582
84669299190,18cyclictest0-21swapper/219:14:582
846689991917,1cyclictest3056-21PK-Backend23:10:461
846689991917,1cyclictest3056-21PK-Backend23:07:281
846689991917,1cyclictest3056-21PK-Backend23:07:281
846689991917,1cyclictest3056-21PK-Backend22:37:131
846689991917,1cyclictest3056-21PK-Backend22:29:441
846689991917,1cyclictest3056-21PK-Backend19:41:041
846689991917,1cyclictest3056-21PK-Backend19:16:101
846689991917,1cyclictest0-21swapper/121:58:141
846689991917,1cyclictest0-21swapper/120:55:211
84668999190,1cyclictest3056-21PK-Backend23:51:501
84668999190,1cyclictest3056-21PK-Backend21:35:281
84668999190,1cyclictest3056-21PK-Backend19:31:251
84668999190,1cyclictest3056-21PK-Backend00:39:481
84668999190,1cyclictest0-21swapper/100:06:561
84668999190,18cyclictest3056-21PK-Backend20:10:111
846696991816,1cyclictest0-21swapper/319:42:353
846696991816,1cyclictest0-21swapper/319:24:503
846696991813,3cyclictest946029-21acpi23:50:103
846696991813,3cyclictest904280-21df_inode21:50:133
846696991812,5cyclictest960074-21ls00:30:013
84669699181,15cyclictest923450-21irqstats22:45:153
846696991810,6cyclictest892726-21cron21:20:003
84669699180,7cyclictest0-21swapper/323:50:013
846692991816,1cyclictest0-21swapper/222:20:132
846692991816,1cyclictest0-21swapper/222:18:182
846692991816,1cyclictest0-21swapper/221:10:312
846692991816,1cyclictest0-21swapper/220:45:442
846692991816,1cyclictest0-21swapper/219:32:022
846692991813,4cyclictest34-21ksoftirqd/200:10:012
846692991812,5cyclictest929318-21sshd23:01:102
846692991811,5cyclictest936314-21cat23:20:202
846689991816,1cyclictest3056-21PK-Backend22:34:431
846689991816,1cyclictest3056-21PK-Backend20:17:071
846696991713,3cyclictest0-21swapper/321:56:323
846696991713,2cyclictest947777-21apt23:55:113
846692991712,3cyclictest901269-21vmstat21:40:232
846689991715,1cyclictest0-21swapper/120:30:261
846689991713,3cyclictest940589-21idleruntime-cro23:35:001
84668999170,5cyclictest3056-21PK-Backend21:20:541
84668999170,5cyclictest3056-21PK-Backend21:20:531
84669699160,4cyclictest876517-21/usr/sbin/munin20:30:213
846692991610,5cyclictest888076-21expr21:05:122
84668999169,5cyclictest0-21swapper/121:08:081
846689991614,1cyclictest3056-21PK-Backend23:16:481
846689991614,1cyclictest3056-21PK-Backend19:45:371
846692991512,2cyclictest886517-21if_err_enp2s021:00:162
846692991511,2cyclictest894447-21sh21:25:002
846692991511,2cyclictest894447-21sh21:24:592
846692991510,4cyclictest34-21ksoftirqd/200:15:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional