You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 12:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat May 16, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
372709121160,2sleep00-21swapper/021:20:210
367963827639,13sleep30-21swapper/319:06:593
367964626840,22sleep10-21swapper/119:07:051
367973026646,14sleep20-21swapper/219:08:112
367967026344,13sleep00-21swapper/019:07:240
37649362530,3sleep33056-21PK-Backend23:10:113
37470752480,4chrt3747077-21apt22:20:001
37837322420,1sleep20-21swapper/200:04:552
37742002390,4sleep20-21swapper/223:35:162
37096462370,3sleep10-21swapper/120:30:221
3679996992821,1cyclictest0-21swapper/323:34:523
367999199270,5cyclictest3056-21PK-Backend19:45:001
367999199254,9cyclictest3757826-21apt-config22:50:011
3679987992513,10cyclictest3713660-21unin-run20:45:000
367999399240,22cyclictest1389-21rtkit-daemon22:47:102
367999699234,17cyclictest3785751-21munin-plugin-st00:10:013
367999199235,5cyclictest3794018-21users00:30:211
367999199235,5cyclictest3783026-21ntp_states00:00:181
367999199234,6cyclictest3683506-21users19:15:221
367999199233,5cyclictest3690110-21ntp_states19:35:161
367999199232,7cyclictest3785145-21python300:05:211
3679991992321,1cyclictest0-21swapper/100:16:121
3679991992314,7cyclictest883-21dbus-daemon23:18:521
139099230,3rtkit-daemon0-21swapper/020:11:380
3679996992220,1cyclictest0-21swapper/322:51:323
3679996992220,1cyclictest0-21swapper/322:34:563
3679996992220,1cyclictest0-21swapper/322:34:563
3679996992220,1cyclictest0-21swapper/321:06:483
3679996992219,2cyclictest3056-21PK-Backend22:08:333
3679996992219,2cyclictest3056-21PK-Backend20:40:263
3679996992219,2cyclictest3056-21PK-Backend20:24:033
3679996992219,2cyclictest3056-21PK-Backend20:09:093
3679993992213,8cyclictest3792708-21unin-run00:30:002
367999199223,6cyclictest3781776-21taskset23:57:181
367999199223,6cyclictest3736418-21seq21:46:511
367999199222,5cyclictest1980-21gvfs-udisks2-vo21:18:031
3679991992220,1cyclictest0-21swapper/121:42:411
3679991992220,1cyclictest0-21swapper/120:16:161
3679991992215,5cyclictest3775063-21cron23:40:001
3679991992212,7cyclictest436-20systemd-journal19:30:001
367999199220,5cyclictest26-21ksoftirqd/122:46:141
367999199220,5cyclictest26-21ksoftirqd/100:38:381
3679987992220,1cyclictest0-21swapper/022:48:420
3679987992220,1cyclictest0-21swapper/022:30:030
3679987992220,1cyclictest0-21swapper/022:30:030
3679987992220,1cyclictest0-21swapper/020:57:220
3679987992220,1cyclictest0-21swapper/020:20:520
3679987992220,1cyclictest0-21swapper/019:44:080
3679987992220,1cyclictest0-21swapper/019:32:390
3679987992215,5cyclictest3724036-21munin-run21:15:010
3679987992214,6cyclictest3780262-21cron23:55:000
367998799220,21cyclictest3795052-21diskstats00:35:150
3679996992119,1cyclictest3056-21PK-Backend23:18:023
3679996992119,1cyclictest3056-21PK-Backend23:08:223
3679996992119,1cyclictest3056-21PK-Backend22:10:363
3679996992119,1cyclictest3056-21PK-Backend21:58:333
3679996992119,1cyclictest3056-21PK-Backend20:59:043
3679996992119,1cyclictest3056-21PK-Backend20:15:053
3679996992119,1cyclictest3056-21PK-Backend19:40:243
3679996992119,1cyclictest3056-21PK-Backend19:28:063
3679996992119,1cyclictest0-21swapper/319:24:003
3679996992119,1cyclictest0-21swapper/300:10:273
3679996992119,1cyclictest0-21swapper/300:10:273
3679996992119,1cyclictest0-21swapper/300:02:503
3679996992118,2cyclictest3056-21PK-Backend23:54:123
3679996992118,2cyclictest3056-21PK-Backend21:13:313
3679996992118,2cyclictest3056-21PK-Backend19:35:163
3679996992118,2cyclictest3056-21PK-Backend00:34:543
3679996992118,2cyclictest3056-21PK-Backend00:21:093
3679996992118,2cyclictest3056-21PK-Backend00:21:093
367999699210,19cyclictest3056-21PK-Backend20:25:043
3679993992112,7cyclictest3771006-21python323:25:212
367999399210,20cyclictest0-21swapper/222:37:542
367999199214,5cyclictest3718720-21seq20:58:511
367999199213,7cyclictest3697727-21taskset19:59:441
367999199213,6cyclictest3748418-21users22:20:221
367999199212,7cyclictest3700254-21kernelversion20:05:161
3679991992119,1cyclictest3056-21PK-Backend21:34:441
3679991992119,1cyclictest3056-21PK-Backend21:08:281
3679991992119,1cyclictest0-21swapper/123:45:441
3679991992119,1cyclictest0-21swapper/123:32:461
3679991992119,1cyclictest0-21swapper/122:11:551
3679991992119,1cyclictest0-21swapper/121:12:201
3679991992119,1cyclictest0-21swapper/119:20:051
3679991992116,3cyclictest3716706-21users20:50:231
367999199210,20cyclictest3056-21PK-Backend22:55:071
367999199210,20cyclictest0-21swapper/121:35:451
367999199210,20cyclictest0-21swapper/120:23:121
367999199210,1cyclictest0-21swapper/120:13:121
3679987992119,1cyclictest0-21swapper/023:42:090
3679987992119,1cyclictest0-21swapper/023:34:310
3679987992119,1cyclictest0-21swapper/023:27:080
3679987992119,1cyclictest0-21swapper/023:20:460
3679987992119,1cyclictest0-21swapper/022:06:300
3679987992119,1cyclictest0-21swapper/022:03:260
3679987992119,1cyclictest0-21swapper/021:59:210
3679987992119,1cyclictest0-21swapper/020:53:300
3679987992119,1cyclictest0-21swapper/020:02:060
3679987992119,1cyclictest0-21swapper/019:49:290
3679987992119,1cyclictest0-21swapper/019:27:320
3679987992119,1cyclictest0-21swapper/019:24:280
3679987992116,3cyclictest3690837-21idleruntime-cro19:40:000
3679987992113,6cyclictest3777443-21df_inode23:45:130
3679987992113,6cyclictest3681917-21python319:15:010
367998799210,20cyclictest0-21swapper/023:00:250
367998799210,1cyclictest0-21swapper/021:47:170
367998799210,19cyclictest3056-21PK-Backend21:31:220
139099210,18rtkit-daemon3681916-21munin-node22:22:222
37465572200,5sleep013-21ksoftirqd/022:15:220
3679996992018,1cyclictest3056-21PK-Backend23:49:523
3679996992018,1cyclictest3056-21PK-Backend21:17:243
3679996992018,1cyclictest3056-21PK-Backend20:36:473
3679996992018,1cyclictest3056-21PK-Backend20:33:083
3679996992018,1cyclictest3056-21PK-Backend20:14:023
3679996992018,1cyclictest3056-21PK-Backend19:31:103
3679996992018,1cyclictest3056-21PK-Backend19:15:163
3679996992018,1cyclictest0-21swapper/319:46:453
3679996992018,1cyclictest0-21swapper/300:39:133
3679996992017,2cyclictest3056-21PK-Backend23:40:123
3679996992017,2cyclictest3056-21PK-Backend22:56:533
3679996992017,2cyclictest3056-21PK-Backend19:55:443
367999699200,1cyclictest3056-21PK-Backend23:59:323
367999699200,19cyclictest3056-21PK-Backend22:38:003
367999699200,19cyclictest3056-21PK-Backend21:46:433
367999699200,19cyclictest3056-21PK-Backend20:53:233
367999699200,19cyclictest3056-21PK-Backend20:04:153
367999699200,19cyclictest3056-21PK-Backend00:29:333
367999699200,19cyclictest0-21swapper/321:31:013
3679993992018,1cyclictest0-21swapper/220:26:542
3679993992013,5cyclictest3692721-21wc19:45:002
3679993992012,6cyclictest3761833-21df_inode23:00:132
367999399200,3cyclictest3722405-21latency_hist21:10:012
367999399200,1cyclictest0-21swapper/223:52:442
367999399200,1cyclictest0-21swapper/220:53:252
367999399200,1cyclictest0-21swapper/220:36:482
367999399200,19cyclictest0-21swapper/223:48:252
367999399200,19cyclictest0-21swapper/222:58:022
367999399200,19cyclictest0-21swapper/220:14:512
367999399200,19cyclictest0-21swapper/219:59:102
367999399200,19cyclictest0-21swapper/219:35:372
367999199203,6cyclictest3792323-21runrttasks00:25:411
367999199203,5cyclictest3719098-21apt21:00:121
367999199202,7cyclictest3695848-21runrttasks19:52:511
367999199202,13cyclictest3680213-21apt19:10:111
3679991992018,1cyclictest0-21swapper/123:43:271
3679991992018,1cyclictest0-21swapper/122:06:481
3679991992018,1cyclictest0-21swapper/121:59:381
3679991992018,1cyclictest0-21swapper/121:29:501
3679991992018,1cyclictest0-21swapper/121:29:501
3679991992018,1cyclictest0-21swapper/100:20:321
3679991992018,1cyclictest0-21swapper/100:20:321
3679991992018,1cyclictest0-21swapper/100:11:531
3679991992018,1cyclictest0-21swapper/100:11:531
3679991992014,4cyclictest1095-21runrttasks21:50:211
3679991992013,5cyclictest3711885-21cron20:40:001
367999199200,6cyclictest26-21ksoftirqd/119:45:221
367999199200,1cyclictest0-21swapper/120:01:231
367999199200,19cyclictest3056-21PK-Backend22:34:121
367999199200,19cyclictest3056-21PK-Backend22:34:121
367999199200,19cyclictest0-21swapper/123:54:221
3679987992018,1cyclictest0-21swapper/023:38:510
3679987992018,1cyclictest0-21swapper/022:20:500
3679987992018,1cyclictest0-21swapper/021:40:210
3679987992018,1cyclictest0-21swapper/020:26:470
3679987992018,1cyclictest0-21swapper/000:22:170
3679987992018,1cyclictest0-21swapper/000:22:170
3679987992018,1cyclictest0-21swapper/000:17:580
3679987992018,1cyclictest0-21swapper/000:13:390
3679987992018,1cyclictest0-21swapper/000:13:390
3679987992012,2cyclictest13-21ksoftirqd/000:05:220
3679987992011,2cyclictest13-21ksoftirqd/000:00:200
3679987992010,9cyclictest3744337-21if_err_enp2s022:10:150
367998799200,1cyclictest0-21swapper/019:12:260
367998799200,19cyclictest0-21swapper/023:15:250
367998799200,19cyclictest0-21swapper/023:10:050
367998799200,19cyclictest0-21swapper/021:07:570
367998799200,19cyclictest0-21swapper/020:36:400
367998799200,19cyclictest0-21swapper/020:05:580
139099200,14rtkit-daemon0-21swapper/119:33:011
37058292190,4sleep23056-21PK-Backend20:20:202
3679996991917,1cyclictest3056-21PK-Backend22:26:453
3679996991917,1cyclictest0-21swapper/321:01:553
367999699190,1cyclictest3056-21PK-Backend23:26:273
367999699190,1cyclictest3056-21PK-Backend22:17:323
367999699190,1cyclictest3056-21PK-Backend21:41:503
367999699190,1cyclictest3056-21PK-Backend21:21:153
367999699190,1cyclictest3056-21PK-Backend00:15:483
367999699190,18cyclictest3056-21PK-Backend22:44:093
367999699190,18cyclictest3056-21PK-Backend22:21:383
367999699190,18cyclictest3056-21PK-Backend21:51:363
367999699190,17cyclictest3695147-21memory19:50:173
3679993991917,1cyclictest0-21swapper/223:19:242
3679993991917,1cyclictest0-21swapper/222:50:392
3679993991917,1cyclictest0-21swapper/221:28:262
3679993991917,1cyclictest0-21swapper/221:28:262
3679993991917,1cyclictest0-21swapper/221:02:102
3679993991917,1cyclictest0-21swapper/220:05:042
3679993991917,1cyclictest0-21swapper/219:47:202
3679993991917,1cyclictest0-21swapper/219:25:232
3679993991917,1cyclictest0-21swapper/200:19:552
3679993991913,4cyclictest3744066-21tr22:10:122
3679993991911,6cyclictest3683135-21awk19:15:182
367999399190,5cyclictest3785766-21idleruntime-cro00:10:012
367999399190,1cyclictest0-21swapper/223:30:062
367999399190,1cyclictest0-21swapper/221:42:182
367999399190,1cyclictest0-21swapper/221:33:192
367999399190,1cyclictest0-21swapper/220:31:212
367999399190,1cyclictest0-21swapper/219:21:192
367999399190,18cyclictest0-21swapper/222:08:272
367999399190,18cyclictest0-21swapper/221:55:232
367999399190,18cyclictest0-21swapper/221:52:182
367999399190,18cyclictest0-21swapper/220:44:052
367999399190,18cyclictest0-21swapper/200:30:362
367999199196,8cyclictest3712167-21apt20:40:111
367999199193,4cyclictest3764310-21runrttasks23:05:421
3679991991917,1cyclictest0-21swapper/123:28:271
3679991991914,3cyclictest3769875-21latency_hist23:25:001
3679991991912,5cyclictest3765512-21memory23:10:161
3679991991910,6cyclictest3741275-21seq22:00:191
367999199190,1cyclictest0-21swapper/120:46:311
367999199190,18cyclictest0-21swapper/122:39:191
3679987991917,1cyclictest0-21swapper/022:53:010
3679987991917,1cyclictest0-21swapper/022:25:570
3679987991917,1cyclictest0-21swapper/021:51:230
3679987991917,1cyclictest0-21swapper/021:38:180
3679987991917,1cyclictest0-21swapper/021:04:050
3679987991917,1cyclictest0-21swapper/019:55:240
3679987991913,5cyclictest3755870-21cat22:45:010
367998799190,1cyclictest0-21swapper/020:31:130
367998799190,18cyclictest0-21swapper/020:49:380
3679996991816,1cyclictest3056-21PK-Backend22:01:373
3679996991816,1cyclictest3056-21PK-Backend21:37:583
3679996991816,1cyclictest3056-21PK-Backend21:27:093
3679996991816,1cyclictest3056-21PK-Backend21:27:093
3679996991816,1cyclictest3056-21PK-Backend20:48:433
3679993991816,1cyclictest0-21swapper/223:14:042
3679993991816,1cyclictest0-21swapper/223:09:452
3679993991816,1cyclictest0-21swapper/222:15:372
3679993991816,1cyclictest0-21swapper/222:00:302
3679993991816,1cyclictest0-21swapper/221:18:392
3679993991816,1cyclictest0-21swapper/220:48:442
3679993991814,2cyclictest3724025-21idleruntime-cro21:15:002
3679993991812,5cyclictest3786457-21cat00:10:152
3679993991812,5cyclictest3786457-21cat00:10:152
3679993991812,5cyclictest3727168-21awk21:20:222
3679993991811,5cyclictest3789631-21cut00:20:122
3679993991811,5cyclictest3789631-21cut00:20:122
3679993991810,6cyclictest3781119-21memory23:55:182
367999399180,3cyclictest141rcu_preempt22:44:552
367999199182,6cyclictest26-21ksoftirqd/122:25:441
367999199182,12cyclictest3727154-21taskset21:20:221
3679991991812,4cyclictest3707102-21smartctl20:25:151
367998799186,2cyclictest141rcu_preempt20:17:010
3679987991816,1cyclictest0-21swapper/019:54:220
3679987991816,1cyclictest0-21swapper/000:25:340
3679987991812,5cyclictest13-21ksoftirqd/023:05:120
3679987991811,6cyclictest3728017-21awk21:25:100
3679987991811,6cyclictest3728017-21awk21:25:100
367999399175,2cyclictest141rcu_preempt22:25:142
367999399172,5cyclictest141rcu_preempt19:14:222
3679993991715,1cyclictest0-21swapper/223:23:442
3679993991715,1cyclictest0-21swapper/220:55:272
3679993991715,1cyclictest0-21swapper/220:18:572
3679993991715,1cyclictest0-21swapper/220:01:122
367999199170,7cyclictest26-21ksoftirqd/122:40:181
3679987991712,4cyclictest13-21ksoftirqd/021:20:010
3679987991711,4cyclictest3752685-21cpuspeed_turbos22:35:130
3679987991710,6cyclictest3794064-21seq00:30:480
139099170,13rtkit-daemon1389-21rtkit-daemon21:39:352
3679996991610,5cyclictest3680728-21grep19:10:163
367999399162,5cyclictest141rcu_preempt21:45:152
367999399161,5cyclictest141rcu_preempt23:40:142
3679993991614,1cyclictest0-21swapper/219:30:302
3679993991612,3cyclictest3750785-21/usr/sbin/munin22:30:152
3679993991612,3cyclictest3750785-21/usr/sbin/munin22:30:152
367999199162,9cyclictest3761468-21apt23:00:111
367998799168,3cyclictest3760480-21sh22:55:180
3679987991611,3cyclictest37819222sleep023:59:090
367999699151,6cyclictest978-21snmpd22:49:593
367999699150,7cyclictest3056-21PK-Backend23:20:193
367999399159,4cyclictest3694698-21grep19:50:142
3679993991511,3cyclictest3795377-21netstat00:35:192
367999699140,12cyclictest3056-21PK-Backend23:35:123
367999699128,3cyclictest1009-21lldpd23:02:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional