You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 00:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Apr 21, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
242220421050,4sleep00-21swapper/008:15:130
245042221010,3sleep00-21swapper/009:35:160
139099840,75rtkit-daemon0-21swapper/107:07:411
239729426741,17sleep20-21swapper/207:05:222
239792426546,13sleep00-21swapper/007:07:050
239790726343,14sleep30-21swapper/307:06:523
25027122470,2sleep30-21swapper/312:05:133
24097912400,3chrt3056-21PK-Backend07:40:141
239829599321,16cyclictest2481057-21munin-run11:05:002
239829599321,16cyclictest2481057-21munin-run11:04:592
2398295992812,5cyclictest2502382-21apt12:05:112
239828999280,10cyclictest2514492-21latency_hist12:40:020
24830492270,12sleep342-21ksoftirqd/311:10:113
2398295992714,11cyclictest2452322-21ntp_states09:40:192
239829599270,15cyclictest2441238-21expr09:10:142
24040542260,3sleep00-21swapper/007:25:010
239829599264,5cyclictest1968-21Xorg09:25:002
2398295992613,11cyclictest2472421-21unin-run10:40:012
239829599261,10cyclictest2450013-21cpuspeed_turbos09:35:132
239829599260,13cyclictest2422407-21sed08:15:162
239828999263,5cyclictest13-21ksoftirqd/009:45:000
239828999260,6cyclictest2427765-21grep08:30:200
139099260,20rtkit-daemon922-21systemd-logind09:48:342
2398298992511,12cyclictest2455039-21dump-pmu-power09:50:003
2398289992512,4cyclictest13-21ksoftirqd/010:07:210
239828999250,7cyclictest2511795-21grep12:30:200
239828999250,6cyclictest2434692-21grep08:50:180
239829599245,5cyclictest2513187-21fw_packets12:35:162
239829599242,8cyclictest2507332-21awk12:20:012
239829599240,10cyclictest2488068-21sed11:25:012
239828999241,5cyclictest2412893-21cron07:50:000
239828999240,6cyclictest2473384-21grep10:40:190
239829599235,4cyclictest2491720-21apt11:35:122
239829599234,5cyclictest2403986-21apt-config07:25:012
2398295992320,2cyclictest0-21swapper/210:30:132
239829599231,9cyclictest2473735-21users10:40:242
239829599231,9cyclictest2408498-21memory07:35:172
239829599231,6cyclictest2490723-21ntp_states11:30:172
239829599230,8cyclictest2475442-21timerwakeupswit10:45:222
239828999233,10cyclictest13-21ksoftirqd/007:45:000
239828999231,6cyclictest2460358-21latency_hist10:05:010
239828999230,7cyclictest2466420-21ntpq10:20:180
239828999230,7cyclictest2400282-21grep07:15:010
239828999230,6cyclictest2456020-21ntp_kernel_pll_09:50:190
24786332220,4chrt0-21swapper/110:55:211
2398298992220,1cyclictest0-21swapper/312:29:533
2398298992220,1cyclictest0-21swapper/310:34:493
2398298992220,1cyclictest0-21swapper/310:10:143
2398298992220,1cyclictest0-21swapper/310:09:143
2398298992220,1cyclictest0-21swapper/310:02:453
2398298992216,4cyclictest2439183-21apt09:05:123
2398298992214,6cyclictest1-21systemd09:30:013
239829599225,5cyclictest34-21ksoftirqd/212:20:222
2398295992220,1cyclictest0-21swapper/208:55:492
2398295992220,1cyclictest0-21swapper/208:46:572
2398295992220,1cyclictest0-21swapper/208:12:302
239829599221,8cyclictest2464839-21python310:15:192
239829599221,20cyclictest331rcuc/209:33:492
239829599221,12cyclictest0-21swapper/208:09:012
239829599220,9cyclictest2497717-21sed11:50:182
239829599220,8cyclictest2484126-21users11:10:222
239829599220,8cyclictest2443189-21latency09:15:172
239829599220,6cyclictest2431260-21ntp_states08:40:192
239829599220,21cyclictest0-21swapper/211:41:512
239829599220,21cyclictest0-21swapper/207:47:562
2398289992213,6cyclictest2491454-21latency_hist11:35:010
2398289992211,4cyclictest13-21ksoftirqd/011:55:090
2398289992211,4cyclictest13-21ksoftirqd/011:08:410
2398289992211,4cyclictest13-21ksoftirqd/010:36:360
2398289992211,4cyclictest13-21ksoftirqd/008:47:340
2398289992211,4cyclictest13-21ksoftirqd/008:04:520
2398289992211,4cyclictest13-21ksoftirqd/007:59:260
2398289992211,3cyclictest13-21ksoftirqd/009:10:110
239828999221,11cyclictest13-21ksoftirqd/011:25:000
239828999220,8cyclictest2476112-21sort10:50:120
239828999220,8cyclictest2476112-21sort10:50:120
239828999220,7cyclictest2446660-21latency09:25:150
139099220,14rtkit-daemon0-21swapper/311:15:573
239829899219,4cyclictest141rcu_preempt12:20:003
2398298992119,1cyclictest0-21swapper/310:38:403
2398298992119,1cyclictest0-21swapper/310:26:173
2398298992119,1cyclictest0-21swapper/309:57:173
2398298992119,1cyclictest0-21swapper/309:36:363
2398298992119,1cyclictest0-21swapper/309:22:093
2398298992119,1cyclictest0-21swapper/309:00:533
2398298992119,1cyclictest0-21swapper/308:35:453
2398298992119,1cyclictest0-21swapper/308:22:343
2398298992119,1cyclictest0-21swapper/307:39:073
2398298992119,1cyclictest0-21swapper/307:27:593
2398298992119,1cyclictest0-21swapper/307:15:553
239829899211,18cyclictest2419831-21cron08:10:003
239829899210,20cyclictest0-21swapper/312:31:093
239829899210,20cyclictest0-21swapper/312:00:203
239829899210,20cyclictest0-21swapper/309:18:163
239829899210,20cyclictest0-21swapper/308:44:093
2398295992119,1cyclictest0-21swapper/212:28:482
2398295992119,1cyclictest0-21swapper/211:57:452
2398295992119,1cyclictest0-21swapper/211:46:242
2398295992119,1cyclictest0-21swapper/209:58:552
2398295992119,1cyclictest0-21swapper/209:51:052
2398295992119,1cyclictest0-21swapper/209:29:352
2398295992119,1cyclictest0-21swapper/207:15:592
2398295992118,2cyclictest0-21swapper/208:51:232
239829599210,5cyclictest0-21swapper/211:05:132
239829599210,5cyclictest0-21swapper/208:30:172
239829599210,20cyclictest0-21swapper/207:30:052
2398293992119,1cyclictest3056-21PK-Backend07:15:451
239829399210,20cyclictest0-21swapper/111:37:041
239829399210,1cyclictest3056-21PK-Backend10:15:591
239829399210,19cyclictest3056-21PK-Backend11:19:251
239829399210,19cyclictest3056-21PK-Backend09:01:291
239829399210,19cyclictest3056-21PK-Backend08:24:261
239829399210,19cyclictest3056-21PK-Backend08:04:031
239828999218,2cyclictest141rcu_preempt09:20:000
2398289992115,5cyclictest13-21ksoftirqd/009:45:020
2398289992110,4cyclictest13-21ksoftirqd/012:28:370
2398289992110,4cyclictest13-21ksoftirqd/012:15:410
2398289992110,4cyclictest13-21ksoftirqd/012:02:220
2398289992110,4cyclictest13-21ksoftirqd/010:47:200
2398289992110,4cyclictest13-21ksoftirqd/010:32:000
2398289992110,4cyclictest13-21ksoftirqd/010:19:330
2398289992110,4cyclictest13-21ksoftirqd/010:11:380
2398289992110,4cyclictest13-21ksoftirqd/009:22:430
2398289992110,4cyclictest13-21ksoftirqd/008:12:490
2398289992110,4cyclictest13-21ksoftirqd/007:26:380
2398289992110,4cyclictest13-21ksoftirqd/007:12:430
239828999210,6cyclictest2407639-21cron07:35:010
239828999210,20cyclictest0-21swapper/009:57:490
239828999210,10cyclictest883-21dbus-daemon07:55:010
139099210,15rtkit-daemon0-21swapper/312:23:133
139099210,15rtkit-daemon0-21swapper/210:00:362
139099210,15rtkit-daemon0-21swapper/209:08:342
139099210,14rtkit-daemon0-21swapper/312:35:093
139099210,14rtkit-daemon0-21swapper/312:10:353
239829899202,16cyclictest2481456-21cpuspeed_turbos11:05:133
2398298992018,1cyclictest0-21swapper/311:29:093
2398298992018,1cyclictest0-21swapper/310:54:093
2398298992018,1cyclictest0-21swapper/310:54:093
2398298992018,1cyclictest0-21swapper/309:52:033
2398298992018,1cyclictest0-21swapper/309:32:233
2398298992018,1cyclictest0-21swapper/309:11:473
2398298992018,1cyclictest0-21swapper/308:59:523
2398298992018,1cyclictest0-21swapper/308:18:353
239829899201,6cyclictest2416798-21cpuspeed_turbos08:00:133
2398298992012,6cyclictest2497150-21apt-get11:50:043
239829899200,19cyclictest0-21swapper/311:55:463
239829899200,19cyclictest0-21swapper/310:44:213
239829899200,19cyclictest0-21swapper/308:30:453
239829899200,19cyclictest0-21swapper/307:31:373
239829599202,7cyclictest0-21swapper/210:20:152
239829599202,5cyclictest0-21swapper/212:00:172
2398295992018,1cyclictest0-21swapper/210:29:102
2398295992017,2cyclictest2460487-21/usr/sbin/munin10:05:112
2398295992013,5cyclictest2418020-21sshd08:04:152
239829599200,1cyclictest0-21swapper/212:30:032
239829599200,19cyclictest0-21swapper/210:11:072
239829399209,5cyclictest26-21ksoftirqd/109:45:011
2398293992018,1cyclictest3056-21PK-Backend10:36:271
2398293992018,1cyclictest3056-21PK-Backend07:55:191
2398293992017,2cyclictest3056-21PK-Backend10:22:281
2398293992017,2cyclictest3056-21PK-Backend08:30:131
239829399200,1cyclictest3056-21PK-Backend11:58:541
239829399200,1cyclictest3056-21PK-Backend10:13:091
239829399200,1cyclictest3056-21PK-Backend09:22:191
239829399200,1cyclictest3056-21PK-Backend08:51:021
239829399200,1cyclictest3056-21PK-Backend07:10:091
239829399200,19cyclictest3056-21PK-Backend10:05:511
239829399200,19cyclictest3056-21PK-Backend10:02:131
239829399200,18cyclictest3056-21PK-Backend10:33:361
239829399200,18cyclictest3056-21PK-Backend09:05:551
239829399200,18cyclictest3056-21PK-Backend08:40:011
239828999209,4cyclictest13-21ksoftirqd/011:04:440
239828999209,4cyclictest13-21ksoftirqd/011:04:430
239828999209,4cyclictest13-21ksoftirqd/010:57:250
239828999209,4cyclictest13-21ksoftirqd/009:31:010
239828999209,4cyclictest13-21ksoftirqd/009:02:100
239828999209,4cyclictest13-21ksoftirqd/008:42:520
239828999209,4cyclictest13-21ksoftirqd/008:25:530
239828999204,2cyclictest141rcu_preempt08:35:180
239828999203,7cyclictest13-21ksoftirqd/008:05:180
2398289992010,3cyclictest13-21ksoftirqd/011:35:200
2398289992010,3cyclictest13-21ksoftirqd/011:12:140
2398289992010,3cyclictest13-21ksoftirqd/008:56:000
239828999200,6cyclictest2468694-21nm-dispatcher10:27:270
239828999200,1cyclictest0-21swapper/009:09:080
239828999200,19cyclictest0-21swapper/011:49:560
239828999200,19cyclictest0-21swapper/011:26:100
139099200,14rtkit-daemon0-21swapper/311:02:023
139099200,14rtkit-daemon0-21swapper/311:02:023
139099200,14rtkit-daemon0-21swapper/310:57:463
139099200,14rtkit-daemon0-21swapper/310:23:433
139099200,14rtkit-daemon0-21swapper/211:27:232
139099200,14rtkit-daemon0-21swapper/210:55:332
139099200,14rtkit-daemon0-21swapper/209:02:452
139099200,13rtkit-daemon0-21swapper/212:11:252
2398298991917,1cyclictest0-21swapper/311:44:253
2398298991917,1cyclictest0-21swapper/310:17:443
2398298991917,1cyclictest0-21swapper/307:50:303
2398298991917,1cyclictest0-21swapper/307:46:373
2398298991917,1cyclictest0-21swapper/307:40:093
2398298991914,3cyclictest2403915-21grep07:24:593
2398298991912,5cyclictest2496690-21gpgv11:50:013
2398298991911,6cyclictest2487255-21ntp_states11:20:183
2398298991911,6cyclictest2434699-21ntpq08:50:183
2398298991910,6cyclictest957-21gdbus08:27:263
239829899190,1cyclictest0-21swapper/311:33:303
239829599197,10cyclictest0-21swapper/207:15:012
2398295991913,4cyclictest2428935-21expr08:35:132
239829599190,1cyclictest0-21swapper/207:51:482
239829599190,18cyclictest0-21swapper/207:40:262
239829599190,18cyclictest0-21swapper/207:29:032
239829399193,2cyclictest141rcu_preempt09:10:211
2398293991917,1cyclictest3056-21PK-Backend12:25:301
2398293991917,1cyclictest3056-21PK-Backend12:01:101
2398293991917,1cyclictest3056-21PK-Backend11:45:411
2398293991917,1cyclictest3056-21PK-Backend07:31:131
2398293991910,4cyclictest26-21ksoftirqd/107:50:231
239829399190,1cyclictest3056-21PK-Backend12:18:261
239829399190,1cyclictest3056-21PK-Backend12:05:291
239829399190,1cyclictest3056-21PK-Backend11:44:261
239829399190,1cyclictest3056-21PK-Backend11:30:421
239829399190,1cyclictest3056-21PK-Backend11:21:291
239829399190,1cyclictest3056-21PK-Backend10:42:561
239829399190,1cyclictest3056-21PK-Backend09:58:341
239829399190,1cyclictest3056-21PK-Backend07:36:391
239829399190,1cyclictest0-21swapper/112:21:591
239829399190,18cyclictest3056-21PK-Backend12:39:221
239829399190,18cyclictest3056-21PK-Backend08:47:381
239829399190,18cyclictest0-21swapper/109:50:581
239828999199,4cyclictest13-21ksoftirqd/012:24:310
239828999198,2cyclictest141rcu_preempt08:20:190
239828999197,4cyclictest13-21ksoftirqd/011:15:260
239828999197,4cyclictest13-21ksoftirqd/007:32:400
2398289991916,2cyclictest3056-21PK-Backend12:10:180
239828999190,5cyclictest1211-21kerneloops11:40:140
239828999190,1cyclictest0-21swapper/012:05:590
2398298991816,1cyclictest0-21swapper/307:56:253
2398298991811,5cyclictest2491711-21awk11:35:123
2398298991811,5cyclictest2451760-21grep09:40:123
2398298991810,6cyclictest2420470-21df08:10:123
239829899180,16cyclictest2399581-21/usr/sbin/munin07:10:233
239829599182,7cyclictest2484758-21/usr/sbin/munin11:15:142
2398293991816,1cyclictest3056-21PK-Backend10:48:371
2398293991816,1cyclictest3056-21PK-Backend09:15:021
2398293991816,1cyclictest3056-21PK-Backend07:22:201
2398293991815,2cyclictest3056-21PK-Backend12:30:161
2398293991815,2cyclictest3056-21PK-Backend11:13:311
2398293991815,2cyclictest3056-21PK-Backend08:58:521
2398293991811,6cyclictest2504502-21awk12:10:141
2398293991810,6cyclictest2451309-21cron09:40:011
2398298991713,2cyclictest2432253-21apt08:45:103
239829899170,15cyclictest2474750-21df10:45:133
239829599171,7cyclictest34-21ksoftirqd/207:57:412
239829599171,6cyclictest34-21ksoftirqd/208:25:192
2398295991710,5cyclictest2424509-21cat08:20:192
239829399178,4cyclictest26-21ksoftirqd/109:45:181
239829399175,3cyclictest26-21ksoftirqd/111:00:191
239829399175,3cyclictest26-21ksoftirqd/111:00:191
2398293991715,1cyclictest3056-21PK-Backend11:54:351
2398293991715,1cyclictest3056-21PK-Backend09:34:551
2398293991715,1cyclictest3056-21PK-Backend09:29:071
2398293991715,1cyclictest3056-21PK-Backend08:44:131
2398293991714,2cyclictest3056-21PK-Backend10:28:561
239829399170,15cyclictest1389-21rtkit-daemon08:10:281
239828999176,2cyclictest141rcu_preempt11:50:170
239829599160,5cyclictest34-21ksoftirqd/210:52:412
239829599160,5cyclictest34-21ksoftirqd/210:52:402
239829399161,6cyclictest26-21ksoftirqd/108:08:091
2398293991614,1cyclictest3056-21PK-Backend11:05:341
2398293991614,1cyclictest3056-21PK-Backend08:15:541
2398293991614,1cyclictest3056-21PK-Backend07:48:371
2398293991612,2cyclictest26-21ksoftirqd/110:50:111
2398293991612,2cyclictest26-21ksoftirqd/110:50:111
239829399152,2cyclictest141rcu_preempt11:25:121
2398293991410,2cyclictest26-21ksoftirqd/108:28:041
239829399140,6cyclictest3056-21PK-Backend07:27:441
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional