You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 23:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu May 28, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
238952622450,9sleep10-21swapper/107:10:001
24684262990,2sleep13056-21PK-Backend10:50:151
238935128138,14sleep00-21swapper/007:08:240
238923826939,23sleep30-21swapper/307:06:593
238937826343,14sleep20-21swapper/207:08:452
24087042470,3sleep30-21swapper/308:00:193
24100262460,3sleep30-21swapper/308:05:123
24706752450,3sleep20-21swapper/210:55:222
238961099280,19cyclictest3056-21PK-Backend08:32:163
24903822270,4sleep13056-21PK-Backend11:50:251
139099260,4rtkit-daemon1389-21rtkit-daemon08:27:311
139099260,4rtkit-daemon1389-21rtkit-daemon07:12:281
238960499250,23cyclictest24284012sleep108:55:421
2389610992415,7cyclictest2489007-21cron11:50:013
238960699241,2cyclictest2492754-21apt12:00:102
2389604992414,6cyclictest26-21ksoftirqd/111:24:591
2389606992319,3cyclictest1788643-21eadedCompositor10:07:152
238960099236,3cyclictest2411171-21turbostat08:09:590
2389600992316,5cyclictest13-21ksoftirqd/011:10:120
238961099226,14cyclictest0-21swapper/310:55:013
2389610992220,1cyclictest3056-21PK-Backend12:28:553
238961099220,20cyclictest2433802-21chrt09:13:133
2389606992218,3cyclictest1095-21runrttasks11:44:552
2389606992214,6cyclictest2501135-21cron12:25:002
2389604992220,1cyclictest0-21swapper/112:20:251
2389604992220,1cyclictest0-21swapper/111:10:071
2389604992219,2cyclictest3056-21PK-Backend10:57:371
2389604992219,2cyclictest3056-21PK-Backend10:42:291
238960499220,21cyclictest0-21swapper/108:35:221
2389600992220,1cyclictest0-21swapper/010:29:510
2389600992220,1cyclictest0-21swapper/010:23:220
2389600992220,1cyclictest0-21swapper/009:33:200
2389600992220,1cyclictest0-21swapper/009:16:580
2389600992220,1cyclictest0-21swapper/008:21:080
2389600992214,7cyclictest2394187-21/usr/sbin/munin07:20:170
2389600992214,6cyclictest2413911-21awk08:15:190
238960099220,21cyclictest0-21swapper/010:00:230
238960099220,21cyclictest0-21swapper/009:39:140
238960099220,20cyclictest0-21swapper/011:22:570
139099220,4rtkit-daemon1389-21rtkit-daemon10:17:590
24447592210,3sleep30-21swapper/309:45:003
24378782210,3sleep22437882-21cpuspeed_turbos09:25:122
2389610992117,3cyclictest1009-21lldpd12:12:403
2389610992117,3cyclictest1009-21lldpd12:12:393
238961099210,20cyclictest3056-21PK-Backend11:27:243
238961099210,20cyclictest2471399-21/usr/sbin/munin11:00:203
238961099210,20cyclictest0-21swapper/311:32:313
238961099210,1cyclictest3056-21PK-Backend08:23:053
238961099210,19cyclictest3056-21PK-Backend08:37:103
238961099210,19cyclictest3056-21PK-Backend08:11:213
238961099210,19cyclictest3056-21PK-Backend07:54:393
238961099210,19cyclictest3056-21PK-Backend07:46:403
238961099210,19cyclictest3056-21PK-Backend07:23:283
238961099210,19cyclictest3056-21PK-Backend07:13:433
238960499212,15cyclictest2430752-21apt09:05:111
2389604992119,1cyclictest3056-21PK-Backend11:44:001
2389604992119,1cyclictest0-21swapper/112:18:011
2389604992119,1cyclictest0-21swapper/110:30:471
2389604992119,1cyclictest0-21swapper/109:32:401
2389604992119,1cyclictest0-21swapper/109:23:271
2389604992119,1cyclictest0-21swapper/109:03:341
2389604992119,1cyclictest0-21swapper/108:45:491
2389604992119,1cyclictest0-21swapper/107:55:471
2389604992119,1cyclictest0-21swapper/107:40:541
2389604992118,2cyclictest3056-21PK-Backend11:31:171
2389604992114,5cyclictest883-21dbus-daemon10:30:001
238960499210,20cyclictest3056-21PK-Backend12:09:421
238960499210,20cyclictest0-21swapper/108:51:371
238960499210,20cyclictest0-21swapper/107:47:361
238960499210,19cyclictest3056-21PK-Backend12:12:401
238960499210,19cyclictest3056-21PK-Backend12:12:401
2389600992119,1cyclictest0-21swapper/012:37:330
2389600992119,1cyclictest0-21swapper/012:17:180
2389600992119,1cyclictest0-21swapper/011:31:010
2389600992119,1cyclictest0-21swapper/011:00:120
2389600992119,1cyclictest0-21swapper/009:47:330
2389600992119,1cyclictest0-21swapper/008:45:490
2389600992119,1cyclictest0-21swapper/007:27:140
238960099210,20cyclictest0-21swapper/012:24:140
238960099210,20cyclictest0-21swapper/008:36:480
238960099210,20cyclictest0-21swapper/007:53:370
238960099210,1cyclictest0-21swapper/007:13:490
139099210,4rtkit-daemon1389-21rtkit-daemon10:40:450
139099210,4rtkit-daemon1389-21rtkit-daemon10:10:560
24476232200,4sleep20-21swapper/209:50:212
238961099209,9cyclictest2494068-21NetworkManager12:03:183
2389610992018,1cyclictest0-21swapper/310:51:073
2389610992018,1cyclictest0-21swapper/310:06:463
2389610992018,1cyclictest0-21swapper/309:53:223
2389610992018,1cyclictest0-21swapper/307:35:323
2389610992017,2cyclictest3056-21PK-Backend09:36:113
2389610992017,2cyclictest3056-21PK-Backend08:46:103
2389610992014,5cyclictest2456008-21if_err_enp2s010:15:163
2389610992014,4cyclictest2478371-21apt11:20:123
2389610992012,6cyclictest2457680-21awk10:20:153
238961099200,1cyclictest3056-21PK-Backend09:08:063
238961099200,19cyclictest3056-21PK-Backend10:32:083
238961099200,19cyclictest2485506-21cnf-update-db11:40:023
238961099200,19cyclictest0-21swapper/312:15:023
238961099200,19cyclictest0-21swapper/312:07:063
238961099200,19cyclictest0-21swapper/311:36:163
238961099200,19cyclictest0-21swapper/310:44:053
238961099200,19cyclictest0-21swapper/310:04:373
238961099200,19cyclictest0-21swapper/309:45:523
238961099200,18cyclictest3056-21PK-Backend12:35:513
2389606992018,1cyclictest0-21swapper/211:06:572
2389606992018,1cyclictest0-21swapper/210:35:482
2389606992018,1cyclictest0-21swapper/210:11:342
2389606992018,1cyclictest0-21swapper/208:50:242
2389606992018,1cyclictest0-21swapper/208:09:402
2389606992013,5cyclictest2488926-21sshd11:49:012
2389606992013,5cyclictest2451702-21sed10:05:012
238960699201,18cyclictest0-21swapper/207:36:222
238960699200,1cyclictest0-21swapper/207:41:022
238960699200,1cyclictest0-21swapper/207:14:532
238960699200,19cyclictest0-21swapper/212:32:212
238960699200,19cyclictest0-21swapper/210:29:062
238960699200,19cyclictest0-21swapper/208:59:232
238960699200,19cyclictest0-21swapper/208:15:552
238960699200,19cyclictest0-21swapper/207:56:092
238960699200,19cyclictest0-21swapper/207:24:042
2389604992018,1cyclictest3056-21PK-Backend10:45:471
2389604992018,1cyclictest3056-21PK-Backend09:47:131
2389604992018,1cyclictest0-21swapper/112:37:271
2389604992018,1cyclictest0-21swapper/108:34:481
2389604992018,1cyclictest0-21swapper/108:21:221
2389604992018,1cyclictest0-21swapper/107:50:391
2389604992018,1cyclictest0-21swapper/107:39:511
2389604992018,1cyclictest0-21swapper/107:19:451
2389604992018,1cyclictest0-21swapper/107:19:441
238960499200,1cyclictest3056-21PK-Backend11:19:061
238960499200,1cyclictest0-21swapper/109:10:031
238960499200,19cyclictest3056-21PK-Backend09:42:551
238960499200,19cyclictest0-21swapper/111:02:171
238960499200,19cyclictest0-21swapper/110:37:161
238960499200,19cyclictest0-21swapper/107:23:571
2389600992018,1cyclictest0-21swapper/012:30:380
2389600992018,1cyclictest0-21swapper/012:28:140
2389600992018,1cyclictest0-21swapper/012:02:240
2389600992018,1cyclictest0-21swapper/011:55:420
2389600992018,1cyclictest0-21swapper/011:54:540
2389600992018,1cyclictest0-21swapper/010:35:450
2389600992018,1cyclictest0-21swapper/010:34:580
2389600992018,1cyclictest0-21swapper/009:59:360
2389600992018,1cyclictest0-21swapper/009:52:060
2389600992018,1cyclictest0-21swapper/009:22:460
2389600992018,1cyclictest0-21swapper/008:58:390
2389600992018,1cyclictest0-21swapper/007:49:310
2389600992018,1cyclictest0-21swapper/007:37:540
2389600992017,2cyclictest0-21swapper/009:05:350
2389600992015,3cyclictest2420095-21munin-run08:35:000
2389600992013,5cyclictest2439157-21perf09:30:000
2389600992012,6cyclictest2417961-21awk08:25:210
238960099200,1cyclictest0-21swapper/009:40:020
238960099200,19cyclictest0-21swapper/012:12:330
238960099200,19cyclictest0-21swapper/012:12:320
238960099200,19cyclictest0-21swapper/011:40:420
238960099200,19cyclictest0-21swapper/011:36:570
238960099200,19cyclictest0-21swapper/010:59:250
238960099200,19cyclictest0-21swapper/010:51:000
238960099200,19cyclictest0-21swapper/010:08:140
238960099200,19cyclictest0-21swapper/008:00:520
2389610991917,1cyclictest3056-21PK-Backend09:00:223
2389610991917,1cyclictest3056-21PK-Backend07:33:433
2389610991917,1cyclictest0-21swapper/311:51:163
2389610991917,1cyclictest0-21swapper/311:17:243
2389610991917,1cyclictest0-21swapper/310:46:153
2389610991913,4cyclictest2435941-21/usr/sbin/munin09:20:153
238961099190,1cyclictest3056-21PK-Backend11:13:053
238961099190,1cyclictest3056-21PK-Backend08:59:343
238961099190,1cyclictest3056-21PK-Backend07:28:303
238961099190,1cyclictest0-21swapper/311:55:023
238961099190,1cyclictest0-21swapper/309:15:433
238961099190,1cyclictest0-21swapper/308:44:133
2389606991917,1cyclictest0-21swapper/212:36:212
2389606991917,1cyclictest0-21swapper/212:15:522
2389606991917,1cyclictest0-21swapper/211:30:572
2389606991917,1cyclictest0-21swapper/211:22:532
2389606991917,1cyclictest0-21swapper/210:54:002
2389606991917,1cyclictest0-21swapper/209:43:562
2389606991917,1cyclictest0-21swapper/209:12:402
2389606991917,1cyclictest0-21swapper/208:47:052
2389606991912,5cyclictest2490580-21sshd11:53:022
2389606991912,5cyclictest2483864-21expr11:35:122
2389606991911,6cyclictest436-20systemd-journal10:19:592
238960699191,16cyclictest2491192-21cut11:55:102
2389606991911,6cyclictest2448736-21cat09:55:142
2389606991910,7cyclictest2466606-21fw_packets10:45:142
238960699190,1cyclictest0-21swapper/211:27:132
238960699190,1cyclictest0-21swapper/209:37:152
238960699190,1cyclictest0-21swapper/209:18:142
238960699190,1cyclictest0-21swapper/209:09:362
238960699190,1cyclictest0-21swapper/208:21:372
238960699190,1cyclictest0-21swapper/207:45:542
238960699190,1cyclictest0-21swapper/207:32:442
238960699190,18cyclictest0-21swapper/212:11:062
238960699190,18cyclictest0-21swapper/212:11:052
238960699190,18cyclictest0-21swapper/211:16:232
238960499191,7cyclictest24378832cpuspeed_turbos09:25:111
2389604991917,1cyclictest3056-21PK-Backend11:36:581
2389604991917,1cyclictest3056-21PK-Backend10:17:421
2389604991917,1cyclictest3056-21PK-Backend10:03:301
2389604991917,1cyclictest3056-21PK-Backend08:07:441
2389604991917,1cyclictest0-21swapper/112:30:311
2389604991917,1cyclictest0-21swapper/107:32:351
2389604991913,4cyclictest2414769-21apt-key08:20:001
238960499190,1cyclictest3056-21PK-Backend11:55:241
238960499190,1cyclictest3056-21PK-Backend08:03:181
238960499190,1cyclictest0-21swapper/111:26:091
238960499190,1cyclictest0-21swapper/109:37:481
238960499190,18cyclictest3056-21PK-Backend12:01:051
238960499190,18cyclictest3056-21PK-Backend09:16:321
238960499190,18cyclictest0-21swapper/112:28:081
2389600991917,1cyclictest0-21swapper/011:48:120
2389600991917,1cyclictest0-21swapper/011:25:540
2389600991917,1cyclictest0-21swapper/008:43:180
2389600991917,1cyclictest0-21swapper/008:10:470
2389600991917,1cyclictest0-21swapper/007:43:350
2389600991912,5cyclictest2426001-21cat08:50:150
238961099188,7cyclictest42-21ksoftirqd/310:25:183
2389610991816,1cyclictest3056-21PK-Backend12:22:463
2389610991816,1cyclictest3056-21PK-Backend08:19:333
2389610991815,2cyclictest3056-21PK-Backend09:31:043
238961099180,3cyclictest2454899-21users10:10:203
238960699189,7cyclictest2495557-21users12:05:212
2389606991816,1cyclictest0-21swapper/209:31:202
2389606991812,5cyclictest2472875-21cron11:05:002
2389606991812,5cyclictest2465707-21basename10:45:002
2389606991811,5cyclictest2418572-21date08:30:012
2389604991816,1cyclictest3056-21PK-Backend09:51:321
2389604991816,1cyclictest0-21swapper/107:26:331
2389604991811,6cyclictest2473897-21ntpq11:05:191
2389604991810,2cyclictest26-21ksoftirqd/111:49:551
238960499180,1cyclictest3056-21PK-Backend10:09:311
2389600991816,1cyclictest0-21swapper/011:19:260
2389600991816,1cyclictest0-21swapper/011:05:540
2389600991816,1cyclictest0-21swapper/009:10:490
2389600991811,5cyclictest13-21ksoftirqd/007:15:100
2389600991811,5cyclictest13-21ksoftirqd/007:15:100
2389610991715,1cyclictest3056-21PK-Backend12:30:313
2389610991715,1cyclictest3056-21PK-Backend11:09:323
2389610991715,1cyclictest3056-21PK-Backend10:35:053
2389610991715,1cyclictest3056-21PK-Backend09:26:453
2389610991711,4cyclictest2449016-21tail09:55:163
238960699179,6cyclictest2422671-21memory08:40:162
2389606991715,1cyclictest0-21swapper/211:14:142
2389606991715,1cyclictest0-21swapper/209:48:292
2389606991711,5cyclictest2421237-21awk08:35:212
2389606991710,6cyclictest883-21dbus-daemon07:53:112
2389606991710,6cyclictest2458396-21seq10:20:342
2389606991710,5cyclictest2501987-21sed12:25:172
238960699170,6cyclictest3056-21PK-Backend07:18:492
238960699170,6cyclictest3056-21PK-Backend07:18:492
2389604991715,1cyclictest3056-21PK-Backend10:21:471
2389604991715,1cyclictest3056-21PK-Backend09:58:351
2389604991714,2cyclictest3056-21PK-Backend10:10:181
238960499171,12cyclictest2412293-21chrt08:10:201
2389600991710,6cyclictest1-21systemd09:04:590
2389610991614,1cyclictest3056-21PK-Backend08:50:023
2389606991614,1cyclictest0-21swapper/210:32:512
2389606991614,1cyclictest0-21swapper/209:24:502
2389606991614,1cyclictest0-21swapper/209:03:422
2389606991614,1cyclictest0-21swapper/208:32:262
2389606991614,1cyclictest0-21swapper/208:00:352
2389606991613,2cyclictest2396976-21turbostat.cron07:29:592
238960099169,5cyclictest436-20systemd-journal07:57:000
139099160,13rtkit-daemon2637-21gmain07:30:560
238961099151,5cyclictest141rcu_preempt08:25:163
2389610991513,1cyclictest3056-21PK-Backend07:43:503
2389610991510,4cyclictest3056-21PK-Backend07:55:463
238960499151,11cyclictest2422093-21apt08:40:121
238960699148,4cyclictest2411620-21tr08:10:112
238960099149,3cyclictest2495024-21latency12:05:160
238960099148,5cyclictest2467197-21grep10:45:210
238961099130,5cyclictest141rcu_preempt07:16:083
238961099130,5cyclictest141rcu_preempt07:16:083
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional