You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-15 - 17:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Fri May 15, 2026 12:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
339524124270,20sleep342-21ksoftirqd/307:05:203
339514323260,11sleep234-21ksoftirqd/207:05:192
346942221260,3sleep30-21swapper/310:35:133
339463126743,15sleep10-21swapper/107:05:101
339607626344,13sleep00-21swapper/007:09:400
34280312490,5sleep30-21swapper/308:35:193
34673222430,4sleep03467311-21munin-run10:30:010
34152112400,4sleep13056-21PK-Backend08:00:131
339626699391,24cyclictest2085813-21snapd09:39:152
3396266993912,17cyclictest1095-21runrttasks08:35:162
339626699371,20cyclictest0-21swapper/211:49:472
339626699363,18cyclictest0-21swapper/208:04:262
339626699363,18cyclictest0-21swapper/207:21:552
339626699362,19cyclictest0-21swapper/208:53:232
339626699361,20cyclictest0-21swapper/212:31:282
339626699361,20cyclictest0-21swapper/210:14:512
339626699361,19cyclictest0-21swapper/209:19:332
339626699353,17cyclictest0-21swapper/210:38:122
339626699353,17cyclictest0-21swapper/210:08:532
339626699352,18cyclictest0-21swapper/211:28:402
339626699352,18cyclictest0-21swapper/210:00:272
339626699351,19cyclictest0-21swapper/211:09:202
339626699351,19cyclictest0-21swapper/208:48:342
339626699351,19cyclictest0-21swapper/208:27:162
339626699351,19cyclictest0-21swapper/207:47:392
339626699346,16cyclictest0-21swapper/209:11:032
339626699344,16cyclictest3418627-21cat08:10:102
339626699343,18cyclictest0-21swapper/210:32:102
339626699343,18cyclictest0-21swapper/209:29:152
339626699343,18cyclictest0-21swapper/208:55:482
339626699343,17cyclictest3472662-21taskset10:43:532
339626699343,17cyclictest0-21swapper/212:25:462
339626699342,18cyclictest0-21swapper/210:27:492
339626699342,18cyclictest0-21swapper/207:52:162
339626699342,17cyclictest0-21swapper/212:14:372
339626699342,17cyclictest0-21swapper/212:00:362
339626699341,20cyclictest0-21swapper/209:08:582
339626699341,19cyclictest0-21swapper/211:42:572
339626699341,19cyclictest0-21swapper/211:42:572
34783342330,2sleep00-21swapper/011:00:120
339626699333,18cyclictest0-21swapper/209:51:492
339626699333,18cyclictest0-21swapper/209:45:562
339626699333,18cyclictest0-21swapper/209:44:322
339626699332,19cyclictest0-21swapper/211:23:562
339626699332,18cyclictest0-21swapper/208:15:382
339626699332,18cyclictest0-21swapper/207:10:292
339626699331,20cyclictest0-21swapper/210:59:252
339626699324,16cyclictest0-21swapper/207:15:272
339626699322,19cyclictest0-21swapper/210:16:322
339626699322,18cyclictest0-21swapper/212:22:222
339626699322,18cyclictest0-21swapper/212:15:572
339626699322,18cyclictest0-21swapper/211:57:272
339626699322,18cyclictest0-21swapper/211:53:432
339626699322,18cyclictest0-21swapper/211:37:022
339626699322,18cyclictest0-21swapper/211:02:022
339626699322,18cyclictest0-21swapper/209:59:302
339626699322,18cyclictest0-21swapper/209:32:482
339626699322,18cyclictest0-21swapper/209:04:212
339626699322,18cyclictest0-21swapper/208:41:032
339626699322,18cyclictest0-21swapper/207:56:332
339626699322,17cyclictest0-21swapper/210:53:012
339626699321,19cyclictest3489752-21runrttasks11:30:482
339626699321,19cyclictest0-21swapper/209:24:152
339626699312,17cyclictest0-21swapper/211:19:592
339626199277,4cyclictest3511684-21cron12:35:000
3396261992717,3cyclictest13-21ksoftirqd/007:55:000
3396261992714,4cyclictest13-21ksoftirqd/008:45:130
339626199268,3cyclictest13-21ksoftirqd/011:30:010
339626199263,2cyclictest141rcu_preempt09:03:490
139099260,4rtkit-daemon1389-21rtkit-daemon10:45:453
139099260,4rtkit-daemon1389-21rtkit-daemon08:24:213
339627199240,22cyclictest3419033-21ntpq08:10:173
3396271992318,3cyclictest3426839-21taskset08:33:143
339626699235,16cyclictest3473761-21ntp_offset10:45:192
339626699234,17cyclictest3502215-21grep12:05:172
339626699231,20cyclictest2174-21JS10:27:482
3396261992318,4cyclictest13-21ksoftirqd/010:40:010
339626199231,3cyclictest3513426-21turbostat12:39:590
3396261992312,4cyclictest13-21ksoftirqd/008:34:000
139099230,19rtkit-daemon465-21systemd-udevd09:11:353
34593852220,3sleep00-21swapper/010:05:190
339627199222,18cyclictest3452777-21uptime09:45:213
3396271992220,1cyclictest0-21swapper/312:04:463
339627199220,20cyclictest0-21swapper/311:10:043
339627199220,20cyclictest0-21swapper/307:37:183
339626699228,13cyclictest3422004-21cat08:20:132
339626699228,12cyclictest3426505-21cat08:30:212
3396266992220,1cyclictest0-21swapper/207:41:122
339626199227,4cyclictest13-21ksoftirqd/011:13:020
3396261992220,1cyclictest0-21swapper/009:10:390
3396261992220,1cyclictest0-21swapper/008:38:010
3396261992220,1cyclictest0-21swapper/007:59:350
3396261992220,1cyclictest0-21swapper/007:33:050
339626199221,8cyclictest3460962-21sort10:10:170
3396261992213,2cyclictest141rcu_preempt10:57:050
3396261992212,2cyclictest141rcu_preempt07:35:570
3396261992211,4cyclictest13-21ksoftirqd/009:15:260
3396261992211,4cyclictest13-21ksoftirqd/008:58:300
3396261992211,4cyclictest13-21ksoftirqd/008:02:410
3396261992210,4cyclictest13-21ksoftirqd/009:29:190
3396261992210,3cyclictest13-21ksoftirqd/008:20:140
3396271992119,1cyclictest3056-21PK-Backend10:50:363
3396271992119,1cyclictest3056-21PK-Backend07:34:073
3396271992119,1cyclictest0-21swapper/312:34:133
3396271992119,1cyclictest0-21swapper/311:49:033
3396271992119,1cyclictest0-21swapper/310:56:243
3396271992119,1cyclictest0-21swapper/310:42:463
3396271992119,1cyclictest0-21swapper/309:51:493
3396271992119,1cyclictest0-21swapper/309:03:153
3396271992119,1cyclictest0-21swapper/307:48:063
3396271992114,6cyclictest3467335-21systemd-run10:30:003
339627199210,20cyclictest3056-21PK-Backend07:23:113
339627199210,20cyclictest0-21swapper/312:25:343
339627199210,20cyclictest0-21swapper/311:00:573
339627199210,20cyclictest0-21swapper/309:21:413
339627199210,19cyclictest3480324-21df11:05:133
339627199210,19cyclictest3056-21PK-Backend12:13:243
339627199210,19cyclictest0-21swapper/310:15:013
339626699210,9cyclictest3416970-21/usr/sbin/munin08:05:142
339626499210,20cyclictest3056-21PK-Backend12:15:431
339626499210,20cyclictest3056-21PK-Backend09:21:171
339626499210,1cyclictest3056-21PK-Backend09:55:241
339626499210,19cyclictest3056-21PK-Backend12:28:261
339626499210,19cyclictest3056-21PK-Backend12:02:441
339626499210,19cyclictest3056-21PK-Backend09:12:031
339626499210,19cyclictest3056-21PK-Backend08:48:241
339626499210,19cyclictest0-21swapper/110:11:591
339626199219,4cyclictest13-21ksoftirqd/011:56:230
339626199219,3cyclictest13-21ksoftirqd/010:51:530
339626199219,2cyclictest141rcu_preempt09:30:160
339626199216,3cyclictest13-21ksoftirqd/011:05:190
3396261992119,1cyclictest0-21swapper/012:23:160
3396261992119,1cyclictest0-21swapper/012:01:410
3396261992119,1cyclictest0-21swapper/011:47:130
3396261992119,1cyclictest0-21swapper/011:44:570
3396261992119,1cyclictest0-21swapper/011:44:570
3396261992119,1cyclictest0-21swapper/010:42:530
3396261992119,1cyclictest0-21swapper/009:56:150
3396261992119,1cyclictest0-21swapper/009:38:380
3396261992111,3cyclictest13-21ksoftirqd/011:23:220
3396261992111,3cyclictest13-21ksoftirqd/008:44:040
3396261992111,3cyclictest13-21ksoftirqd/008:19:500
3396261992111,3cyclictest13-21ksoftirqd/008:12:160
3396261992110,4cyclictest13-21ksoftirqd/012:19:440
3396261992110,4cyclictest13-21ksoftirqd/010:31:420
3396261992110,4cyclictest13-21ksoftirqd/010:18:500
3396261992110,4cyclictest13-21ksoftirqd/009:20:560
3396261992110,4cyclictest13-21ksoftirqd/007:13:320
339626199210,20cyclictest0-21swapper/010:47:110
339626199210,20cyclictest0-21swapper/008:07:050
339626199210,20cyclictest0-21swapper/007:42:520
339626199210,20cyclictest0-21swapper/007:16:140
339626199210,10cyclictest13-21ksoftirqd/011:50:120
339627199202,5cyclictest3496723-21diskstats11:50:133
3396271992018,1cyclictest0-21swapper/311:58:103
3396271992018,1cyclictest0-21swapper/311:42:153
3396271992018,1cyclictest0-21swapper/311:42:143
3396271992018,1cyclictest0-21swapper/311:37:423
3396271992018,1cyclictest0-21swapper/311:31:063
3396271992018,1cyclictest0-21swapper/311:22:133
3396271992018,1cyclictest0-21swapper/310:11:443
3396271992018,1cyclictest0-21swapper/308:56:533
3396271992018,1cyclictest0-21swapper/308:53:433
3396271992018,1cyclictest0-21swapper/308:47:353
3396271992015,3cyclictest3413477-21df07:55:133
3396271992013,6cyclictest3430050-21sed08:40:223
3396271992012,6cyclictest42-21ksoftirqd/308:25:013
3396271992011,7cyclictest2277-21gvfsd-trash09:39:213
339627199200,1cyclictest0-21swapper/308:09:293
339627199200,19cyclictest3056-21PK-Backend07:29:203
339627199200,19cyclictest0-21swapper/312:19:593
339627199200,19cyclictest0-21swapper/308:16:123
339627199200,19cyclictest0-21swapper/307:10:473
339627199200,18cyclictest436-20systemd-journal09:43:533
339626699200,8cyclictest3402330-21cat07:25:112
339626699200,1cyclictest0-21swapper/207:34:352
339626699200,19cyclictest0-21swapper/210:20:442
3396264992018,1cyclictest3056-21PK-Backend10:53:301
3396264992018,1cyclictest3056-21PK-Backend10:15:171
3396264992017,2cyclictest3056-21PK-Backend09:26:301
3396264992014,4cyclictest1095-21runrttasks11:15:191
3396264992010,4cyclictest26-21ksoftirqd/111:10:161
339626499200,1cyclictest3056-21PK-Backend09:03:051
339626499200,1cyclictest0-21swapper/112:10:211
339626499200,19cyclictest3056-21PK-Backend07:18:011
339626499200,19cyclictest0-21swapper/107:25:311
339626499200,19cyclictest0-21swapper/107:23:551
339626499200,18cyclictest3056-21PK-Backend12:21:041
339626499200,18cyclictest3056-21PK-Backend10:55:471
339626499200,18cyclictest3056-21PK-Backend08:34:191
339626199209,4cyclictest13-21ksoftirqd/011:30:330
339626199209,4cyclictest13-21ksoftirqd/010:04:440
3396261992018,1cyclictest0-21swapper/008:51:180
3396261992018,1cyclictest0-21swapper/008:26:030
3396261992018,1cyclictest0-21swapper/007:29:060
3396261992015,4cyclictest13-21ksoftirqd/007:50:000
3396261992010,3cyclictest13-21ksoftirqd/011:38:550
339626199200,19cyclictest0-21swapper/012:27:360
339626199200,19cyclictest0-21swapper/011:17:330
339626199200,19cyclictest0-21swapper/007:22:100
339626199200,10cyclictest13-21ksoftirqd/009:49:590
3396271991917,1cyclictest3056-21PK-Backend10:24:403
3396271991917,1cyclictest0-21swapper/312:38:323
3396271991917,1cyclictest0-21swapper/309:33:113
3396271991917,1cyclictest0-21swapper/309:26:073
3396271991917,1cyclictest0-21swapper/309:19:053
3396271991917,1cyclictest0-21swapper/308:01:373
3396271991914,3cyclictest3502029-21irqstats12:05:173
339627199191,16cyclictest436-20systemd-journal09:59:593
3396271991911,6cyclictest3507502-21ntpq12:20:203
339627199190,7cyclictest3467937-21df10:30:133
339627199190,1cyclictest0-21swapper/310:05:493
339627199190,18cyclictest0-21swapper/310:02:583
339627199190,18cyclictest0-21swapper/307:16:283
339626699191,4cyclictest436-20systemd-journal07:35:002
339626699190,3cyclictest34831102sleep211:14:342
339626499195,6cyclictest26-21ksoftirqd/108:15:141
339626499194,2cyclictest141rcu_preempt09:48:411
3396264991917,1cyclictest3056-21PK-Backend12:05:011
3396264991917,1cyclictest3056-21PK-Backend11:00:191
3396264991917,1cyclictest3056-21PK-Backend10:37:131
3396264991917,1cyclictest3056-21PK-Backend10:00:511
3396264991917,1cyclictest3056-21PK-Backend09:31:441
3396264991917,1cyclictest0-21swapper/108:12:551
3396264991916,2cyclictest3056-21PK-Backend10:06:181
3396264991916,2cyclictest3056-21PK-Backend07:51:401
3396264991913,4cyclictest436-20systemd-journal10:20:541
3396264991912,5cyclictest3406650-21sed07:35:211
339626499190,1cyclictest3056-21PK-Backend11:50:491
339626499190,1cyclictest3056-21PK-Backend08:23:231
339626499190,18cyclictest3056-21PK-Backend11:57:241
339626499190,18cyclictest3056-21PK-Backend10:46:401
339626499190,18cyclictest3056-21PK-Backend08:59:061
339626199199,4cyclictest13-21ksoftirqd/009:41:420
339626199198,2cyclictest141rcu_preempt09:50:210
339626199190,1cyclictest0-21swapper/009:05:390
3396271991816,1cyclictest0-21swapper/311:27:473
3396271991816,1cyclictest0-21swapper/309:09:523
3396271991810,6cyclictest3411750-21df_inode07:50:123
3396264991816,1cyclictest3056-21PK-Backend11:34:201
3396264991816,1cyclictest3056-21PK-Backend11:30:001
3396264991816,1cyclictest3056-21PK-Backend09:43:261
3396264991816,1cyclictest3056-21PK-Backend09:36:091
3396264991816,1cyclictest3056-21PK-Backend09:18:391
3396264991816,1cyclictest3056-21PK-Backend08:25:471
3396264991816,1cyclictest3056-21PK-Backend08:07:011
3396264991816,1cyclictest0-21swapper/111:40:541
3396264991816,1cyclictest0-21swapper/111:40:541
3396264991812,5cyclictest3466832-21cat10:25:211
339626499180,1cyclictest3056-21PK-Backend08:52:571
3396261991811,5cyclictest3463944-21mktemp10:20:010
34858822170,2sleep13056-21PK-Backend11:20:181
3396271991714,2cyclictest3056-21PK-Backend07:44:423
3396264991715,1cyclictest3056-21PK-Backend10:30:051
3396264991715,1cyclictest3056-21PK-Backend08:44:391
3396264991713,3cyclictest26-21ksoftirqd/107:45:001
3396264991712,3cyclictest3511692-21latency_hist12:34:591
3396264991711,5cyclictest3404521-21tr07:30:191
3396261991710,5cyclictest3504762-21idleruntime-cro12:15:000
339626199170,8cyclictest3056-21PK-Backend12:05:140
3396264991614,1cyclictest3056-21PK-Backend11:09:111
3396264991614,1cyclictest3056-21PK-Backend09:53:341
3396264991614,1cyclictest3056-21PK-Backend07:58:431
3396264991614,1cyclictest3056-21PK-Backend07:12:131
339626499161,3cyclictest3410053-21grep07:45:181
3396264991612,2cyclictest26-21ksoftirqd/109:08:301
3396271991513,1cyclictest3056-21PK-Backend11:18:553
3396264991511,2cyclictest26-21ksoftirqd/112:39:591
3396264991511,2cyclictest26-21ksoftirqd/111:48:091
3396264991511,2cyclictest26-21ksoftirqd/111:35:051
3396264991510,4cyclictest3428064-21cut08:35:201
339626499147,2cyclictest26-21ksoftirqd/110:40:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional