You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 22:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Tue Jun 16, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
52711422420,8sleep20-21swapper/207:05:162
58905521300,2sleep10-21swapper/110:00:111
59180121090,8sleep126-21ksoftirqd/110:05:201
1390991020,79rtkit-daemon0-21swapper/107:05:121
5437572790,3sleep30-21swapper/307:50:173
52802127859,13sleep00-21swapper/007:08:090
52703726945,16sleep30-21swapper/307:05:143
6386662420,1sleep30-21swapper/312:20:133
5914182400,2sleep30-21swapper/310:05:143
5752142380,3sleep20-21swapper/209:20:142
5893832370,3sleep20-21swapper/210:00:132
5361102280,3chrt0-21swapper/307:30:003
139099270,18rtkit-daemon551264-21python308:10:223
6073572250,2sleep00-21swapper/010:50:190
528322992519,4cyclictest542942-21turbostat07:50:003
52831299240,22cyclictest596234-21/usr/sbin/munin10:20:130
528322992317,4cyclictest42-21ksoftirqd/310:03:213
528320992317,5cyclictest622531-21expr11:35:122
528315992310,11cyclictest565719-21cron08:55:011
52831599230,21cyclictest0-21swapper/109:40:121
52831299237,2cyclictest5771462cut09:25:160
52832299225,15cyclictest592903-21cpuspeed_turbos10:10:133
52832299221,19cyclictest554612-21ntp_states08:20:193
52832299220,20cyclictest3056-21PK-Backend07:58:453
528320992220,1cyclictest0-21swapper/209:32:372
528320992220,1cyclictest0-21swapper/208:16:132
528320992220,1cyclictest0-21swapper/208:03:282
528315992220,1cyclictest0-21swapper/111:22:331
528315992220,1cyclictest0-21swapper/110:11:021
528315992214,6cyclictest572785-21sshd09:13:291
528312992219,2cyclictest617931-21smartctl11:20:230
528312992215,6cyclictest529977-21cron07:15:010
528322992119,1cyclictest0-21swapper/311:45:163
528322992113,6cyclictest571585-21grep09:10:113
52832299210,2cyclictest606533-21latency_hist10:50:003
52832299210,20cyclictest3056-21PK-Backend08:32:193
52832299210,19cyclictest3056-21PK-Backend08:36:593
528320992119,1cyclictest3056-21PK-Backend09:10:212
528320992119,1cyclictest0-21swapper/212:21:232
528320992119,1cyclictest0-21swapper/211:46:552
528320992119,1cyclictest0-21swapper/211:31:562
528320992119,1cyclictest0-21swapper/211:24:172
528320992119,1cyclictest0-21swapper/210:52:482
528320992119,1cyclictest0-21swapper/210:25:442
528320992119,1cyclictest0-21swapper/210:13:132
528320992119,1cyclictest0-21swapper/210:07:062
528320992119,1cyclictest0-21swapper/209:39:212
528320992119,1cyclictest0-21swapper/207:22:122
528320992114,6cyclictest539465-21cron07:39:592
52832099210,20cyclictest0-21swapper/211:02:272
52832099210,20cyclictest0-21swapper/210:49:302
52832099210,20cyclictest0-21swapper/210:33:552
52832099210,20cyclictest0-21swapper/209:51:572
52832099210,20cyclictest0-21swapper/209:06:432
52832099210,20cyclictest0-21swapper/208:55:402
528315992119,1cyclictest3056-21PK-Backend12:18:021
528315992119,1cyclictest3056-21PK-Backend11:50:581
528315992119,1cyclictest3056-21PK-Backend11:47:541
528315992119,1cyclictest3056-21PK-Backend11:44:511
528315992119,1cyclictest3056-21PK-Backend10:25:411
528315992119,1cyclictest3056-21PK-Backend08:29:021
528315992119,1cyclictest3056-21PK-Backend07:31:571
528315992119,1cyclictest3056-21PK-Backend07:31:571
528315992119,1cyclictest0-21swapper/111:10:501
528315992119,1cyclictest0-21swapper/109:25:391
528315992119,1cyclictest0-21swapper/109:22:491
528315992119,1cyclictest0-21swapper/109:04:161
528315992119,1cyclictest0-21swapper/108:10:561
528315992119,1cyclictest0-21swapper/108:07:181
528315992119,1cyclictest0-21swapper/108:01:021
528315992119,1cyclictest0-21swapper/107:42:511
528315992119,1cyclictest0-21swapper/107:12:021
528315992118,2cyclictest612963-21sshd11:05:291
52831599210,20cyclictest0-21swapper/108:46:051
52831599210,19cyclictest3056-21PK-Backend10:35:281
52831599210,19cyclictest3056-21PK-Backend09:19:111
528312992118,1cyclictest0-21swapper/009:24:220
52831299210,20cyclictest0-21swapper/008:17:100
52831299210,19cyclictest3056-21PK-Backend12:20:170
52831299210,19cyclictest3056-21PK-Backend09:30:030
139099210,15rtkit-daemon3056-21PK-Backend08:42:103
139099210,14rtkit-daemon3056-21PK-Backend11:14:453
139099210,14rtkit-daemon0-21swapper/310:25:033
139099210,14rtkit-daemon0-21swapper/208:05:112
52832299201,8cyclictest587995-21/usr/sbin/munin09:55:193
528322992018,1cyclictest0-21swapper/312:28:563
528322992018,1cyclictest0-21swapper/309:17:473
528322992018,1cyclictest0-21swapper/307:16:423
528322992014,4cyclictest2156-21gnome-shell10:15:093
52832299200,1cyclictest3056-21PK-Backend11:21:513
52832299200,1cyclictest0-21swapper/312:30:593
52832299200,19cyclictest3056-21PK-Backend08:07:383
52832299200,19cyclictest3056-21PK-Backend08:03:113
52832299200,19cyclictest0-21swapper/311:54:443
52832299200,19cyclictest0-21swapper/310:33:393
52832299200,19cyclictest0-21swapper/309:48:503
52832299200,19cyclictest0-21swapper/309:32:353
52832299200,19cyclictest0-21swapper/308:29:273
528320992018,1cyclictest0-21swapper/212:01:432
528320992018,1cyclictest0-21swapper/211:54:192
528320992018,1cyclictest0-21swapper/211:27:352
528320992018,1cyclictest0-21swapper/211:27:352
528320992018,1cyclictest0-21swapper/211:13:222
528320992018,1cyclictest0-21swapper/211:05:462
528320992018,1cyclictest0-21swapper/210:41:192
528320992018,1cyclictest0-21swapper/207:30:512
528320992018,1cyclictest0-21swapper/207:30:502
528320992018,1cyclictest0-21swapper/207:27:332
528320992018,1cyclictest0-21swapper/207:15:292
528320992018,1cyclictest0-21swapper/207:13:402
528320992012,6cyclictest436-20systemd-journal12:40:002
52832099200,1cyclictest0-21swapper/210:18:202
52832099200,1cyclictest0-21swapper/208:36:402
52832099200,19cyclictest0-21swapper/212:29:352
52832099200,19cyclictest0-21swapper/211:16:402
52832099200,19cyclictest0-21swapper/209:56:512
52832099200,19cyclictest0-21swapper/208:41:352
52832099200,19cyclictest0-21swapper/208:32:482
52832099200,19cyclictest0-21swapper/207:50:042
52831599209,7cyclictest6004952sleep110:30:201
528315992018,1cyclictest3056-21PK-Backend12:21:541
528315992018,1cyclictest3056-21PK-Backend11:38:561
528315992018,1cyclictest3056-21PK-Backend09:34:441
528315992018,1cyclictest0-21swapper/111:16:121
528315992018,1cyclictest0-21swapper/109:06:051
528315992018,1cyclictest0-21swapper/108:58:281
528315992018,1cyclictest0-21swapper/108:33:401
528315992018,1cyclictest0-21swapper/107:52:451
528315992013,5cyclictest560903-21cat08:40:121
52831599200,1cyclictest3056-21PK-Backend09:45:401
52831599200,19cyclictest3056-21PK-Backend12:27:351
52831599200,19cyclictest3056-21PK-Backend10:54:141
52831599200,19cyclictest0-21swapper/107:38:181
528312992018,1cyclictest0-21swapper/011:17:040
528312992018,1cyclictest0-21swapper/010:30:400
528312992018,1cyclictest0-21swapper/009:56:550
528312992018,1cyclictest0-21swapper/009:46:190
528312992018,1cyclictest0-21swapper/009:11:380
528312992018,1cyclictest0-21swapper/008:30:090
528312992018,1cyclictest0-21swapper/008:09:060
528312992018,1cyclictest0-21swapper/008:02:510
528312992013,5cyclictest540466-21ntpq07:40:200
52831299200,1cyclictest0-21swapper/012:15:110
52831299200,1cyclictest0-21swapper/012:11:050
52831299200,1cyclictest0-21swapper/010:09:580
52831299200,19cyclictest0-21swapper/012:02:530
52831299200,19cyclictest0-21swapper/010:38:510
52831299200,19cyclictest0-21swapper/010:15:050
52831299200,19cyclictest0-21swapper/008:40:290
52831299200,19cyclictest0-21swapper/007:47:090
139099200,14rtkit-daemon3056-21PK-Backend08:45:333
5381752190,4sleep33056-21PK-Backend07:35:143
52832299192,15cyclictest630941-21perf12:00:003
528322991917,1cyclictest3056-21PK-Backend11:09:073
528322991917,1cyclictest3056-21PK-Backend08:55:303
528322991914,3cyclictest608124-21idleruntime-cro10:55:003
528322991914,3cyclictest1095-21runrttasks07:31:413
528322991914,3cyclictest1095-21runrttasks07:31:413
528322991912,5cyclictest633665-21ntpq12:05:183
528322991911,6cyclictest533558-21awk07:20:233
528322991910,6cyclictest436-20systemd-journal11:31:343
52832299190,1cyclictest0-21swapper/312:36:063
52832299190,1cyclictest0-21swapper/312:15:103
52832299190,1cyclictest0-21swapper/311:00:433
52832299190,1cyclictest0-21swapper/309:44:583
52832299190,1cyclictest0-21swapper/309:35:253
52832299190,18cyclictest3056-21PK-Backend10:42:513
52832299190,18cyclictest0-21swapper/312:03:083
52832299190,18cyclictest0-21swapper/310:39:463
52832299190,18cyclictest0-21swapper/308:17:313
528320991917,1cyclictest0-21swapper/210:57:072
528320991917,1cyclictest0-21swapper/209:15:482
528320991917,1cyclictest0-21swapper/208:14:242
528320991917,1cyclictest0-21swapper/207:41:592
528320991912,6cyclictest553846-21find08:20:112
52832099190,1cyclictest0-21swapper/212:09:532
52832099190,1cyclictest0-21swapper/211:41:352
52832099190,1cyclictest0-21swapper/207:59:502
52832099190,1cyclictest0-21swapper/207:46:322
528315991917,1cyclictest3056-21PK-Backend10:23:381
528315991917,1cyclictest3056-21PK-Backend10:17:451
528315991917,1cyclictest3056-21PK-Backend07:24:591
528315991916,2cyclictest3056-21PK-Backend12:11:201
528315991916,2cyclictest3056-21PK-Backend11:30:581
528315991916,2cyclictest3056-21PK-Backend11:26:521
528315991916,2cyclictest3056-21PK-Backend11:26:521
52831599190,1cyclictest3056-21PK-Backend12:33:161
52831599190,1cyclictest3056-21PK-Backend12:02:341
52831599190,1cyclictest3056-21PK-Backend11:56:531
52831599190,1cyclictest3056-21PK-Backend09:58:241
52831599190,1cyclictest3056-21PK-Backend09:53:441
52831599190,1cyclictest3056-21PK-Backend09:36:341
52831599190,1cyclictest0-21swapper/110:55:151
52831599190,1cyclictest0-21swapper/108:38:201
52831599190,18cyclictest0-21swapper/107:56:371
528312991917,1cyclictest0-21swapper/011:34:210
528312991917,1cyclictest0-21swapper/011:13:460
528312991917,1cyclictest0-21swapper/010:57:450
528312991917,1cyclictest0-21swapper/008:53:470
528312991917,1cyclictest0-21swapper/008:22:370
528312991917,1cyclictest0-21swapper/007:32:420
528312991917,1cyclictest0-21swapper/007:32:420
528312991917,1cyclictest0-21swapper/007:29:240
528312991915,2cyclictest622423-21tr11:35:120
528312991912,5cyclictest531548-21cut07:15:190
52831299190,1cyclictest0-21swapper/012:32:010
52831299190,1cyclictest0-21swapper/011:50:230
52831299190,1cyclictest0-21swapper/009:38:350
52831299190,1cyclictest0-21swapper/007:58:240
52831299190,1cyclictest0-21swapper/007:23:020
52831299190,18cyclictest0-21swapper/012:06:590
52831299190,18cyclictest0-21swapper/010:43:580
52831299190,18cyclictest0-21swapper/009:19:420
528322991816,1cyclictest3056-21PK-Backend09:03:423
528322991815,2cyclictest3056-21PK-Backend09:06:193
528322991813,4cyclictest42-21ksoftirqd/311:35:223
528322991810,6cyclictest597118-21python310:20:213
52832299180,1cyclictest0-21swapper/309:54:513
52832299180,16cyclictest635520-21python312:10:213
528320991816,1cyclictest0-21swapper/209:04:052
528320991816,1cyclictest0-21swapper/208:28:092
528320991815,2cyclictest0-21swapper/212:15:142
528320991814,3cyclictest584144-21cat09:45:152
528320991813,3cyclictest1095-21runrttasks10:35:112
528320991812,5cyclictest643454-21seq12:33:282
528315991816,1cyclictest3056-21PK-Backend10:48:201
528315991816,1cyclictest3056-21PK-Backend08:17:451
52831599180,17cyclictest3056-21PK-Backend12:39:461
528312991816,1cyclictest0-21swapper/012:26:260
528312991816,1cyclictest0-21swapper/008:39:270
528312991813,3cyclictest5438842sleep007:50:190
528312991812,4cyclictest610296-21cut11:00:120
528312991812,4cyclictest537915-21date07:35:010
528312991811,5cyclictest569908-21expr09:05:130
52831299180,1cyclictest0-21swapper/011:29:140
52831299180,1cyclictest0-21swapper/011:29:140
52831299180,17cyclictest0-21swapper/009:50:120
528322991711,5cyclictest624880-21awk11:40:223
528322991711,4cyclictest2156-21gnome-shell11:25:103
528322991711,4cyclictest2156-21gnome-shell11:25:093
528322991710,5cyclictest42-21ksoftirqd/309:20:143
52832299170,3cyclictest608750-21df10:55:133
52832299170,15cyclictest5298262sleep307:13:093
528320991715,1cyclictest0-21swapper/209:26:562
528320991713,3cyclictest34-21ksoftirqd/208:50:012
528320991713,2cyclictest34-21ksoftirqd/210:20:182
528320991712,3cyclictest582737-21swap09:40:232
52831599179,5cyclictest922-21systemd-logind07:15:011
52831599177,7cyclictest611040-21/usr/sbin/munin11:00:211
528315991715,1cyclictest3056-21PK-Backend08:23:481
528315991715,1cyclictest3056-21PK-Backend07:28:511
528315991715,1cyclictest0-21swapper/110:41:231
528315991712,3cyclictest541623-21tr07:45:111
52831299179,6cyclictest612730-21python311:05:200
52831299178,7cyclictest566001-21apt08:55:120
528312991715,1cyclictest0-21swapper/011:58:480
528312991715,1cyclictest0-21swapper/010:13:020
528312991715,1cyclictest0-21swapper/008:29:060
528312991715,1cyclictest0-21swapper/008:12:440
528312991714,2cyclictest0-21swapper/011:44:010
528312991712,3cyclictest627209-21perf11:50:010
528312991712,3cyclictest569449-21latency_hist09:05:020
528312991711,4cyclictest599634-21sed10:30:010
528322991614,1cyclictest3056-21PK-Backend09:29:443
52832099168,2cyclictest34-21ksoftirqd/208:50:132
528320991610,5cyclictest634746-21ls12:10:122
52831299169,5cyclictest562532-21apt08:45:100
528312991614,1cyclictest0-21swapper/012:38:290
528312991613,2cyclictest3056-21PK-Backend10:00:590
528312991613,2cyclictest3056-21PK-Backend09:40:380
528312991610,4cyclictest605726-21cut10:45:200
52832299155,9cyclictest3056-21PK-Backend07:41:363
528322991510,3cyclictest615387-21cpuspeed11:15:123
52832099158,5cyclictest630391-21sed11:55:232
528315991513,1cyclictest3056-21PK-Backend12:07:291
528322991410,3cyclictest3056-21PK-Backend08:54:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional