You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 15:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sat Jun 27, 2026 12:44:46)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
257392327838,33sleep10-21swapper/107:07:061
257267227740,14sleep30-21swapper/307:05:073
257412426443,14sleep00-21swapper/007:09:410
257382226141,14sleep20-21swapper/207:05:462
26084182350,2sleep00-21swapper/008:45:010
26178552340,5sleep10-21swapper/109:10:151
257431799320,21cyclictest2660074-21ntp_states11:10:200
2574317992913,6cyclictest2613481-21apt-get08:59:590
257431799287,6cyclictest2628960-21awk09:40:220
257431799286,8cyclictest0-21swapper/010:50:130
2574317992812,4cyclictest2643307-21apt-key10:25:010
2574327992714,11cyclictest2588665-21unin-run07:50:003
2574327992714,11cyclictest2588665-21unin-run07:50:003
257431799278,5cyclictest2599496-21gdbus08:20:020
2574317992713,4cyclictest2583386-21idleruntime-cro07:35:010
257432799260,7cyclictest2689710-21diskstats12:35:153
2574319992615,8cyclictest2631117-21systemd09:50:011
257431799269,4cyclictest2609276-21grep08:45:190
257431799268,6cyclictest883-21dbus-daemon08:15:010
257431799268,5cyclictest2668432-21if_enp2s011:35:170
257431799266,6cyclictest2620242-21sed09:15:210
2574317992613,4cyclictest2606581-21systemd-run08:40:000
2574317992611,3cyclictest2614620-21missed_timers09:00:170
2574317992610,3cyclictest1095-21runrttasks11:04:070
257431799258,4cyclictest2657801-21cstates11:05:140
257431799255,6cyclictest2644193-21cat10:25:180
2574317992511,2cyclictest2674671-21latency_hist11:55:010
2574317992511,2cyclictest2674671-21latency_hist11:55:010
257431799251,10cyclictest2623429-21ntp_states09:25:210
139099250,4rtkit-daemon2633875-21ps09:55:201
2574317992411,2cyclictest2634793-21/usr/sbin/munin10:00:170
257431799241,10cyclictest2632098-21ntp_states09:50:200
2574317992410,3cyclictest3056-21PK-Backend12:28:400
2574317992410,2cyclictest2678682-21gpgconf12:05:000
26176912230,6sleep00-21swapper/009:10:150
257432799238,13cyclictest2662691-21latency_hist11:20:003
2574322992317,4cyclictest2579898-21cron07:25:002
2574319992315,5cyclictest2643708-21apt-get10:25:041
2574317992310,3cyclictest1095-21runrttasks07:28:400
139099230,16rtkit-daemon3056-21PK-Backend12:30:351
2574327992220,1cyclictest3056-21PK-Backend07:28:003
2574322992220,1cyclictest0-21swapper/211:07:542
2574322992220,1cyclictest0-21swapper/209:36:022
257432299220,21cyclictest1788598-21PressureMonitor10:29:212
257432299220,21cyclictest0-21swapper/210:55:572
257432299220,20cyclictest0-21swapper/212:26:082
257431999220,3cyclictest2616119-21diskstats09:05:151
257431999220,21cyclictest0-21swapper/111:09:581
257431999220,20cyclictest2620263-21cat09:15:221
257431799229,4cyclictest2683878-21awk12:20:010
257431799220,3cyclictest2627624-21turbostat09:40:010
2574327992119,1cyclictest3056-21PK-Backend12:09:053
2574327992119,1cyclictest3056-21PK-Backend11:50:143
2574327992119,1cyclictest3056-21PK-Backend11:50:133
2574327992119,1cyclictest3056-21PK-Backend11:42:153
2574327992119,1cyclictest3056-21PK-Backend08:59:593
2574327992119,1cyclictest3056-21PK-Backend07:31:253
2574327992118,2cyclictest3056-21PK-Backend11:10:193
2574327992118,2cyclictest3056-21PK-Backend08:07:333
2574327992116,3cyclictest2657340-21munin-plugin-st11:05:003
257432799211,18cyclictest2639019-21ntp_states10:10:193
257432799210,19cyclictest3056-21PK-Backend10:36:393
257432799210,19cyclictest3056-21PK-Backend10:07:393
257432799210,19cyclictest3056-21PK-Backend09:50:013
257432799210,19cyclictest3056-21PK-Backend09:46:223
257432799210,19cyclictest3056-21PK-Backend09:06:013
257432799210,19cyclictest3056-21PK-Backend08:38:293
257432299213,8cyclictest141rcu_preempt10:30:342
2574322992119,1cyclictest0-21swapper/212:05:002
2574322992119,1cyclictest0-21swapper/211:44:182
2574322992119,1cyclictest0-21swapper/211:17:482
2574322992119,1cyclictest0-21swapper/210:07:522
2574322992119,1cyclictest0-21swapper/209:41:092
2574322992119,1cyclictest0-21swapper/208:30:182
2574322992119,1cyclictest0-21swapper/208:13:002
2574322992119,1cyclictest0-21swapper/207:56:112
2574322992119,1cyclictest0-21swapper/207:49:352
2574322992119,1cyclictest0-21swapper/207:49:342
2574322992119,1cyclictest0-21swapper/207:43:542
2574322992119,1cyclictest0-21swapper/207:34:552
2574322992119,1cyclictest0-21swapper/207:12:112
2574322992118,2cyclictest3056-21PK-Backend09:30:202
257432299210,20cyclictest0-21swapper/211:10:322
257432299210,20cyclictest0-21swapper/209:54:002
257432299210,20cyclictest0-21swapper/209:09:382
257432299210,20cyclictest0-21swapper/208:39:242
257432299210,20cyclictest0-21swapper/208:29:442
257432299210,20cyclictest0-21swapper/207:26:102
257432299210,19cyclictest3056-21PK-Backend08:51:542
257432299210,19cyclictest3056-21PK-Backend08:51:542
2574319992114,5cyclictest1095-21runrttasks08:23:241
2574319992112,6cyclictest436-20systemd-journal08:50:001
257431999210,20cyclictest3056-21PK-Backend07:23:211
257431999210,19cyclictest0-21swapper/111:15:131
2574317992119,1cyclictest0-21swapper/012:08:260
2574317992119,1cyclictest0-21swapper/011:57:110
2574317992119,1cyclictest0-21swapper/011:40:340
2574317992119,1cyclictest0-21swapper/010:15:510
2574317992119,1cyclictest0-21swapper/007:38:550
139099210,14rtkit-daemon0-21swapper/110:43:571
139099210,14rtkit-daemon0-21swapper/109:01:441
139099210,14rtkit-daemon0-21swapper/008:30:450
2574327992018,1cyclictest3056-21PK-Backend12:00:143
2574327992018,1cyclictest3056-21PK-Backend09:32:043
2574327992018,1cyclictest3056-21PK-Backend08:54:393
2574327992018,1cyclictest3056-21PK-Backend08:54:383
2574327992018,1cyclictest3056-21PK-Backend08:30:453
2574327992018,1cyclictest0-21swapper/309:56:373
2574327992018,1cyclictest0-21swapper/307:16:513
2574327992013,6cyclictest2688635-21sed12:30:233
257432799201,17cyclictest82332-21update-manager09:20:303
2574327992011,7cyclictest2591006-21df_inode07:55:143
257432799200,1cyclictest3056-21PK-Backend10:32:133
257432799200,1cyclictest3056-21PK-Backend08:44:253
257432799200,19cyclictest3056-21PK-Backend10:29:163
257432799200,19cyclictest3056-21PK-Backend09:15:213
257432799200,19cyclictest3056-21PK-Backend08:14:443
257432799200,19cyclictest0-21swapper/311:35:133
257432799200,19cyclictest0-21swapper/310:16:303
257432799200,19cyclictest0-21swapper/307:52:063
257432799200,19cyclictest0-21swapper/307:41:173
2574322992018,1cyclictest3056-21PK-Backend11:30:182
2574322992018,1cyclictest3056-21PK-Backend09:22:232
2574322992018,1cyclictest3056-21PK-Backend08:58:372
2574322992018,1cyclictest0-21swapper/211:46:552
2574322992018,1cyclictest0-21swapper/211:27:412
2574322992018,1cyclictest0-21swapper/211:02:402
2574322992018,1cyclictest0-21swapper/209:17:572
2574322992018,1cyclictest0-21swapper/208:42:002
2574322992018,1cyclictest0-21swapper/207:39:422
2574322992012,6cyclictest2689389-21awk12:35:142
257432299200,1cyclictest0-21swapper/208:46:402
257432299200,19cyclictest0-21swapper/211:20:252
257432299200,19cyclictest0-21swapper/210:37:332
257432299200,19cyclictest0-21swapper/210:20:502
257432299200,19cyclictest0-21swapper/209:14:592
257432299200,18cyclictest3056-21PK-Backend10:51:532
2574319992018,1cyclictest0-21swapper/109:53:141
2574319992018,1cyclictest0-21swapper/108:25:331
2574319992012,6cyclictest2668299-21expr11:35:131
257431999201,14cyclictest2648300-21sshd10:35:231
2574319992010,7cyclictest2646940-21perf10:35:011
2574319992010,7cyclictest1-21systemd10:25:011
257431999200,19cyclictest3056-21PK-Backend07:56:531
257431999200,19cyclictest3056-21PK-Backend07:25:031
257431999200,19cyclictest0-21swapper/112:06:031
257431999200,19cyclictest0-21swapper/108:54:331
257431999200,19cyclictest0-21swapper/108:54:321
257431999200,19cyclictest0-21swapper/108:35:341
2574317992018,1cyclictest0-21swapper/007:46:060
2574317992018,1cyclictest0-21swapper/007:46:060
257431799201,5cyclictest2624425-21apt09:30:110
257431799201,17cyclictest2631083-21perf09:49:590
257431799200,1cyclictest0-21swapper/008:00:110
257431799200,19cyclictest0-21swapper/012:11:240
257431799200,19cyclictest0-21swapper/010:46:200
257431799200,19cyclictest0-21swapper/009:58:080
139099200,3rtkit-daemon0-21swapper/012:33:380
139099200,14rtkit-daemon0-21swapper/010:58:440
26418212190,4sleep33056-21PK-Backend10:20:103
257432799194,13cyclictest2579451-21timerandwakeup07:20:223
2574327991917,1cyclictest3056-21PK-Backend11:56:433
2574327991917,1cyclictest3056-21PK-Backend11:31:413
2574327991917,1cyclictest3056-21PK-Backend10:02:533
2574327991916,2cyclictest3056-21PK-Backend10:59:443
2574327991912,5cyclictest2623624-21cat09:25:223
2574327991912,5cyclictest2593307-21seq08:00:213
257432799190,1cyclictest3056-21PK-Backend12:10:343
257432799190,1cyclictest3056-21PK-Backend10:51:253
257432799190,1cyclictest3056-21PK-Backend09:44:343
257432799190,1cyclictest3056-21PK-Backend08:16:263
257432799190,18cyclictest3056-21PK-Backend12:16:163
257432799190,18cyclictest3056-21PK-Backend10:42:343
257432799190,18cyclictest0-21swapper/311:25:333
2574322991917,1cyclictest0-21swapper/212:11:412
2574322991917,1cyclictest0-21swapper/212:05:342
2574322991917,1cyclictest0-21swapper/211:56:142
2574322991917,1cyclictest0-21swapper/211:39:382
2574322991917,1cyclictest0-21swapper/210:49:502
2574322991917,1cyclictest0-21swapper/210:43:412
2574322991917,1cyclictest0-21swapper/209:00:052
2574322991917,1cyclictest0-21swapper/208:23:152
2574322991917,1cyclictest0-21swapper/208:07:542
2574322991913,4cyclictest436-20systemd-journal08:04:552
257432299190,1cyclictest3056-21PK-Backend09:58:252
257432299190,1cyclictest0-21swapper/212:21:082
257432299190,1cyclictest0-21swapper/210:15:502
257432299190,1cyclictest0-21swapper/209:45:212
257432299190,1cyclictest0-21swapper/207:50:092
257432299190,1cyclictest0-21swapper/207:18:122
2574319991917,1cyclictest0-21swapper/112:35:301
2574319991917,1cyclictest0-21swapper/112:27:321
2574319991917,1cyclictest0-21swapper/111:10:331
2574319991917,1cyclictest0-21swapper/109:39:271
2574319991917,1cyclictest0-21swapper/108:33:311
2574319991916,2cyclictest3056-21PK-Backend07:52:341
257431999190,1cyclictest3056-21PK-Backend07:48:161
257431999190,1cyclictest3056-21PK-Backend07:48:161
257431999190,1cyclictest3056-21PK-Backend07:44:511
257431999190,1cyclictest0-21swapper/112:22:321
257431999190,1cyclictest0-21swapper/111:01:141
257431999190,1cyclictest0-21swapper/109:43:401
257431999190,1cyclictest0-21swapper/109:29:351
257431999190,1cyclictest0-21swapper/108:16:551
257431999190,1cyclictest0-21swapper/108:14:041
257431999190,18cyclictest0-21swapper/111:57:511
257431999190,18cyclictest0-21swapper/111:42:581
257431999190,18cyclictest0-21swapper/111:26:011
2574317991917,1cyclictest0-21swapper/008:29:590
2574317991917,1cyclictest0-21swapper/008:09:320
2574317991911,6cyclictest2649607-21grep10:40:170
2574317991911,6cyclictest2575230-21ntpq07:10:200
257431799190,18cyclictest0-21swapper/009:09:550
257431799190,18cyclictest0-21swapper/007:54:040
139099190,13rtkit-daemon3056-21PK-Backend11:24:321
139099190,13rtkit-daemon0-21swapper/010:09:490
25763582180,1sleep00-21swapper/007:15:020
2574327991816,1cyclictest3056-21PK-Backend09:10:143
2574327991816,1cyclictest3056-21PK-Backend07:13:053
2574327991815,2cyclictest3056-21PK-Backend10:46:043
2574327991815,2cyclictest3056-21PK-Backend08:26:543
2574327991810,6cyclictest2663446-21cat11:20:193
2574322991816,1cyclictest0-21swapper/211:53:362
2574322991816,1cyclictest0-21swapper/211:53:362
2574322991815,2cyclictest0-21swapper/210:00:282
257431999188,8cyclictest2621411-21cat09:20:181
2574319991816,1cyclictest3056-21PK-Backend10:10:581
2574319991816,1cyclictest0-21swapper/112:15:091
2574319991816,1cyclictest0-21swapper/111:51:431
2574319991816,1cyclictest0-21swapper/111:51:421
2574319991816,1cyclictest0-21swapper/110:55:401
2574319991816,1cyclictest0-21swapper/110:05:031
2574319991816,1cyclictest0-21swapper/110:00:371
2574319991815,2cyclictest3056-21PK-Backend10:15:241
2574319991811,5cyclictest2625172-21ntpq09:30:181
257431999181,13cyclictest1095-21runrttasks08:55:111
2574319991810,6cyclictest2677670-21ntpq12:00:191
2574319991810,6cyclictest2576269-21python307:15:021
2574319991810,5cyclictest2650809-21grep10:45:131
257431999180,1cyclictest0-21swapper/111:47:041
2574317991816,1cyclictest0-21swapper/011:47:240
2574317991813,3cyclictest2665569-21timerwakeupswit11:25:240
2574317991812,5cyclictest2667327-21sed11:30:240
2574317991810,6cyclictest2597932-21df08:15:140
257431799180,1cyclictest3056-21PK-Backend11:15:200
25818952170,3sleep10-21swapper/107:30:101
2574327991715,1cyclictest3056-21PK-Backend12:21:373
2574327991715,1cyclictest0-21swapper/311:47:153
2574327991714,2cyclictest3056-21PK-Backend09:38:463
2574327991711,4cyclictest2585251-21cat07:40:003
2574322991712,3cyclictest2683858-21munin-plugin-st12:20:002
2574322991712,3cyclictest2638891-21missed_timers10:10:172
2574322991711,5cyclictest2598008-21awk08:15:152
2574322991711,4cyclictest2688638-21sshd12:30:242
257431999179,6cyclictest2652769-21df10:50:151
2574319991715,1cyclictest3056-21PK-Backend07:38:561
2574317991715,1cyclictest0-21swapper/007:40:040
2574317991711,5cyclictest2648674-21cron10:40:010
2574317991710,6cyclictest2685027-21sed12:20:210
2574317991710,5cyclictest436-20systemd-journal08:50:250
2574317991710,5cyclictest436-20systemd-journal08:50:240
2574317991710,5cyclictest2638443-21tr10:10:110
2574317991710,5cyclictest2621650-21ntp_kernel_err09:20:190
139099170,3rtkit-daemon0-21swapper/007:55:170
2574327991614,1cyclictest3056-21PK-Backend12:26:573
2574327991614,1cyclictest3056-21PK-Backend08:47:023
2574327991612,2cyclictest2614875-21python309:00:213
257432799160,2cyclictest141rcu_preempt08:23:093
257431999169,5cyclictest2680921-21cat12:10:141
257431999168,6cyclictest2667171-21cat11:30:231
2574319991614,1cyclictest3056-21PK-Backend08:03:301
2574319991614,1cyclictest0-21swapper/107:11:181
2574319991612,2cyclictest2607599-21proc_pri08:40:211
257431799168,6cyclictest2579314-21python307:20:210
257431799158,5cyclictest2646439-21sed10:30:220
257431799150,1cyclictest0-21swapper/012:35:020
257431799150,14cyclictest0-21swapper/011:21:350
139099150,3rtkit-daemon0-21swapper/108:06:531
2574327991411,2cyclictest3056-21PK-Backend11:08:503
257432299149,4cyclictest2623307-21grep09:25:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional