You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-30 - 21:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Mon Mar 30, 2026 12:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
249972228059,15sleep00-21swapper/007:05:130
250102027339,12sleep10-21swapper/107:08:111
250082626940,22sleep30-21swapper/307:05:393
250085526545,14sleep20-21swapper/207:06:032
25230072520,4sleep00-21swapper/008:10:130
25161532490,4sleep13056-21PK-Backend07:50:121
25200812430,2sleep20-21swapper/208:00:172
2501313993910,15cyclictest2548078-21/usr/sbin/munin09:20:152
25854262370,4sleep20-21swapper/211:05:212
250131399348,14cyclictest0-21swapper/209:44:052
250131399345,14cyclictest0-21swapper/209:09:592
25263692320,4sleep13056-21PK-Backend08:20:011
250131399329,14cyclictest0-21swapper/210:25:522
250131399329,14cyclictest0-21swapper/209:30:512
250131399328,14cyclictest0-21swapper/208:33:132
250131399328,14cyclictest0-21swapper/208:22:462
250131399327,14cyclictest0-21swapper/207:47:202
250131399327,12cyclictest2533581-21turbostat08:44:542
250131399319,9cyclictest0-21swapper/210:34:542
250131399319,10cyclictest0-21swapper/207:38:072
250131399318,13cyclictest0-21swapper/211:31:562
250131399317,13cyclictest0-21swapper/209:28:502
250131399308,12cyclictest0-21swapper/210:48:142
250131399308,10cyclictest0-21swapper/210:36:072
250131399307,6cyclictest0-21swapper/207:31:092
250131399305,15cyclictest2501506-21apt07:10:102
250131399298,9cyclictest0-21swapper/212:00:072
250131399298,9cyclictest0-21swapper/208:12:042
250131399298,10cyclictest0-21swapper/212:22:442
250131399297,10cyclictest0-21swapper/212:18:362
250131399297,10cyclictest0-21swapper/212:18:362
250131999284,14cyclictest0-21swapper/311:06:593
2501313992817,9cyclictest2545705-21perf09:15:002
250131999273,14cyclictest0-21swapper/312:13:153
250131999273,14cyclictest0-21swapper/309:37:003
250131999273,13cyclictest0-21swapper/311:49:563
250131999272,16cyclictest0-21swapper/311:39:513
250131399279,6cyclictest2506799-21chrt07:23:362
250131399279,5cyclictest0-21swapper/210:00:542
250131399277,8cyclictest0-21swapper/207:40:182
250131999267,17cyclictest2569552-21wc10:20:193
250131999263,13cyclictest0-21swapper/309:44:583
250131399269,5cyclictest0-21swapper/211:16:212
250131399269,5cyclictest0-21swapper/208:58:232
250131399269,5cyclictest0-21swapper/208:09:272
250131399268,6cyclictest0-21swapper/209:17:082
250131399267,7cyclictest0-21swapper/211:47:202
250131399265,7cyclictest2528099-21apt-get08:25:022
250131399262,13cyclictest0-21swapper/211:57:182
250131999252,15cyclictest0-21swapper/310:33:503
250131999252,13cyclictest0-21swapper/307:11:373
2501319992517,6cyclictest883-21dbus-daemon12:19:593
2501319992517,6cyclictest883-21dbus-daemon12:19:593
250131399259,5cyclictest0-21swapper/211:12:402
250131399258,6cyclictest0-21swapper/210:12:172
250131399258,5cyclictest0-21swapper/212:06:572
250131399258,5cyclictest0-21swapper/209:49:482
250131399258,5cyclictest0-21swapper/208:36:382
250131399257,6cyclictest0-21swapper/209:39:122
250131399252,13cyclictest0-21swapper/209:52:322
250131399252,12cyclictest0-21swapper/208:47:482
2501313992518,4cyclictest2613814-21cron12:29:592
2501313992516,5cyclictest2589747-21cpuspeed_turbos11:20:122
2501313992516,5cyclictest2589747-21cpuspeed_turbos11:20:122
2501313992515,7cyclictest2581020-21cpuspeed_turbos10:55:132
250131999249,14cyclictest2601437-21perf11:55:003
250131999243,9cyclictest0-21swapper/307:19:193
250131999242,14cyclictest0-21swapper/310:58:033
250131999242,10cyclictest0-21swapper/311:26:353
250131999241,9cyclictest0-21swapper/311:30:563
2501319992414,5cyclictest2605385-21apt-get12:05:043
2501319992413,9cyclictest2612119-21taskset12:25:013
2501319992410,10cyclictest0-21swapper/307:40:123
250131399248,5cyclictest0-21swapper/211:28:592
250131399248,5cyclictest0-21swapper/210:23:202
250131399247,5cyclictest0-21swapper/212:37:132
2501313992418,4cyclictest1095-21runrttasks07:55:092
2501313992417,5cyclictest2597945-21cron11:45:002
2501313992416,4cyclictest2582769-21cpuspeed_turbos11:00:112
2501313992415,6cyclictest2567968-21find10:15:232
2501313992414,8cyclictest2564335-21/usr/sbin/munin10:05:192
250131399241,13cyclictest0-21swapper/210:51:272
2501319992312,7cyclictest2545709-21awk09:15:003
2501319992312,7cyclictest2526113-21apt-get08:20:033
2501313992315,5cyclictest2540890-21cpuspeed_turbos09:00:122
2501313992314,7cyclictest2517448-21perf07:55:002
2501313992314,6cyclictest2524821-21grep08:15:112
2501313992313,6cyclictest2576635-21find10:40:212
2501309992320,2cyclictest3056-21PK-Backend12:23:301
2501309992315,6cyclictest883-21dbus-daemon07:46:341
2501307992315,6cyclictest2562186-21df10:00:120
250131999222,8cyclictest0-21swapper/307:33:573
2501319992218,3cyclictest25211542sleep308:05:103
2501319992214,6cyclictest883-21dbus-daemon10:55:003
2501319992214,6cyclictest2588203-21diskstats11:15:163
2501319992213,5cyclictest2528601-21apt-get08:25:033
250131999220,20cyclictest2567417-21/usr/sbin/munin10:15:163
2501313992216,3cyclictest2561337-21munin-run10:00:012
2501313992215,5cyclictest2503209-21nin-node07:15:022
2501313992214,6cyclictest1095-21runrttasks12:10:222
2501313992214,5cyclictest2615554-21cron12:35:002
2501313992213,5cyclictest2594739-21/usr/sbin/munin11:35:222
2501309992220,1cyclictest3056-21PK-Backend10:07:431
2501309992220,1cyclictest0-21swapper/112:30:251
2501309992219,2cyclictest3056-21PK-Backend11:55:381
2501309992219,2cyclictest3056-21PK-Backend11:22:191
2501309992219,2cyclictest3056-21PK-Backend11:22:191
2501309992219,2cyclictest3056-21PK-Backend09:22:411
2501307992220,1cyclictest0-21swapper/008:03:470
2501307992220,1cyclictest0-21swapper/007:10:080
2501307992219,2cyclictest3056-21PK-Backend11:04:030
250130799220,21cyclictest0-21swapper/012:11:490
250131999218,11cyclictest2547438-21/usr/sbin/munin09:20:003
250131999216,3cyclictest2510944-21expr07:35:133
250131999214,4cyclictest2551440-21expr09:30:113
2501319992119,1cyclictest0-21swapper/308:58:043
2501319992114,5cyclictest2537036-21cron08:50:003
2501319992113,6cyclictest2613818-21cron12:30:003
2501319992111,7cyclictest2524388-21perf08:15:003
250131999210,20cyclictest0-21swapper/307:53:143
2501313992114,5cyclictest2538786-21awk08:55:012
2501313992113,4cyclictest2507168-21/usr/sbin/munin07:25:142
2501313992112,6cyclictest2600908-21threads11:50:252
2501309992119,1cyclictest3056-21PK-Backend10:36:011
2501309992119,1cyclictest3056-21PK-Backend10:23:251
2501309992119,1cyclictest3056-21PK-Backend10:10:191
2501309992119,1cyclictest3056-21PK-Backend09:27:141
2501309992119,1cyclictest3056-21PK-Backend07:23:501
2501309992118,2cyclictest3056-21PK-Backend12:05:041
2501309992118,2cyclictest3056-21PK-Backend11:36:241
2501309992118,2cyclictest3056-21PK-Backend09:52:151
2501309992118,2cyclictest3056-21PK-Backend09:45:251
250130999210,20cyclictest3056-21PK-Backend08:48:351
250130999210,1cyclictest3056-21PK-Backend10:27:171
250130999210,1cyclictest3056-21PK-Backend07:56:431
250130999210,19cyclictest3056-21PK-Backend12:16:201
250130999210,19cyclictest3056-21PK-Backend12:16:201
250130999210,19cyclictest3056-21PK-Backend12:13:151
250130999210,19cyclictest3056-21PK-Backend11:18:141
250130999210,19cyclictest3056-21PK-Backend10:41:421
250130999210,19cyclictest3056-21PK-Backend10:01:421
250130999210,19cyclictest3056-21PK-Backend08:50:091
250130999210,19cyclictest3056-21PK-Backend08:43:141
250130999210,19cyclictest3056-21PK-Backend08:35:441
250130999210,19cyclictest3056-21PK-Backend08:02:581
2501307992119,1cyclictest0-21swapper/010:12:450
2501307992119,1cyclictest0-21swapper/009:56:420
2501307992119,1cyclictest0-21swapper/009:54:050
2501307992119,1cyclictest0-21swapper/009:23:230
2501307992119,1cyclictest0-21swapper/009:05:260
2501307992119,1cyclictest0-21swapper/008:55:180
2501307992119,1cyclictest0-21swapper/008:39:170
2501307992119,1cyclictest0-21swapper/008:33:070
2501307992119,1cyclictest0-21swapper/008:29:440
2501307992119,1cyclictest0-21swapper/008:06:580
2501307992119,1cyclictest0-21swapper/007:42:050
2501307992119,1cyclictest0-21swapper/007:28:180
250130799211,8cyclictest2597971-21perf11:44:590
250130799210,20cyclictest0-21swapper/011:59:320
250130799210,20cyclictest0-21swapper/011:32:490
250130799210,20cyclictest0-21swapper/011:27:550
250130799210,20cyclictest0-21swapper/011:12:000
250130799210,20cyclictest0-21swapper/010:55:040
250130799210,20cyclictest0-21swapper/010:16:100
250130799210,20cyclictest0-21swapper/007:45:570
250130799210,20cyclictest0-21swapper/007:17:180
250131999208,8cyclictest2515106-21/usr/sbin/munin07:45:223
2501319992015,3cyclictest2562224-21hddtemp_smartct10:00:133
2501319992013,6cyclictest2538180-21awk08:50:223
2501319992013,5cyclictest2564263-21ntp_kernel_err10:05:193
2501319992013,5cyclictest2558478-21diskstats09:50:143
2501319992012,6cyclictest2540973-21cstates09:00:113
2501319992012,4cyclictest2604294-21python312:00:213
2501319992010,8cyclictest2603177-21perf12:00:013
2501319992010,6cyclictest2557381-21timerwakeupswit09:45:243
250131999200,19cyclictest42-21ksoftirqd/310:25:233
250131999200,19cyclictest0-21swapper/311:43:173
250131999200,19cyclictest0-21swapper/309:56:573
250131999200,19cyclictest0-21swapper/309:22:303
250131999200,19cyclictest0-21swapper/308:32:353
250131999200,19cyclictest0-21swapper/307:20:363
2501309992018,1cyclictest3056-21PK-Backend12:36:201
2501309992018,1cyclictest3056-21PK-Backend12:29:251
2501309992018,1cyclictest3056-21PK-Backend08:32:191
2501309992018,1cyclictest3056-21PK-Backend07:19:391
2501309992017,2cyclictest3056-21PK-Backend07:25:411
250130999200,18cyclictest3056-21PK-Backend11:42:201
250130999200,18cyclictest3056-21PK-Backend10:31:091
250130999200,18cyclictest3056-21PK-Backend09:35:261
250130999200,18cyclictest3056-21PK-Backend07:30:541
2501307992018,1cyclictest0-21swapper/011:23:020
2501307992018,1cyclictest0-21swapper/011:23:010
2501307992018,1cyclictest0-21swapper/010:22:110
2501307992018,1cyclictest0-21swapper/009:01:260
2501307992013,5cyclictest436-20systemd-journal11:10:010
2501307992012,6cyclictest436-20systemd-journal07:40:000
250130799200,1cyclictest0-21swapper/011:54:240
250130799200,19cyclictest0-21swapper/012:08:450
250130799200,19cyclictest0-21swapper/010:49:210
250130799200,18cyclictest0-21swapper/010:41:040
139099200,3rtkit-daemon1389-21rtkit-daemon12:16:230
139099200,3rtkit-daemon1389-21rtkit-daemon12:16:230
139099200,18rtkit-daemon1997-21gvfs-afc-volume09:17:010
250131999199,7cyclictest2524824-21cut08:15:123
250131999192,15cyclictest0-21swapper/307:55:143
2501319991917,1cyclictest0-21swapper/307:28:273
2501319991912,5cyclictest2577817-21awk10:45:143
2501319991911,7cyclictest2616494-21cut12:35:183
2501319991911,6cyclictest2589744-21grep11:20:113
2501319991911,6cyclictest2589744-21grep11:20:113
2501319991911,6cyclictest2549800-21df_inode09:25:133
2501319991910,7cyclictest2566139-21/usr/sbin/munin10:10:213
2501319991910,7cyclictest2533048-21awk08:35:203
250131999190,7cyclictest2582630-21apt11:00:113
250131999190,1cyclictest0-21swapper/309:07:033
250131999190,1cyclictest0-21swapper/308:42:083
250131999190,18cyclictest0-21swapper/310:37:133
250131999190,16cyclictest2586095-21apt11:10:123
2501309991917,1cyclictest0-21swapper/108:24:411
2501309991916,2cyclictest3056-21PK-Backend10:55:561
2501309991910,7cyclictest2552026-21python309:30:191
250130999190,1cyclictest3056-21PK-Backend11:13:051
250130999190,1cyclictest3056-21PK-Backend10:50:151
250130999190,1cyclictest3056-21PK-Backend09:55:401
250130999190,1cyclictest3056-21PK-Backend09:41:141
250130999190,1cyclictest3056-21PK-Backend09:15:101
2501307991917,1cyclictest3056-21PK-Backend11:38:440
2501307991917,1cyclictest3056-21PK-Backend09:30:200
2501307991917,1cyclictest0-21swapper/012:36:290
2501307991917,1cyclictest0-21swapper/012:32:380
2501307991917,1cyclictest0-21swapper/012:22:510
2501307991917,1cyclictest0-21swapper/007:21:300
2501307991914,3cyclictest2533976-21tr08:40:120
2501307991912,5cyclictest2527643-21sed08:20:230
250130799190,1cyclictest0-21swapper/010:25:020
250130799190,1cyclictest0-21swapper/008:16:320
250130799190,18cyclictest0-21swapper/010:54:010
250130799190,18cyclictest0-21swapper/010:32:330
250130799190,18cyclictest0-21swapper/008:52:560
139099190,15rtkit-daemon2501302-21cyclictest12:26:160
250131999180,1cyclictest0-21swapper/310:42:073
250130999187,2cyclictest141rcu_preempt09:00:011
2501309991816,1cyclictest3056-21PK-Backend11:46:261
2501309991816,1cyclictest3056-21PK-Backend11:26:251
2501309991816,1cyclictest3056-21PK-Backend08:05:221
2501309991816,1cyclictest3056-21PK-Backend07:42:031
2501309991815,2cyclictest3056-21PK-Backend10:16:211
2501307991816,1cyclictest0-21swapper/010:36:250
2501307991816,1cyclictest0-21swapper/010:09:210
2501307991816,1cyclictest0-21swapper/008:47:350
2501307991816,1cyclictest0-21swapper/007:56:450
2501307991816,1cyclictest0-21swapper/007:54:420
2501307991816,1cyclictest0-21swapper/007:34:350
2501307991813,3cyclictest2544382-21cpuspeed_turbos09:10:130
2501307991811,5cyclictest2556560-21cut09:45:140
250131999179,6cyclictest2614147-21runrttasks12:30:133
250131999179,6cyclictest2519447-21apt08:00:113
250130999179,6cyclictest2542233-21cron09:05:011
2501309991715,1cyclictest3056-21PK-Backend10:45:351
2501309991715,1cyclictest3056-21PK-Backend08:28:531
2501307991715,1cyclictest0-21swapper/011:45:390
2501307991715,1cyclictest0-21swapper/011:15:030
2501307991711,4cyclictest2554604-21chrt09:40:070
2501309991614,1cyclictest3056-21PK-Backend11:05:091
2501309991611,4cyclictest3056-21PK-Backend11:54:531
250130799168,7cyclictest2550101-21grep09:25:170
2501307991610,5cyclictest2553883-21cat09:35:210
139099160,13rtkit-daemon0-21swapper/111:01:381
2501309991513,1cyclictest3056-21PK-Backend11:30:291
2501309991513,1cyclictest3056-21PK-Backend07:10:061
2501309991511,2cyclictest26-21ksoftirqd/108:10:221
2501307991510,4cyclictest2603554-21tr12:00:130
139099150,12rtkit-daemon3056-21PK-Backend12:00:541
2501309991410,3cyclictest3056-21PK-Backend09:12:581
250130999139,2cyclictest26-21ksoftirqd/109:05:191
2501309991311,1cyclictest3056-21PK-Backend07:35:341
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional