You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 19:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Sun Sep 20, 2026 12:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32861121290,3sleep10-21swapper/111:20:151
139099960,76rtkit-daemon0-21swapper/007:05:150
23958427639,14sleep30-21swapper/307:07:433
23828627447,17sleep20-21swapper/207:05:092
23906826643,15sleep10-21swapper/107:05:281
3110412550,4sleep00-21swapper/010:30:140
2942682480,1sleep30-21swapper/309:40:243
3336842450,2sleep1333277-21/usr/sbin/munin11:35:151
3279542450,2sleep33056-21PK-Backend11:20:013
2703522430,2sleep00-21swapper/008:35:130
3335802410,4sleep33056-21PK-Backend11:35:133
3370832350,3sleep10-21swapper/111:45:131
239897993319,12cyclictest287740-21unin-run09:25:010
239899992614,8cyclictest1095-21runrttasks11:15:111
23989999251,22cyclictest255723-21runrttasks07:51:421
23989799250,9cyclictest3512942grep12:25:130
3297822240,4chrt0-21swapper/211:25:002
23990799241,9cyclictest322645-21perf11:05:003
2580032230,5sleep00-21swapper/008:00:120
239907992316,5cyclictest292915-21awk09:40:013
239902992319,3cyclictest324863-21expr11:10:142
239897992314,7cyclictest356106-21perf12:39:590
23989799230,21cyclictest296367-21apt-get09:49:590
139099230,16rtkit-daemon3056-21PK-Backend09:48:401
139099230,16rtkit-daemon0-21swapper/109:55:091
139099230,14rtkit-daemon0-21swapper/107:25:171
2884542220,4chrt0-21swapper/109:25:191
2710852220,3sleep10-21swapper/108:35:221
23990799220,1cyclictest3056-21PK-Backend11:10:133
23990299221,2cyclictest0-21swapper/207:40:122
239899992220,1cyclictest3056-21PK-Backend09:42:141
239899992220,1cyclictest0-21swapper/110:22:571
239899992220,1cyclictest0-21swapper/109:14:571
239899992220,1cyclictest0-21swapper/109:07:461
239899992219,2cyclictest3056-21PK-Backend07:21:411
239899992214,6cyclictest332271-21ntp_kernel_err11:30:181
239899992210,10cyclictest284882-21diskstats09:15:141
23989999220,21cyclictest0-21swapper/112:18:491
239897992220,1cyclictest0-21swapper/012:31:490
239897992220,1cyclictest0-21swapper/012:04:110
239897992220,1cyclictest0-21swapper/009:00:390
239897992220,1cyclictest0-21swapper/008:50:110
239897992220,1cyclictest0-21swapper/008:33:350
239897992220,1cyclictest0-21swapper/007:28:250
239897992211,9cyclictest266423-21unin-run08:25:010
23989799220,1cyclictest0-21swapper/011:59:110
139099220,14rtkit-daemon0-21swapper/108:52:531
239907992119,1cyclictest3056-21PK-Backend07:58:173
239907992118,2cyclictest3056-21PK-Backend12:25:443
23990799211,2cyclictest287953-21/usr/sbin/munin09:25:163
23990799210,20cyclictest3056-21PK-Backend10:28:573
23990799210,1cyclictest3056-21PK-Backend08:26:083
23990799210,19cyclictest3056-21PK-Backend11:50:503
23990799210,19cyclictest3056-21PK-Backend10:36:413
23990799210,19cyclictest3056-21PK-Backend09:00:293
239902992110,4cyclictest319173-21grep10:55:002
239902992110,4cyclictest319173-21grep10:54:592
239899992119,1cyclictest3056-21PK-Backend07:11:461
239899992119,1cyclictest0-21swapper/111:40:031
239899992119,1cyclictest0-21swapper/111:13:471
239899992119,1cyclictest0-21swapper/111:00:001
239899992119,1cyclictest0-21swapper/109:50:261
239899992119,1cyclictest0-21swapper/109:00:371
239899992119,1cyclictest0-21swapper/108:42:451
239899992119,1cyclictest0-21swapper/108:08:531
239899992119,1cyclictest0-21swapper/107:43:031
239899992115,5cyclictest243071-21sshd07:15:201
23989999210,20cyclictest0-21swapper/112:33:151
23989999210,20cyclictest0-21swapper/110:31:411
23989999210,19cyclictest0-21swapper/112:09:371
239897992119,1cyclictest3056-21PK-Backend10:25:420
239897992119,1cyclictest3056-21PK-Backend07:36:160
239897992119,1cyclictest0-21swapper/010:57:260
239897992119,1cyclictest0-21swapper/010:45:010
239897992119,1cyclictest0-21swapper/010:23:400
239897992119,1cyclictest0-21swapper/008:28:420
239897992114,6cyclictest328369-21expr11:20:110
239897992112,7cyclictest339959-21cron11:55:000
23989799210,20cyclictest337434-21strings11:45:190
23989799210,20cyclictest3056-21PK-Backend10:50:430
23989799210,20cyclictest3056-21PK-Backend10:50:430
23989799210,20cyclictest0-21swapper/010:16:420
23989799210,20cyclictest0-21swapper/008:11:590
23989799210,20cyclictest0-21swapper/007:51:430
23989799210,20cyclictest0-21swapper/007:31:500
139099210,4rtkit-daemon1389-21rtkit-daemon12:08:310
139099210,15rtkit-daemon3056-21PK-Backend07:37:521
139099210,15rtkit-daemon0-21swapper/107:45:241
23990799205,5cyclictest299500-21/usr/sbin/munin09:55:213
239907992018,1cyclictest3056-21PK-Backend12:15:433
239907992018,1cyclictest3056-21PK-Backend10:08:563
239907992017,2cyclictest3056-21PK-Backend10:19:093
23990799201,5cyclictest341337-21awk11:55:213
239907992015,3cyclictest240107-21apt07:10:123
239907992012,6cyclictest285977-21cron09:19:593
239907992011,7cyclictest436-20systemd-journal12:05:003
239907992011,7cyclictest305293-21perf10:15:003
239907992011,6cyclictest266407-21basename08:24:593
23990799200,1cyclictest3056-21PK-Backend11:47:113
23990799200,19cyclictest3056-21PK-Backend10:40:333
23990799200,19cyclictest3056-21PK-Backend10:20:113
23990799200,19cyclictest3056-21PK-Backend08:56:233
23990799200,19cyclictest3056-21PK-Backend08:37:443
23990799200,19cyclictest3056-21PK-Backend08:31:023
23990799200,19cyclictest3056-21PK-Backend08:04:183
23990799200,19cyclictest0-21swapper/311:33:403
23990799200,19cyclictest0-21swapper/309:51:333
23990799200,19cyclictest0-21swapper/307:16:063
23990299206,13cyclictest269851-21cron08:34:592
239902992018,1cyclictest0-21swapper/209:45:172
239902992014,4cyclictest886-21NetworkManager07:15:002
239902992012,6cyclictest279504-21ntpq09:00:192
239899992018,1cyclictest3056-21PK-Backend12:12:141
239899992018,1cyclictest3056-21PK-Backend10:58:591
239899992018,1cyclictest0-21swapper/111:50:581
239899992018,1cyclictest0-21swapper/110:39:251
239899992018,1cyclictest0-21swapper/108:11:571
239899992012,6cyclictest351514-21df12:25:131
23989999200,19cyclictest3056-21PK-Backend12:35:531
23989999200,19cyclictest3056-21PK-Backend12:01:111
23989999200,19cyclictest3056-21PK-Backend08:24:481
23989999200,19cyclictest0-21swapper/110:40:281
23989999200,19cyclictest0-21swapper/110:18:511
23989999200,19cyclictest0-21swapper/108:26:511
239897992018,1cyclictest0-21swapper/012:24:120
239897992018,1cyclictest0-21swapper/011:43:280
239897992018,1cyclictest0-21swapper/011:33:360
239897992018,1cyclictest0-21swapper/010:36:170
239897992018,1cyclictest0-21swapper/009:59:470
239897992018,1cyclictest0-21swapper/009:51:010
239897992018,1cyclictest0-21swapper/009:41:000
239897992018,1cyclictest0-21swapper/009:16:130
239897992015,3cyclictest251228-21cpuspeed_turbos07:40:120
239897992013,5cyclictest327825-21cron11:20:010
23989799200,1cyclictest0-21swapper/008:19:550
23989799200,19cyclictest3056-21PK-Backend10:11:490
23989799200,19cyclictest0-21swapper/011:35:240
23989799200,19cyclictest0-21swapper/009:37:570
23989799200,19cyclictest0-21swapper/007:49:550
23990799193,14cyclictest295558-21grep09:45:183
239907991917,1cyclictest3056-21PK-Backend11:08:253
239907991917,1cyclictest3056-21PK-Backend09:06:373
239907991917,1cyclictest3056-21PK-Backend08:51:153
239907991917,1cyclictest0-21swapper/307:25:393
239907991917,1cyclictest0-21swapper/307:21:593
239907991916,2cyclictest3056-21PK-Backend08:40:493
239907991916,2cyclictest3056-21PK-Backend07:34:443
239907991915,3cyclictest42-21ksoftirqd/307:36:343
239907991914,3cyclictest291186-21munin-run09:35:013
239907991912,6cyclictest355379-21ntp_states12:35:213
23990799190,7cyclictest320200-21/usr/sbin/munin10:55:203
23990799190,1cyclictest3056-21PK-Backend11:28:123
23990799190,1cyclictest3056-21PK-Backend11:21:093
23990799190,1cyclictest3056-21PK-Backend08:06:213
23990799190,1cyclictest0-21swapper/312:12:183
23990799190,1cyclictest0-21swapper/308:18:113
23990799190,18cyclictest3056-21PK-Backend12:34:573
239902991918,1cyclictest0-21swapper/212:00:162
239902991917,1cyclictest0-21swapper/207:25:112
239902991913,5cyclictest299037-21smartctl09:55:142
239902991913,5cyclictest266899-21expr08:25:142
23990299190,1cyclictest0-21swapper/211:30:012
23990299190,1cyclictest0-21swapper/208:59:412
23990299190,18cyclictest0-21swapper/211:07:032
239899991917,1cyclictest0-21swapper/112:21:261
239899991917,1cyclictest0-21swapper/110:54:061
239899991917,1cyclictest0-21swapper/110:54:061
239899991912,5cyclictest436-20systemd-journal07:59:551
239899991911,6cyclictest292444-21sed09:35:231
23989999190,1cyclictest3056-21PK-Backend09:21:191
23989999190,1cyclictest0-21swapper/111:07:181
23989999190,1cyclictest0-21swapper/108:02:581
23989999190,18cyclictest0-21swapper/111:29:071
23989999190,18cyclictest0-21swapper/110:48:121
23989999190,18cyclictest0-21swapper/110:27:501
23989999190,18cyclictest0-21swapper/107:31:481
23989799192,12cyclictest271865-21apt08:40:130
239897991917,1cyclictest0-21swapper/011:11:110
239897991917,1cyclictest0-21swapper/010:05:550
239897991911,6cyclictest301229-21python310:00:210
23989799190,1cyclictest0-21swapper/011:06:310
23989799190,1cyclictest0-21swapper/010:42:590
23989799190,1cyclictest0-21swapper/009:08:500
139099190,13rtkit-daemon3056-21PK-Backend10:14:281
23990799185,10cyclictest312240-21sh10:35:013
239907991816,1cyclictest3056-21PK-Backend07:52:153
239907991816,1cyclictest0-21swapper/310:48:163
239907991813,3cyclictest344654-21netstat12:05:183
239907991813,3cyclictest251150-21cpuspeed07:40:113
239907991810,6cyclictest286994-21ntpq09:20:183
239902991814,3cyclictest34-21ksoftirqd/212:15:012
239902991812,5cyclictest317263-21seq10:48:022
239902991811,6cyclictest344970-21expr12:05:222
239902991811,6cyclictest289531-21sed09:30:002
239902991811,5cyclictest290872-21sshd09:31:192
239899991816,1cyclictest0-21swapper/111:56:581
239899991816,1cyclictest0-21swapper/108:15:011
239897991816,1cyclictest0-21swapper/011:02:060
239897991813,4cyclictest283316-21wc09:10:200
239897991812,5cyclictest290055-21cat09:30:130
239897991810,6cyclictest348388-21ntpq12:15:180
139099180,12rtkit-daemon0-21swapper/108:31:561
239907991715,1cyclictest3056-21PK-Backend10:00:593
239907991715,1cyclictest3056-21PK-Backend08:13:173
239907991715,1cyclictest0-21swapper/307:47:013
23990799170,4cyclictest349414-21awk12:20:113
23990799170,15cyclictest317707-21apt10:50:123
23990799170,15cyclictest317707-21apt10:50:123
23990299179,6cyclictest436-20systemd-journal08:25:002
239902991715,1cyclictest0-21swapper/211:31:372
239902991713,3cyclictest270288-21grep08:35:132
239902991713,2cyclictest334760-21idleruntime-cro11:40:012
239902991712,3cyclictest320890-21cron11:00:002
239902991712,3cyclictest242276-21cstates07:15:032
239902991711,5cyclictest292979-21cut09:40:012
239902991711,5cyclictest275615-21expr08:50:132
239902991710,6cyclictest351855-21awk12:25:192
239902991710,5cyclictest308765-21ls10:25:002
239899991715,1cyclictest0-21swapper/110:00:391
239899991710,5cyclictest277311-21ls08:55:141
239897991714,2cyclictest3056-21PK-Backend07:19:260
239907991614,1cyclictest3056-21PK-Backend09:10:433
239907991613,2cyclictest3056-21PK-Backend11:42:443
23990299169,5cyclictest340850-21expr11:55:132
23990299169,5cyclictest309650-21ls10:25:172
23990299169,5cyclictest247537-21awk07:30:112
23990299168,6cyclictest321904-21cat11:00:202
23990299164,4cyclictest34-21ksoftirqd/208:00:122
23990299163,9cyclictest244064-21apt07:20:122
23990299162,11cyclictest293179-21apt09:40:132
239902991612,3cyclictest355483-21rt-features12:35:222
239902991612,3cyclictest280969-21tr09:05:132
239902991612,3cyclictest261197-21idleruntime-cro08:10:002
239902991611,4cyclictest34-21ksoftirqd/211:49:592
239902991611,4cyclictest310940-21cut10:30:132
239902991611,3cyclictest305303-21latency_hist10:15:012
239902991610,5cyclictest306565-21cat10:15:232
239902991610,5cyclictest285825-21sshd09:18:192
239902991610,4cyclictest297203-21cat09:50:122
23989999169,5cyclictest273839-21expr08:45:131
23989999168,6cyclictest289930-21expr09:30:131
239899991614,1cyclictest0-21swapper/110:05:461
23989799169,5cyclictest256563-21cat07:55:140
23989799168,6cyclictest277060-21apt08:55:130
239897991612,3cyclictest274461-21python308:45:210
239897991612,3cyclictest245532-21latency_hist07:24:590
239897991610,5cyclictest260733-21sed08:05:230
23990299159,4cyclictest257231-21sed07:55:232
23990299158,6cyclictest339478-21wc11:50:222
23990299158,5cyclictest302109-21grep10:05:122
23990299157,6cyclictest0-21swapper/211:18:532
239902991513,1cyclictest0-21swapper/208:42:382
239902991513,1cyclictest0-21swapper/207:52:492
239902991510,4cyclictest34-21ksoftirqd/212:18:432
23990299150,14cyclictest0-21swapper/209:21:592
23990299150,14cyclictest0-21swapper/209:14:562
23989799158,6cyclictest288228-21expr09:25:150
23989799158,5cyclictest330187-21expr11:25:130
23990799149,4cyclictest3056-21PK-Backend08:46:193
23990299149,3cyclictest1095-21runrttasks10:36:502
23990299148,5cyclictest253431-21cat07:45:182
23990299146,6cyclictest263623-21cat08:15:172
239902991412,1cyclictest0-21swapper/208:49:272
23990299141,10cyclictest250462-21file07:36:192
239902991410,3cyclictest314215-21/usr/sbin/munin10:40:192
23990299140,1cyclictest0-21swapper/212:30:312
23990299140,1cyclictest0-21swapper/211:41:302
23989799140,6cyclictest3056-21PK-Backend12:10:140
23990299138,3cyclictest0-21swapper/210:00:112
23990299136,5cyclictest349655-21expr12:20:112
23990299132,10cyclictest262348-21switchtime08:10:232
23989799120,11cyclictest3056-21PK-Backend07:11:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional