You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-27 - 12:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Mar 27, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2673328876,8sleep00-21swapper/019:08:020
2674628471,9sleep20-21swapper/219:08:112
2688328264,13sleep30-21swapper/319:09:573
2653827564,6sleep10-21swapper/119:05:291
272202580,0sleep30-21swapper/323:54:183
26979992920,4cyclictest39-21ksoftirqd/300:31:313
2697699290,19cyclictest0-21swapper/222:15:272
26979992827,1cyclictest0-21swapper/300:08:463
26979992723,3cyclictest39-21ksoftirqd/300:02:423
26979992624,1cyclictest39-21ksoftirqd/322:42:253
26979992624,1cyclictest39-21ksoftirqd/321:21:563
26979992522,3cyclictest39-21ksoftirqd/319:45:003
26979992521,1cyclictest39-21ksoftirqd/322:00:513
26979992519,1cyclictest39-21ksoftirqd/300:14:373
26979992513,4cyclictest39-21ksoftirqd/322:21:443
2697699250,25cyclictest0-21swapper/200:29:052
26979992412,5cyclictest39-21ksoftirqd/321:36:153
2697699240,24cyclictest0-21swapper/221:52:482
2697699240,24cyclictest0-21swapper/221:38:232
2697699240,24cyclictest0-21swapper/200:33:152
2697699240,23cyclictest0-21swapper/223:30:202
26979992318,2cyclictest39-21ksoftirqd/321:40:163
2697699230,23cyclictest0-21swapper/223:20:342
2697699230,23cyclictest0-21swapper/221:11:072
2697699230,0cyclictest0-21swapper/223:56:152
26979992219,3cyclictest39-21ksoftirqd/322:11:363
26979992217,3cyclictest39-21ksoftirqd/300:20:423
26979992216,2cyclictest39-21ksoftirqd/323:46:583
2697699220,22cyclictest0-21swapper/223:06:102
2697699220,22cyclictest0-21swapper/200:21:472
2697699220,1cyclictest0-21swapper/223:48:242
2696999220,1cyclictest0-21swapper/022:34:350
2697699210,20cyclictest0-21swapper/223:28:242
2697699210,20cyclictest0-21swapper/223:02:012
2696999210,20cyclictest0-21swapper/000:34:220
26979992012,2cyclictest141rcu_preempt21:15:193
26976992019,1cyclictest0-21swapper/222:45:582
2697699200,20cyclictest0-21swapper/222:23:032
2697699200,1cyclictest0-21swapper/223:13:392
2697499203,9cyclictest0-21swapper/119:50:141
2697499203,2cyclictest141rcu_preempt21:55:041
2697499200,20cyclictest0-21swapper/123:56:401
2697499200,1cyclictest19247-21diskmemload23:47:121
2697999191,7cyclictest39-21ksoftirqd/323:55:153
26979991913,2cyclictest0-21swapper/321:29:203
2697699190,19cyclictest0-21swapper/222:51:142
2697699190,19cyclictest0-21swapper/221:24:042
2697699190,19cyclictest0-21swapper/200:12:402
2697699190,18cyclictest0-21swapper/223:36:452
2697699190,18cyclictest0-21swapper/223:36:452
2697499191,1cyclictest141rcu_preempt00:37:181
2697499190,1cyclictest141rcu_preempt21:25:351
26969991919,0cyclictest0-21swapper/021:41:570
2696999190,1cyclictest0-21swapper/021:46:470
2697999189,5cyclictest39-21ksoftirqd/322:19:003
26979991817,1cyclictest39-21ksoftirqd/323:41:523
26979991817,1cyclictest0-21swapper/323:14:163
26979991816,2cyclictest39-21ksoftirqd/323:00:373
26979991811,6cyclictest39-21ksoftirqd/322:48:463
26979991810,2cyclictest141rcu_preempt23:29:193
2697999180,4cyclictest141rcu_preempt20:35:013
2697999180,2cyclictest0-21swapper/323:05:183
2697999180,14cyclictest39-21ksoftirqd/322:32:343
2697699180,18cyclictest0-21swapper/200:18:202
2697499180,1cyclictest141rcu_preempt23:07:021
2697499180,0cyclictest0-21swapper/122:27:271
2697999175,1cyclictest141rcu_preempt00:18:393
2697999172,7cyclictest39-21ksoftirqd/322:39:493
2697999171,2cyclictest141rcu_preempt21:50:123
2697999170,1cyclictest0-21swapper/322:58:203
2697699170,17cyclictest0-21swapper/221:59:222
2697699170,16cyclictest0-21swapper/222:09:382
2697499179,4cyclictest141rcu_preempt21:34:311
2697499176,10cyclictest25-21ksoftirqd/121:18:481
26974991716,1cyclictest7719-21kworker/u8:221:41:261
2697499170,2cyclictest0-21swapper/122:44:261
2697499170,1cyclictest141rcu_preempt21:13:401
2697999168,7cyclictest19247-21diskmemload00:38:493
2697999168,2cyclictest141rcu_preempt23:16:003
2697999161,13cyclictest39-21ksoftirqd/319:15:123
26979991610,4cyclictest381rcuc/321:57:493
26976991616,0cyclictest0-21swapper/223:41:112
2697699160,16cyclictest0-21swapper/223:18:422
2697699160,16cyclictest0-21swapper/222:36:382
2697699160,16cyclictest0-21swapper/221:17:232
2697699160,15cyclictest0-21swapper/222:00:022
2697499163,4cyclictest19247-21diskmemload23:50:261
2697499162,1cyclictest141rcu_preempt22:18:561
26974991614,1cyclictest25-21ksoftirqd/122:04:091
2697499161,1cyclictest19247-21diskmemload00:29:591
26969991615,1cyclictest0-21swapper/000:10:510
2697999157,1cyclictest141rcu_preempt23:24:003
2697999152,1cyclictest141rcu_preempt22:07:123
2697999150,2cyclictest0-21swapper/322:54:043
2697999150,1cyclictest0-21swapper/323:39:473
2697999150,1cyclictest0-21swapper/323:39:473
2697999150,14cyclictest0-21swapper/319:10:133
2697699156,3cyclictest14590-21apt-get22:30:142
26976991513,1cyclictest21363-21strings23:50:202
26976991512,2cyclictest30204-21grep20:25:012
2697699150,15cyclictest0-21swapper/200:04:222
2697499159,1cyclictest141rcu_preempt22:45:121
26974991511,2cyclictest25-21ksoftirqd/123:02:461
26974991511,1cyclictest19247-21diskmemload23:12:571
2697499150,3cyclictest0-21swapper/119:20:091
2696999151,8cyclictest7210-21perf20:44:590
26969991513,1cyclictest0-21swapper/021:11:070
2696999150,1cyclictest0-21swapper/023:28:060
2696999150,1cyclictest0-21swapper/000:36:130
2696999150,0cyclictest0-21swapper/023:53:360
49112140,4sleep00-21swapper/023:02:120
2697999144,1cyclictest141rcu_preempt20:15:213
2697999141,1cyclictest141rcu_preempt21:33:133
26979991411,2cyclictest24932-21grep20:10:253
2697999140,3cyclictest0-21swapper/320:50:153
2697699144,5cyclictest0-21swapper/220:00:122
26976991413,1cyclictest0-21swapper/221:30:082
2697699140,1cyclictest0-21swapper/221:48:072
2697699140,14cyclictest0-21swapper/222:14:222
2697699140,13cyclictest0-21swapper/222:44:592
26974991413,1cyclictest0-21swapper/120:31:161
26974991412,1cyclictest25-21ksoftirqd/123:26:241
2697499140,1cyclictest0-21swapper/122:36:201
2697499140,14cyclictest0-21swapper/123:18:481
2696999142,9cyclictest16163-21/usr/sbin/munin00:25:220
26969991413,1cyclictest0-21swapper/022:58:540
2697999133,2cyclictest21457-21cat21:10:243
2697999131,4cyclictest141rcu_preempt19:20:133
2697999131,2cyclictest141rcu_preempt00:25:263
26979991310,2cyclictest5821-21grep20:40:163
26979991310,2cyclictest22455-21ls20:05:283
26979991310,2cyclictest19518-21wc20:00:193
26979991310,2cyclictest17092-21wc19:55:173
26976991310,2cyclictest14846-21grep21:00:142
2697699130,13cyclictest0-21swapper/221:28:482
2697699130,11cyclictest0-21swapper/222:59:492
2697499137,2cyclictest225-21systemd-journal00:20:001
2697499136,5cyclictest0-21swapper/100:02:091
2697499132,11cyclictest22981-21latency_hist22:15:011
26974991313,0cyclictest0-21swapper/122:54:211
2697499131,2cyclictest141rcu_preempt19:25:121
2697499131,12cyclictest31927-21ssh21:35:471
26974991310,2cyclictest141rcu_preempt23:40:121
2697499130,11cyclictest0-21swapper/121:20:471
2696999138,1cyclictest0-21swapper/023:15:140
26969991311,1cyclictest0-21swapper/021:50:220
26969991310,2cyclictest30126-21cron20:25:010
26969991310,2cyclictest29298-21sh19:15:000
26969991310,2cyclictest1975-21sed20:30:280
26969991310,2cyclictest19568-21ssh00:08:430
2696999130,13cyclictest0-21swapper/022:23:340
2697999129,2cyclictest6491-21tr22:25:153
2697999129,2cyclictest20697-21grep23:30:253
2697999129,2cyclictest14020-21tr19:50:153
2697999123,6cyclictest39-21ksoftirqd/321:46:153
26979991210,1cyclictest16406-21grep21:00:263
2697999120,1cyclictest141rcu_preempt20:25:193
2697699129,2cyclictest5876-21sed19:30:242
2697699129,2cyclictest5822-21tail20:40:162
2697699129,2cyclictest5493-21cut21:40:122
2697699129,2cyclictest32568-21grep20:30:002
2697699129,2cyclictest26403-21cut20:15:192
2697699129,2cyclictest1485-21wc00:35:252
2697699129,2cyclictest14513-21awk21:00:002
2697699129,2cyclictest14512-21wc00:05:222
2697699129,2cyclictest13010-21tail19:45:242
2697699127,4cyclictest1661-21idleruntime-cro19:24:592
2697699124,5cyclictest19233-21cron21:09:592
2697699120,12cyclictest0-21swapper/222:27:042
2697499129,2cyclictest3697-21wc20:35:201
2697499129,2cyclictest14514-21ssh00:05:221
2697499125,3cyclictest25-21ksoftirqd/122:23:011
2697499125,2cyclictest29540-21/usr/sbin/munin19:15:131
2697499123,9cyclictest0-21swapper/120:20:141
2697499122,2cyclictest16872-21cron21:05:001
26974991211,1cyclictest25-21ksoftirqd/122:06:121
26974991210,1cyclictest25-21ksoftirqd/123:35:041
26974991210,1cyclictest25-21ksoftirqd/123:35:031
2697499120,8cyclictest0-21swapper/122:58:191
2697499120,0cyclictest141rcu_preempt23:34:021
2696999129,2cyclictest8478-21grep00:20:260
2696999129,2cyclictest5364-21grep19:30:200
2696999129,2cyclictest32670-21grep19:20:140
2696999129,2cyclictest30208-21cut19:15:230
2696999129,2cyclictest29921-21ssh23:55:220
2696999129,2cyclictest29584-21head21:15:210
2696999129,2cyclictest24451-21cut22:35:220
2696999129,2cyclictest22293-21cat20:05:280
2696999129,2cyclictest16817-21grep19:55:160
2696999129,2cyclictest15656-21cat22:50:000
2696999121,10cyclictest0-21swapper/023:07:100
26969991210,2cyclictest9988-21ssh23:25:000
26969991210,1cyclictest5185-21grep23:40:250
26969991210,1cyclictest13461-21cron19:49:590
2696999120,11cyclictest30711-21grep21:35:200
2697999118,2cyclictest39-21ksoftirqd/320:40:003
2697999118,2cyclictest28110-21grep20:20:153
2697999118,2cyclictest2643-21chrt19:25:163
2697999118,2cyclictest13403-21sed20:55:193
2697999118,2cyclictest11983-21cut19:45:153
2697699118,2cyclictest6897-21tr19:35:162
2697699118,2cyclictest599-21tr20:30:152
2697699118,2cyclictest30259-21cut19:15:232
2697699118,2cyclictest21567-21grep20:05:202
2697699118,2cyclictest17292-21cut19:55:202
2697699118,2cyclictest14525-21grep19:50:182
2697699118,2cyclictest10756-21tail20:50:152
2697699114,3cyclictest225-21systemd-journal20:40:012
2697499119,1cyclictest25-21ksoftirqd/120:16:161
2697499119,1cyclictest24951-21grep20:10:261
2697499119,1cyclictest21614-21head20:05:201
2697499118,2cyclictest8471-21sed00:20:261
2697499118,2cyclictest7368-21ssh23:23:031
2697499118,2cyclictest31185-21grep20:25:171
2697499118,2cyclictest21679-21ssh00:10:011
2697499118,2cyclictest19197-21tail20:00:151
2697499118,2cyclictest12197-21wc20:54:591
2697499117,2cyclictest25-21ksoftirqd/100:30:151
2697499111,2cyclictest19247-21diskmemload21:54:251
2697499110,2cyclictest0-21swapper/122:32:061
2696999119,1cyclictest6559-21cat00:00:300
2696999119,1cyclictest6506-21sh22:25:150
2696999119,1cyclictest4490-21grep21:20:140
2696999119,1cyclictest31139-21grep20:25:170
2696999119,1cyclictest22249-21strings21:30:200
2696999119,1cyclictest20746-21grep23:30:260
2696999119,1cyclictest19673-21grep20:00:190
2696999119,1cyclictest13343-21grep21:25:150
2696999119,1cyclictest13111-21grep20:55:160
2696999119,1cyclictest11130-21awk19:45:000
2696999119,1cyclictest0-21swapper/020:15:220
2696999118,2cyclictest7490-21wget20:45:150
2696999118,2cyclictest31386-21grep00:15:180
2696999118,2cyclictest29108-21grep23:35:240
2696999118,2cyclictest29108-21grep23:35:230
2696999118,2cyclictest19210-21taskset23:49:420
2696999118,2cyclictest17869-21sed22:50:250
2696999118,2cyclictest17689-21df21:05:130
2696999118,2cyclictest11526-21cat20:50:240
2697999108,1cyclictest60252sleep319:30:483
2697999107,2cyclictest7718-21tr20:45:153
2697999107,2cyclictest6731-21cut19:35:153
2697999107,2cyclictest18091-21cut21:05:173
2697699107,2cyclictest9277-21tr19:40:142
2697699107,2cyclictest7495-21grep20:45:152
2697699107,2cyclictest28309-21head19:10:192
2697699107,2cyclictest2652-21cut19:25:162
2697699102,2cyclictest225-21systemd-journal20:11:102
2697499108,2cyclictest13840-21ssh21:45:121
2697499108,2cyclictest11062-21chrt19:44:331
2697499107,2cyclictest11627-21grep19:45:131
2697499105,2cyclictest225-21systemd-journal21:00:001
2697499104,5cyclictest18701-21cat21:05:231
2697499100,9cyclictest6704-21grep19:35:151
2697499100,10cyclictest9212-21taskset20:45:461
2696999108,2cyclictest23959-21ssh22:15:370
2696999108,2cyclictest20850-21rm23:11:300
2696999108,2cyclictest18555-21ssh22:11:420
2696999108,1cyclictest7378-21grep19:35:170
2696999108,1cyclictest4484-21ssh22:01:070
2696999108,1cyclictest32222-21ssh22:40:150
2696999108,1cyclictest24985-21grep20:10:260
2696999107,2cyclictest9771-21taskset22:05:040
2696999107,2cyclictest3439-21grep20:35:180
2696999107,2cyclictest29206-21ssh21:56:440
2696999107,2cyclictest15834-21cat21:00:200
269749997,1cyclictest6760-21grep20:40:241
269749997,1cyclictest27483-21tr19:10:141
269749996,2cyclictest5003-21grep19:30:171
269749996,2cyclictest16164-21basename19:55:131
269699997,2cyclictest15600-21taskset19:52:040
269699997,1cyclictest2047-21egrep19:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional