You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 13:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Dec 07, 2025 00:43:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3067210187,9sleep30-21swapper/319:08:183
2865210087,9sleep10-21swapper/119:05:431
164828163,11sleep00-21swapper/019:05:180
291827867,6sleep20-21swapper/219:06:242
305592590,0sleep10-21swapper/123:13:341
300772300,0sleep20-21swapper/221:59:502
328799260,25cyclictest0-21swapper/221:50:172
3279992625,1cyclictest241rcuc/123:18:311
327999244,9cyclictest25-21ksoftirqd/121:52:241
328999230,22cyclictest0-21swapper/323:35:463
327999226,2cyclictest241rcuc/122:19:551
327999223,4cyclictest25-21ksoftirqd/122:13:521
3279992219,2cyclictest25-21ksoftirqd/123:00:001
327999221,5cyclictest25-21ksoftirqd/100:30:081
3279992120,1cyclictest0-21swapper/100:13:341
3279992119,1cyclictest241rcuc/119:10:171
3279992018,2cyclictest0-21swapper/123:52:531
3279992018,1cyclictest241rcuc/121:25:551
3279992016,3cyclictest28558-21diskmemload00:25:341
327999201,2cyclictest0-21swapper/121:48:261
3279992012,7cyclictest241rcuc/122:05:241
328999190,19cyclictest0-21swapper/300:21:093
328799190,18cyclictest0-21swapper/221:20:542
3279991918,1cyclictest0-21swapper/100:39:261
327999190,19cyclictest0-21swapper/122:45:071
3289991817,1cyclictest0-21swapper/323:54:293
328999181,5cyclictest0-21swapper/321:17:493
328999180,18cyclictest0-21swapper/323:43:293
328999180,18cyclictest0-21swapper/322:17:093
328999180,18cyclictest0-21swapper/300:13:293
328999180,12cyclictest0-21swapper/320:35:143
328799182,15cyclictest7566-21chrt21:41:592
3287991815,2cyclictest0-21swapper/219:25:232
328799180,18cyclictest0-21swapper/221:33:262
3279991815,3cyclictest0-21swapper/123:59:351
3279991815,2cyclictest28558-21diskmemload23:43:401
3279991814,3cyclictest19932-21turbostat19:45:001
327999180,2cyclictest0-21swapper/122:28:361
3289991716,1cyclictest0-21swapper/322:50:293
3289991716,1cyclictest0-21swapper/322:13:093
3289991711,5cyclictest0-21swapper/300:18:393
3287991716,1cyclictest29160-21kworker/2:223:54:092
3287991716,1cyclictest28395-21kworker/2:022:27:492
328799170,15cyclictest0-21swapper/221:47:302
327999179,7cyclictest0-21swapper/123:29:481
327999176,11cyclictest0-21swapper/122:39:211
327999173,14cyclictest25-21ksoftirqd/100:20:141
327999171,13cyclictest25-21ksoftirqd/121:24:211
3287991615,1cyclictest29160-21kworker/2:223:40:492
3287991614,1cyclictest23555-21kworker/2:221:17:492
327599164,6cyclictest27078-21ssh00:00:000
3289991513,2cyclictest94-21kswapd021:49:293
3287991515,0cyclictest0-21swapper/223:30:372
3287991514,1cyclictest28395-21kworker/2:022:21:292
328799150,1cyclictest0-21swapper/222:11:302
328799150,1cyclictest0-21swapper/200:05:412
328799150,14cyclictest0-21swapper/222:53:152
327999159,2cyclictest25-21ksoftirqd/121:12:261
327999155,9cyclictest11348-21ssh22:35:001
3279991513,1cyclictest29554-21ssh22:24:021
3279991512,3cyclictest25-21ksoftirqd/122:44:461
327599154,7cyclictest1291-21runrttasks23:31:280
327599150,13cyclictest0-21swapper/019:35:140
3289991410,3cyclictest8945-21grep23:45:273
3289991410,3cyclictest24619-21grep21:00:183
328799148,5cyclictest8004-21cron19:20:002
3287991413,1cyclictest0-21swapper/223:26:072
3287991413,1cyclictest0-21swapper/220:45:162
3287991410,3cyclictest6108-21sh23:20:002
3287991410,2cyclictest25725-21grep21:00:282
327999149,4cyclictest28558-21diskmemload00:02:551
327999143,8cyclictest28558-21diskmemload23:46:291
327999142,10cyclictest0-21swapper/123:36:341
3279991412,1cyclictest28558-21diskmemload23:01:401
3279991410,2cyclictest12290-21grep19:25:271
327999140,2cyclictest0-21swapper/123:30:121
327599142,7cyclictest4805-21cut20:20:140
327599142,2cyclictest224-21systemd-journal21:30:010
3275991410,2cyclictest26144-21wc22:45:220
328999139,3cyclictest2610-21tr20:15:163
328999139,3cyclictest24378-21cut22:20:153
328999139,3cyclictest23338-21wc21:55:003
328999139,3cyclictest15157-21idleruntime-cro19:35:003
328999139,2cyclictest20125-21sed23:05:183
3289991310,2cyclictest6344-21grep22:55:153
3289991310,2cyclictest39-21ksoftirqd/319:10:193
3289991310,2cyclictest28727-21cut00:25:173
3289991310,2cyclictest27114-21uniq21:05:183
3289991310,2cyclictest27113-21cron20:00:003
3289991310,2cyclictest18582-21head19:40:193
3289991310,2cyclictest12526-21grep22:35:173
328799135,7cyclictest0-21swapper/200:30:152
328799131,11cyclictest0-21swapper/220:35:142
3287991311,1cyclictest0-21swapper/220:05:142
3287991310,2cyclictest5412-21sort20:20:172
3287991310,2cyclictest3436-21grep20:15:212
328799130,3cyclictest0-21swapper/220:25:352
328799130,11cyclictest0-21swapper/200:27:062
327999139,3cyclictest27170-21wc20:00:001
327999139,3cyclictest1653-21sed20:10:301
327999139,2cyclictest8414-21grep23:20:271
327999135,8cyclictest0-21swapper/121:43:391
3279991310,2cyclictest9013-21tail19:20:171
3279991310,2cyclictest705-21grep22:50:221
3279991310,2cyclictest6642-21grep19:15:161
3279991310,2cyclictest23319-21grep19:50:181
3279991310,2cyclictest21441-21wc20:55:001
327999130,12cyclictest16908-21grep20:45:171
327999130,11cyclictest0-21swapper/121:30:121
327599139,3cyclictest25303-21cron23:10:000
327599139,3cyclictest25273-21grep21:00:230
327599139,3cyclictest24444-21grep00:20:280
327599139,3cyclictest20444-21grep19:45:160
327599139,3cyclictest18826-21ssh23:53:400
327599139,3cyclictest10895-21cron22:10:000
3275991310,2cyclictest9431-21ntpq19:20:200
3275991310,2cyclictest3972-21grep20:15:270
3275991310,2cyclictest31265-21wc20:05:260
3275991310,2cyclictest24251-21grep21:55:140
3275991310,2cyclictest22062-21cron00:20:000
3275991310,2cyclictest16920-21grep20:45:170
327599130,12cyclictest26500-21cut23:35:170
328999129,2cyclictest8763-21wc00:10:003
328999129,2cyclictest31962-21tail22:25:203
328999129,2cyclictest31003-21grep20:05:223
328999129,2cyclictest2318-21df_abs00:30:203
328999129,2cyclictest18401-21basename22:40:163
328999129,2cyclictest17359-21head20:45:193
328999129,2cyclictest15037-21grep23:25:253
328999129,2cyclictest11197-21head20:30:293
328999129,2cyclictest10547-21ssh23:22:363
328999124,3cyclictest224-21systemd-journal21:40:003
328999121,9cyclictest8950-21fschecks_time19:20:173
3289991212,0cyclictest0-21swapper/322:09:493
3289991211,1cyclictest0-21swapper/321:59:093
328999120,11cyclictest712-21head20:10:213
328799129,2cyclictest9020-21head19:20:172
328799129,2cyclictest586-21grep20:10:212
328799129,2cyclictest32597-21grep22:00:282
328799129,2cyclictest24266-21grep19:50:282
328799129,2cyclictest23248-21grep20:55:282
328799126,4cyclictest0-21swapper/223:20:172
328799126,2cyclictest0-21swapper/219:35:142
328799124,4cyclictest32-21ksoftirqd/200:05:002
328799121,4cyclictest13612-21sed22:35:262
327999129,2cyclictest5496-21grep20:20:171
327999129,2cyclictest3271-21grep00:05:221
327999129,2cyclictest31987-21grep21:35:261
327999129,2cyclictest14151-21grep19:30:211
327999128,2cyclictest19607-21cut23:05:151
327999126,4cyclictest25-21ksoftirqd/119:38:271
327599129,2cyclictest5167-21grep19:10:260
327599129,2cyclictest29835-21head21:10:200
327599129,2cyclictest18960-21cat23:05:010
327599129,2cyclictest15151-21cron19:35:000
327599129,2cyclictest15098-21sed23:25:250
327599129,2cyclictest13002-21taskset22:10:300
327599129,2cyclictest11501-21ssh21:20:200
327599128,3cyclictest13526-21tail20:35:260
327599128,2cyclictest11887-21grep22:35:140
327599126,4cyclictest0-21swapper/023:23:320
328999119,1cyclictest84462sleep321:42:563
328999118,2cyclictest6092-21tail22:30:173
328999118,2cyclictest3736-21ssh22:04:223
328999118,2cyclictest32721-21expr23:15:213
328999118,2cyclictest32119-21ssh21:11:393
328999118,2cyclictest28316-21df00:00:163
328999118,2cyclictest28193-21head20:00:193
328999118,2cyclictest27138-21grep23:10:233
328999118,2cyclictest26155-21ssh21:31:013
328999118,2cyclictest24878-21cut22:45:143
328999118,2cyclictest24264-21sed19:50:283
328999118,2cyclictest23511-21chrt20:57:403
328999118,2cyclictest22218-21taskset19:49:013
328999118,2cyclictest21226-21ssh23:30:253
328999118,2cyclictest20454-21ssh23:55:053
328999118,2cyclictest17247-21tr21:25:153
328999118,2cyclictest16417-21sh23:02:203
328999118,2cyclictest16383-21wc19:35:193
328999118,2cyclictest14583-21expr20:40:183
328999118,2cyclictest12612-21taskset19:27:543
328999111,9cyclictest0-21swapper/321:21:383
328999110,10cyclictest12124-21sh00:38:083
328799119,1cyclictest0-21swapper/223:10:182
328799118,2cyclictest8831-21head00:35:172
328799118,2cyclictest8096-21ssh22:55:332
328799118,2cyclictest7177-21sed22:30:272
328799118,2cyclictest29817-21head21:10:202
328799118,2cyclictest27505-21hostname20:00:162
328799118,2cyclictest22040-21ssh22:42:312
328799118,2cyclictest20981-21sed19:45:182
328799118,2cyclictest19923-21cron19:45:002
328799118,2cyclictest18248-21ssh21:25:222
328799118,2cyclictest17812-21ssh22:15:122
328799118,2cyclictest10733-21ssh22:09:472
328799117,3cyclictest9216-21grep00:10:182
328799117,3cyclictest30394-21tr21:35:142
328799117,3cyclictest27518-21cat21:05:212
328799116,4cyclictest11215-21ssh23:47:522
328799111,9cyclictest27637-21ssh22:46:242
328799110,1cyclictest0-21swapper/200:16:052
328799110,10cyclictest25122-21rm00:21:162
328799110,0cyclictest0-21swapper/223:55:042
327999119,1cyclictest28558-21diskmemload22:03:431
327999118,2cyclictest4602-21tr21:15:211
327999118,2cyclictest15819-21ssh00:15:141
327999118,2cyclictest10723-21tr20:30:241
327999116,4cyclictest28558-21diskmemload21:56:441
327999116,2cyclictest25-21ksoftirqd/120:20:001
327999115,4cyclictest25-21ksoftirqd/120:41:341
327999111,4cyclictest224-21systemd-journal20:26:431
327599118,2cyclictest9768-21expr20:30:170
327599118,2cyclictest8961-21cat00:35:190
327599118,2cyclictest7899-21ssh22:31:130
327599118,2cyclictest7145-21grep20:25:140
327599118,2cyclictest6809-21ssh00:08:140
327599118,2cyclictest651-21grep20:10:210
327599118,2cyclictest5155-21grep21:15:280
327599118,2cyclictest5067-21head21:40:180
327599118,2cyclictest3574-21ssh22:04:120
327599118,2cyclictest3518-21ls00:30:300
327599118,2cyclictest32698-21rm22:25:260
327599118,2cyclictest30511-21ssh23:13:320
327599118,2cyclictest29941-21ssh21:35:050
327599118,2cyclictest29934-21head00:25:300
327599118,2cyclictest28936-21cat20:00:270
327599118,2cyclictest27194-21head21:05:180
327599118,2cyclictest26690-21ssh21:31:370
327599118,2cyclictest24041-21sed22:20:000
327599118,2cyclictest22824-21head20:55:230
327599118,2cyclictest20632-21ssh21:51:570
327599118,2cyclictest19819-21awk20:50:180
327599118,2cyclictest1612-21cat23:40:190
327599118,2cyclictest1259-21grep23:15:260
327599118,2cyclictest12380-21sed19:25:270
327599118,2cyclictest10642-21rm22:58:190
327599117,3cyclictest28078-21tr00:00:150
327599117,3cyclictest10866-21cut21:45:150
327599110,8cyclictest6920-21ssh23:45:010
327599110,10cyclictest902-21ssh22:50:240
223912110,0sleep20-21swapper/223:06:472
328999108,1cyclictest20553-21mailstats20:50:243
328999107,2cyclictest8034-21uname20:25:193
328999107,2cyclictest6084-21cut20:20:233
328999107,2cyclictest5866-21chrt19:15:103
328799108,1cyclictest26161-21ssh23:35:062
328799108,1cyclictest21004-21chrt20:50:582
328799108,1cyclictest15572-21grep20:40:252
328799108,1cyclictest14960-21sh23:00:482
328799107,2cyclictest4000-21df19:10:152
328799107,2cyclictest25487-21df19:55:162
328799107,2cyclictest14912-21taskset19:31:582
328799107,2cyclictest10088-21sort20:30:192
327999107,2cyclictest30588-21hostname20:05:181
327999107,2cyclictest27545-21grep20:00:161
327999107,2cyclictest22701-21grep20:55:221
327999107,2cyclictest21388-21pmu-power19:45:231
327999105,4cyclictest25-21ksoftirqd/121:00:151
327999101,6cyclictest25-21ksoftirqd/120:35:141
327999100,5cyclictest224-21systemd-journal21:10:011
327599108,2cyclictest26630-21ssh22:20:510
327599108,1cyclictest15118-21if_enp0s31f620:40:210
327599107,2cyclictest9615-21expr00:10:200
327599107,2cyclictest7493-21date19:15:250
327599107,2cyclictest18304-21sh19:40:170
327599106,3cyclictest18534-21tr22:40:170
32759997,1cyclictest25197-21ls19:55:150
32759997,1cyclictest23177-21grep19:50:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional