You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-09 - 04:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Dec 09, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5593210779,9sleep10-21swapper/119:07:281
575929683,9sleep30-21swapper/319:09:393
567428671,10sleep00-21swapper/019:08:310
572628164,13sleep20-21swapper/219:09:142
588799221,4cyclictest13070-21sed20:30:172
588399223,10cyclictest224-21systemd-journal23:06:371
588399220,21cyclictest0-21swapper/100:03:141
5889992118,2cyclictest0-21swapper/321:35:493
588399214,16cyclictest0-21swapper/123:45:141
588399210,21cyclictest0-21swapper/122:08:251
588799200,2cyclictest0-21swapper/200:15:142
588399199,3cyclictest2862-21wget21:35:161
588999182,3cyclictest141rcu_preempt23:01:073
5887991816,1cyclictest0-21swapper/219:55:182
5883991811,3cyclictest10643-21df21:40:161
5883991810,2cyclictest649-21grep21:10:241
587999184,7cyclictest10609-21munin-run19:20:010
587999180,1cyclictest0-21swapper/023:33:370
587999180,16cyclictest0-21swapper/023:55:140
588999173,12cyclictest16298-21unixbench_multi23:35:263
5889991716,1cyclictest0-21swapper/322:37:283
588999170,13cyclictest0-21swapper/321:15:153
5887991715,2cyclictest0-21swapper/223:58:442
588799170,2cyclictest25054-21grep21:50:152
588799170,17cyclictest0-21swapper/222:44:322
588799170,15cyclictest0-21swapper/223:48:362
5883991710,3cyclictest14911-21ssh22:50:211
588399170,1cyclictest0-21swapper/123:10:051
588399170,12cyclictest0-21swapper/122:10:141
588399170,0cyclictest0-21swapper/123:41:421
587999175,6cyclictest13413-21awk22:50:010
587999172,1cyclictest141rcu_preempt22:57:530
588999161,6cyclictest31585-21kworker/u8:121:10:173
5889991615,1cyclictest0-21swapper/321:32:293
588999161,1cyclictest0-21swapper/323:26:203
588999160,16cyclictest0-21swapper/322:21:363
588799163,6cyclictest224-21systemd-journal00:30:012
588799163,1cyclictest141rcu_preempt23:13:162
588799161,1cyclictest31363-21diskmemload00:24:492
588799160,16cyclictest0-21swapper/221:35:352
588799160,15cyclictest0-21swapper/221:40:152
588399160,3cyclictest0-21swapper/122:19:091
587999160,1cyclictest0-21swapper/022:09:380
588999152,10cyclictest0-21swapper/300:10:153
5889991514,1cyclictest0-21swapper/300:07:283
5887991514,1cyclictest0-21swapper/200:07:522
5887991513,1cyclictest17660-21cpuspeed_turbos22:30:172
588799150,13cyclictest0-21swapper/221:18:412
588399150,15cyclictest0-21swapper/121:56:481
5879991510,3cyclictest13332-21ls22:04:590
587999150,1cyclictest0-21swapper/022:18:580
5889991410,2cyclictest23296-21grep19:45:143
5889991410,2cyclictest15549-21wc00:20:203
588999140,1cyclictest0-21swapper/323:49:433
588799144,8cyclictest6115-21dump-pmu-power22:00:002
5887991411,2cyclictest32029-21grep20:00:282
588799140,2cyclictest0-21swapper/222:57:452
588799140,1cyclictest0-21swapper/221:49:122
588799140,14cyclictest0-21swapper/223:29:552
588799140,14cyclictest0-21swapper/222:15:332
588799140,13cyclictest0-21swapper/219:35:162
588799140,12cyclictest0-21swapper/222:00:132
5883991410,3cyclictest28693-21sed00:30:231
5883991410,3cyclictest25516-21grep22:35:201
5883991410,3cyclictest18473-21tr19:35:151
587999149,3cyclictest2242-21cron21:34:590
5879991411,2cyclictest9535-21idleruntime-cro21:40:000
5879991410,3cyclictest974-21ntpq23:25:220
5879991410,2cyclictest31180-21sed20:00:180
587999140,1cyclictest0-21swapper/022:12:170
588999139,3cyclictest29928-21idleruntime-cro19:59:593
588999139,3cyclictest21983-21ntp_kernel_err21:25:213
588999139,3cyclictest14103-21sendmail-msp00:20:003
588999139,3cyclictest12995-21sh19:25:013
588999137,4cyclictest20634-21ssh22:10:003
588999131,8cyclictest26856-21apache_volume21:00:143
5889991310,2cyclictest5412-21tr20:15:143
5889991310,2cyclictest16378-21grep19:30:193
5889991310,1cyclictest11035-21tr23:55:153
588999130,12cyclictest0-21swapper/322:10:063
588799139,4cyclictest2907-21kworker/2:023:53:572
588799139,3cyclictest7116-21tr22:45:152
588799139,3cyclictest1710-21rm00:10:232
588799139,2cyclictest14886-21grep20:35:152
5887991313,0cyclictest0-21swapper/223:01:082
5887991310,2cyclictest26692-21wc19:50:222
5887991310,2cyclictest26127-21cat23:20:222
5887991310,2cyclictest12169-21sed22:25:282
5887991310,2cyclictest0-21swapper/222:21:502
588799130,12cyclictest0-21swapper/219:40:152
588399139,3cyclictest6845-21tr22:00:141
588399139,3cyclictest20720-21head20:45:231
588399139,3cyclictest17541-21grep21:45:161
588399139,3cyclictest14671-21grep21:20:221
588399136,4cyclictest7069-21/usr/sbin/munin22:45:151
5883991310,2cyclictest15441-21cat19:30:011
5883991310,2cyclictest11509-21sort23:55:181
587999139,2cyclictest14778-21grep00:20:140
587999135,3cyclictest26397-21sendmail-msp21:00:000
587999131,6cyclictest4874-21munin-run20:15:000
5879991312,1cyclictest31363-21diskmemload22:22:100
5879991310,2cyclictest32680-21sed21:10:200
5879991310,2cyclictest19158-21ssh00:00:190
5879991310,2cyclictest15777-21grep19:30:150
588999129,2cyclictest31645-21grep23:25:003
588999129,2cyclictest26206-21head19:50:173
588999129,2cyclictest22098-21cat19:40:273
588999129,2cyclictest21628-21grep22:55:193
588999129,2cyclictest18850-21grep00:00:183
588999129,2cyclictest14252-21grep21:20:193
588999129,2cyclictest10877-21grep21:40:183
588999128,3cyclictest15440-21sed20:35:173
588999128,2cyclictest9116-21cut19:15:193
5889991210,1cyclictest6363-21grep19:10:183
5889991210,1cyclictest598-21grep20:05:163
588999120,3cyclictest224-21systemd-journal20:35:003
588999120,11cyclictest14516-21sh22:50:193
588999120,11cyclictest0-21swapper/323:08:413
588799129,2cyclictest8024-21df20:20:152
588799129,2cyclictest7485-21cat00:40:012
588799129,2cyclictest2093-21sed20:05:282
588799129,2cyclictest20780-21basename21:25:152
588799129,2cyclictest20093-21grep20:45:172
588799129,2cyclictest13019-21basename19:25:012
588799129,2cyclictest11613-21grep20:25:272
588799128,3cyclictest3708-21ssh23:05:182
588799127,4cyclictest5094-21ssh21:14:172
588799124,7cyclictest13772-21cpuspeed_turbos21:20:162
588799123,9cyclictest31363-21diskmemload23:40:112
5887991211,1cyclictest0-21swapper/222:37:282
5887991211,1cyclictest0-21swapper/221:00:152
5887991210,2cyclictest32-21ksoftirqd/223:38:352
588799120,8cyclictest224-21systemd-journal21:00:012
588399129,2cyclictest8630-21grep20:20:201
588399129,2cyclictest8393-21cut00:15:191
588399129,2cyclictest8149-21sed23:30:201
588399129,2cyclictest3219-21head00:35:271
588399129,2cyclictest28380-21ssh23:00:131
588399129,2cyclictest26176-21grep00:05:191
588399129,2cyclictest25026-21head20:55:221
588399129,2cyclictest23967-21chrt22:56:311
588399129,2cyclictest21161-21grep23:39:591
588399129,2cyclictest16771-21tr19:30:211
588399128,3cyclictest6214-21ssh00:14:081
588399128,3cyclictest14391-21cron20:35:001
587999129,2cyclictest3682-21grep23:05:180
587999129,2cyclictest32051-21basename22:40:140
587999129,2cyclictest29915-21cron20:00:000
587999129,2cyclictest27481-21grep21:00:190
587999129,2cyclictest2553-21sleep23:05:000
587999129,2cyclictest22309-21sed19:40:260
587999129,2cyclictest18254-21sed22:30:180
587999128,2cyclictest6440-21ssh21:15:150
587999127,2cyclictest25442-21cut00:05:160
5879991211,1cyclictest13-21ksoftirqd/023:38:110
5879991210,2cyclictest13-21ksoftirqd/021:20:500
5879991210,1cyclictest2485-21munin-plugin-st20:10:000
110952122,1sleep211116-21ssh23:32:202
588999119,1cyclictest29611-21cut22:15:223
588999118,2cyclictest8519-21cat22:45:263
588999118,2cyclictest7637-21grep20:20:143
588999118,2cyclictest6498-21ssh22:00:103
588999118,2cyclictest5480-21ssh23:50:293
588999118,2cyclictest4761-21ssh00:37:073
588999118,2cyclictest32078-21tr21:55:153
588999118,2cyclictest31723-21processes20:00:243
588999118,2cyclictest30205-21ssh00:31:243
588999118,2cyclictest24380-21ssh21:50:073
588999118,2cyclictest23826-21sh00:26:343
588999118,2cyclictest22369-21grep23:40:183
588999118,2cyclictest22043-21rm22:32:563
588999118,2cyclictest21069-21ssh21:47:093
588999118,2cyclictest19370-21ntp_offset19:35:203
588999118,2cyclictest18844-21chrt20:41:013
588999118,2cyclictest17563-21sh23:15:103
588999118,2cyclictest15711-21ssh23:13:413
588999118,2cyclictest15141-21ssh22:28:103
588999118,2cyclictest13987-21grep19:25:163
588999118,2cyclictest11606-21ssh23:32:473
588999116,4cyclictest26343-21cron21:00:013
588999113,6cyclictest3339-21entropy20:10:173
588999110,10cyclictest31760-21sh22:40:043
588799118,2cyclictest311-21ssh21:33:152
588799118,2cyclictest3034-21grep20:10:162
588799118,2cyclictest27173-21sh00:30:032
588799118,2cyclictest21393-21grep22:10:162
588799118,2cyclictest19486-21rm22:09:002
588799118,2cyclictest16860-21grep19:30:232
588799117,3cyclictest19932-21wc00:00:272
588399119,1cyclictest32217-21grep23:25:161
588399119,1cyclictest302422sleep120:00:001
588399118,2cyclictest9592-21grep19:15:221
588399118,2cyclictest8194-21ssh23:53:001
588399118,2cyclictest7498-21ssh21:15:211
588399118,2cyclictest6719-21cut19:10:191
588399118,2cyclictest5869-21grep20:15:161
588399118,2cyclictest4909-21ssh22:20:261
588399118,2cyclictest3353-21ssh22:42:101
588399118,2cyclictest29496-21tr21:05:161
588399118,2cyclictest29417-21tr21:30:231
588399118,2cyclictest29218-21ssh23:22:431
588399118,2cyclictest27501-21cut21:00:191
588399118,2cyclictest25812-21ssh00:28:461
588399118,2cyclictest24766-21grep19:45:261
588399118,2cyclictest24421-21ssh21:50:081
588399118,2cyclictest23092-21ssh22:33:541
588399118,2cyclictest21522-21ssh23:17:171
588399118,2cyclictest21079-21cat21:25:161
588399118,2cyclictest14500-21ssh00:20:111
588399118,2cyclictest12201-21sed22:25:281
587999118,2cyclictest8960-21cat20:20:250
587999118,2cyclictest5760-21ssh00:38:150
587999118,2cyclictest30226-21ssh21:54:000
587999118,2cyclictest29974-21sort23:45:210
587999118,2cyclictest27447-21ssh23:44:080
587999118,2cyclictest25762-21sed23:20:180
587999118,2cyclictest2427-21ssh00:10:340
587999118,2cyclictest23347-21cut19:45:150
587999118,2cyclictest23135-21ssh21:49:080
587999118,2cyclictest22241-21ssh00:25:240
587999118,2cyclictest20800-21ssh23:16:350
587999118,2cyclictest20767-21basename21:25:150
587999118,2cyclictest17732-21grep20:40:170
587999118,2cyclictest15759-21grep22:50:280
587999118,2cyclictest14577-21date20:35:010
587999118,2cyclictest12500-21ssh22:25:390
587999118,2cyclictest1198-21grep21:55:260
587999118,2cyclictest11740-21cut23:10:220
587999118,2cyclictest11028-21grep19:20:140
587999110,10cyclictest9255-21ssh00:15:260
588999108,1cyclictest22117-21expr20:50:143
588999108,1cyclictest119012sleep320:28:343
588999107,2cyclictest20286-21hddtemp_smartct20:45:193
588799108,2cyclictest17530-21expr20:40:152
588799108,1cyclictest24001-21cut19:45:182
588799108,1cyclictest218332sleep220:50:102
588799108,1cyclictest17546-21rm23:15:102
588799107,2cyclictest5185-21aten2_r7power_e20:15:132
588799107,2cyclictest29340-21wget21:05:162
588799107,2cyclictest10348-21chrt19:16:582
588799100,9cyclictest14518-21df_inode22:50:192
588399108,1cyclictest12985-21chrt19:24:571
588399107,2cyclictest27276-21chrt19:51:431
588399107,2cyclictest22893-21wc20:50:201
588399107,2cyclictest21607-21uname19:40:201
588399107,2cyclictest17031-21basename20:40:121
588399107,2cyclictest16594-21taskset20:37:521
588399107,2cyclictest10512-21sort20:25:181
588399106,3cyclictest1079-21head20:05:191
588399104,2cyclictest29922-21latency_hist20:00:001
587999108,1cyclictest8226-21apt19:15:000
587999108,1cyclictest3372-21ssh23:50:000
587999108,1cyclictest10384-21cat21:40:150
587999107,2cyclictest25670-21cat22:35:210
587999107,2cyclictest25398-21chrt19:50:090
587999107,2cyclictest19638-21cat20:45:160
587999107,2cyclictest14952-21grep20:35:160
587999107,2cyclictest10511-21cut20:25:190
587999106,3cyclictest5199-21cut20:15:140
587999106,2cyclictest29342-21cut21:05:150
587999104,3cyclictest28724-21head00:30:230
58899997,1cyclictest30128-21if_err_enp2s021:05:203
58879998,1cyclictest13284-21grep19:25:142
58879996,2cyclictest6948-21cat19:10:202
58839997,1cyclictest3838-21needreboot20:10:211
58799997,1cyclictest18115-21apt-config19:35:010
58799997,1cyclictest15562-21ls19:30:010
58799996,2cyclictest21869-21apache_volume20:50:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional